[GitHub] [airflow] milton0825 commented on issue #5126: [AIRFLOW-4342] Use @cached_property instead of re-implementing it each time
milton0825 commented on issue #5126: [AIRFLOW-4342] Use @cached_property instead of re-implementing it each time URL: https://github.com/apache/airflow/pull/5126#issuecomment-484365457 LGTM. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] milton0825 opened a new pull request #5130: Bigquery extra links
milton0825 opened a new pull request #5130: Bigquery extra links
URL: https://github.com/apache/airflow/pull/5130
Make sure you have checked _all_ steps below.
### Jira
- [ ] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-XXX
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
- In case you are proposing a fundamental code change, you need to create
an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
- In case you are adding a dependency, check if the license complies with
the [ASF 3rd Party License
Policy](https://www.apache.org/legal/resolved.html#category-x).
### Description
- [ ] Here are some details about my PR, including screenshots of any UI
changes:
### Tests
- [ ] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [ ] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [ ] In case of new functionality, my PR adds documentation that describes
how to use it.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
### Code Quality
- [ ] Passes `flake8`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Created] (AIRFLOW-4348) Add a GCP console link in BigQueryOperator
Chao-Han Tsai created AIRFLOW-4348: -- Summary: Add a GCP console link in BigQueryOperator Key: AIRFLOW-4348 URL: https://issues.apache.org/jira/browse/AIRFLOW-4348 Project: Apache Airflow Issue Type: New Feature Reporter: Chao-Han Tsai Assignee: Chao-Han Tsai Add a GCP console link in BigQueryOperator -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-3870) SFTPOperator to push filepath into xcom
[ https://issues.apache.org/jira/browse/AIRFLOW-3870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jack updated AIRFLOW-3870: -- Fix Version/s: 1.10.4 > SFTPOperator to push filepath into xcom > --- > > Key: AIRFLOW-3870 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3870 > Project: Apache Airflow > Issue Type: Improvement >Reporter: jack >Priority: Minor > Fix For: 1.10.4 > > > push filepath into xcom and update info messages. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-4347) airflow webserver can not load webserver_config.py
[
https://issues.apache.org/jira/browse/AIRFLOW-4347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16820733#comment-16820733
]
ASF GitHub Bot commented on AIRFLOW-4347:
-
sumous commented on pull request #5129: [AIRFLOW-4347] fix WEBSERVER_CONFIG
when AIRFLOW_HOME=./
URL: https://github.com/apache/airflow/pull/5129
Make sure you have checked _all_ steps below.
### Jira
- [ ] My PR addresses the following
[AIRFLOW-4347](https://issues.apache.org/jira/browse/AIRFLOW-4347) issues and
references them in the PR title.
### Description
- [ ] Here are some details about my PR, including screenshots of any UI
changes:
I set AIRFLOW_HOME=./
command **airflow initdb** is ok.
And command **airflow webserver** got this:
```
/Users/guoguo/Downloads/env_tmp/lib/python3.6/site-packages/flask_sqlalchemy/__init__.py:774:
UserWarning: Neither SQLALCHEMY_DATABASE_URI nor SQLALCHEMY_BINDS is set.
Defaulting SQLALCHEMY_DATABASE_URI to "sqlite:///:memory:".
'Neither SQLALCHEMY_DATABASE_URI nor SQLALCHEMY_BINDS is set. '
```
Airflow v1.10.3 code load RBAC WEBSERVER_CONFIG config in
https://github.com/apache/airflow/blob/v1-10-stable/airflow/www_rbac/app.py#L51.
When **AIRFLOW_HOME=./**, then
**settings.WEBSERVER_CONFIG=.//webserver_config.py**. It will got the error
path of RBAC_CODE_PATH+.//webserver_config.py
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> airflow webserver can not load webserver_config.py
> --
>
> Key: AIRFLOW-4347
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4347
> Project: Apache Airflow
> Issue Type: Bug
> Components: webserver
>Affects Versions: 1.10.3
>Reporter: sumous
>Assignee: sumous
>Priority: Minor
> Fix For: 1.10.4
>
>
> I set AIRFLOW_HOME=./
> command *airflow initdb* is ok.
> And command *airflow webserver* got this:
> {code:shell}
> /Users/guoguo/Downloads/env_tmp/lib/python3.6/site-packages/flask_sqlalchemy/__init__.py:774:
> UserWarning: Neither SQLALCHEMY_DATABASE_URI nor SQLALCHEMY_BINDS is set.
> Defaulting SQLALCHEMY_DATABASE_URI to "sqlite:///:memory:".
> 'Neither SQLALCHEMY_DATABASE_URI nor SQLALCHEMY_BINDS is set. '
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Closed] (AIRFLOW-3169) Indicate in the main UI if the scheduler is NOT working.
[ https://issues.apache.org/jira/browse/AIRFLOW-3169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jack closed AIRFLOW-3169. - Resolution: Duplicate > Indicate in the main UI if the scheduler is NOT working. > > > Key: AIRFLOW-3169 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3169 > Project: Apache Airflow > Issue Type: Improvement >Affects Versions: 1.10.0 >Reporter: jack >Priority: Major > Labels: observability > > I came to work today and took a look at Airflow UI. > Everything was green (success) - it took me a while to notice that the dates > of tasks are from Thursday. The scheduler was offline whole weekend. > Only when I restarted the scheduler tasks has began to run. I don't know why > the scheduler stopped but I think it would be great if the UI would indicate > in the main screen when the scheduler is offline. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-4328) Link to view task instances from Pool broken
[
https://issues.apache.org/jira/browse/AIRFLOW-4328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16820718#comment-16820718
]
ASF GitHub Bot commented on AIRFLOW-4328:
-
XD-DENG commented on pull request #5124: [AIRFLOW-4328] Fix link to task
instances from Pool page
URL: https://github.com/apache/airflow/pull/5124
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Link to view task instances from Pool broken
>
>
> Key: AIRFLOW-4328
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4328
> Project: Apache Airflow
> Issue Type: Improvement
> Components: pools
>Affects Versions: 1.10.3
>Reporter: t oo
>Assignee: Ash Berlin-Taylor
>Priority: Major
> Fix For: 1.10.4
>
>
> create some pool names (using cli) with 70 or 80 character length
>
> Q 1. UI does not allow creating > 50 length but why does cli?
>
> [https://domain:8080/admin/pool/]
> click on one of the pool names listed (link is cut to 50 char name:
> [https://domain:8080/admin/airflow/task?flt1_pool_equals=qjfdal_CRCE_INTERCONNECTION_FORECAST_TNC_EJFLSA_LP)]
> get below error:
> [2019-04-15 22:40:12,554] \{app.py:1761} ERROR - Exception on
> /admin/airflow/task [GET]
> Traceback (most recent call last):
> File "/home/ec2-user/venv/local/lib64/python2.7/site-packages/flask/app.py",
> line 2292, in wsgi_app
> response = self.full_dispatch_request()
> File "/home/ec2-user/venv/local/lib64/python2.7/site-packages/flask/app.py",
> line 1815, in full_dispatch_request
> rv = self.handle_user_exception(e)
> File "/home/ec2-user/venv/local/lib64/python2.7/site-packages/flask/app.py",
> line 1718, in handle_user_exception
> reraise(exc_type, exc_value, tb)
> File "/home/ec2-user/venv/local/lib64/python2.7/site-packages/flask/app.py",
> line 1813, in full_dispatch_request
> rv = self.dispatch_request()
> File "/home/ec2-user/venv/local/lib64/python2.7/site-packages/flask/app.py",
> line 1799, in dispatch_request
> return self.view_functions[rule.endpoint](**req.view_args)
> File
> "/home/ec2-user/venv/local/lib64/python2.7/site-packages/flask_admin/base.py",
> line 69, in inner
> return self._run_view(f, *args, **kwargs)
> File
> "/home/ec2-user/venv/local/lib64/python2.7/site-packages/flask_admin/base.py",
> line 368, in _run_view
> return fn(self, *args, **kwargs)
> File
> "/home/ec2-user/venv/local/lib64/python2.7/site-packages/flask_login/utils.py",
> line 261, in decorated_view
> return func(*args, **kwargs)
> File
> "/home/ec2-user/venv/local/lib/python2.7/site-packages/airflow/www/utils.py",
> line 275, in wrapper
> return f(*args, **kwargs)
> File
> "/home/ec2-user/venv/local/lib/python2.7/site-packages/airflow/www/views.py",
> line 893, in task
> dttm = pendulum.parse(execution_date)
> File
> "/home/ec2-user/venv/local/lib64/python2.7/site-packages/pendulum/parser.py",
> line 75, in parse
> return Parser(**options).parse(text)
> File
> "/home/ec2-user/venv/local/lib64/python2.7/site-packages/pendulum/parser.py",
> line 31, in parse
> parsed = super(Parser, self).parse(text)
> File
> "/home/ec2-user/venv/local/lib64/python2.7/site-packages/pendulum/parsing/parser.py",
> line 297, in parse
> return self.normalize(self._parse(text))
> File
> "/home/ec2-user/venv/local/lib64/python2.7/site-packages/pendulum/parsing/parser.py",
> line 342, in _parse
> parsed = self.parse_common(text)
> File
> "/home/ec2-user/venv/local/lib64/python2.7/site-packages/pendulum/parsing/parser.py",
> line 104, in parse_common
> m = self.COMMON.match(text)
> TypeError: expected string or buffer
> If click 'edit' it shows full 80chars in Description but cut 50chars in Pool
>
> Q2. why this large error? i believe its related to the fact that this is new
> airflow instance where no DAGs/tasks have run yet so it is expecting some
> execution_date value but none is there. Confirmed this as creating a new 3
> letter pool name still gives the large error after clicking on the name of
> the created pool
> Q3. why limit to 50 length at all?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AIRFLOW-4328) Link to view task instances from Pool broken
[
https://issues.apache.org/jira/browse/AIRFLOW-4328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16820719#comment-16820719
]
ASF subversion and git services commented on AIRFLOW-4328:
--
Commit c61321fca9ac663b3f6660b0d35548d642a0cd8e in airflow's branch
refs/heads/v1-10-stable from Ash Berlin-Taylor
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=c61321f ]
[AIRFLOW-4328] Fix link to task instances from Pool page (#5124)
We want to link to the list of task instances that ran against this
pool, not to a specific task. (The link as it was wouldn't work, it was
missing lots of parameters, and wasn't meant to be. I messed up a
previous commit)
> Link to view task instances from Pool broken
>
>
> Key: AIRFLOW-4328
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4328
> Project: Apache Airflow
> Issue Type: Improvement
> Components: pools
>Affects Versions: 1.10.3
>Reporter: t oo
>Assignee: Ash Berlin-Taylor
>Priority: Major
> Fix For: 1.10.4
>
>
> create some pool names (using cli) with 70 or 80 character length
>
> Q 1. UI does not allow creating > 50 length but why does cli?
>
> [https://domain:8080/admin/pool/]
> click on one of the pool names listed (link is cut to 50 char name:
> [https://domain:8080/admin/airflow/task?flt1_pool_equals=qjfdal_CRCE_INTERCONNECTION_FORECAST_TNC_EJFLSA_LP)]
> get below error:
> [2019-04-15 22:40:12,554] \{app.py:1761} ERROR - Exception on
> /admin/airflow/task [GET]
> Traceback (most recent call last):
> File "/home/ec2-user/venv/local/lib64/python2.7/site-packages/flask/app.py",
> line 2292, in wsgi_app
> response = self.full_dispatch_request()
> File "/home/ec2-user/venv/local/lib64/python2.7/site-packages/flask/app.py",
> line 1815, in full_dispatch_request
> rv = self.handle_user_exception(e)
> File "/home/ec2-user/venv/local/lib64/python2.7/site-packages/flask/app.py",
> line 1718, in handle_user_exception
> reraise(exc_type, exc_value, tb)
> File "/home/ec2-user/venv/local/lib64/python2.7/site-packages/flask/app.py",
> line 1813, in full_dispatch_request
> rv = self.dispatch_request()
> File "/home/ec2-user/venv/local/lib64/python2.7/site-packages/flask/app.py",
> line 1799, in dispatch_request
> return self.view_functions[rule.endpoint](**req.view_args)
> File
> "/home/ec2-user/venv/local/lib64/python2.7/site-packages/flask_admin/base.py",
> line 69, in inner
> return self._run_view(f, *args, **kwargs)
> File
> "/home/ec2-user/venv/local/lib64/python2.7/site-packages/flask_admin/base.py",
> line 368, in _run_view
> return fn(self, *args, **kwargs)
> File
> "/home/ec2-user/venv/local/lib64/python2.7/site-packages/flask_login/utils.py",
> line 261, in decorated_view
> return func(*args, **kwargs)
> File
> "/home/ec2-user/venv/local/lib/python2.7/site-packages/airflow/www/utils.py",
> line 275, in wrapper
> return f(*args, **kwargs)
> File
> "/home/ec2-user/venv/local/lib/python2.7/site-packages/airflow/www/views.py",
> line 893, in task
> dttm = pendulum.parse(execution_date)
> File
> "/home/ec2-user/venv/local/lib64/python2.7/site-packages/pendulum/parser.py",
> line 75, in parse
> return Parser(**options).parse(text)
> File
> "/home/ec2-user/venv/local/lib64/python2.7/site-packages/pendulum/parser.py",
> line 31, in parse
> parsed = super(Parser, self).parse(text)
> File
> "/home/ec2-user/venv/local/lib64/python2.7/site-packages/pendulum/parsing/parser.py",
> line 297, in parse
> return self.normalize(self._parse(text))
> File
> "/home/ec2-user/venv/local/lib64/python2.7/site-packages/pendulum/parsing/parser.py",
> line 342, in _parse
> parsed = self.parse_common(text)
> File
> "/home/ec2-user/venv/local/lib64/python2.7/site-packages/pendulum/parsing/parser.py",
> line 104, in parse_common
> m = self.COMMON.match(text)
> TypeError: expected string or buffer
> If click 'edit' it shows full 80chars in Description but cut 50chars in Pool
>
> Q2. why this large error? i believe its related to the fact that this is new
> airflow instance where no DAGs/tasks have run yet so it is expecting some
> execution_date value but none is there. Confirmed this as creating a new 3
> letter pool name still gives the large error after clicking on the name of
> the created pool
> Q3. why limit to 50 length at all?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] [airflow] XD-DENG merged pull request #5124: [AIRFLOW-4328] Fix link to task instances from Pool page
XD-DENG merged pull request #5124: [AIRFLOW-4328] Fix link to task instances from Pool page URL: https://github.com/apache/airflow/pull/5124 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] jmcarp commented on a change in pull request #4821: [AIRFLOW-3998] Use nested commands in cli.
jmcarp commented on a change in pull request #4821: [AIRFLOW-3998] Use nested
commands in cli.
URL: https://github.com/apache/airflow/pull/4821#discussion_r276511864
##
File path: airflow/bin/cli.py
##
@@ -258,38 +259,57 @@ def delete_dag(args):
print("Bail.")
+def _tabulate_pools(pools):
+return "\n%s" % tabulate(pools, ['Pool', 'Slots', 'Description'],
+ tablefmt="fancy_grid")
+
+
@cli_utils.action_logging
-def pool(args):
+def pool_list(args):
log = LoggingMixin().log
+pools = api_client.get_pools()
+log.info(_tabulate_pools(pools=pools))
-def _tabulate(pools):
-return "\n%s" % tabulate(pools, ['Pool', 'Slots', 'Description'],
- tablefmt="fancy_grid")
-try:
-imp = getattr(args, 'import')
-if args.get is not None:
-pools = [api_client.get_pool(name=args.get)]
-elif args.set:
-pools = [api_client.create_pool(name=args.set[0],
-slots=args.set[1],
-description=args.set[2])]
-elif args.delete:
-pools = [api_client.delete_pool(name=args.delete)]
-elif imp:
-if os.path.exists(imp):
-pools = pool_import_helper(imp)
-else:
-print("Missing pools file.")
-pools = api_client.get_pools()
-elif args.export:
-pools = pool_export_helper(args.export)
-else:
-pools = api_client.get_pools()
-except (AirflowException, IOError) as err:
-log.error(err)
+@cli_utils.action_logging
Review comment:
Thanks, I dropped the decorator from list/get functions that I added here. I
see it's being used for other functions that don't appear to change state--I
can clean those up as well or save for another patch.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] jmcarp commented on a change in pull request #4821: [AIRFLOW-3998] Use nested commands in cli.
jmcarp commented on a change in pull request #4821: [AIRFLOW-3998] Use nested
commands in cli.
URL: https://github.com/apache/airflow/pull/4821#discussion_r276511890
##
File path: airflow/bin/cli.py
##
@@ -2186,73 +2140,217 @@ class CLIFactory(object):
'autoscale': Arg(
('-a', '--autoscale'),
help="Minimum and Maximum number of worker to autoscale"),
-
}
subparsers = (
{
-'func': backfill,
-'help': "Run subsections of a DAG for a specified date range. "
-"If reset_dag_run option is used,"
-" backfill will first prompt users whether airflow "
-"should clear all the previous dag_run and task_instances "
-"within the backfill date range. "
-"If rerun_failed_tasks is used, backfill "
-"will auto re-run the previous failed task instances"
-" within the backfill date range.",
-'args': (
-'dag_id', 'task_regex', 'start_date', 'end_date',
-'mark_success', 'local', 'donot_pickle',
-'bf_ignore_dependencies', 'bf_ignore_first_depends_on_past',
-'subdir', 'pool', 'delay_on_limit', 'dry_run', 'verbose',
'conf',
-'reset_dag_run', 'rerun_failed_tasks', 'run_backwards'
-)
-}, {
-'func': list_dag_runs,
-'help': "List dag runs given a DAG id. If state option is given,
it will only"
-"search for all the dagruns with the given state. "
-"If no_backfill option is given, it will filter out"
-"all backfill dagruns for given dag id.",
-'args': (
-'dag_id', 'no_backfill', 'state'
-)
-}, {
-'func': list_tasks,
-'help': "List the tasks within a DAG",
-'args': ('dag_id', 'tree', 'subdir'),
-}, {
-'func': list_jobs,
-'help': "List the jobs",
-'args': ('dag_id_opt', 'state', 'limit'),
-}, {
-'func': clear,
-'help': "Clear a set of task instance, as if they never ran",
-'args': (
-'dag_id', 'task_regex', 'start_date', 'end_date', 'subdir',
-'upstream', 'downstream', 'no_confirm', 'only_failed',
-'only_running', 'exclude_subdags', 'exclude_parentdag',
'dag_regex'),
-}, {
-'func': pause,
-'help': "Pause a DAG",
-'args': ('dag_id', 'subdir'),
-}, {
-'func': unpause,
-'help': "Resume a paused DAG",
-'args': ('dag_id', 'subdir'),
-}, {
-'func': trigger_dag,
-'help': "Trigger a DAG run",
-'args': ('dag_id', 'subdir', 'run_id', 'conf', 'exec_date'),
+'help': 'DAGs',
Review comment:
Updated.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] jmcarp commented on a change in pull request #4821: [AIRFLOW-3998] Use nested commands in cli.
jmcarp commented on a change in pull request #4821: [AIRFLOW-3998] Use nested
commands in cli.
URL: https://github.com/apache/airflow/pull/4821#discussion_r276511788
##
File path: airflow/bin/cli.py
##
@@ -330,34 +350,44 @@ def pool_export_helper(filepath):
@cli_utils.action_logging
-def variables(args):
-if args.get:
-try:
-var = Variable.get(args.get,
- deserialize_json=args.json,
- default_var=args.default)
-print(var)
-except ValueError as e:
-print(e)
-if args.delete:
-Variable.delete(args.delete)
-if args.set:
-Variable.set(args.set[0], args.set[1])
-# Work around 'import' as a reserved keyword
-imp = getattr(args, 'import')
-if imp:
-if os.path.exists(imp):
-import_helper(imp)
-else:
-print("Missing variables file.")
-if args.export:
-export_helper(args.export)
-if not (args.set or args.get or imp or args.export or args.delete):
-# list all variables
-with db.create_session() as session:
-vars = session.query(Variable)
-msg = "\n".join(var.key for var in vars)
-print(msg)
+def variables_list(args):
+with db.create_session() as session:
+vars = session.query(Variable)
+print("\n".join(var.key for var in vars))
+
+
+@cli_utils.action_logging
+def variables_get(args):
+try:
+var = Variable.get(args.key,
+ deserialize_json=args.json,
+ default_var=args.default)
+print(var)
+except ValueError as e:
+print(e)
+
+
+@cli_utils.action_logging
+def variables_set(args):
+Variable.set(args.key, args.value, serialize_json=args.json)
+
+
+@cli_utils.action_logging
+def variables_delete(args):
+Variable.delete(args.key)
+
+
+@cli_utils.action_logging
+def variables_import(args):
+if os.path.exists(args.file):
+import_helper(args.file)
+else:
+print("Missing variables file.")
+
+
+@cli_utils.action_logging
+def variables_export(args):
+export_helper(args.file)
Review comment:
Made this change.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Comment Edited] (AIRFLOW-3513) Pakegecloud
[ https://issues.apache.org/jira/browse/AIRFLOW-3513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16820682#comment-16820682 ] Aizhamal Nurmamat kyzy edited comment on AIRFLOW-3513 at 4/18/19 3:20 AM: -- This issue seems automatically generated, and it doesn't contain any useful information. I'll close it for now. [~pakegecloud] please feel free to reopen if there is further information. EDIT: It looks like I dont have enough privileges to close Jira issues. [~ash] do you think they could be expanded please? was (Author: aizhamal): This issue seems automatically generated, and it doesn't contain any useful information. I'll close it for now. [~pakegecloud] please feel free to reopen if there is further information. > Pakegecloud > --- > > Key: AIRFLOW-3513 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3513 > Project: Apache Airflow > Issue Type: Improvement > Components: api, authentication, configuration, core, database, > Dataflow, db, docker >Reporter: pakegecloud.atlassian.net >Assignee: Aizhamal Nurmamat kyzy >Priority: Major > Original Estimate: 1,311h > Remaining Estimate: 1,311h > > pakegecloud.atlassian.net -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (AIRFLOW-3513) Pakegecloud
[ https://issues.apache.org/jira/browse/AIRFLOW-3513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aizhamal Nurmamat kyzy reassigned AIRFLOW-3513: --- Assignee: Aizhamal Nurmamat kyzy > Pakegecloud > --- > > Key: AIRFLOW-3513 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3513 > Project: Apache Airflow > Issue Type: Improvement > Components: api, authentication, configuration, core, database, > Dataflow, db, docker >Reporter: pakegecloud.atlassian.net >Assignee: Aizhamal Nurmamat kyzy >Priority: Major > Original Estimate: 1,311h > Remaining Estimate: 1,311h > > pakegecloud.atlassian.net -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-4347) airflow webserver can not load webserver_config.py
sumous created AIRFLOW-4347:
---
Summary: airflow webserver can not load webserver_config.py
Key: AIRFLOW-4347
URL: https://issues.apache.org/jira/browse/AIRFLOW-4347
Project: Apache Airflow
Issue Type: Bug
Components: webserver
Affects Versions: 1.10.3
Reporter: sumous
Assignee: sumous
Fix For: 1.10.4
I set AIRFLOW_HOME=./
command *airflow initdb* is ok.
And command *airflow webserver* got this:
{code:shell}
/Users/guoguo/Downloads/env_tmp/lib/python3.6/site-packages/flask_sqlalchemy/__init__.py:774:
UserWarning: Neither SQLALCHEMY_DATABASE_URI nor SQLALCHEMY_BINDS is set.
Defaulting SQLALCHEMY_DATABASE_URI to "sqlite:///:memory:".
'Neither SQLALCHEMY_DATABASE_URI nor SQLALCHEMY_BINDS is set. '
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] [airflow] codecov-io edited a comment on issue #5094: [AIRFLOW-4306] Global operator extra links
codecov-io edited a comment on issue #5094: [AIRFLOW-4306] Global operator extra links URL: https://github.com/apache/airflow/pull/5094#issuecomment-483039818 # [Codecov](https://codecov.io/gh/apache/airflow/pull/5094?src=pr=h1) Report > Merging [#5094](https://codecov.io/gh/apache/airflow/pull/5094?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/a71d4b861302db6101a399db9ef562d716f3c26e?src=pr=desc) will **decrease** coverage by `1.29%`. > The diff coverage is `95.65%`. [](https://codecov.io/gh/apache/airflow/pull/5094?src=pr=tree) ```diff @@ Coverage Diff@@ ## master #5094 +/- ## - Coverage 77.69% 76.4% -1.3% Files 465 463 -2 Lines 29816 29816 - Hits23167 22780-387 - Misses 66497036+387 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/5094?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/plugins\_manager.py](https://codecov.io/gh/apache/airflow/pull/5094/diff?src=pr=tree#diff-YWlyZmxvdy9wbHVnaW5zX21hbmFnZXIucHk=) | `87.38% <100%> (+0.47%)` | :arrow_up: | | [airflow/contrib/operators/qubole\_operator.py](https://codecov.io/gh/apache/airflow/pull/5094/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9xdWJvbGVfb3BlcmF0b3IucHk=) | `87.03% <100%> (+0.24%)` | :arrow_up: | | [airflow/models/baseoperator.py](https://codecov.io/gh/apache/airflow/pull/5094/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvYmFzZW9wZXJhdG9yLnB5) | `93.95% <93.75%> (-0.08%)` | :arrow_down: | | [airflow/contrib/operators/vertica\_operator.py](https://codecov.io/gh/apache/airflow/pull/5094/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy92ZXJ0aWNhX29wZXJhdG9yLnB5) | `0% <0%> (-100%)` | :arrow_down: | | [airflow/contrib/hooks/salesforce\_hook.py](https://codecov.io/gh/apache/airflow/pull/5094/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2hvb2tzL3NhbGVzZm9yY2VfaG9vay5weQ==) | `0% <0%> (-91.79%)` | :arrow_down: | | [airflow/contrib/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/5094/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2t1YmVybmV0ZXMvdm9sdW1lX21vdW50LnB5) | `33.33% <0%> (-66.67%)` | :arrow_down: | | [airflow/contrib/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/5094/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2t1YmVybmV0ZXMvcG9kX2xhdW5jaGVyLnB5) | `38.21% <0%> (-52.04%)` | :arrow_down: | | [airflow/hooks/webhdfs\_hook.py](https://codecov.io/gh/apache/airflow/pull/5094/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy93ZWJoZGZzX2hvb2sucHk=) | `36.36% <0%> (-51.4%)` | :arrow_down: | | [airflow/contrib/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/5094/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2t1YmVybmV0ZXMvdm9sdW1lLnB5) | `50% <0%> (-50%)` | :arrow_down: | | [airflow/contrib/operators/winrm\_operator.py](https://codecov.io/gh/apache/airflow/pull/5094/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy93aW5ybV9vcGVyYXRvci5weQ==) | `0% <0%> (-44.07%)` | :arrow_down: | | ... and [28 more](https://codecov.io/gh/apache/airflow/pull/5094/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/5094?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/5094?src=pr=footer). Last update [a71d4b8...997a405](https://codecov.io/gh/apache/airflow/pull/5094?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #5094: [AIRFLOW-4306] Global operator extra links
codecov-io edited a comment on issue #5094: [AIRFLOW-4306] Global operator extra links URL: https://github.com/apache/airflow/pull/5094#issuecomment-483039818 # [Codecov](https://codecov.io/gh/apache/airflow/pull/5094?src=pr=h1) Report > Merging [#5094](https://codecov.io/gh/apache/airflow/pull/5094?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/a71d4b861302db6101a399db9ef562d716f3c26e?src=pr=desc) will **decrease** coverage by `1.29%`. > The diff coverage is `95.65%`. [](https://codecov.io/gh/apache/airflow/pull/5094?src=pr=tree) ```diff @@ Coverage Diff@@ ## master #5094 +/- ## - Coverage 77.69% 76.4% -1.3% Files 465 463 -2 Lines 29816 29816 - Hits23167 22780-387 - Misses 66497036+387 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/5094?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/plugins\_manager.py](https://codecov.io/gh/apache/airflow/pull/5094/diff?src=pr=tree#diff-YWlyZmxvdy9wbHVnaW5zX21hbmFnZXIucHk=) | `87.38% <100%> (+0.47%)` | :arrow_up: | | [airflow/contrib/operators/qubole\_operator.py](https://codecov.io/gh/apache/airflow/pull/5094/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9xdWJvbGVfb3BlcmF0b3IucHk=) | `87.03% <100%> (+0.24%)` | :arrow_up: | | [airflow/models/baseoperator.py](https://codecov.io/gh/apache/airflow/pull/5094/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvYmFzZW9wZXJhdG9yLnB5) | `93.95% <93.75%> (-0.08%)` | :arrow_down: | | [airflow/contrib/operators/vertica\_operator.py](https://codecov.io/gh/apache/airflow/pull/5094/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy92ZXJ0aWNhX29wZXJhdG9yLnB5) | `0% <0%> (-100%)` | :arrow_down: | | [airflow/contrib/hooks/salesforce\_hook.py](https://codecov.io/gh/apache/airflow/pull/5094/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2hvb2tzL3NhbGVzZm9yY2VfaG9vay5weQ==) | `0% <0%> (-91.79%)` | :arrow_down: | | [airflow/contrib/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/5094/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2t1YmVybmV0ZXMvdm9sdW1lX21vdW50LnB5) | `33.33% <0%> (-66.67%)` | :arrow_down: | | [airflow/contrib/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/5094/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2t1YmVybmV0ZXMvcG9kX2xhdW5jaGVyLnB5) | `38.21% <0%> (-52.04%)` | :arrow_down: | | [airflow/hooks/webhdfs\_hook.py](https://codecov.io/gh/apache/airflow/pull/5094/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy93ZWJoZGZzX2hvb2sucHk=) | `36.36% <0%> (-51.4%)` | :arrow_down: | | [airflow/contrib/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/5094/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2t1YmVybmV0ZXMvdm9sdW1lLnB5) | `50% <0%> (-50%)` | :arrow_down: | | [airflow/contrib/operators/winrm\_operator.py](https://codecov.io/gh/apache/airflow/pull/5094/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy93aW5ybV9vcGVyYXRvci5weQ==) | `0% <0%> (-44.07%)` | :arrow_down: | | ... and [28 more](https://codecov.io/gh/apache/airflow/pull/5094/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/5094?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/5094?src=pr=footer). Last update [a71d4b8...997a405](https://codecov.io/gh/apache/airflow/pull/5094?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Assigned] (AIRFLOW-4324) Partial search in RBAC UI no longer works
[ https://issues.apache.org/jira/browse/AIRFLOW-4324?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sumous reassigned AIRFLOW-4324: --- Assignee: sumous > Partial search in RBAC UI no longer works > - > > Key: AIRFLOW-4324 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4324 > Project: Apache Airflow > Issue Type: Bug > Components: ui >Affects Versions: 1.10.3 >Reporter: Robin Edwards >Assignee: sumous >Priority: Minor > Fix For: 1.10.4 > > > In airflow 1.10.2 RBAC UI searching for part of a dag name worked. For > example if I had multiple dags named: > foo_bar_1 > foo_bar_2 > foo_bar_3 > Searching for "foo" would return all 3 dags. As of upgrading to 1.10.3 the > same search yields no results. On the result page after no results are > returned the search autocomplete feature no longer suggest dags as I type -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] [airflow] XD-DENG edited a comment on issue #5127: [AIRFLOW-4343] Show warning in UI if scheduler is not running
XD-DENG edited a comment on issue #5127: [AIRFLOW-4343] Show warning in UI if scheduler is not running URL: https://github.com/apache/airflow/pull/5127#issuecomment-484307268 Awesome feature Will have a deeper dive later. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] XD-DENG commented on issue #5127: [AIRFLOW-4343] Show warning in UI if scheduler is not running
XD-DENG commented on issue #5127: [AIRFLOW-4343] Show warning in UI if scheduler is not running URL: https://github.com/apache/airflow/pull/5127#issuecomment-484307268 Awesome feature This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] ryanyuan commented on issue #5117: [AIRFLOW-4335] Add default num_retries to GCP connection
ryanyuan commented on issue #5117: [AIRFLOW-4335] Add default num_retries to GCP connection URL: https://github.com/apache/airflow/pull/5117#issuecomment-484304059 @kaxil PTAL This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] ryanyuan commented on issue #5117: [AIRFLOW-4335] Add default num_retries to GCP connection
ryanyuan commented on issue #5117: [AIRFLOW-4335] Add default num_retries to GCP connection URL: https://github.com/apache/airflow/pull/5117#issuecomment-484303817 #5043 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (AIRFLOW-4346) Kubernetes Executor Fails for Large Wide DAGs
Vincent Castaneda created AIRFLOW-4346: -- Summary: Kubernetes Executor Fails for Large Wide DAGs Key: AIRFLOW-4346 URL: https://issues.apache.org/jira/browse/AIRFLOW-4346 Project: Apache Airflow Issue Type: Bug Components: DAG, executor, kubernetes Affects Versions: 1.10.3, 1.10.2 Reporter: Vincent Castaneda Attachments: configmap-airflow-share.yaml, wide_dag_bash_test.py, wide_dag_test_100_300.py, wide_dag_test_300_300.py When running large DAGs–those with parallelism of over 100 task instances to be running concurrently--several tasks fail on the executor and are reported to the database, but the scheduler is never aware of them failing. Attached are: - A test DAG that we can use to replicate the issue. - The configmap-airflow.yaml file I will be available to answer any other questions that are raised about our configuration. We are running this on GKE and giving the scheduler and web pod a base 100m for execution. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-4345) UI - admin/taskinstance (and other admin/* urls) is missing year portion in the date column values
[ https://issues.apache.org/jira/browse/AIRFLOW-4345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] t oo updated AIRFLOW-4345: -- Summary: UI - admin/taskinstance (and other admin/* urls) is missing year portion in the date column values (was: UI - admin/taskinstance is missing year portion in the date column values) > UI - admin/taskinstance (and other admin/* urls) is missing year portion in > the date column values > -- > > Key: AIRFLOW-4345 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4345 > Project: Apache Airflow > Issue Type: Bug > Components: ui >Affects Versions: 1.10.3 >Reporter: t oo >Priority: Major > Attachments: missingyear.png > > > see attached values are starting at MM-DD without a value showing in > each of the Execution Date, Start Date...etc columns > > || > | | | > | || > | | | > | |01-03T05:00:00+00:00| > | | | > | || > | | | > | || > | | | > | | | > | | | > | || > | | | > | || > | | | > | |04-17T22:00:24.286685+00:00| > | | | > | || > | | | > | || > | | | > | |04-17T22:00:24.286709+00:00| -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-4345) UI - admin/taskinstance (and other admin/* urls) is missing year portion in the date column values
[ https://issues.apache.org/jira/browse/AIRFLOW-4345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] t oo updated AIRFLOW-4345: -- Description: see attached values are starting at MM-DD without a value showing in each of the Execution Date, Start Date...etc columns. Seems to affect all date values in the ui across all pages under /admin/ || | | | | || | | | | |01-03T05:00:00+00:00| | | | | || | | | | || | | | | | | | | | | || | | | | || | | | | |04-17T22:00:24.286685+00:00| | | | | || | | | | || | | | | |04-17T22:00:24.286709+00:00| was: see attached values are starting at MM-DD without a value showing in each of the Execution Date, Start Date...etc columns || | | | | || | | | | |01-03T05:00:00+00:00| | | | | || | | | | || | | | | | | | | | | || | | | | || | | | | |04-17T22:00:24.286685+00:00| | | | | || | | | | || | | | | |04-17T22:00:24.286709+00:00| > UI - admin/taskinstance (and other admin/* urls) is missing year portion in > the date column values > -- > > Key: AIRFLOW-4345 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4345 > Project: Apache Airflow > Issue Type: Bug > Components: ui >Affects Versions: 1.10.3 >Reporter: t oo >Priority: Major > Attachments: missingyear.png > > > see attached values are starting at MM-DD without a value showing in > each of the Execution Date, Start Date...etc columns. Seems to affect all > date values in the ui across all pages under /admin/ > > || > | | | > | || > | | | > | |01-03T05:00:00+00:00| > | | | > | || > | | | > | || > | | | > | | | > | | | > | || > | | | > | || > | | | > | |04-17T22:00:24.286685+00:00| > | | | > | || > | | | > | || > | | | > | |04-17T22:00:24.286709+00:00| -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] [airflow] codecov-io commented on issue #4064: AIRFLOW-3149 Support dataproc cluster deletion on ERROR
codecov-io commented on issue #4064: AIRFLOW-3149 Support dataproc cluster deletion on ERROR URL: https://github.com/apache/airflow/pull/4064#issuecomment-484285488 # [Codecov](https://codecov.io/gh/apache/airflow/pull/4064?src=pr=h1) Report > Merging [#4064](https://codecov.io/gh/apache/airflow/pull/4064?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/b93f2649aed7fe566b987ec4cb135a2ec643acd6?src=pr=desc) will **increase** coverage by `1.5%`. > The diff coverage is `19.35%`. [](https://codecov.io/gh/apache/airflow/pull/4064?src=pr=tree) ```diff @@Coverage Diff@@ ## master#4064 +/- ## = + Coverage 76.36% 77.86% +1.5% = Files 471 465 -6 Lines 3029029859-431 = + Hits2313023251+121 + Misses 7160 6608-552 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/4064?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/contrib/hooks/gcp\_dataproc\_hook.py](https://codecov.io/gh/apache/airflow/pull/4064/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2hvb2tzL2djcF9kYXRhcHJvY19ob29rLnB5) | `34.93% <18.75%> (-2%)` | :arrow_down: | | [airflow/contrib/operators/dataproc\_operator.py](https://codecov.io/gh/apache/airflow/pull/4064/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9kYXRhcHJvY19vcGVyYXRvci5weQ==) | `66.6% <19.56%> (-2.92%)` | :arrow_down: | | [airflow/hooks/hdfs\_hook.py](https://codecov.io/gh/apache/airflow/pull/4064/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy9oZGZzX2hvb2sucHk=) | `30% <0%> (-62.5%)` | :arrow_down: | | [airflow/contrib/kubernetes/kube\_client.py](https://codecov.io/gh/apache/airflow/pull/4064/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2t1YmVybmV0ZXMva3ViZV9jbGllbnQucHk=) | `65.51% <0%> (-13.8%)` | :arrow_down: | | [airflow/utils/decorators.py](https://codecov.io/gh/apache/airflow/pull/4064/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9kZWNvcmF0b3JzLnB5) | `87.93% <0%> (-5.18%)` | :arrow_down: | | [airflow/contrib/hooks/pinot\_hook.py](https://codecov.io/gh/apache/airflow/pull/4064/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2hvb2tzL3Bpbm90X2hvb2sucHk=) | `77.5% <0%> (-5%)` | :arrow_down: | | [airflow/utils/timezone.py](https://codecov.io/gh/apache/airflow/pull/4064/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy90aW1lem9uZS5weQ==) | `89.58% <0%> (-1.73%)` | :arrow_down: | | [airflow/task/task\_runner/base\_task\_runner.py](https://codecov.io/gh/apache/airflow/pull/4064/diff?src=pr=tree#diff-YWlyZmxvdy90YXNrL3Rhc2tfcnVubmVyL2Jhc2VfdGFza19ydW5uZXIucHk=) | `75% <0%> (-1.67%)` | :arrow_down: | | [airflow/configuration.py](https://codecov.io/gh/apache/airflow/pull/4064/diff?src=pr=tree#diff-YWlyZmxvdy9jb25maWd1cmF0aW9uLnB5) | `90.75% <0%> (-1.37%)` | :arrow_down: | | [airflow/contrib/hooks/gcp\_dataflow\_hook.py](https://codecov.io/gh/apache/airflow/pull/4064/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2hvb2tzL2djcF9kYXRhZmxvd19ob29rLnB5) | `74.54% <0%> (-1.22%)` | :arrow_down: | | ... and [67 more](https://codecov.io/gh/apache/airflow/pull/4064/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/4064?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/4064?src=pr=footer). Last update [b93f264...51ce786](https://codecov.io/gh/apache/airflow/pull/4064?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Resolved] (AIRFLOW-4338) Change k8s pod_request_factory to use yaml safe_load
[
https://issues.apache.org/jira/browse/AIRFLOW-4338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Felix Uellendall resolved AIRFLOW-4338.
---
Resolution: Fixed
Fix Version/s: 1.10.4
> Change k8s pod_request_factory to use yaml safe_load
>
>
> Key: AIRFLOW-4338
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4338
> Project: Apache Airflow
> Issue Type: Improvement
>Reporter: Felix Uellendall
>Assignee: Felix Uellendall
>Priority: Major
> Fix For: 1.10.4
>
>
> Before Python yaml version 5.1 yaml has used the {{UnsafeLoader that has a
> potential security vulnerability. From now on there is a FullLoader}} that
> will be used by default that fixes these vulnerability. To suppress the
> warning we can explicitly set the loader.
> See [https://github.com/yaml/pyyaml/wiki/PyYAML-yaml.load(input)-Deprecation]
> for more details.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (AIRFLOW-4345) UI - admin/taskinstance is missing year portion in the date column values
t oo created AIRFLOW-4345: - Summary: UI - admin/taskinstance is missing year portion in the date column values Key: AIRFLOW-4345 URL: https://issues.apache.org/jira/browse/AIRFLOW-4345 Project: Apache Airflow Issue Type: Bug Components: ui Affects Versions: 1.10.3 Reporter: t oo Attachments: missingyear.png see attached values are starting at MM-DD without a value showing in each of the Execution Date, Start Date...etc columns || | | | | || | | | | |01-03T05:00:00+00:00| | | | | || | | | | || | | | | | | | | | | || | | | | || | | | | |04-17T22:00:24.286685+00:00| | | | | || | | | | || | | | | |04-17T22:00:24.286709+00:00| -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] [airflow] codecov-io commented on issue #5126: [AIRFLOW-4342] Use @cached_property instead of re-implementing it each time
codecov-io commented on issue #5126: [AIRFLOW-4342] Use @cached_property instead of re-implementing it each time URL: https://github.com/apache/airflow/pull/5126#issuecomment-484273651 # [Codecov](https://codecov.io/gh/apache/airflow/pull/5126?src=pr=h1) Report > Merging [#5126](https://codecov.io/gh/apache/airflow/pull/5126?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/a511b4877b88c5154474309e3acd7a60706415c9?src=pr=desc) will **increase** coverage by `<.01%`. > The diff coverage is `71.42%`. [](https://codecov.io/gh/apache/airflow/pull/5126?src=pr=tree) ```diff @@Coverage Diff @@ ## master#5126 +/- ## == + Coverage 77.96% 77.96% +<.01% == Files 465 465 Lines 2981629797 -19 == - Hits2324523231 -14 + Misses 6571 6566 -5 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/5126?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/utils/decorators.py](https://codecov.io/gh/apache/airflow/pull/5126/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9kZWNvcmF0b3JzLnB5) | `85.41% <ø> (-2.52%)` | :arrow_down: | | [airflow/utils/log/gcs\_task\_handler.py](https://codecov.io/gh/apache/airflow/pull/5126/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9sb2cvZ2NzX3Rhc2tfaGFuZGxlci5weQ==) | `0% <0%> (ø)` | :arrow_up: | | [airflow/utils/log/s3\_task\_handler.py](https://codecov.io/gh/apache/airflow/pull/5126/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9sb2cvczNfdGFza19oYW5kbGVyLnB5) | `98.5% <100%> (-0.07%)` | :arrow_down: | | [airflow/utils/log/wasb\_task\_handler.py](https://codecov.io/gh/apache/airflow/pull/5126/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9sb2cvd2FzYl90YXNrX2hhbmRsZXIucHk=) | `32.87% <100%> (+1.29%)` | :arrow_up: | | [airflow/contrib/hooks/gcp\_vision\_hook.py](https://codecov.io/gh/apache/airflow/pull/5126/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2hvb2tzL2djcF92aXNpb25faG9vay5weQ==) | `91.51% <100%> (ø)` | :arrow_up: | | [airflow/models/taskinstance.py](https://codecov.io/gh/apache/airflow/pull/5126/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvdGFza2luc3RhbmNlLnB5) | `92.42% <0%> (-0.18%)` | :arrow_down: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/5126?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/5126?src=pr=footer). Last update [a511b48...396a11b](https://codecov.io/gh/apache/airflow/pull/5126?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #5125: [AIRFLOW-4341] Remove `View.render()` already exists in fab.BaseView
codecov-io edited a comment on issue #5125: [AIRFLOW-4341] Remove `View.render()` already exists in fab.BaseView URL: https://github.com/apache/airflow/pull/5125#issuecomment-484148040 # [Codecov](https://codecov.io/gh/apache/airflow/pull/5125?src=pr=h1) Report > Merging [#5125](https://codecov.io/gh/apache/airflow/pull/5125?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/a511b4877b88c5154474309e3acd7a60706415c9?src=pr=desc) will **decrease** coverage by `<.01%`. > The diff coverage is `79.16%`. [](https://codecov.io/gh/apache/airflow/pull/5125?src=pr=tree) ```diff @@Coverage Diff @@ ## master#5125 +/- ## == - Coverage 77.96% 77.95% -0.01% == Files 465 465 Lines 2981629814 -2 == - Hits2324523242 -3 - Misses 6571 6572 +1 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/5125?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/www/views.py](https://codecov.io/gh/apache/airflow/pull/5125/diff?src=pr=tree#diff-YWlyZmxvdy93d3cvdmlld3MucHk=) | `76.35% <79.16%> (-0.04%)` | :arrow_down: | | [airflow/models/taskinstance.py](https://codecov.io/gh/apache/airflow/pull/5125/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvdGFza2luc3RhbmNlLnB5) | `92.42% <0%> (-0.18%)` | :arrow_down: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/5125?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/5125?src=pr=footer). Last update [a511b48...1b93ce6](https://codecov.io/gh/apache/airflow/pull/5125?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-4299) Upgrade to Celery 4.3.0 to fix crashing workers
[ https://issues.apache.org/jira/browse/AIRFLOW-4299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16820513#comment-16820513 ] Teresa Martyny commented on AIRFLOW-4299: - Damn. Thanks for the rapid turn around!! > Upgrade to Celery 4.3.0 to fix crashing workers > --- > > Key: AIRFLOW-4299 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4299 > Project: Apache Airflow > Issue Type: Bug > Components: celery >Affects Versions: 1.10.2 >Reporter: Teresa Martyny >Assignee: Kaxil Naik >Priority: Major > Fix For: 1.10.4 > > > Celery needs to be updated to 4.3.0 to fix this issue: > [https://github.com/celery/billiard/issues/260] > When doing warm restarts all Celery workers crash. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-1063) A manually-created DAG run can prevent a scheduled run to be created
[
https://issues.apache.org/jira/browse/AIRFLOW-1063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16820521#comment-16820521
]
t oo commented on AIRFLOW-1063:
---
#this works
airflow trigger_dag -e 20190412 mycooldag --conf '\{"hourstr":"1"}'
#then below gets error --> airflow.exceptions.DagRunAlreadyExists: Run id
manual__2019-04-12T00:00:00+00:00 already exists for dag id mycooldag
airflow trigger_dag -e 20190412 mycooldag --conf '\{"hourstr":"2"}'
any workaround for this?
> A manually-created DAG run can prevent a scheduled run to be created
>
>
> Key: AIRFLOW-1063
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1063
> Project: Apache Airflow
> Issue Type: Improvement
> Components: scheduler
>Affects Versions: 1.7.1.3
>Reporter: Vitor Baptista
>Priority: Major
>
> I manually created a DAG Run with the {{execution_date}} as {{2017-03-01
> 00:00:00}} on a monthly-recurrent DAG. After a while, I noticed that the
> scheduled run was never created and checked the scheduler's logs, finding
> this traceback:
> {quote}
> Process Process-475397:
> Traceback (most recent call last):
> File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in
> _bootstrap
> self.run()
> File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
> self._target(*self._args, **self._kwargs)
> File "/usr/local/lib/python2.7/dist-packages/airflow/jobs.py", line 664, in
> _do_dags
> dag = dagbag.get_dag(dag.dag_id)
> File "/usr/local/lib/python2.7/dist-packages/airflow/models.py", line 188,
> in get_dag
> orm_dag = DagModel.get_current(root_dag_id)
> File "/usr/local/lib/python2.7/dist-packages/airflow/models.py", line 2320,
> in get_current
> obj = session.query(cls).filter(cls.dag_id == dag_id).first()
> File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line
> 2690, in first
> ret = list(self[0:1])
> File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line
> 2482, in __getitem__
> return list(res)
> File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line
> 2790, in __iter__
> return self._execute_and_instances(context)
> File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line
> 2811, in _execute_and_instances
> close_with_result=True)
> File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line
> 2820, in _get_bind_args
> **kw
> File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line
> 2802, in _connection_from_session
> conn = self.session.connection(**kw)
> File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py",
> line 966, in connection
> execution_options=execution_options)
> File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py",
> line 971, in _connection_for_bind
> engine, execution_options)
> File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py",
> line 382, in _connection_for_bind
> self._assert_active()
> File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py",
> line 276, in _assert_active
> % self._rollback_exception
> InvalidRequestError: This Session's transaction has been rolled back due to a
> previous exception during flush. To begin a new transaction with this
> Session, first issue Session.rollback(). Original exception was:
> (psycopg2.IntegrityError)
> duplicate key value violates unique constraint
> "dag_run_dag_id_execution_date_key"
> DETAIL: Key (dag_id, execution_date)=(nct, 2017-03-01 00:00:00) already
> exists.
> [SQL: 'INSERT INTO dag_run (dag_id, execution_date, start_date, end_date,
> state, run_id, external_trigger, conf) VALUES (%(dag_id)s,
> %(execution_date)s, %(start_date)s, %(end_date)s, %(state)s, %(run_id)s,
> %(external_trigger)s, %(conf)s)
> RETURNING dag_run.id'] [parameters: {'end_date': None, 'run_id':
> u'scheduled__2017-03-01T00:00:00', 'execution_date': datetime.datetime(2017,
> 3, 1, 0, 0), 'external_trigger': False, 'state': u'running', 'conf': None,
> 'start_date': dateti
> me.datetime(2017, 4, 3, 13, 48, 39, 168456), 'dag_id': 'nct'}]
> {quote}
> The problem is that the {{dag_runs}} table require the {{(dag_id,
> execution_date)}} pair to be unique, so the scheduler was stuck in a loop
> where it tried creating a new scheduled dag run but failed, as I had already
> created one on the same {{execution_date}}. This was surprising. As a user, I
> would expect that it would either schedule the run normally, even if there's
> a manual one on the same date, or maybe it would skip that execution date.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] [airflow] ashb commented on issue #5127: [AIRFLOW-4343] Show warning in UI if scheduler is not running
ashb commented on issue #5127: [AIRFLOW-4343] Show warning in UI if scheduler is not running URL: https://github.com/apache/airflow/pull/5127#issuecomment-484262466 @XD-DENG PTAL too - I've subtly changed the /health endpoint here to use functions from airflow.jobs. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-4338) Change k8s pod_request_factory to use yaml safe_load
[
https://issues.apache.org/jira/browse/AIRFLOW-4338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16820502#comment-16820502
]
ASF GitHub Bot commented on AIRFLOW-4338:
-
dimberman commented on pull request #5120: [AIRFLOW-4338] Change k8s
pod_request_factory to use yaml safe_load
URL: https://github.com/apache/airflow/pull/5120
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Change k8s pod_request_factory to use yaml safe_load
>
>
> Key: AIRFLOW-4338
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4338
> Project: Apache Airflow
> Issue Type: Improvement
>Reporter: Felix Uellendall
>Assignee: Felix Uellendall
>Priority: Major
>
> Before Python yaml version 5.1 yaml has used the {{UnsafeLoader that has a
> potential security vulnerability. From now on there is a FullLoader}} that
> will be used by default that fixes these vulnerability. To suppress the
> warning we can explicitly set the loader.
> See [https://github.com/yaml/pyyaml/wiki/PyYAML-yaml.load(input)-Deprecation]
> for more details.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AIRFLOW-4338) Change k8s pod_request_factory to use yaml safe_load
[
https://issues.apache.org/jira/browse/AIRFLOW-4338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16820503#comment-16820503
]
ASF subversion and git services commented on AIRFLOW-4338:
--
Commit a71d4b861302db6101a399db9ef562d716f3c26e in airflow's branch
refs/heads/master from Felix Uellendall
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=a71d4b8 ]
[AIRFLOW-4338] Change k8s pod_request_factory to use yaml safe_load (#5120)
- fixes yaml.load deprecation warning
> Change k8s pod_request_factory to use yaml safe_load
>
>
> Key: AIRFLOW-4338
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4338
> Project: Apache Airflow
> Issue Type: Improvement
>Reporter: Felix Uellendall
>Assignee: Felix Uellendall
>Priority: Major
>
> Before Python yaml version 5.1 yaml has used the {{UnsafeLoader that has a
> potential security vulnerability. From now on there is a FullLoader}} that
> will be used by default that fixes these vulnerability. To suppress the
> warning we can explicitly set the loader.
> See [https://github.com/yaml/pyyaml/wiki/PyYAML-yaml.load(input)-Deprecation]
> for more details.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] [airflow] dimberman merged pull request #5120: [AIRFLOW-4338] Change k8s pod_request_factory to use yaml safe_load
dimberman merged pull request #5120: [AIRFLOW-4338] Change k8s pod_request_factory to use yaml safe_load URL: https://github.com/apache/airflow/pull/5120 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] dimberman commented on issue #5120: [AIRFLOW-4338] Change k8s pod_request_factory to use yaml safe_load
dimberman commented on issue #5120: [AIRFLOW-4338] Change k8s pod_request_factory to use yaml safe_load URL: https://github.com/apache/airflow/pull/5120#issuecomment-484261105 SG. Yeah if the tests are passing we should be fine. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io commented on issue #5128: [AIRFLOW-4307] Backfill respects concurrency limit
codecov-io commented on issue #5128: [AIRFLOW-4307] Backfill respects concurrency limit URL: https://github.com/apache/airflow/pull/5128#issuecomment-484258404 # [Codecov](https://codecov.io/gh/apache/airflow/pull/5128?src=pr=h1) Report > Merging [#5128](https://codecov.io/gh/apache/airflow/pull/5128?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/a511b4877b88c5154474309e3acd7a60706415c9?src=pr=desc) will **increase** coverage by `<.01%`. > The diff coverage is `100%`. [](https://codecov.io/gh/apache/airflow/pull/5128?src=pr=tree) ```diff @@Coverage Diff @@ ## master#5128 +/- ## == + Coverage 77.96% 77.97% +<.01% == Files 465 465 Lines 2981629823 +7 == + Hits2324523253 +8 + Misses 6571 6570 -1 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/5128?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/exceptions.py](https://codecov.io/gh/apache/airflow/pull/5128/diff?src=pr=tree#diff-YWlyZmxvdy9leGNlcHRpb25zLnB5) | `100% <100%> (ø)` | :arrow_up: | | [airflow/models/dag.py](https://codecov.io/gh/apache/airflow/pull/5128/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvZGFnLnB5) | `93.36% <100%> (+0.17%)` | :arrow_up: | | [airflow/jobs.py](https://codecov.io/gh/apache/airflow/pull/5128/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzLnB5) | `78.59% <100%> (+0.05%)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/5128?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/5128?src=pr=footer). Last update [a511b48...0c957fb](https://codecov.io/gh/apache/airflow/pull/5128?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] feluelle commented on issue #5120: [AIRFLOW-4338] Change k8s pod_request_factory to use yaml safe_load
feluelle commented on issue #5120: [AIRFLOW-4338] Change k8s pod_request_factory to use yaml safe_load URL: https://github.com/apache/airflow/pull/5120#issuecomment-484258154 The issue I had was only related to mypy (I still don't get) but has been worked-around by using a shortcut function where mypy don't need to check the Loader classes. - So yes it is fixed. Just to summarize it for you: If we use the `FullLoader` / `load_full` we don't change any functionality. It just will fix the deprecation warning. > Loads the full YAML language. Avoids arbitrary code execution. This is currently (PyYAML 5.1) the default loader called by yaml.load(input) (after issuing the warning). If we use the `SafeLoader` / `load_safe` we may change functionality - but tests run successfully so it is probably fine. > Loads a subset of the YAML language, safely. This is recommended for loading untrusted input. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Resolved] (AIRFLOW-808) SqlToS3 Operator
[ https://issues.apache.org/jira/browse/AIRFLOW-808?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Imberman resolved AIRFLOW-808. - Resolution: Won't Do > SqlToS3 Operator > > > Key: AIRFLOW-808 > URL: https://issues.apache.org/jira/browse/AIRFLOW-808 > Project: Apache Airflow > Issue Type: New Feature > Components: contrib, operators >Reporter: Abner S Rolim >Assignee: Abner S Rolim >Priority: Minor > > As an user, i need to export query results to s3 in csv format. Optionally, > it will be good some compression option (zip). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (AIRFLOW-1117) Increase the default value of min_file_process_interval
[
https://issues.apache.org/jira/browse/AIRFLOW-1117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Imberman closed AIRFLOW-1117.
Resolution: Duplicate
> Increase the default value of min_file_process_interval
> ---
>
> Key: AIRFLOW-1117
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1117
> Project: Apache Airflow
> Issue Type: Wish
> Components: scheduler
>Affects Versions: 1.8.0
>Reporter: Keisuke Nishida
>Priority: Minor
> Attachments: screenshot-1.png
>
>
> I observed high CPU usage after upgrading Airflow from 1.7.1.3 to 1.8.0.
> I found Airflow is loading DAG files repeatedly, which consumed most of the
> CPU time in my Airflow instance. I realized Airflow 1.8 introduced a new
> configuration variable {{min_file_process_interval}} with default value 0.
> This means Airflow reloads DAG files one after another without any interval.
> Can you increase the default value of {{min_file_process_interval}} to at
> least the same value of {{scheduler_heartbeat_sec}}, which is 5 seconds?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] [airflow] dimberman closed pull request #2825: [AIRFLOW-1117] Change default min_file_process_interval
dimberman closed pull request #2825: [AIRFLOW-1117] Change default min_file_process_interval URL: https://github.com/apache/airflow/pull/2825 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] dimberman commented on issue #2825: [AIRFLOW-1117] Change default min_file_process_interval
dimberman commented on issue #2825: [AIRFLOW-1117] Change default min_file_process_interval URL: https://github.com/apache/airflow/pull/2825#issuecomment-484256659 Closing since this is a duplicate This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-1117) Increase the default value of min_file_process_interval
[
https://issues.apache.org/jira/browse/AIRFLOW-1117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16820492#comment-16820492
]
ASF GitHub Bot commented on AIRFLOW-1117:
-
dimberman commented on pull request #2825: [AIRFLOW-1117] Change default
min_file_process_interval
URL: https://github.com/apache/airflow/pull/2825
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Increase the default value of min_file_process_interval
> ---
>
> Key: AIRFLOW-1117
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1117
> Project: Apache Airflow
> Issue Type: Wish
> Components: scheduler
>Affects Versions: 1.8.0
>Reporter: Keisuke Nishida
>Priority: Minor
> Attachments: screenshot-1.png
>
>
> I observed high CPU usage after upgrading Airflow from 1.7.1.3 to 1.8.0.
> I found Airflow is loading DAG files repeatedly, which consumed most of the
> CPU time in my Airflow instance. I realized Airflow 1.8 introduced a new
> configuration variable {{min_file_process_interval}} with default value 0.
> This means Airflow reloads DAG files one after another without any interval.
> Can you increase the default value of {{min_file_process_interval}} to at
> least the same value of {{scheduler_heartbeat_sec}}, which is 5 seconds?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] [airflow] dimberman commented on issue #2443: [AIRFLOW-1411] Auto-create from the newest version
dimberman commented on issue #2443: [AIRFLOW-1411] Auto-create from the newest version URL: https://github.com/apache/airflow/pull/2443#issuecomment-484255644 Hi @peterjdolan! I'm going to close this ticket/PR but please let me know if you are interested in working on this and I will re-open it. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-1411) Support versioned model export directories in the CloudMLVersionOperator
[ https://issues.apache.org/jira/browse/AIRFLOW-1411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16820491#comment-16820491 ] ASF GitHub Bot commented on AIRFLOW-1411: - dimberman commented on pull request #2443: [AIRFLOW-1411] Auto-create from the newest version URL: https://github.com/apache/airflow/pull/2443 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Support versioned model export directories in the CloudMLVersionOperator > > > Key: AIRFLOW-1411 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1411 > Project: Apache Airflow > Issue Type: Improvement > Components: contrib >Affects Versions: 1.9.0 >Reporter: Peter Dolan >Assignee: Peter Dolan >Priority: Major > > Some TensorFlow libraries export models into versioned > subdirectories - some using incrementing versions, some using > the unix timestamp - making it difficult to create a version > when the full deployment Uri must be known in advance. > The CloudMLVersionOperator should support specifying the base path in which > the TensorFlow models are exported, and to create a CloudML > version using the exported model with the highest value (which > should be interpreted as the latest model export). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (AIRFLOW-1411) Support versioned model export directories in the CloudMLVersionOperator
[ https://issues.apache.org/jira/browse/AIRFLOW-1411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Imberman closed AIRFLOW-1411. Resolution: Won't Do > Support versioned model export directories in the CloudMLVersionOperator > > > Key: AIRFLOW-1411 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1411 > Project: Apache Airflow > Issue Type: Improvement > Components: contrib >Affects Versions: 1.9.0 >Reporter: Peter Dolan >Assignee: Peter Dolan >Priority: Major > > Some TensorFlow libraries export models into versioned > subdirectories - some using incrementing versions, some using > the unix timestamp - making it difficult to create a version > when the full deployment Uri must be known in advance. > The CloudMLVersionOperator should support specifying the base path in which > the TensorFlow models are exported, and to create a CloudML > version using the exported model with the highest value (which > should be interpreted as the latest model export). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] [airflow] dimberman closed pull request #2443: [AIRFLOW-1411] Auto-create from the newest version
dimberman closed pull request #2443: [AIRFLOW-1411] Auto-create from the newest version URL: https://github.com/apache/airflow/pull/2443 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-808) SqlToS3 Operator
[ https://issues.apache.org/jira/browse/AIRFLOW-808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16820489#comment-16820489 ] ASF GitHub Bot commented on AIRFLOW-808: dimberman commented on pull request #2040: [AIRFLOW-808] - SqlToS3 Operator URL: https://github.com/apache/airflow/pull/2040 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > SqlToS3 Operator > > > Key: AIRFLOW-808 > URL: https://issues.apache.org/jira/browse/AIRFLOW-808 > Project: Apache Airflow > Issue Type: New Feature > Components: contrib, operators >Reporter: Abner S Rolim >Assignee: Abner S Rolim >Priority: Minor > > As an user, i need to export query results to s3 in csv format. Optionally, > it will be good some compression option (zip). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] [airflow] morgendave commented on issue #4923: [AIRFLOW-4092] Add gRPCOperator, unit test and added to auto doc
morgendave commented on issue #4923: [AIRFLOW-4092] Add gRPCOperator, unit test and added to auto doc URL: https://github.com/apache/airflow/pull/4923#issuecomment-484255196 > Hey @morgendave if you don't mind I have a few comments. :) Thanks a lot, these should be fixed now This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] dimberman closed pull request #2040: [AIRFLOW-808] - SqlToS3 Operator
dimberman closed pull request #2040: [AIRFLOW-808] - SqlToS3 Operator URL: https://github.com/apache/airflow/pull/2040 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] dimberman commented on issue #2040: [AIRFLOW-808] - SqlToS3 Operator
dimberman commented on issue #2040: [AIRFLOW-808] - SqlToS3 Operator URL: https://github.com/apache/airflow/pull/2040#issuecomment-484254824 Hi @abnerrolim! I'm going to close this PR but please let me know if you still interested in doing it and I will gladly re-open. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] piffall edited a comment on issue #4064: AIRFLOW-3149 Support dataproc cluster deletion on ERROR
piffall edited a comment on issue #4064: AIRFLOW-3149 Support dataproc cluster deletion on ERROR URL: https://github.com/apache/airflow/pull/4064#issuecomment-484245591 > @OmerJog I went over that test in detail and it seems like the `with self.assertRaises(TypeError)` was a defensive measure to catch an exception within the test and not that raising a TypeError was part of the test specification. I've deleted that line to see if the rest of the test is successful. > > cc @piffall who added that line about 11 months ago Hi @dossett, in fact, I didn't add that line, I just keep [that test with a minor change](https://github.com/apache/airflow/pull/3357/files#diff-f7804530ca88cb2e43e9492695ccde2cL320). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] piffall edited a comment on issue #4064: AIRFLOW-3149 Support dataproc cluster deletion on ERROR
piffall edited a comment on issue #4064: AIRFLOW-3149 Support dataproc cluster deletion on ERROR URL: https://github.com/apache/airflow/pull/4064#issuecomment-484245591 > @OmerJog I went over that test in detail and it seems like the `with self.assertRaises(TypeError)` was a defensive measure to catch an exception within the test and not that raising a TypeError was part of the test specification. I've deleted that line to see if the rest of the test is successful. > > cc @piffall who added that line about 11 months ago Hi @dossett, in fact, I didn't add that line, I just keep [that test with a minor change] (https://github.com/apache/airflow/pull/3357/files#diff-f7804530ca88cb2e43e9492695ccde2cL320). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] piffall edited a comment on issue #4064: AIRFLOW-3149 Support dataproc cluster deletion on ERROR
piffall edited a comment on issue #4064: AIRFLOW-3149 Support dataproc cluster deletion on ERROR URL: https://github.com/apache/airflow/pull/4064#issuecomment-484245591 > @OmerJog I went over that test in detail and it seems like the `with self.assertRaises(TypeError)` was a defensive measure to catch an exception within the test and not that raising a TypeError was part of the test specification. I've deleted that line to see if the rest of the test is successful. > > cc @piffall who added that line about 11 months ago Hi @dossett, in fact, I didn't add that line, I just keep [that test with a minor change] (https://github.com/apache/airflow/pull/3357/files#diff-f7804530ca88cb2e43e9492695ccde2cL320) in a similar test. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] piffall commented on issue #4064: AIRFLOW-3149 Support dataproc cluster deletion on ERROR
piffall commented on issue #4064: AIRFLOW-3149 Support dataproc cluster deletion on ERROR URL: https://github.com/apache/airflow/pull/4064#issuecomment-484245591 > @OmerJog I went over that test in detail and it seems like the `with self.assertRaises(TypeError)` was a defensive measure to catch an exception within the test and not that raising a TypeError was part of the test specification. I've deleted that line to see if the rest of the test is successful. > > cc @piffall who added that line about 11 months ago Hi @dossett, in fact, I didn't add that line, I just keep [that test with a minor change](https://github.com/apache/airflow/pull/3357/files#diff-f7804530ca88cb2e43e9492695ccde2cL320). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-4307) Backfill respect concurrency limit
[
https://issues.apache.org/jira/browse/AIRFLOW-4307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16820462#comment-16820462
]
ASF GitHub Bot commented on AIRFLOW-4307:
-
milton0825 commented on pull request #5128: [AIRFLOW-4307] Backfill respects
concurrency limit
URL: https://github.com/apache/airflow/pull/5128
Make sure you have checked _all_ steps below.
### Jira
- [X] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-4307
### Description
- [X] Here are some details about my PR, including screenshots of any UI
changes:
Airflow backfill should respect the concurrency limit specified in the `DAG`.
### Tests
- [X] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [X] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [X] In case of new functionality, my PR adds documentation that describes
how to use it.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
### Code Quality
- [X] Passes `flake8`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Backfill respect concurrency limit
> --
>
> Key: AIRFLOW-4307
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4307
> Project: Apache Airflow
> Issue Type: New Feature
>Reporter: Chao-Han Tsai
>Assignee: Chao-Han Tsai
>Priority: Major
>
> Currently backfill respects `pool` limit and `max_active_runs`. It is
> probably a good idea to make it respect concurrency limit so that we won't
> launch a big backfill that occupied all the resources.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] [airflow] milton0825 opened a new pull request #5128: [AIRFLOW-4307] Backfill respects concurrency limit
milton0825 opened a new pull request #5128: [AIRFLOW-4307] Backfill respects
concurrency limit
URL: https://github.com/apache/airflow/pull/5128
Make sure you have checked _all_ steps below.
### Jira
- [X] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-4307
### Description
- [X] Here are some details about my PR, including screenshots of any UI
changes:
Airflow backfill should respect the concurrency limit specified in the `DAG`.
### Tests
- [X] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [X] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [X] In case of new functionality, my PR adds documentation that describes
how to use it.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
### Code Quality
- [X] Passes `flake8`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io commented on issue #5120: [AIRFLOW-4338] Change k8s pod_request_factory to use yaml safe_load
codecov-io commented on issue #5120: [AIRFLOW-4338] Change k8s pod_request_factory to use yaml safe_load URL: https://github.com/apache/airflow/pull/5120#issuecomment-484227217 # [Codecov](https://codecov.io/gh/apache/airflow/pull/5120?src=pr=h1) Report > Merging [#5120](https://codecov.io/gh/apache/airflow/pull/5120?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/a511b4877b88c5154474309e3acd7a60706415c9?src=pr=desc) will **decrease** coverage by `<.01%`. > The diff coverage is `100%`. [](https://codecov.io/gh/apache/airflow/pull/5120?src=pr=tree) ```diff @@Coverage Diff @@ ## master#5120 +/- ## == - Coverage 77.96% 77.95% -0.01% == Files 465 465 Lines 2981629816 == - Hits2324523244 -1 - Misses 6571 6572 +1 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/5120?src=pr=tree) | Coverage Δ | | |---|---|---| | [.../kubernetes\_request\_factory/pod\_request\_factory.py](https://codecov.io/gh/apache/airflow/pull/5120/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2t1YmVybmV0ZXMva3ViZXJuZXRlc19yZXF1ZXN0X2ZhY3RvcnkvcG9kX3JlcXVlc3RfZmFjdG9yeS5weQ==) | `100% <100%> (ø)` | :arrow_up: | | [airflow/models/taskinstance.py](https://codecov.io/gh/apache/airflow/pull/5120/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvdGFza2luc3RhbmNlLnB5) | `92.42% <0%> (-0.18%)` | :arrow_down: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/5120?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/5120?src=pr=footer). Last update [a511b48...6b57282](https://codecov.io/gh/apache/airflow/pull/5120?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-4305) DAGs not showing in Web UI if Scheduler is not run at least once
[
https://issues.apache.org/jira/browse/AIRFLOW-4305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16820408#comment-16820408
]
Andreas Költringer commented on AIRFLOW-4305:
-
[~ash] yes! Looks great. Thx for your effort. Let me know if I can help!
> DAGs not showing in Web UI if Scheduler is not run at least once
>
>
> Key: AIRFLOW-4305
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4305
> Project: Apache Airflow
> Issue Type: Bug
> Components: webapp
>Affects Versions: 1.10.3
> Environment: Linux (archlinux), Pyton 3.7
>Reporter: Andreas Költringer
>Priority: Major
> Fix For: 1.10.4
>
> Attachments: Screen Shot 2019-04-17 at 18.47.56.png
>
>
> Dags do not show up in the Web UI for new airflow instances.
> Steps to reproduce:
> {{mkdir temp && cd temp}}
> {{python -m venv venv}}
> {{source venv/bin/activate}}
> {{pip install
> apache-airflow[celery,rabbitmq,crypto,password,postgres]==1.10.3}}
> {{export AIRFLOW_HOME=$(pwd)}}
> {{airflow upgradedb}}
> {{airflow webserver}}
> Neither the "example dags" nor any dags dropped into {{dags/}} show up.
> Some more facts:
> * Interestingly, when replacing {{airflow upgradedb}} with {{airflow
> initdb}} it works as expected. I thought, using {{airflow upgradedb}} was
> safe b/c it is mentioned here:
> [https://medium.com/datareply/airflow-lesser-known-tips-tricks-and-best-practises-cf4d4a90f8f]
> * Also, the version with {{airflow upgradedb}} works as expected with
> {{v1.10.2}}.
>
> Updates:
> * {{initdb}} does insert DAGs into the database, while {{upgradedb}} does
> not: [https://github.com/apache/airflow/blob/master/airflow/utils/db.py#L292]
> *
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AIRFLOW-4343) Show warning in UI if scheduler is not running.
[
https://issues.apache.org/jira/browse/AIRFLOW-4343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16820389#comment-16820389
]
ASF GitHub Bot commented on AIRFLOW-4343:
-
ashb commented on pull request #5127: [AIRFLOW-4343] Show warning in UI if
scheduler is not running
URL: https://github.com/apache/airflow/pull/5127
First commit is from #5125, look at the second commit only for now.
Make sure you have checked _all_ steps below.
### Jira
- [x]
- https://issues.apache.org/jira/browse/AIRFLOW-4343
- In case you are adding a dependency, check if the license complies with
the [ASF 3rd Party License
Policy](https://www.apache.org/legal/resolved.html#category-x). - yes, Cat A
### Description
- [x] Now that the webserver is more stateless, if the scheduler is not
running the list of dags won't populate, making it harder for new
starters to work out what is going on.
**When scheduler has never run (I cleared Jobs table):**
**When scheduler not running (either crashed, or has stopped successfully)
(the time has a tooltip to display the full ISO8601 heartbeat)
Login screen/anything not subclassed from AirflowBaseView doesn't display
the message.
On Login (not changed, doesn't show warning):
### Tests
- [x] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason: added tests for helper functions on BaseJob
### Commits
- [x] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [x] In case of new functionality, my PR adds documentation that describes
how to use it.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
### Code Quality
- [x] Passes `flake8`
/cc @Fokko as discussed.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Show warning in UI if scheduler is not running.
> ---
>
> Key: AIRFLOW-4343
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4343
> Project: Apache Airflow
> Issue Type: Sub-task
> Components: ui
>Reporter: Ash Berlin-Taylor
>Assignee: Ash Berlin-Taylor
>Priority: Major
> Fix For: 1.10.4
>
>
> Now that the webserver is more stateless, if the scheduler is not running the
> list of dags won't populate, making it harder for new starters to work out
> what is going on.
> We should display a warning if the scheduler isn't running (based on the job
> table heartbeat column)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] [airflow] ashb opened a new pull request #5127: [AIRFLOW-4343] Show warning in UI if scheduler is not running
ashb opened a new pull request #5127: [AIRFLOW-4343] Show warning in UI if
scheduler is not running
URL: https://github.com/apache/airflow/pull/5127
First commit is from #5125, look at the second commit only for now.
Make sure you have checked _all_ steps below.
### Jira
- [x]
- https://issues.apache.org/jira/browse/AIRFLOW-4343
- In case you are adding a dependency, check if the license complies with
the [ASF 3rd Party License
Policy](https://www.apache.org/legal/resolved.html#category-x). - yes, Cat A
### Description
- [x] Now that the webserver is more stateless, if the scheduler is not
running the list of dags won't populate, making it harder for new
starters to work out what is going on.
**When scheduler has never run (I cleared Jobs table):**
**When scheduler not running (either crashed, or has stopped successfully)
(the time has a tooltip to display the full ISO8601 heartbeat)
Login screen/anything not subclassed from AirflowBaseView doesn't display
the message.
On Login (not changed, doesn't show warning):
### Tests
- [x] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason: added tests for helper functions on BaseJob
### Commits
- [x] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [x] In case of new functionality, my PR adds documentation that describes
how to use it.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
### Code Quality
- [x] Passes `flake8`
/cc @Fokko as discussed.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Created] (AIRFLOW-4343) Show warning in UI if scheduler is not running.
Ash Berlin-Taylor created AIRFLOW-4343: -- Summary: Show warning in UI if scheduler is not running. Key: AIRFLOW-4343 URL: https://issues.apache.org/jira/browse/AIRFLOW-4343 Project: Apache Airflow Issue Type: Sub-task Components: ui Reporter: Ash Berlin-Taylor Fix For: 1.10.4 Now that the webserver is more stateless, if the scheduler is not running the list of dags won't populate, making it harder for new starters to work out what is going on. We should display a warning if the scheduler isn't running (based on the job table heartbeat column) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-4344) Sync DAGs to DB from upgradedb calls
Ash Berlin-Taylor created AIRFLOW-4344:
--
Summary: Sync DAGs to DB from upgradedb calls
Key: AIRFLOW-4344
URL: https://issues.apache.org/jira/browse/AIRFLOW-4344
Project: Apache Airflow
Issue Type: Sub-task
Reporter: Ash Berlin-Taylor
Since the webserver is now stateless, the webserver won't get an up-to-date
list of dags if the scheduler is not running.
To make it slightly easier for users we should add sync dags in {{airflow
upgradedb}} (with a flag to disable it) too.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work started] (AIRFLOW-4343) Show warning in UI if scheduler is not running.
[ https://issues.apache.org/jira/browse/AIRFLOW-4343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on AIRFLOW-4343 started by Ash Berlin-Taylor. -- > Show warning in UI if scheduler is not running. > --- > > Key: AIRFLOW-4343 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4343 > Project: Apache Airflow > Issue Type: Sub-task > Components: ui >Reporter: Ash Berlin-Taylor >Assignee: Ash Berlin-Taylor >Priority: Major > Fix For: 1.10.4 > > > Now that the webserver is more stateless, if the scheduler is not running the > list of dags won't populate, making it harder for new starters to work out > what is going on. > We should display a warning if the scheduler isn't running (based on the job > table heartbeat column) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-4305) DAGs not showing in Web UI if Scheduler is not run at least once
[
https://issues.apache.org/jira/browse/AIRFLOW-4305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16820350#comment-16820350
]
Ash Berlin-Taylor commented on AIRFLOW-4305:
[~akoeltringer] Would this help? !Screen Shot 2019-04-17 at 18.47.56.png!
> DAGs not showing in Web UI if Scheduler is not run at least once
>
>
> Key: AIRFLOW-4305
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4305
> Project: Apache Airflow
> Issue Type: Bug
> Components: webapp
>Affects Versions: 1.10.3
> Environment: Linux (archlinux), Pyton 3.7
>Reporter: Andreas Költringer
>Priority: Major
> Fix For: 1.10.4
>
> Attachments: Screen Shot 2019-04-17 at 18.47.56.png
>
>
> Dags do not show up in the Web UI for new airflow instances.
> Steps to reproduce:
> {{mkdir temp && cd temp}}
> {{python -m venv venv}}
> {{source venv/bin/activate}}
> {{pip install
> apache-airflow[celery,rabbitmq,crypto,password,postgres]==1.10.3}}
> {{export AIRFLOW_HOME=$(pwd)}}
> {{airflow upgradedb}}
> {{airflow webserver}}
> Neither the "example dags" nor any dags dropped into {{dags/}} show up.
> Some more facts:
> * Interestingly, when replacing {{airflow upgradedb}} with {{airflow
> initdb}} it works as expected. I thought, using {{airflow upgradedb}} was
> safe b/c it is mentioned here:
> [https://medium.com/datareply/airflow-lesser-known-tips-tricks-and-best-practises-cf4d4a90f8f]
> * Also, the version with {{airflow upgradedb}} works as expected with
> {{v1.10.2}}.
>
> Updates:
> * {{initdb}} does insert DAGs into the database, while {{upgradedb}} does
> not: [https://github.com/apache/airflow/blob/master/airflow/utils/db.py#L292]
> *
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (AIRFLOW-4338) Change k8s pod_request_factory to use yaml safe_load
[
https://issues.apache.org/jira/browse/AIRFLOW-4338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Felix Uellendall updated AIRFLOW-4338:
--
Summary: Change k8s pod_request_factory to use yaml safe_load (was: Fix
yaml.load deprecation warning)
> Change k8s pod_request_factory to use yaml safe_load
>
>
> Key: AIRFLOW-4338
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4338
> Project: Apache Airflow
> Issue Type: Improvement
>Reporter: Felix Uellendall
>Assignee: Felix Uellendall
>Priority: Major
>
> Before Python yaml version 5.1 yaml has used the {{UnsafeLoader that has a
> potential security vulnerability. From now on there is a FullLoader}} that
> will be used by default that fixes these vulnerability. To suppress the
> warning we can explicitly set the loader.
> See [https://github.com/yaml/pyyaml/wiki/PyYAML-yaml.load(input)-Deprecation]
> for more details.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (AIRFLOW-4305) DAGs not showing in Web UI if Scheduler is not run at least once
[
https://issues.apache.org/jira/browse/AIRFLOW-4305?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ash Berlin-Taylor updated AIRFLOW-4305:
---
Attachment: Screen Shot 2019-04-17 at 18.47.56.png
> DAGs not showing in Web UI if Scheduler is not run at least once
>
>
> Key: AIRFLOW-4305
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4305
> Project: Apache Airflow
> Issue Type: Bug
> Components: webapp
>Affects Versions: 1.10.3
> Environment: Linux (archlinux), Pyton 3.7
>Reporter: Andreas Költringer
>Priority: Major
> Fix For: 1.10.4
>
> Attachments: Screen Shot 2019-04-17 at 18.47.56.png
>
>
> Dags do not show up in the Web UI for new airflow instances.
> Steps to reproduce:
> {{mkdir temp && cd temp}}
> {{python -m venv venv}}
> {{source venv/bin/activate}}
> {{pip install
> apache-airflow[celery,rabbitmq,crypto,password,postgres]==1.10.3}}
> {{export AIRFLOW_HOME=$(pwd)}}
> {{airflow upgradedb}}
> {{airflow webserver}}
> Neither the "example dags" nor any dags dropped into {{dags/}} show up.
> Some more facts:
> * Interestingly, when replacing {{airflow upgradedb}} with {{airflow
> initdb}} it works as expected. I thought, using {{airflow upgradedb}} was
> safe b/c it is mentioned here:
> [https://medium.com/datareply/airflow-lesser-known-tips-tricks-and-best-practises-cf4d4a90f8f]
> * Also, the version with {{airflow upgradedb}} works as expected with
> {{v1.10.2}}.
>
> Updates:
> * {{initdb}} does insert DAGs into the database, while {{upgradedb}} does
> not: [https://github.com/apache/airflow/blob/master/airflow/utils/db.py#L292]
> *
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] [airflow] dossett commented on issue #4064: AIRFLOW-3149 Support dataproc cluster deletion on ERROR
dossett commented on issue #4064: AIRFLOW-3149 Support dataproc cluster deletion on ERROR URL: https://github.com/apache/airflow/pull/4064#issuecomment-484193027 @OmerJog I went over that test in detail and it seems like the `with self.assertRaises(TypeError)` was a defensive measure to catch an exception within the test and not that raising a TypeError was part of the test specification. I've deleted that line to see if the rest of the test is successful. cc @piffall who added that line about 11 months ago This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] feluelle commented on issue #5120: [AIRFLOW-4338] Fix yaml.load deprecation warning
feluelle commented on issue #5120: [AIRFLOW-4338] Fix yaml.load deprecation warning URL: https://github.com/apache/airflow/pull/5120#issuecomment-484186539 Yes, the shortcut function would fix it I am fine with `SafeLoader` as well. @dimberman what do you think of this? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] ashb edited a comment on issue #5126: [AIRFLOW-4342] Use @cached_property instead of re-implementing it each time
ashb edited a comment on issue #5126: [AIRFLOW-4342] Use @cached_property instead of re-implementing it each time URL: https://github.com/apache/airflow/pull/5126#issuecomment-484171179 Oh yes good point. That was only used in one place (that I haven't looked at in a while) @milton0825 Replaced that now. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] ashb edited a comment on issue #5125: [AIRFLOW-4341] Remove `View.render()` already exists in fab.BaseView
ashb edited a comment on issue #5125: [AIRFLOW-4341] Remove `View.render()` already exists in fab.BaseView URL: https://github.com/apache/airflow/pull/5125#issuecomment-484170507 This breaks 404 it seems. Fixed that now. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] ashb commented on issue #5126: [AIRFLOW-4342] Use @cached_property instead of re-implementing it each time
ashb commented on issue #5126: [AIRFLOW-4342] Use @cached_property instead of re-implementing it each time URL: https://github.com/apache/airflow/pull/5126#issuecomment-484171179 Oh yes good point. That was only used in one place (that I haven't looked at in a while) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] ashb commented on issue #5125: [AIRFLOW-4341] Remove `View.render()` already exists in fab.BaseView
ashb commented on issue #5125: [AIRFLOW-4341] Remove `View.render()` already exists in fab.BaseView URL: https://github.com/apache/airflow/pull/5125#issuecomment-484170507 This breaks 404 it seems. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] feng-tao commented on issue #5118: [AIRFLOW-4315] Add monitoring API's to airflow
feng-tao commented on issue #5118: [AIRFLOW-4315] Add monitoring API's to airflow URL: https://github.com/apache/airflow/pull/5118#issuecomment-484168995 FYI, there is an ongoing effort to revamp the API framework(https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-13%3A+OpenAPI+3+based+API+definition). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] milton0825 commented on issue #5126: [AIRFLOW-4342] Use @cached_property instead of re-implementing it each time
milton0825 commented on issue #5126: [AIRFLOW-4342] Use @cached_property instead of re-implementing it each time URL: https://github.com/apache/airflow/pull/5126#issuecomment-484168441 We should probably replace this: https://github.com/apache/airflow/blob/6970b233964ee254bbb343ed8bdc906c2f7bd974/airflow/utils/decorators.py#L108 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] milton0825 commented on issue #5094: [AIRFLOW-4306] Global operator extra links
milton0825 commented on issue #5094: [AIRFLOW-4306] Global operator extra links URL: https://github.com/apache/airflow/pull/5094#issuecomment-484166658 PTAL @XD-DENG @Fokko This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] ashb commented on a change in pull request #5126: [AIRFLOW-4342] Use @cached_property instead of re-implementing it each time
ashb commented on a change in pull request #5126: [AIRFLOW-4342] Use @cached_property instead of re-implementing it each time URL: https://github.com/apache/airflow/pull/5126#discussion_r276332283 ## File path: setup.py ## @@ -309,6 +309,7 @@ def do_setup(): scripts=['airflow/bin/airflow'], install_requires=[ 'alembic>=0.9, <1.0', +'cached_property~=1.5', Review comment: BSD, which is Cat-A http://www.apache.org/legal/resolved.html#category-a This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] milton0825 commented on a change in pull request #5126: [AIRFLOW-4342] Use @cached_property instead of re-implementing it each time
milton0825 commented on a change in pull request #5126: [AIRFLOW-4342] Use @cached_property instead of re-implementing it each time URL: https://github.com/apache/airflow/pull/5126#discussion_r276331279 ## File path: setup.py ## @@ -309,6 +309,7 @@ def do_setup(): scripts=['airflow/bin/airflow'], install_requires=[ 'alembic>=0.9, <1.0', +'cached_property~=1.5', Review comment: Is it an Apache approved library? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] milton0825 commented on a change in pull request #5102: [AIRFLOW-4318] Create Google Cloud Translate Speech Operator
milton0825 commented on a change in pull request #5102: [AIRFLOW-4318] Create
Google Cloud Translate Speech Operator
URL: https://github.com/apache/airflow/pull/5102#discussion_r276329389
##
File path: airflow/contrib/operators/gcp_translate_speech_operator.py
##
@@ -0,0 +1,156 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from google.protobuf.json_format import MessageToDict
+
+
+from airflow import AirflowException
+from airflow.contrib.hooks.gcp_speech_to_text_hook import GCPSpeechToTextHook
+from airflow.contrib.hooks.gcp_translate_hook import CloudTranslateHook
+from airflow.models import BaseOperator
+from airflow.utils.decorators import apply_defaults
+
+
+class GcpTranslateSpeechOperator(BaseOperator):
+"""
+Recognizes speech in audio input and translates it.
+
+Note that it uses the first result from the recognition api response - the
one with the highest confidence
+In order to see other possible results please use
+:ref:`howto/operator:GcpSpeechToTextRecognizeSpeechOperator`
+and
+:ref:`howto/operator:CloudTranslateTextOperator`
+separately
+
+.. seealso::
+For more information on how to use this operator, take a look at the
guide:
+:ref:`howto/operator:GcpTranslateSpeechOperator`
+
+See https://cloud.google.com/translate/docs/translating-text
+
+Execute method returns string object with the translation
+
+This is a list of dictionaries queried value.
+Dictionary typically contains three keys (though not
+all will be present in all cases).
+
+* ``detectedSourceLanguage``: The detected language (as an
+ ISO 639-1 language code) of the text.
+* ``translatedText``: The translation of the text into the
+ target language.
+* ``input``: The corresponding input value.
+* ``model``: The model used to translate the text.
+
+Dictionary is set as XCom return value.
+
+:param audio: audio data to be recognized. See more:
+
https://googleapis.github.io/google-cloud-python/latest/speech/gapic/v1/types.html#google.cloud.speech_v1.types.RecognitionAudio
+:type audio: dict or google.cloud.speech_v1.types.RecognitionAudio
+
+:param config: information to the recognizer that specifies how to process
the request. See more:
+
https://googleapis.github.io/google-cloud-python/latest/speech/gapic/v1/types.html#google.cloud.speech_v1.types.RecognitionConfig
+:type config: dict or google.cloud.speech_v1.types.RecognitionConfig
+
+:type target_language: str
+:param target_language: The language to translate results into. This
+ is required by the API and defaults to
+ the target language of the current instance.
+
+:type format_: str or None
+:param format_: (Optional) One of ``text`` or ``html``, to specify
+ if the input text is plain text or HTML.
+
+:type source_language: str or None
+:param source_language: (Optional) The language of the text to
+ be translated.
+
+:type model: str or None
+:param model: (Optional) The model used to translate the text, such
+ as ``'base'`` or ``'nmt'``.
+
+:param project_id: Optional, Google Cloud Platform Project ID where the
Compute
+Engine Instance exists. If set to None or missing, the default
project_id from the GCP connection is
+used.
+:type project_id: str
+
+:param gcp_conn_id: Optional, The connection ID used to connect to Google
Cloud
+Platform. Defaults to 'google_cloud_default'.
+:type gcp_conn_id: str
+"""
+
+# [START translate_speech_template_fields]
+template_fields = ('target_language', 'format_', 'source_language',
'model', 'project_id', 'gcp_conn_id')
+# [END translate_speech_template_fields]
+
+@apply_defaults
+def __init__(
+self,
+audio,
+config,
+target_language,
+format_,
+source_language,
+model,
+project_id=None,
+gcp_conn_id='google_cloud_default',
+*args,
+**kwargs
+):
+super(GcpTranslateSpeechOperator, self).__init__(*args, **kwargs)
+self.audio = audio
+self.config = config
+
[GitHub] [airflow] milton0825 commented on issue #5102: [AIRFLOW-4318] Create Google Cloud Translate Speech Operator
milton0825 commented on issue #5102: [AIRFLOW-4318] Create Google Cloud Translate Speech Operator URL: https://github.com/apache/airflow/pull/5102#issuecomment-484164150 LGTM This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (AIRFLOW-4339) Cleared backfill tasks do not get picked by the scheduler
[ https://issues.apache.org/jira/browse/AIRFLOW-4339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ignas updated AIRFLOW-4339: --- Priority: Minor (was: Blocker) > Cleared backfill tasks do not get picked by the scheduler > - > > Key: AIRFLOW-4339 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4339 > Project: Apache Airflow > Issue Type: Bug > Components: backfill, scheduler >Affects Versions: 1.10.1 >Reporter: Ignas >Priority: Minor > > When a task in a successful backfill run is cleared DagRun with running state > is created but it is not picked by the scheduler. > If there are many of such DagRuns and the backfill command is executed, all > the DagRuns will be ran at the same time, not following the max_active_runs > parameter. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-4339) Cleared backfill tasks do not get picked by the scheduler
[ https://issues.apache.org/jira/browse/AIRFLOW-4339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16820264#comment-16820264 ] Ignas commented on AIRFLOW-4339: I am running LocalExecutor and they are picked by the backfill command, the main problem was that when tasks are cleared the backfill will pick all the DagRuns at once, ignoring the max_active_runs parameter. The workaround with --reset_dagruns, --ignore_dependencies and --task-regex is ok for me, this way I do not need to clear the tasks. > Cleared backfill tasks do not get picked by the scheduler > - > > Key: AIRFLOW-4339 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4339 > Project: Apache Airflow > Issue Type: Bug > Components: backfill, scheduler >Affects Versions: 1.10.1 >Reporter: Ignas >Priority: Blocker > > When a task in a successful backfill run is cleared DagRun with running state > is created but it is not picked by the scheduler. > If there are many of such DagRuns and the backfill command is executed, all > the DagRuns will be ran at the same time, not following the max_active_runs > parameter. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-4339) Cleared backfill tasks do not get picked by the scheduler
[ https://issues.apache.org/jira/browse/AIRFLOW-4339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16820258#comment-16820258 ] Tao Feng commented on AIRFLOW-4339: --- [~Ignas] I am in a conference, could take a look in the night, but all the clear tasks should be picked by scheduler. Which executor do you use for backfill? The patch has been running in our prod without any issues so far. > Cleared backfill tasks do not get picked by the scheduler > - > > Key: AIRFLOW-4339 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4339 > Project: Apache Airflow > Issue Type: Bug > Components: backfill, scheduler >Affects Versions: 1.10.1 >Reporter: Ignas >Priority: Blocker > > When a task in a successful backfill run is cleared DagRun with running state > is created but it is not picked by the scheduler. > If there are many of such DagRuns and the backfill command is executed, all > the DagRuns will be ran at the same time, not following the max_active_runs > parameter. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (AIRFLOW-4339) Cleared backfill tasks do not get picked by the scheduler
[ https://issues.apache.org/jira/browse/AIRFLOW-4339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16820207#comment-16820207 ] Ignas edited comment on AIRFLOW-4339 at 4/17/19 3:53 PM: - Thanks Ash, I thought that --reset_dagruns will reset all the tasks in the dag no matter if --ignore_dependencies and task regex is used or not was (Author: ignas): Together with --reset_dagruns? I do use task-regex, the problem is that after clearing the tasks I have a few hundreds of "running" DagRuns, and if I then run backfill command it will start executing all of it which completely hangs the whole server > Cleared backfill tasks do not get picked by the scheduler > - > > Key: AIRFLOW-4339 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4339 > Project: Apache Airflow > Issue Type: Bug > Components: backfill, scheduler >Affects Versions: 1.10.1 >Reporter: Ignas >Priority: Blocker > > When a task in a successful backfill run is cleared DagRun with running state > is created but it is not picked by the scheduler. > If there are many of such DagRuns and the backfill command is executed, all > the DagRuns will be ran at the same time, not following the max_active_runs > parameter. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AIRFLOW-3769) Open Redirect Vulnerability in Admin Create Variable Page
[ https://issues.apache.org/jira/browse/AIRFLOW-3769?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ash Berlin-Taylor resolved AIRFLOW-3769. Resolution: Fixed Fix Version/s: 1.10.3 I fixed some URL handling/validation in 1.10.3 > Open Redirect Vulnerability in Admin Create Variable Page > - > > Key: AIRFLOW-3769 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3769 > Project: Apache Airflow > Issue Type: Bug > Components: security >Affects Versions: 1.10.1 >Reporter: Media Rest >Assignee: Media Rest >Priority: Critical > Fix For: 1.10.3 > > > In the /admin/variable/new page, it is possible to inject an open redirect > URL into the URL query parameter which is executed in the List anchor of the > page. This can be exploited to redirect an admin to a malicious domain. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] [airflow] codecov-io commented on issue #5125: [AIRFLOW-4341] Remove `View.render()` already exists in fab.BaseView
codecov-io commented on issue #5125: [AIRFLOW-4341] Remove `View.render()` already exists in fab.BaseView URL: https://github.com/apache/airflow/pull/5125#issuecomment-484148040 # [Codecov](https://codecov.io/gh/apache/airflow/pull/5125?src=pr=h1) Report > Merging [#5125](https://codecov.io/gh/apache/airflow/pull/5125?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/a511b4877b88c5154474309e3acd7a60706415c9?src=pr=desc) will **decrease** coverage by `<.01%`. > The diff coverage is `80.95%`. [](https://codecov.io/gh/apache/airflow/pull/5125?src=pr=tree) ```diff @@Coverage Diff @@ ## master#5125 +/- ## == - Coverage 77.96% 77.95% -0.01% == Files 465 465 Lines 2981629814 -2 == - Hits2324523242 -3 - Misses 6571 6572 +1 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/5125?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/www/views.py](https://codecov.io/gh/apache/airflow/pull/5125/diff?src=pr=tree#diff-YWlyZmxvdy93d3cvdmlld3MucHk=) | `76.35% <80.95%> (-0.04%)` | :arrow_down: | | [airflow/models/taskinstance.py](https://codecov.io/gh/apache/airflow/pull/5125/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvdGFza2luc3RhbmNlLnB5) | `92.42% <0%> (-0.18%)` | :arrow_down: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/5125?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/5125?src=pr=footer). Last update [a511b48...43ae410](https://codecov.io/gh/apache/airflow/pull/5125?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Assigned] (AIRFLOW-3769) Open Redirect Vulnerability in Admin Create Variable Page
[ https://issues.apache.org/jira/browse/AIRFLOW-3769?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anonymous reassigned AIRFLOW-3769: -- Assignee: Ash Berlin-Taylor (was: Media Rest) > Open Redirect Vulnerability in Admin Create Variable Page > - > > Key: AIRFLOW-3769 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3769 > Project: Apache Airflow > Issue Type: Bug > Components: security >Affects Versions: 1.10.1 >Reporter: Media Rest >Assignee: Ash Berlin-Taylor >Priority: Critical > Fix For: 1.10.3 > > > In the /admin/variable/new page, it is possible to inject an open redirect > URL into the URL query parameter which is executed in the List anchor of the > page. This can be exploited to redirect an admin to a malicious domain. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (AIRFLOW-3769) Open Redirect Vulnerability in Admin Create Variable Page
[ https://issues.apache.org/jira/browse/AIRFLOW-3769?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anonymous reassigned AIRFLOW-3769: -- Assignee: Media Rest > Open Redirect Vulnerability in Admin Create Variable Page > - > > Key: AIRFLOW-3769 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3769 > Project: Apache Airflow > Issue Type: Bug > Components: security >Affects Versions: 1.10.1 >Reporter: Media Rest >Assignee: Media Rest >Priority: Critical > > In the /admin/variable/new page, it is possible to inject an open redirect > URL into the URL query parameter which is executed in the List anchor of the > page. This can be exploited to redirect an admin to a malicious domain. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-55) Add HDFS Log Support
[ https://issues.apache.org/jira/browse/AIRFLOW-55?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16820208#comment-16820208 ] Ash Berlin-Taylor commented on AIRFLOW-55: -- It seems to have been closed without merging - I find no reference to [AIRFLOW-55] in our commit history anyway > Add HDFS Log Support > > > Key: AIRFLOW-55 > URL: https://issues.apache.org/jira/browse/AIRFLOW-55 > Project: Apache Airflow > Issue Type: New Feature > Components: hooks >Affects Versions: 1.7.0 >Reporter: Wu Xiang >Assignee: Daniel Huang >Priority: Major > Labels: features > Fix For: 1.8.0 > > > To support save task logs on HDFS. > PR: > https://github.com/apache/incubator-airflow/pull/1409 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] [airflow] ashb commented on issue #5088: [AIRFLOW-4330] - Change _create_or_update_secret to work with python3
ashb commented on issue #5088: [AIRFLOW-4330] - Change _create_or_update_secret to work with python3 URL: https://github.com/apache/airflow/pull/5088#issuecomment-484144305 You say "covered by preexisting tests" except they weren't failing before on Python3, which makes me think this entire code path isn't covered by tests. Would you be able to have a look at how much work it is to at some minimal tests that hit this? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Comment Edited] (AIRFLOW-4297) Manually triggerd DAG with no schedule_interval breaks scheduler
[
https://issues.apache.org/jira/browse/AIRFLOW-4297?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16820215#comment-16820215
]
Xiaodong DENG edited comment on AIRFLOW-4297 at 4/17/19 3:38 PM:
-
I realise the current implementation of sla_manage() is quite "poor".
* if there is no successful DagRun at all, SLA will not work. This is because
only the DagRuns after the `max_tis` will be checked, while `max_tis` is based
on SUCCESS/SKIPPED records.
* the `while` loop will create SLA miss records even if for the DagRuns which
are not created yet.
We may need to do a relatively big refactoring on this part.
was (Author: xd-deng):
I realise the current implementation of sla_manage() is quite "poor".
* if there is no successful DagRun at all, SLA will not work. This is because
only the DagRuns after the `max_tis` will be checked, while `max_tis` is based
on SUCCESS/SKIPPED records.
* the `while` loop will create SLA miss records even if for the DagRuns which
are not created yet.
> Manually triggerd DAG with no schedule_interval breaks scheduler
>
>
> Key: AIRFLOW-4297
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4297
> Project: Apache Airflow
> Issue Type: Bug
>Affects Versions: 1.10.3
>Reporter: Ash Berlin-Taylor
>Assignee: Xiaodong DENG
>Priority: Blocker
> Fix For: 1.10.4
>
>
> {code:title=example_dag.py}
> timezone = "UTC"
> local_tz = pendulum.timezone(timezone)
> start_date = datetime.datetime.strptime('2019-03-28 07:57:00', "%Y-%m-%d
> %H:%M:%S")
> start_date = start_date.replace(tzinfo=local_tz)
> default_args = {
> 'owner': 'DaniRC'
> }
> dag = DAG('testsla',
> default_args=default_args,
> start_date=start_date,
> concurrency=1,
> max_active_runs=1,
> default_view='tree',
> orientation='TB',
> catchup=False,
> schedule_interval=None
> )
> {code}
> If this DAG is triggered, then this error occurs:
> {noformat}
> Process DagFileProcessor5303-Process:
> Traceback (most recent call last):
> File "/usr/lib/python3.6/multiprocessing/process.py", line 258, in
> _bootstrap
> self.run()
> File "/usr/lib/python3.6/multiprocessing/process.py", line 93, in run
> self._target(*self._args, **self._kwargs)
> File "/usr/lib/python3.6/site-packages/airflow/jobs.py", line 402, in helper
> pickle_dags)
> File "/usr/lib/python3.6/site-packages/airflow/utils/db.py", line 73, in
> wrapper
> return func(*args, **kwargs)
> File "/usr/lib/python3.6/site-packages/airflow/jobs.py", line 1760, in
> process_file
> self._process_dags(dagbag, dags, ti_keys_to_schedule)
> File "/usr/lib/python3.6/site-packages/airflow/jobs.py", line 1452, in
> _process_dags
> self.manage_slas(dag)
> File "/usr/lib/python3.6/site-packages/airflow/utils/db.py", line 73, in
> wrapper
> return func(*args, **kwargs)
> File "/usr/lib/python3.6/site-packages/airflow/jobs.py", line 662, in
> manage_slas
> while dttm < timezone.utcnow():
> TypeError: '<' not supported between instances of 'NoneType' and
> 'datetime.datetime'
> {noformat}
> After this happens the scheduler also won't schedule any further tasks for
> _ANY_ dag.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AIRFLOW-4297) Manually triggerd DAG with no schedule_interval breaks scheduler
[
https://issues.apache.org/jira/browse/AIRFLOW-4297?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16820215#comment-16820215
]
Xiaodong DENG commented on AIRFLOW-4297:
I realise the current implementation of sla_manage() is quite "poor".
* if there is no successful DagRun at all, SLA will not work. This is because
only the DagRuns after the `max_tis` will be checked, while `max_tis` is based
on SUCCESS/SKIPPED records.
* the `while` loop will create SLA miss records even if for the DagRuns which
are not created yet.
> Manually triggerd DAG with no schedule_interval breaks scheduler
>
>
> Key: AIRFLOW-4297
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4297
> Project: Apache Airflow
> Issue Type: Bug
>Affects Versions: 1.10.3
>Reporter: Ash Berlin-Taylor
>Assignee: Xiaodong DENG
>Priority: Blocker
> Fix For: 1.10.4
>
>
> {code:title=example_dag.py}
> timezone = "UTC"
> local_tz = pendulum.timezone(timezone)
> start_date = datetime.datetime.strptime('2019-03-28 07:57:00', "%Y-%m-%d
> %H:%M:%S")
> start_date = start_date.replace(tzinfo=local_tz)
> default_args = {
> 'owner': 'DaniRC'
> }
> dag = DAG('testsla',
> default_args=default_args,
> start_date=start_date,
> concurrency=1,
> max_active_runs=1,
> default_view='tree',
> orientation='TB',
> catchup=False,
> schedule_interval=None
> )
> {code}
> If this DAG is triggered, then this error occurs:
> {noformat}
> Process DagFileProcessor5303-Process:
> Traceback (most recent call last):
> File "/usr/lib/python3.6/multiprocessing/process.py", line 258, in
> _bootstrap
> self.run()
> File "/usr/lib/python3.6/multiprocessing/process.py", line 93, in run
> self._target(*self._args, **self._kwargs)
> File "/usr/lib/python3.6/site-packages/airflow/jobs.py", line 402, in helper
> pickle_dags)
> File "/usr/lib/python3.6/site-packages/airflow/utils/db.py", line 73, in
> wrapper
> return func(*args, **kwargs)
> File "/usr/lib/python3.6/site-packages/airflow/jobs.py", line 1760, in
> process_file
> self._process_dags(dagbag, dags, ti_keys_to_schedule)
> File "/usr/lib/python3.6/site-packages/airflow/jobs.py", line 1452, in
> _process_dags
> self.manage_slas(dag)
> File "/usr/lib/python3.6/site-packages/airflow/utils/db.py", line 73, in
> wrapper
> return func(*args, **kwargs)
> File "/usr/lib/python3.6/site-packages/airflow/jobs.py", line 662, in
> manage_slas
> while dttm < timezone.utcnow():
> TypeError: '<' not supported between instances of 'NoneType' and
> 'datetime.datetime'
> {noformat}
> After this happens the scheduler also won't schedule any further tasks for
> _ANY_ dag.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (AIRFLOW-55) Add HDFS Log Support
[ https://issues.apache.org/jira/browse/AIRFLOW-55?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ash Berlin-Taylor updated AIRFLOW-55: - Affects Version/s: (was: 1.7.0) Fix Version/s: (was: 1.8.0) Component/s: logging > Add HDFS Log Support > > > Key: AIRFLOW-55 > URL: https://issues.apache.org/jira/browse/AIRFLOW-55 > Project: Apache Airflow > Issue Type: New Feature > Components: hooks, logging >Reporter: Wu Xiang >Assignee: Daniel Huang >Priority: Major > Labels: features > > To support save task logs on HDFS. > PR: > https://github.com/apache/incubator-airflow/pull/1409 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-4339) Cleared backfill tasks do not get picked by the scheduler
[ https://issues.apache.org/jira/browse/AIRFLOW-4339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16820207#comment-16820207 ] Ignas commented on AIRFLOW-4339: Together with --reset_dagruns? I do use task-regex, the problem is that after clearing the tasks I have a few hundreds of "running" DagRuns, and if I then run backfill command it will start executing all of it which completely hangs the whole server > Cleared backfill tasks do not get picked by the scheduler > - > > Key: AIRFLOW-4339 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4339 > Project: Apache Airflow > Issue Type: Bug > Components: backfill, scheduler >Affects Versions: 1.10.1 >Reporter: Ignas >Priority: Blocker > > When a task in a successful backfill run is cleared DagRun with running state > is created but it is not picked by the scheduler. > If there are many of such DagRuns and the backfill command is executed, all > the DagRuns will be ran at the same time, not following the max_active_runs > parameter. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (AIRFLOW-4339) Cleared backfill tasks do not get picked by the scheduler
[
https://issues.apache.org/jira/browse/AIRFLOW-4339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16820201#comment-16820201
]
Ash Berlin-Taylor edited comment on AIRFLOW-4339 at 4/17/19 3:30 PM:
-
In which case a combo of {{\-\-ignore_dependencies}} and {{--task-regex}}
was (Author: ashb):
In which case a combo of {{--ignore_dependencies}} and {{--task-regex}}
> Cleared backfill tasks do not get picked by the scheduler
> -
>
> Key: AIRFLOW-4339
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4339
> Project: Apache Airflow
> Issue Type: Bug
> Components: backfill, scheduler
>Affects Versions: 1.10.1
>Reporter: Ignas
>Priority: Blocker
>
> When a task in a successful backfill run is cleared DagRun with running state
> is created but it is not picked by the scheduler.
> If there are many of such DagRuns and the backfill command is executed, all
> the DagRuns will be ran at the same time, not following the max_active_runs
> parameter.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AIRFLOW-4339) Cleared backfill tasks do not get picked by the scheduler
[
https://issues.apache.org/jira/browse/AIRFLOW-4339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16820201#comment-16820201
]
Ash Berlin-Taylor commented on AIRFLOW-4339:
In which case a combo of {{--ignore_dependencies}} and {{--task-regex}}
> Cleared backfill tasks do not get picked by the scheduler
> -
>
> Key: AIRFLOW-4339
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4339
> Project: Apache Airflow
> Issue Type: Bug
> Components: backfill, scheduler
>Affects Versions: 1.10.1
>Reporter: Ignas
>Priority: Blocker
>
> When a task in a successful backfill run is cleared DagRun with running state
> is created but it is not picked by the scheduler.
> If there are many of such DagRuns and the backfill command is executed, all
> the DagRuns will be ran at the same time, not following the max_active_runs
> parameter.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AIRFLOW-4339) Cleared backfill tasks do not get picked by the scheduler
[ https://issues.apache.org/jira/browse/AIRFLOW-4339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16820199#comment-16820199 ] Ignas commented on AIRFLOW-4339: --reset_dagruns will backfill all the tasks in the dag, in my case I only need to backfill one of the tasks. > Cleared backfill tasks do not get picked by the scheduler > - > > Key: AIRFLOW-4339 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4339 > Project: Apache Airflow > Issue Type: Bug > Components: backfill, scheduler >Affects Versions: 1.10.1 >Reporter: Ignas >Priority: Blocker > > When a task in a successful backfill run is cleared DagRun with running state > is created but it is not picked by the scheduler. > If there are many of such DagRuns and the backfill command is executed, all > the DagRuns will be ran at the same time, not following the max_active_runs > parameter. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-4299) Upgrade to Celery 4.3.0 to fix crashing workers
[ https://issues.apache.org/jira/browse/AIRFLOW-4299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16820193#comment-16820193 ] ASF subversion and git services commented on AIRFLOW-4299: -- Commit 85ba1f1b0be0d34e07df008ac3248b5c93e1e867 in airflow's branch refs/heads/v1-10-stable from Kaxil Naik [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=85ba1f1 ] [AIRFLOW-4299] Upgrade to Celery 4.3.0 to fix crashing workers (#5116) > Upgrade to Celery 4.3.0 to fix crashing workers > --- > > Key: AIRFLOW-4299 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4299 > Project: Apache Airflow > Issue Type: Bug > Components: celery >Affects Versions: 1.10.2 >Reporter: Teresa Martyny >Assignee: Kaxil Naik >Priority: Major > Fix For: 1.10.4 > > > Celery needs to be updated to 4.3.0 to fix this issue: > [https://github.com/celery/billiard/issues/260] > When doing warm restarts all Celery workers crash. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-3241) Incorrect Connection Parameter & Indentation in Gcs Sensor docstrings
[
https://issues.apache.org/jira/browse/AIRFLOW-3241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16820183#comment-16820183
]
ASF subversion and git services commented on AIRFLOW-3241:
--
Commit 59499f09f911cc3799b2e8248b04cce750b4b188 in airflow's branch
refs/heads/v1-10-test from Kaxil Naik
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=59499f0 ]
[AIRFLOW-3241] Remove Invalid template ext in GCS Sensors (#4076)
- Remove `template_ext = ('.sql',)`
- Fix Docstrings (Incorrect connection name and indentation)
> Incorrect Connection Parameter & Indentation in Gcs Sensor docstrings
> -
>
> Key: AIRFLOW-3241
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3241
> Project: Apache Airflow
> Issue Type: Improvement
> Components: docs, gcp
>Reporter: Kaxil Naik
>Assignee: Kaxil Naik
>Priority: Trivial
> Fix For: 1.10.4
>
>
> * It uses `google_cloud_storage_conn_id` instead of `google_cloud_conn_id`
> * Docstrings have unncessary indentation which causes the docs to not parse it
> * The GCS sensor also contains an incorrect template extension
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
