[GitHub] XD-DENG commented on issue #3939: [AIRFLOW-3104] Add .airflowignore info into doc
XD-DENG commented on issue #3939: [AIRFLOW-3104] Add .airflowignore info into doc URL: https://github.com/apache/incubator-airflow/pull/3939#issuecomment-424966969 Hi bad @ashb who didn't write test, may you have another look? ;-) The a few points you highlighted are covered: - Patterns will not only be checked for file name itself. The path will also be one part of the pattern matching (this is also reflected in my example) - Each subfolder can have its own `.airflowignore` file. This is described in the last paragraph when I introduce the scope of `.airflowignore` file. I also tried to run some local tests and confirmed it's working as expected. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] JacobHayes commented on a change in pull request #3952: [AIRFLOW-XXX] Update GCS logging docs for latest code
JacobHayes commented on a change in pull request #3952: [AIRFLOW-XXX] Update
GCS logging docs for latest code
URL: https://github.com/apache/incubator-airflow/pull/3952#discussion_r220782677
##
File path: docs/howto/write-logs.rst
##
@@ -89,54 +89,21 @@ Writing Logs to Google Cloud Storage
Follow the steps below to enable Google Cloud Storage logging.
-#. Airflow's logging system requires a custom .py file to be located in the
``PYTHONPATH``, so that it's importable from Airflow. Start by creating a
directory to store the config file. ``$AIRFLOW_HOME/config`` is recommended.
-#. Create empty files called ``$AIRFLOW_HOME/config/log_config.py`` and
``$AIRFLOW_HOME/config/__init__.py``.
-#. Copy the contents of ``airflow/config_templates/airflow_local_settings.py``
into the ``log_config.py`` file that was just created in the step above.
-#. Customize the following portions of the template:
-
-.. code-block:: bash
-
-# Add this variable to the top of the file. Note the trailing slash.
-GCS_LOG_FOLDER = 'gs:///'
-
-# Rename DEFAULT_LOGGING_CONFIG to LOGGING CONFIG
-LOGGING_CONFIG = ...
-
-# Add a GCSTaskHandler to the 'handlers' block of the LOGGING_CONFIG
variable
-'gcs.task': {
-'class': 'airflow.utils.log.gcs_task_handler.GCSTaskHandler',
-'formatter': 'airflow.task',
-'base_log_folder': os.path.expanduser(BASE_LOG_FOLDER),
-'gcs_log_folder': GCS_LOG_FOLDER,
-'filename_template': FILENAME_TEMPLATE,
-},
-
-# Update the airflow.task and airflow.task_runner blocks to be
'gcs.task' instead of 'file.task'.
-'loggers': {
-'airflow.task': {
-'handlers': ['gcs.task'],
-...
-},
-'airflow.task_runner': {
-'handlers': ['gcs.task'],
-...
-},
-'airflow': {
-'handlers': ['console'],
-...
-},
-}
-
-#. Make sure a Google Cloud Platform connection hook has been defined in
Airflow. The hook should have read and write access to the Google Cloud Storage
bucket defined above in ``GCS_LOG_FOLDER``.
-
-#. Update ``$AIRFLOW_HOME/airflow.cfg`` to contain:
+To enable this feature, ``airflow.cfg`` must be configured as in this
+example:
-.. code-block:: bash
+.. code-block:: bash
-task_log_reader = gcs.task
-logging_config_class = log_config.LOGGING_CONFIG
-remote_log_conn_id =
+[core]
+# Airflow can store logs remotely in AWS S3. Users must supply a remote
+# location URL (starting with either 's3://...') and an Airflow connection
+# id that provides access to the storage location.
+remote_logging_enabled = True
Review comment:
This variable should be `remote_logging`
https://github.com/apache/incubator-airflow/blob/53b89b98371c7bb993b242c341d3941e9ce09f9a/airflow/config_templates/airflow_local_settings.py#L173
Also, the comment above this line should probably be changed to reference
GCS (and `gs://`) instead of S3.
Other than that, this seems to match my working setup with airflow GCS
logging.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] jmcarp closed pull request #3959: [WIP] Backfill unit test for postgres operator.
jmcarp closed pull request #3959: [WIP] Backfill unit test for postgres
operator.
URL: https://github.com/apache/incubator-airflow/pull/3959
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/tests/operators/test_postgres_operator.py
b/tests/operators/test_postgres_operator.py
new file mode 100644
index 00..9a1dc5db7e
--- /dev/null
+++ b/tests/operators/test_postgres_operator.py
@@ -0,0 +1,51 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import unittest
+
+from airflow.operators.postgres_operator import PostgresOperator
+
+try:
+from unittest import mock
+except ImportError:
+try:
+import mock
+except ImportError:
+mock = None
+
+
+class TestPostgresOperator(unittest.TestCase):
+
+@mock.patch('airflow.operators.postgres_operator.PostgresHook')
+def test_execute(self, mock_hook):
+operator = PostgresOperator(
+task_id='postgres',
+sql='select 1',
+)
+
+operator.execute(None)
+mock_hook.assert_called_once_with(
+postgres_conn_id='postgres_default',
+schema=None,
+)
+mock_hook.return_value.run.assert_called_once_with(
+'select 1',
+False,
+parameters=None,
+)
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Commented] (AIRFLOW-668) Configuration parsing doesn't work properly with python 3
[
https://issues.apache.org/jira/browse/AIRFLOW-668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16629663#comment-16629663
]
ASF GitHub Bot commented on AIRFLOW-668:
stale[bot] closed pull request #1918: [AIRFLOW-668] Fix TypeError of
run_command on python3
URL: https://github.com/apache/incubator-airflow/pull/1918
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Configuration parsing doesn't work properly with python 3
> -
>
> Key: AIRFLOW-668
> URL: https://issues.apache.org/jira/browse/AIRFLOW-668
> Project: Apache Airflow
> Issue Type: Bug
> Environment: Airflow version: v1.7.1.3
> - Airflow components: webserver and scheduler with a postgres database and
> CeleryExecutor
> - Python Version: 3.4.5
>Reporter: Rafael Gomes Fernandes
>Assignee: Rafael Gomes Fernandes
>Priority: Major
>
> The problem is: if you use python3 and the '_cmd' on the config file airflow
> will not start due the error:
> {noformat}
> File
> "~/test/env/airflow3/lib/python3.4/site-packages/airflow/configuration.py",
> line 447, in _validate
> "sqlite" in self.get('core', 'sql_alchemy_conn')):
> TypeError: 'str' does not support the buffer interface
> {noformat}
> To reproduce the problem change the following line on airflow.cfg:
> {code:title=airflow.cfg|borderStyle=solid}
> sql_alchemy_conn_cmd = echo sqlite:~/airflow/airflow.db
> {code}
> The solution is change the following run_command method's line on
> airflow/configuration.py:
> {code:title=airflow/configuration.py|borderStyle=solid}
> command.split(), stdout=subprocess.PIPE, stderr=subprocess.PIPE,
> universal_newlines=True)
> {code}
> By setting the universal_newlines to true the file objects stdout and stderr
> are opened as text files and treated as string in python 2 and python 3
> avoiding the error.
> run_command with universal_newlines=True:
> When using python 3 output type: and no error.
> When using python 2 output type: and no error.
> run_command as it is :
> When using python 3 output type: and TypeError.
> When using python 2 output type: and no error.
> I have tested the change with Travis CI and it passed, but when I tried to
> run tox or unit test I found this problems:
> {code:title=run_unit_tests.sh|borderStyle=solid}
> ./run_unit_tests.sh -q -c airflow.cfg
> Initializing the DB
> Starting the unit tests with the following nose arguments: -q -c airflow.cfg
> ==
> ERROR: Failure: OperationalError ((sqlite3.OperationalError) no such table:
> task_instance [SQL: 'DELETE FROM task_instance WHERE task_instance.dag_id =
> ?'] [parameters: ('unit_tests',)])
> --
> Traceback (most recent call last):
> File
> "/home/varrun/test/env/airflow3/lib/python3.4/site-packages/nose/failure.py",
> line 39, in runTest
> raise self.exc_val.with_traceback(self.tb)
> File
> "/home/varrun/test/env/airflow3/lib/python3.4/site-packages/nose/loader.py",
> line 418, in loadTestsFromName
> addr.filename, addr.module)
> File
> "/home/varrun/test/env/airflow3/lib/python3.4/site-packages/nose/importer.py",
> line 47, in importFromPath
> return self.importFromDir(dir_path, fqname)
> File
> "/home/varrun/test/env/airflow3/lib/python3.4/site-packages/nose/importer.py",
> line 94, in importFromDir
> mod = load_module(part_fqname, fh, filename, desc)
> File "/home/varrun/test/env/airflow3/lib/python3.4/imp.py", line 245, in
> load_module
> return load_package(name, filename)
> File "/home/varrun/test/env/airflow3/lib/python3.4/imp.py", line 217, in
> load_package
> return methods.load()
> File "", line 1220, in load
> File "", line 1200, in _load_unlocked
> File "", line 1129, in _exec
> File "", line 1471, in exec_module
> File "", line 321, in _call_with_frames_removed
> File "/home/varrun/dev/incubator-airflow/tests/__init__.py", line 18, in
>
> from .contrib import *
> File "/home/varrun/dev/incubator-airflow/tests/contrib/__init__.py", line
> 16, in
> from .operators import *
> File
> "/home/varrun/dev/incubator-airflow/tests/contrib/operators/__init__.py",
> line 17, in
> from .ssh_execute_operator import *
> File
> "/home/varrun/dev/incubator-airflow/tests/contrib/operators/ssh_execute_operator.py",
> line 37, in
> reset()
> File
> "/home/varrun/dev/incubator-airflow/tests/contrib/operators/ssh_execute_operator.py",
> line 33,
[jira] [Commented] (AIRFLOW-419) Setting task instance status through UI causes exception
[
https://issues.apache.org/jira/browse/AIRFLOW-419?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16629664#comment-16629664
]
ASF GitHub Bot commented on AIRFLOW-419:
stale[bot] closed pull request #2537: [AIRFLOW-419] Fix microsecond-precision
datetime decode issue
URL: https://github.com/apache/incubator-airflow/pull/2537
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Setting task instance status through UI causes exception
>
>
> Key: AIRFLOW-419
> URL: https://issues.apache.org/jira/browse/AIRFLOW-419
> Project: Apache Airflow
> Issue Type: Bug
> Components: ui
>Affects Versions: 1.7.1
> Environment: Using the Celery executor and PostgreSQL as the backing
> DB.
>Reporter: Mjumbe Poe
>Priority: Major
>
> When using postgres, datetimes are returned down to the microsecond, and
> breaks the datetime parsing in
> {{TaskInstanceModelView.set_task_instance_state}} at
> https://github.com/mjumbewu/incubator-airflow/blob/55985ef/airflow/www/views.py#L2252.
> This blocks users from updating the task instance status through the UI.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AIRFLOW-428) Scheduler restart (via SIGKILL) leaves orphaned celery and airflow run processes
[ https://issues.apache.org/jira/browse/AIRFLOW-428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16629665#comment-16629665 ] ASF GitHub Bot commented on AIRFLOW-428: stale[bot] closed pull request #1732: [AIRFLOW-428] Clean shutdown celery on SIGTERM URL: https://github.com/apache/incubator-airflow/pull/1732 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Scheduler restart (via SIGKILL) leaves orphaned celery and airflow run > processes > > > Key: AIRFLOW-428 > URL: https://issues.apache.org/jira/browse/AIRFLOW-428 > Project: Apache Airflow > Issue Type: Bug >Reporter: Li Xuanji >Assignee: Li Xuanji >Priority: Major > > Airflow worker currently becomes a celery process. Sending a SIGTERM to it > performs a warm shutdown which waits for tasks to exit. We use runit which > will send a SIGKILL after some time passes; the SIGKILL causes the airflow > run commands to be orphaned (parented onto PID 1 init, celery does not know > about them). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] stale[bot] closed pull request #2490: User should be able to decide TemporaryDirectory root in a BashOperator
stale[bot] closed pull request #2490: User should be able to decide TemporaryDirectory root in a BashOperator URL: https://github.com/apache/incubator-airflow/pull/2490 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] stale[bot] closed pull request #1918: [AIRFLOW-668] Fix TypeError of run_command on python3
stale[bot] closed pull request #1918: [AIRFLOW-668] Fix TypeError of run_command on python3 URL: https://github.com/apache/incubator-airflow/pull/1918 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] stale[bot] closed pull request #2666: Updating column header in table to match column.
stale[bot] closed pull request #2666: Updating column header in table to match column. URL: https://github.com/apache/incubator-airflow/pull/2666 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] stale[bot] closed pull request #1732: [AIRFLOW-428] Clean shutdown celery on SIGTERM
stale[bot] closed pull request #1732: [AIRFLOW-428] Clean shutdown celery on SIGTERM URL: https://github.com/apache/incubator-airflow/pull/1732 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] jmcarp opened a new pull request #3959: [WIP] Backfill unit test for postgres operator.
jmcarp opened a new pull request #3959: [WIP] Backfill unit test for postgres
operator.
URL: https://github.com/apache/incubator-airflow/pull/3959
Make sure you have checked _all_ steps below.
### Jira
- [ ] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-XXX
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
### Description
- [ ] Here are some details about my PR, including screenshots of any UI
changes:
### Tests
- [ ] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [ ] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [ ] In case of new functionality, my PR adds documentation that describes
how to use it.
- When adding new operators/hooks/sensors, the autoclass documentation
generation needs to be added.
### Code Quality
- [ ] Passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] akosel opened a new pull request #3958: [AIRFLOW-3124] Fix RBAC webserver debug mode
akosel opened a new pull request #3958: [AIRFLOW-3124] Fix RBAC webserver debug
mode
URL: https://github.com/apache/incubator-airflow/pull/3958
Make sure you have checked _all_ steps below.
### Jira
- [x] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW-3124) issues and references
them in the PR title.
### Description
- [x] Here are some details about my PR, including screenshots of any UI
changes:
The command `airflow webserver -d` crashes when `settings.RBAC == True`.
This is because `create_app` currently returns `app, appbuilder`, which causes
the app to crash with an `AttributeError`. Reading through the code, it doesn't
look like we rely on `appbuilder` being returned by `create_app` anywhere, so
it should be safe to remove this.
### Tests
- [x] My PR adds the following unit tests
I set up the CLI to pass in the `unit_test_mode` setting to the flask app so
we can disable the werkzeug reloader when running tests. If the reloader runs,
there is an extra monitoring process left running after killing the main
webserver process. This causes conflicts if multiple tests rely on being able
to start a webserver (as the process will not be able to bind to the default
socket).
### Commits
- [x] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Code Quality
- [x] Passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Commented] (AIRFLOW-3124) Broken webserver debug mode (RBAC)
[
https://issues.apache.org/jira/browse/AIRFLOW-3124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16629528#comment-16629528
]
ASF GitHub Bot commented on AIRFLOW-3124:
-
akosel opened a new pull request #3958: [AIRFLOW-3124] Fix RBAC webserver debug
mode
URL: https://github.com/apache/incubator-airflow/pull/3958
Make sure you have checked _all_ steps below.
### Jira
- [x] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW-3124) issues and references
them in the PR title.
### Description
- [x] Here are some details about my PR, including screenshots of any UI
changes:
The command `airflow webserver -d` crashes when `settings.RBAC == True`.
This is because `create_app` currently returns `app, appbuilder`, which causes
the app to crash with an `AttributeError`. Reading through the code, it doesn't
look like we rely on `appbuilder` being returned by `create_app` anywhere, so
it should be safe to remove this.
### Tests
- [x] My PR adds the following unit tests
I set up the CLI to pass in the `unit_test_mode` setting to the flask app so
we can disable the werkzeug reloader when running tests. If the reloader runs,
there is an extra monitoring process left running after killing the main
webserver process. This causes conflicts if multiple tests rely on being able
to start a webserver (as the process will not be able to bind to the default
socket).
### Commits
- [x] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Code Quality
- [x] Passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Broken webserver debug mode (RBAC)
> --
>
> Key: AIRFLOW-3124
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3124
> Project: Apache Airflow
> Issue Type: Bug
> Components: webapp, webserver
>Reporter: Aaron Kosel
>Assignee: Aaron Kosel
>Priority: Minor
>
> {code:java}
> Traceback (most recent call last):
> File "/usr/local/bin/airflow", line 7, in
> exec(compile(f.read(), __file__, 'exec'))
> File "/airflow/airflow/bin/airflow", line 32, in
> args.func(args)
> File "/airflow/airflow/utils/cli.py", line 74, in wrapper
> return f(*args, **kwargs)
> File "/airflow/airflow/bin/cli.py", line 875, in webserver
> app.run(debug=True, port=args.port, host=args.hostname,
> AttributeError: 'tuple' object has no attribute 'run'
> {code}
> Nearly the same issue as https://issues.apache.org/jira/browse/AIRFLOW-2204,
> but only affecting RBAC debug mode. The problem is that `create_app` returns
> a tuple, but the `cli` script expects to just receive the flask app back
> without the appbuilder.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (AIRFLOW-3124) Broken webserver debug mode (RBAC)
Aaron Kosel created AIRFLOW-3124:
Summary: Broken webserver debug mode (RBAC)
Key: AIRFLOW-3124
URL: https://issues.apache.org/jira/browse/AIRFLOW-3124
Project: Apache Airflow
Issue Type: Bug
Components: webapp, webserver
Reporter: Aaron Kosel
Assignee: Aaron Kosel
{code:java}
Traceback (most recent call last):
File "/usr/local/bin/airflow", line 7, in
exec(compile(f.read(), __file__, 'exec'))
File "/airflow/airflow/bin/airflow", line 32, in
args.func(args)
File "/airflow/airflow/utils/cli.py", line 74, in wrapper
return f(*args, **kwargs)
File "/airflow/airflow/bin/cli.py", line 875, in webserver
app.run(debug=True, port=args.port, host=args.hostname,
AttributeError: 'tuple' object has no attribute 'run'
{code}
Nearly the same issue as https://issues.apache.org/jira/browse/AIRFLOW-2204,
but only affecting RBAC debug mode. The problem is that `create_app` returns a
tuple, but the `cli` script expects to just receive the flask app back without
the appbuilder.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] newtonle commented on a change in pull request #3956: [AIRFLOW-3123] Use a stack for DAG context management
newtonle commented on a change in pull request #3956: [AIRFLOW-3123] Use a stack for DAG context management URL: https://github.com/apache/incubator-airflow/pull/3956#discussion_r220743204 ## File path: airflow/models.py ## @@ -3389,7 +3389,7 @@ def __init__( self.on_success_callback = on_success_callback self.on_failure_callback = on_failure_callback -self._context_manager_set = False +self._old_context_manager_dags = [] Review comment: Actually this doesn't contain the current one. It just contains what was in `_CONTEXT_MANAGER_DAG` before it is set to the current DAG. The `old` naming convention was existing, and the only difference here is making into a list. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] aoen commented on a change in pull request #3956: [AIRFLOW-3123] Use a stack for DAG context management
aoen commented on a change in pull request #3956: [AIRFLOW-3123] Use a stack for DAG context management URL: https://github.com/apache/incubator-airflow/pull/3956#discussion_r220735705 ## File path: airflow/models.py ## @@ -3389,7 +3389,7 @@ def __init__( self.on_success_callback = on_success_callback self.on_failure_callback = on_failure_callback -self._context_manager_set = False +self._old_context_manager_dags = [] Review comment: Nit: How about just context_manager_dags since it includes the current one? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] dimberman commented on issue #3957: [AIRFLOW-2952] Fix Kubernetes CI
dimberman commented on issue #3957: [AIRFLOW-2952] Fix Kubernetes CI URL: https://github.com/apache/incubator-airflow/pull/3957#issuecomment-424881819 +1 thank you @kaxil! This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-2952) Dockerized CI pipeline has silently broken integration testing for KubernetesExecutor
[
https://issues.apache.org/jira/browse/AIRFLOW-2952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16629441#comment-16629441
]
ASF GitHub Bot commented on AIRFLOW-2952:
-
kaxil opened a new pull request #3957: [AIRFLOW-2952] Fix Kubernetes CI
URL: https://github.com/apache/incubator-airflow/pull/3957
Make sure you have checked _all_ steps below.
### Jira
- [x] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title.
- https://issues.apache.org/jira/browse/AIRFLOW-2952
### Description
- [x] Here are some details about my PR, including screenshots of any UI
changes:
- Update outdated cli command to create user
- Remove `airflow/example_dags_kubernetes` as the dag already exists in
`contrib/example_dags/`
- Update the path to copy K8s dags
### Tests
- [x] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [x] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [x] In case of new functionality, my PR adds documentation that describes
how to use it.
- When adding new operators/hooks/sensors, the autoclass documentation
generation needs to be added.
### Code Quality
- [x] Passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Dockerized CI pipeline has silently broken integration testing for
> KubernetesExecutor
> -
>
> Key: AIRFLOW-2952
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2952
> Project: Apache Airflow
> Issue Type: Bug
>Reporter: Daniel Imberman
>Assignee: Daniel Imberman
>Priority: Blocker
> Fix For: 2.0.0
>
>
> [~gcuriel] [~bolke] [~Fokko]
> Looking at all recent builds the new CI pipeline is silently reverting the
> kubernetes tests to the normal airflow tests.
> Before
> https://travis-ci.org/apache/incubator-airflow/jobs/418914949#L1007
> After:
> [https://travis-ci.org/apache/incubator-airflow/jobs/419062412#L4970]
> This means that kubernetes builds will pass without actually testing on a
> kubernetes cluster.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] kaxil opened a new pull request #3957: [AIRFLOW-2952] Fix Kubernetes CI
kaxil opened a new pull request #3957: [AIRFLOW-2952] Fix Kubernetes CI
URL: https://github.com/apache/incubator-airflow/pull/3957
Make sure you have checked _all_ steps below.
### Jira
- [x] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title.
- https://issues.apache.org/jira/browse/AIRFLOW-2952
### Description
- [x] Here are some details about my PR, including screenshots of any UI
changes:
- Update outdated cli command to create user
- Remove `airflow/example_dags_kubernetes` as the dag already exists in
`contrib/example_dags/`
- Update the path to copy K8s dags
### Tests
- [x] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [x] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [x] In case of new functionality, my PR adds documentation that describes
how to use it.
- When adding new operators/hooks/sensors, the autoclass documentation
generation needs to be added.
### Code Quality
- [x] Passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] newtonle commented on issue #3956: [AIRFLOW-3123] Use a stack for DAG context management
newtonle commented on issue #3956: [AIRFLOW-3123] Use a stack for DAG context management URL: https://github.com/apache/incubator-airflow/pull/3956#issuecomment-424878741 @aoen This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-3123) DAG context manager fails in nested cases
[
https://issues.apache.org/jira/browse/AIRFLOW-3123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16629440#comment-16629440
]
ASF GitHub Bot commented on AIRFLOW-3123:
-
newtonle opened a new pull request #3956: [AIRFLOW-3123] Use a stack for DAG
context management
URL: https://github.com/apache/incubator-airflow/pull/3956
Make sure you have checked _all_ steps below.
### Jira
- [x] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-3123
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
### Description
- [x] Here are some details about my PR, including screenshots of any UI
changes:
Implemented a stack to manage the DAG context. Entering a DAG context pushes
the past context on the stack. Exiting pops the stack to reset the context.
This allows arbitrary nesting of contexts, particularly if the same DAG object
is used in the nesting.
### Tests
- [x] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
Modified `test_dag_as_context_manager` in `models.py` to capture this
condition.
### Commits
- [x] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [ ] In case of new functionality, my PR adds documentation that describes
how to use it.
- When adding new operators/hooks/sensors, the autoclass documentation
generation needs to be added.
### Code Quality
- [x] Passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> DAG context manager fails in nested cases
> -
>
> Key: AIRFLOW-3123
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3123
> Project: Apache Airflow
> Issue Type: Bug
> Components: DAG
>Reporter: Newton Le
>Assignee: Newton Le
>Priority: Major
>
> DAG context manager fails under some cases with nested contexts:
> {code:python}
> with DAG( ... ) as dag:
> op1 = Operator()
> with dag:
> op2 = Operator()
> op3 = Operator
> {code}
> op3 will not continue to be assigned the original DAG after exiting the
> nested context.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] newtonle opened a new pull request #3956: [AIRFLOW-3123] Use a stack for DAG context management
newtonle opened a new pull request #3956: [AIRFLOW-3123] Use a stack for DAG
context management
URL: https://github.com/apache/incubator-airflow/pull/3956
Make sure you have checked _all_ steps below.
### Jira
- [x] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-3123
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
### Description
- [x] Here are some details about my PR, including screenshots of any UI
changes:
Implemented a stack to manage the DAG context. Entering a DAG context pushes
the past context on the stack. Exiting pops the stack to reset the context.
This allows arbitrary nesting of contexts, particularly if the same DAG object
is used in the nesting.
### Tests
- [x] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
Modified `test_dag_as_context_manager` in `models.py` to capture this
condition.
### Commits
- [x] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [ ] In case of new functionality, my PR adds documentation that describes
how to use it.
- When adding new operators/hooks/sensors, the autoclass documentation
generation needs to be added.
### Code Quality
- [x] Passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Created] (AIRFLOW-3123) DAG context manager fails in nested cases
Newton Le created AIRFLOW-3123:
--
Summary: DAG context manager fails in nested cases
Key: AIRFLOW-3123
URL: https://issues.apache.org/jira/browse/AIRFLOW-3123
Project: Apache Airflow
Issue Type: Bug
Components: DAG
Reporter: Newton Le
Assignee: Newton Le
DAG context manager fails under some cases with nested contexts:
{code:python}
with DAG( ... ) as dag:
op1 = Operator()
with dag:
op2 = Operator()
op3 = Operator
{code}
op3 will not continue to be assigned the original DAG after exiting the nested
context.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AIRFLOW-3118) DAGs not successful on new installation
[
https://issues.apache.org/jira/browse/AIRFLOW-3118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16629431#comment-16629431
]
Kaxil Naik commented on AIRFLOW-3118:
-
I know the reason probably. Can you try and upload any DAG file in
~/airflow/dags folder and restart both webserver and scheduler?
> DAGs not successful on new installation
> ---
>
> Key: AIRFLOW-3118
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3118
> Project: Apache Airflow
> Issue Type: Bug
> Components: DAG
>Affects Versions: 1.10.0
> Environment: Ubuntu 18.04
> Python 3.6
>Reporter: Brylie Christopher Oxley
>Priority: Blocker
> Attachments: Screenshot_20180926_161837.png,
> image-2018-09-26-12-39-03-094.png
>
>
> When trying out Airflow, on localhost, none of the DAG runs are getting to
> the 'success' state. They are getting stuck in 'running', or I manually label
> them as failed:
> !image-2018-09-26-12-39-03-094.png!
> h2. Steps to reproduce
> # create new conda environment
> ** conda create -n airflow
> ** source activate airflow
> # install airflow
> ** pip install apache-airflow
> # initialize Airflow db
> ** airflow initdb
> # disable default paused setting in airflow.cfg
> ** dags_are_paused_at_creation = False
> # {color:#6a8759}run airflow and airflow scheduler (in separate
> terminal){color}
> ** {color:#6a8759}airflow scheduler{color}
> ** {color:#6a8759}airflow webserver{color}
> # {color:#6a8759}unpause example_bash_operator{color}
> ** {color:#6a8759}airflow unpause example_bash_operator{color}
> # {color:#6a8759}log in to Airflow UI{color}
> # {color:#6a8759}turn on example_bash_operator{color}
> # {color:#6a8759}click "Trigger DAG" in `example_bash_operator` row{color}
> h2. {color:#6a8759}Observed result{color}
> {color:#6a8759}The `example_bash_operator` never leaves the "running"
> state.{color}
> h2. {color:#6a8759}Expected result{color}
> {color:#6a8759}The `example_bash_operator` would quickly enter the "success"
> state{color}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (AIRFLOW-3036) Upgrading to Airflow 1.10 not possible using GCP Cloud SQL for MYSQL
[
https://issues.apache.org/jira/browse/AIRFLOW-3036?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Smith Mathieu reassigned AIRFLOW-3036:
--
Assignee: (was: Iuliia Volkova)
> Upgrading to Airflow 1.10 not possible using GCP Cloud SQL for MYSQL
>
>
> Key: AIRFLOW-3036
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3036
> Project: Apache Airflow
> Issue Type: Bug
> Components: core, db
>Affects Versions: 1.10.0
> Environment: Google Cloud Platform, Google Kubernetes Engine, Airflow
> 1.10 on Debian Stretch, Google Cloud SQL MySQL
>Reporter: Smith Mathieu
>Priority: Blocker
> Labels: 1.10, google, google-cloud-sql
> Fix For: 2.0.0
>
>
> The upgrade path to airflow 1.10 seems impossible for users of MySQL in
> Google's Cloud SQL service given new mysql requirements for 1.10.
>
> When executing "airflow upgradedb"
> ```
> INFO [alembic.runtime.migration] Running upgrade d2ae31099d61 ->
> 0e2a74e0fc9f, Add time zone awareness
> Traceback (most recent call last):
> File "/usr/local/bin/airflow", line 32, in
> args.func(args)
> File "/usr/local/lib/python3.6/site-packages/airflow/bin/cli.py", line 1002,
> in initdb
> db_utils.initdb(settings.RBAC)
> File "/usr/local/lib/python3.6/site-packages/airflow/utils/db.py", line 92,
> in initdb
> upgradedb()
> File "/usr/local/lib/python3.6/site-packages/airflow/utils/db.py", line 346,
> in upgradedb
> command.upgrade(config, 'heads')
> File "/usr/local/lib/python3.6/site-packages/alembic/command.py", line 174,
> in upgrade
> script.run_env()
> File "/usr/local/lib/python3.6/site-packages/alembic/script/base.py", line
> 416, in run_env
> util.load_python_file(self.dir, 'env.py')
> File "/usr/local/lib/python3.6/site-packages/alembic/util/pyfiles.py", line
> 93, in load_python_file
> module = load_module_py(module_id, path)
> File "/usr/local/lib/python3.6/site-packages/alembic/util/compat.py", line
> 68, in load_module_py
> module_id, path).load_module(module_id)
> File "", line 399, in
> _check_name_wrapper
> File "", line 823, in load_module
> File "", line 682, in load_module
> File "", line 265, in _load_module_shim
> File "", line 684, in _load
> File "", line 665, in _load_unlocked
> File "", line 678, in exec_module
> File "", line 219, in _call_with_frames_removed
> File "/usr/local/lib/python3.6/site-packages/airflow/migrations/env.py",
> line 91, in
> run_migrations_online()
> File "/usr/local/lib/python3.6/site-packages/airflow/migrations/env.py",
> line 86, in run_migrations_online
> context.run_migrations()
> File "", line 8, in run_migrations
> File
> "/usr/local/lib/python3.6/site-packages/alembic/runtime/environment.py", line
> 807, in run_migrations
> self.get_context().run_migrations(**kw)
> File "/usr/local/lib/python3.6/site-packages/alembic/runtime/migration.py",
> line 321, in run_migrations
> step.migration_fn(**kw)
> File
> "/usr/local/lib/python3.6/site-packages/airflow/migrations/versions/0e2a74e0fc9f_add_time_zone_awareness.py",
> line 46, in upgrade
> raise Exception("Global variable explicit_defaults_for_timestamp needs to be
> on (1) for mysql")
> Exception: Global variable explicit_defaults_for_timestamp needs to be on
> (1) for mysql
> ```
>
> Reading documentation for upgrading to airflow 1.10, it seems the requirement
> for explicit_defaults_for_timestamp=1 was intentional.
>
> However, MySQL on Google Cloud SQL does not support configuring this
> variable and it is off by default. Users of MySQL and Cloud SQL do not have
> an upgrade path to 1.10. Alas, so close to the mythical Kubernetes Executor.
> In GCP, Cloud SQL is _the_ hosted MySQL solution.
> [https://cloud.google.com/sql/docs/mysql/flags]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (AIRFLOW-3122) Logs not being printed to STDOUT when running `airflow test` in airflow >= 1.10
[
https://issues.apache.org/jira/browse/AIRFLOW-3122?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Fred Israel updated AIRFLOW-3122:
-
Summary: Logs not being printed to STDOUT when running `airflow test` in
airflow >= 1.10 (was: Logs not being printed to STDOUT when running `airflow
test` in airflow > 1.9)
> Logs not being printed to STDOUT when running `airflow test` in airflow >=
> 1.10
> ---
>
> Key: AIRFLOW-3122
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3122
> Project: Apache Airflow
> Issue Type: Bug
>Affects Versions: 1.10.0
> Environment: Linux
>Reporter: Fred Israel
>Priority: Major
> Attachments: airflow_bug.zip
>
>
>
> In Version 1.10 it seems like there is a regression, as logs in bash operator
> are not being sent to stdout in `airflow test` command. This looks wrong to
> me, as it makes the airflow test command a bit useless and impossible to
> debug. Take a look at my tests below:
> (code is in attachment, inside docker for easy testing)
> dag.py
>
> {code:java}
> from datetime import datetime
> from airflow import DAG
> from airflow.operators.bash_operator import BashOperator
> dag = DAG('test')
> start = datetime.now()
> with dag:
> BashOperator(task_id='test', bash_command='echo TEST', start_date=start)
> BashOperator(task_id='test_failed', bash_command='echo TEST ; exit 1',
> start_date=start)
> {code}
>
>
> $ VERSION=1.10.0 docker-compose run --rm bug airflow test test test 2000-1-1
> {code:java}
> [2018-09-26 20:23:21,614] {__init__.py:51} INFO - Using executor
> SequentialExecutor
> [2018-09-26 20:23:21,756] {models.py:258} INFO - Filling up the DagBag from
> /root/airflow/dags
> [2018-09-26 20:23:21,842] {example_kubernetes_operator.py:54} WARNING - Could
> not import KubernetesPodOperator: No module named 'kubernetes'
> [2018-09-26 20:23:21,842] {example_kubernetes_operator.py:55} WARNING -
> Install kubernetes dependencies with: pip install airflow['kubernetes']{code}
> $ VERSION=1.9.0 docker-compose run --rm bug airflow test test test 2000-1-1
> {code:java}
> [2018-09-26 20:23:28,941] {__init__.py:45} INFO - Using executor
> SequentialExecutor
> [2018-09-26 20:23:28,994] {models.py:189} INFO - Filling up the DagBag from
> /root/airflow/dags
> [2018-09-26 20:23:29,094] {bash_operator.py:70} INFO - Tmp dir root location:
> /tmp
> [2018-09-26 20:23:29,094] {bash_operator.py:80} INFO - Temporary script
> location: /tmp/airflowtmphywv1usk//tmp/airflowtmphywv1usk/testoxl3mg4r
> [2018-09-26 20:23:29,095] {bash_operator.py:88} INFO - Running command: echo
> TEST
> [2018-09-26 20:23:29,098] {bash_operator.py:97} INFO - Output:
> [2018-09-26 20:23:29,099] {bash_operator.py:101} INFO - TEST
> [2018-09-26 20:23:29,099] {bash_operator.py:105} INFO - Command exited with
> return code 0{code}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (AIRFLOW-3122) Logs not being printed to STDOUT when running `airflow test` in airflow > 1.9
Fred Israel created AIRFLOW-3122:
Summary: Logs not being printed to STDOUT when running `airflow
test` in airflow > 1.9
Key: AIRFLOW-3122
URL: https://issues.apache.org/jira/browse/AIRFLOW-3122
Project: Apache Airflow
Issue Type: Bug
Affects Versions: 1.10.0
Environment: Linux
Reporter: Fred Israel
Attachments: airflow_bug.zip
In Version 1.10 it seems like there is a regression, as logs in bash operator
are not being sent to stdout in `airflow test` command. This looks wrong to me,
as it makes the airflow test command a bit useless and impossible to debug.
Take a look at my tests below:
(code is in attachment, inside docker for easy testing)
dag.py
{code:java}
from datetime import datetime
from airflow import DAG
from airflow.operators.bash_operator import BashOperator
dag = DAG('test')
start = datetime.now()
with dag:
BashOperator(task_id='test', bash_command='echo TEST', start_date=start)
BashOperator(task_id='test_failed', bash_command='echo TEST ; exit 1',
start_date=start)
{code}
$ VERSION=1.10.0 docker-compose run --rm bug airflow test test test 2000-1-1
{code:java}
[2018-09-26 20:23:21,614] {__init__.py:51} INFO - Using executor
SequentialExecutor
[2018-09-26 20:23:21,756] {models.py:258} INFO - Filling up the DagBag from
/root/airflow/dags
[2018-09-26 20:23:21,842] {example_kubernetes_operator.py:54} WARNING - Could
not import KubernetesPodOperator: No module named 'kubernetes'
[2018-09-26 20:23:21,842] {example_kubernetes_operator.py:55} WARNING - Install
kubernetes dependencies with: pip install airflow['kubernetes']{code}
$ VERSION=1.9.0 docker-compose run --rm bug airflow test test test 2000-1-1
{code:java}
[2018-09-26 20:23:28,941] {__init__.py:45} INFO - Using executor
SequentialExecutor
[2018-09-26 20:23:28,994] {models.py:189} INFO - Filling up the DagBag from
/root/airflow/dags
[2018-09-26 20:23:29,094] {bash_operator.py:70} INFO - Tmp dir root location:
/tmp
[2018-09-26 20:23:29,094] {bash_operator.py:80} INFO - Temporary script
location: /tmp/airflowtmphywv1usk//tmp/airflowtmphywv1usk/testoxl3mg4r
[2018-09-26 20:23:29,095] {bash_operator.py:88} INFO - Running command: echo
TEST
[2018-09-26 20:23:29,098] {bash_operator.py:97} INFO - Output:
[2018-09-26 20:23:29,099] {bash_operator.py:101} INFO - TEST
[2018-09-26 20:23:29,099] {bash_operator.py:105} INFO - Command exited with
return code 0{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AIRFLOW-3036) Upgrading to Airflow 1.10 not possible using GCP Cloud SQL for MYSQL
[
https://issues.apache.org/jira/browse/AIRFLOW-3036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16629341#comment-16629341
]
Iuliia Volkova commented on AIRFLOW-3036:
-
[~smith-m] please set task unassigned
> Upgrading to Airflow 1.10 not possible using GCP Cloud SQL for MYSQL
>
>
> Key: AIRFLOW-3036
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3036
> Project: Apache Airflow
> Issue Type: Bug
> Components: core, db
>Affects Versions: 1.10.0
> Environment: Google Cloud Platform, Google Kubernetes Engine, Airflow
> 1.10 on Debian Stretch, Google Cloud SQL MySQL
>Reporter: Smith Mathieu
>Assignee: Iuliia Volkova
>Priority: Blocker
> Labels: 1.10, google, google-cloud-sql
> Fix For: 2.0.0
>
>
> The upgrade path to airflow 1.10 seems impossible for users of MySQL in
> Google's Cloud SQL service given new mysql requirements for 1.10.
>
> When executing "airflow upgradedb"
> ```
> INFO [alembic.runtime.migration] Running upgrade d2ae31099d61 ->
> 0e2a74e0fc9f, Add time zone awareness
> Traceback (most recent call last):
> File "/usr/local/bin/airflow", line 32, in
> args.func(args)
> File "/usr/local/lib/python3.6/site-packages/airflow/bin/cli.py", line 1002,
> in initdb
> db_utils.initdb(settings.RBAC)
> File "/usr/local/lib/python3.6/site-packages/airflow/utils/db.py", line 92,
> in initdb
> upgradedb()
> File "/usr/local/lib/python3.6/site-packages/airflow/utils/db.py", line 346,
> in upgradedb
> command.upgrade(config, 'heads')
> File "/usr/local/lib/python3.6/site-packages/alembic/command.py", line 174,
> in upgrade
> script.run_env()
> File "/usr/local/lib/python3.6/site-packages/alembic/script/base.py", line
> 416, in run_env
> util.load_python_file(self.dir, 'env.py')
> File "/usr/local/lib/python3.6/site-packages/alembic/util/pyfiles.py", line
> 93, in load_python_file
> module = load_module_py(module_id, path)
> File "/usr/local/lib/python3.6/site-packages/alembic/util/compat.py", line
> 68, in load_module_py
> module_id, path).load_module(module_id)
> File "", line 399, in
> _check_name_wrapper
> File "", line 823, in load_module
> File "", line 682, in load_module
> File "", line 265, in _load_module_shim
> File "", line 684, in _load
> File "", line 665, in _load_unlocked
> File "", line 678, in exec_module
> File "", line 219, in _call_with_frames_removed
> File "/usr/local/lib/python3.6/site-packages/airflow/migrations/env.py",
> line 91, in
> run_migrations_online()
> File "/usr/local/lib/python3.6/site-packages/airflow/migrations/env.py",
> line 86, in run_migrations_online
> context.run_migrations()
> File "", line 8, in run_migrations
> File
> "/usr/local/lib/python3.6/site-packages/alembic/runtime/environment.py", line
> 807, in run_migrations
> self.get_context().run_migrations(**kw)
> File "/usr/local/lib/python3.6/site-packages/alembic/runtime/migration.py",
> line 321, in run_migrations
> step.migration_fn(**kw)
> File
> "/usr/local/lib/python3.6/site-packages/airflow/migrations/versions/0e2a74e0fc9f_add_time_zone_awareness.py",
> line 46, in upgrade
> raise Exception("Global variable explicit_defaults_for_timestamp needs to be
> on (1) for mysql")
> Exception: Global variable explicit_defaults_for_timestamp needs to be on
> (1) for mysql
> ```
>
> Reading documentation for upgrading to airflow 1.10, it seems the requirement
> for explicit_defaults_for_timestamp=1 was intentional.
>
> However, MySQL on Google Cloud SQL does not support configuring this
> variable and it is off by default. Users of MySQL and Cloud SQL do not have
> an upgrade path to 1.10. Alas, so close to the mythical Kubernetes Executor.
> In GCP, Cloud SQL is _the_ hosted MySQL solution.
> [https://cloud.google.com/sql/docs/mysql/flags]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AIRFLOW-3036) Upgrading to Airflow 1.10 not possible using GCP Cloud SQL for MYSQL
[
https://issues.apache.org/jira/browse/AIRFLOW-3036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16629339#comment-16629339
]
Iuliia Volkova commented on AIRFLOW-3036:
-
[~smith-m] I'm not sure what somebody could resolve it without Bolke
[~bolke]
> Upgrading to Airflow 1.10 not possible using GCP Cloud SQL for MYSQL
>
>
> Key: AIRFLOW-3036
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3036
> Project: Apache Airflow
> Issue Type: Bug
> Components: core, db
>Affects Versions: 1.10.0
> Environment: Google Cloud Platform, Google Kubernetes Engine, Airflow
> 1.10 on Debian Stretch, Google Cloud SQL MySQL
>Reporter: Smith Mathieu
>Assignee: Iuliia Volkova
>Priority: Blocker
> Labels: 1.10, google, google-cloud-sql
> Fix For: 2.0.0
>
>
> The upgrade path to airflow 1.10 seems impossible for users of MySQL in
> Google's Cloud SQL service given new mysql requirements for 1.10.
>
> When executing "airflow upgradedb"
> ```
> INFO [alembic.runtime.migration] Running upgrade d2ae31099d61 ->
> 0e2a74e0fc9f, Add time zone awareness
> Traceback (most recent call last):
> File "/usr/local/bin/airflow", line 32, in
> args.func(args)
> File "/usr/local/lib/python3.6/site-packages/airflow/bin/cli.py", line 1002,
> in initdb
> db_utils.initdb(settings.RBAC)
> File "/usr/local/lib/python3.6/site-packages/airflow/utils/db.py", line 92,
> in initdb
> upgradedb()
> File "/usr/local/lib/python3.6/site-packages/airflow/utils/db.py", line 346,
> in upgradedb
> command.upgrade(config, 'heads')
> File "/usr/local/lib/python3.6/site-packages/alembic/command.py", line 174,
> in upgrade
> script.run_env()
> File "/usr/local/lib/python3.6/site-packages/alembic/script/base.py", line
> 416, in run_env
> util.load_python_file(self.dir, 'env.py')
> File "/usr/local/lib/python3.6/site-packages/alembic/util/pyfiles.py", line
> 93, in load_python_file
> module = load_module_py(module_id, path)
> File "/usr/local/lib/python3.6/site-packages/alembic/util/compat.py", line
> 68, in load_module_py
> module_id, path).load_module(module_id)
> File "", line 399, in
> _check_name_wrapper
> File "", line 823, in load_module
> File "", line 682, in load_module
> File "", line 265, in _load_module_shim
> File "", line 684, in _load
> File "", line 665, in _load_unlocked
> File "", line 678, in exec_module
> File "", line 219, in _call_with_frames_removed
> File "/usr/local/lib/python3.6/site-packages/airflow/migrations/env.py",
> line 91, in
> run_migrations_online()
> File "/usr/local/lib/python3.6/site-packages/airflow/migrations/env.py",
> line 86, in run_migrations_online
> context.run_migrations()
> File "", line 8, in run_migrations
> File
> "/usr/local/lib/python3.6/site-packages/alembic/runtime/environment.py", line
> 807, in run_migrations
> self.get_context().run_migrations(**kw)
> File "/usr/local/lib/python3.6/site-packages/alembic/runtime/migration.py",
> line 321, in run_migrations
> step.migration_fn(**kw)
> File
> "/usr/local/lib/python3.6/site-packages/airflow/migrations/versions/0e2a74e0fc9f_add_time_zone_awareness.py",
> line 46, in upgrade
> raise Exception("Global variable explicit_defaults_for_timestamp needs to be
> on (1) for mysql")
> Exception: Global variable explicit_defaults_for_timestamp needs to be on
> (1) for mysql
> ```
>
> Reading documentation for upgrading to airflow 1.10, it seems the requirement
> for explicit_defaults_for_timestamp=1 was intentional.
>
> However, MySQL on Google Cloud SQL does not support configuring this
> variable and it is off by default. Users of MySQL and Cloud SQL do not have
> an upgrade path to 1.10. Alas, so close to the mythical Kubernetes Executor.
> In GCP, Cloud SQL is _the_ hosted MySQL solution.
> [https://cloud.google.com/sql/docs/mysql/flags]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] yeluolei commented on issue #3683: [AIRFLOW-2770] kubernetes: add support for dag folder in the docker i…
yeluolei commented on issue #3683: [AIRFLOW-2770] kubernetes: add support for dag folder in the docker i… URL: https://github.com/apache/incubator-airflow/pull/3683#issuecomment-424853136 @odracci please merge change #3770 first, I will try to rebase this commit if needed. if you can add this feature in your change, it will be better. thanks! This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] yeluolei commented on a change in pull request #3683: [AIRFLOW-2770] kubernetes: add support for dag folder in the docker i…
yeluolei commented on a change in pull request #3683: [AIRFLOW-2770]
kubernetes: add support for dag folder in the docker i…
URL: https://github.com/apache/incubator-airflow/pull/3683#discussion_r220705601
##
File path: airflow/contrib/kubernetes/worker_configuration.py
##
@@ -121,32 +121,19 @@ def _construct_volume(name, claim):
return volume
volumes = [
-_construct_volume(
-dags_volume_name,
-self.kube_config.dags_volume_claim
-),
_construct_volume(
logs_volume_name,
self.kube_config.logs_volume_claim
)
]
-dag_volume_mount_path = ""
-
-if self.kube_config.dags_volume_claim:
-dag_volume_mount_path = self.worker_airflow_dags
-else:
-dag_volume_mount_path = os.path.join(
-self.worker_airflow_dags,
-self.kube_config.git_subpath
+if not self.kube_config.dags_in_docker:
+volumes.append(
+_construct_volume(
+dags_volume_name,
+self.kube_config.dags_volume_claim
+)
)
-dags_volume_mount = {
-'name': dags_volume_name,
-'mountPath': dag_volume_mount_path,
-'readOnly': True,
-}
-if self.kube_config.dags_volume_subpath:
-dags_volume_mount['subPath'] = self.kube_config.dags_volume_subpath
logs_volume_mount = {
Review comment:
I agree with this, actually I need this.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] aoen commented on issue #3955: [AIRFLOW-3121] Define closed property on StreamLogWriter
aoen commented on issue #3955: [AIRFLOW-3121] Define closed property on StreamLogWriter URL: https://github.com/apache/incubator-airflow/pull/3955#issuecomment-424848684 LGTM once CI passes This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-3121) StreamLogWriter should extend io.IOBase
[
https://issues.apache.org/jira/browse/AIRFLOW-3121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16629309#comment-16629309
]
ASF GitHub Bot commented on AIRFLOW-3121:
-
jinnovation opened a new pull request #3955: [AIRFLOW-3121] Define closed
property on LoggingMixin
URL: https://github.com/apache/incubator-airflow/pull/3955
### Jira
- [X] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-3121
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
### Description
- [X] Here are some details about my PR, including screenshots of any UI
changes:
This PR adds the `closed` property to the `StreamLogWriter` class—a stub
function that simply returns `False` for compatibility reasons.
### Tests
- [X] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
This PR adds a straightforward property.
### Commits
- [X] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [X] In case of new functionality, my PR adds documentation that describes
how to use it.
- When adding new operators/hooks/sensors, the autoclass documentation
generation needs to be added.
### Code Quality
- [X] Passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> StreamLogWriter should extend io.IOBase
> ---
>
> Key: AIRFLOW-3121
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3121
> Project: Apache Airflow
> Issue Type: Bug
>Reporter: Jonathan
>Assignee: Jonathan
>Priority: Minor
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] jinnovation opened a new pull request #3955: [AIRFLOW-3121] Define closed property on LoggingMixin
jinnovation opened a new pull request #3955: [AIRFLOW-3121] Define closed
property on LoggingMixin
URL: https://github.com/apache/incubator-airflow/pull/3955
### Jira
- [X] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-3121
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
### Description
- [X] Here are some details about my PR, including screenshots of any UI
changes:
This PR adds the `closed` property to the `StreamLogWriter` class—a stub
function that simply returns `False` for compatibility reasons.
### Tests
- [X] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
This PR adds a straightforward property.
### Commits
- [X] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [X] In case of new functionality, my PR adds documentation that describes
how to use it.
- When adding new operators/hooks/sensors, the autoclass documentation
generation needs to be added.
### Code Quality
- [X] Passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Created] (AIRFLOW-3121) StreamLogWriter should extend io.IOBase
Jonathan created AIRFLOW-3121: - Summary: StreamLogWriter should extend io.IOBase Key: AIRFLOW-3121 URL: https://issues.apache.org/jira/browse/AIRFLOW-3121 Project: Apache Airflow Issue Type: Bug Reporter: Jonathan Assignee: Jonathan -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] jakahn commented on issue #3805: [AIRFLOW-2062] Add per-connection KMS encryption.
jakahn commented on issue #3805: [AIRFLOW-2062] Add per-connection KMS encryption. URL: https://github.com/apache/incubator-airflow/pull/3805#issuecomment-424837465 @Fokko No problem! Rebased and pushed, Travis running now. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] dimberman commented on a change in pull request #3683: [AIRFLOW-2770] kubernetes: add support for dag folder in the docker i…
dimberman commented on a change in pull request #3683: [AIRFLOW-2770]
kubernetes: add support for dag folder in the docker i…
URL: https://github.com/apache/incubator-airflow/pull/3683#discussion_r220668743
##
File path: airflow/contrib/kubernetes/worker_configuration.py
##
@@ -121,32 +121,19 @@ def _construct_volume(name, claim):
return volume
volumes = [
-_construct_volume(
-dags_volume_name,
-self.kube_config.dags_volume_claim
-),
_construct_volume(
logs_volume_name,
self.kube_config.logs_volume_claim
)
]
-dag_volume_mount_path = ""
-
-if self.kube_config.dags_volume_claim:
-dag_volume_mount_path = self.worker_airflow_dags
-else:
-dag_volume_mount_path = os.path.join(
-self.worker_airflow_dags,
-self.kube_config.git_subpath
+if not self.kube_config.dags_in_docker:
+volumes.append(
+_construct_volume(
+dags_volume_name,
+self.kube_config.dags_volume_claim
+)
)
-dags_volume_mount = {
-'name': dags_volume_name,
-'mountPath': dag_volume_mount_path,
-'readOnly': True,
-}
-if self.kube_config.dags_volume_subpath:
-dags_volume_mount['subPath'] = self.kube_config.dags_volume_subpath
logs_volume_mount = {
Review comment:
Could we please modify this s.t. users can determine not to use a volume for
logs? Would be great to completely separate the necessity of volumes for
launching.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Commented] (AIRFLOW-3079) initdb fails on Microsoft SQL Server
[
https://issues.apache.org/jira/browse/AIRFLOW-3079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16629173#comment-16629173
]
Brad Holmes commented on AIRFLOW-3079:
--
OK, I already have the fixes locally -- I just need to create a PR, and as this
is my first one, it will take me a little while to find the time and figure out
how to do it.
> initdb fails on Microsoft SQL Server
>
>
> Key: AIRFLOW-3079
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3079
> Project: Apache Airflow
> Issue Type: Bug
> Components: database
>Affects Versions: 1.10.0
>Reporter: Morten Post
>Priority: Major
>
> airflow initdb fails using Microsoft SQL Server 17 backend. Problem does not
> exist in 1.9.0.
> [*@ airflow]$ airflow initdb
> [2018-09-17 14:08:28,744] \{settings.py:174} INFO - setting.configure_orm():
> Using pool settings. pool_size=5, pool_recycle=1800
> [2018-09-17 14:08:28,865] \{__init__.py:51} INFO - Using executor
> SequentialExecutor
> DB: DB: mssql+pyodbc://***/Airflow?driver=ODBC Driver 17 for SQL
> Server
> [2018-09-17 14:08:28,967] \{db.py:338} INFO - Creating tables
> INFO [alembic.runtime.migration] Context impl MSSQLImpl.
> INFO [alembic.runtime.migration] Will assume transactional DDL.
> INFO [alembic.runtime.migration] Running upgrade -> e3a246e0dc1, current
> schema
> INFO [alembic.runtime.migration] Running upgrade e3a246e0dc1 -> 1507a7289a2f,
> create is_encrypted
> INFO [alembic.runtime.migration] Running upgrade 1507a7289a2f ->
> 13eb55f81627, maintain history for compatibility with earlier migrations
> INFO [alembic.runtime.migration] Running upgrade 13eb55f81627 ->
> 338e90f54d61, More logging into task_isntance
> INFO [alembic.runtime.migration] Running upgrade 338e90f54d61 -> 52d714495f0,
> job_id indices
> INFO [alembic.runtime.migration] Running upgrade 52d714495f0 -> 502898887f84,
> Adding extra to Log
> INFO [alembic.runtime.migration] Running upgrade 502898887f84 ->
> 1b38cef5b76e, add dagrun
> INFO [alembic.runtime.migration] Running upgrade 1b38cef5b76e ->
> 2e541a1dcfed, task_duration
> INFO [alembic.runtime.migration] Running upgrade 2e541a1dcfed ->
> 40e67319e3a9, dagrun_config
> INFO [alembic.runtime.migration] Running upgrade 40e67319e3a9 ->
> 561833c1c74b, add password column to user
> INFO [alembic.runtime.migration] Running upgrade 561833c1c74b -> 4446e08588,
> dagrun start end
> INFO [alembic.runtime.migration] Running upgrade 4446e08588 -> bbc73705a13e,
> Add notification_sent column to sla_miss
> INFO [alembic.runtime.migration] Running upgrade bbc73705a13e ->
> bba5a7cfc896, Add a column to track the encryption state of the 'Extra' field
> in connection
> INFO [alembic.runtime.migration] Running upgrade bba5a7cfc896 ->
> 1968acfc09e3, add is_encrypted column to variable table
> INFO [alembic.runtime.migration] Running upgrade 1968acfc09e3 ->
> 2e82aab8ef20, rename user table
> INFO [alembic.runtime.migration] Running upgrade 2e82aab8ef20 ->
> 211e584da130, add TI state index
> INFO [alembic.runtime.migration] Running upgrade 211e584da130 ->
> 64de9cddf6c9, add task fails journal table
> INFO [alembic.runtime.migration] Running upgrade 64de9cddf6c9 ->
> f2ca10b85618, add dag_stats table
> INFO [alembic.runtime.migration] Running upgrade f2ca10b85618 ->
> 4addfa1236f1, Add fractional seconds to mysql tables
> INFO [alembic.runtime.migration] Running upgrade 4addfa1236f1 ->
> 8504051e801b, xcom dag task indices
> INFO [alembic.runtime.migration] Running upgrade 8504051e801b ->
> 5e7d17757c7a, add pid field to TaskInstance
> INFO [alembic.runtime.migration] Running upgrade 5e7d17757c7a ->
> 127d2bf2dfa7, Add dag_id/state index on dag_run table
> INFO [alembic.runtime.migration] Running upgrade 127d2bf2dfa7 ->
> cc1e65623dc7, add max tries column to task instance
> INFO [alembic.runtime.migration] Running upgrade cc1e65623dc7 ->
> bdaa763e6c56, Make xcom value column a large binary
> INFO [alembic.runtime.migration] Running upgrade bdaa763e6c56 ->
> 947454bf1dff, add ti job_id index
> INFO [alembic.runtime.migration] Running upgrade 947454bf1dff ->
> d2ae31099d61, Increase text size for MySQL (not relevant for other DBs' text
> types)
> INFO [alembic.runtime.migration] Running upgrade d2ae31099d61 ->
> 0e2a74e0fc9f, Add time zone awareness
> Traceback (most recent call last):
> File "/bin/airflow", line 32, in
> args.func(args)
> File "/usr/lib/python2.7/site-packages/airflow/bin/cli.py", line 1002, in
> initdb
> db_utils.initdb(settings.RBAC)
> File "/usr/lib/python2.7/site-packages/airflow/utils/db.py", line 92, in
> initdb
> upgradedb()
> File "/usr/lib/python2.7/site-packages/airflow/utils/db.py", line 346, in
> upgradedb
> command.upgrade(config, 'heads')
> File
[jira] [Commented] (AIRFLOW-3079) initdb fails on Microsoft SQL Server
[
https://issues.apache.org/jira/browse/AIRFLOW-3079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16629169#comment-16629169
]
Brad Holmes commented on AIRFLOW-3079:
--
Technically, this ticket duplicates AIRFLOW-1877, as it came later. But, the
activity is here, so we'll say it is duplicated by AIRFLOW-1877, which should
probably be closed in favor of this ticket.
> initdb fails on Microsoft SQL Server
>
>
> Key: AIRFLOW-3079
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3079
> Project: Apache Airflow
> Issue Type: Bug
> Components: database
>Affects Versions: 1.10.0
>Reporter: Morten Post
>Priority: Major
>
> airflow initdb fails using Microsoft SQL Server 17 backend. Problem does not
> exist in 1.9.0.
> [*@ airflow]$ airflow initdb
> [2018-09-17 14:08:28,744] \{settings.py:174} INFO - setting.configure_orm():
> Using pool settings. pool_size=5, pool_recycle=1800
> [2018-09-17 14:08:28,865] \{__init__.py:51} INFO - Using executor
> SequentialExecutor
> DB: DB: mssql+pyodbc://***/Airflow?driver=ODBC Driver 17 for SQL
> Server
> [2018-09-17 14:08:28,967] \{db.py:338} INFO - Creating tables
> INFO [alembic.runtime.migration] Context impl MSSQLImpl.
> INFO [alembic.runtime.migration] Will assume transactional DDL.
> INFO [alembic.runtime.migration] Running upgrade -> e3a246e0dc1, current
> schema
> INFO [alembic.runtime.migration] Running upgrade e3a246e0dc1 -> 1507a7289a2f,
> create is_encrypted
> INFO [alembic.runtime.migration] Running upgrade 1507a7289a2f ->
> 13eb55f81627, maintain history for compatibility with earlier migrations
> INFO [alembic.runtime.migration] Running upgrade 13eb55f81627 ->
> 338e90f54d61, More logging into task_isntance
> INFO [alembic.runtime.migration] Running upgrade 338e90f54d61 -> 52d714495f0,
> job_id indices
> INFO [alembic.runtime.migration] Running upgrade 52d714495f0 -> 502898887f84,
> Adding extra to Log
> INFO [alembic.runtime.migration] Running upgrade 502898887f84 ->
> 1b38cef5b76e, add dagrun
> INFO [alembic.runtime.migration] Running upgrade 1b38cef5b76e ->
> 2e541a1dcfed, task_duration
> INFO [alembic.runtime.migration] Running upgrade 2e541a1dcfed ->
> 40e67319e3a9, dagrun_config
> INFO [alembic.runtime.migration] Running upgrade 40e67319e3a9 ->
> 561833c1c74b, add password column to user
> INFO [alembic.runtime.migration] Running upgrade 561833c1c74b -> 4446e08588,
> dagrun start end
> INFO [alembic.runtime.migration] Running upgrade 4446e08588 -> bbc73705a13e,
> Add notification_sent column to sla_miss
> INFO [alembic.runtime.migration] Running upgrade bbc73705a13e ->
> bba5a7cfc896, Add a column to track the encryption state of the 'Extra' field
> in connection
> INFO [alembic.runtime.migration] Running upgrade bba5a7cfc896 ->
> 1968acfc09e3, add is_encrypted column to variable table
> INFO [alembic.runtime.migration] Running upgrade 1968acfc09e3 ->
> 2e82aab8ef20, rename user table
> INFO [alembic.runtime.migration] Running upgrade 2e82aab8ef20 ->
> 211e584da130, add TI state index
> INFO [alembic.runtime.migration] Running upgrade 211e584da130 ->
> 64de9cddf6c9, add task fails journal table
> INFO [alembic.runtime.migration] Running upgrade 64de9cddf6c9 ->
> f2ca10b85618, add dag_stats table
> INFO [alembic.runtime.migration] Running upgrade f2ca10b85618 ->
> 4addfa1236f1, Add fractional seconds to mysql tables
> INFO [alembic.runtime.migration] Running upgrade 4addfa1236f1 ->
> 8504051e801b, xcom dag task indices
> INFO [alembic.runtime.migration] Running upgrade 8504051e801b ->
> 5e7d17757c7a, add pid field to TaskInstance
> INFO [alembic.runtime.migration] Running upgrade 5e7d17757c7a ->
> 127d2bf2dfa7, Add dag_id/state index on dag_run table
> INFO [alembic.runtime.migration] Running upgrade 127d2bf2dfa7 ->
> cc1e65623dc7, add max tries column to task instance
> INFO [alembic.runtime.migration] Running upgrade cc1e65623dc7 ->
> bdaa763e6c56, Make xcom value column a large binary
> INFO [alembic.runtime.migration] Running upgrade bdaa763e6c56 ->
> 947454bf1dff, add ti job_id index
> INFO [alembic.runtime.migration] Running upgrade 947454bf1dff ->
> d2ae31099d61, Increase text size for MySQL (not relevant for other DBs' text
> types)
> INFO [alembic.runtime.migration] Running upgrade d2ae31099d61 ->
> 0e2a74e0fc9f, Add time zone awareness
> Traceback (most recent call last):
> File "/bin/airflow", line 32, in
> args.func(args)
> File "/usr/lib/python2.7/site-packages/airflow/bin/cli.py", line 1002, in
> initdb
> db_utils.initdb(settings.RBAC)
> File "/usr/lib/python2.7/site-packages/airflow/utils/db.py", line 92, in
> initdb
> upgradedb()
> File "/usr/lib/python2.7/site-packages/airflow/utils/db.py", line 346, in
> upgradedb
> command.upgrade(config, 'heads')
> File
[jira] [Reopened] (AIRFLOW-3036) Upgrading to Airflow 1.10 not possible using GCP Cloud SQL for MYSQL
[
https://issues.apache.org/jira/browse/AIRFLOW-3036?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Smith Mathieu reopened AIRFLOW-3036:
Assignee: Iuliia Volkova (was: Josh Carp)
The PR that autoclosed this issue was misidentified as being associated with
AIRFLOW-3036 (this issue). The PR that closed this issue actually addressed
AIRFLOW-3074.
This issue has not been addressed
> Upgrading to Airflow 1.10 not possible using GCP Cloud SQL for MYSQL
>
>
> Key: AIRFLOW-3036
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3036
> Project: Apache Airflow
> Issue Type: Bug
> Components: core, db
>Affects Versions: 1.10.0
> Environment: Google Cloud Platform, Google Kubernetes Engine, Airflow
> 1.10 on Debian Stretch, Google Cloud SQL MySQL
>Reporter: Smith Mathieu
>Assignee: Iuliia Volkova
>Priority: Blocker
> Labels: 1.10, google, google-cloud-sql
> Fix For: 2.0.0
>
>
> The upgrade path to airflow 1.10 seems impossible for users of MySQL in
> Google's Cloud SQL service given new mysql requirements for 1.10.
>
> When executing "airflow upgradedb"
> ```
> INFO [alembic.runtime.migration] Running upgrade d2ae31099d61 ->
> 0e2a74e0fc9f, Add time zone awareness
> Traceback (most recent call last):
> File "/usr/local/bin/airflow", line 32, in
> args.func(args)
> File "/usr/local/lib/python3.6/site-packages/airflow/bin/cli.py", line 1002,
> in initdb
> db_utils.initdb(settings.RBAC)
> File "/usr/local/lib/python3.6/site-packages/airflow/utils/db.py", line 92,
> in initdb
> upgradedb()
> File "/usr/local/lib/python3.6/site-packages/airflow/utils/db.py", line 346,
> in upgradedb
> command.upgrade(config, 'heads')
> File "/usr/local/lib/python3.6/site-packages/alembic/command.py", line 174,
> in upgrade
> script.run_env()
> File "/usr/local/lib/python3.6/site-packages/alembic/script/base.py", line
> 416, in run_env
> util.load_python_file(self.dir, 'env.py')
> File "/usr/local/lib/python3.6/site-packages/alembic/util/pyfiles.py", line
> 93, in load_python_file
> module = load_module_py(module_id, path)
> File "/usr/local/lib/python3.6/site-packages/alembic/util/compat.py", line
> 68, in load_module_py
> module_id, path).load_module(module_id)
> File "", line 399, in
> _check_name_wrapper
> File "", line 823, in load_module
> File "", line 682, in load_module
> File "", line 265, in _load_module_shim
> File "", line 684, in _load
> File "", line 665, in _load_unlocked
> File "", line 678, in exec_module
> File "", line 219, in _call_with_frames_removed
> File "/usr/local/lib/python3.6/site-packages/airflow/migrations/env.py",
> line 91, in
> run_migrations_online()
> File "/usr/local/lib/python3.6/site-packages/airflow/migrations/env.py",
> line 86, in run_migrations_online
> context.run_migrations()
> File "", line 8, in run_migrations
> File
> "/usr/local/lib/python3.6/site-packages/alembic/runtime/environment.py", line
> 807, in run_migrations
> self.get_context().run_migrations(**kw)
> File "/usr/local/lib/python3.6/site-packages/alembic/runtime/migration.py",
> line 321, in run_migrations
> step.migration_fn(**kw)
> File
> "/usr/local/lib/python3.6/site-packages/airflow/migrations/versions/0e2a74e0fc9f_add_time_zone_awareness.py",
> line 46, in upgrade
> raise Exception("Global variable explicit_defaults_for_timestamp needs to be
> on (1) for mysql")
> Exception: Global variable explicit_defaults_for_timestamp needs to be on
> (1) for mysql
> ```
>
> Reading documentation for upgrading to airflow 1.10, it seems the requirement
> for explicit_defaults_for_timestamp=1 was intentional.
>
> However, MySQL on Google Cloud SQL does not support configuring this
> variable and it is off by default. Users of MySQL and Cloud SQL do not have
> an upgrade path to 1.10. Alas, so close to the mythical Kubernetes Executor.
> In GCP, Cloud SQL is _the_ hosted MySQL solution.
> [https://cloud.google.com/sql/docs/mysql/flags]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] smith-m commented on issue #3908: [AIRFLOW-3036] Add relevant ECS options to ECS operator.
smith-m commented on issue #3908: [AIRFLOW-3036] Add relevant ECS options to ECS operator. URL: https://github.com/apache/incubator-airflow/pull/3908#issuecomment-424795372 In that case, the airflow-3036 needs to be reopened and 3074 should be closed This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] r39132 commented on issue #3954: Update README.md
r39132 commented on issue #3954: Update README.md URL: https://github.com/apache/incubator-airflow/pull/3954#issuecomment-424793393 Closing dupe of https://github.com/apache/incubator-airflow/pull/3953 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] r39132 opened a new pull request #3954: Update README.md
r39132 opened a new pull request #3954: Update README.md
URL: https://github.com/apache/incubator-airflow/pull/3954
Make sure you have checked _all_ steps below.
### Jira
- [ ] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-XXX
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
### Description
- [ ] Here are some details about my PR, including screenshots of any UI
changes:
### Tests
- [ ] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [ ] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [ ] In case of new functionality, my PR adds documentation that describes
how to use it.
- When adding new operators/hooks/sensors, the autoclass documentation
generation needs to be added.
### Code Quality
- [ ] Passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] r39132 closed pull request #3954: Update README.md
r39132 closed pull request #3954: Update README.md URL: https://github.com/apache/incubator-airflow/pull/3954 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] r39132 closed pull request #3953: [AIRFLOW-XXX] Add Fathom Health to readme
r39132 closed pull request #3953: [AIRFLOW-XXX] Add Fathom Health to readme URL: https://github.com/apache/incubator-airflow/pull/3953 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] r39132 opened a new pull request #3953: [AIRFLOW-XXX] Add Fathom Health to readme
r39132 opened a new pull request #3953: [AIRFLOW-XXX] Add Fathom Health to
readme
URL: https://github.com/apache/incubator-airflow/pull/3953
Make sure you have checked _all_ steps below.
### Jira
- [x] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-XXX
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
### Description
- [x] Here are some details about my PR, including screenshots of any UI
changes:
### Tests
- [x] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [x] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [x] In case of new functionality, my PR adds documentation that describes
how to use it.
- When adding new operators/hooks/sensors, the autoclass documentation
generation needs to be added.
### Code Quality
- [x] Passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] r39132 closed pull request #3951: [AIRFLOW-XXX] Add Square to the companies lists
r39132 closed pull request #3951: [AIRFLOW-XXX] Add Square to the companies lists URL: https://github.com/apache/incubator-airflow/pull/3951 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] jmcarp commented on issue #3917: [AIRFLOW-3086] Add extras group for google auth to setup.py.
jmcarp commented on issue #3917: [AIRFLOW-3086] Add extras group for google auth to setup.py. URL: https://github.com/apache/incubator-airflow/pull/3917#issuecomment-424785093 Does this make sense @Fokko? Should be quick to review when somebody has time. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] akshayi1 opened a new pull request #3952: [AIRFLOW-XXX] Update GCS logging docs for latest code
akshayi1 opened a new pull request #3952: [AIRFLOW-XXX] Update GCS logging docs for latest code URL: https://github.com/apache/incubator-airflow/pull/3952 Updated documentation for the steps to set up remote logging to Google Cloud Storage. In discussion with @ashb on the Slack channel, this does not need a JIRA ticket. Please take a look and approve this PR. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Assigned] (AIRFLOW-2794) Add delete support for Azure blob
[ https://issues.apache.org/jira/browse/AIRFLOW-2794?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anonymous reassigned AIRFLOW-2794: -- Assignee: Bart Eijk > Add delete support for Azure blob > - > > Key: AIRFLOW-2794 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2794 > Project: Apache Airflow > Issue Type: Wish > Components: hooks, operators >Reporter: Bart Eijk >Assignee: Bart Eijk >Priority: Trivial > > As a developer, I would like to have the ability to create tasks that can > delete files in Azure blob storage. > Nice to have: the ability to delete a "folder", i.e. a prefix. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-3120) Missing dependency for gcp_api authentification
Victor created AIRFLOW-3120: --- Summary: Missing dependency for gcp_api authentification Key: AIRFLOW-3120 URL: https://issues.apache.org/jira/browse/AIRFLOW-3120 Project: Apache Airflow Issue Type: Bug Components: authentication Affects Versions: 1.10.0 Reporter: Victor In order to use Google Auth with Airflow, the package Flask-OAuthlib is needed. I was expecting that passing the option gcp_api during pip install would add the dependency. Another solution would be to add a new option google_auth or something like that of course. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] r39132 opened a new pull request #3951: [AIRFLOW-XXX] Add Square to the companies lists
r39132 opened a new pull request #3951: [AIRFLOW-XXX] Add Square to the
companies lists
URL: https://github.com/apache/incubator-airflow/pull/3951
Make sure you have checked _all_ steps below.
### Jira
- [x] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-XXX
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
### Description
- [x] Here are some details about my PR, including screenshots of any UI
changes:
### Tests
- [x] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [x] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [x] In case of new functionality, my PR adds documentation that describes
how to use it.
- When adding new operators/hooks/sensors, the autoclass documentation
generation needs to be added.
### Code Quality
- [x] Passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] ckljohn commented on issue #3945: [AIRFLOW-3112] Make SFTP hook to inherit SSH hook
ckljohn commented on issue #3945: [AIRFLOW-3112] Make SFTP hook to inherit SSH hook URL: https://github.com/apache/incubator-airflow/pull/3945#issuecomment-424740292 Should the backward compatibility be deprecated on the next major release? Should I also include a deprecate note? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-3079) initdb fails on Microsoft SQL Server
[
https://issues.apache.org/jira/browse/AIRFLOW-3079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16628834#comment-16628834
]
Ash Berlin-Taylor commented on AIRFLOW-3079:
Ah, "thanks SQLServer!"
Probably fixing the migration in question - it is already specialised to cope
with different DB engines, so extending it to cope with SQLServer is probably
not a lot of work.
> initdb fails on Microsoft SQL Server
>
>
> Key: AIRFLOW-3079
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3079
> Project: Apache Airflow
> Issue Type: Bug
> Components: database
>Affects Versions: 1.10.0
>Reporter: Morten Post
>Priority: Major
>
> airflow initdb fails using Microsoft SQL Server 17 backend. Problem does not
> exist in 1.9.0.
> [*@ airflow]$ airflow initdb
> [2018-09-17 14:08:28,744] \{settings.py:174} INFO - setting.configure_orm():
> Using pool settings. pool_size=5, pool_recycle=1800
> [2018-09-17 14:08:28,865] \{__init__.py:51} INFO - Using executor
> SequentialExecutor
> DB: DB: mssql+pyodbc://***/Airflow?driver=ODBC Driver 17 for SQL
> Server
> [2018-09-17 14:08:28,967] \{db.py:338} INFO - Creating tables
> INFO [alembic.runtime.migration] Context impl MSSQLImpl.
> INFO [alembic.runtime.migration] Will assume transactional DDL.
> INFO [alembic.runtime.migration] Running upgrade -> e3a246e0dc1, current
> schema
> INFO [alembic.runtime.migration] Running upgrade e3a246e0dc1 -> 1507a7289a2f,
> create is_encrypted
> INFO [alembic.runtime.migration] Running upgrade 1507a7289a2f ->
> 13eb55f81627, maintain history for compatibility with earlier migrations
> INFO [alembic.runtime.migration] Running upgrade 13eb55f81627 ->
> 338e90f54d61, More logging into task_isntance
> INFO [alembic.runtime.migration] Running upgrade 338e90f54d61 -> 52d714495f0,
> job_id indices
> INFO [alembic.runtime.migration] Running upgrade 52d714495f0 -> 502898887f84,
> Adding extra to Log
> INFO [alembic.runtime.migration] Running upgrade 502898887f84 ->
> 1b38cef5b76e, add dagrun
> INFO [alembic.runtime.migration] Running upgrade 1b38cef5b76e ->
> 2e541a1dcfed, task_duration
> INFO [alembic.runtime.migration] Running upgrade 2e541a1dcfed ->
> 40e67319e3a9, dagrun_config
> INFO [alembic.runtime.migration] Running upgrade 40e67319e3a9 ->
> 561833c1c74b, add password column to user
> INFO [alembic.runtime.migration] Running upgrade 561833c1c74b -> 4446e08588,
> dagrun start end
> INFO [alembic.runtime.migration] Running upgrade 4446e08588 -> bbc73705a13e,
> Add notification_sent column to sla_miss
> INFO [alembic.runtime.migration] Running upgrade bbc73705a13e ->
> bba5a7cfc896, Add a column to track the encryption state of the 'Extra' field
> in connection
> INFO [alembic.runtime.migration] Running upgrade bba5a7cfc896 ->
> 1968acfc09e3, add is_encrypted column to variable table
> INFO [alembic.runtime.migration] Running upgrade 1968acfc09e3 ->
> 2e82aab8ef20, rename user table
> INFO [alembic.runtime.migration] Running upgrade 2e82aab8ef20 ->
> 211e584da130, add TI state index
> INFO [alembic.runtime.migration] Running upgrade 211e584da130 ->
> 64de9cddf6c9, add task fails journal table
> INFO [alembic.runtime.migration] Running upgrade 64de9cddf6c9 ->
> f2ca10b85618, add dag_stats table
> INFO [alembic.runtime.migration] Running upgrade f2ca10b85618 ->
> 4addfa1236f1, Add fractional seconds to mysql tables
> INFO [alembic.runtime.migration] Running upgrade 4addfa1236f1 ->
> 8504051e801b, xcom dag task indices
> INFO [alembic.runtime.migration] Running upgrade 8504051e801b ->
> 5e7d17757c7a, add pid field to TaskInstance
> INFO [alembic.runtime.migration] Running upgrade 5e7d17757c7a ->
> 127d2bf2dfa7, Add dag_id/state index on dag_run table
> INFO [alembic.runtime.migration] Running upgrade 127d2bf2dfa7 ->
> cc1e65623dc7, add max tries column to task instance
> INFO [alembic.runtime.migration] Running upgrade cc1e65623dc7 ->
> bdaa763e6c56, Make xcom value column a large binary
> INFO [alembic.runtime.migration] Running upgrade bdaa763e6c56 ->
> 947454bf1dff, add ti job_id index
> INFO [alembic.runtime.migration] Running upgrade 947454bf1dff ->
> d2ae31099d61, Increase text size for MySQL (not relevant for other DBs' text
> types)
> INFO [alembic.runtime.migration] Running upgrade d2ae31099d61 ->
> 0e2a74e0fc9f, Add time zone awareness
> Traceback (most recent call last):
> File "/bin/airflow", line 32, in
> args.func(args)
> File "/usr/lib/python2.7/site-packages/airflow/bin/cli.py", line 1002, in
> initdb
> db_utils.initdb(settings.RBAC)
> File "/usr/lib/python2.7/site-packages/airflow/utils/db.py", line 92, in
> initdb
> upgradedb()
> File "/usr/lib/python2.7/site-packages/airflow/utils/db.py", line 346, in
> upgradedb
> command.upgrade(config, 'heads')
> File
[jira] [Commented] (AIRFLOW-3119) Enable loglevel on celery worker and inherit from airflow.cfg
[
https://issues.apache.org/jira/browse/AIRFLOW-3119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16628831#comment-16628831
]
ASF GitHub Bot commented on AIRFLOW-3119:
-
cloneluke opened a new pull request #3950: [AIRFLOW-3119] Enable loglevel on
celery worker and inherit from airflow.cfg
URL: https://github.com/apache/incubator-airflow/pull/3950
Make sure you have checked _all_ steps below.
### Jira
- [x ] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-XXX
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
### Description
- [ x] Enable debugging on my celery worker, this story will enable
--loglevel when launching a celery worker:
http://docs.celeryproject.org/en/latest/reference/celery.bin.worker.html
and inherit that loglevel setting from airflow.cfg
### Tests
- [x ] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [x ] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [x ] In case of new functionality, my PR adds documentation that describes
how to use it.
- When adding new operators/hooks/sensors, the autoclass documentation
generation needs to be added.
### Code Quality
- [ ] Passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Enable loglevel on celery worker and inherit from airflow.cfg
> -
>
> Key: AIRFLOW-3119
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3119
> Project: Apache Airflow
> Issue Type: Improvement
> Components: celery, worker
>Affects Versions: 1.9.0, 1.10.0
>Reporter: Luke Bodeen
>Assignee: Luke Bodeen
>Priority: Minor
> Fix For: 1.10.1
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> I would like to enable debugging on my celery worker, this story will enable
> --loglevel when launching a celery worker:
> [http://docs.celeryproject.org/en/latest/reference/celery.bin.worker.html]
>
> and inherit that loglevel setting from airflow.cfg
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] cloneluke opened a new pull request #3950: [AIRFLOW-3119] Enable loglevel on celery worker and inherit from airflow.cfg
cloneluke opened a new pull request #3950: [AIRFLOW-3119] Enable loglevel on
celery worker and inherit from airflow.cfg
URL: https://github.com/apache/incubator-airflow/pull/3950
Make sure you have checked _all_ steps below.
### Jira
- [x ] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-XXX
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
### Description
- [ x] Enable debugging on my celery worker, this story will enable
--loglevel when launching a celery worker:
http://docs.celeryproject.org/en/latest/reference/celery.bin.worker.html
and inherit that loglevel setting from airflow.cfg
### Tests
- [x ] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [x ] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [x ] In case of new functionality, my PR adds documentation that describes
how to use it.
- When adding new operators/hooks/sensors, the autoclass documentation
generation needs to be added.
### Code Quality
- [ ] Passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Created] (AIRFLOW-3119) Enable loglevel on celery worker and inherit from airflow.cfg
Luke Bodeen created AIRFLOW-3119: Summary: Enable loglevel on celery worker and inherit from airflow.cfg Key: AIRFLOW-3119 URL: https://issues.apache.org/jira/browse/AIRFLOW-3119 Project: Apache Airflow Issue Type: Improvement Components: celery, worker Affects Versions: 1.10.0, 1.9.0 Reporter: Luke Bodeen Assignee: Luke Bodeen Fix For: 1.10.1 I would like to enable debugging on my celery worker, this story will enable --loglevel when launching a celery worker: [http://docs.celeryproject.org/en/latest/reference/celery.bin.worker.html] and inherit that loglevel setting from airflow.cfg -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-3079) initdb fails on Microsoft SQL Server
[
https://issues.apache.org/jira/browse/AIRFLOW-3079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16628785#comment-16628785
]
Brad Holmes commented on AIRFLOW-3079:
--
I also encountered this problem. The root cause is that {{TIMESTAMP}} for
Microsoft SQL Server ({{mssql}}) is actually a rowid, where as for others, it
is what it sounds like: a timestamp. See
https://stackoverflow.com/questions/41682748/sqlalchemy-fails-to-insert-timestamp-into-mssql,
especially this comment:
{quote}
Microsoft SQL Server's timestamp data type is not, actually, a timestamp - it
is more of a sequence (it is an ancient holdover, and horribly misnamed). If
you need to update a date time value, use a datetime data type.
{quote}
I am unsure if we should apply the fix here in Airflow (I have a patch that
works) or try to get a fix into {{sqlalchemy}}.
> initdb fails on Microsoft SQL Server
>
>
> Key: AIRFLOW-3079
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3079
> Project: Apache Airflow
> Issue Type: Bug
> Components: database
>Affects Versions: 1.10.0
>Reporter: Morten Post
>Priority: Major
>
> airflow initdb fails using Microsoft SQL Server 17 backend. Problem does not
> exist in 1.9.0.
> [*@ airflow]$ airflow initdb
> [2018-09-17 14:08:28,744] \{settings.py:174} INFO - setting.configure_orm():
> Using pool settings. pool_size=5, pool_recycle=1800
> [2018-09-17 14:08:28,865] \{__init__.py:51} INFO - Using executor
> SequentialExecutor
> DB: DB: mssql+pyodbc://***/Airflow?driver=ODBC Driver 17 for SQL
> Server
> [2018-09-17 14:08:28,967] \{db.py:338} INFO - Creating tables
> INFO [alembic.runtime.migration] Context impl MSSQLImpl.
> INFO [alembic.runtime.migration] Will assume transactional DDL.
> INFO [alembic.runtime.migration] Running upgrade -> e3a246e0dc1, current
> schema
> INFO [alembic.runtime.migration] Running upgrade e3a246e0dc1 -> 1507a7289a2f,
> create is_encrypted
> INFO [alembic.runtime.migration] Running upgrade 1507a7289a2f ->
> 13eb55f81627, maintain history for compatibility with earlier migrations
> INFO [alembic.runtime.migration] Running upgrade 13eb55f81627 ->
> 338e90f54d61, More logging into task_isntance
> INFO [alembic.runtime.migration] Running upgrade 338e90f54d61 -> 52d714495f0,
> job_id indices
> INFO [alembic.runtime.migration] Running upgrade 52d714495f0 -> 502898887f84,
> Adding extra to Log
> INFO [alembic.runtime.migration] Running upgrade 502898887f84 ->
> 1b38cef5b76e, add dagrun
> INFO [alembic.runtime.migration] Running upgrade 1b38cef5b76e ->
> 2e541a1dcfed, task_duration
> INFO [alembic.runtime.migration] Running upgrade 2e541a1dcfed ->
> 40e67319e3a9, dagrun_config
> INFO [alembic.runtime.migration] Running upgrade 40e67319e3a9 ->
> 561833c1c74b, add password column to user
> INFO [alembic.runtime.migration] Running upgrade 561833c1c74b -> 4446e08588,
> dagrun start end
> INFO [alembic.runtime.migration] Running upgrade 4446e08588 -> bbc73705a13e,
> Add notification_sent column to sla_miss
> INFO [alembic.runtime.migration] Running upgrade bbc73705a13e ->
> bba5a7cfc896, Add a column to track the encryption state of the 'Extra' field
> in connection
> INFO [alembic.runtime.migration] Running upgrade bba5a7cfc896 ->
> 1968acfc09e3, add is_encrypted column to variable table
> INFO [alembic.runtime.migration] Running upgrade 1968acfc09e3 ->
> 2e82aab8ef20, rename user table
> INFO [alembic.runtime.migration] Running upgrade 2e82aab8ef20 ->
> 211e584da130, add TI state index
> INFO [alembic.runtime.migration] Running upgrade 211e584da130 ->
> 64de9cddf6c9, add task fails journal table
> INFO [alembic.runtime.migration] Running upgrade 64de9cddf6c9 ->
> f2ca10b85618, add dag_stats table
> INFO [alembic.runtime.migration] Running upgrade f2ca10b85618 ->
> 4addfa1236f1, Add fractional seconds to mysql tables
> INFO [alembic.runtime.migration] Running upgrade 4addfa1236f1 ->
> 8504051e801b, xcom dag task indices
> INFO [alembic.runtime.migration] Running upgrade 8504051e801b ->
> 5e7d17757c7a, add pid field to TaskInstance
> INFO [alembic.runtime.migration] Running upgrade 5e7d17757c7a ->
> 127d2bf2dfa7, Add dag_id/state index on dag_run table
> INFO [alembic.runtime.migration] Running upgrade 127d2bf2dfa7 ->
> cc1e65623dc7, add max tries column to task instance
> INFO [alembic.runtime.migration] Running upgrade cc1e65623dc7 ->
> bdaa763e6c56, Make xcom value column a large binary
> INFO [alembic.runtime.migration] Running upgrade bdaa763e6c56 ->
> 947454bf1dff, add ti job_id index
> INFO [alembic.runtime.migration] Running upgrade 947454bf1dff ->
> d2ae31099d61, Increase text size for MySQL (not relevant for other DBs' text
> types)
> INFO [alembic.runtime.migration] Running upgrade d2ae31099d61 ->
> 0e2a74e0fc9f, Add
[jira] [Commented] (AIRFLOW-3118) DAGs not successful on new installation
[
https://issues.apache.org/jira/browse/AIRFLOW-3118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16628764#comment-16628764
]
Brylie Christopher Oxley commented on AIRFLOW-3118:
---
Here is a screenshot of the example_bash_operator DAG logs table:
!Screenshot_20180926_161837.png!
> DAGs not successful on new installation
> ---
>
> Key: AIRFLOW-3118
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3118
> Project: Apache Airflow
> Issue Type: Bug
> Components: DAG
>Affects Versions: 1.10.0
> Environment: Ubuntu 18.04
> Python 3.6
>Reporter: Brylie Christopher Oxley
>Priority: Blocker
> Attachments: Screenshot_20180926_161837.png,
> image-2018-09-26-12-39-03-094.png
>
>
> When trying out Airflow, on localhost, none of the DAG runs are getting to
> the 'success' state. They are getting stuck in 'running', or I manually label
> them as failed:
> !image-2018-09-26-12-39-03-094.png!
> h2. Steps to reproduce
> # create new conda environment
> ** conda create -n airflow
> ** source activate airflow
> # install airflow
> ** pip install apache-airflow
> # initialize Airflow db
> ** airflow initdb
> # disable default paused setting in airflow.cfg
> ** dags_are_paused_at_creation = False
> # {color:#6a8759}run airflow and airflow scheduler (in separate
> terminal){color}
> ** {color:#6a8759}airflow scheduler{color}
> ** {color:#6a8759}airflow webserver{color}
> # {color:#6a8759}unpause example_bash_operator{color}
> ** {color:#6a8759}airflow unpause example_bash_operator{color}
> # {color:#6a8759}log in to Airflow UI{color}
> # {color:#6a8759}turn on example_bash_operator{color}
> # {color:#6a8759}click "Trigger DAG" in `example_bash_operator` row{color}
> h2. {color:#6a8759}Observed result{color}
> {color:#6a8759}The `example_bash_operator` never leaves the "running"
> state.{color}
> h2. {color:#6a8759}Expected result{color}
> {color:#6a8759}The `example_bash_operator` would quickly enter the "success"
> state{color}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AIRFLOW-3118) DAGs not successful on new installation
[
https://issues.apache.org/jira/browse/AIRFLOW-3118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16628750#comment-16628750
]
Brylie Christopher Oxley commented on AIRFLOW-3118:
---
The start_date for the example_bash_operator DAG is 'two days ago':
{code:java}
args = {
'owner': 'airflow',
'start_date': airflow.utils.dates.days_ago(2)
}
{code}
> DAGs not successful on new installation
> ---
>
> Key: AIRFLOW-3118
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3118
> Project: Apache Airflow
> Issue Type: Bug
> Components: DAG
>Affects Versions: 1.10.0
> Environment: Ubuntu 18.04
> Python 3.6
>Reporter: Brylie Christopher Oxley
>Priority: Blocker
> Attachments: image-2018-09-26-12-39-03-094.png
>
>
> When trying out Airflow, on localhost, none of the DAG runs are getting to
> the 'success' state. They are getting stuck in 'running', or I manually label
> them as failed:
> !image-2018-09-26-12-39-03-094.png!
> h2. Steps to reproduce
> # create new conda environment
> ** conda create -n airflow
> ** source activate airflow
> # install airflow
> ** pip install apache-airflow
> # initialize Airflow db
> ** airflow initdb
> # disable default paused setting in airflow.cfg
> ** dags_are_paused_at_creation = False
> # {color:#6a8759}run airflow and airflow scheduler (in separate
> terminal){color}
> ** {color:#6a8759}airflow scheduler{color}
> ** {color:#6a8759}airflow webserver{color}
> # {color:#6a8759}unpause example_bash_operator{color}
> ** {color:#6a8759}airflow unpause example_bash_operator{color}
> # {color:#6a8759}log in to Airflow UI{color}
> # {color:#6a8759}turn on example_bash_operator{color}
> # {color:#6a8759}click "Trigger DAG" in `example_bash_operator` row{color}
> h2. {color:#6a8759}Observed result{color}
> {color:#6a8759}The `example_bash_operator` never leaves the "running"
> state.{color}
> h2. {color:#6a8759}Expected result{color}
> {color:#6a8759}The `example_bash_operator` would quickly enter the "success"
> state{color}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AIRFLOW-3118) DAGs not successful on new installation
[
https://issues.apache.org/jira/browse/AIRFLOW-3118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16628761#comment-16628761
]
Ash Berlin-Taylor commented on AIRFLOW-3118:
If no logs appear is the scheduler still running? Alternatively do you get any
output/error messages when running {{airflow scheduler}}? They might be rather
verbose and repeating, so only the first few (100? 50?) lines are all that
interesting probably.
> DAGs not successful on new installation
> ---
>
> Key: AIRFLOW-3118
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3118
> Project: Apache Airflow
> Issue Type: Bug
> Components: DAG
>Affects Versions: 1.10.0
> Environment: Ubuntu 18.04
> Python 3.6
>Reporter: Brylie Christopher Oxley
>Priority: Blocker
> Attachments: image-2018-09-26-12-39-03-094.png
>
>
> When trying out Airflow, on localhost, none of the DAG runs are getting to
> the 'success' state. They are getting stuck in 'running', or I manually label
> them as failed:
> !image-2018-09-26-12-39-03-094.png!
> h2. Steps to reproduce
> # create new conda environment
> ** conda create -n airflow
> ** source activate airflow
> # install airflow
> ** pip install apache-airflow
> # initialize Airflow db
> ** airflow initdb
> # disable default paused setting in airflow.cfg
> ** dags_are_paused_at_creation = False
> # {color:#6a8759}run airflow and airflow scheduler (in separate
> terminal){color}
> ** {color:#6a8759}airflow scheduler{color}
> ** {color:#6a8759}airflow webserver{color}
> # {color:#6a8759}unpause example_bash_operator{color}
> ** {color:#6a8759}airflow unpause example_bash_operator{color}
> # {color:#6a8759}log in to Airflow UI{color}
> # {color:#6a8759}turn on example_bash_operator{color}
> # {color:#6a8759}click "Trigger DAG" in `example_bash_operator` row{color}
> h2. {color:#6a8759}Observed result{color}
> {color:#6a8759}The `example_bash_operator` never leaves the "running"
> state.{color}
> h2. {color:#6a8759}Expected result{color}
> {color:#6a8759}The `example_bash_operator` would quickly enter the "success"
> state{color}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (AIRFLOW-3118) DAGs not successful on new installation
[
https://issues.apache.org/jira/browse/AIRFLOW-3118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brylie Christopher Oxley updated AIRFLOW-3118:
--
Attachment: Screenshot_20180926_161837.png
> DAGs not successful on new installation
> ---
>
> Key: AIRFLOW-3118
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3118
> Project: Apache Airflow
> Issue Type: Bug
> Components: DAG
>Affects Versions: 1.10.0
> Environment: Ubuntu 18.04
> Python 3.6
>Reporter: Brylie Christopher Oxley
>Priority: Blocker
> Attachments: Screenshot_20180926_161837.png,
> image-2018-09-26-12-39-03-094.png
>
>
> When trying out Airflow, on localhost, none of the DAG runs are getting to
> the 'success' state. They are getting stuck in 'running', or I manually label
> them as failed:
> !image-2018-09-26-12-39-03-094.png!
> h2. Steps to reproduce
> # create new conda environment
> ** conda create -n airflow
> ** source activate airflow
> # install airflow
> ** pip install apache-airflow
> # initialize Airflow db
> ** airflow initdb
> # disable default paused setting in airflow.cfg
> ** dags_are_paused_at_creation = False
> # {color:#6a8759}run airflow and airflow scheduler (in separate
> terminal){color}
> ** {color:#6a8759}airflow scheduler{color}
> ** {color:#6a8759}airflow webserver{color}
> # {color:#6a8759}unpause example_bash_operator{color}
> ** {color:#6a8759}airflow unpause example_bash_operator{color}
> # {color:#6a8759}log in to Airflow UI{color}
> # {color:#6a8759}turn on example_bash_operator{color}
> # {color:#6a8759}click "Trigger DAG" in `example_bash_operator` row{color}
> h2. {color:#6a8759}Observed result{color}
> {color:#6a8759}The `example_bash_operator` never leaves the "running"
> state.{color}
> h2. {color:#6a8759}Expected result{color}
> {color:#6a8759}The `example_bash_operator` would quickly enter the "success"
> state{color}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AIRFLOW-3118) DAGs not successful on new installation
[
https://issues.apache.org/jira/browse/AIRFLOW-3118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16628753#comment-16628753
]
Brylie Christopher Oxley commented on AIRFLOW-3118:
---
Although the logs/scheduler directory is created, with two sub-directories
(latest and 2018-09-26), no logs appear.
> DAGs not successful on new installation
> ---
>
> Key: AIRFLOW-3118
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3118
> Project: Apache Airflow
> Issue Type: Bug
> Components: DAG
>Affects Versions: 1.10.0
> Environment: Ubuntu 18.04
> Python 3.6
>Reporter: Brylie Christopher Oxley
>Priority: Blocker
> Attachments: image-2018-09-26-12-39-03-094.png
>
>
> When trying out Airflow, on localhost, none of the DAG runs are getting to
> the 'success' state. They are getting stuck in 'running', or I manually label
> them as failed:
> !image-2018-09-26-12-39-03-094.png!
> h2. Steps to reproduce
> # create new conda environment
> ** conda create -n airflow
> ** source activate airflow
> # install airflow
> ** pip install apache-airflow
> # initialize Airflow db
> ** airflow initdb
> # disable default paused setting in airflow.cfg
> ** dags_are_paused_at_creation = False
> # {color:#6a8759}run airflow and airflow scheduler (in separate
> terminal){color}
> ** {color:#6a8759}airflow scheduler{color}
> ** {color:#6a8759}airflow webserver{color}
> # {color:#6a8759}unpause example_bash_operator{color}
> ** {color:#6a8759}airflow unpause example_bash_operator{color}
> # {color:#6a8759}log in to Airflow UI{color}
> # {color:#6a8759}turn on example_bash_operator{color}
> # {color:#6a8759}click "Trigger DAG" in `example_bash_operator` row{color}
> h2. {color:#6a8759}Observed result{color}
> {color:#6a8759}The `example_bash_operator` never leaves the "running"
> state.{color}
> h2. {color:#6a8759}Expected result{color}
> {color:#6a8759}The `example_bash_operator` would quickly enter the "success"
> state{color}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AIRFLOW-3118) DAGs not successful on new installation
[
https://issues.apache.org/jira/browse/AIRFLOW-3118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16628740#comment-16628740
]
Brylie Christopher Oxley commented on AIRFLOW-3118:
---
The example_bash_operator DAG looks like this:
{code:java}
dag = DAG(
dag_id='example_bash_operator', default_args=args,
schedule_interval='0 0 * * *',
dagrun_timeout=timedelta(minutes=60))
{code}
It seems to be scheduled on an interval, rather than at a date.
> DAGs not successful on new installation
> ---
>
> Key: AIRFLOW-3118
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3118
> Project: Apache Airflow
> Issue Type: Bug
> Components: DAG
>Affects Versions: 1.10.0
> Environment: Ubuntu 18.04
> Python 3.6
>Reporter: Brylie Christopher Oxley
>Priority: Blocker
> Attachments: image-2018-09-26-12-39-03-094.png
>
>
> When trying out Airflow, on localhost, none of the DAG runs are getting to
> the 'success' state. They are getting stuck in 'running', or I manually label
> them as failed:
> !image-2018-09-26-12-39-03-094.png!
> h2. Steps to reproduce
> # create new conda environment
> ** conda create -n airflow
> ** source activate airflow
> # install airflow
> ** pip install apache-airflow
> # initialize Airflow db
> ** airflow initdb
> # disable default paused setting in airflow.cfg
> ** dags_are_paused_at_creation = False
> # {color:#6a8759}run airflow and airflow scheduler (in separate
> terminal){color}
> ** {color:#6a8759}airflow scheduler{color}
> ** {color:#6a8759}airflow webserver{color}
> # {color:#6a8759}unpause example_bash_operator{color}
> ** {color:#6a8759}airflow unpause example_bash_operator{color}
> # {color:#6a8759}log in to Airflow UI{color}
> # {color:#6a8759}turn on example_bash_operator{color}
> # {color:#6a8759}click "Trigger DAG" in `example_bash_operator` row{color}
> h2. {color:#6a8759}Observed result{color}
> {color:#6a8759}The `example_bash_operator` never leaves the "running"
> state.{color}
> h2. {color:#6a8759}Expected result{color}
> {color:#6a8759}The `example_bash_operator` would quickly enter the "success"
> state{color}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AIRFLOW-3118) DAGs not successful on new installation
[
https://issues.apache.org/jira/browse/AIRFLOW-3118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16628735#comment-16628735
]
Brylie Christopher Oxley commented on AIRFLOW-3118:
---
Here is the output from airlow list_dags -r
{noformat}
---
DAGS
---
example_bash_operator
example_branch_dop_operator_v3
example_branch_operator
example_http_operator
example_kubernetes_executor
example_passing_params_via_test_command
example_python_operator
example_short_circuit_operator
example_skip_dag
example_subdag_operator
example_subdag_operator.section-1
example_subdag_operator.section-2
example_trigger_controller_dag
example_trigger_target_dag
example_xcom
latest_only
latest_only_with_trigger
test_utils
tutorial
---
DagBag loading stats for /home/brylie/airflow/dags
---
Number of DAGs: 0
Total task number: 0
DagBag parsing time: 0
None
{noformat}
I am trying to run any of the example DAGs that ship with Airflow.
> DAGs not successful on new installation
> ---
>
> Key: AIRFLOW-3118
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3118
> Project: Apache Airflow
> Issue Type: Bug
> Components: DAG
>Affects Versions: 1.10.0
> Environment: Ubuntu 18.04
> Python 3.6
>Reporter: Brylie Christopher Oxley
>Priority: Blocker
> Attachments: image-2018-09-26-12-39-03-094.png
>
>
> When trying out Airflow, on localhost, none of the DAG runs are getting to
> the 'success' state. They are getting stuck in 'running', or I manually label
> them as failed:
> !image-2018-09-26-12-39-03-094.png!
> h2. Steps to reproduce
> # create new conda environment
> ** conda create -n airflow
> ** source activate airflow
> # install airflow
> ** pip install apache-airflow
> # initialize Airflow db
> ** airflow initdb
> # disable default paused setting in airflow.cfg
> ** dags_are_paused_at_creation = False
> # {color:#6a8759}run airflow and airflow scheduler (in separate
> terminal){color}
> ** {color:#6a8759}airflow scheduler{color}
> ** {color:#6a8759}airflow webserver{color}
> # {color:#6a8759}unpause example_bash_operator{color}
> ** {color:#6a8759}airflow unpause example_bash_operator{color}
> # {color:#6a8759}log in to Airflow UI{color}
> # {color:#6a8759}turn on example_bash_operator{color}
> # {color:#6a8759}click "Trigger DAG" in `example_bash_operator` row{color}
> h2. {color:#6a8759}Observed result{color}
> {color:#6a8759}The `example_bash_operator` never leaves the "running"
> state.{color}
> h2. {color:#6a8759}Expected result{color}
> {color:#6a8759}The `example_bash_operator` would quickly enter the "success"
> state{color}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] ashb commented on issue #3873: [Airflow-2760] Decouple DAG parsing loop from scheduler loop
ashb commented on issue #3873: [Airflow-2760] Decouple DAG parsing loop from scheduler loop URL: https://github.com/apache/incubator-airflow/pull/3873#issuecomment-424696610 Please add more docs, possibly including the diagram - having more detail about how the scheduler works in our docs would be a Very Good Thing. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] ron819 commented on issue #2334: [AIRFLOW-1252] API - Fix when conf is in JSON body
ron819 commented on issue #2334: [AIRFLOW-1252] API - Fix when conf is in JSON body URL: https://github.com/apache/incubator-airflow/pull/2334#issuecomment-424689055 @Fokko any news? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] ron819 commented on issue #2015: [AIRFLOW-765] Auto detect dag dependency files, variables, and resour…
ron819 commented on issue #2015: [AIRFLOW-765] Auto detect dag dependency files, variables, and resour… URL: https://github.com/apache/incubator-airflow/pull/2015#issuecomment-424687630 @artwr @aminghadersohi is this still active? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] mishikaSingh commented on issue #3650: [AIRFLOW-2703] exceptions from scheduler's heartbeat is handled so that scheduler does not crash
mishikaSingh commented on issue #3650: [AIRFLOW-2703] exceptions from scheduler's heartbeat is handled so that scheduler does not crash URL: https://github.com/apache/incubator-airflow/pull/3650#issuecomment-424680530 @ashb As we are handling only OperationalError so this will handle only "Exception raised for errors that are related to the database's operation and not necessarily under the control of the programmer, e.g. an unexpected disconnect occurs, the data source name is not found, a transaction could not be processed, a memory allocation error occurred during processing, etc." we think that these intermittent issues should not bring the whole scheduler down. Any other DB related issue will bring the scheduler down as they will come under different class . |__Error |__InterfaceError |__DatabaseError |__DataError |___OperationalError_ |__IntegrityError |__InternalError |__ProgrammingError |__NotSupportedError This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] BasPH commented on issue #3941: [AIRFLOW-3106] Validate Postgres connection after saving it
BasPH commented on issue #3941: [AIRFLOW-3106] Validate Postgres connection after saving it URL: https://github.com/apache/incubator-airflow/pull/3941#issuecomment-424671583 Thanks. Will see if I can process @Fokko 's comments tonight. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-3118) DAGs not successful on new installation
[
https://issues.apache.org/jira/browse/AIRFLOW-3118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16628555#comment-16628555
]
Ash Berlin-Taylor commented on AIRFLOW-3118:
Can you check in the scheduler log? See if there is anything "interesting"
there (or attach a small sample there).
One other thing to check: In the terminal where you will run the scheduler run
{{airflow list_dags -r}} - it should list your example_bash_operator. Does it?
What start date have you set on the dag?
> DAGs not successful on new installation
> ---
>
> Key: AIRFLOW-3118
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3118
> Project: Apache Airflow
> Issue Type: Bug
> Components: DAG
>Affects Versions: 1.10.0
> Environment: Ubuntu 18.04
> Python 3.6
>Reporter: Brylie Christopher Oxley
>Priority: Blocker
> Attachments: image-2018-09-26-12-39-03-094.png
>
>
> When trying out Airflow, on localhost, none of the DAG runs are getting to
> the 'success' state. They are getting stuck in 'running', or I manually label
> them as failed:
> !image-2018-09-26-12-39-03-094.png!
> h2. Steps to reproduce
> # create new conda environment
> ** conda create -n airflow
> ** source activate airflow
> # install airflow
> ** pip install apache-airflow
> # initialize Airflow db
> ** airflow initdb
> # disable default paused setting in airflow.cfg
> ** dags_are_paused_at_creation = False
> # {color:#6a8759}run airflow and airflow scheduler (in separate
> terminal){color}
> ** {color:#6a8759}airflow scheduler{color}
> ** {color:#6a8759}airflow webserver{color}
> # {color:#6a8759}unpause example_bash_operator{color}
> ** {color:#6a8759}airflow unpause example_bash_operator{color}
> # {color:#6a8759}log in to Airflow UI{color}
> # {color:#6a8759}turn on example_bash_operator{color}
> # {color:#6a8759}click "Trigger DAG" in `example_bash_operator` row{color}
> h2. {color:#6a8759}Observed result{color}
> {color:#6a8759}The `example_bash_operator` never leaves the "running"
> state.{color}
> h2. {color:#6a8759}Expected result{color}
> {color:#6a8759}The `example_bash_operator` would quickly enter the "success"
> state{color}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AIRFLOW-3118) DAGs not successful on new installation
[
https://issues.apache.org/jira/browse/AIRFLOW-3118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16628551#comment-16628551
]
Brylie Christopher Oxley commented on AIRFLOW-3118:
---
We have, so far, tried to set up Airflow on two computers (Linux and Mac). In
both cases, the example DAGs never get out of the 'running' state.
> DAGs not successful on new installation
> ---
>
> Key: AIRFLOW-3118
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3118
> Project: Apache Airflow
> Issue Type: Bug
> Components: DAG
>Affects Versions: 1.10.0
> Environment: Ubuntu 18.04
> Python 3.6
>Reporter: Brylie Christopher Oxley
>Priority: Blocker
> Attachments: image-2018-09-26-12-39-03-094.png
>
>
> When trying out Airflow, on localhost, none of the DAG runs are getting to
> the 'success' state. They are getting stuck in 'running', or I manually label
> them as failed:
> !image-2018-09-26-12-39-03-094.png!
> h2. Steps to reproduce
> # create new conda environment
> ** conda create -n airflow
> ** source activate airflow
> # install airflow
> ** pip install apache-airflow
> # initialize Airflow db
> ** airflow initdb
> # disable default paused setting in airflow.cfg
> ** dags_are_paused_at_creation = False
> # {color:#6a8759}run airflow and airflow scheduler (in separate
> terminal){color}
> ** {color:#6a8759}airflow scheduler{color}
> ** {color:#6a8759}airflow webserver{color}
> # {color:#6a8759}unpause example_bash_operator{color}
> ** {color:#6a8759}airflow unpause example_bash_operator{color}
> # {color:#6a8759}log in to Airflow UI{color}
> # {color:#6a8759}turn on example_bash_operator{color}
> # {color:#6a8759}click "Trigger DAG" in `example_bash_operator` row{color}
> h2. {color:#6a8759}Observed result{color}
> {color:#6a8759}The `example_bash_operator` never leaves the "running"
> state.{color}
> h2. {color:#6a8759}Expected result{color}
> {color:#6a8759}The `example_bash_operator` would quickly enter the "success"
> state{color}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (AIRFLOW-3118) DAGs not successful on new installation
[
https://issues.apache.org/jira/browse/AIRFLOW-3118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brylie Christopher Oxley updated AIRFLOW-3118:
--
Description:
When trying out Airflow, on localhost, none of the DAG runs are getting to the
'success' state. They are getting stuck in 'running', or I manually label them
as failed:
!image-2018-09-26-12-39-03-094.png!
h2. Steps to reproduce
# create new conda environment
** conda create -n airflow
** source activate airflow
# install airflow
** pip install apache-airflow
# initialize Airflow db
** airflow initdb
# disable default paused setting in airflow.cfg
** dags_are_paused_at_creation = False
# {color:#6a8759}run airflow and airflow scheduler (in separate
terminal){color}
** {color:#6a8759}airflow scheduler{color}
** {color:#6a8759}airflow webserver{color}
# {color:#6a8759}unpause example_bash_operator{color}
** {color:#6a8759}airflow unpause example_bash_operator{color}
# {color:#6a8759}log in to Airflow UI{color}
# {color:#6a8759}turn on example_bash_operator{color}
# {color:#6a8759}click "Trigger DAG" in `example_bash_operator` row{color}
h2. {color:#6a8759}Observed result{color}
{color:#6a8759}The `example_bash_operator` never leaves the "running"
state.{color}
h2. {color:#6a8759}Expected result{color}
{color:#6a8759}The `example_bash_operator` would quickly enter the "success"
state{color}
was:
When trying out Airflow, on localhost, none of the DAG runs are getting to the
'success' state. They are getting stuck in 'running', or I manually label them
as failed:
!image-2018-09-26-12-39-03-094.png!
h2. Steps to reproduce
# create new conda environment
** conda create -n airflow
** source activate airflow
# install airflow
** pip install apache-airflow
# initialize Airflow db
** airflow initdb
# disable default paused setting in airflow.cfg
** dags_are_paused_at_creation = False
# {color:#6a8759}run airflow and airflow scheduler (in separate
terminal){color}
** {color:#6a8759}airflow scheduler{color}
** {color:#6a8759}`airflow server`{color}
# {color:#6a8759}unpause example_bash_operator{color}
** {color:#6a8759}airflow unpause example_bash_operator{color}
# {color:#6a8759}log in to Airflow UI{color}
# {color:#6a8759}turn on example_bash_operator{color}
# {color:#6a8759}click "Trigger DAG" in `example_bash_operator` row{color}
h2. {color:#6a8759}Observed result{color}
{color:#6a8759}The `example_bash_operator` never leaves the "running"
state.{color}
h2. {color:#6a8759}Expected result{color}
{color:#6a8759}The `example_bash_operator` would quickly enter the "success"
state{color}
> DAGs not successful on new installation
> ---
>
> Key: AIRFLOW-3118
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3118
> Project: Apache Airflow
> Issue Type: Bug
> Components: DAG
>Affects Versions: 1.10.0
> Environment: Ubuntu 18.04
> Python 3.6
>Reporter: Brylie Christopher Oxley
>Priority: Blocker
> Attachments: image-2018-09-26-12-39-03-094.png
>
>
> When trying out Airflow, on localhost, none of the DAG runs are getting to
> the 'success' state. They are getting stuck in 'running', or I manually label
> them as failed:
> !image-2018-09-26-12-39-03-094.png!
> h2. Steps to reproduce
> # create new conda environment
> ** conda create -n airflow
> ** source activate airflow
> # install airflow
> ** pip install apache-airflow
> # initialize Airflow db
> ** airflow initdb
> # disable default paused setting in airflow.cfg
> ** dags_are_paused_at_creation = False
> # {color:#6a8759}run airflow and airflow scheduler (in separate
> terminal){color}
> ** {color:#6a8759}airflow scheduler{color}
> ** {color:#6a8759}airflow webserver{color}
> # {color:#6a8759}unpause example_bash_operator{color}
> ** {color:#6a8759}airflow unpause example_bash_operator{color}
> # {color:#6a8759}log in to Airflow UI{color}
> # {color:#6a8759}turn on example_bash_operator{color}
> # {color:#6a8759}click "Trigger DAG" in `example_bash_operator` row{color}
> h2. {color:#6a8759}Observed result{color}
> {color:#6a8759}The `example_bash_operator` never leaves the "running"
> state.{color}
> h2. {color:#6a8759}Expected result{color}
> {color:#6a8759}The `example_bash_operator` would quickly enter the "success"
> state{color}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (AIRFLOW-3106) Validate Postgres connection when saving
[ https://issues.apache.org/jira/browse/AIRFLOW-3106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ash Berlin-Taylor updated AIRFLOW-3106: --- Priority: Minor (was: Major) Summary: Validate Postgres connection when saving (was: Validate connection when saving connection) > Validate Postgres connection when saving > > > Key: AIRFLOW-3106 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3106 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Bas Harenslak >Priority: Minor > > I've encountered failures in DAG runs at various occasions due to invalid > connection credentials, or a domain was unreachable from the Airflow instance. > It'd be nice to validate a connection when saving it, to directly know if a > given connection can be made or not. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] ashb commented on issue #3941: [AIRFLOW-3106] Validate Postgres connection after saving it
ashb commented on issue #3941: [AIRFLOW-3106] Validate Postgres connection after saving it URL: https://github.com/apache/incubator-airflow/pull/3941#issuecomment-424664381 Oh yes, I was basing my comment on the title of the this Jira - I've changed that to be more specific and linked the two issues. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (AIRFLOW-3111) Confusing comments and instructions for log templates in UPDATING.md and default_airflow.cfg
[
https://issues.apache.org/jira/browse/AIRFLOW-3111?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ash Berlin-Taylor updated AIRFLOW-3111:
---
Fix Version/s: 1.10.1
> Confusing comments and instructions for log templates in UPDATING.md and
> default_airflow.cfg
>
>
> Key: AIRFLOW-3111
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3111
> Project: Apache Airflow
> Issue Type: Bug
>Reporter: Eric Chang
>Assignee: Eric Chang
>Priority: Minor
> Fix For: 1.10.1
>
>
> The new 1.10 release includes instructions on updating the *airflow.cfg*
> options core.log_filename_template and
> elasticsearch.elasticsearch_log_id_template that are technically incorrect:
>
> {code:java}
> // UPDATING.md
> ...the following defaults need to be added.
> [core]
> log_filename_template = ti.dag_id / ti.task_id / ts
> / try_number .log
> [elasticsearch]
> elasticsearch_log_id_template =
> {{dag_id}}-{{task_id}}-{{execution_date}}-{{try_number}}{code}
>
> Inserting the above options into an existing *airflow.cfg* will result in a
> jinja parse failure because `` is not a valid template string. The reason
> the extra braces are necessary is because the default *airflow.cfg* copied to
> AIRFLOW_HOME is a rendered version of *default_airflow.cfg* (rendered by
> *airflow.configuration.parameterized_config*).
> The confusion is compounded by several comments copied over to *airflow.cfg*
> stating that
>
> {code:java}
> # we need to escape the curly braces by adding an additional curly brace
> {code}
> But that's is only true for *default_airflow.cfg* and doesn't apply to the
> actual *airflow.cfg*.
> I'll submit a PR that corrects this.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (AIRFLOW-3109) Default user permission should contain 'can_clear'
[ https://issues.apache.org/jira/browse/AIRFLOW-3109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ash Berlin-Taylor resolved AIRFLOW-3109. Resolution: Fixed Fix Version/s: 1.10.1 > Default user permission should contain 'can_clear' > -- > > Key: AIRFLOW-3109 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3109 > Project: Apache Airflow > Issue Type: Bug >Reporter: Joy Gao >Assignee: Joy Gao >Priority: Major > Fix For: 1.10.1 > > > The default user role is missing 'can_clear' permission which allows user to > clear DAG runs. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] kaxil commented on a change in pull request #3043: AIRFLOW-2107 add time_partitioning to run_query on BigQueryBaseCursor
kaxil commented on a change in pull request #3043: AIRFLOW-2107 add
time_partitioning to run_query on BigQueryBaseCursor
URL: https://github.com/apache/incubator-airflow/pull/3043#discussion_r220499641
##
File path: airflow/contrib/hooks/bigquery_hook.py
##
@@ -1485,3 +1490,17 @@ def var_print(var_name):
project_id = default_project_id
return project_id, dataset_id, table_id
+
+
+def _cleanse_time_partitioning(destination_dataset_table,
time_partitioning_in):
+# if it is a partitioned table ($ is in the table name) add partition load
option
+time_partitioning_out = {}
+if destination_dataset_table and '$' in destination_dataset_table:
+assert not time_partitioning_in.get('field'), (
Review comment:
Sorry, silly me. I meant this will be available in **Airflow 1.10.1** (Next
Release) . :)
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] kaxil commented on a change in pull request #3043: AIRFLOW-2107 add time_partitioning to run_query on BigQueryBaseCursor
kaxil commented on a change in pull request #3043: AIRFLOW-2107 add
time_partitioning to run_query on BigQueryBaseCursor
URL: https://github.com/apache/incubator-airflow/pull/3043#discussion_r220499641
##
File path: airflow/contrib/hooks/bigquery_hook.py
##
@@ -1485,3 +1490,17 @@ def var_print(var_name):
project_id = default_project_id
return project_id, dataset_id, table_id
+
+
+def _cleanse_time_partitioning(destination_dataset_table,
time_partitioning_in):
+# if it is a partitioned table ($ is in the table name) add partition load
option
+time_partitioning_out = {}
+if destination_dataset_table and '$' in destination_dataset_table:
+assert not time_partitioning_in.get('field'), (
Review comment:
Sorry, silly me. I meant this will be available in **Airflow 1.10.1** . :)
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] kaxil commented on a change in pull request #3043: AIRFLOW-2107 add time_partitioning to run_query on BigQueryBaseCursor
kaxil commented on a change in pull request #3043: AIRFLOW-2107 add
time_partitioning to run_query on BigQueryBaseCursor
URL: https://github.com/apache/incubator-airflow/pull/3043#discussion_r220499641
##
File path: airflow/contrib/hooks/bigquery_hook.py
##
@@ -1485,3 +1490,17 @@ def var_print(var_name):
project_id = default_project_id
return project_id, dataset_id, table_id
+
+
+def _cleanse_time_partitioning(destination_dataset_table,
time_partitioning_in):
+# if it is a partitioned table ($ is in the table name) add partition load
option
+time_partitioning_out = {}
+if destination_dataset_table and '$' in destination_dataset_table:
+assert not time_partitioning_in.get('field'), (
Review comment:
Sorry, silly me. I meant this will be available in *Airflow 1.10.1*. :)
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] kaxil commented on a change in pull request #3043: AIRFLOW-2107 add time_partitioning to run_query on BigQueryBaseCursor
kaxil commented on a change in pull request #3043: AIRFLOW-2107 add
time_partitioning to run_query on BigQueryBaseCursor
URL: https://github.com/apache/incubator-airflow/pull/3043#discussion_r220342170
##
File path: airflow/contrib/hooks/bigquery_hook.py
##
@@ -1485,3 +1490,17 @@ def var_print(var_name):
project_id = default_project_id
return project_id, dataset_id, table_id
+
+
+def _cleanse_time_partitioning(destination_dataset_table,
time_partitioning_in):
+# if it is a partitioned table ($ is in the table name) add partition load
option
+time_partitioning_out = {}
+if destination_dataset_table and '$' in destination_dataset_table:
+assert not time_partitioning_in.get('field'), (
Review comment:
This was fixed in https://github.com/apache/incubator-airflow/pull/3901 and
will be available in **Airflow 1.10.1**
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Updated] (AIRFLOW-3118) DAGs not successful on new installation
[
https://issues.apache.org/jira/browse/AIRFLOW-3118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brylie Christopher Oxley updated AIRFLOW-3118:
--
Description:
When trying out Airflow, on localhost, none of the DAG runs are getting to the
'success' state. They are getting stuck in 'running', or I manually label them
as failed:
!image-2018-09-26-12-39-03-094.png!
h2. Steps to reproduce
# create new conda environment
** conda create -n airflow
** source activate airflow
# install airflow
** pip install apache-airflow
# initialize Airflow db
** airflow initdb
# disable default paused setting in airflow.cfg
** dags_are_paused_at_creation = False
# {color:#6a8759}run airflow and airflow scheduler (in separate
terminal){color}
** {color:#6a8759}airflow scheduler{color}
** {color:#6a8759}`airflow server`{color}
# {color:#6a8759}unpause example_bash_operator{color}
** {color:#6a8759}airflow unpause example_bash_operator{color}
# {color:#6a8759}log in to Airflow UI{color}
# {color:#6a8759}turn on example_bash_operator{color}
# {color:#6a8759}click "Trigger DAG" in `example_bash_operator` row{color}
h2. {color:#6a8759}Observed result{color}
{color:#6a8759}The `example_bash_operator` never leaves the "running"
state.{color}
h2. {color:#6a8759}Expected result{color}
{color:#6a8759}The `example_bash_operator` would quickly enter the "success"
state{color}
was:
When trying out Airflow, on localhost, none of the DAG runs are getting to the
'success' state. They are getting stuck in 'running', or I manually label them
as failed:
!image-2018-09-26-12-39-03-094.png!
h2. Steps to reproduce
# create new conda environment
**
{code:java}
conda create -n airflow{code}
**
{code:java}
source activate airflow{code}
# install airflow
**
{code:java}
pip install apache-airflow{code}
# initialize Airflow db
**
{code:java}
airflow initdb{code}
# disable default paused setting in airflow.cfg
##
{code:java}
dags_are_paused_at_creation = False{code}
# {color:#6a8759}run airflow and airflow scheduler (in separate
terminal){color}
** {color:#6a8759}airflow scheduler{color}
** {color:#6a8759}`airflow server`{color}
# {color:#6a8759}unpause `exam`ple_bash_operator`{color}
** {color:#6a8759}`airflow unpause example_bash_operator`{color}
# {color:#6a8759}log in to Airflow UI{color}
# {color:#6a8759}turn on `example_bash_operator`{color}
# {color:#6a8759}click "Trigger DAG" in `example_bash_operator` row{color}
h2. {color:#6a8759}Observed result{color}
{color:#6a8759}The `example_bash_operator` never leaves the "running"
state.{color}
h2. {color:#6a8759}Expected result{color}
{color:#6a8759}The `example_bash_operator` would quickly enter the "success"
state{color}
> DAGs not successful on new installation
> ---
>
> Key: AIRFLOW-3118
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3118
> Project: Apache Airflow
> Issue Type: Bug
> Components: DAG
>Affects Versions: 1.10.0
> Environment: Ubuntu 18.04
> Python 3.6
>Reporter: Brylie Christopher Oxley
>Priority: Blocker
> Attachments: image-2018-09-26-12-39-03-094.png
>
>
> When trying out Airflow, on localhost, none of the DAG runs are getting to
> the 'success' state. They are getting stuck in 'running', or I manually label
> them as failed:
> !image-2018-09-26-12-39-03-094.png!
> h2. Steps to reproduce
> # create new conda environment
> ** conda create -n airflow
> ** source activate airflow
> # install airflow
> ** pip install apache-airflow
> # initialize Airflow db
> ** airflow initdb
> # disable default paused setting in airflow.cfg
> ** dags_are_paused_at_creation = False
> # {color:#6a8759}run airflow and airflow scheduler (in separate
> terminal){color}
> ** {color:#6a8759}airflow scheduler{color}
> ** {color:#6a8759}`airflow server`{color}
> # {color:#6a8759}unpause example_bash_operator{color}
> ** {color:#6a8759}airflow unpause example_bash_operator{color}
> # {color:#6a8759}log in to Airflow UI{color}
> # {color:#6a8759}turn on example_bash_operator{color}
> # {color:#6a8759}click "Trigger DAG" in `example_bash_operator` row{color}
> h2. {color:#6a8759}Observed result{color}
> {color:#6a8759}The `example_bash_operator` never leaves the "running"
> state.{color}
> h2. {color:#6a8759}Expected result{color}
> {color:#6a8759}The `example_bash_operator` would quickly enter the "success"
> state{color}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (AIRFLOW-3118) DAGs not successful on new installation
[
https://issues.apache.org/jira/browse/AIRFLOW-3118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brylie Christopher Oxley updated AIRFLOW-3118:
--
Description:
When trying out Airflow, on localhost, none of the DAG runs are getting to the
'success' state. They are getting stuck in 'running', or I manually label them
as failed:
!image-2018-09-26-12-39-03-094.png!
h2. Steps to reproduce
# create new conda environment
**
{code:java}
conda create -n airflow{code}
**
{code:java}
source activate airflow{code}
# install airflow
**
{code:java}
pip install apache-airflow{code}
# initialize Airflow db
**
{code:java}
airflow initdb{code}
# disable default paused setting in airflow.cfg
##
{code:java}
dags_are_paused_at_creation = False{code}
# {color:#6a8759}run airflow and airflow scheduler (in separate
terminal){color}
** {color:#6a8759}airflow scheduler{color}
** {color:#6a8759}`airflow server`{color}
# {color:#6a8759}unpause `exam`ple_bash_operator`{color}
** {color:#6a8759}`airflow unpause example_bash_operator`{color}
# {color:#6a8759}log in to Airflow UI{color}
# {color:#6a8759}turn on `example_bash_operator`{color}
# {color:#6a8759}click "Trigger DAG" in `example_bash_operator` row{color}
h2. {color:#6a8759}Observed result{color}
{color:#6a8759}The `example_bash_operator` never leaves the "running"
state.{color}
h2. {color:#6a8759}Expected result{color}
{color:#6a8759}The `example_bash_operator` would quickly enter the "success"
state{color}
was:
When trying out Airflow, on localhost, none of the DAG runs are getting to the
'success' state. They are getting stuck in 'running', or I manually label them
as failed:
!image-2018-09-26-12-39-03-094.png!
h2. Steps to reproduce
# create new conda environment
** `conda create -n airflow`
** `source activate airflow`
# install airflow
** `pip install apache-airflow`
# initialize Airflow db
** `airflow initdb`
# disable default paused setting in airflow.cfg
## `dags_are_paused_at_creation = {color:#6a8759}False`{color}
# {color:#6a8759}run airflow and airflow scheduler (in separate
terminal){color}
** {color:#6a8759}`airflow scheduler`{color}
** {color:#6a8759}`airflow server`{color}
# {color:#6a8759}unpause `exam`ple_bash_operator`{color}
** {color:#6a8759}`airflow unpause example_bash_operator`{color}
# {color:#6a8759}log in to Airflow UI{color}
# {color:#6a8759}turn on `example_bash_operator`{color}
# {color:#6a8759}click "Trigger DAG" in `example_bash_operator` row{color}
h2. {color:#6a8759}Observed result{color}
{color:#6a8759}The `example_bash_operator` never leaves the "running"
state.{color}
h2. {color:#6a8759}Expected result{color}
{color:#6a8759}The `example_bash_operator` would quickly enter the "success"
state{color}
> DAGs not successful on new installation
> ---
>
> Key: AIRFLOW-3118
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3118
> Project: Apache Airflow
> Issue Type: Bug
> Components: DAG
>Affects Versions: 1.10.0
> Environment: Ubuntu 18.04
> Python 3.6
>Reporter: Brylie Christopher Oxley
>Priority: Blocker
> Attachments: image-2018-09-26-12-39-03-094.png
>
>
> When trying out Airflow, on localhost, none of the DAG runs are getting to
> the 'success' state. They are getting stuck in 'running', or I manually label
> them as failed:
> !image-2018-09-26-12-39-03-094.png!
> h2. Steps to reproduce
> # create new conda environment
> **
> {code:java}
> conda create -n airflow{code}
> **
> {code:java}
> source activate airflow{code}
> # install airflow
> **
> {code:java}
> pip install apache-airflow{code}
> # initialize Airflow db
> **
> {code:java}
> airflow initdb{code}
> # disable default paused setting in airflow.cfg
> ##
> {code:java}
> dags_are_paused_at_creation = False{code}
> # {color:#6a8759}run airflow and airflow scheduler (in separate
> terminal){color}
> ** {color:#6a8759}airflow scheduler{color}
> ** {color:#6a8759}`airflow server`{color}
> # {color:#6a8759}unpause `exam`ple_bash_operator`{color}
> ** {color:#6a8759}`airflow unpause example_bash_operator`{color}
> # {color:#6a8759}log in to Airflow UI{color}
> # {color:#6a8759}turn on `example_bash_operator`{color}
> # {color:#6a8759}click "Trigger DAG" in `example_bash_operator` row{color}
> h2. {color:#6a8759}Observed result{color}
> {color:#6a8759}The `example_bash_operator` never leaves the "running"
> state.{color}
> h2. {color:#6a8759}Expected result{color}
> {color:#6a8759}The `example_bash_operator` would quickly enter the "success"
> state{color}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (AIRFLOW-3118) DAGs not successful on new installation
Brylie Christopher Oxley created AIRFLOW-3118:
-
Summary: DAGs not successful on new installation
Key: AIRFLOW-3118
URL: https://issues.apache.org/jira/browse/AIRFLOW-3118
Project: Apache Airflow
Issue Type: Bug
Components: DAG
Affects Versions: 1.10.0
Environment: Ubuntu 18.04
Python 3.6
Reporter: Brylie Christopher Oxley
Attachments: image-2018-09-26-12-39-03-094.png
When trying out Airflow, on localhost, none of the DAG runs are getting to the
'success' state. They are getting stuck in 'running', or I manually label them
as failed:
!image-2018-09-26-12-39-03-094.png!
h2. Steps to reproduce
# create new conda environment
** `conda create -n airflow`
** `source activate airflow`
# install airflow
** `pip install apache-airflow`
# initialize Airflow db
** `airflow initdb`
# disable default paused setting in airflow.cfg
## `dags_are_paused_at_creation = {color:#6a8759}False`{color}
# {color:#6a8759}run airflow and airflow scheduler (in separate
terminal){color}
** {color:#6a8759}`airflow scheduler`{color}
** {color:#6a8759}`airflow server`{color}
# {color:#6a8759}unpause `exam`ple_bash_operator`{color}
** {color:#6a8759}`airflow unpause example_bash_operator`{color}
# {color:#6a8759}log in to Airflow UI{color}
# {color:#6a8759}turn on `example_bash_operator`{color}
# {color:#6a8759}click "Trigger DAG" in `example_bash_operator` row{color}
h2. {color:#6a8759}Observed result{color}
{color:#6a8759}The `example_bash_operator` never leaves the "running"
state.{color}
h2. {color:#6a8759}Expected result{color}
{color:#6a8759}The `example_bash_operator` would quickly enter the "success"
state{color}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (AIRFLOW-2979) Deprecated Celery Option not in Options list
[
https://issues.apache.org/jira/browse/AIRFLOW-2979?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ash Berlin-Taylor updated AIRFLOW-2979:
---
Fix Version/s: 1.10.1
> Deprecated Celery Option not in Options list
> -
>
> Key: AIRFLOW-2979
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2979
> Project: Apache Airflow
> Issue Type: Bug
> Components: celery
>Affects Versions: 1.10.0
>Reporter: Micheal Ascah
>Assignee: Kaxil Naik
>Priority: Critical
> Fix For: 2.0.0, 1.10.1
>
>
> References AIRFLOW-1840
> In airflow/configuration.py
> {code:java}
> # A two-level mapping of (section -> new_name -> old_name). When reading
> # new_name, the old_name will be checked to see if it exists. If it does a
> # DeprecationWarning will be issued and the old name will be used instead
> deprecated_options = {
> 'celery': {
> # Remove these keys in Airflow 1.11
> 'worker_concurrency': 'celeryd_concurrency',
> 'broker_url': 'celery_broker_url',
> 'ssl_active': 'celery_ssl_active',
> 'ssl_cert': 'celery_ssl_cert',
> 'ssl_key': 'celery_ssl_key',
> }
> }
> {code}
> This block is missing the renaming of celery_result_backend to just
> result_backed.
>
> When setting this through an environment variable, the deprecated config name
> is not being used and instead the default value in the file is being used.
> This is obviously remedied by the reading the UPDATING and setting the new
> name, but this change has broken back compat as far as I can tell.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (AIRFLOW-2716) Replace new Python 3.7 keywords
[ https://issues.apache.org/jira/browse/AIRFLOW-2716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ash Berlin-Taylor updated AIRFLOW-2716: --- Affects Version/s: (was: 2.0.0) 1.10.0 Fix Version/s: 1.10.1 > Replace new Python 3.7 keywords > --- > > Key: AIRFLOW-2716 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2716 > Project: Apache Airflow > Issue Type: Bug >Affects Versions: 1.10.0 >Reporter: Jacob Hayes >Assignee: Jacob Hayes >Priority: Major > Fix For: 2.0.0, 1.10.1 > > > Python 3.7 added `async` and `await` as reserved keywords, so they need to be > replaced with alternative names. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AIRFLOW-3117) Add installation instructions for including unidecode GPL dependency
[ https://issues.apache.org/jira/browse/AIRFLOW-3117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ash Berlin-Taylor resolved AIRFLOW-3117. Resolution: Fixed Fix Version/s: 2.0.0 > Add installation instructions for including unidecode GPL dependency > > > Key: AIRFLOW-3117 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3117 > Project: Apache Airflow > Issue Type: Improvement > Components: Documentation >Reporter: Brylie Christopher Oxley >Assignee: Brylie Christopher Oxley >Priority: Trivial > Labels: newbie > Fix For: 2.0.0 > > > The installation instructions do not currently describe how to explictly > _allow_ the GPL dependency. In fairness, the documentation should describe > both scenarios, allowing and denying the GPL dependency. That way, end-users > can make an informed decision. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] ashb closed pull request #3949: [AIRFLOW-3117] Add instructions to allow GPL dependency
ashb closed pull request #3949: [AIRFLOW-3117] Add instructions to allow GPL
dependency
URL: https://github.com/apache/incubator-airflow/pull/3949
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/docs/installation.rst b/docs/installation.rst
index 921f2c8d46..3db48e45dd 100644
--- a/docs/installation.rst
+++ b/docs/installation.rst
@@ -19,6 +19,9 @@ You can also install Airflow with support for extra features
like ``s3`` or ``po
.. note:: GPL dependency
One of the dependencies of Apache Airflow by default pulls in a GPL
library ('unidecode').
+
+If you are not concerned about the GPL dependency, export the following
environment variable prior to installing airflow: ``export
AIRFLOW_GPL_UNIDECODE=yes``.
+
In case this is a concern you can force a non GPL library by issuing
``export SLUGIFY_USES_TEXT_UNIDECODE=yes`` and then proceed with the
normal installation.
Please note that this needs to be specified at every upgrade. Also note
that if `unidecode`
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Commented] (AIRFLOW-3117) Add installation instructions for including unidecode GPL dependency
[
https://issues.apache.org/jira/browse/AIRFLOW-3117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16628471#comment-16628471
]
ASF GitHub Bot commented on AIRFLOW-3117:
-
ashb closed pull request #3949: [AIRFLOW-3117] Add instructions to allow GPL
dependency
URL: https://github.com/apache/incubator-airflow/pull/3949
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/docs/installation.rst b/docs/installation.rst
index 921f2c8d46..3db48e45dd 100644
--- a/docs/installation.rst
+++ b/docs/installation.rst
@@ -19,6 +19,9 @@ You can also install Airflow with support for extra features
like ``s3`` or ``po
.. note:: GPL dependency
One of the dependencies of Apache Airflow by default pulls in a GPL
library ('unidecode').
+
+If you are not concerned about the GPL dependency, export the following
environment variable prior to installing airflow: ``export
AIRFLOW_GPL_UNIDECODE=yes``.
+
In case this is a concern you can force a non GPL library by issuing
``export SLUGIFY_USES_TEXT_UNIDECODE=yes`` and then proceed with the
normal installation.
Please note that this needs to be specified at every upgrade. Also note
that if `unidecode`
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Add installation instructions for including unidecode GPL dependency
>
>
> Key: AIRFLOW-3117
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3117
> Project: Apache Airflow
> Issue Type: Improvement
> Components: Documentation
>Reporter: Brylie Christopher Oxley
>Assignee: Brylie Christopher Oxley
>Priority: Trivial
> Labels: newbie
>
> The installation instructions do not currently describe how to explictly
> _allow_ the GPL dependency. In fairness, the documentation should describe
> both scenarios, allowing and denying the GPL dependency. That way, end-users
> can make an informed decision.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] ashb commented on issue #3650: [AIRFLOW-2703] exceptions from scheduler's heartbeat is handled so that scheduler does not crash
ashb commented on issue #3650: [AIRFLOW-2703] exceptions from scheduler's heartbeat is handled so that scheduler does not crash URL: https://github.com/apache/incubator-airflow/pull/3650#issuecomment-424648004 > If there is any continuous issue with the connection to the DB then we were expecting the external process supervisor to take care of it and restart the scheduler. Does that throw up as a different exception class? I guess my question is does this change now mask that issue and prevent the scheduler from failing over in the first case? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] ron819 commented on issue #3941: [AIRFLOW-3106] Validate Postgres connection after saving it
ron819 commented on issue #3941: [AIRFLOW-3106] Validate Postgres connection after saving it URL: https://github.com/apache/incubator-airflow/pull/3941#issuecomment-424642655 @ashb not sure. This PR offers solution only for PostgreSQL. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] ashb commented on issue #3939: [AIRFLOW-3104] Add .airflowignore info into doc
ashb commented on issue #3939: [AIRFLOW-3104] Add .airflowignore info into doc URL: https://github.com/apache/incubator-airflow/pull/3939#issuecomment-424641567 So extra thing about this - every directory is checked for an `.airflowignore`. I.e. ``` $ find dags -type f dags/.airflowignore dags/x_ignored.py dags/y_not_ignored.py dags/ignored_dir/this/whole/path/is/not/even/walked dags/lib/.airflowignore dags/lib/x_ignored.py dags/lib/y_ignored.py $ cat dags/..airflowignore ignored_dir$ x_.*\.py $ cat dags/libs/.airflowignore y_.*\.py ``` Though I've done this from memory (and sadly didn't write tests when I fixed the behaviour. Bad Ash) so a quick check I've said the right thing would be good. The key points being that the patterns are also checked against directories (and if a dir is ignored then we don't do an `os.walk` in to it, which can speed up things) and that patterns from a parent folder are also checked against the contents of sub folders.. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-3117) Add installation instructions for including unidecode GPL dependency
[ https://issues.apache.org/jira/browse/AIRFLOW-3117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16628426#comment-16628426 ] ASF GitHub Bot commented on AIRFLOW-3117: - brylie opened a new pull request #3949: [AIRFLOW-3117] Add instructions to allow GPL dependency URL: https://github.com/apache/incubator-airflow/pull/3949 The installation instructions failed to mention how to proceed with the GPL dependency. For those who are not concerned by GPL, it is useful to know how to proceed with GPL dependency. ### Description Add instructions for allowing the GPL software installation during installation process. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add installation instructions for including unidecode GPL dependency > > > Key: AIRFLOW-3117 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3117 > Project: Apache Airflow > Issue Type: Improvement > Components: Documentation >Reporter: Brylie Christopher Oxley >Assignee: Brylie Christopher Oxley >Priority: Trivial > Labels: newbie > > The installation instructions do not currently describe how to explictly > _allow_ the GPL dependency. In fairness, the documentation should describe > both scenarios, allowing and denying the GPL dependency. That way, end-users > can make an informed decision. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] brylie opened a new pull request #3949: [AIRFLOW-3117] Add instructions to allow GPL dependency
brylie opened a new pull request #3949: [AIRFLOW-3117] Add instructions to allow GPL dependency URL: https://github.com/apache/incubator-airflow/pull/3949 The installation instructions failed to mention how to proceed with the GPL dependency. For those who are not concerned by GPL, it is useful to know how to proceed with GPL dependency. ### Description Add instructions for allowing the GPL software installation during installation process. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (AIRFLOW-3117) Add installation instructions for including unidecode GPL dependency
Brylie Christopher Oxley created AIRFLOW-3117: - Summary: Add installation instructions for including unidecode GPL dependency Key: AIRFLOW-3117 URL: https://issues.apache.org/jira/browse/AIRFLOW-3117 Project: Apache Airflow Issue Type: Improvement Components: Documentation Reporter: Brylie Christopher Oxley Assignee: Brylie Christopher Oxley The installation instructions do not currently describe how to explictly _allow_ the GPL dependency. In fairness, the documentation should describe both scenarios, allowing and denying the GPL dependency. That way, end-users can make an informed decision. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] verdan commented on issue #3804: [AIRFLOW-2866] Fix missing CSRF token header when using RBAC UI
verdan commented on issue #3804: [AIRFLOW-2866] Fix missing CSRF token header when using RBAC UI URL: https://github.com/apache/incubator-airflow/pull/3804#issuecomment-424625200 @ashb exactly! This block was misplaced during the `npm` PR. This PR would probably be included in 2.0.0 (along with the `manage deps via npm`) This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] ron819 opened a new pull request #3948: clear Val of Variable from the UI
ron819 opened a new pull request #3948: clear Val of Variable from the UI URL: https://github.com/apache/incubator-airflow/pull/3948 ### Jira - [ ] My PR addresses the following [Airflow Jira](https://issues.apache.org/jira/browse/AIRFLOW-3071) issues and references them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR" - https://issues.apache.org/jira/browse/AIRFLOW-3071 - In case you are fixing a typo in the documentation you can prepend your commit with \[AIRFLOW-XXX\], code changes always need a Jira issue. ### Description Allows the user to save Variable (Key,Val) where Val is empty string. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (AIRFLOW-3116) Import / export connections from CLI (like for Variables)
jack created AIRFLOW-3116: - Summary: Import / export connections from CLI (like for Variables) Key: AIRFLOW-3116 URL: https://issues.apache.org/jira/browse/AIRFLOW-3116 Project: Apache Airflow Issue Type: Improvement Affects Versions: 1.10.0 Reporter: jack Variables have import/export from Json: airflow variables |-i, --import|Import variables from JSON file| |-e, --export|Export variables to JSON file| This behavior doesn't exist for connections [https://airflow.apache.org/cli.html] Add this behavior also for connections. Also requested on stackoverflow https://stackoverflow.com/questions/48636783/how-to-migrate-airflow-variables-between-dev-and-prod-environments -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-3115) Add comments column to Variable page in the UI
jack created AIRFLOW-3115: - Summary: Add comments column to Variable page in the UI Key: AIRFLOW-3115 URL: https://issues.apache.org/jira/browse/AIRFLOW-3115 Project: Apache Airflow Issue Type: Task Affects Versions: 1.10.0 Reporter: jack In Admin -> Variables there are Key,Val,Is Encrypted columns Please add another column that can store comment about the variables. This will also require changes with the CLI command and the Create Variable page. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] ron819 commented on issue #2369: [AIRFLOW-1159] upgrading to docker >2.0
ron819 commented on issue #2369: [AIRFLOW-1159] upgrading to docker >2.0 URL: https://github.com/apache/incubator-airflow/pull/2369#issuecomment-424602966 @elipapa any progress with this? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
