[jira] [Updated] (AIRFLOW-2814) Default Arg "file_process_interval" for class SchedulerJob is inconsistent with doc
[ https://issues.apache.org/jira/browse/AIRFLOW-2814?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaodong DENG updated AIRFLOW-2814: --- External issue URL: https://github.com/apache/incubator-airflow/pull/3659 > Default Arg "file_process_interval" for class SchedulerJob is inconsistent > with doc > --- > > Key: AIRFLOW-2814 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2814 > Project: Apache Airflow > Issue Type: Bug > Components: scheduler >Reporter: Xiaodong DENG >Assignee: Xiaodong DENG >Priority: Critical > > h2. Backgrond > In > [https://github.com/XD-DENG/incubator-airflow/blob/master/airflow/jobs.py#L592] > , it was mentioned the default value of argument *file_process_interval* > should be 3 minutes (*file_process_interval:* Parse and schedule each file no > faster than this interval). > The value is normally parsed from the default configuration. However, in the > default config_template, its value is 0 rather than 180 seconds > ([https://github.com/XD-DENG/incubator-airflow/blob/master/airflow/config_templates/default_airflow.cfg#L432] > ). > h2. Issue > This means that actually that each file is parsed and scheduled without > letting Airflow "rest". This conflicts with the design purpose (by default > let it be 180 seconds) and may affect performance significantly. > h2. My Proposal > Change the value in the config template from 0 to 180. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AIRFLOW-2563) Pig Hook Doesn't work for Python 3
[ https://issues.apache.org/jira/browse/AIRFLOW-2563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arthur Wiedmer resolved AIRFLOW-2563. - Resolution: Fixed Fix Version/s: 2.0.0 Fixed by PR #3594 > Pig Hook Doesn't work for Python 3 > -- > > Key: AIRFLOW-2563 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2563 > Project: Apache Airflow > Issue Type: Bug >Reporter: Murium Iqbal >Assignee: Jasper Kahn >Priority: Major > Fix For: 2.0.0 > > > Pig Hook doesn't work in Python3 due to differences in handling string and > bytes as described in this stackO post > https://stackoverflow.com/questions/50652034/pig-hook-in-airflow-doesnt-work-for-python3 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2563) Pig Hook Doesn't work for Python 3
[ https://issues.apache.org/jira/browse/AIRFLOW-2563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560455#comment-16560455 ] ASF subversion and git services commented on AIRFLOW-2563: -- Commit 3e7e42f028279a628d9e15d1ae4b6005593f8afb in incubator-airflow's branch refs/heads/master from [~jakahn] [ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=3e7e42f ] [AIRFLOW-2563] Fix PigCliHook Python 3 string/bytes use Unit tests added for PigCliHook as well to prevent future issues. Closes #3594 from jakahn/master > Pig Hook Doesn't work for Python 3 > -- > > Key: AIRFLOW-2563 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2563 > Project: Apache Airflow > Issue Type: Bug >Reporter: Murium Iqbal >Assignee: Jasper Kahn >Priority: Major > > Pig Hook doesn't work in Python3 due to differences in handling string and > bytes as described in this stackO post > https://stackoverflow.com/questions/50652034/pig-hook-in-airflow-doesnt-work-for-python3 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
incubator-airflow git commit: [AIRFLOW-2563] Fix PigCliHook Python 3 string/bytes use
Repository: incubator-airflow
Updated Branches:
refs/heads/master 096ba9ecd -> 3e7e42f02
[AIRFLOW-2563] Fix PigCliHook Python 3 string/bytes use
Unit tests added for PigCliHook as well to prevent
future issues.
Closes #3594 from jakahn/master
Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/3e7e42f0
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/3e7e42f0
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/3e7e42f0
Branch: refs/heads/master
Commit: 3e7e42f028279a628d9e15d1ae4b6005593f8afb
Parents: 096ba9e
Author: Jasper Kahn
Authored: Fri Jul 27 16:08:32 2018 -0700
Committer: Arthur Wiedmer
Committed: Fri Jul 27 16:08:32 2018 -0700
--
airflow/hooks/pig_hook.py| 6 +-
tests/hooks/test_pig_hook.py | 137 ++
2 files changed, 140 insertions(+), 3 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/3e7e42f0/airflow/hooks/pig_hook.py
--
diff --git a/airflow/hooks/pig_hook.py b/airflow/hooks/pig_hook.py
index a3836b1..fcfcc7f 100644
--- a/airflow/hooks/pig_hook.py
+++ b/airflow/hooks/pig_hook.py
@@ -55,7 +55,7 @@ class PigCliHook(BaseHook):
with TemporaryDirectory(prefix='airflow_pigop_') as tmp_dir:
with NamedTemporaryFile(dir=tmp_dir) as f:
-f.write(pig)
+f.write(pig.encode('utf-8'))
f.flush()
fname = f.name
pig_bin = 'pig'
@@ -76,8 +76,8 @@ class PigCliHook(BaseHook):
close_fds=True)
self.sp = sp
stdout = ''
-for line in iter(sp.stdout.readline, ''):
-stdout += line
+for line in iter(sp.stdout.readline, b''):
+stdout += line.decode('utf-8')
if verbose:
self.log.info(line.strip())
sp.wait()
http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/3e7e42f0/tests/hooks/test_pig_hook.py
--
diff --git a/tests/hooks/test_pig_hook.py b/tests/hooks/test_pig_hook.py
new file mode 100644
index 000..c250d23
--- /dev/null
+++ b/tests/hooks/test_pig_hook.py
@@ -0,0 +1,137 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import unittest
+from airflow.hooks.pig_hook import PigCliHook
+
+try:
+from unittest import mock
+except ImportError:
+try:
+import mock
+except ImportError:
+mock = None
+
+
+class TestPigCliHook(unittest.TestCase):
+
+def setUp(self):
+super(TestPigCliHook, self).setUp()
+
+self.extra_dejson = mock.MagicMock()
+self.extra_dejson.get.return_value = None
+self.conn = mock.MagicMock()
+self.conn.extra_dejson = self.extra_dejson
+conn = self.conn
+
+class SubPigCliHook(PigCliHook):
+def get_connection(self, id):
+return conn
+
+self.pig_hook = SubPigCliHook
+
+def test_init(self):
+self.pig_hook()
+self.extra_dejson.get.assert_called_with('pig_properties', '')
+
+@mock.patch('subprocess.Popen')
+def test_run_cli_success(self, popen_mock):
+proc_mock = mock.MagicMock()
+proc_mock.returncode = 0
+proc_mock.stdout.readline.return_value = b''
+popen_mock.return_value = proc_mock
+
+hook = self.pig_hook()
+stdout = hook.run_cli("")
+
+self.assertEqual(stdout, "")
+
+@mock.patch('subprocess.Popen')
+def test_run_cli_fail(self, popen_mock):
+proc_mock = mock.MagicMock()
+proc_mock.returncode = 1
+proc_mock.stdout.readline.return_value = b''
+popen_mock.return_value = proc_mock
+
+hook = self.pig_hook()
+
+from airflow.exceptions import AirflowException
+self.assertRaises(AirflowException,
[jira] [Commented] (AIRFLOW-2145) Deadlock after clearing a running task
[
https://issues.apache.org/jira/browse/AIRFLOW-2145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560442#comment-16560442
]
Abdul Nimeri commented on AIRFLOW-2145:
---
Opened a PR: https://github.com/apache/incubator-airflow/pull/3657
> Deadlock after clearing a running task
> --
>
> Key: AIRFLOW-2145
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2145
> Project: Apache Airflow
> Issue Type: Bug
>Affects Versions: 1.9.0
>Reporter: George Roldugin
>Priority: Minor
> Attachments: image-2018-02-23-18-59-11-828.png,
> image-2018-02-23-19-00-37-741.png, image-2018-02-23-19-00-55-630.png,
> image-2018-02-23-19-01-45-012.png, image-2018-02-23-19-01-57-498.png,
> image-2018-02-23-19-02-18-837.png
>
>
> TL;DR The essense of the issue is that whenever a currently running ask is
> cleared, the dagrun enters a deadlocked state and fails.
>
> We see this in production with Celery executors and {{TimeDeltaSensor}}, and
> I've been able to reproduce it locally with both {{TimeDeltaSensor}} and
> {{WebHDFSSensor}}.
> Here's the minimal example:
> {code:java}
> from datetime import datetime, timedelta
> import airflow
> from airflow.operators.sensors import TimeDeltaSensor
> from airflow.operators.dummy_operator import DummyOperator
> with airflow.DAG(
> 'foo',
> schedule_interval='@daily',
> start_date=datetime(2018, 1, 1)) as dag:
> wait_for_upstream_sla = TimeDeltaSensor(
> task_id="wait_for_upstream_sla",
> delta=timedelta(days=365*10)
> )
> do_work = DummyOperator(task_id='do_work')
> dag >> wait_for_upstream_sla >> do_work
> {code}
>
> Sequence of actions, relevant DEBUG level logs, and some UI screenshots
> {code:java}
> airflow clear foo -e 2018-02-22 --no_confirm && airflow backfill foo -s
> 2018-02-22 -e 2018-02-22{code}
> {code:java}
> [2018-02-23 17:17:45,983] {__init__.py:45} INFO - Using executor
> SequentialExecutor
> [2018-02-23 17:17:46,069] {models.py:189} INFO - Filling up the DagBag from
> /Users/grol/Drive/dev/reporting/dags
> ...
> [2018-02-23 17:17:47,563] {jobs.py:2180} DEBUG - Task instance to run
>
> state scheduled
> ...
> {code}
> !image-2018-02-23-18-59-11-828.png|width=418,height=87!
> Now we clear all DAG's tasks externally:
> {code:java}
> airflow clear foo -e 2018-02-22 --no_confirm
> {code}
> This causes the following:
> {code:java}
> [2018-02-23 17:17:55,258] {base_task_runner.py:98} INFO - Subtask:
> [2018-02-23 17:17:55,258] {sensors.py:629} INFO - Checking if the time
> (2018-02-23 16:19:00) has come
> [2018-02-23 17:17:58,844] {jobs.py:184} DEBUG - [heart] Boom.
> [2018-02-23 17:18:03,848] {jobs.py:184} DEBUG - [heart] Boom.
> [2018-02-23 17:18:08,856] {jobs.py:2585} WARNING - State of this instance has
> been externally set to shutdown. Taking the poison pill.
> [2018-02-23 17:18:08,874] {helpers.py:266} DEBUG - There are no descendant
> processes to kill
> [2018-02-23 17:18:08,875] {jobs.py:184} DEBUG - [heart] Boom.
> [2018-02-23 17:18:08,900] {helpers.py:266} DEBUG - There are no descendant
> processes to kill
> [2018-02-23 17:18:08,922] {helpers.py:266} DEBUG - There are no descendant
> processes to kill
> [2018-02-23 17:18:09,005] {sequential_executor.py:47} ERROR - Failed to
> execute task Command 'airflow run foo wait_for_upstream_sla
> 2018-02-22T00:00:00 --local -sd DAGS_FOLDER/foo.py' returned non-zero exit
> status 1.
> [2018-02-23 17:18:09,012] {jobs.py:2004} DEBUG - Executor state: failed task
>
> [2018-02-23 17:18:09,018] {models.py:4584} INFO - Updating state for foo @ 2018-02-22 00:00:00: backfill_2018-02-22T00:00:00, externally
> triggered: False> considering 2 task(s)
> [2018-02-23 17:18:09,021] {models.py:1215} DEBUG - 2018-02-22 00:00:00 [None]> dependency 'Previous Dagrun State' PASSED: True,
> The task did not have depends_on_past set.
> [2018-02-23 17:18:09,021] {models.py:1215} DEBUG - 2018-02-22 00:00:00 [None]> dependency 'Not In Retry Period' PASSED: True,
> The context specified that being in a retry period was permitted.
> [2018-02-23 17:18:09,027] {models.py:1215} DEBUG - 2018-02-22 00:00:00 [None]> dependency 'Trigger Rule' PASSED: False, Task's
> trigger rule 'all_success' requires all upstream tasks to have succeeded, but
> found 1 non-success(es). upstream_tasks_state={'skipped': 0, 'successes': 0,
> 'failed': 0, 'upstream_failed': 0, 'done': 0, 'total': 1},
> upstream_task_ids=['wait_for_upstream_sla']
> [2018-02-23 17:18:09,029] {models.py:4643} INFO - Deadlock; marking run
> triggered: False> failed
> [2018-02-23 17:18:09,045] {jobs.py:2125} INFO - [backfill progress] |
> finished run 1 of 1 | tasks waiting: 1 | succeeded: 0 | kicked_off: 1 |
> failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 1
> [2018-02-23 17:18:09,045] {jobs.py:2129} DEBUG - Finished dag
[jira] [Commented] (AIRFLOW-2145) Deadlock after clearing a running task
[
https://issues.apache.org/jira/browse/AIRFLOW-2145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560427#comment-16560427
]
Abdul Nimeri commented on AIRFLOW-2145:
---
I think I ran into the same issue. In my case, it was because the upstream task
moved into the `shutdown` state while it's being cleared, and then the dag run
believed it was deadlocked because:
* the only "unfinished" task is the downstream (e.g. `do_work`)
* all "unfinished" tasks don't have their dependencies met (because its
upstream isn't done)
I think the bug is that `shutdown` isn't included as an "unfinished" state (see
[https://github.com/apache/incubator-airflow/blob/c7a472ed6b0d8a4720f57ba1140c8cf665757167/airflow/utils/state.py#L110)]
and is considered a "finished" state. Which doesn't seem right considering
you'd never want a DAG be considered complete while you have an instance that's
`shutdown` (and might have some retries left)
> Deadlock after clearing a running task
> --
>
> Key: AIRFLOW-2145
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2145
> Project: Apache Airflow
> Issue Type: Bug
>Affects Versions: 1.9.0
>Reporter: George Roldugin
>Priority: Minor
> Attachments: image-2018-02-23-18-59-11-828.png,
> image-2018-02-23-19-00-37-741.png, image-2018-02-23-19-00-55-630.png,
> image-2018-02-23-19-01-45-012.png, image-2018-02-23-19-01-57-498.png,
> image-2018-02-23-19-02-18-837.png
>
>
> TL;DR The essense of the issue is that whenever a currently running ask is
> cleared, the dagrun enters a deadlocked state and fails.
>
> We see this in production with Celery executors and {{TimeDeltaSensor}}, and
> I've been able to reproduce it locally with both {{TimeDeltaSensor}} and
> {{WebHDFSSensor}}.
> Here's the minimal example:
> {code:java}
> from datetime import datetime, timedelta
> import airflow
> from airflow.operators.sensors import TimeDeltaSensor
> from airflow.operators.dummy_operator import DummyOperator
> with airflow.DAG(
> 'foo',
> schedule_interval='@daily',
> start_date=datetime(2018, 1, 1)) as dag:
> wait_for_upstream_sla = TimeDeltaSensor(
> task_id="wait_for_upstream_sla",
> delta=timedelta(days=365*10)
> )
> do_work = DummyOperator(task_id='do_work')
> dag >> wait_for_upstream_sla >> do_work
> {code}
>
> Sequence of actions, relevant DEBUG level logs, and some UI screenshots
> {code:java}
> airflow clear foo -e 2018-02-22 --no_confirm && airflow backfill foo -s
> 2018-02-22 -e 2018-02-22{code}
> {code:java}
> [2018-02-23 17:17:45,983] {__init__.py:45} INFO - Using executor
> SequentialExecutor
> [2018-02-23 17:17:46,069] {models.py:189} INFO - Filling up the DagBag from
> /Users/grol/Drive/dev/reporting/dags
> ...
> [2018-02-23 17:17:47,563] {jobs.py:2180} DEBUG - Task instance to run
>
> state scheduled
> ...
> {code}
> !image-2018-02-23-18-59-11-828.png|width=418,height=87!
> Now we clear all DAG's tasks externally:
> {code:java}
> airflow clear foo -e 2018-02-22 --no_confirm
> {code}
> This causes the following:
> {code:java}
> [2018-02-23 17:17:55,258] {base_task_runner.py:98} INFO - Subtask:
> [2018-02-23 17:17:55,258] {sensors.py:629} INFO - Checking if the time
> (2018-02-23 16:19:00) has come
> [2018-02-23 17:17:58,844] {jobs.py:184} DEBUG - [heart] Boom.
> [2018-02-23 17:18:03,848] {jobs.py:184} DEBUG - [heart] Boom.
> [2018-02-23 17:18:08,856] {jobs.py:2585} WARNING - State of this instance has
> been externally set to shutdown. Taking the poison pill.
> [2018-02-23 17:18:08,874] {helpers.py:266} DEBUG - There are no descendant
> processes to kill
> [2018-02-23 17:18:08,875] {jobs.py:184} DEBUG - [heart] Boom.
> [2018-02-23 17:18:08,900] {helpers.py:266} DEBUG - There are no descendant
> processes to kill
> [2018-02-23 17:18:08,922] {helpers.py:266} DEBUG - There are no descendant
> processes to kill
> [2018-02-23 17:18:09,005] {sequential_executor.py:47} ERROR - Failed to
> execute task Command 'airflow run foo wait_for_upstream_sla
> 2018-02-22T00:00:00 --local -sd DAGS_FOLDER/foo.py' returned non-zero exit
> status 1.
> [2018-02-23 17:18:09,012] {jobs.py:2004} DEBUG - Executor state: failed task
>
> [2018-02-23 17:18:09,018] {models.py:4584} INFO - Updating state for foo @ 2018-02-22 00:00:00: backfill_2018-02-22T00:00:00, externally
> triggered: False> considering 2 task(s)
> [2018-02-23 17:18:09,021] {models.py:1215} DEBUG - 2018-02-22 00:00:00 [None]> dependency 'Previous Dagrun State' PASSED: True,
> The task did not have depends_on_past set.
> [2018-02-23 17:18:09,021] {models.py:1215} DEBUG - 2018-02-22 00:00:00 [None]> dependency 'Not In Retry Period' PASSED: True,
> The context specified that being in a retry period was permitted.
> [2018-02-23 17:18:09,027] {models.py:1215} DEBUG - 2018-02-22 00:00:00
incubator-airflow git commit: [AIRFLOW-XXX] Add Strongmind to the Readme
Repository: incubator-airflow Updated Branches: refs/heads/master a338f3276 -> 096ba9ecd [AIRFLOW-XXX] Add Strongmind to the Readme Closes #3655 from tomchapin/master Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/096ba9ec Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/096ba9ec Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/096ba9ec Branch: refs/heads/master Commit: 096ba9ecd961cdaebd062599f408571ffb21165a Parents: a338f32 Author: Tom Chapin Authored: Fri Jul 27 12:39:24 2018 -0700 Committer: r39132 Committed: Fri Jul 27 12:39:29 2018 -0700 -- README.md | 1 + 1 file changed, 1 insertion(+) -- http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/096ba9ec/README.md -- diff --git a/README.md b/README.md index f5a5133..7256565 100644 --- a/README.md +++ b/README.md @@ -231,6 +231,7 @@ Currently **officially** using Airflow: 1. [Spotify](https://github.com/spotify) [[@znichols](https://github.com/znichols)] 1. [Stackspace](https://beta.stackspace.io/) 1. [Stripe](https://stripe.com) [[@jbalogh](https://github.com/jbalogh)] +1. [Strongmind](https://www.strongmind.com) [[@tomchapin](https://github.com/tomchapin) & [@wongstein](https://github.com/wongstein)] 1. [Tails.com](https://tails.com/) [[@alanmcruickshank](https://github.com/alanmcruickshank)] 1. [Thinking Machines](https://thinkingmachin.es) [[@marksteve](https://github.com/marksteve)] 1. [Thinknear](https://www.thinknear.com/) [[@d3cay1](https://github.com/d3cay1), [@ccson](https://github.com/ccson), & [@ababian](https://github.com/ababian)]
[jira] [Commented] (AIRFLOW-2813) `pip install apache-airflow` fails
[ https://issues.apache.org/jira/browse/AIRFLOW-2813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560147#comment-16560147 ] Jeff Schwab commented on AIRFLOW-2813: -- Possibly related: https://github.com/pypa/pipenv/issues/956 Any suggested work-around would be welcome, including building from source (for which I do not see instructions, even in the repo). > `pip install apache-airflow` fails > -- > > Key: AIRFLOW-2813 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2813 > Project: Apache Airflow > Issue Type: Bug >Affects Versions: 1.9.0 > Environment: Mac OS, Linux, Windows >Reporter: Jeff Schwab >Priority: Major > > `pip install apache-airflow` fails with a SyntaxError on Mac OS, and with a > different (extremely verbose) error on Linux. This happens both on my > MacBook and on a fresh Alpine Linux Docker image, and with both pip2 and > pip3; a friend just tried `pip install apache-airflow` for me on his Windows > box, and it died with yet another error. Googling quickly found someone else > seeing the same issue over a week ago: > https://gitter.im/apache/incubator-airflow?at=5b5130bac86c4f0b47201af0 > Here's a Dockerfile that consistently fails at `pip install apache-airflow`: > https://gist.github.com/jeffs/cff7cb01d4ea3969fa5a65a8bd434e5d > Please let me know what further information you would like, and/or what I am > doing wrong. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2813) `pip install apache-airflow` fails
[
https://issues.apache.org/jira/browse/AIRFLOW-2813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560121#comment-16560121
]
Jeff Schwab commented on AIRFLOW-2813:
--
Same issue with 1.10:
{code}
$ pip install git+https://github.com/apache/incubator-airflow.git@v1-10-stable
Collecting git+https://github.com/apache/incubator-airflow.git@v1-10-stable
Cloning https://github.com/apache/incubator-airflow.git (to revision
v1-10-stable) to
/private/var/folders/90/qh7gxrf543xfhmqrn7gjxbr0gn/T/pip-req-build-z7f00dvq
Complete output from command python setup.py egg_info:
Traceback (most recent call last):
File "", line 1, in
File
"/private/var/folders/90/qh7gxrf543xfhmqrn7gjxbr0gn/T/pip-req-build-z7f00dvq/setup.py",
line 105
async = [
^
SyntaxError: invalid syntax
Command "python setup.py egg_info" failed with error code 1 in
/private/var/folders/90/qh7gxrf543xfhmqrn7gjxbr0gn/T/pip-req-build-z7f00dvq/
{code}
> `pip install apache-airflow` fails
> --
>
> Key: AIRFLOW-2813
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2813
> Project: Apache Airflow
> Issue Type: Bug
>Affects Versions: 1.9.0
> Environment: Mac OS, Linux, Windows
>Reporter: Jeff Schwab
>Priority: Major
>
> `pip install apache-airflow` fails with a SyntaxError on Mac OS, and with a
> different (extremely verbose) error on Linux. This happens both on my
> MacBook and on a fresh Alpine Linux Docker image, and with both pip2 and
> pip3; a friend just tried `pip install apache-airflow` for me on his Windows
> box, and it died with yet another error. Googling quickly found someone else
> seeing the same issue over a week ago:
> https://gitter.im/apache/incubator-airflow?at=5b5130bac86c4f0b47201af0
> Here's a Dockerfile that consistently fails at `pip install apache-airflow`:
> https://gist.github.com/jeffs/cff7cb01d4ea3969fa5a65a8bd434e5d
> Please let me know what further information you would like, and/or what I am
> doing wrong.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (AIRFLOW-2813) `pip install apache-airflow` fails
[
https://issues.apache.org/jira/browse/AIRFLOW-2813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560114#comment-16560114
]
Jeff Schwab edited comment on AIRFLOW-2813 at 7/27/18 6:39 PM:
---
On macOS High Sierra 10.13.6:
{code}
$ pip install git+https://github.com/apache/incubator-airflow.git@v1-9-stable
Collecting git+https://github.com/apache/incubator-airflow.git@v1-9-stable
Cloning https://github.com/apache/incubator-airflow.git (to revision
v1-9-stable) to
/private/var/folders/90/qh7gxrf543xfhmqrn7gjxbr0gn/T/pip-req-build-0gngd_me
Complete output from command python setup.py egg_info:
Traceback (most recent call last):
File "", line 1, in
File
"/private/var/folders/90/qh7gxrf543xfhmqrn7gjxbr0gn/T/pip-req-build-0gngd_me/setup.py",
line 102
async = [
^
SyntaxError: invalid syntax
Command "python setup.py egg_info" failed with error code 1 in
/private/var/folders/90/qh7gxrf543xfhmqrn7gjxbr0gn/T/pip-req-build-0gngd_me/
{code}
was (Author: unbuggy):
{code}
$ pip install git+https://github.com/apache/incubator-airflow.git@v1-9-stable
Collecting git+https://github.com/apache/incubator-airflow.git@v1-9-stable
Cloning https://github.com/apache/incubator-airflow.git (to revision
v1-9-stable) to
/private/var/folders/90/qh7gxrf543xfhmqrn7gjxbr0gn/T/pip-req-build-0gngd_me
Complete output from command python setup.py egg_info:
Traceback (most recent call last):
File "", line 1, in
File
"/private/var/folders/90/qh7gxrf543xfhmqrn7gjxbr0gn/T/pip-req-build-0gngd_me/setup.py",
line 102
async = [
^
SyntaxError: invalid syntax
Command "python setup.py egg_info" failed with error code 1 in
/private/var/folders/90/qh7gxrf543xfhmqrn7gjxbr0gn/T/pip-req-build-0gngd_me/
{code}
> `pip install apache-airflow` fails
> --
>
> Key: AIRFLOW-2813
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2813
> Project: Apache Airflow
> Issue Type: Bug
>Affects Versions: 1.9.0
> Environment: Mac OS, Linux, Windows
>Reporter: Jeff Schwab
>Priority: Major
>
> `pip install apache-airflow` fails with a SyntaxError on Mac OS, and with a
> different (extremely verbose) error on Linux. This happens both on my
> MacBook and on a fresh Alpine Linux Docker image, and with both pip2 and
> pip3; a friend just tried `pip install apache-airflow` for me on his Windows
> box, and it died with yet another error. Googling quickly found someone else
> seeing the same issue over a week ago:
> https://gitter.im/apache/incubator-airflow?at=5b5130bac86c4f0b47201af0
> Here's a Dockerfile that consistently fails at `pip install apache-airflow`:
> https://gist.github.com/jeffs/cff7cb01d4ea3969fa5a65a8bd434e5d
> Please let me know what further information you would like, and/or what I am
> doing wrong.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2813) `pip install apache-airflow` fails
[
https://issues.apache.org/jira/browse/AIRFLOW-2813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560114#comment-16560114
]
Jeff Schwab commented on AIRFLOW-2813:
--
{{$ pip install git+https://github.com/apache/incubator-airflow.git@v1-9-stable
Collecting git+https://github.com/apache/incubator-airflow.git@v1-9-stable
Cloning https://github.com/apache/incubator-airflow.git (to revision
v1-9-stable) to
/private/var/folders/90/qh7gxrf543xfhmqrn7gjxbr0gn/T/pip-req-build-0gngd_me
Complete output from command python setup.py egg_info:
Traceback (most recent call last):
File "", line 1, in
File
"/private/var/folders/90/qh7gxrf543xfhmqrn7gjxbr0gn/T/pip-req-build-0gngd_me/setup.py",
line 102
async = [
^
SyntaxError: invalid syntax
Command "python setup.py egg_info" failed with error code 1 in
/private/var/folders/90/qh7gxrf543xfhmqrn7gjxbr0gn/T/pip-req-build-0gngd_me/
}}
> `pip install apache-airflow` fails
> --
>
> Key: AIRFLOW-2813
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2813
> Project: Apache Airflow
> Issue Type: Bug
>Affects Versions: 1.9.0
> Environment: Mac OS, Linux, Windows
>Reporter: Jeff Schwab
>Priority: Major
>
> `pip install apache-airflow` fails with a SyntaxError on Mac OS, and with a
> different (extremely verbose) error on Linux. This happens both on my
> MacBook and on a fresh Alpine Linux Docker image, and with both pip2 and
> pip3; a friend just tried `pip install apache-airflow` for me on his Windows
> box, and it died with yet another error. Googling quickly found someone else
> seeing the same issue over a week ago:
> https://gitter.im/apache/incubator-airflow?at=5b5130bac86c4f0b47201af0
> Here's a Dockerfile that consistently fails at `pip install apache-airflow`:
> https://gist.github.com/jeffs/cff7cb01d4ea3969fa5a65a8bd434e5d
> Please let me know what further information you would like, and/or what I am
> doing wrong.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (AIRFLOW-2813) `pip install apache-airflow` fails
Jeff Schwab created AIRFLOW-2813: Summary: `pip install apache-airflow` fails Key: AIRFLOW-2813 URL: https://issues.apache.org/jira/browse/AIRFLOW-2813 Project: Apache Airflow Issue Type: Bug Affects Versions: 1.9.0 Environment: Mac OS, Linux, Windows Reporter: Jeff Schwab `pip install apache-airflow` fails with a SyntaxError on Mac OS, and with a different (extremely verbose) error on Linux. This happens both on my MacBook and on a fresh Alpine Linux Docker image, and with both pip2 and pip3; a friend just tried `pip install apache-airflow` for me on his Windows box, and it died with yet another error. Googling quickly found someone else seeing the same issue over a week ago: https://gitter.im/apache/incubator-airflow?at=5b5130bac86c4f0b47201af0 Here's a Dockerfile that consistently fails at `pip install apache-airflow`: https://gist.github.com/jeffs/cff7cb01d4ea3969fa5a65a8bd434e5d Please let me know what further information you would like, and/or what I am doing wrong. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2803) Fix all ESLint issues
[ https://issues.apache.org/jira/browse/AIRFLOW-2803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560075#comment-16560075 ] Taylor Edmiston commented on AIRFLOW-2803: -- Opened up an initial PR for this here - [https://github.com/apache/incubator-airflow/pull/3656.] I'll move discussion over to GitHub for the PR. > Fix all ESLint issues > - > > Key: AIRFLOW-2803 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2803 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Verdan Mahmood >Assignee: Taylor Edmiston >Priority: Major > > Most of the JS code in Apache Airflow has linting issues which are > highlighted after the integration of ESLint. > Once AIRFLOW-2783 merged in master branch, please fix all the javascript > styling issues that we have in .js and .html files. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2810) Typo in Xcom model timestamp field
[ https://issues.apache.org/jira/browse/AIRFLOW-2810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560015#comment-16560015 ] ASF subversion and git services commented on AIRFLOW-2810: -- Commit a338f3276835af45765d24a6e6d43ad4ba4d66ba in incubator-airflow's branch refs/heads/master from [~andywilco] [ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=a338f32 ] [AIRFLOW-2810] Fix typo in Xcom model timestamp Fix typo in Xcom model timestamp field No new testing - the field is already represented in migrations Closes #3652 from andywilco/fix_datetime_typo > Typo in Xcom model timestamp field > -- > > Key: AIRFLOW-2810 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2810 > Project: Apache Airflow > Issue Type: Bug > Components: models >Affects Versions: 1.10 >Reporter: Andy Wilcox >Assignee: Andy Wilcox >Priority: Minor > > > Looks like a find/replace error. Should be type UtcDateTime, is legacy > DateTime -- This message was sent by Atlassian JIRA (v7.6.3#76005)
incubator-airflow git commit: [AIRFLOW-2810] Fix typo in Xcom model timestamp
Repository: incubator-airflow Updated Branches: refs/heads/master 233056e0d -> a338f3276 [AIRFLOW-2810] Fix typo in Xcom model timestamp Fix typo in Xcom model timestamp field No new testing - the field is already represented in migrations Closes #3652 from andywilco/fix_datetime_typo Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/a338f327 Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/a338f327 Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/a338f327 Branch: refs/heads/master Commit: a338f3276835af45765d24a6e6d43ad4ba4d66ba Parents: 233056e Author: Andy Wilcox Authored: Fri Jul 27 13:03:30 2018 -0400 Committer: Dan Davydov Committed: Fri Jul 27 13:03:36 2018 -0400 -- airflow/models.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) -- http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/a338f327/airflow/models.py -- diff --git a/airflow/models.py b/airflow/models.py index 1d832ab..cf7eb0a 100755 --- a/airflow/models.py +++ b/airflow/models.py @@ -4482,7 +4482,7 @@ class XCom(Base, LoggingMixin): key = Column(String(512)) value = Column(LargeBinary) timestamp = Column( -DateTime, default=timezone.utcnow, nullable=False) +UtcDateTime, default=timezone.utcnow, nullable=False) execution_date = Column(UtcDateTime, nullable=False) # source information
[jira] [Commented] (AIRFLOW-2803) Fix all ESLint issues
[ https://issues.apache.org/jira/browse/AIRFLOW-2803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16559908#comment-16559908 ] Taylor Edmiston commented on AIRFLOW-2803: -- Are you also running something like node v8.11.3 and npm v5.6.0? > Fix all ESLint issues > - > > Key: AIRFLOW-2803 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2803 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Verdan Mahmood >Assignee: Taylor Edmiston >Priority: Major > > Most of the JS code in Apache Airflow has linting issues which are > highlighted after the integration of ESLint. > Once AIRFLOW-2783 merged in master branch, please fix all the javascript > styling issues that we have in .js and .html files. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2803) Fix all ESLint issues
[
https://issues.apache.org/jira/browse/AIRFLOW-2803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16559895#comment-16559895
]
Taylor Edmiston commented on AIRFLOW-2803:
--
Wow - you're right that would have saved a lot of time. I'll re-run it that
way and see what's left.
I misunderstood the output of eslint when I started on this to mean "0 things
fixable with --fix" when really it meant 0 warnings fixable but many errors
fixable.
{code:java}
✖ 1056 problems (1003 errors, 53 warnings)
697 errors, 0 warnings potentially fixable with the `--fix` option.{code}
> Fix all ESLint issues
> -
>
> Key: AIRFLOW-2803
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2803
> Project: Apache Airflow
> Issue Type: Improvement
>Reporter: Verdan Mahmood
>Assignee: Taylor Edmiston
>Priority: Major
>
> Most of the JS code in Apache Airflow has linting issues which are
> highlighted after the integration of ESLint.
> Once AIRFLOW-2783 merged in master branch, please fix all the javascript
> styling issues that we have in .js and .html files.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (AIRFLOW-2812) Fix error in Updating.md for upgrading to 1.10
[
https://issues.apache.org/jira/browse/AIRFLOW-2812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Hughes updated AIRFLOW-2812:
-
Affects Version/s: 1.10
> Fix error in Updating.md for upgrading to 1.10
> --
>
> Key: AIRFLOW-2812
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2812
> Project: Apache Airflow
> Issue Type: Bug
>Affects Versions: 1.10
>Reporter: Nick Hughes
>Assignee: Nick Hughes
>Priority: Minor
>
> Error in Updating.md directions under Logging Configuration having too many
> brackets which results in errors:
> * {{log_filename_template}}
> * {{log_processor_filename_template}}{{}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (AIRFLOW-2812) Fix error in Updating.md for upgrading to 1.10
Nick Hughes created AIRFLOW-2812:
Summary: Fix error in Updating.md for upgrading to 1.10
Key: AIRFLOW-2812
URL: https://issues.apache.org/jira/browse/AIRFLOW-2812
Project: Apache Airflow
Issue Type: Bug
Reporter: Nick Hughes
Assignee: Nick Hughes
Error in Updating.md directions under Logging Configuration having too many
brackets which results in errors:
* {{log_filename_template}}
* {{log_processor_filename_template}}{{}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (AIRFLOW-2811) Fix scheduler_ops_metrics.py to work
Kengo Seki created AIRFLOW-2811:
---
Summary: Fix scheduler_ops_metrics.py to work
Key: AIRFLOW-2811
URL: https://issues.apache.org/jira/browse/AIRFLOW-2811
Project: Apache Airflow
Issue Type: Bug
Reporter: Kengo Seki
Assignee: Kengo Seki
I tried to run {{scripts/perf/scheduler_ops_metrics.py}} but it failed with the
following error:
{code}
$ python scripts/perf/scheduler_ops_metrics.py
(snip)
Traceback (most recent call last):
File "scripts/perf/scheduler_ops_metrics.py", line 192, in
main()
File "scripts/perf/scheduler_ops_metrics.py", line 188, in main
job.run()
File "/home/sekikn/dev/incubator-airflow/airflow/jobs.py", line 202, in run
self._execute()
File "/home/sekikn/dev/incubator-airflow/airflow/jobs.py", line 1584, in
_execute
self._execute_helper(processor_manager)
File "/home/sekikn/dev/incubator-airflow/airflow/jobs.py", line 1714, in
_execute_helper
self.heartbeat()
File "scripts/perf/scheduler_ops_metrics.py", line 121, in heartbeat
for dag in dags for task in dag.tasks])
TypeError: can't subtract offset-naive and offset-aware datetimes
{code}
Also, it'd be nice if {{MAX_RUNTIME_SECS}} were configurable, since the default
value (6 seconds) is too short for all TaskInstances to finish in my
environment.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (AIRFLOW-2810) Typo in Xcom model timestamp field
Andy Wilcox created AIRFLOW-2810: Summary: Typo in Xcom model timestamp field Key: AIRFLOW-2810 URL: https://issues.apache.org/jira/browse/AIRFLOW-2810 Project: Apache Airflow Issue Type: Bug Components: models Affects Versions: 1.10 Reporter: Andy Wilcox Assignee: Andy Wilcox Looks like a find/replace error. Should be type UtcDateTime, is legacy DateTime -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-2809) Fix security issue regarding Flask SECRET_KEY
[ https://issues.apache.org/jira/browse/AIRFLOW-2809?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaodong DENG updated AIRFLOW-2809: --- Summary: Fix security issue regarding Flask SECRET_KEY (was: Security Issue Regarding Flask SECRET_KEY) > Fix security issue regarding Flask SECRET_KEY > - > > Key: AIRFLOW-2809 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2809 > Project: Apache Airflow > Issue Type: Bug > Components: webserver >Reporter: Xiaodong DENG >Assignee: Xiaodong DENG >Priority: Major > > h2. Background > Currently there is a configuration item *secret_key* in the configuration > .cfg file, with a default value "temporary_key". > h2. Issue > Most admins would ignore it and just use the default value "temporary_key". > However, this may be very dangerous. User may modify the cookie if they try > the default SECRET_KEY while the admin didn't change it. > In Flask documentation, it's suggested to have a SECRET_KEY which is as > random as possible ([http://flask.pocoo.org/docs/1.0/quickstart/] ). > h2. My Proposal > If Admin explicitly specified the SECRET_KEY in *.cfg* file, we use this > SECRET_KEY given by Admin. > If the default SECRET_KEY is not changed in *.cfg* file, randomly generate > SECRET_KEY. Meanwhile, print INFO to remind that a randomly generated > SECRET_KEY is used. > This solution will not affect user experience at all. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
incubator-airflow git commit: [AIRFLOW-XXX] Document dag_dir_list_interval in cfg
Repository: incubator-airflow Updated Branches: refs/heads/master 25f31c9b7 -> 233056e0d [AIRFLOW-XXX] Document dag_dir_list_interval in cfg Document dag_dir_list_interval in default_airflow.cfg https://github.com/apache/incubator-airflow/blob/m aster/UPDATING.md#dag_dir_list_interval Closes #3645 from elgalu/patch-2 Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/233056e0 Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/233056e0 Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/233056e0 Branch: refs/heads/master Commit: 233056e0ddcaf3033e95881d5c7a26fc35fa408f Parents: 25f31c9 Author: Leo Gallucci Authored: Fri Jul 27 11:02:38 2018 +0200 Committer: Fokko Driesprong Committed: Fri Jul 27 11:02:43 2018 +0200 -- airflow/config_templates/default_airflow.cfg | 1 + 1 file changed, 1 insertion(+) -- http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/233056e0/airflow/config_templates/default_airflow.cfg -- diff --git a/airflow/config_templates/default_airflow.cfg b/airflow/config_templates/default_airflow.cfg index 592c9d7..29a6e6b 100644 --- a/airflow/config_templates/default_airflow.cfg +++ b/airflow/config_templates/default_airflow.cfg @@ -434,6 +434,7 @@ min_file_process_interval = 0 # How many seconds to wait between file-parsing loops to prevent the logs from being spammed. min_file_parsing_loop_time = 1 +# How often (in seconds) to scan the DAGs directory for new files. Default to 5 minutes. dag_dir_list_interval = 300 # How often should stats be printed to the logs
[jira] [Commented] (AIRFLOW-2703) Scheduler crashes if Mysql Connectivity is lost
[ https://issues.apache.org/jira/browse/AIRFLOW-2703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16559422#comment-16559422 ] Mishika Singh commented on AIRFLOW-2703: raised PR for this: [https://github.com/apache/incubator-airflow/pull/3650] > Scheduler crashes if Mysql Connectivity is lost > --- > > Key: AIRFLOW-2703 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2703 > Project: Apache Airflow > Issue Type: Bug > Components: scheduler >Affects Versions: Airflow 2.0, 1.9.0 >Reporter: raman >Priority: Major > > Airflow scheduler crashes if connectivity to Mysql is lost. > Below is the stack Trace > Traceback (most recent call last): File > "/usr/src/venv/local/lib/python2.7/site-packages/airflow/jobs.py", line 371, > in helper pickle_dags) File > "/usr/src/venv/local/lib/python2.7/site-packages/airflow/utils/db.py", line > 50, in wrapper result = func(*args, **kwargs) File > "/usr/src/venv/local/lib/python2.7/site-packages/airflow/jobs.py", line 1762, > in process_file dag.sync_to_db() File > "/usr/src/venv/local/lib/python2.7/site-packages/airflow/utils/db.py", line > 50, in wrapper result = func(*args, **kwargs) File > "/usr/src/venv/local/lib/python2.7/site-packages/airflow/models.py", line > 3816, in sync_to_db session.commit() File > "/usr/src/venv/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", > line 943, in commit self.transaction.commit() File > "/usr/src/venv/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", > line 471, in commit t[1].commit() File > "/usr/src/venv/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", > line 1643, in commit self._do_commit() File > "/usr/src/venv/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", > line 1674, in _do_commit self.connection._commit_impl() File > "/usr/src/venv/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", > line 726, in _commit_impl self._handle_dbapi_exception(e, None, None, None, > None) File > "/usr/src/venv/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", > line 1413, in _handle_dbapi_exception exc_info File > "/usr/src/venv/local/lib/python2.7/site-packages/sqlalchemy/util/compat.py", > line 203, in raise_from_cause reraise(type(exception), exception, tb=exc_tb, > cause=cause) File > "/usr/src/venv/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", > line 724, in _commit_impl self.engine.dialect.do_commit(self.connection) File > "/usr/src/venv/local/lib/python2.7/site-packages/sqlalchemy/dialects/mysql/base.py", > line 1784, in do_commit dbapi_connection.commit() OperationalError: > (_mysql_exceptions.OperationalError) (2013, 'Lost connection to MySQL server > during query') (Background on this error at: http://sqlalche.me/e/e3q8) > Process DagFileProcessor141318-Process: -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2803) Fix all ESLint issues
[ https://issues.apache.org/jira/browse/AIRFLOW-2803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16559418#comment-16559418 ] Verdan Mahmood commented on AIRFLOW-2803: - [~tedmiston] I believe you'd be able to resolve most of them by running `npm run lint:fix` command. > Fix all ESLint issues > - > > Key: AIRFLOW-2803 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2803 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Verdan Mahmood >Assignee: Taylor Edmiston >Priority: Major > > Most of the JS code in Apache Airflow has linting issues which are > highlighted after the integration of ESLint. > Once AIRFLOW-2783 merged in master branch, please fix all the javascript > styling issues that we have in .js and .html files. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-2808) Plugin duplication checking is not working
[
https://issues.apache.org/jira/browse/AIRFLOW-2808?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaodong DENG updated AIRFLOW-2808:
---
Component/s: plugins
> Plugin duplication checking is not working
> --
>
> Key: AIRFLOW-2808
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2808
> Project: Apache Airflow
> Issue Type: Bug
> Components: plugins
>Reporter: Xiaodong DENG
>Assignee: Xiaodong DENG
>Priority: Major
>
> h2. *Background*
> A plugin duplication checking was designed in *plugins_manager.py*
> [https://github.com/apache/incubator-airflow/blob/master/airflow/plugins_manager.py#L93]
> .
> Corresponding commit was
> [https://github.com/apache/incubator-airflow/commit/3f38dec9bf44717a275412d1fe155e8252e45ee5|https://github.com/apache/incubator-airflow/commit/3f38dec9bf44717a275412d1fe155e8252e45ee5.]
>
> However, it turns out that this checking is not really working (reason:
> plugin method object name is formed using plugin file path + plugin file name
> + Plugin Class name. It will never be duplicated given there will not be two
> files with the same name in the same directory).
> h2. *Issue*
> In my production environment, there are two plugin files with the same name
> and operator names in the new _AirflowPlugin_ classes defined inside.
> However, they passed the check without any warning or exception.
> For example, I have a plugin *file_sensor_1.py* as below,
> {code:java}
> from airflow.plugins_manager import AirflowPlugin
> from airflow.operators.sensors import BaseSensorOperator
> from airflow.utils.decorators import apply_defaults
> import os
> class local_file_sensor(BaseSensorOperator):
> @apply_defaults
> def __init__(self, file_path, *args, **kwargs):
> super(local_file_sensor, self).__init__(*args, **kwargs)
> self.file_path = file_path
> def poke(self, context):
> self.log.info('A-Poking: %s', self.file_path)
> return os.path.exists(self.file_path)
> class AirflowLocalFileSensorPlugin(AirflowPlugin):
> name = "local_file_sensor_plugin"
> operators = [local_file_sensor]
> {code}
>
> I copy & paste it into another plugin file *file_sensor_2.py*, and make the
> only change to change the log info from "_A-Poking_" to "_B-Poking_" (to help
> me check which one is picked).
> Only one plugin would be loaded eventually (because the earlier loaded one
> will be overwritten by the later loaded one
> [https://github.com/apache/incubator-airflow/blob/master/airflow/plugins_manager.py#L101]
> ). However, which one? We don't know. It's indeterminate. So far the file
> name seems to be the only factor affecting which one would be picked by
> Airflow.
> h2. *My proposal*
> Give WARNING to the users when they launch the Airflow. (Or should we give
> error msg and fail the launching?)
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (AIRFLOW-2809) Security Issue Regarding Flask SECRET_KEY
[ https://issues.apache.org/jira/browse/AIRFLOW-2809?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaodong DENG updated AIRFLOW-2809: --- Issue Type: Bug (was: Improvement) > Security Issue Regarding Flask SECRET_KEY > - > > Key: AIRFLOW-2809 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2809 > Project: Apache Airflow > Issue Type: Bug > Components: webserver >Reporter: Xiaodong DENG >Assignee: Xiaodong DENG >Priority: Major > > h2. Background > Currently there is a configuration item *secret_key* in the configuration > .cfg file, with a default value "temporary_key". > h2. Issue > Most admins would ignore it and just use the default value "temporary_key". > However, this may be very dangerous. User may modify the cookie if they try > the default SECRET_KEY while the admin didn't change it. > In Flask documentation, it's suggested to have a SECRET_KEY which is as > random as possible ([http://flask.pocoo.org/docs/1.0/quickstart/] ). > h2. My Proposal > If Admin explicitly specified the SECRET_KEY in *.cfg* file, we use this > SECRET_KEY given by Admin. > If the default SECRET_KEY is not changed in *.cfg* file, randomly generate > SECRET_KEY. Meanwhile, print INFO to remind that a randomly generated > SECRET_KEY is used. > This solution will not affect user experience at all. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-2809) Security Issue Regarding Flask SECRET_KEY
[ https://issues.apache.org/jira/browse/AIRFLOW-2809?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaodong DENG updated AIRFLOW-2809: --- External issue URL: https://github.com/apache/incubator-airflow/pull/3651 Description: h2. Background Currently there is a configuration item *secret_key* in the configuration .cfg file, with a default value "temporary_key". h2. Issue Most admins would ignore it and just use the default value "temporary_key". However, this may be very dangerous. User may modify the cookie if they try the default SECRET_KEY while the admin didn't change it. In Flask documentation, it's suggested to have a SECRET_KEY which is as random as possible ([http://flask.pocoo.org/docs/1.0/quickstart/] ). h2. My Proposal If Admin explicitly specified the SECRET_KEY in *.cfg* file, we use this SECRET_KEY given by Admin. If the default SECRET_KEY is not changed in *.cfg* file, randomly generate SECRET_KEY. Meanwhile, print INFO to remind that a randomly generated SECRET_KEY is used. This solution will not affect user experience at all. was: h2. Background Currently there is a configuration item *secret_key* in the configuration .cfg file, with a default value "temporary_key". h2. Issue Most users would ignore it and just use the default value "temporary_key". However, this may be very dangerous. User may modify the cookie if they try the default SECRET_KEY while the admin didn't change it. In Flask documentation, it's suggested to have a SECRET_KEY which is as random as possible ([http://flask.pocoo.org/docs/1.0/quickstart/] ). h2. My Proposal If Admin explicitly specified the SECRET_KEY in *.cfg* file, we use this SECRET_KEY given by Admin. If the default SECRET_KEY is not changed in *.cfg* file, randomly generate SECRET_KEY. Meanwhile, print INFO to remind that a randomly generated SECRET_KEY is used. This solution will not affect user experience at all. > Security Issue Regarding Flask SECRET_KEY > - > > Key: AIRFLOW-2809 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2809 > Project: Apache Airflow > Issue Type: Improvement > Components: webserver >Reporter: Xiaodong DENG >Assignee: Xiaodong DENG >Priority: Major > > h2. Background > Currently there is a configuration item *secret_key* in the configuration > .cfg file, with a default value "temporary_key". > h2. Issue > Most admins would ignore it and just use the default value "temporary_key". > However, this may be very dangerous. User may modify the cookie if they try > the default SECRET_KEY while the admin didn't change it. > In Flask documentation, it's suggested to have a SECRET_KEY which is as > random as possible ([http://flask.pocoo.org/docs/1.0/quickstart/] ). > h2. My Proposal > If Admin explicitly specified the SECRET_KEY in *.cfg* file, we use this > SECRET_KEY given by Admin. > If the default SECRET_KEY is not changed in *.cfg* file, randomly generate > SECRET_KEY. Meanwhile, print INFO to remind that a randomly generated > SECRET_KEY is used. > This solution will not affect user experience at all. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-2809) Security Issue Regarding Flask SECRET_KEY
Xiaodong DENG created AIRFLOW-2809: -- Summary: Security Issue Regarding Flask SECRET_KEY Key: AIRFLOW-2809 URL: https://issues.apache.org/jira/browse/AIRFLOW-2809 Project: Apache Airflow Issue Type: Improvement Components: webserver Reporter: Xiaodong DENG Assignee: Xiaodong DENG h2. Background Currently there is a configuration item *secret_key* in the configuration .cfg file, with a default value "temporary_key". h2. Issue Most users would ignore it and just use the default value "temporary_key". However, this may be very dangerous. User may modify the cookie if they try the default SECRET_KEY while the admin didn't change it. In Flask documentation, it's suggested to have a SECRET_KEY which is as random as possible ([http://flask.pocoo.org/docs/1.0/quickstart/] ). h2. My Proposal If Admin explicitly specified the SECRET_KEY in *.cfg* file, we use this SECRET_KEY given by Admin. If the default SECRET_KEY is not changed in *.cfg* file, randomly generate SECRET_KEY. Meanwhile, print INFO to remind that a randomly generated SECRET_KEY is used. This solution will not affect user experience at all. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
