Brecht De Vlieger created AIRFLOW-3397:
------------------------------------------
Summary: Adding a new role to rbac makes airflow crash
Key: AIRFLOW-3397
URL: https://issues.apache.org/jira/browse/AIRFLOW-3397
Project: Apache Airflow
Issue Type: Bug
Components: security
Affects Versions: 2.0.0
Reporter: Brecht De Vlieger
Here are the steps to reproduce:
- Start airflow with rbac enabled
- Create a new role
- Wait until one of the gunicorn workers is restarted
- Airflow crashes with the following exception:
```
[2018-11-26 09:59:17,353] \{__init__.py:51} INFO - Using executor
KubernetesExecutor
[2018-11-26 09:59:17,617] \{models.py:286} INFO - Filling up the DagBag from
/root/airflow/dags
[2018-11-26 09:59:18,134] \{security.py:435} INFO - Start syncing user roles.
[2018-11-26 09:59:18,270] \{security.py:194} INFO - Existing permissions for
the role:Viewer within the database will persist.
/usr/local/lib/python2.7/dist-packages/flask_appbuilder/fields.py:152:
UserWarning: allow_blank=True does not do anything for QuerySelectMultipleField.
warnings.warn('allow_blank=True does not do anything for
QuerySelectMultipleField.')
[2018-11-26 09:59:18,384] \{security.py:194} INFO - Existing permissions for
the role:User within the database will persist.
[2018-11-26 09:59:18,491] \{security.py:194} INFO - Existing permissions for
the role:Op within the database will persist.
[2018-11-26 09:59:18,491] \{security.py:344} INFO - Fetching a set of all
permission, view_menu from FAB meta-table
172.17.0.1 - - [26/Nov/2018:09:59:18 +0000] "GET /roles/list/ HTTP/1.1" 200
48607 "http://localhost/home"; "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:63.0)
Gecko/20100101 Firefox/63.0"
[2018-11-26 09:59:18 +0000] [37] [ERROR] Exception in worker process
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/gunicorn/arbiter.py", line 583,
in spawn_worker
worker.init_process()
File "/usr/local/lib/python2.7/dist-packages/gunicorn/workers/base.py", line
129, in init_process
self.load_wsgi()
File "/usr/local/lib/python2.7/dist-packages/gunicorn/workers/base.py", line
138, in load_wsgi
self.wsgi = self.app.wsgi()
File "/usr/local/lib/python2.7/dist-packages/gunicorn/app/base.py", line 67,
in wsgi
self.callable = self.load()
File "/usr/local/lib/python2.7/dist-packages/gunicorn/app/wsgiapp.py", line
52, in load
return self.load_wsgiapp()
File "/usr/local/lib/python2.7/dist-packages/gunicorn/app/wsgiapp.py", line
41, in load_wsgiapp
return util.import_app(self.app_uri)
File "/usr/local/lib/python2.7/dist-packages/gunicorn/util.py", line 362, in
import_app
app = eval(obj, vars(mod))
File "<string>", line 1, in <module>
File "/usr/local/lib/python2.7/dist-packages/airflow/www_rbac/app.py", line
207, in cached_app
app, _ = create_app(config, session, testing)
File "/usr/local/lib/python2.7/dist-packages/airflow/www_rbac/app.py", line
167, in create_app
security_manager.sync_roles()
File "/usr/local/lib/python2.7/dist-packages/airflow/www_rbac/security.py",
line 443, in sync_roles
self.create_custom_dag_permission_view()
File "/usr/local/lib/python2.7/dist-packages/airflow/www_rbac/security.py",
line 406, in create_custom_dag_permission_view
self.get_session.execute(ab_perm_view_role.insert(), update_perm_views)
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/scoping.py", line
157, in do
return getattr(self.registry(), name)(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line
1160, in execute
bind, close_with_result=True).execute(clause, params or {})
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line
945, in execute
return meth(self, multiparams, params)
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/sql/elements.py",
line 263, in _execute_on_connection
return connection._execute_clauseelement(self, multiparams, params)
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line
1053, in _execute_clauseelement
compiled_sql, distilled_params
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line
1189, in _execute_context
context)
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line
1402, in _handle_dbapi_exception
exc_info
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/util/compat.py", line
203, in raise_from_cause
reraise(type(exception), exception, tb=exc_tb, cause=cause)
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line
1159, in _execute_context
context)
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/default.py",
line 467, in do_executemany
cursor.executemany(statement, parameters)
IntegrityError: (psycopg2.IntegrityError) insert or update on table
"ab_permission_view_role" violates foreign key constraint
"ab_permission_view_role_role_id_fkey"
DETAIL: Key (role_id)=(347) is not present in table "ab_role".
[SQL: "INSERT INTO ab_permission_view_role (id, permission_view_id, role_id)
VALUES (nextval('ab_permission_view_role_id_seq'), %(permission_view_id)s,
%(role_id)s)"] [parameters: (\{'permission_view_id': 36, 'role_id': 347},
\{'permission_view_id': 37, 'role_id': 347}, \{'permission_view_id': 38,
'role_id': 347}, \{'permission_view_id': 39, 'role_id': 347},
\{'permission_view_id': 40, 'role_id': 347}, \{'permission_view_id': 41,
'role_id': 347}, \{'permission_view_id': 42, 'role_id': 347},
\{'permission_view_id': 43, 'role_id': 347} ... displaying 10 of 54 total
bound parameter sets ... \{'permission_view_id': 122, 'role_id': 347},
\{'permission_view_id': 123, 'role_id': 347})]
[2018-11-26 09:59:18 +0000] [37] [INFO] Worker exiting (pid: 37)
[2018-11-26 09:59:19 +0000] [33] [INFO] Worker exiting (pid: 33)
[2018-11-26 09:59:19 +0000] [15] [INFO] Worker exiting (pid: 15)
[2018-11-26 09:59:19 +0000] [16] [INFO] Worker exiting (pid: 16)
[2018-11-26 09:59:19 +0000] [29] [INFO] Worker exiting (pid: 29)
[2018-11-26 09:59:19 +0000] [8] [INFO] Shutting down: Master
[2018-11-26 09:59:19 +0000] [8] [INFO] Reason: Worker failed to boot.
[2018-11-26 09:59:19,963] \{cli.py:818} ERROR - [0 / 0] some workers seem to
have died and gunicorndid not restart them as expected
```
Solution:
This is caused because of name shadowing in a for loop.
The following diff solves the problem:
```
diff --git a/airflow/www_rbac/security.py b/airflow/www_rbac/security.py
index 8f9b6287..369ec71c 100644
--- a/airflow/www_rbac/security.py
+++ b/airflow/www_rbac/security.py
@@ -395,8 +395,8 @@ class AirflowSecurityManager(SecurityManager):
existing_perm_view_by_user =
self.get_session.query(ab_perm_view_role)\
.filter(ab_perm_view_role.columns.role_id == role.id)
- existing_perms_views = set([role.permission_view_id
- for role in
existing_perm_view_by_user])
+ existing_perms_views = set([pv.permission_view_id
+ for pv in existing_perm_view_by_user])
missing_perm_views = all_perm_views - existing_perms_views
for perm_view_id in missing_perm_views:
```
I have no time atm to create a PR, but I can do it later if required.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
