[
https://issues.apache.org/jira/browse/AIRFLOW-3397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kaxil Naik resolved AIRFLOW-3397.
---------------------------------
Resolution: Fixed
Fix Version/s: 1.10.2
> Adding a new role to rbac makes airflow crash
> ---------------------------------------------
>
> Key: AIRFLOW-3397
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3397
> Project: Apache Airflow
> Issue Type: Bug
> Components: security
> Affects Versions: 2.0.0
> Reporter: Brecht De Vlieger
> Priority: Major
> Fix For: 1.10.2
>
>
> Here are the steps to reproduce:
> - Start airflow with rbac enabled
> - Create a new role
> - Wait until one of the gunicorn workers is restarted
> - Airflow crashes with the following exception:
> ```
> [2018-11-26 09:59:17,353] \{__init__.py:51} INFO - Using executor
> KubernetesExecutor
> [2018-11-26 09:59:17,617] \{models.py:286} INFO - Filling up the DagBag from
> /root/airflow/dags
> [2018-11-26 09:59:18,134] \{security.py:435} INFO - Start syncing user roles.
> [2018-11-26 09:59:18,270] \{security.py:194} INFO - Existing permissions for
> the role:Viewer within the database will persist.
> /usr/local/lib/python2.7/dist-packages/flask_appbuilder/fields.py:152:
> UserWarning: allow_blank=True does not do anything for
> QuerySelectMultipleField.
> warnings.warn('allow_blank=True does not do anything for
> QuerySelectMultipleField.')
> [2018-11-26 09:59:18,384] \{security.py:194} INFO - Existing permissions for
> the role:User within the database will persist.
> [2018-11-26 09:59:18,491] \{security.py:194} INFO - Existing permissions for
> the role:Op within the database will persist.
> [2018-11-26 09:59:18,491] \{security.py:344} INFO - Fetching a set of all
> permission, view_menu from FAB meta-table
> 172.17.0.1 - - [26/Nov/2018:09:59:18 +0000] "GET /roles/list/ HTTP/1.1" 200
> 48607 "http://localhost/home"; "Mozilla/5.0 (X11; Ubuntu; Linux x86_64;
> rv:63.0) Gecko/20100101 Firefox/63.0"
> [2018-11-26 09:59:18 +0000] [37] [ERROR] Exception in worker process
> Traceback (most recent call last):
> File "/usr/local/lib/python2.7/dist-packages/gunicorn/arbiter.py", line
> 583, in spawn_worker
> worker.init_process()
> File "/usr/local/lib/python2.7/dist-packages/gunicorn/workers/base.py",
> line 129, in init_process
> self.load_wsgi()
> File "/usr/local/lib/python2.7/dist-packages/gunicorn/workers/base.py",
> line 138, in load_wsgi
> self.wsgi = self.app.wsgi()
> File "/usr/local/lib/python2.7/dist-packages/gunicorn/app/base.py", line
> 67, in wsgi
> self.callable = self.load()
> File "/usr/local/lib/python2.7/dist-packages/gunicorn/app/wsgiapp.py", line
> 52, in load
> return self.load_wsgiapp()
> File "/usr/local/lib/python2.7/dist-packages/gunicorn/app/wsgiapp.py", line
> 41, in load_wsgiapp
> return util.import_app(self.app_uri)
> File "/usr/local/lib/python2.7/dist-packages/gunicorn/util.py", line 362,
> in import_app
> app = eval(obj, vars(mod))
> File "<string>", line 1, in <module>
> File "/usr/local/lib/python2.7/dist-packages/airflow/www_rbac/app.py", line
> 207, in cached_app
> app, _ = create_app(config, session, testing)
> File "/usr/local/lib/python2.7/dist-packages/airflow/www_rbac/app.py", line
> 167, in create_app
> security_manager.sync_roles()
> File "/usr/local/lib/python2.7/dist-packages/airflow/www_rbac/security.py",
> line 443, in sync_roles
> self.create_custom_dag_permission_view()
> File "/usr/local/lib/python2.7/dist-packages/airflow/www_rbac/security.py",
> line 406, in create_custom_dag_permission_view
> self.get_session.execute(ab_perm_view_role.insert(), update_perm_views)
> File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/scoping.py",
> line 157, in do
> return getattr(self.registry(), name)(*args, **kwargs)
> File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py",
> line 1160, in execute
> bind, close_with_result=True).execute(clause, params or {})
> File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py",
> line 945, in execute
> return meth(self, multiparams, params)
> File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/sql/elements.py",
> line 263, in _execute_on_connection
> return connection._execute_clauseelement(self, multiparams, params)
> File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py",
> line 1053, in _execute_clauseelement
> compiled_sql, distilled_params
> File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py",
> line 1189, in _execute_context
> context)
> File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py",
> line 1402, in _handle_dbapi_exception
> exc_info
> File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/util/compat.py",
> line 203, in raise_from_cause
> reraise(type(exception), exception, tb=exc_tb, cause=cause)
> File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py",
> line 1159, in _execute_context
> context)
> File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/default.py",
> line 467, in do_executemany
> cursor.executemany(statement, parameters)
> IntegrityError: (psycopg2.IntegrityError) insert or update on table
> "ab_permission_view_role" violates foreign key constraint
> "ab_permission_view_role_role_id_fkey"
> DETAIL: Key (role_id)=(347) is not present in table "ab_role".
> [SQL: "INSERT INTO ab_permission_view_role (id, permission_view_id, role_id)
> VALUES (nextval('ab_permission_view_role_id_seq'), %(permission_view_id)s,
> %(role_id)s)"] [parameters: (\{'permission_view_id': 36, 'role_id': 347},
> \{'permission_view_id': 37, 'role_id': 347}, \{'permission_view_id': 38,
> 'role_id': 347}, \{'permission_view_id': 39, 'role_id': 347},
> \{'permission_view_id': 40, 'role_id': 347}, \{'permission_view_id': 41,
> 'role_id': 347}, \{'permission_view_id': 42, 'role_id': 347},
> \{'permission_view_id': 43, 'role_id': 347} ... displaying 10 of 54 total
> bound parameter sets ... \{'permission_view_id': 122, 'role_id': 347},
> \{'permission_view_id': 123, 'role_id': 347})]
> [2018-11-26 09:59:18 +0000] [37] [INFO] Worker exiting (pid: 37)
> [2018-11-26 09:59:19 +0000] [33] [INFO] Worker exiting (pid: 33)
> [2018-11-26 09:59:19 +0000] [15] [INFO] Worker exiting (pid: 15)
> [2018-11-26 09:59:19 +0000] [16] [INFO] Worker exiting (pid: 16)
> [2018-11-26 09:59:19 +0000] [29] [INFO] Worker exiting (pid: 29)
> [2018-11-26 09:59:19 +0000] [8] [INFO] Shutting down: Master
> [2018-11-26 09:59:19 +0000] [8] [INFO] Reason: Worker failed to boot.
> [2018-11-26 09:59:19,963] \{cli.py:818} ERROR - [0 / 0] some workers seem to
> have died and gunicorndid not restart them as expected
> ```
> Solution:
> This is caused because of name shadowing in a for loop.
> The following diff solves the problem:
> ```
> diff --git a/airflow/www_rbac/security.py b/airflow/www_rbac/security.py
> index 8f9b6287..369ec71c 100644
> --- a/airflow/www_rbac/security.py
> +++ b/airflow/www_rbac/security.py
> @@ -395,8 +395,8 @@ class AirflowSecurityManager(SecurityManager):
> existing_perm_view_by_user =
> self.get_session.query(ab_perm_view_role)\
> .filter(ab_perm_view_role.columns.role_id == role.id)
>
> - existing_perms_views = set([role.permission_view_id
> - for role in
> existing_perm_view_by_user])
> + existing_perms_views = set([pv.permission_view_id
> + for pv in
> existing_perm_view_by_user])
> missing_perm_views = all_perm_views - existing_perms_views
>
> for perm_view_id in missing_perm_views:
> ```
> I have no time atm to create a PR, but I can do it later if required.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
