betodealmeida opened a new pull request #15435: URL: https://github.com/apache/superset/pull/15435
### SUMMARY <!--- Describe the change below, including rationale and design decisions --> https://github.com/apache/superset/pull/15405/ builds the configuration for Superset using `pydash.objects.merge` to combine `superset_config.py` into `superset/config.py`. This fails when a `RESULTS_BACKEND` is configured, since `pydash` tries to deepcopy all the objects, but a `cachelib.redis.RedisCache` (a valid backend) can't be deepcopied because it has a lock: ```python Traceback (most recent call last): File "/Users/beto/.pyenv/versions/superset/bin/superset", line 33, in <module> sys.exit(load_entry_point('apache-superset', 'console_scripts', 'superset')()) File "/Users/beto/.pyenv/versions/3.8.8/envs/superset/lib/python3.8/site-packages/click/core.py", line 829, in __call__ return self.main(*args, **kwargs) File "/Users/beto/.pyenv/versions/3.8.8/envs/superset/lib/python3.8/site-packages/flask/cli.py", line 586, in main return super(FlaskGroup, self).main(*args, **kwargs) File "/Users/beto/.pyenv/versions/3.8.8/envs/superset/lib/python3.8/site-packages/click/core.py", line 782, in main rv = self.invoke(ctx) File "/Users/beto/.pyenv/versions/3.8.8/envs/superset/lib/python3.8/site-packages/click/core.py", line 1256, in invoke Command.invoke(self, ctx) File "/Users/beto/.pyenv/versions/3.8.8/envs/superset/lib/python3.8/site-packages/click/core.py", line 1066, in invoke return ctx.invoke(self.callback, **ctx.params) File "/Users/beto/.pyenv/versions/3.8.8/envs/superset/lib/python3.8/site-packages/click/core.py", line 610, in invoke return callback(*args, **kwargs) File "/Users/beto/.pyenv/versions/3.8.8/envs/superset/lib/python3.8/site-packages/click/decorators.py", line 21, in new_func return f(get_current_context(), *args, **kwargs) File "/Users/beto/.pyenv/versions/3.8.8/envs/superset/lib/python3.8/site-packages/flask/cli.py", line 425, in decorator with __ctx.ensure_object(ScriptInfo).load_app().app_context(): File "/Users/beto/.pyenv/versions/3.8.8/envs/superset/lib/python3.8/site-packages/flask/cli.py", line 381, in load_app app = call_factory(self, self.create_app) File "/Users/beto/.pyenv/versions/3.8.8/envs/superset/lib/python3.8/site-packages/flask/cli.py", line 119, in call_factory return app_factory() File "/Users/beto/Projects/incubator-superset/superset/app.py", line 52, in create_app raise ex File "/Users/beto/Projects/incubator-superset/superset/app.py", line 41, in create_app config = init_config() File "/Users/beto/Projects/incubator-superset/superset/app.py", line 62, in init_config return merge(config, override_conf) File "/Users/beto/.pyenv/versions/3.8.8/envs/superset/lib/python3.8/site-packages/pydash-5.0.1-py3.8.egg/pydash/objects.py", line 859, in merge return merge_with(obj, *sources) File "/Users/beto/.pyenv/versions/3.8.8/envs/superset/lib/python3.8/site-packages/pydash-5.0.1-py3.8.egg/pydash/objects.py", line 907, in merge_with sources = [copy.deepcopy(source) for source in sources] File "/Users/beto/.pyenv/versions/3.8.8/envs/superset/lib/python3.8/site-packages/pydash-5.0.1-py3.8.egg/pydash/objects.py", line 907, in <listcomp> sources = [copy.deepcopy(source) for source in sources] File "/Users/beto/.pyenv/versions/3.8.8/lib/python3.8/copy.py", line 146, in deepcopy y = copier(x, memo) File "/Users/beto/.pyenv/versions/3.8.8/lib/python3.8/copy.py", line 230, in _deepcopy_dict y[deepcopy(key, memo)] = deepcopy(value, memo) File "/Users/beto/.pyenv/versions/3.8.8/lib/python3.8/copy.py", line 172, in deepcopy y = _reconstruct(x, memo, *rv) File "/Users/beto/.pyenv/versions/3.8.8/lib/python3.8/copy.py", line 270, in _reconstruct state = deepcopy(state, memo) File "/Users/beto/.pyenv/versions/3.8.8/lib/python3.8/copy.py", line 146, in deepcopy y = copier(x, memo) File "/Users/beto/.pyenv/versions/3.8.8/lib/python3.8/copy.py", line 230, in _deepcopy_dict y[deepcopy(key, memo)] = deepcopy(value, memo) File "/Users/beto/.pyenv/versions/3.8.8/lib/python3.8/copy.py", line 172, in deepcopy y = _reconstruct(x, memo, *rv) File "/Users/beto/.pyenv/versions/3.8.8/lib/python3.8/copy.py", line 270, in _reconstruct state = deepcopy(state, memo) File "/Users/beto/.pyenv/versions/3.8.8/lib/python3.8/copy.py", line 146, in deepcopy y = copier(x, memo) File "/Users/beto/.pyenv/versions/3.8.8/lib/python3.8/copy.py", line 230, in _deepcopy_dict y[deepcopy(key, memo)] = deepcopy(value, memo) File "/Users/beto/.pyenv/versions/3.8.8/lib/python3.8/copy.py", line 172, in deepcopy y = _reconstruct(x, memo, *rv) File "/Users/beto/.pyenv/versions/3.8.8/lib/python3.8/copy.py", line 270, in _reconstruct state = deepcopy(state, memo) File "/Users/beto/.pyenv/versions/3.8.8/lib/python3.8/copy.py", line 146, in deepcopy y = copier(x, memo) File "/Users/beto/.pyenv/versions/3.8.8/lib/python3.8/copy.py", line 230, in _deepcopy_dict y[deepcopy(key, memo)] = deepcopy(value, memo) File "/Users/beto/.pyenv/versions/3.8.8/lib/python3.8/copy.py", line 161, in deepcopy rv = reductor(4) TypeError: cannot pickle '_thread.lock' object Exception ignored in: <function Redis.__del__ at 0x1109a7790> Traceback (most recent call last): File "/Users/beto/.pyenv/versions/3.8.8/envs/superset/lib/python3.8/site-packages/redis/client.py", line 885, in __del__ self.close() File "/Users/beto/.pyenv/versions/3.8.8/envs/superset/lib/python3.8/site-packages/redis/client.py", line 888, in close conn = self.connection AttributeError: 'Redis' object has no attribute 'connection' ``` Additionally, the merge strategy is not what we want to do. For example, we can have languages configured in superset/config.py: ```python LANGUAGES = { "en": {"flag": "us", "name": "English"}, "es": {"flag": "es", "name": "Spanish"}, "it": {"flag": "it", "name": "Italian"}, "fr": {"flag": "fr", "name": "French"}, "zh": {"flag": "cn", "name": "Chinese"}, "ja": {"flag": "jp", "name": "Japanese"}, "de": {"flag": "de", "name": "German"}, "pt": {"flag": "pt", "name": "Portuguese"}, "pt_BR": {"flag": "br", "name": "Brazilian Portuguese"}, "ru": {"flag": "ru", "name": "Russian"}, "ko": {"flag": "kr", "name": "Korean"}, "sl": {"flag": "si", "name": "Slovenian"}, } ``` If a user wants to have only Brazilian Portuguese in their Superset instance they would add to their config: ```python LANGUAGES = { "pt_BR": {"flag": "br", "name": "Brazilian Portuguese"}, } ``` But the `merge` would bring back all languages, resulting in a config that looks like this: ```python LANGUAGES = { "en": {"flag": "us", "name": "English"}, "es": {"flag": "es", "name": "Spanish"}, "it": {"flag": "it", "name": "Italian"}, "fr": {"flag": "fr", "name": "French"}, "zh": {"flag": "cn", "name": "Chinese"}, "ja": {"flag": "jp", "name": "Japanese"}, "de": {"flag": "de", "name": "German"}, "pt": {"flag": "pt", "name": "Portuguese"}, "pt_BR": {"flag": "br", "name": "Brazilian Portuguese"}, "ru": {"flag": "ru", "name": "Russian"}, "ko": {"flag": "kr", "name": "Korean"}, "sl": {"flag": "si", "name": "Slovenian"}, } ``` ### BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF <!--- Skip this if not applicable --> ### TESTING INSTRUCTIONS <!--- Required! What steps can be taken to manually verify the changes? --> ### ADDITIONAL INFORMATION <!--- Check any relevant boxes with "x" --> <!--- HINT: Include "Fixes #nnn" if you are fixing an existing issue --> - [ ] Has associated issue: - [ ] Changes UI - [ ] Includes DB Migration (follow approval process in [SIP-59](https://github.com/apache/superset/issues/13351)) - [ ] Migration is atomic, supports rollback & is backwards-compatible - [ ] Confirm DB migration upgrade and downgrade tested - [ ] Runtime estimates and downtime expectations provided - [ ] Introduces new feature or API - [ ] Removes existing feature or API -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
