EBoisseauSierra opened a new issue #14052: URL: https://github.com/apache/superset/issues/14052
I am trying to follow the documentation to setup my VM and successfully run the test suite, yet I'm facing multiple hurdles along the way. ### Expected results * follow [the instructions](https://github.com/apache/superset/blob/master/CONTRIBUTING.md#setup-local-environment-for-development) only, * run the test suite successfully. ### Actual results I'm facing some issues: 1. one OS dependency is not specified, 2. one needs an admin actually named `admin` to load examples 2. examples can't be all loaded in one go, 3. tests fail #### How to reproduce the bug On a new VM running Fedora 32: 1. OSError: mysql_config not found ```shell # Following https://github.com/apache/superset/blob/master/CONTRIBUTING.md#setup-local-environment-for-development git clone [email protected]:apache/superset.git cd superset git checkout 1.0.1 # to make things reproducible sudo dnf install -y gcc gcc-c++ libffi-devel python-devel python-pip python-wheel openssl-devel cyrus-sasl-devel openldap-devel # as per https://superset.apache.org/docs/installation/installing-superset-from-scratch#os-dependencies python3.8 -m venv venv source venv/bin/activate pip install -r requirements/local.txt ``` This throws an `OSError: mysql_config not found` As per [this SO question](https://stackoverflow.com/q/22949654/5433628), **the solution is to install `mariadb-devel` — what actually solves the issue.** 2. Fail to load the examples ```shell # Following https://github.com/apache/superset/blob/master/CONTRIBUTING.md#setup-local-environment-for-development git clone [email protected]:apache/superset.git cd superset git checkout 1.0.1 # to make things reproducible sudo dnf install -y gcc gcc-c++ libffi-devel python-devel python-pip python-wheel openssl-devel cyrus-sasl-devel openldap-devel # as per https://superset.apache.org/docs/installation/installing-superset-from-scratch#os-dependencies sudo dnf -y install mariadb-devel # FIX as per https://stackoverflow.com/q/22949654/5433628 python3.8 -m venv venv source venv/bin/activate pip install -r requirements/local.txt pip install . superset fab create-admin --username adent --firstname arthur --lastname dent --email [email protected] --password 42 superset db upgrade superset init superset load-examples ``` This returns the following error: ``` INFO:superset.utils.core:Creating database reference for examples Loading examples metadata and related data into examples Traceback (most recent call last): File "/home/fedora/superset/venv/bin/superset", line 11, in <module> load_entry_point('apache-superset==1.0.1', 'console_scripts', 'superset')() File "/home/fedora/superset/venv/lib64/python3.8/site-packages/click/core.py", line 829, in __call__ return self.main(*args, **kwargs) File "/home/fedora/superset/venv/lib64/python3.8/site-packages/flask/cli.py", line 586, in main return super(FlaskGroup, self).main(*args, **kwargs) File "/home/fedora/superset/venv/lib64/python3.8/site-packages/click/core.py", line 782, in main rv = self.invoke(ctx) File "/home/fedora/superset/venv/lib64/python3.8/site-packages/click/core.py", line 1259, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/home/fedora/superset/venv/lib64/python3.8/site-packages/click/core.py", line 1066, in invoke return ctx.invoke(self.callback, **ctx.params) File "/home/fedora/superset/venv/lib64/python3.8/site-packages/click/core.py", line 610, in invoke return callback(*args, **kwargs) File "/home/fedora/superset/venv/lib64/python3.8/site-packages/click/decorators.py", line 21, in new_func return f(get_current_context(), *args, **kwargs) File "/home/fedora/superset/venv/lib64/python3.8/site-packages/flask/cli.py", line 426, in decorator return __ctx.invoke(f, *args, **kwargs) File "/home/fedora/superset/venv/lib64/python3.8/site-packages/click/core.py", line 610, in invoke return callback(*args, **kwargs) File "/home/fedora/superset/venv/lib64/python3.8/site-packages/superset/cli.py", line 187, in load_examples load_examples_run(load_test_data, only_metadata, force) File "/home/fedora/superset/venv/lib64/python3.8/site-packages/superset/cli.py", line 120, in load_examples_run from superset import examples File "/home/fedora/superset/venv/lib64/python3.8/site-packages/superset/examples/__init__.py", line 18, in <module> from .birth_names import load_birth_names File "/home/fedora/superset/venv/lib64/python3.8/site-packages/superset/examples/birth_names.py", line 46, in <module> raise NoDataException( superset.exceptions.NoDataException: Admin user does not exist. Please, check if test users are properly loaded (`superset load_test_users`). ``` We can then explicitely create an admin user: ```shell # Following https://github.com/apache/superset/blob/master/CONTRIBUTING.md#setup-local-environment-for-development git clone [email protected]:apache/superset.git cd superset git checkout 1.0.1 # to make things reproducible sudo dnf install -y gcc gcc-c++ libffi-devel python-devel python-pip python-wheel openssl-devel cyrus-sasl-devel openldap-devel # as per https://superset.apache.org/docs/installation/installing-superset-from-scratch#os-dependencies sudo dnf -y install mariadb-devel # FIX as per https://stackoverflow.com/q/22949654/5433628 python3.8 -m venv venv source venv/bin/activate pip install -r requirements/local.txt pip install . superset fab create-admin --username admin --firstname Ad --lastname Ministrator --email [email protected] --password secret # FIX explicitely create an admin user with username exactly `admin` superset db upgrade superset init superset load-examples ``` But the following error is still returned ``` […] INFO:superset.models.helpers:Updating table_columns gender Loading data inside the import transaction INFO:superset.datasets.commands.importers.v1.utils:Loading data inside the import transaction Traceback (most recent call last): File "/home/fedora/superset/venv/lib64/python3.8/site-packages/sqlalchemy/engine/base.py", line 1276, in _execute_context self.dialect.do_execute( File "/home/fedora/superset/venv/lib64/python3.8/site-packages/sqlalchemy/engine/default.py", line 593, in do_execute cursor.execute(statement, parameters) sqlite3.OperationalError: too many SQL variables The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/home/fedora/superset/venv/lib64/python3.8/site-packages/superset/commands/importers/v1/examples.py", line 67, in run self._import(db.session, self._configs, self.overwrite, self.force_data) File "/home/fedora/superset/venv/lib64/python3.8/site-packages/superset/commands/importers/v1/examples.py", line 99, in _import dataset = import_dataset( File "/home/fedora/superset/venv/lib64/python3.8/site-packages/superset/datasets/commands/importers/v1/utils.py", line 119, in import_dataset load_data(data_uri, dataset, example_database, session) File "/home/fedora/superset/venv/lib64/python3.8/site-packages/superset/datasets/commands/importers/v1/utils.py", line 146, in load_data df.to_sql( File "/home/fedora/superset/venv/lib64/python3.8/site-packages/pandas/core/generic.py", line 2605, in to_sql sql.to_sql( File "/home/fedora/superset/venv/lib64/python3.8/site-packages/pandas/io/sql.py", line 589, in to_sql pandas_sql.to_sql( File "/home/fedora/superset/venv/lib64/python3.8/site-packages/pandas/io/sql.py", line 1406, in to_sql raise err File "/home/fedora/superset/venv/lib64/python3.8/site-packages/pandas/io/sql.py", line 1398, in to_sql table.insert(chunksize, method=method) File "/home/fedora/superset/venv/lib64/python3.8/site-packages/pandas/io/sql.py", line 830, in insert exec_insert(conn, keys, chunk_iter) File "/home/fedora/superset/venv/lib64/python3.8/site-packages/pandas/io/sql.py", line 758, in _execute_insert_multi conn.execute(self.table.insert(data)) File "/home/fedora/superset/venv/lib64/python3.8/site-packages/sqlalchemy/engine/base.py", line 1011, in execute return meth(self, multiparams, params) File "/home/fedora/superset/venv/lib64/python3.8/site-packages/sqlalchemy/sql/elements.py", line 298, in _execute_on_connection return connection._execute_clauseelement(self, multiparams, params) File "/home/fedora/superset/venv/lib64/python3.8/site-packages/sqlalchemy/engine/base.py", line 1124, in _execute_clauseelement ret = self._execute_context( File "/home/fedora/superset/venv/lib64/python3.8/site-packages/sqlalchemy/engine/base.py", line 1316, in _execute_context self._handle_dbapi_exception( File "/home/fedora/superset/venv/lib64/python3.8/site-packages/sqlalchemy/engine/base.py", line 1510, in _handle_dbapi_exception util.raise_( File "/home/fedora/superset/venv/lib64/python3.8/site-packages/sqlalchemy/util/compat.py", line 182, in raise_ raise exception File "/home/fedora/superset/venv/lib64/python3.8/site-packages/sqlalchemy/engine/base.py", line 1276, in _execute_context self.dialect.do_execute( File "/home/fedora/superset/venv/lib64/python3.8/site-packages/sqlalchemy/engine/default.py", line 593, in do_execute cursor.execute(statement, parameters) sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) too many SQL variables [SQL: INSERT INTO "FCC 2018 Survey" ("ID", is_software_dev, is_first_dev_job, months_job_search, job_pref, job_intr_fllstck, job_intr_backend, job_intr_frntend, job_intr_mobile, job_intr_devops, job_intr_datasci, job_intr_teacher, job_intr_qa_engn, job_intr_ux_engn, job_intr_projm, job_intr_gamedev, job_intr_infosec, job_intr_dataengn, job_intr_other, when_appl_job, expected_earn, job_lctn_pref, job_relocate, reasons_to_code, reasons_to_code_other, rsrc_fcc, rsrc_mdn, rsrc_so, rsrc_edx, rsrc_coursera, rsrc_khan_acdm, rsrc_pluralsght, rsrc_codeacdm, rsrc_udacity, rsrc_udemy, rsrc_code_wars, rsrc_treehouse, rsrc_hackerrank, rsrc_frntendmstr, rsrc_lynda, rsrc_egghead, rsrc_css_tricks, rsrc_other, codeevnt_fcc, codeevnt_hackthn, codeevnt_confs, codeevnt_workshps, codeevnt_startupwknd, codeevnt_nodeschl, codeevnt_womenwc, codeevnt_girldevit, codeevnt_coderdojo, codeevnt_meetup, codeevnt_railsbrdg, codeevnt_gamejam, codeevnt_railsgrls, codeevnt_djangogrls, codeevnt_wkndbtcmp, codeevnt_o ther, podcast_fcc, podcast_codenewbie, podcast_changelog, podcast_sedaily, podcast_js_jabber, podcast_syntaxfm, podcast_ltcwm, podcast_fullstckrd, podcast_frnthppyhr, podcast_codingblcks, podcast_shoptalk, podcast_devtea, podcast_progthrwdwn, podcast_geekspeak, podcast_hanselmnts, podcast_talkpythonme, podcast_rubyrogues, podcast_codepenrd, podcast_seradio, podcast_other, yt_mit_ocw, yt_fcc, yt_computerphile, yt_devtips, yt_cs_dojo, yt_engn_truth, yt_learncodeacdm, yt_lvluptuts, yt_funfunfunct, yt_codingtuts360, yt_codingtrain, yt_derekbanas, yt_simplilearn, yt_simpleprog, yt_mozillahacks, yt_googledevs, yt_other, hours_learning, months_programming, bootcamp_attend, bootcamp_name, bootcamp_finished, bootcamp_have_loan, bootcamp_recommend, money_for_learning, age, gender, gender_other, country_citizen, country_live, live_city_population, is_ethnic_minority, lang_at_home, school_degree, school_major, marital_status, has_finance_depends, has_children, num_children, do_finance_support, debt_amt, home_mrtg_has, home_mrtg_owe, student_debt_has, student_debt_amt, curr_emplymnt, curr_emplymnt_other, curr_field, last_yr_income, communite_time, is_self_employed, has_served_military, is_recv_disab_bnft, has_high_spd_ntnet, time_start, time_end, network_id, time_total_sec) VALUES […] (Background on this error at: http://sqlalche.me/e/13/e3q8) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/fedora/superset/venv/bin/superset", line 11, in <module> load_entry_point('apache-superset==1.0.1', 'console_scripts', 'superset')() File "/home/fedora/superset/venv/lib64/python3.8/site-packages/click/core.py", line 829, in __call__ return self.main(*args, **kwargs) File "/home/fedora/superset/venv/lib64/python3.8/site-packages/flask/cli.py", line 586, in main return super(FlaskGroup, self).main(*args, **kwargs) File "/home/fedora/superset/venv/lib64/python3.8/site-packages/click/core.py", line 782, in main rv = self.invoke(ctx) File "/home/fedora/superset/venv/lib64/python3.8/site-packages/click/core.py", line 1259, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/home/fedora/superset/venv/lib64/python3.8/site-packages/click/core.py", line 1066, in invoke return ctx.invoke(self.callback, **ctx.params) File "/home/fedora/superset/venv/lib64/python3.8/site-packages/click/core.py", line 610, in invoke return callback(*args, **kwargs) File "/home/fedora/superset/venv/lib64/python3.8/site-packages/click/decorators.py", line 21, in new_func return f(get_current_context(), *args, **kwargs) File "/home/fedora/superset/venv/lib64/python3.8/site-packages/flask/cli.py", line 426, in decorator return __ctx.invoke(f, *args, **kwargs) File "/home/fedora/superset/venv/lib64/python3.8/site-packages/click/core.py", line 610, in invoke return callback(*args, **kwargs) File "/home/fedora/superset/venv/lib64/python3.8/site-packages/superset/cli.py", line 187, in load_examples load_examples_run(load_test_data, only_metadata, force) File "/home/fedora/superset/venv/lib64/python3.8/site-packages/superset/cli.py", line 171, in load_examples_run examples.load_from_configs(force) File "/home/fedora/superset/venv/lib64/python3.8/site-packages/superset/examples/utils.py", line 30, in load_from_configs command.run() File "/home/fedora/superset/venv/lib64/python3.8/site-packages/superset/commands/importers/v1/examples.py", line 71, in run raise self.import_error() superset.commands.exceptions.CommandException ``` Weirdly, the solution seems to be… running `superset load-examples` again. 3. Test suite not passing ```shell # Following https://github.com/apache/superset/blob/master/CONTRIBUTING.md#setup-local-environment-for-development git clone [email protected]:apache/superset.git cd superset git checkout 1.0.1 # to make things reproducible sudo dnf install -y gcc gcc-c++ libffi-devel python-devel python-pip python-wheel openssl-devel cyrus-sasl-devel openldap-devel # as per https://superset.apache.org/docs/installation/installing-superset-from-scratch#os-dependencies sudo dnf -y install mariadb-devel # FIX as per https://stackoverflow.com/q/22949654/5433628 python3.8 -m venv venv source venv/bin/activate pip install -r requirements/local.txt pip install . superset fab create-admin --username admin --firstname Ad --lastname Ministrator --email [email protected] --password secret # FIX explicitely create an admin user with username exactly `admin` superset db upgrade superset init superset load-examples || superset load-examples # FIX pip install -r requirements/testing.txt # FIX as, btw, this isn't specified but otherwise `tox` isn't installed tox -e py38 ``` and this fails: ``` […] ======================================================= test session starts ======================================================= platform linux -- Python 3.8.7, pytest-6.1.2, py-1.9.0, pluggy-0.13.1 cachedir: .tox/py38/.pytest_cache rootdir: /home/fedora/superset, configfile: pytest.ini, testpaths: tests plugins: cov-2.10.1, celery-4.4.7 collected 1194 items […] ======================== 60 failed, 1007 passed, 16 skipped, 220 warnings, 115 errors in 931.35s (0:15:31) ======================== ERROR: InvocationError for command /home/fedora/superset/.tox/py38/bin/pytest -s (exited with code 1) ``` One could also run the [shell script to run tests faster](https://github.com/apache/superset/tree/master/scripts/tests): ```shell # Following https://github.com/apache/superset/blob/master/CONTRIBUTING.md#setup-local-environment-for-development git clone [email protected]:apache/superset.git cd superset git checkout 1.0.1 # to make things reproducible sudo dnf install -y gcc gcc-c++ libffi-devel python-devel python-pip python-wheel openssl-devel cyrus-sasl-devel openldap-devel # as per https://superset.apache.org/docs/installation/installing-superset-from-scratch#os-dependencies sudo dnf -y install mariadb-devel # FIX as per https://stackoverflow.com/q/22949654/5433628 python3.8 -m venv venv source venv/bin/activate pip install -r requirements/local.txt pip install . superset fab create-admin --username admin --firstname Ad --lastname Ministrator --email [email protected] --password secret # FIX explicitely create an admin user with username exactly `admin` superset db upgrade superset init superset load-examples || superset load-examples # FIX pip install -r requirements/testing.txt # FIX as, btw, this isn't specified but otherwise `tox` isn't installed scripts/tests/run.sh ``` but then it misses some OS dependencies: ``` ------------------------------------ DB_URI=postgresql+psycopg2://superset:superset@localhost/test Superset config module=tests.superset_test_config Run init procedures=1 Run reset DB=1 Test to run:tests ------------------------------------ -------------------- Reseting test DB -------------------- scripts/tests/run.sh: line 29: docker-compose: command not found scripts/tests/run.sh: line 40: docker: command not found ``` So after installing it and setting it up: ```shell # Following https://github.com/apache/superset/blob/master/CONTRIBUTING.md#setup-local-environment-for-development git clone [email protected]:apache/superset.git cd superset git checkout 1.0.1 # to make things reproducible sudo dnf install -y gcc gcc-c++ libffi-devel python-devel python-pip python-wheel openssl-devel cyrus-sasl-devel openldap-devel # as per https://superset.apache.org/docs/installation/installing-superset-from-scratch#os-dependencies sudo dnf -y install mariadb-devel # FIX as per https://stackoverflow.com/q/22949654/5433628 # FIX install docker as per https://docs.docker.com/engine/install/fedora/ sudo dnf -y install dnf-plugins-core sudo dnf config-manager --add-repo https://download.docker.com/linux/fedora/docker-ce.repo sudo dnf install docker-ce docker-ce-cli containerd.io sudo systemctl start docker sudo systemctl enable docker sudo groupadd docker sudo usermod -aG docker $USER sudo curl -L "https://github.com/docker/compose/releases/download/1.29.0/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose sudo reboot python3.8 -m venv venv source venv/bin/activate pip install -r requirements/local.txt pip install . superset fab create-admin --username admin --firstname Ad --lastname Ministrator --email [email protected] --password secret # FIX explicitely create an admin user with username exactly `admin` superset db upgrade superset init superset load-examples || superset load-examples # FIX pip install -r requirements/testing.txt # FIX as, btw, this isn't specified but otherwise `tox` isn't installed scripts/tests/run.sh ``` But I get ``` scripts/tests/run.sh ------------------------------------ DB_URI=postgresql+psycopg2://superset:superset@localhost/test Superset config module=tests.superset_test_config Run init procedures=1 Run reset DB=1 Test to run:tests ------------------------------------ -------------------- Reseting test DB -------------------- Error: No such container: superset_db ``` ### Environment (please complete the following information): - superset version: 1.0.1 (tested with `667eb83` — latest commit passing CI at 2021-04-09T14:58:35Z — as well) - python version: 3.8.7 - node.js version: n/a ### Checklist Make sure to follow these steps before submitting your issue - thank you! - [x] I have checked the superset logs for python stacktraces and included it here as text if there are any. - [x] I have reproduced the issue with at least the latest released version of superset. - [x] I have checked the issue tracker for the same issue and I haven't found one similar. ### Additional information It may well be that I'm not reading the instructions correctly! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
