jonasjancarik opened a new issue #13809:
URL: https://github.com/apache/superset/issues/13809
When installing Superset from scratch on Amazon Linux 2 (essentially the
default for Amazon EC2), `superset load_examples` will fail due to too many SQL
variables being used in a SQLite query. This probably also affects Fedora and
other RHEL-derivative Linux distributions.
### Expected results
Example datasets to install correctly with `superset load_examples`.
### Actual results
```Traceback (most recent call last):
File
"/home/jonasjancarik/venv/lib64/python3.7/site-packages/sqlalchemy/engine/base.py",
line 1277, in _execute_context
cursor, statement, parameters, context
File
"/home/jonasjancarik/venv/lib64/python3.7/site-packages/sqlalchemy/engine/default.py",
line 608, in do_execute
cursor.execute(statement, parameters)
sqlite3.OperationalError: too many SQL variables
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/jonasjancarik/venv/bin/superset", line 11, in <module>
load_entry_point('apache-superset==1.0.1', 'console_scripts',
'superset')()
File
"/home/jonasjancarik/venv/lib64/python3.7/site-packages/click/core.py", line
829, in __call__
return self.main(*args, **kwargs)
File
"/home/jonasjancarik/venv/lib64/python3.7/site-packages/flask/cli.py", line
586, in main
return super(FlaskGroup, self).main(*args, **kwargs)
File
"/home/jonasjancarik/venv/lib64/python3.7/site-packages/click/core.py", line
782, in main
rv = self.invoke(ctx)
File
"/home/jonasjancarik/venv/lib64/python3.7/site-packages/click/core.py", line
1259, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File
"/home/jonasjancarik/venv/lib64/python3.7/site-packages/click/core.py", line
1066, in invoke
return ctx.invoke(self.callback, **ctx.params)
File
"/home/jonasjancarik/venv/lib64/python3.7/site-packages/click/core.py", line
610, in invoke
return callback(*args, **kwargs)
File
"/home/jonasjancarik/venv/lib64/python3.7/site-packages/click/decorators.py",
line 21, in new_func
return f(get_current_context(), *args, **kwargs)
File
"/home/jonasjancarik/venv/lib64/python3.7/site-packages/flask/cli.py", line
426, in decorator
return __ctx.invoke(f, *args, **kwargs)
File
"/home/jonasjancarik/venv/lib64/python3.7/site-packages/click/core.py", line
610, in invoke
return callback(*args, **kwargs)
File
"/home/jonasjancarik/venv/lib64/python3.7/site-packages/superset/cli.py", line
187, in load_examples
load_examples_run(load_test_data, only_metadata, force)
File
"/home/jonasjancarik/venv/lib64/python3.7/site-packages/superset/cli.py", line
128, in load_examples_run
examples.load_world_bank_health_n_pop(only_metadata, force)
File
"/home/jonasjancarik/venv/lib64/python3.7/site-packages/superset/examples/world_bank.py",
line 77, in load_world_bank_health_n_pop
index=False,
File
"/home/jonasjancarik/venv/lib64/python3.7/site-packages/pandas/core/generic.py",
line 2615, in to_sql
method=method,
File
"/home/jonasjancarik/venv/lib64/python3.7/site-packages/pandas/io/sql.py", line
598, in to_sql
method=method,
File
"/home/jonasjancarik/venv/lib64/python3.7/site-packages/pandas/io/sql.py", line
1406, in to_sql
raise err
File
"/home/jonasjancarik/venv/lib64/python3.7/site-packages/pandas/io/sql.py", line
1398, in to_sql
table.insert(chunksize, method=method)
File
"/home/jonasjancarik/venv/lib64/python3.7/site-packages/pandas/io/sql.py", line
830, in insert
exec_insert(conn, keys, chunk_iter)
File
"/home/jonasjancarik/venv/lib64/python3.7/site-packages/pandas/io/sql.py", line
758, in _execute_insert_multi
conn.execute(self.table.insert(data))
File
"/home/jonasjancarik/venv/lib64/python3.7/site-packages/sqlalchemy/engine/base.py",
line 1011, in execute
return meth(self, multiparams, params)
File
"/home/jonasjancarik/venv/lib64/python3.7/site-packages/sqlalchemy/sql/elements.py",
line 298, in _execute_on_connection
return connection._execute_clauseelement(self, multiparams, params)
File
"/home/jonasjancarik/venv/lib64/python3.7/site-packages/sqlalchemy/engine/base.py",
line 1130, in _execute_clauseelement
distilled_params,
File
"/home/jonasjancarik/venv/lib64/python3.7/site-packages/sqlalchemy/engine/base.py",
line 1317, in _execute_context
e, statement, parameters, cursor, context
File
"/home/jonasjancarik/venv/lib64/python3.7/site-packages/sqlalchemy/engine/base.py",
line 1511, in _handle_dbapi_exception
sqlalchemy_exception, with_traceback=exc_info[2], from_=e
File
"/home/jonasjancarik/venv/lib64/python3.7/site-packages/sqlalchemy/util/compat.py",
line 182, in raise_
raise exception
File
"/home/jonasjancarik/venv/lib64/python3.7/site-packages/sqlalchemy/engine/base.py",
line 1277, in _execute_context
cursor, statement, parameters, context
File
"/home/jonasjancarik/venv/lib64/python3.7/site-packages/sqlalchemy/engine/default.py",
line 608, in do_execute
cursor.execute(statement, parameters)
sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) too many SQL
variables
```
#### How to reproduce the bug
1. Install Superset from scratch following the docs on Amazon Linux 2 (e.g.
on a fresh Amazon EC2 instance)
2. Run `superset load_examples`
4. See the error `sqlite3.OperationalError: too many SQL variables`
### Environment
(please complete the following information):
- superset version: `1.0.1`
- python version: `3.7.9`
### Checklist
Make sure to follow these steps before submitting your issue - thank you!
- [x] I have checked the superset logs for python stacktraces and included
it here as text if there are any.
- [x] I have reproduced the issue with at least the latest released version
of superset.
- [x] I have checked the issue tracker for the same issue and I haven't
found one similar.
### Additional context
This is most likely due to the fact that SQLite on Amazon Linux 2 (and other
RHEL-derivative Linux distributions) is compiled with the default
SQLITE_MAX_VARIABLE_NUMBER option, which limits the number of variables in
queries to 999 - see https://bugzilla.redhat.com/show_bug.cgi?id=1798134 and
https://github.com/optuna/optuna/issues/1457#issuecomment-653587345. This limit
is much lower than the limit used in other distros (250000).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
