This is an automated email from the ASF dual-hosted git repository. villebro pushed a commit to branch 1.0 in repository https://gitbox.apache.org/repos/asf/superset.git
commit 10ca58f89a1151a48ba94a54414fbb4b8041f081 Author: Rob DiCiuccio <[email protected]> AuthorDate: Tue Jan 19 13:57:25 2021 -0800 Add docs for GLOBAL_ASYNC_QUERIES (SIP-39) (#12573) --- CONTRIBUTING.md | 24 ++++++++++++++++++++++++ docs/installation.rst | 26 ++++++++++++++++++++++++++ 2 files changed, 50 insertions(+) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 2980c59..b0670eb 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -81,6 +81,7 @@ little bit helps, and credit will always be given. - [Adding a DB migration](#adding-a-db-migration) - [Merging DB migrations](#merging-db-migrations) - [SQL Lab Async](#sql-lab-async) + - [Async Chart Queries](#async-chart-queries) - [Chart Parameters](#chart-parameters) - [Datasource & Chart Type](#datasource--chart-type) - [Time](#time) @@ -969,6 +970,29 @@ Note that: to your production environment, and use the similar broker as well as results backend configuration +### Async Chart Queries + +It's possible to configure database queries for charts to operate in `async` mode. This is especially useful for dashboards with many charts that may otherwise be affected by browser connection limits. To enable async queries for dashboards and Explore, the following dependencies are required: + +- Redis 5.0+ (the feature utilizes [Redis Streams](https://redis.io/topics/streams-intro)) +- Cache backends enabled via the `CACHE_CONFIG` and `DATA_CACHE_CONFIG` config settings +- Celery workers configured and running to process async tasks + +The following configuration settings are available for async queries (see config.py for default values) + +- `GLOBAL_ASYNC_QUERIES` (feature flag) - enable or disable async query operation +- `GLOBAL_ASYNC_QUERIES_REDIS_CONFIG` - Redis connection info +- `GLOBAL_ASYNC_QUERIES_REDIS_STREAM_PREFIX` - the prefix used with Redis Streams +- `GLOBAL_ASYNC_QUERIES_REDIS_STREAM_LIMIT` - the maximum number of events for each user-specific event stream (FIFO eviction) +- `GLOBAL_ASYNC_QUERIES_REDIS_STREAM_LIMIT_FIREHOSE` - the maximum number of events for all users (FIFO eviction) +- `GLOBAL_ASYNC_QUERIES_JWT_COOKIE_NAME` - the async query feature uses a [JWT](https://tools.ietf.org/html/rfc7519) cookie for authentication, this setting is the cookie's name +- `GLOBAL_ASYNC_QUERIES_JWT_COOKIE_SECURE` - JWT cookie secure option +- `GLOBAL_ASYNC_QUERIES_JWT_SECRET` - JWT's use a secret key to sign and validate the contents. This value should be at least 32 bytes and have sufficient randomness for proper security +- `GLOBAL_ASYNC_QUERIES_TRANSPORT` - currently the only available option is (HTTP) `polling`, but support for a WebSocket will be added in future versions +- `GLOBAL_ASYNC_QUERIES_POLLING_DELAY` - the time (in ms) between polling requests + +More information on the async query feature can be found in [SIP-39](https://github.com/apache/superset/issues/9190). + ## Chart Parameters Chart parameters are stored as a JSON encoded string the `slices.params` column and are often referenced throughout the code as form-data. Currently the form-data is neither versioned nor typed as thus is somewhat free-formed. Note in the future there may be merit in using something like [JSON Schema](https://json-schema.org/) to both annotate and validate the JSON object in addition to using a Mypy `TypedDict` (introduced in Python 3.8) for typing the form-data in the backend. This sect [...] diff --git a/docs/installation.rst b/docs/installation.rst index 095f6be..e283116 100644 --- a/docs/installation.rst +++ b/docs/installation.rst @@ -1113,6 +1113,32 @@ serialization. This can be disabled by setting ``RESULTS_BACKEND_USE_MSGPACK = F in your configuration, should any issues arise. Please clear your existing results cache store when upgrading an existing environment. +**Async queries for dashboards and Explore** + +It's also possible to configure database queries for charts to operate in `async` mode. +This is especially useful for dashboards with many charts that may otherwise be affected +by browser connection limits. To enable async queries for dashboards and Explore, the +following dependencies are required: + +- Redis 5.0+ (the feature utilizes `Redis Streams <https://redis.io/topics/streams-intro>`_) +- Cache backends enabled via the ``CACHE_CONFIG`` and ``DATA_CACHE_CONFIG`` config settings +- Celery workers configured and running to process async tasks + +The following configuration settings are available for async queries (see config.py for default values) + +- ``GLOBAL_ASYNC_QUERIES`` (feature flag) - enable or disable async query operation +- ``GLOBAL_ASYNC_QUERIES_REDIS_CONFIG`` - Redis connection info +- ``GLOBAL_ASYNC_QUERIES_REDIS_STREAM_PREFIX`` - the prefix used with Redis Streams +- ``GLOBAL_ASYNC_QUERIES_REDIS_STREAM_LIMIT`` - the maximum number of events for each user-specific event stream (FIFO eviction) +- ``GLOBAL_ASYNC_QUERIES_REDIS_STREAM_LIMIT_FIREHOSE`` - the maximum number of events for all users (FIFO eviction) +- ``GLOBAL_ASYNC_QUERIES_JWT_COOKIE_NAME`` - the async query feature uses a `JWT <https://tools.ietf.org/html/rfc7519>`_ cookie for authentication, this setting is the cookie's name +- ``GLOBAL_ASYNC_QUERIES_JWT_COOKIE_SECURE`` - JWT cookie secure option +- ``GLOBAL_ASYNC_QUERIES_JWT_SECRET`` - JWT's use a secret key to sign and validate the contents. This value should be at least 32 bytes and have sufficient randomness for proper security +- ``GLOBAL_ASYNC_QUERIES_TRANSPORT`` - currently the only available option is (HTTP) `polling`, but support for a WebSocket will be added in future versions +- ``GLOBAL_ASYNC_QUERIES_POLLING_DELAY`` - the time (in ms) between polling requests + +More information on the async query feature can be found in `SIP-39 <https://github.com/apache/superset/issues/9190>`_. + **Important notes** * It is important that all the worker nodes and web servers in
