This is an automated email from the ASF dual-hosted git repository. rusackas pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/incubator-superset-site.git
commit d416edfb7c76a23836aec671a24a99b80470f80d Author: Maxime Beauchemin <[email protected]> AuthorDate: Sat Jan 26 17:05:41 2019 -0800 As of ~0.31 --- _modules/superset/jinja_context.html | 35 +++++- _sources/druid.rst.txt | 19 ++- _sources/faq.rst.txt | 19 ++- _sources/gallery.rst.txt | 17 +++ _sources/import_export_datasources.rst.txt | 17 +++ _sources/index.rst.txt | 17 +++ _sources/installation.rst.txt | 193 +++++++++++++++++++++++------ _sources/misc.rst.txt | 17 +++ _sources/security.rst.txt | 17 +++ _sources/sqllab.rst.txt | 19 ++- _sources/tutorial.rst.txt | 17 +++ _sources/videos.rst.txt | 17 +++ _sources/visualization.rst.txt | 124 +++++++++++++++++- _static/docs.css | 18 +++ druid.html | 2 +- faq.html | 2 +- index.html | 4 + installation.html | 177 ++++++++++++++++++++------ searchindex.js | 2 +- sqllab.html | 15 ++- visualization.html | 162 +++++++++++++++++++++++- 21 files changed, 817 insertions(+), 93 deletions(-) diff --git a/_modules/superset/jinja_context.html b/_modules/superset/jinja_context.html index b70d86a..41f20a4 100644 --- a/_modules/superset/jinja_context.html +++ b/_modules/superset/jinja_context.html @@ -149,7 +149,23 @@ <div itemprop="articleBody"> <h1>Source code for superset.jinja_context</h1><div class="highlight"><pre> -<span></span><span class="c1"># pylint: disable=C,R,W</span> +<span></span><span class="c1"># Licensed to the Apache Software Foundation (ASF) under one</span> +<span class="c1"># or more contributor license agreements. See the NOTICE file</span> +<span class="c1"># distributed with this work for additional information</span> +<span class="c1"># regarding copyright ownership. The ASF licenses this file</span> +<span class="c1"># to you under the Apache License, Version 2.0 (the</span> +<span class="c1"># "License"); you may not use this file except in compliance</span> +<span class="c1"># with the License. You may obtain a copy of the License at</span> +<span class="c1">#</span> +<span class="c1"># http://www.apache.org/licenses/LICENSE-2.0</span> +<span class="c1">#</span> +<span class="c1"># Unless required by applicable law or agreed to in writing,</span> +<span class="c1"># software distributed under the License is distributed on an</span> +<span class="c1"># "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY</span> +<span class="c1"># KIND, either express or implied. See the License for the</span> +<span class="c1"># specific language governing permissions and limitations</span> +<span class="c1"># under the License.</span> +<span class="c1"># pylint: disable=C,R,W</span> <span class="sd">"""Defines the templating context for SQL Lab"""</span> <span class="kn">from</span> <span class="nn">datetime</span> <span class="k">import</span> <span class="n">datetime</span><span class="p">,</span> <span class="n">timedelta</span> <span class="kn">import</span> <span class="nn">inspect</span> @@ -177,7 +193,18 @@ <div class="viewcode-block" id="url_param"><a class="viewcode-back" href="../../sqllab.html#superset.jinja_context.url_param">[docs]</a><span class="k">def</span> <span class="nf">url_param</span><span class="p">(</span><span class="n">param</span><span class="p">,</span> <span class="n">default</span><span class="o">=</span><span class="kc">None</span><span class="p">):</span> - <span class="sd">"""Get a url or post data parameter</span> + <span class="sd">"""Read a url or post parameter and use it in your SQL Lab query</span> + +<span class="sd"> When in SQL Lab, it's possible to add arbitrary URL "query string"</span> +<span class="sd"> parameters, and use those in your SQL code. For instance you can</span> +<span class="sd"> alter your url and add `?foo=bar`, as in</span> +<span class="sd"> `{domain}/superset/sqllab?foo=bar`. Then if your query is something like</span> +<span class="sd"> SELECT * FROM foo = '{{ url_param('foo') }}', it will be parsed at</span> +<span class="sd"> runtime and replaced by the value in the URL.</span> + +<span class="sd"> As you create a visualization form this SQL Lab query, you can pass</span> +<span class="sd"> parameters in the explore view as well as from the dashboard, and</span> +<span class="sd"> it should carry through to your queries.</span> <span class="sd"> :param param: the parameter to lookup</span> <span class="sd"> :type param: str</span> @@ -189,7 +216,7 @@ <span class="c1"># Supporting POST as well as get</span> <span class="k">if</span> <span class="n">request</span><span class="o">.</span><span class="n">form</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s1">'form_data'</span><span class="p">):</span> <span class="n">form_data</span> <span class="o">=</span> <span class="n">json</span><span class="o">.</span><span class="n">loads</span><span class="p">(</span><span class="n">request</span><span class="o">.</span><span class="n">form</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s1">'form_data'</span><span class="p">))</span> - <span class="n">url_params</span> <span class="o">=</span> <span class="n">form_data</span><span class="p">[</span><span class="s1">'url_params'</span><span class="p">]</span> <span class="ow">or</span> <span class="p">{}</span> + <span class="n">url_params</span> <span class="o">=</span> <span class="n">form_data</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s1">'url_params'</span><span class="p">)</span> <span class="ow">or</span> <span class="p">{}</span> <span class="k">return</span> <span class="n">url_params</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">param</span><span class="p">,</span> <span class="n">default</span><span class="p">)</span> <span class="k">return</span> <span class="n">default</span></div> @@ -219,7 +246,7 @@ <span class="sd"> Usage example:</span> <span class="sd"> SELECT action, count(*) as times</span> <span class="sd"> FROM logs</span> -<span class="sd"> WHERE action in ( {{ "'" + "','".join(filter_values('action_type')) + "'" )</span> +<span class="sd"> WHERE action in ( {{ "'" + "','".join(filter_values('action_type')) + "'" }} )</span> <span class="sd"> GROUP BY 1</span> <span class="sd"> :param column: column/filter name to lookup</span> diff --git a/_sources/druid.rst.txt b/_sources/druid.rst.txt index 5d14999..74139d6 100644 --- a/_sources/druid.rst.txt +++ b/_sources/druid.rst.txt @@ -1,3 +1,20 @@ +.. Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + +.. http://www.apache.org/licenses/LICENSE-2.0 + +.. Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + Druid ===== @@ -33,7 +50,7 @@ Post-Aggregations ----------------- Druid supports post aggregation and this works in Superset. All you have to -do is creating a metric, much like you would create an aggregation manually, +do is create a metric, much like you would create an aggregation manually, but specify ``postagg`` as a ``Metric Type``. You then have to provide a valid json post-aggregation definition (as specified in the Druid docs) in the Json field. diff --git a/_sources/faq.rst.txt b/_sources/faq.rst.txt index 3b69044..648bb86 100644 --- a/_sources/faq.rst.txt +++ b/_sources/faq.rst.txt @@ -1,3 +1,20 @@ +.. Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + +.. http://www.apache.org/licenses/LICENSE-2.0 + +.. Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + FAQ === @@ -88,7 +105,7 @@ It's easy: use the ``Filter Box`` widget, build a slice, and add it to your dashboard. The ``Filter Box`` widget allows you to define a query to populate dropdowns -that can be use for filtering. To build the list of distinct values, we +that can be used for filtering. To build the list of distinct values, we run a query, and sort the result by the metric you provide, sorting descending. diff --git a/_sources/gallery.rst.txt b/_sources/gallery.rst.txt index 5fa212d..27b9c2a 100644 --- a/_sources/gallery.rst.txt +++ b/_sources/gallery.rst.txt @@ -1,3 +1,20 @@ +.. Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + +.. http://www.apache.org/licenses/LICENSE-2.0 + +.. Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + Visualizations Gallery ====================== diff --git a/_sources/import_export_datasources.rst.txt b/_sources/import_export_datasources.rst.txt index d398428..9c786ce 100644 --- a/_sources/import_export_datasources.rst.txt +++ b/_sources/import_export_datasources.rst.txt @@ -1,3 +1,20 @@ +.. Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + +.. http://www.apache.org/licenses/LICENSE-2.0 + +.. Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + Importing and Exporting Datasources =================================== diff --git a/_sources/index.rst.txt b/_sources/index.rst.txt index 08e1a64..926031f 100644 --- a/_sources/index.rst.txt +++ b/_sources/index.rst.txt @@ -1,3 +1,20 @@ +.. Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + +.. http://www.apache.org/licenses/LICENSE-2.0 + +.. Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + |apache_img| |superset_img| .. |apache_img| image:: images/apache_feather.png diff --git a/_sources/installation.rst.txt b/_sources/installation.rst.txt index 75f6d25..f1a6d76 100644 --- a/_sources/installation.rst.txt +++ b/_sources/installation.rst.txt @@ -1,3 +1,20 @@ +.. Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + +.. http://www.apache.org/licenses/LICENSE-2.0 + +.. Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + Installation & Configuration ============================ @@ -43,22 +60,29 @@ If you know docker, then you're lucky, we have shortcut road for you to initialize development environment: :: git clone https://github.com/apache/incubator-superset/ - cd incubator-superset - cp contrib/docker/{docker-build.sh,docker-compose.yml,docker-entrypoint.sh,docker-init.sh,Dockerfile} . - cp contrib/docker/superset_config.py superset/ - bash -x docker-build.sh - docker-compose up -d - docker-compose exec superset bash - bash docker-init.sh + cd incubator-superset/contrib/docker + # prefix with SUPERSET_LOAD_EXAMPLES=yes to load examples: + docker-compose run --rm superset ./docker-init.sh + # you can run this command everytime you need to start superset now: + docker-compose up After several minutes for superset initialization to finish, you can open a browser and view `http://localhost:8088` to start your journey. +From there, the container server will reload on modification of the superset python +and javascript source code. +Don't forget to reload the page to take the new frontend into account though. + +See also `CONTRIBUTING.md <https://github.com/apache/incubator-superset/blob/master/CONTRIBUTING.md#webpack-dev-server>`_, +for alternative way of serving the frontend. + +It is also possible to run Superset in non-development mode: in the `docker-compose.yml` file remove +the volumes needed for development and change the variable `SUPERSET_ENV` to `production`. + If you are attempting to build on a Mac and it exits with 137 you need to increase your docker resources. OSX instructions: https://docs.docker.com/docker-for-mac/#advanced (Search for memory) -Or if you're curious and want to install superset from bottom up, then go -ahead. +Or if you're curious and want to install superset from bottom up, then go ahead. OS dependencies --------------- @@ -78,9 +102,9 @@ the required dependencies are installed: :: sudo apt-get install build-essential libssl-dev libffi-dev python-dev python-pip libsasl2-dev libldap2-dev -**Ubuntu 16.04** If you have python3.5 installed alongside with python2.7, as is default on **Ubuntu 16.04 LTS**, run this command also: :: +**Ubuntu 18.04** If you have python3.6 installed alongside with python2.7, as is default on **Ubuntu 18.04 LTS**, run this command also: :: - sudo apt-get install build-essential libssl-dev libffi-dev python3.5-dev python-pip libsasl2-dev libldap2-dev + sudo apt-get install build-essential libssl-dev libffi-dev python3.6-dev python-pip libsasl2-dev libldap2-dev otherwise build for ``cryptography`` fails. @@ -93,7 +117,7 @@ that the required dependencies are installed: :: **OSX**, system python is not recommended. brew's python also ships with pip :: brew install pkg-config libffi openssl python - env LDFLAGS="-L$(brew --prefix openssl)/lib" CFLAGS="-I$(brew --prefix openssl)/include" pip install cryptography==1.9 + env LDFLAGS="-L$(brew --prefix openssl)/lib" CFLAGS="-I$(brew --prefix openssl)/include" pip install cryptography==2.4.2 **Windows** isn't officially supported at this point, but if you want to attempt it, download `get-pip.py <https://bootstrap.pypa.io/get-pip.py>`_, and run ``python get-pip.py`` which may need admin access. Then run the following: :: @@ -105,17 +129,18 @@ attempt it, download `get-pip.py <https://bootstrap.pypa.io/get-pip.py>`_, and r Python virtualenv ----------------- -It is recommended to install Superset inside a virtualenv. Python 3 already ships virtualenv, for -Python 2 you need to install it. If it's packaged for your operating systems install it from there -otherwise you can install from pip: :: +It is recommended to install Superset inside a virtualenv. Python 3 already ships virtualenv. +But if it's not installed in your environment for some reason, you can install it +via the package for your operating systems, otherwise you can install from pip: :: pip install virtualenv You can create and activate a virtualenv by: :: - # virtualenv is shipped in Python 3 as pyvenv - virtualenv venv - . ./venv/bin/activate + # virtualenv is shipped in Python 3.6+ as venv instead of pyvenv. + # See https://docs.python.org/3.6/library/venv.html + python3 -m venv venv + . venv/bin/activate On windows the syntax for activating it is a bit different: :: @@ -396,10 +421,6 @@ Make sure the user has privileges to access and use all required databases/schemas/tables/views/warehouses, as the Snowflake SQLAlchemy engine does not test for user rights during engine creation. -*Note*: At the time of writing, there is a regression in the current stable version (1.1.2) of -snowflake-sqlalchemy package that causes problems when used with Superset. It is recommended to -use version 1.1.0 or try a newer version. - See `Snowflake SQLAlchemy <https://github.com/snowflakedb/snowflake-sqlalchemy>`_. Teradata @@ -551,6 +572,18 @@ The following keys in `superset_config.py` can be specified to configure CORS: * ``CORS_OPTIONS``: options passed to Flask-CORS (`documentation <http://flask-cors.corydolphin.com/en/latest/api.html#extension>`) +DOMAIN SHARDING +--------------- + +Chrome allows up to 6 open connections per domain at a time. When there are more +than 6 slices in dashboard, a lot of time fetch requests are queued up and wait for +next available socket. PR (`#5039 <https://github.com/apache/incubator-superset/pull/5039>`) adds domain sharding to Superset, +and this feature will be enabled by configuration only (by default Superset +doesn't allow cross-domain request). + +*``SUPERSET_WEBSERVER_DOMAINS``: list of allowed hostnames for domain sharding feature. default `None` + + MIDDLEWARE ---------- @@ -583,14 +616,12 @@ Upgrading should be as straightforward as running:: superset db upgrade superset init -SQL Lab -------- -SQL Lab is a powerful SQL IDE that works with all SQLAlchemy compatible -databases. By default, queries are executed in the scope of a web -request so they -may eventually timeout as queries exceed the maximum duration of a web -request in your environment, whether it'd be a reverse proxy or the Superset -server itself. +Celery Tasks +------------ +On large analytic databases, it's common to run background jobs, reports +and/or queries that execute for minutes or hours. In certain cases, we need +to support long running tasks that execute beyond the typical web request's +timeout (30-60 seconds). On large analytic databases, it's common to run queries that execute for minutes or hours. @@ -614,15 +645,41 @@ have the same configuration. class CeleryConfig(object): BROKER_URL = 'redis://localhost:6379/0' - CELERY_IMPORTS = ('superset.sql_lab', ) + CELERY_IMPORTS = ( + 'superset.sql_lab', + 'superset.tasks', + ) CELERY_RESULT_BACKEND = 'redis://localhost:6379/0' - CELERY_ANNOTATIONS = {'tasks.add': {'rate_limit': '10/s'}} + CELERYD_LOG_LEVEL = 'DEBUG' + CELERYD_PREFETCH_MULTIPLIER = 10 + CELERY_ACKS_LATE = True + CELERY_ANNOTATIONS = { + 'sql_lab.get_sql_results': { + 'rate_limit': '100/s', + }, + 'email_reports.send': { + 'rate_limit': '1/s', + 'time_limit': 120, + 'soft_time_limit': 150, + 'ignore_result': True, + }, + } + CELERYBEAT_SCHEDULE = { + 'email_reports.schedule_hourly': { + 'task': 'email_reports.schedule_hourly', + 'schedule': crontab(minute=1, hour='*'), + }, + } CELERY_CONFIG = CeleryConfig -To start a Celery worker to leverage the configuration run: :: +* To start a Celery worker to leverage the configuration run: :: + + celery worker --app=superset.tasks.celery_app:app --pool=prefork -Ofair -c 4 + +* To start a job which schedules periodic background jobs, run :: - celery worker --app=superset.sql_lab:celery_app --pool=gevent -Ofair + celery beat --app=superset.tasks.celery_app:app To setup a result backend, you need to pass an instance of a derivative of ``werkzeug.contrib.cache.BaseCache`` to the ``RESULTS_BACKEND`` @@ -645,11 +702,65 @@ look something like: RESULTS_BACKEND = RedisCache( host='localhost', port=6379, key_prefix='superset_results') -Note that it's important that all the worker nodes and web servers in -the Superset cluster share a common metadata database. -This means that SQLite will not work in this context since it has -limited support for concurrency and -typically lives on the local file system. +**Important notes** + +* It is important that all the worker nodes and web servers in + the Superset cluster share a common metadata database. + This means that SQLite will not work in this context since it has + limited support for concurrency and + typically lives on the local file system. + +* There should only be one instance of ``celery beat`` running in your + entire setup. If not, background jobs can get scheduled multiple times + resulting in weird behaviors like duplicate delivery of reports, + higher than expected load / traffic etc. + + +Email Reports +------------- +Email reports allow users to schedule email reports for + +* slice and dashboard visualization (Attachment or inline) +* slice data (CSV attachment on inline table) + +Schedules are defined in crontab format and each schedule +can have a list of recipients (all of them can receive a single mail, +or separate mails). For audit purposes, all outgoing mails can have a +mandatory bcc. + +**Requirements** + +* A selenium compatible driver & headless browser + + * `geckodriver <https://github.com/mozilla/geckodriver>`_ and Firefox is preferred + * `chromedriver <http://chromedriver.chromium.org/>`_ is a good option too +* Run `celery worker` and `celery beat` as follows :: + + celery worker --app=superset.tasks.celery_app:app --pool=prefork -Ofair -c 4 + celery beat --app=superset.tasks.celery_app:app + +**Important notes** + +* Be mindful of the concurrency setting for celery (using ``-c 4``). + Selenium/webdriver instances can consume a lot of CPU / memory on your servers. + +* In some cases, if you notice a lot of leaked ``geckodriver`` processes, try running + your celery processes with :: + + celery worker --pool=prefork --max-tasks-per-child=128 ... + +* It is recommended to run separate workers for ``sql_lab`` and + ``email_reports`` tasks. Can be done by using ``queue`` field in ``CELERY_ANNOTATIONS`` + +SQL Lab +------- +SQL Lab is a powerful SQL IDE that works with all SQLAlchemy compatible +databases. By default, queries are executed in the scope of a web +request so they may eventually timeout as queries exceed the maximum duration of a web +request in your environment, whether it'd be a reverse proxy or the Superset +server itself. In such cases, it is preferred to use ``celery`` to run the queries +in the background. Please follow the examples/notes mentioned above to get your +celery setup working. Also note that SQL Lab supports Jinja templating in queries and that it's possible to overload @@ -664,6 +775,8 @@ in this dictionary are made available for users to use in their SQL. } +Celery Flower +------------- Flower is a web based tool for monitoring the Celery cluster which you can install from pip: :: @@ -671,7 +784,7 @@ install from pip: :: and run via: :: - celery flower --app=superset.sql_lab:celery_app + celery flower --app=superset.tasks.celery_app:app Building from source --------------------- diff --git a/_sources/misc.rst.txt b/_sources/misc.rst.txt index fb953dd..840f17b 100644 --- a/_sources/misc.rst.txt +++ b/_sources/misc.rst.txt @@ -1,3 +1,20 @@ +.. Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + +.. http://www.apache.org/licenses/LICENSE-2.0 + +.. Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + Misc ---- diff --git a/_sources/security.rst.txt b/_sources/security.rst.txt index 7b11cf1..8894b29 100644 --- a/_sources/security.rst.txt +++ b/_sources/security.rst.txt @@ -1,3 +1,20 @@ +.. Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + +.. http://www.apache.org/licenses/LICENSE-2.0 + +.. Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + Security ======== Security in Superset is handled by Flask AppBuilder (FAB). FAB is a diff --git a/_sources/sqllab.rst.txt b/_sources/sqllab.rst.txt index 2ba9ac2..6d12971 100644 --- a/_sources/sqllab.rst.txt +++ b/_sources/sqllab.rst.txt @@ -1,3 +1,20 @@ +.. Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + +.. http://www.apache.org/licenses/LICENSE-2.0 + +.. Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + SQL Lab ======= @@ -19,7 +36,7 @@ Feature Overview - Browse database metadata: tables, columns, indexes, partitions - Support for long-running queries - - uses the `Celery distributed queue <http://www.python.org/>`_ + - uses the `Celery distributed queue <http://www.celeryproject.org/>`_ to dispatch query handling to workers - supports defining a "results backend" to persist query results diff --git a/_sources/tutorial.rst.txt b/_sources/tutorial.rst.txt index b6e64b4..9aa0c01 100644 --- a/_sources/tutorial.rst.txt +++ b/_sources/tutorial.rst.txt @@ -1,3 +1,20 @@ +.. Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + +.. http://www.apache.org/licenses/LICENSE-2.0 + +.. Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + Tutorial - Creating your first dashboard ======================================== diff --git a/_sources/videos.rst.txt b/_sources/videos.rst.txt index 9c00b41..ba41fd2 100644 --- a/_sources/videos.rst.txt +++ b/_sources/videos.rst.txt @@ -1,3 +1,20 @@ +.. Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + +.. http://www.apache.org/licenses/LICENSE-2.0 + +.. Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + Videos ====== diff --git a/_sources/visualization.rst.txt b/_sources/visualization.rst.txt index 0b8c3ee..b8c133a 100644 --- a/_sources/visualization.rst.txt +++ b/_sources/visualization.rst.txt @@ -1,3 +1,20 @@ +.. Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + +.. http://www.apache.org/licenses/LICENSE-2.0 + +.. Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + Visualization Tools =================== @@ -711,6 +728,108 @@ List of Countries |IT-VI |Vicenza | +------+------------------------------------+ + +* Japan + ++-------+----------------+ +| ISO | Name of region | ++=======+================+ +| JP-01 | Hokkaido | ++-------+----------------+ +| JP-02 | Aomori | ++-------+----------------+ +| JP-03 | Iwate | ++-------+----------------+ +| JP-04 | Miyagi | ++-------+----------------+ +| JP-05 | Akita | ++-------+----------------+ +| JP-06 | Yamagata | ++-------+----------------+ +| JP-07 | Fukushima | ++-------+----------------+ +| JP-08 | Ibaraki | ++-------+----------------+ +| JP-09 | Tochigi | ++-------+----------------+ +| JP-10 | Gunma | ++-------+----------------+ +| JP-11 | Saitama | ++-------+----------------+ +| JP-12 | Chiba | ++-------+----------------+ +| JP-13 | Tokyo | ++-------+----------------+ +| JP-14 | Kanagawa | ++-------+----------------+ +| JP-15 | Niigata | ++-------+----------------+ +| JP-16 | Toyama | ++-------+----------------+ +| JP-17 | Ishikawa | ++-------+----------------+ +| JP-18 | Fukui | ++-------+----------------+ +| JP-19 | Yamanashi | ++-------+----------------+ +| JP-20 | Nagano | ++-------+----------------+ +| JP-21 | Gifu | ++-------+----------------+ +| JP-22 | Shizuoka | ++-------+----------------+ +| JP-23 | Aichi | ++-------+----------------+ +| JP-24 | Mie | ++-------+----------------+ +| JP-25 | Shiga | ++-------+----------------+ +| JP-26 | Kyoto | ++-------+----------------+ +| JP-27 | Osaka | ++-------+----------------+ +| JP-28 | Hyogo | ++-------+----------------+ +| JP-29 | Nara | ++-------+----------------+ +| JP-30 | Wakayama | ++-------+----------------+ +| JP-31 | Tottori | ++-------+----------------+ +| JP-32 | Shimane | ++-------+----------------+ +| JP-33 | Okayama | ++-------+----------------+ +| JP-34 | Hiroshima | ++-------+----------------+ +| JP-35 | Yamaguchi | ++-------+----------------+ +| JP-36 | Tokushima | ++-------+----------------+ +| JP-37 | Kagawa | ++-------+----------------+ +| JP-38 | Ehime | ++-------+----------------+ +| JP-39 | Kochi | ++-------+----------------+ +| JP-40 | Fukuoka | ++-------+----------------+ +| JP-41 | Saga | ++-------+----------------+ +| JP-42 | Nagasaki | ++-------+----------------+ +| JP-43 | Kumamoto | ++-------+----------------+ +| JP-44 | Oita | ++-------+----------------+ +| JP-45 | Miyazaki | ++-------+----------------+ +| JP-46 | Kagoshima | ++-------+----------------+ +| JP-47 | Okinawa | ++-------+----------------+ + + * Morocco +-------+------------------------------+ @@ -1726,11 +1845,11 @@ To add a new country in country map tools, we need to follow the following steps 3. You need to convert shapefile to geojson file. This action can make with ogr2ogr tools: http://www.gdal.org/ogr2ogr.html -4. Put your geojson file in next folder : superset/assets/visualizations/countries with the next name : nameofyourcountries.geojson +4. Put your geojson file in next folder : superset/assets/src/visualizations/CountryMap/countries with the next name : nameofyourcountries.geojson 5. You can to reduce size of geojson file on this site: http://mapshaper.org/ -6. Go in file superset/assets/javascripts/explorev2/stores/controls.jsx +6. Go in file superset/assets/src/explore/controls.jsx 7. Add your country in component 'select_country' Example : @@ -1749,6 +1868,7 @@ To add a new country in country map tools, we need to follow the following steps 'France', 'Germany', 'Italy', + 'Japan', 'Morocco', 'Netherlands', 'Russia', diff --git a/_static/docs.css b/_static/docs.css index f592257..d15b7f9 100644 --- a/_static/docs.css +++ b/_static/docs.css @@ -1,3 +1,21 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ body { padding-top: 0px; } diff --git a/druid.html b/druid.html index bf96246..ede90b6 100644 --- a/druid.html +++ b/druid.html @@ -185,7 +185,7 @@ documentation.</p> <div class="section" id="post-aggregations"> <h2>Post-Aggregations<a class="headerlink" href="#post-aggregations" title="Permalink to this headline">¶</a></h2> <p>Druid supports post aggregation and this works in Superset. All you have to -do is creating a metric, much like you would create an aggregation manually, +do is create a metric, much like you would create an aggregation manually, but specify <code class="docutils literal notranslate"><span class="pre">postagg</span></code> as a <code class="docutils literal notranslate"><span class="pre">Metric</span> <span class="pre">Type</span></code>. You then have to provide a valid json post-aggregation definition (as specified in the Druid docs) in the Json field.</p> diff --git a/faq.html b/faq.html index 2447bd9..94761b0 100644 --- a/faq.html +++ b/faq.html @@ -236,7 +236,7 @@ other on the <code class="docutils literal notranslate"><span class="pre">Source <p>It’s easy: use the <code class="docutils literal notranslate"><span class="pre">Filter</span> <span class="pre">Box</span></code> widget, build a slice, and add it to your dashboard.</p> <p>The <code class="docutils literal notranslate"><span class="pre">Filter</span> <span class="pre">Box</span></code> widget allows you to define a query to populate dropdowns -that can be use for filtering. To build the list of distinct values, we +that can be used for filtering. To build the list of distinct values, we run a query, and sort the result by the metric you provide, sorting descending.</p> <p>The widget also has a checkbox <code class="docutils literal notranslate"><span class="pre">Date</span> <span class="pre">Filter</span></code>, which enables time filtering diff --git a/index.html b/index.html index 79f151c..38ee5b9 100644 --- a/index.html +++ b/index.html @@ -258,9 +258,13 @@ to the user</li> <li class="toctree-l2"><a class="reference internal" href="installation.html#ssl-access-to-databases">SSL Access to databases</a></li> <li class="toctree-l2"><a class="reference internal" href="installation.html#druid">Druid</a></li> <li class="toctree-l2"><a class="reference internal" href="installation.html#cors">CORS</a></li> +<li class="toctree-l2"><a class="reference internal" href="installation.html#domain-sharding">DOMAIN SHARDING</a></li> <li class="toctree-l2"><a class="reference internal" href="installation.html#middleware">MIDDLEWARE</a></li> <li class="toctree-l2"><a class="reference internal" href="installation.html#upgrading">Upgrading</a></li> +<li class="toctree-l2"><a class="reference internal" href="installation.html#celery-tasks">Celery Tasks</a></li> +<li class="toctree-l2"><a class="reference internal" href="installation.html#email-reports">Email Reports</a></li> <li class="toctree-l2"><a class="reference internal" href="installation.html#sql-lab">SQL Lab</a></li> +<li class="toctree-l2"><a class="reference internal" href="installation.html#celery-flower">Celery Flower</a></li> <li class="toctree-l2"><a class="reference internal" href="installation.html#building-from-source">Building from source</a></li> <li class="toctree-l2"><a class="reference internal" href="installation.html#blueprints">Blueprints</a></li> <li class="toctree-l2"><a class="reference internal" href="installation.html#statsd-logging">StatsD logging</a></li> diff --git a/installation.html b/installation.html index 505e4af..cd58b11 100644 --- a/installation.html +++ b/installation.html @@ -103,9 +103,13 @@ <li class="toctree-l2"><a class="reference internal" href="#ssl-access-to-databases">SSL Access to databases</a></li> <li class="toctree-l2"><a class="reference internal" href="#druid">Druid</a></li> <li class="toctree-l2"><a class="reference internal" href="#cors">CORS</a></li> +<li class="toctree-l2"><a class="reference internal" href="#domain-sharding">DOMAIN SHARDING</a></li> <li class="toctree-l2"><a class="reference internal" href="#middleware">MIDDLEWARE</a></li> <li class="toctree-l2"><a class="reference internal" href="#upgrading">Upgrading</a></li> +<li class="toctree-l2"><a class="reference internal" href="#celery-tasks">Celery Tasks</a></li> +<li class="toctree-l2"><a class="reference internal" href="#email-reports">Email Reports</a></li> <li class="toctree-l2"><a class="reference internal" href="#sql-lab">SQL Lab</a></li> +<li class="toctree-l2"><a class="reference internal" href="#celery-flower">Celery Flower</a></li> <li class="toctree-l2"><a class="reference internal" href="#building-from-source">Building from source</a></li> <li class="toctree-l2"><a class="reference internal" href="#blueprints">Blueprints</a></li> <li class="toctree-l2"><a class="reference internal" href="#statsd-logging">StatsD logging</a></li> @@ -220,21 +224,25 @@ as needed.</p> <p>If you know docker, then you’re lucky, we have shortcut road for you to initialize development environment:</p> <div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">git</span> <span class="n">clone</span> <span class="n">https</span><span class="p">:</span><span class="o">//</span><span class="n">github</span><span class="o">.</span><span class="n">com</span><span class="o">/</span><span class="n">apache</span><span class="o">/</span><span class="n">incubator</span><span class="o">-</span><span class="n">superset</span><span class="o">/</span> -<span class="n">cd</span> <span class="n">incubator</span><span class="o">-</span><span class="n">superset</span> -<span class="n">cp</span> <span class="n">contrib</span><span class="o">/</span><span class="n">docker</span><span class="o">/</span><span class="p">{</span><span class="n">docker</span><span class="o">-</span><span class="n">build</span><span class="o">.</span><span class="n">sh</span><span class="p">,</span><span class="n">docker</span><span class="o">-</span><span class="n">compose</span><span class="o">.</span><span class="n">yml</span><span class="p">,</span><span class="n">docker</ [...] -<span class="n">cp</span> <span class="n">contrib</span><span class="o">/</span><span class="n">docker</span><span class="o">/</span><span class="n">superset_config</span><span class="o">.</span><span class="n">py</span> <span class="n">superset</span><span class="o">/</span> -<span class="n">bash</span> <span class="o">-</span><span class="n">x</span> <span class="n">docker</span><span class="o">-</span><span class="n">build</span><span class="o">.</span><span class="n">sh</span> -<span class="n">docker</span><span class="o">-</span><span class="n">compose</span> <span class="n">up</span> <span class="o">-</span><span class="n">d</span> -<span class="n">docker</span><span class="o">-</span><span class="n">compose</span> <span class="n">exec</span> <span class="n">superset</span> <span class="n">bash</span> -<span class="n">bash</span> <span class="n">docker</span><span class="o">-</span><span class="n">init</span><span class="o">.</span><span class="n">sh</span> +<span class="n">cd</span> <span class="n">incubator</span><span class="o">-</span><span class="n">superset</span><span class="o">/</span><span class="n">contrib</span><span class="o">/</span><span class="n">docker</span> +<span class="c1"># prefix with SUPERSET_LOAD_EXAMPLES=yes to load examples:</span> +<span class="n">docker</span><span class="o">-</span><span class="n">compose</span> <span class="n">run</span> <span class="o">--</span><span class="n">rm</span> <span class="n">superset</span> <span class="o">./</span><span class="n">docker</span><span class="o">-</span><span class="n">init</span><span class="o">.</span><span class="n">sh</span> +<span class="c1"># you can run this command everytime you need to start superset now:</span> +<span class="n">docker</span><span class="o">-</span><span class="n">compose</span> <span class="n">up</span> </pre></div> </div> <p>After several minutes for superset initialization to finish, you can open a browser and view <cite>http://localhost:8088</cite> to start your journey.</p> +<p>From there, the container server will reload on modification of the superset python +and javascript source code. +Don’t forget to reload the page to take the new frontend into account though.</p> +<p>See also <a class="reference external" href="https://github.com/apache/incubator-superset/blob/master/CONTRIBUTING.md#webpack-dev-server";>CONTRIBUTING.md</a>, +for alternative way of serving the frontend.</p> +<p>It is also possible to run Superset in non-development mode: in the <cite>docker-compose.yml</cite> file remove +the volumes needed for development and change the variable <cite>SUPERSET_ENV</cite> to <cite>production</cite>.</p> <p>If you are attempting to build on a Mac and it exits with 137 you need to increase your docker resources. OSX instructions: <a class="reference external" href="https://docs.docker.com/docker-for-mac/#advanced";>https://docs.docker.com/docker-for-mac/#advanced</a> (Search for memory)</p> -<p>Or if you’re curious and want to install superset from bottom up, then go -ahead.</p> +<p>Or if you’re curious and want to install superset from bottom up, then go ahead.</p> </div> <div class="section" id="os-dependencies"> <h2>OS dependencies<a class="headerlink" href="#os-dependencies" title="Permalink to this headline">¶</a></h2> @@ -250,8 +258,8 @@ the required dependencies are installed:</p> <div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">sudo</span> <span class="n">apt</span><span class="o">-</span><span class="n">get</span> <span class="n">install</span> <span class="n">build</span><span class="o">-</span><span class="n">essential</span> <span class="n">libssl</span><span class="o">-</span><span class="n">dev</span> <span class="n">libffi</span><span class="o">-</span><span class="n">dev</span> <span class="n">python</spa [...] </pre></div> </div> -<p><strong>Ubuntu 16.04</strong> If you have python3.5 installed alongside with python2.7, as is default on <strong>Ubuntu 16.04 LTS</strong>, run this command also:</p> -<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">sudo</span> <span class="n">apt</span><span class="o">-</span><span class="n">get</span> <span class="n">install</span> <span class="n">build</span><span class="o">-</span><span class="n">essential</span> <span class="n">libssl</span><span class="o">-</span><span class="n">dev</span> <span class="n">libffi</span><span class="o">-</span><span class="n">dev</span> <span class="n">python3</sp [...] +<p><strong>Ubuntu 18.04</strong> If you have python3.6 installed alongside with python2.7, as is default on <strong>Ubuntu 18.04 LTS</strong>, run this command also:</p> +<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">sudo</span> <span class="n">apt</span><span class="o">-</span><span class="n">get</span> <span class="n">install</span> <span class="n">build</span><span class="o">-</span><span class="n">essential</span> <span class="n">libssl</span><span class="o">-</span><span class="n">dev</span> <span class="n">libffi</span><span class="o">-</span><span class="n">dev</span> <span class="n">python3</sp [...] </pre></div> </div> <p>otherwise build for <code class="docutils literal notranslate"><span class="pre">cryptography</span></code> fails.</p> @@ -263,7 +271,7 @@ that the required dependencies are installed:</p> </div> <p><strong>OSX</strong>, system python is not recommended. brew’s python also ships with pip</p> <div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">brew</span> <span class="n">install</span> <span class="n">pkg</span><span class="o">-</span><span class="n">config</span> <span class="n">libffi</span> <span class="n">openssl</span> <span class="n">python</span> -<span class="n">env</span> <span class="n">LDFLAGS</span><span class="o">=</span><span class="s2">"-L$(brew --prefix openssl)/lib"</span> <span class="n">CFLAGS</span><span class="o">=</span><span class="s2">"-I$(brew --prefix openssl)/include"</span> <span class="n">pip</span> <span class="n">install</span> <span class="n">cryptography</span><span class="o">==</span><span class="mf">1.9</span> +<span class="n">env</span> <span class="n">LDFLAGS</span><span class="o">=</span><span class="s2">"-L$(brew --prefix openssl)/lib"</span> <span class="n">CFLAGS</span><span class="o">=</span><span class="s2">"-I$(brew --prefix openssl)/include"</span> <span class="n">pip</span> <span class="n">install</span> <span class="n">cryptography</span><span class="o">==</span><span class="mf">2.4</span><span class="o">.</span><span class="mi">2</span> </pre></div> </div> <p><strong>Windows</strong> isn’t officially supported at this point, but if you want to @@ -277,16 +285,17 @@ attempt it, download <a class="reference external" href="https://bootstrap.pypa. </div> <div class="section" id="python-virtualenv"> <h2>Python virtualenv<a class="headerlink" href="#python-virtualenv" title="Permalink to this headline">¶</a></h2> -<p>It is recommended to install Superset inside a virtualenv. Python 3 already ships virtualenv, for -Python 2 you need to install it. If it’s packaged for your operating systems install it from there -otherwise you can install from pip:</p> +<p>It is recommended to install Superset inside a virtualenv. Python 3 already ships virtualenv. +But if it’s not installed in your environment for some reason, you can install it +via the package for your operating systems, otherwise you can install from pip:</p> <div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">pip</span> <span class="n">install</span> <span class="n">virtualenv</span> </pre></div> </div> <p>You can create and activate a virtualenv by:</p> -<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="c1"># virtualenv is shipped in Python 3 as pyvenv</span> -<span class="n">virtualenv</span> <span class="n">venv</span> -<span class="o">.</span> <span class="o">./</span><span class="n">venv</span><span class="o">/</span><span class="nb">bin</span><span class="o">/</span><span class="n">activate</span> +<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="c1"># virtualenv is shipped in Python 3.6+ as venv instead of pyvenv.</span> +<span class="c1"># See https://docs.python.org/3.6/library/venv.html</span> +<span class="n">python3</span> <span class="o">-</span><span class="n">m</span> <span class="n">venv</span> <span class="n">venv</span> +<span class="o">.</span> <span class="n">venv</span><span class="o">/</span><span class="nb">bin</span><span class="o">/</span><span class="n">activate</span> </pre></div> </div> <p>On windows the syntax for activating it is a bit different:</p> @@ -595,9 +604,6 @@ The role and warehouse can be omitted if defaults are defined for the user, i.e. <p>Make sure the user has privileges to access and use all required databases/schemas/tables/views/warehouses, as the Snowflake SQLAlchemy engine does not test for user rights during engine creation.</p> -<p><em>Note</em>: At the time of writing, there is a regression in the current stable version (1.1.2) of -snowflake-sqlalchemy package that causes problems when used with Superset. It is recommended to -use version 1.1.0 or try a newer version.</p> <p>See <a class="reference external" href="https://github.com/snowflakedb/snowflake-sqlalchemy";>Snowflake SQLAlchemy</a>.</p> </div> <div class="section" id="teradata"> @@ -727,6 +733,15 @@ metadata from your Druid cluster(s)</p> <li><code class="docutils literal notranslate"><span class="pre">CORS_OPTIONS</span></code>: options passed to Flask-CORS (<cite>documentation <http://flask-cors.corydolphin.com/en/latest/api.html#extension></cite>)</li> </ul> </div> +<div class="section" id="domain-sharding"> +<h2>DOMAIN SHARDING<a class="headerlink" href="#domain-sharding" title="Permalink to this headline">¶</a></h2> +<p>Chrome allows up to 6 open connections per domain at a time. When there are more +than 6 slices in dashboard, a lot of time fetch requests are queued up and wait for +next available socket. PR (<cite>#5039 <https://github.com/apache/incubator-superset/pull/5039></cite>) adds domain sharding to Superset, +and this feature will be enabled by configuration only (by default Superset +doesn’t allow cross-domain request).</p> +<p><a href="#id1"><span class="problematic" id="id2">*</span></a><code class="docutils literal notranslate"><span class="pre">SUPERSET_WEBSERVER_DOMAINS</span></code>: list of allowed hostnames for domain sharding feature. default <cite>None</cite></p> +</div> <div class="section" id="middleware"> <h2>MIDDLEWARE<a class="headerlink" href="#middleware" title="Permalink to this headline">¶</a></h2> <p>Superset allows you to add your own middleware. To add your own middleware, update the <code class="docutils literal notranslate"><span class="pre">ADDITIONAL_MIDDLEWARE</span></code> key in @@ -756,14 +771,12 @@ environment variable:</p> </pre></div> </div> </div> -<div class="section" id="sql-lab"> -<h2>SQL Lab<a class="headerlink" href="#sql-lab" title="Permalink to this headline">¶</a></h2> -<p>SQL Lab is a powerful SQL IDE that works with all SQLAlchemy compatible -databases. By default, queries are executed in the scope of a web -request so they -may eventually timeout as queries exceed the maximum duration of a web -request in your environment, whether it’d be a reverse proxy or the Superset -server itself.</p> +<div class="section" id="celery-tasks"> +<h2>Celery Tasks<a class="headerlink" href="#celery-tasks" title="Permalink to this headline">¶</a></h2> +<p>On large analytic databases, it’s common to run background jobs, reports +and/or queries that execute for minutes or hours. In certain cases, we need +to support long running tasks that execute beyond the typical web request’s +timeout (30-60 seconds).</p> <p>On large analytic databases, it’s common to run queries that execute for minutes or hours. To enable support for long running queries that @@ -783,17 +796,47 @@ results</li> have the same configuration.</p> <div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="k">class</span> <span class="nc">CeleryConfig</span><span class="p">(</span><span class="nb">object</span><span class="p">):</span> <span class="n">BROKER_URL</span> <span class="o">=</span> <span class="s1">'redis://localhost:6379/0'</span> - <span class="n">CELERY_IMPORTS</span> <span class="o">=</span> <span class="p">(</span><span class="s1">'superset.sql_lab'</span><span class="p">,</span> <span class="p">)</span> + <span class="n">CELERY_IMPORTS</span> <span class="o">=</span> <span class="p">(</span> + <span class="s1">'superset.sql_lab'</span><span class="p">,</span> + <span class="s1">'superset.tasks'</span><span class="p">,</span> + <span class="p">)</span> <span class="n">CELERY_RESULT_BACKEND</span> <span class="o">=</span> <span class="s1">'redis://localhost:6379/0'</span> - <span class="n">CELERY_ANNOTATIONS</span> <span class="o">=</span> <span class="p">{</span><span class="s1">'tasks.add'</span><span class="p">:</span> <span class="p">{</span><span class="s1">'rate_limit'</span><span class="p">:</span> <span class="s1">'10/s'</span><span class="p">}}</span> + <span class="n">CELERYD_LOG_LEVEL</span> <span class="o">=</span> <span class="s1">'DEBUG'</span> + <span class="n">CELERYD_PREFETCH_MULTIPLIER</span> <span class="o">=</span> <span class="mi">10</span> + <span class="n">CELERY_ACKS_LATE</span> <span class="o">=</span> <span class="bp">True</span> + <span class="n">CELERY_ANNOTATIONS</span> <span class="o">=</span> <span class="p">{</span> + <span class="s1">'sql_lab.get_sql_results'</span><span class="p">:</span> <span class="p">{</span> + <span class="s1">'rate_limit'</span><span class="p">:</span> <span class="s1">'100/s'</span><span class="p">,</span> + <span class="p">},</span> + <span class="s1">'email_reports.send'</span><span class="p">:</span> <span class="p">{</span> + <span class="s1">'rate_limit'</span><span class="p">:</span> <span class="s1">'1/s'</span><span class="p">,</span> + <span class="s1">'time_limit'</span><span class="p">:</span> <span class="mi">120</span><span class="p">,</span> + <span class="s1">'soft_time_limit'</span><span class="p">:</span> <span class="mi">150</span><span class="p">,</span> + <span class="s1">'ignore_result'</span><span class="p">:</span> <span class="bp">True</span><span class="p">,</span> + <span class="p">},</span> + <span class="p">}</span> + <span class="n">CELERYBEAT_SCHEDULE</span> <span class="o">=</span> <span class="p">{</span> + <span class="s1">'email_reports.schedule_hourly'</span><span class="p">:</span> <span class="p">{</span> + <span class="s1">'task'</span><span class="p">:</span> <span class="s1">'email_reports.schedule_hourly'</span><span class="p">,</span> + <span class="s1">'schedule'</span><span class="p">:</span> <span class="n">crontab</span><span class="p">(</span><span class="n">minute</span><span class="o">=</span><span class="mi">1</span><span class="p">,</span> <span class="n">hour</span><span class="o">=</span><span class="s1">'*'</span><span class="p">),</span> + <span class="p">},</span> + <span class="p">}</span> <span class="n">CELERY_CONFIG</span> <span class="o">=</span> <span class="n">CeleryConfig</span> </pre></div> </div> -<p>To start a Celery worker to leverage the configuration run:</p> -<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">celery</span> <span class="n">worker</span> <span class="o">--</span><span class="n">app</span><span class="o">=</span><span class="n">superset</span><span class="o">.</span><span class="n">sql_lab</span><span class="p">:</span><span class="n">celery_app</span> <span class="o">--</span><span class="n">pool</span><span class="o">=</span><span class="n">gevent</span> <span class="o">-</span> [...] +<ul> +<li><p class="first">To start a Celery worker to leverage the configuration run:</p> +<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">celery</span> <span class="n">worker</span> <span class="o">--</span><span class="n">app</span><span class="o">=</span><span class="n">superset</span><span class="o">.</span><span class="n">tasks</span><span class="o">.</span><span class="n">celery_app</span><span class="p">:</span><span class="n">app</span> <span class="o">--</span><span class="n">pool</span><span class="o">=</span><span [...] </pre></div> </div> +</li> +<li><p class="first">To start a job which schedules periodic background jobs, run</p> +<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">celery</span> <span class="n">beat</span> <span class="o">--</span><span class="n">app</span><span class="o">=</span><span class="n">superset</span><span class="o">.</span><span class="n">tasks</span><span class="o">.</span><span class="n">celery_app</span><span class="p">:</span><span class="n">app</span> +</pre></div> +</div> +</li> +</ul> <p>To setup a result backend, you need to pass an instance of a derivative of <code class="docutils literal notranslate"><span class="pre">werkzeug.contrib.cache.BaseCache</span></code> to the <code class="docutils literal notranslate"><span class="pre">RESULTS_BACKEND</span></code> configuration key in your <code class="docutils literal notranslate"><span class="pre">superset_config.py</span></code>. It’s possible to use @@ -813,11 +856,70 @@ look something like:</p> <span class="n">host</span><span class="o">=</span><span class="s1">'localhost'</span><span class="p">,</span> <span class="n">port</span><span class="o">=</span><span class="mi">6379</span><span class="p">,</span> <span class="n">key_prefix</span><span class="o">=</span><span class="s1">'superset_results'</span><span class="p">)</span> </pre></div> </div> -<p>Note that it’s important that all the worker nodes and web servers in +<p><strong>Important notes</strong></p> +<ul class="simple"> +<li>It is important that all the worker nodes and web servers in the Superset cluster share a common metadata database. This means that SQLite will not work in this context since it has limited support for concurrency and -typically lives on the local file system.</p> +typically lives on the local file system.</li> +<li>There should only be one instance of <code class="docutils literal notranslate"><span class="pre">celery</span> <span class="pre">beat</span></code> running in your +entire setup. If not, background jobs can get scheduled multiple times +resulting in weird behaviors like duplicate delivery of reports, +higher than expected load / traffic etc.</li> +</ul> +</div> +<div class="section" id="email-reports"> +<h2>Email Reports<a class="headerlink" href="#email-reports" title="Permalink to this headline">¶</a></h2> +<p>Email reports allow users to schedule email reports for</p> +<ul class="simple"> +<li>slice and dashboard visualization (Attachment or inline)</li> +<li>slice data (CSV attachment on inline table)</li> +</ul> +<p>Schedules are defined in crontab format and each schedule +can have a list of recipients (all of them can receive a single mail, +or separate mails). For audit purposes, all outgoing mails can have a +mandatory bcc.</p> +<p><strong>Requirements</strong></p> +<ul> +<li><p class="first">A selenium compatible driver & headless browser</p> +<ul class="simple"> +<li><a class="reference external" href="https://github.com/mozilla/geckodriver";>geckodriver</a> and Firefox is preferred</li> +<li><a class="reference external" href="http://chromedriver.chromium.org/";>chromedriver</a> is a good option too</li> +</ul> +</li> +<li><p class="first">Run <cite>celery worker</cite> and <cite>celery beat</cite> as follows</p> +<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">celery</span> <span class="n">worker</span> <span class="o">--</span><span class="n">app</span><span class="o">=</span><span class="n">superset</span><span class="o">.</span><span class="n">tasks</span><span class="o">.</span><span class="n">celery_app</span><span class="p">:</span><span class="n">app</span> <span class="o">--</span><span class="n">pool</span><span class="o">=</span><span [...] +<span class="n">celery</span> <span class="n">beat</span> <span class="o">--</span><span class="n">app</span><span class="o">=</span><span class="n">superset</span><span class="o">.</span><span class="n">tasks</span><span class="o">.</span><span class="n">celery_app</span><span class="p">:</span><span class="n">app</span> +</pre></div> +</div> +</li> +</ul> +<p><strong>Important notes</strong></p> +<ul> +<li><p class="first">Be mindful of the concurrency setting for celery (using <code class="docutils literal notranslate"><span class="pre">-c</span> <span class="pre">4</span></code>). +Selenium/webdriver instances can consume a lot of CPU / memory on your servers.</p> +</li> +<li><p class="first">In some cases, if you notice a lot of leaked <code class="docutils literal notranslate"><span class="pre">geckodriver</span></code> processes, try running +your celery processes with</p> +<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">celery</span> <span class="n">worker</span> <span class="o">--</span><span class="n">pool</span><span class="o">=</span><span class="n">prefork</span> <span class="o">--</span><span class="nb">max</span><span class="o">-</span><span class="n">tasks</span><span class="o">-</span><span class="n">per</span><span class="o">-</span><span class="n">child</span><span class="o">=</span><span class [...] +</pre></div> +</div> +</li> +<li><p class="first">It is recommended to run separate workers for <code class="docutils literal notranslate"><span class="pre">sql_lab</span></code> and +<code class="docutils literal notranslate"><span class="pre">email_reports</span></code> tasks. Can be done by using <code class="docutils literal notranslate"><span class="pre">queue</span></code> field in <code class="docutils literal notranslate"><span class="pre">CELERY_ANNOTATIONS</span></code></p> +</li> +</ul> +</div> +<div class="section" id="sql-lab"> +<h2>SQL Lab<a class="headerlink" href="#sql-lab" title="Permalink to this headline">¶</a></h2> +<p>SQL Lab is a powerful SQL IDE that works with all SQLAlchemy compatible +databases. By default, queries are executed in the scope of a web +request so they may eventually timeout as queries exceed the maximum duration of a web +request in your environment, whether it’d be a reverse proxy or the Superset +server itself. In such cases, it is preferred to use <code class="docutils literal notranslate"><span class="pre">celery</span></code> to run the queries +in the background. Please follow the examples/notes mentioned above to get your +celery setup working.</p> <p>Also note that SQL Lab supports Jinja templating in queries and that it’s possible to overload the default Jinja context in your environment by defining the @@ -828,13 +930,16 @@ in this dictionary are made available for users to use in their SQL.</p> <span class="p">}</span> </pre></div> </div> +</div> +<div class="section" id="celery-flower"> +<h2>Celery Flower<a class="headerlink" href="#celery-flower" title="Permalink to this headline">¶</a></h2> <p>Flower is a web based tool for monitoring the Celery cluster which you can install from pip:</p> <div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">pip</span> <span class="n">install</span> <span class="n">flower</span> </pre></div> </div> <p>and run via:</p> -<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">celery</span> <span class="n">flower</span> <span class="o">--</span><span class="n">app</span><span class="o">=</span><span class="n">superset</span><span class="o">.</span><span class="n">sql_lab</span><span class="p">:</span><span class="n">celery_app</span> +<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">celery</span> <span class="n">flower</span> <span class="o">--</span><span class="n">app</span><span class="o">=</span><span class="n">superset</span><span class="o">.</span><span class="n">tasks</span><span class="o">.</span><span class="n">celery_app</span><span class="p">:</span><span class="n">app</span> </pre></div> </div> </div> diff --git a/searchindex.js b/searchindex.js index b8f58fa..41f6ba4 100644 --- a/searchindex.js +++ b/searchindex.js @@ -1 +1 @@ -Search.setIndex({docnames:["druid","faq","gallery","import_export_datasources","index","installation","misc","security","sqllab","tutorial","videos","visualization"],envversion:{"sphinx.domains.c":1,"sphinx.domains.changeset":1,"sphinx.domains.cpp":1,"sphinx.domains.javascript":1,"sphinx.domains.math":1,"sphinx.domains.python":1,"sphinx.domains.rst":1,"sphinx.domains.std":1,"sphinx.ext.viewcode":1,sphinx:55},filenames:["druid.rst","faq.rst","gallery.rst","import_export_datasources.rst"," [...] \ No newline at end of file +Search.setIndex({docnames:["druid","faq","gallery","import_export_datasources","index","installation","misc","security","sqllab","tutorial","videos","visualization"],envversion:{"sphinx.domains.c":1,"sphinx.domains.changeset":1,"sphinx.domains.cpp":1,"sphinx.domains.javascript":1,"sphinx.domains.math":1,"sphinx.domains.python":1,"sphinx.domains.rst":1,"sphinx.domains.std":1,"sphinx.ext.viewcode":1,sphinx:55},filenames:["druid.rst","faq.rst","gallery.rst","import_export_datasources.rst"," [...] \ No newline at end of file diff --git a/sqllab.html b/sqllab.html index cf993ad..0b8b30d 100644 --- a/sqllab.html +++ b/sqllab.html @@ -175,7 +175,7 @@ visualization capabilities</li> <li>Browse database metadata: tables, columns, indexes, partitions</li> <li>Support for long-running queries<ul> -<li>uses the <a class="reference external" href="http://www.python.org/";>Celery distributed queue</a> +<li>uses the <a class="reference external" href="http://www.celeryproject.org/";>Celery distributed queue</a> to dispatch query handling to workers</li> <li>supports defining a “results backend” to persist query results</li> </ul> @@ -226,7 +226,16 @@ jinja context as in <code class="docutils literal notranslate"><span class="pre" <dl class="function"> <dt id="superset.jinja_context.url_param"> <code class="descclassname">superset.jinja_context.</code><code class="descname">url_param</code><span class="sig-paren">(</span><em>param</em>, <em>default=None</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/superset/jinja_context.html#url_param"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#superset.jinja_context.url_param" title="Permalink to this definition">¶</a></dt> -<dd><p>Get a url or post data parameter</p> +<dd><p>Read a url or post parameter and use it in your SQL Lab query</p> +<p>When in SQL Lab, it’s possible to add arbitrary URL “query string” +parameters, and use those in your SQL code. For instance you can +alter your url and add <cite>?foo=bar</cite>, as in +<cite>{domain}/superset/sqllab?foo=bar</cite>. Then if your query is something like +SELECT * FROM foo = ‘{{ url_param(‘foo’) }}’, it will be parsed at +runtime and replaced by the value in the URL.</p> +<p>As you create a visualization form this SQL Lab query, you can pass +parameters in the explore view as well as from the dashboard, and +it should carry through to your queries.</p> <table class="docutils field-list" frame="void" rules="none"> <col class="field-name" /> <col class="field-body" /> @@ -259,7 +268,7 @@ column doesn’t match the one in the select statement</li> <dt>Usage example:</dt> <dd>SELECT action, count(*) as times FROM logs -WHERE action in ( {{ “’” + “’,’”.join(filter_values(‘action_type’)) + “’” ) +WHERE action in ( {{ “’” + “’,’”.join(filter_values(‘action_type’)) + “’” }} ) GROUP BY 1</dd> </dl> <table class="docutils field-list" frame="void" rules="none"> diff --git a/visualization.html b/visualization.html index b43ce72..0387f2f 100644 --- a/visualization.html +++ b/visualization.html @@ -1255,6 +1255,163 @@ The second part is a string of up to three alphanumeric characters, which is usu </tbody> </table> <ul class="simple"> +<li>Japan</li> +</ul> +<table border="1" class="docutils"> +<colgroup> +<col width="30%" /> +<col width="70%" /> +</colgroup> +<thead valign="bottom"> +<tr class="row-odd"><th class="head">ISO</th> +<th class="head">Name of region</th> +</tr> +</thead> +<tbody valign="top"> +<tr class="row-even"><td>JP-01</td> +<td>Hokkaido</td> +</tr> +<tr class="row-odd"><td>JP-02</td> +<td>Aomori</td> +</tr> +<tr class="row-even"><td>JP-03</td> +<td>Iwate</td> +</tr> +<tr class="row-odd"><td>JP-04</td> +<td>Miyagi</td> +</tr> +<tr class="row-even"><td>JP-05</td> +<td>Akita</td> +</tr> +<tr class="row-odd"><td>JP-06</td> +<td>Yamagata</td> +</tr> +<tr class="row-even"><td>JP-07</td> +<td>Fukushima</td> +</tr> +<tr class="row-odd"><td>JP-08</td> +<td>Ibaraki</td> +</tr> +<tr class="row-even"><td>JP-09</td> +<td>Tochigi</td> +</tr> +<tr class="row-odd"><td>JP-10</td> +<td>Gunma</td> +</tr> +<tr class="row-even"><td>JP-11</td> +<td>Saitama</td> +</tr> +<tr class="row-odd"><td>JP-12</td> +<td>Chiba</td> +</tr> +<tr class="row-even"><td>JP-13</td> +<td>Tokyo</td> +</tr> +<tr class="row-odd"><td>JP-14</td> +<td>Kanagawa</td> +</tr> +<tr class="row-even"><td>JP-15</td> +<td>Niigata</td> +</tr> +<tr class="row-odd"><td>JP-16</td> +<td>Toyama</td> +</tr> +<tr class="row-even"><td>JP-17</td> +<td>Ishikawa</td> +</tr> +<tr class="row-odd"><td>JP-18</td> +<td>Fukui</td> +</tr> +<tr class="row-even"><td>JP-19</td> +<td>Yamanashi</td> +</tr> +<tr class="row-odd"><td>JP-20</td> +<td>Nagano</td> +</tr> +<tr class="row-even"><td>JP-21</td> +<td>Gifu</td> +</tr> +<tr class="row-odd"><td>JP-22</td> +<td>Shizuoka</td> +</tr> +<tr class="row-even"><td>JP-23</td> +<td>Aichi</td> +</tr> +<tr class="row-odd"><td>JP-24</td> +<td>Mie</td> +</tr> +<tr class="row-even"><td>JP-25</td> +<td>Shiga</td> +</tr> +<tr class="row-odd"><td>JP-26</td> +<td>Kyoto</td> +</tr> +<tr class="row-even"><td>JP-27</td> +<td>Osaka</td> +</tr> +<tr class="row-odd"><td>JP-28</td> +<td>Hyogo</td> +</tr> +<tr class="row-even"><td>JP-29</td> +<td>Nara</td> +</tr> +<tr class="row-odd"><td>JP-30</td> +<td>Wakayama</td> +</tr> +<tr class="row-even"><td>JP-31</td> +<td>Tottori</td> +</tr> +<tr class="row-odd"><td>JP-32</td> +<td>Shimane</td> +</tr> +<tr class="row-even"><td>JP-33</td> +<td>Okayama</td> +</tr> +<tr class="row-odd"><td>JP-34</td> +<td>Hiroshima</td> +</tr> +<tr class="row-even"><td>JP-35</td> +<td>Yamaguchi</td> +</tr> +<tr class="row-odd"><td>JP-36</td> +<td>Tokushima</td> +</tr> +<tr class="row-even"><td>JP-37</td> +<td>Kagawa</td> +</tr> +<tr class="row-odd"><td>JP-38</td> +<td>Ehime</td> +</tr> +<tr class="row-even"><td>JP-39</td> +<td>Kochi</td> +</tr> +<tr class="row-odd"><td>JP-40</td> +<td>Fukuoka</td> +</tr> +<tr class="row-even"><td>JP-41</td> +<td>Saga</td> +</tr> +<tr class="row-odd"><td>JP-42</td> +<td>Nagasaki</td> +</tr> +<tr class="row-even"><td>JP-43</td> +<td>Kumamoto</td> +</tr> +<tr class="row-odd"><td>JP-44</td> +<td>Oita</td> +</tr> +<tr class="row-even"><td>JP-45</td> +<td>Miyazaki</td> +</tr> +<tr class="row-odd"><td>JP-46</td> +<td>Kagoshima</td> +</tr> +<tr class="row-even"><td>JP-47</td> +<td>Okinawa</td> +</tr> +</tbody> +</table> +<ul class="simple"> <li>Morocco</li> </ul> <table border="1" class="docutils"> @@ -2818,9 +2975,9 @@ You can get this file on this site: <a class="reference external" href="http://w It’s important because it’s a norm for mapping your data with geojson file</li> <li>You need to convert shapefile to geojson file. This action can make with ogr2ogr tools: <a class="reference external" href="http://www.gdal.org/ogr2ogr.html";>http://www.gdal.org/ogr2ogr.html</a></li> -<li>Put your geojson file in next folder : superset/assets/visualizations/countries with the next name : nameofyourcountries.geojson</li> +<li>Put your geojson file in next folder : superset/assets/src/visualizations/CountryMap/countries with the next name : nameofyourcountries.geojson</li> <li>You can to reduce size of geojson file on this site: <a class="reference external" href="http://mapshaper.org/";>http://mapshaper.org/</a></li> -<li>Go in file superset/assets/javascripts/explorev2/stores/controls.jsx</li> +<li>Go in file superset/assets/src/explore/controls.jsx</li> <li>Add your country in component ‘select_country’ Example :</li> </ol> @@ -2836,6 +2993,7 @@ Example :</li> <span class="s1">'France'</span><span class="p">,</span> <span class="s1">'Germany'</span><span class="p">,</span> <span class="s1">'Italy'</span><span class="p">,</span> + <span class="s1">'Japan'</span><span class="p">,</span> <span class="s1">'Morocco'</span><span class="p">,</span> <span class="s1">'Netherlands'</span><span class="p">,</span> <span class="s1">'Russia'</span><span class="p">,</span>
