This is an automated email from the ASF dual-hosted git repository.
xhsun pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git
The following commit(s) were added to refs/heads/master by this push:
new 50aaaf9 [TE] added ThirdEye configuration documents (#4429)
50aaaf9 is described below
commit 50aaaf93c6ac51cf6b5faf86e586a5fc47e8e184
Author: Xiaohui Sun <[email protected]>
AuthorDate: Fri Jul 12 13:10:56 2019 -0700
[TE] added ThirdEye configuration documents (#4429)
* [TE] added ThirdEye configuration documents
* [TE] update documents on configuration
* [TE] update documents on configuration
---
thirdeye/docs/configuration.rst | 210 ++++++++++++++++++++++++++++++++++++++++
thirdeye/docs/index.rst | 2 +-
thirdeye/docs/production.rst | 27 ------
thirdeye/docs/quick_start.rst | 8 +-
4 files changed, 216 insertions(+), 31 deletions(-)
diff --git a/thirdeye/docs/configuration.rst b/thirdeye/docs/configuration.rst
new file mode 100644
index 0000000..56decf0
--- /dev/null
+++ b/thirdeye/docs/configuration.rst
@@ -0,0 +1,210 @@
+..
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+..
+.. http://www.apache.org/licenses/LICENSE-2.0
+..
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+..
+
+.. _configurations:
+
+Configuration
+=======================
+
+Overview
+---------
+ThirdEye could be deployed on single machine or deployed in clusters.
+
+- Dashboard servers are used to host web applications.
+
+The ``org.apache.pinot.thirdeye.dashboard.ThirdEyeDashboardApplication`` class
is the entry point.
+
+:ref:`dashboard.yml` and :ref:`rca.yml` are used to configure dashboard
servers.
+
+- Backend servers are used to schedule tasks or run the tasks.
+
+The ``org.apache.pinot.thirdeye.anomaly.ThirdEyeAnomalyApplication`` class is
the entry point.
+
+:ref:`detector.yml`, :ref:`persistence.yml` and :ref:`data-sources-config.yml`
are used to configure backend servers.
+
+.. image::
https://user-images.githubusercontent.com/44730481/61093367-fd684300-a3fe-11e9-943e-d20ba9651528.png
+ :width: 400
+
+.. _detector.yml:
+
+detector.yml
+--------------
+
+ThirdEye uses this file to configure the backend server.
+
+You can deploy ThirdEye to multiple nodes with specific modules enabled in
each node.
+
+Here are a list of modules you can configure in this file:
+
+- autoload: Load Pinot metrics metadata automatically.
+- holidayEventsLoader: Load holiday events from Google Calendar.
+- monitor: Used to do clean up tasks. By default ThirdEye will delete -
detection tasks that are older than 30 days and alert tasks that are older than
10 days.
+- worker: Handles the actual tasks to do anomaly detection or alerting. You
can deploy multiple workers to share the load.
+- detectionPipeline: Scheduler to generate detection tasks.
+- detectionAlert: Scheduler to generate alert tasks.
+
+.. image::
https://user-images.githubusercontent.com/44730481/61093449-6d76c900-a3ff-11e9-85e6-01201ea71eeb.png
+ :width: 500
+
+To enable one module, you can change the module's value to "true".
+
+For example, below configures a node with worker and scheduler enabled.
+
+.. code-block:: yaml
+
+ holidayEventsLoader: false
+ monitor: false
+ pinotProxy: false
+ worker: true
+ detectionPipeline: true
+ detectionAlert: true
+
+To have the minimum system running you need to enable "worker", "monitor",
"detectionPipeline" and "detectionAlert".
+
+Besides the module configuration you can configure the other followings in
this file:
+
+- SMTP configuration: Configure SMTP server which is used to send alert mail.
+- Log configuration: SLF4J configurations.
+- Server ports: Endpoint ports for backend servers.
+- Swagger configuration.
+- PhantomJSPath: PhantomJS is used to generate anomaly metrics screenshots
which are attached in alert mail.
+
+.. _persistence.yml:
+
+persistence.yml
+------------------
+
+ThirdEye uses MySQL to store all the metadata. This file is used to configure
MySQL database instance.
+
+databaseConfiguration: url: user: password: driver: com.mysql.jdbc.Driver
+Here is an example:
+
+databaseConfiguration: url: jdbc:mysql:///thirdeye?autoReconnect=true user:
te_dev password: xxxxx driver: com.mysql.jdbc.Driver
+
+
+.. _data-sources-config.yml:
+
+data-sources-config.yml
+------------------------
+
+ThirdEye doesn't store the actual metrics but will pull the metrics using data
source loaders. This file controls the metrics data sources.
+
+Here is an example used in ThirdEye production which connects to two data
sources: PinotThirdEyeDataSource and SqlThirdEyeDataSource.
+
+Please note ThirdEye support MySQL data source, and this configuration is
different with persistence.yml.
+
+.. code-block:: yaml
+
+ dataSourceConfigs:
+ - className:
org.apache.pinot.thirdeye.datasource.pinot.PinotThirdEyeDataSource
+ properties:
+ zookeeperUrl: '<zookeeperurl>'
+ clusterName: '<clustername>'
+ controllerConnectionScheme: 'https'
+ controllerHost: '<hostname>'
+ controllerPort: <port>
+ cacheLoaderClassName:
org.apache.pinot.thirdeye.datasource.pinot.PinotD2ResponseCacheLoader
+ metadataSourceConfigs:
+ - className:
org.apache.pinot.thirdeye.auto.onboard.AutoOnboardPinotMetadataSource
+ - className:
org.apache.pinot.thirdeye.datasource.sql.SqlThirdEyeDataSource
+ properties:
+ MySQL:
+ - db:
+ te: 'jdbc:mysql://<mysqlurl>/thirdeye?autoReconnect=true'
+ user: 'thirdeye'
+ password: '<password>'
+
+For more examples on datasource configurations please check :ref:`alert-setup`.
+
+.. _dashboard.yml:
+
+dashboard.yml
+------------------
+
+Controls settings relate to web application servers. The followings are
configured here:
+
+- LDAP authentication. To enable LDAP authentication, change "authEnabled" to
"true".
+
+.. code-block:: yaml
+
+ authConfig:
+ authEnabled: true
+ authKey: <authentication_key>
+ ldapUrl: <ldap_url>
+ domainSuffix:
+ - linkedin.biz
+ cacheTTL: 3600
+ cookieTTL: 604800
+ adminUsers:
+ - user1
+ - user2
+
+- Root cause analysis (RCA) configuration: Control thread pool size for RCA
pipelines. Default is 5.
+- Dashboard host and endpoints configuration.
+- Swagger configuration.
+
+.. _rca.yml:
+
+rca.yml
+------------------
+This configures the RCA pipelines, which is used to either do metrics analysis
or loads events from different systems.
+
+These pipelines are called online and not pre-loaded.
+
+Each pipeline derives from org.apache.pinot.thirdeye.rootcause.Pipeline class,
and has "inputNames", "outputName", "className" and "properties". One pipeline
can take another pipeline's output as input and it is a DAG.
+
+The "className" is used to create instances using reflection.
+
+
+FAQ
+-----------
+
+**How to add a new application?**
+
+Application is a group of users. ThirdEye can show anomalies grouped by
applications.
+
+There are two ways to add a new application:
+
+1.Send a post message
+
+curl -vX POST '.../thirdeye/entity?entityType=APPLICATION' -H "Content-Type:
application/json" -H 'Cookie: te_auth=[REPLACE_TOKEN_HERE]' --data-binary
@application.json
+
+Example payload could be:
+
+.. code-block:: json
+
+ {
+ "application": "replace-application-name",
+ "recipients": "[email protected]"
+ }
+
+2.From admin page
+
+a. Navigate to admin page: http://localhost:1426/thirdeye-admin
+b. Select an existing application from "Entity Editor".
+
+.. image::
https://user-images.githubusercontent.com/44730481/61093646-61d7d200-a400-11e9-8517-0b46bd33fe2a.png
+ :width: 500
+
+c. Put the application and recipients in the editor, and leave id, version etc
unspecified.
+
+.. image::
https://user-images.githubusercontent.com/44730481/61093659-6c926700-a400-11e9-8690-6a1742671e5e.png
+ :width: 500
+
+d. Click "load to editor" then click "submit".
diff --git a/thirdeye/docs/index.rst b/thirdeye/docs/index.rst
index 313a6a3..0446113 100644
--- a/thirdeye/docs/index.rst
+++ b/thirdeye/docs/index.rst
@@ -25,6 +25,6 @@ ThirdEye
:maxdepth: 2
introduction
+ configuration
datasources
- production
alert_setup
\ No newline at end of file
diff --git a/thirdeye/docs/production.rst b/thirdeye/docs/production.rst
deleted file mode 100644
index cecbb91..0000000
--- a/thirdeye/docs/production.rst
+++ /dev/null
@@ -1,27 +0,0 @@
-..
-.. Licensed to the Apache Software Foundation (ASF) under one
-.. or more contributor license agreements. See the NOTICE file
-.. distributed with this work for additional information
-.. regarding copyright ownership. The ASF licenses this file
-.. to you under the Apache License, Version 2.0 (the
-.. "License"); you may not use this file except in compliance
-.. with the License. You may obtain a copy of the License at
-..
-.. http://www.apache.org/licenses/LICENSE-2.0
-..
-.. Unless required by applicable law or agreed to in writing,
-.. software distributed under the License is distributed on an
-.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-.. KIND, either express or implied. See the License for the
-.. specific language governing permissions and limitations
-.. under the License.
-..
-
-.. _production:
-
-Production Settings
-====================
-
-ThirdEye relies on a central meta data store to coordinate its workers and
frontend processes. The first step towards moving ThirdEye into production
should therefore be the setup of a dedicated (MySQL) database instance. You can
use the ``thirdeye-pinot/src/main/resources/schema/create-schema.sql`` script
to create your tables. Then, update the
``thirdeye-pinot/config/persistence.yml`` file with path and credentials. Once
you have a dedicated database instance, you can run backend and f [...]
-
-The next step could be the configuration of the holiday auto-loader. The
holiday auto loader connects to the Google Calendar API. Once you obtain an API
token, place it in ``thirdeye-pinot/config/holiday-loader-key.json`` and in
``thirdeye-pinot/config/detector.yml`` set holidayEventsLoader: true. Once the
backend worker is restarted, it will periodically update the local cache of
holiday events for ThirdEye's detection and Root-Cause Analysis components.
diff --git a/thirdeye/docs/quick_start.rst b/thirdeye/docs/quick_start.rst
index 845bf2f..58cb623 100644
--- a/thirdeye/docs/quick_start.rst
+++ b/thirdeye/docs/quick_start.rst
@@ -27,7 +27,7 @@ ThirdEye supports an interactive demo mode for the analysis
dashboard. These ste
**1: Prerequisites**
-You'll need Java 8+, Maven 3+, and NPM 3.10+
+You'll need Java 8+, Maven 3.6+, and NPM 3.10+
**2: Build ThirdEye**
@@ -43,7 +43,7 @@ You'll need Java 8+, Maven 3+, and NPM 3.10+
Note: The build of thirdeye-frontend may take several minutes
-**3: Run ThirdEye frontend**
+**3: Run frontend**
.. code-block:: bash
@@ -60,6 +60,7 @@ Note: ThirdEye in demo mode will accept any credentials
You will find the root cause analysis page like below:
.. image::
https://user-images.githubusercontent.com/44730481/59537777-674b0680-8eac-11e9-8c8b-314bf64d5914.png
+ :width: 500
**5: Have fun**
@@ -106,10 +107,11 @@ Click ``Run Preview`` button, the anomalies will be
detected. Then you can play
You will find the alert preview page like below:
.. image::
https://user-images.githubusercontent.com/44730481/59538138-9f9f1480-8ead-11e9-9b2f-bbb3475f27ed.png
+ :width: 500
If you want to preview the hourly data, just change ``dataset: H2.H2.daily``
to ``dataset: H2.H2.hourly``, and rerun the preview.
-If you want to setup and run real anomaly detection rules, you need to see
:ref:`production` and :ref:`alert-setup`.
+If you want to setup in production, you need to see :ref:`configurations` and
:ref:`alert-setup`.
**7: Shutdown**
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
