Repository: kylin Updated Branches: refs/heads/document 8e639ce01 -> 8989d052c
update faq and advanced_settings Project: http://git-wip-us.apache.org/repos/asf/kylin/repo Commit: http://git-wip-us.apache.org/repos/asf/kylin/commit/8989d052 Tree: http://git-wip-us.apache.org/repos/asf/kylin/tree/8989d052 Diff: http://git-wip-us.apache.org/repos/asf/kylin/diff/8989d052 Branch: refs/heads/document Commit: 8989d052ce79dfa2e9f2afce3c31062844aaebe5 Parents: 8e639ce Author: shaofengshi <shaofeng...@apache.org> Authored: Tue Oct 18 10:30:12 2016 +0800 Committer: shaofengshi <shaofeng...@apache.org> Committed: Tue Oct 18 10:30:12 2016 +0800 ---------------------------------------------------------------------- website/_docs15/gettingstarted/faq.md | 48 ++++++++++++++++-------- website/_docs15/install/advance_settings.md | 12 ++++++ 2 files changed, 45 insertions(+), 15 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/kylin/blob/8989d052/website/_docs15/gettingstarted/faq.md ---------------------------------------------------------------------- diff --git a/website/_docs15/gettingstarted/faq.md b/website/_docs15/gettingstarted/faq.md index 4306813..e070159 100644 --- a/website/_docs15/gettingstarted/faq.md +++ b/website/_docs15/gettingstarted/faq.md @@ -6,21 +6,41 @@ permalink: /docs15/gettingstarted/faq.html since: v0.6.x --- -#### Build cube failed due to "error check status" +#### 1. "bin/find-hive-dependency.sh" can locate hive/hcat jars in local, but Kylin reports error like "java.lang.NoClassDefFoundError: org/apache/hive/hcatalog/mapreduce/HCatInputFormat" + + * Kylin need many dependent jars (hadoop/hive/hcat/hbase/kafka) on classpath to work, but Kylin doesn't ship them. It will seek these jars from your local machine by running commands like `hbase classpath`, `hive -e set` etc. The founded jars' path will be appended to the environment variable *HBASE_CLASSPATH* (Kylin uses `hbase` shell command to start up, which will read this). But in some Hadoop distribution (like EMR 5.0), the `hbase` shell doesn't keep the origin `HBASE_CLASSPATH` value, that causes the "NoClassDefFoundError". + + * To fix this, find the hbase shell script (in hbase/bin folder), and search *HBASE_CLASSPATH*, check whether it overwrite the value like : + + {% highlight Groff markup %} + export HBASE_CLASSPATH=$HADOOP_CONF:$HADOOP_HOME/*:$HADOOP_HOME/lib/*:$ZOOKEEPER_HOME/*:$ZOOKEEPER_HOME/lib/* + {% endhighlight %} + + * If true, change it to keep the origin value like: + + {% highlight Groff markup %} + export HBASE_CLASSPATH=$HADOOP_CONF:$HADOOP_HOME/*:$HADOOP_HOME/lib/*:$ZOOKEEPER_HOME/*:$ZOOKEEPER_HOME/lib/*:$HBASE_CLASSPATH + {% endhighlight %} + +#### 2. Get "java.lang.IllegalArgumentException: Too high cardinality is not suitable for dictionary -- cardinality: 5220674" in "Build Dimension Dictionary" step + + * Kylin uses "Dictionary" encoding to encode/decode the dimension values (check [this blog](/blog/2015/08/13/kylin-dictionary/)); Usually a dimension's cardinality is less than millions, so the "Dict" encoding is good to use. As dictionary need be persisted and loaded into memory, if a dimension's cardinality is very high, the memory footprint will be tremendous, so Kylin add a check on this. If you see this error, suggest to identify the UHC dimension first and then re-evaluate the design (whether need to make that as dimension?). If must keep it, you can by-pass this error with couple ways: 1) change to use other encoding (like `fixed_length`, `integer`) 2) or set a bigger value for `kylin.dictionary.max.cardinality` in `conf/kylin.properties`. + +#### 3. Build cube failed due to "error check status" * Check if `kylin.log` contains *yarn.resourcemanager.webapp.address:http://0.0.0.0:8088* and *java.net.ConnectException: Connection refused* * If yes, then the problem is the address of resource manager was not available in yarn-site.xml * A workaround is update `kylin.properties`, set `kylin.job.yarn.app.rest.check.status.url=http://YOUR_RM_NODE:8088/ws/v1/cluster/apps/${job_id}?anonymous=true` -#### HBase cannot get master address from ZooKeeper on Hortonworks Sandbox +#### 4. HBase cannot get master address from ZooKeeper on Hortonworks Sandbox * By default hortonworks disables hbase, you'll have to start hbase in ambari homepage first. -#### Map Reduce Job information cannot display on Hortonworks Sandbox +#### 5. Map Reduce Job information cannot display on Hortonworks Sandbox * Check out [https://github.com/KylinOLAP/Kylin/issues/40](https://github.com/KylinOLAP/Kylin/issues/40) -#### Install Kylin on CDH 5.2 or Hadoop 2.5.x +#### 6. How to Install Kylin on CDH 5.2 or Hadoop 2.5.x * Check out discussion: [https://groups.google.com/forum/?utm_medium=email&utm_source=footer#!msg/kylin-olap/X0GZfsX1jLc/nzs6xAhNpLkJ](https://groups.google.com/forum/?utm_medium=email&utm_source=footer#!msg/kylin-olap/X0GZfsX1jLc/nzs6xAhNpLkJ) @@ -34,19 +54,17 @@ since: v0.6.x My Cluster is running on Cloudera Distribution CDH 5.2.0. {% endhighlight %} -#### Unable to load a big cube as HTable, with java.lang.OutOfMemoryError: unable to create new native thread - * HBase (as of writing) allocates one thread per region when bulk loading a HTable. Try reduce the number of regions of your cube by setting its "capacity" to "MEDIUM" or "LARGE". Also tweaks OS & JVM can allow more threads, for example see [this article](http://blog.egilh.com/2006/06/2811aspx.html). -#### SUM(field) returns a negtive result while all the numbers in this field are > 0 +#### 7. SUM(field) returns a negtive result while all the numbers in this field are > 0 * If a column is declared as integer in Hive, the SQL engine (calcite) will use column's type (integer) as the data type for "SUM(field)", while the aggregated value on this field may exceed the scope of integer; in that case the cast will cause a negtive value be returned; The workround is, alter that column's type to BIGINT in hive, and then sync the table schema to Kylin (the cube doesn't need rebuild); Keep in mind that, always declare as BIGINT in hive for an integer column which would be used as a measure in Kylin; See hive number types: [https://cwiki.apache.org/confluence/display/Hive/LanguageManual+Types#LanguageManualTypes-NumericTypes](https://cwiki.apache.org/confluence/display/Hive/LanguageManual+Types#LanguageManualTypes-NumericTypes) -#### Why Kylin need extract the distinct columns from Fact Table before building cube? +#### 8. Why Kylin need extract the distinct columns from Fact Table before building cube? * Kylin uses dictionary to encode the values in each column, this greatly reduce the cube's storage size. To build the dictionary, Kylin need fetch the distinct values for each column. -#### Why Kylin calculate the HIVE table cardinality? +#### 9. Why Kylin calculate the HIVE table cardinality? * The cardinality of dimensions is an important measure of cube complexity. The higher the cardinality, the bigger the cube, and thus the longer to build and the slower to query. Cardinality > 1,000 is worth attention and > 1,000,000 should be avoided at best effort. For optimal cube performance, try reduce high cardinality by categorize values or derive features. -#### How to add new user or change the default password? +#### 10. How to add new user or change the default password? * Kylin web's security is implemented with Spring security framework, where the kylinSecurity.xml is the main configuration file: {% highlight Groff markup %} @@ -54,9 +72,9 @@ since: v0.6.x {% endhighlight %} * The password hash for pre-defined test users can be found in the profile "sandbox,testing" part; To change the default password, you need generate a new hash and then update it here, please refer to the code snippet in: [https://stackoverflow.com/questions/25844419/spring-bcryptpasswordencoder-generate-different-password-for-same-input](https://stackoverflow.com/questions/25844419/spring-bcryptpasswordencoder-generate-different-password-for-same-input) - * When you deploy Kylin for more users, switch to LDAP authentication is recommended; To enable LDAP authentication, update "kylin.sandbox" in conf/kylin.properties to false, and also configure the ldap.* properties in ${KYLIN_HOME}/conf/kylin.properties + * When you deploy Kylin for more users, switch to LDAP authentication is recommended. -#### Using sub-query for un-supported SQL +#### 11. Using sub-query for un-supported SQL {% highlight Groff markup %} Original SQL: @@ -85,15 +103,15 @@ from ( group by a.slr_sgmt {% endhighlight %} -#### Build kylin meet NPM errors (ä¸å½å¤§éå°åºç¨æ·è¯·ç¹å«æ³¨ææ¤é®é¢)? +#### 12. Build kylin meet NPM errors (ä¸å½å¤§éå°åºç¨æ·è¯·ç¹å«æ³¨ææ¤é®é¢) - * Please add proxy for your NPM (请为NPM设置代ç): + * Please add proxy for your NPM: `npm config set proxy http://YOUR_PROXY_IP` * Please update your local NPM repository to using any mirror of npmjs.org, like Taobao NPM (请æ´æ°æ¨æ¬å°çNPMä»åºä»¥ä½¿ç¨å½å çNPMéåï¼ä¾å¦æ·å®NPMéå) : [http://npm.taobao.org](http://npm.taobao.org) -#### Failed to run BuildCubeWithEngineTest, saying failed to connect to hbase while hbase is active +#### 13. Failed to run BuildCubeWithEngineTest, saying failed to connect to hbase while hbase is active * User may get this error when first time run hbase client, please check the error trace to see whether there is an error saying couldn't access a folder like "/hadoop/hbase/local/jars"; If that folder doesn't exist, create it. http://git-wip-us.apache.org/repos/asf/kylin/blob/8989d052/website/_docs15/install/advance_settings.md ---------------------------------------------------------------------- diff --git a/website/_docs15/install/advance_settings.md b/website/_docs15/install/advance_settings.md index 2abd6a4..5b978c0 100644 --- a/website/_docs15/install/advance_settings.md +++ b/website/_docs15/install/advance_settings.md @@ -61,6 +61,15 @@ The compression settings define in `kylin_job_conf.xml` and `kylin_job_conf_inme Compression settings only take effect after restarting Kylin server instance. +## Allocate more memory to Kylin instance + +Open `bin/setenv.sh`, which has two sample settings for `KYLIN_JVM_SETTINGS` environment variable; The default setting is small (4GB at max.), you can comment it and then un-comment the next line to allocate 16GB: + +{% highlight Groff markup %} +export KYLIN_JVM_SETTINGS="-Xms1024M -Xmx4096M -Xss1024K -XX:MaxPermSize=128M -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc:$KYLIN_HOME/logs/kylin.gc.$$ -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=64M" +# export KYLIN_JVM_SETTINGS="-Xms16g -Xmx16g -XX:MaxPermSize=512m -XX:NewSize=3g -XX:MaxNewSize=3g -XX:SurvivorRatio=4 -XX:+CMSClassUnloadingEnabled -XX:+CMSParallelRemarkEnabled -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -XX:CMSInitiatingOccupancyFraction=70 -XX:+DisableExplicitGC -XX:+HeapDumpOnOutOfMemoryError" +{% endhighlight %} + ## Enable LDAP or SSO authentication Check [How to Enable Security with LDAP and SSO](../howto/howto_ldap_and_sso.html) @@ -75,6 +84,9 @@ mail.host=your-smtp-server mail.username=your-smtp-account mail.password=your-smtp-pwd mail.sender=your-sender-address +kylin.job.admin.dls=adminstrator-address {% endhighlight %} Restart Kylin server to take effective. To disable, set `mail.enabled` back to `false`. + +Administrator will get notifications for all jobs. Modeler and Analyst need enter email address into the "Notification List" at the first page of cube wizard, and then will get notified for that cube.
