date:20181101

svn commit: r1845476 - in /kylin/site: ./ cn/docs/howto/ docs/gettingstarted/ docs/howto/

2018-11-01 Thread lidong

Author: lidong
Date: Thu Nov  1 14:08:38 2018
New Revision: 1845476

URL: http://svn.apache.org/viewvc?rev=1845476=rev
Log:
Update FAQ, update storage and metadata cleanup doc, add the online video 
section

Modified:
kylin/site/cn/docs/howto/howto_backup_metadata.html
kylin/site/cn/docs/howto/howto_cleanup_storage.html
kylin/site/docs/gettingstarted/best_practices.html
kylin/site/docs/gettingstarted/faq.html
kylin/site/docs/howto/howto_backup_metadata.html
kylin/site/docs/howto/howto_cleanup_storage.html
kylin/site/feed.xml

Modified: kylin/site/cn/docs/howto/howto_backup_metadata.html
URL: 
http://svn.apache.org/viewvc/kylin/site/cn/docs/howto/howto_backup_metadata.html?rev=1845476=1845475=1845476=diff
==
--- kylin/site/cn/docs/howto/howto_backup_metadata.html (original)
+++ kylin/site/cn/docs/howto/howto_backup_metadata.html Thu Nov  1 14:08:38 2018
@@ -221,15 +221,15 @@ mkdir /path/to/restore_new/cubeä» metadata store æ¸çæ ç¨çèµæº
 éçè¿è¡æ¶é´å¢é¿ï¼ç±»ä¼¼åå¸ãè¡¨å¿«ç
§çèµæºåå¾æ²¡æç¨ï¼cube segmentè¢«ä¸¢å¼æè
åå¹¶äºï¼ï¼ä½æ¯å®ä»¬ä¾æ§å ç¨ç©ºé´ï¼ä½ 
å¯ä»¥è¿è¡å½ä»¤æ¥æ¾å°å¹¶æ¸é¤å®ä»¬ï¼
 
-é¦åï¼è¿è¡ä¸ä¸ªæ£æ¥ï¼è¿æ¯å®å¨çå 
ä¸ºå®ä¸ä¼æ¹åä»»ä½ä¸è¥¿ï¼
+é¦åï¼è¿è¡ä¸ä¸ªæ£æ¥ï¼è¿æ¯å®å¨çå 
ä¸ºå®ä¸ä¼æ¹åä»»ä½ä¸è¥¿ï¼éè¿æ·»å  ââjobThreshold 
30(é»è®¤å¼ï¼æ¨å¯ä»¥æ¹ä¸ºä»»ä½æ°å)â 
åæ°ï¼æ¨å¯ä»¥è®¾ç½®è¦ä¿çç metadata resource å¤©æ°ï¼
 
-./bin/metastore.sh clean
+./bin/metastore.sh clean --jobThreshold 30
 
 å°è¦è¢«å é¤çèµæºä¼è¢«ååºæ¥ï¼
 
 æ¥ä¸æ¥ï¼å¢å  ââdelete trueâ åæ°æ¥æ¸
çè¿äºèµæºï¼å¨è¿ä¹åï¼ä½ åºè¯¥ç¡®ä¿å·²ç»å¤ä»½ metadata 
storeï¼
 
-./bin/metastore.sh clean --delete true
+./bin/metastore.sh clean --delete true --jobThreshold 
30
 
 


Modified: kylin/site/cn/docs/howto/howto_cleanup_storage.html
URL: 
http://svn.apache.org/viewvc/kylin/site/cn/docs/howto/howto_cleanup_storage.html?rev=1845476=1845475=1845476=diff
==
--- kylin/site/cn/docs/howto/howto_cleanup_storage.html (original)
+++ kylin/site/cn/docs/howto/howto_cleanup_storage.html Thu Nov  1 14:08:38 2018
@@ -166,7 +166,7 @@ var _hmt = _hmt || [];

æ¸çåå¨

-   
Kylinå¨æå»ºcubeæé´ä¼å¨HDFSä¸çæä¸é´æä»¶ï¼é¤æ¤ä¹å¤ï¼å½æ¸
ç/å 
é¤/åå¹¶cubeæ¶ï¼ä¸äºHBaseè¡¨å¯è½è¢«éçå¨HBaseå´ä»¥ååä¹ä¸ä¼è¢«æ¥è¯¢ï¼è½ç¶Kylinå·²ç»å¼å§åèªå¨åçåå¾åæ¶ï¼ä½ä¸ä¸å®è½è¦çå°ææçæ
åµï¼ä½ å¯ä»¥å®æåç¦»çº¿çåå¨æ¸çï¼
+   Kylin å¨æå»º cube 
æé´ä¼å¨ HDFS ä¸çæä¸é´æä»¶ï¼é¤æ¤ä¹å¤ï¼å½æ¸ç/å 
é¤/åå¹¶ cube æ¶ï¼ä¸äº HBase è¡¨å¯è½è¢«éçå¨ HBase 
å´ä»¥ååä¹ä¸ä¼è¢«æ¥è¯¢ï¼è½ç¶ Kylin 
å·²ç»å¼å§åèªå¨åçåå¾åæ¶ï¼ä½ä¸ä¸å®è½è¦çå°ææçæ
åµï¼ä½ å¯ä»¥å®æåç¦»çº¿çåå¨æ¸çï¼
 
 æ¥éª¤ï¼
 1. æ£æ¥åªäºèµæºå¯ä»¥æ¸çï¼è¿ä¸æ¥ä¸ä¼å é¤ä»»ä½ä¸è¥¿ï¼
@@ -174,12 +174,17 @@ var _hmt = _hmt || [];
 export KYLIN_HOME=/path/to/kylin_home
 ${KYLIN_HOME}/bin/kylin.sh org.apache.kylin.tool.StorageCleanupJob --delete 
false
 
-è¯·å°è¿éç (version) æ¿æ¢ä¸ºä½ å®è£çKylin jarçæ¬ã
+è¯·å°è¿éç (version) æ¿æ¢ä¸ºä½ å®è£ç Kylin jar çæ¬ã
 2. ä½ 
å¯ä»¥æ½æ¥ä¸ä¸¤ä¸ªèµæºæ¥æ£æ¥å®ä»¬æ¯å¦å·²ç»æ²¡æè¢«å¼ç¨äºï¼ç¶åå
 ä¸ââdelete trueâéé¡¹è¿è¡æ¸çã
 
 ${KYLIN_HOME}/bin/kylin.sh 
org.apache.kylin.tool.StorageCleanupJob --delete true
 
-å®æåï¼ä¸é´HDFSä¸çä¸é´æä»¶åHTableä¼è¢«ç§»é¤ã
+å®æåï¼Hive éçä¸é´è¡¨, HDFS ä¸çä¸é´æä»¶å HBase ä¸ç 
HTables é½ä¼è¢«ç§»é¤ã
+3. å¦ææ¨æ³è¦å é¤ææèµæºï¼å¯æ·»å  ââforce trueâ 
éé¡¹ï¼
+
+${KYLIN_HOME}/bin/kylin.sh 
org.apache.kylin.tool.StorageCleanupJob --force true --delete 
true
+
+å®æåï¼Hive ä¸ææçä¸é´è¡¨, HDFS ä¸ææçä¸é´æä»¶å 
HBase ä¸ç HTables é½ä¼è¢«ç§»é¤ã
 



Modified: kylin/site/docs/gettingstarted/best_practices.html
URL: 
http://svn.apache.org/viewvc/kylin/site/docs/gettingstarted/best_practices.html?rev=1845476=1845475=1845476=diff
==
--- kylin/site/docs/gettingstarted/best_practices.html (original)
+++ kylin/site/docs/gettingstarted/best_practices.html Thu Nov  1 14:08:38 2018
@@ -6047,6

[kylin] branch master updated: KYLIN-3611 upgrade to apache tomcat 7.0.91, update version in pom

2018-11-01 Thread shaofengshi

This is an automated email from the ASF dual-hosted git repository.

shaofengshi pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/kylin.git


The following commit(s) were added to refs/heads/master by this push:
 new 3fabbe6  KYLIN-3611 upgrade to apache tomcat 7.0.91, update version in 
pom
3fabbe6 is described below

commit 3fabbe6d2b1199bedb7b74083feabc6586a6d644
Author: shaofengshi 
AuthorDate: Thu Nov 1 15:37:00 2018 +0800

KYLIN-3611 upgrade to apache tomcat 7.0.91, update version in pom
---
 pom.xml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/pom.xml b/pom.xml
index 6cb6716..aa0ca0c 100644
--- a/pom.xml
+++ b/pom.xml
@@ -121,7 +121,7 @@
 3.2.4
 2.4.0
 2.5
-7.0.90
+7.0.91
 3.1
 2.3.23
 5.9.2

[kylin] branch 2.5.x updated: KYLIN-3611 upgrade to apache tomcat 7.0.91, update version in pom

2018-11-01 Thread shaofengshi

This is an automated email from the ASF dual-hosted git repository.

shaofengshi pushed a commit to branch 2.5.x
in repository https://gitbox.apache.org/repos/asf/kylin.git


The following commit(s) were added to refs/heads/2.5.x by this push:
 new 995e0fd  KYLIN-3611 upgrade to apache tomcat 7.0.91, update version in 
pom
995e0fd is described below

commit 995e0fde0b8f62aa94180db8ffc912b006cd5260
Author: shaofengshi 
AuthorDate: Thu Nov 1 15:35:27 2018 +0800

KYLIN-3611 upgrade to apache tomcat 7.0.91, update version in pom
---
 pom.xml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/pom.xml b/pom.xml
index c8f717a..315a403 100644
--- a/pom.xml
+++ b/pom.xml
@@ -119,7 +119,7 @@
 3.2.4
 2.4.0
 2.5
-7.0.90
+7.0.91
 3.1
 2.3.23
 5.9.2

[kylin] branch master updated: KYLIN-3644 Fix SparkFactDistinct step NumberFormatException

2018-11-01 Thread shaofengshi

This is an automated email from the ASF dual-hosted git repository.

shaofengshi pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/kylin.git


The following commit(s) were added to refs/heads/master by this push:
 new 6544143  KYLIN-3644 Fix SparkFactDistinct step NumberFormatException
6544143 is described below

commit 65441434e0aff410050009b4538ef3bf9e7af93f
Author: chao long 
AuthorDate: Thu Nov 1 16:43:11 2018 +0800

KYLIN-3644 Fix SparkFactDistinct step NumberFormatException
---
 .../src/main/java/org/apache/kylin/cube/util/KeyValueBuilder.java| 2 +-
 .../main/java/org/apache/kylin/engine/spark/SparkFactDistinct.java   | 5 -
 2 files changed, 5 insertions(+), 2 deletions(-)

diff --git 
a/core-cube/src/main/java/org/apache/kylin/cube/util/KeyValueBuilder.java 
b/core-cube/src/main/java/org/apache/kylin/cube/util/KeyValueBuilder.java
index 0ba4fd8..0636a5c 100644
--- a/core-cube/src/main/java/org/apache/kylin/cube/util/KeyValueBuilder.java
+++ b/core-cube/src/main/java/org/apache/kylin/cube/util/KeyValueBuilder.java
@@ -56,7 +56,7 @@ public class KeyValueBuilder implements Serializable {
 }
 }
 
-protected boolean isNull(String v) {
+public boolean isNull(String v) {
 return nullStrs.contains(v);
 }
 
diff --git 
a/engine-spark/src/main/java/org/apache/kylin/engine/spark/SparkFactDistinct.java
 
b/engine-spark/src/main/java/org/apache/kylin/engine/spark/SparkFactDistinct.java
index 5cfd2d7..cdd0ac2 100644
--- 
a/engine-spark/src/main/java/org/apache/kylin/engine/spark/SparkFactDistinct.java
+++ 
b/engine-spark/src/main/java/org/apache/kylin/engine/spark/SparkFactDistinct.java
@@ -63,6 +63,7 @@ import org.apache.kylin.cube.DimensionRangeInfo;
 import org.apache.kylin.cube.cuboid.CuboidUtil;
 import org.apache.kylin.cube.model.CubeDesc;
 import org.apache.kylin.cube.model.CubeJoinedFlatTableEnrich;
+import org.apache.kylin.cube.util.KeyValueBuilder;
 import org.apache.kylin.dict.DictionaryGenerator;
 import org.apache.kylin.dict.IDictionaryBuilder;
 import org.apache.kylin.engine.EngineFactory;
@@ -250,6 +251,7 @@ public class SparkFactDistinct extends AbstractApplication 
implements Serializab
 private Map dimensionRangeInfoMap;
 private transient ByteBuffer tmpbuf;
 private LongAccumulator bytesWritten;
+private KeyValueBuilder keyValueBuilder;
 
 public FlatOutputFucntion(String cubeName, String segmentId, String 
metaurl, SerializableConfiguration conf,
 int samplingPercent, LongAccumulator bytesWritten) {
@@ -272,6 +274,7 @@ public class SparkFactDistinct extends AbstractApplication 
implements Serializab
 CubeJoinedFlatTableEnrich intermediateTableDesc = new 
CubeJoinedFlatTableEnrich(
 EngineFactory.getJoinedFlatTableDesc(cubeSegment), 
cubeDesc);
 
+keyValueBuilder = new KeyValueBuilder(intermediateTableDesc);
 reducerMapping = new 
FactDistinctColumnsReducerMapping(cubeInstance);
 tmpbuf = ByteBuffer.allocate(4096);
 
@@ -317,7 +320,7 @@ public class SparkFactDistinct extends AbstractApplication 
implements Serializab
 
 for (int i = 0; i < allCols.size(); i++) {
 String fieldValue = row[columnIndex[i]];
-if (fieldValue == null)
+if (fieldValue == null || 
keyValueBuilder.isNull(fieldValue))
 continue;
 
 final DataType type = allCols.get(i).getType();

[kylin] branch 2.5.x updated: KYLIN-3644 Fix SparkFactDistinct step NumberFormatException

2018-11-01 Thread shaofengshi

This is an automated email from the ASF dual-hosted git repository.

shaofengshi pushed a commit to branch 2.5.x
in repository https://gitbox.apache.org/repos/asf/kylin.git


The following commit(s) were added to refs/heads/2.5.x by this push:
 new 0411360  KYLIN-3644 Fix SparkFactDistinct step NumberFormatException
0411360 is described below

commit 0411360557ec2355448fa67c3ae18a9473179afa
Author: chao long 
AuthorDate: Thu Nov 1 16:43:11 2018 +0800

KYLIN-3644 Fix SparkFactDistinct step NumberFormatException
---
 .../src/main/java/org/apache/kylin/cube/util/KeyValueBuilder.java| 2 +-
 .../main/java/org/apache/kylin/engine/spark/SparkFactDistinct.java   | 5 -
 2 files changed, 5 insertions(+), 2 deletions(-)

diff --git 
a/core-cube/src/main/java/org/apache/kylin/cube/util/KeyValueBuilder.java 
b/core-cube/src/main/java/org/apache/kylin/cube/util/KeyValueBuilder.java
index 0ba4fd8..0636a5c 100644
--- a/core-cube/src/main/java/org/apache/kylin/cube/util/KeyValueBuilder.java
+++ b/core-cube/src/main/java/org/apache/kylin/cube/util/KeyValueBuilder.java
@@ -56,7 +56,7 @@ public class KeyValueBuilder implements Serializable {
 }
 }
 
-protected boolean isNull(String v) {
+public boolean isNull(String v) {
 return nullStrs.contains(v);
 }
 
diff --git 
a/engine-spark/src/main/java/org/apache/kylin/engine/spark/SparkFactDistinct.java
 
b/engine-spark/src/main/java/org/apache/kylin/engine/spark/SparkFactDistinct.java
index 213cdfd..7f05234 100644
--- 
a/engine-spark/src/main/java/org/apache/kylin/engine/spark/SparkFactDistinct.java
+++ 
b/engine-spark/src/main/java/org/apache/kylin/engine/spark/SparkFactDistinct.java
@@ -58,6 +58,7 @@ import org.apache.kylin.cube.DimensionRangeInfo;
 import org.apache.kylin.cube.cuboid.CuboidUtil;
 import org.apache.kylin.cube.model.CubeDesc;
 import org.apache.kylin.cube.model.CubeJoinedFlatTableEnrich;
+import org.apache.kylin.cube.util.KeyValueBuilder;
 import org.apache.kylin.dict.DictionaryGenerator;
 import org.apache.kylin.dict.IDictionaryBuilder;
 import org.apache.kylin.engine.EngineFactory;
@@ -238,6 +239,7 @@ public class SparkFactDistinct extends AbstractApplication 
implements Serializab
 private Map dimensionRangeInfoMap;
 private ByteBuffer tmpbuf;
 private LongAccumulator bytesWritten;
+private KeyValueBuilder keyValueBuilder;
 
 public FlatOutputFucntion(String cubeName, String segmentId, String 
metaurl, SerializableConfiguration conf, int samplingPercent, LongAccumulator 
bytesWritten) {
 this.cubeName = cubeName;
@@ -259,6 +261,7 @@ public class SparkFactDistinct extends AbstractApplication 
implements Serializab
 CubeJoinedFlatTableEnrich intermediateTableDesc = new 
CubeJoinedFlatTableEnrich(
 EngineFactory.getJoinedFlatTableDesc(cubeSegment), 
cubeDesc);
 
+keyValueBuilder = new KeyValueBuilder(intermediateTableDesc);
 reducerMapping = new 
FactDistinctColumnsReducerMapping(cubeInstance);
 tmpbuf = ByteBuffer.allocate(4096);
 
@@ -304,7 +307,7 @@ public class SparkFactDistinct extends AbstractApplication 
implements Serializab
 
 for (int i = 0; i < allCols.size(); i++) {
 String fieldValue = row[columnIndex[i]];
-if (fieldValue == null)
+if (fieldValue == null || 
keyValueBuilder.isNull(fieldValue))
 continue;
 
 final DataType type = allCols.get(i).getType();

[kylin] branch master updated: KYLIN-3662

2018-11-01 Thread shaofengshi

This is an automated email from the ASF dual-hosted git repository.

shaofengshi pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/kylin.git


The following commit(s) were added to refs/heads/master by this push:
 new 2793a94  KYLIN-3662
2793a94 is described below

commit 2793a94331d85a35ca7144a4584dd6ac0e4fb901
Author: chitin 
AuthorDate: Thu Nov 1 16:40:51 2018 +0800

KYLIN-3662
---
 .../src/main/java/org/apache/kylin/rest/service/QueryService.java   | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git 
a/server-base/src/main/java/org/apache/kylin/rest/service/QueryService.java 
b/server-base/src/main/java/org/apache/kylin/rest/service/QueryService.java
index d8e397e..bd4049d 100644
--- a/server-base/src/main/java/org/apache/kylin/rest/service/QueryService.java
+++ b/server-base/src/main/java/org/apache/kylin/rest/service/QueryService.java
@@ -375,7 +375,7 @@ public class QueryService extends BasicService {
 // project not found
 ProjectManager mgr = 
ProjectManager.getInstance(KylinConfig.getInstanceFromEnv());
 if (mgr.getProject(sqlRequest.getProject()) == null) {
-throw new BadRequestException(msg.getPROJECT_NOT_FOUND());
+throw new BadRequestException(String.format(Locale.ROOT, 
msg.getPROJECT_NOT_FOUND(), sqlRequest.getProject()));
 }
 if (StringUtils.isBlank(sqlRequest.getSql())) {
 throw new BadRequestException(msg.getNULL_EMPTY_SQL());

[kylin] branch document updated: Update FAQ, update storage and metadata cleanup doc, add the online video section

2018-11-01 Thread shaofengshi

This is an automated email from the ASF dual-hosted git repository.

shaofengshi pushed a commit to branch document
in repository https://gitbox.apache.org/repos/asf/kylin.git


The following commit(s) were added to refs/heads/document by this push:
 new 183a868  Update FAQ, update storage and metadata cleanup doc, add the 
online video section
183a868 is described below

commit 183a86862234cf6c18b117dfb40c742aab5ebf56
Author: GinaZhai 
AuthorDate: Thu Nov 1 17:58:48 2018 +0800

Update FAQ, update storage and metadata cleanup doc, add the online video 
section
---
 website/_docs/gettingstarted/best_practices.md  | 17 -
 website/_docs/gettingstarted/faq.md |  4 
 website/_docs/howto/howto_backup_metadata.cn.md |  6 +++---
 website/_docs/howto/howto_backup_metadata.md|  6 +++---
 website/_docs/howto/howto_cleanup_storage.cn.md | 11 ---
 website/_docs/howto/howto_cleanup_storage.md|  7 ++-
 6 files changed, 40 insertions(+), 11 deletions(-)

diff --git a/website/_docs/gettingstarted/best_practices.md 
b/website/_docs/gettingstarted/best_practices.md
index 191530e..118a388 100644
--- a/website/_docs/gettingstarted/best_practices.md
+++ b/website/_docs/gettingstarted/best_practices.md
@@ -24,4 +24,19 @@ List of articles about Kylin best practices contributed by 
community. Some of th
 
 * [Kylin and Qlik 
Integration](https://github.com/albertoRamon/Kylin/tree/master/KylinWithQlik) 
(Thanks to [Ramón Portolés, 
Alberto](https://www.linkedin.com/in/alberto-ramon-portoles-a02b523b))
 
-* [How to use Hue with 
Kylin](https://github.com/albertoRamon/Kylin/tree/master/KylinWithHue) (Thanks 
to [Ramón Portolés, 
Alberto](https://www.linkedin.com/in/alberto-ramon-portoles-a02b523b))
\ No newline at end of file
+* [How to use Hue with 
Kylin](https://github.com/albertoRamon/Kylin/tree/master/KylinWithHue) (Thanks 
to [Ramón Portolés, 
Alberto](https://www.linkedin.com/in/alberto-ramon-portoles-a02b523b))
+
+
+**Here are online tutorials for self-studying Kylin:**
+
+- Free Kylin tutorial (registration needed), from the core developers, in 
Chinese:
+[https://www.chinahadoop.cn/search?q=kylin](https://www.chinahadoop.cn/search?q=kylin)
+
+- A paid Kylin tutorial on Udemy, in English:
+[https://www.udemy.com/apache-kylin-implementing-olap-on-the-hadoop-platform](https://www.udemy.com/apache-kylin-implementing-olap-on-the-hadoop-platform)
+
+- Tutorial of Kylin on Tableau, in Spanish: 
+[https://www.youtube.com/watch?v=k6fBw8yA1NI](https://www.youtube.com/watch?v=k6fBw8yA1NI)
+
+Besides, there is also some video of Kylin on the conferences; You can search 
for them on Youtube:
+[https://www.youtube.com/results?search_query=Apache+Kylin](https://www.youtube.com/results?search_query=Apache+Kylin)
diff --git a/website/_docs/gettingstarted/faq.md 
b/website/_docs/gettingstarted/faq.md
index def9851..0540857 100644
--- a/website/_docs/gettingstarted/faq.md
+++ b/website/_docs/gettingstarted/faq.md
@@ -135,6 +135,10 @@ When the new cube is built, please disable or drop the old 
one.
 
 If you can accept the absence of new dimensions for historical data, you can 
build the new cube since the end time of the old cube. And then create a hybrid 
model over the old and new cube.
 
+ How to solve the data security problem of Tableau connection client?
+  
+  * Kylin's ACL control can solve this problem. Different analysts have the 
authority to work on different projects for Kylin. When you create a Kylin ODBC 
DSN, you can map different links to different analyst accounts.
+
  The query result is not exactly matched with that in Hive, what's the 
possible reason?
 
   * Possible reasons:
diff --git a/website/_docs/howto/howto_backup_metadata.cn.md 
b/website/_docs/howto/howto_backup_metadata.cn.md
index 66ab3ed..56db4c8 100644
--- a/website/_docs/howto/howto_backup_metadata.cn.md
+++ b/website/_docs/howto/howto_backup_metadata.cn.md
@@ -73,14 +73,14 @@ cd $KYLIN_HOME
 ## 从 metadata store 清理无用的资源
 随着运行时间增长，类似字典、表快照的资源变得没有用（cube segment被丢弃或者合并了），但是它们依旧占用空间，你可以运行命令来找到并清除它们：
 
-首先，运行一个检查，这是安全的因为它不会改变任何东西：
+首先，运行一个检查，这是安全的因为它不会改变任何东西，通过添加 "--jobThreshold 30(默认值，您可以改为任何数字)" 
参数，您可以设置要保留的 metadata resource 天数：
 {% highlight Groff markup %}
-./bin/metastore.sh clean
+./bin/metastore.sh clean --jobThreshold 30
 {% endhighlight %}
 
 将要被删除的资源会被列出来：
 
 接下来，增加 “--delete true” 参数来清理这些资源；在这之前，你应该确保已经备份 metadata store：
 {% highlight Groff markup %}
-./bin/metastore.sh clean --delete true
+./bin/metastore.sh clean --delete true --jobThreshold 30
 {% endhighlight %}
diff --git a/website/_docs/howto/howto_backup_metadata.md 
b/website/_docs/howto/howto_backup_metadata.md
index bbc0d0f..deaaa70 100644
--- a/website/_docs/howto/howto_backup_metadata.md
+++ b/website/_docs/howto/howto_backup_metadata.md
@@ -74,14 +74,14 @@ When developing/debugging Kylin, typically you have a dev 
machine with an IDE, a
 ## Cleanup unused resources from metadata store
 As time goes on, some resources

[kylin] branch 2.5.1-release updated: [maven-release-plugin] prepare for next development iteration

2018-11-01 Thread shaofengshi

This is an automated email from the ASF dual-hosted git repository.

shaofengshi pushed a commit to branch 2.5.1-release
in repository https://gitbox.apache.org/repos/asf/kylin.git


The following commit(s) were added to refs/heads/2.5.1-release by this push:
 new 4b08881  [maven-release-plugin] prepare for next development iteration
4b08881 is described below

commit 4b08881118348d6586af1dc2a7e4924280c269d4
Author: shaofeng shi 
AuthorDate: Fri Nov 2 05:38:08 2018 +

[maven-release-plugin] prepare for next development iteration
---
 assembly/pom.xml   | 2 +-
 atopcalcite/pom.xml| 2 +-
 core-common/pom.xml| 2 +-
 core-cube/pom.xml  | 2 +-
 core-dictionary/pom.xml| 2 +-
 core-job/pom.xml   | 2 +-
 core-metadata/pom.xml  | 2 +-
 core-metrics/pom.xml   | 2 +-
 core-storage/pom.xml   | 2 +-
 engine-mr/pom.xml  | 2 +-
 engine-spark/pom.xml   | 2 +-
 jdbc/pom.xml   | 2 +-
 kylin-it/pom.xml   | 2 +-
 metrics-reporter-hive/pom.xml  | 2 +-
 metrics-reporter-kafka/pom.xml | 2 +-
 pom.xml| 4 ++--
 query/pom.xml  | 2 +-
 server-base/pom.xml| 2 +-
 server/pom.xml | 2 +-
 source-hive/pom.xml| 2 +-
 source-kafka/pom.xml   | 2 +-
 storage-hbase/pom.xml  | 2 +-
 tomcat-ext/pom.xml | 2 +-
 tool-assembly/pom.xml  | 2 +-
 tool/pom.xml   | 2 +-
 25 files changed, 26 insertions(+), 26 deletions(-)

diff --git a/assembly/pom.xml b/assembly/pom.xml
index 2816c6b..0b5090b 100644
--- a/assembly/pom.xml
+++ b/assembly/pom.xml
@@ -26,7 +26,7 @@
 
 kylin
 org.apache.kylin
-2.5.1
+2.5.2-SNAPSHOT
 
 
 
diff --git a/atopcalcite/pom.xml b/atopcalcite/pom.xml
index 0c9b218..e6ab6d3 100644
--- a/atopcalcite/pom.xml
+++ b/atopcalcite/pom.xml
@@ -28,7 +28,7 @@
 
 org.apache.kylin
 kylin
-2.5.1
+2.5.2-SNAPSHOT
 
 
 
diff --git a/core-common/pom.xml b/core-common/pom.xml
index 57e8b2e..2545fc7 100644
--- a/core-common/pom.xml
+++ b/core-common/pom.xml
@@ -28,7 +28,7 @@
 
 org.apache.kylin
 kylin
-2.5.1
+2.5.2-SNAPSHOT
 
 
 
diff --git a/core-cube/pom.xml b/core-cube/pom.xml
index 54efcdd..b41e8bf 100644
--- a/core-cube/pom.xml
+++ b/core-cube/pom.xml
@@ -28,7 +28,7 @@
 
 org.apache.kylin
 kylin
-2.5.1
+2.5.2-SNAPSHOT
 
 
 
diff --git a/core-dictionary/pom.xml b/core-dictionary/pom.xml
index ee748b6..1f9c96e 100644
--- a/core-dictionary/pom.xml
+++ b/core-dictionary/pom.xml
@@ -28,7 +28,7 @@
 
 org.apache.kylin
 kylin
-2.5.1
+2.5.2-SNAPSHOT
 
 
 
diff --git a/core-job/pom.xml b/core-job/pom.xml
index 1e42410..89af797 100644
--- a/core-job/pom.xml
+++ b/core-job/pom.xml
@@ -28,7 +28,7 @@
 
 org.apache.kylin
 kylin
-2.5.1
+2.5.2-SNAPSHOT
 
 
 
diff --git a/core-metadata/pom.xml b/core-metadata/pom.xml
index 5430b4f..e5d3319 100644
--- a/core-metadata/pom.xml
+++ b/core-metadata/pom.xml
@@ -28,7 +28,7 @@
 
 org.apache.kylin
 kylin
-2.5.1
+2.5.2-SNAPSHOT
 
 
 
diff --git a/core-metrics/pom.xml b/core-metrics/pom.xml
index 3fac13d..10959e1 100644
--- a/core-metrics/pom.xml
+++ b/core-metrics/pom.xml
@@ -28,7 +28,7 @@
 
 kylin
 org.apache.kylin
-2.5.1
+2.5.2-SNAPSHOT
 
 
 
diff --git a/core-storage/pom.xml b/core-storage/pom.xml
index dda19b1..f0390dd 100644
--- a/core-storage/pom.xml
+++ b/core-storage/pom.xml
@@ -28,7 +28,7 @@
 
 org.apache.kylin
 kylin
-2.5.1
+2.5.2-SNAPSHOT
 
 
 
diff --git a/engine-mr/pom.xml b/engine-mr/pom.xml
index 453821b..10b7720 100644
--- a/engine-mr/pom.xml
+++ b/engine-mr/pom.xml
@@ -28,7 +28,7 @@
 
 org.apache.kylin
 kylin
-2.5.1
+2.5.2-SNAPSHOT
 
 
 
diff --git a/engine-spark/pom.xml b/engine-spark/pom.xml
index e44c8ae..27bd121 100644
--- a/engine-spark/pom.xml
+++ b/engine-spark/pom.xml
@@ -28,7 +28,7 @@
 
 org.apache.kylin
 kylin
-2.5.1
+2.5.2-SNAPSHOT
 
 
 
diff --git a/jdbc/pom.xml b/jdbc/pom.xml
index 5083ae4..6d356be 100644
--- a/jdbc/pom.xml
+++ b/jdbc/pom.xml
@@ -28,7 +28,7 @@
 
 org.apache.kylin
 kylin
-2.5.1
+2.5.2-SNAPSHOT
 
 
 
diff --git a/kylin-it/pom.xml b/kylin-it/pom.xml
index 2555fa2..fd63e94 100644
--- a/kylin-it/pom.xml
+++ b/kylin-it/pom.xml
@@ -29,7 +29,7 @@
 
 kylin
 org.apache.kylin
-2.5.1
+2.5.2-SNAPSHOT
 
 
 
diff --git a/metrics-reporter-hive/pom.xml b/metrics-reporter-hive/pom.xml
index 4cdee56..d1fe86e 100644
--- a/metrics-reporter-hive/pom.xml
+++

[kylin] 01/01: [maven-release-plugin] prepare release kylin-2.5.1

2018-11-01 Thread shaofengshi

This is an automated email from the ASF dual-hosted git repository.

shaofengshi pushed a commit to branch 2.5.1-release
in repository https://gitbox.apache.org/repos/asf/kylin.git

commit 24e2452309a450ec4ef62339b003343eabe23016
Author: shaofeng shi 
AuthorDate: Fri Nov 2 05:37:48 2018 +

[maven-release-plugin] prepare release kylin-2.5.1
---
 assembly/pom.xml   | 2 +-
 atopcalcite/pom.xml| 2 +-
 core-common/pom.xml| 2 +-
 core-cube/pom.xml  | 2 +-
 core-dictionary/pom.xml| 2 +-
 core-job/pom.xml   | 2 +-
 core-metadata/pom.xml  | 2 +-
 core-metrics/pom.xml   | 2 +-
 core-storage/pom.xml   | 2 +-
 engine-mr/pom.xml  | 2 +-
 engine-spark/pom.xml   | 2 +-
 jdbc/pom.xml   | 2 +-
 kylin-it/pom.xml   | 2 +-
 metrics-reporter-hive/pom.xml  | 2 +-
 metrics-reporter-kafka/pom.xml | 2 +-
 pom.xml| 4 ++--
 query/pom.xml  | 2 +-
 server-base/pom.xml| 2 +-
 server/pom.xml | 2 +-
 source-hive/pom.xml| 2 +-
 source-kafka/pom.xml   | 2 +-
 storage-hbase/pom.xml  | 2 +-
 tomcat-ext/pom.xml | 2 +-
 tool-assembly/pom.xml  | 2 +-
 tool/pom.xml   | 2 +-
 25 files changed, 26 insertions(+), 26 deletions(-)

diff --git a/assembly/pom.xml b/assembly/pom.xml
index 02a535e..2816c6b 100644
--- a/assembly/pom.xml
+++ b/assembly/pom.xml
@@ -26,7 +26,7 @@
 
 kylin
 org.apache.kylin
-2.5.1-SNAPSHOT
+2.5.1
 
 
 
diff --git a/atopcalcite/pom.xml b/atopcalcite/pom.xml
index 1b975cb..0c9b218 100644
--- a/atopcalcite/pom.xml
+++ b/atopcalcite/pom.xml
@@ -28,7 +28,7 @@
 
 org.apache.kylin
 kylin
-2.5.1-SNAPSHOT
+2.5.1
 
 
 
diff --git a/core-common/pom.xml b/core-common/pom.xml
index fd02274..57e8b2e 100644
--- a/core-common/pom.xml
+++ b/core-common/pom.xml
@@ -28,7 +28,7 @@
 
 org.apache.kylin
 kylin
-2.5.1-SNAPSHOT
+2.5.1
 
 
 
diff --git a/core-cube/pom.xml b/core-cube/pom.xml
index df13d1e..54efcdd 100644
--- a/core-cube/pom.xml
+++ b/core-cube/pom.xml
@@ -28,7 +28,7 @@
 
 org.apache.kylin
 kylin
-2.5.1-SNAPSHOT
+2.5.1
 
 
 
diff --git a/core-dictionary/pom.xml b/core-dictionary/pom.xml
index 25a10e7..ee748b6 100644
--- a/core-dictionary/pom.xml
+++ b/core-dictionary/pom.xml
@@ -28,7 +28,7 @@
 
 org.apache.kylin
 kylin
-2.5.1-SNAPSHOT
+2.5.1
 
 
 
diff --git a/core-job/pom.xml b/core-job/pom.xml
index 491be34..1e42410 100644
--- a/core-job/pom.xml
+++ b/core-job/pom.xml
@@ -28,7 +28,7 @@
 
 org.apache.kylin
 kylin
-2.5.1-SNAPSHOT
+2.5.1
 
 
 
diff --git a/core-metadata/pom.xml b/core-metadata/pom.xml
index 88dd284..5430b4f 100644
--- a/core-metadata/pom.xml
+++ b/core-metadata/pom.xml
@@ -28,7 +28,7 @@
 
 org.apache.kylin
 kylin
-2.5.1-SNAPSHOT
+2.5.1
 
 
 
diff --git a/core-metrics/pom.xml b/core-metrics/pom.xml
index 635c38b..3fac13d 100644
--- a/core-metrics/pom.xml
+++ b/core-metrics/pom.xml
@@ -28,7 +28,7 @@
 
 kylin
 org.apache.kylin
-2.5.1-SNAPSHOT
+2.5.1
 
 
 
diff --git a/core-storage/pom.xml b/core-storage/pom.xml
index 374eae4..dda19b1 100644
--- a/core-storage/pom.xml
+++ b/core-storage/pom.xml
@@ -28,7 +28,7 @@
 
 org.apache.kylin
 kylin
-2.5.1-SNAPSHOT
+2.5.1
 
 
 
diff --git a/engine-mr/pom.xml b/engine-mr/pom.xml
index a7efef4..453821b 100644
--- a/engine-mr/pom.xml
+++ b/engine-mr/pom.xml
@@ -28,7 +28,7 @@
 
 org.apache.kylin
 kylin
-2.5.1-SNAPSHOT
+2.5.1
 
 
 
diff --git a/engine-spark/pom.xml b/engine-spark/pom.xml
index b609244..e44c8ae 100644
--- a/engine-spark/pom.xml
+++ b/engine-spark/pom.xml
@@ -28,7 +28,7 @@
 
 org.apache.kylin
 kylin
-2.5.1-SNAPSHOT
+2.5.1
 
 
 
diff --git a/jdbc/pom.xml b/jdbc/pom.xml
index 316fcd4..5083ae4 100644
--- a/jdbc/pom.xml
+++ b/jdbc/pom.xml
@@ -28,7 +28,7 @@
 
 org.apache.kylin
 kylin
-2.5.1-SNAPSHOT
+2.5.1
 
 
 
diff --git a/kylin-it/pom.xml b/kylin-it/pom.xml
index 8a4ab88..2555fa2 100644
--- a/kylin-it/pom.xml
+++ b/kylin-it/pom.xml
@@ -29,7 +29,7 @@
 
 kylin
 org.apache.kylin
-2.5.1-SNAPSHOT
+2.5.1
 
 
 
diff --git a/metrics-reporter-hive/pom.xml b/metrics-reporter-hive/pom.xml
index 6c64b2a..4cdee56 100644
--- a/metrics-reporter-hive/pom.xml
+++ b/metrics-reporter-hive/pom.xml
@@ -28,7 +28,7 @@
 
 kylin
 org.apache.kylin
-2.5.1-SNAPSHOT
+2.5.1
 
 
 
diff --git a/metrics-reporter-kafka/pom.xml

[kylin] annotated tag kylin-2.5.1 updated (24e2452 -> 3c71bd8)

2018-11-01 Thread shaofengshi

This is an automated email from the ASF dual-hosted git repository.

shaofengshi pushed a change to annotated tag kylin-2.5.1
in repository https://gitbox.apache.org/repos/asf/kylin.git.


*** WARNING: tag kylin-2.5.1 was modified! ***

from 24e2452  (commit)
  to 3c71bd8  (tag)
 tagging 24e2452309a450ec4ef62339b003343eabe23016 (commit)
 replaces kylin-1.2
  by shaofeng shi
  on Fri Nov 2 05:38:03 2018 +

- Log -
[maven-release-plugin] copy for tag kylin-2.5.1
---


No new revisions were added by this update.

Summary of changes:

[kylin] branch 2.5.1-release created (now 24e2452)

2018-11-01 Thread shaofengshi

This is an automated email from the ASF dual-hosted git repository.

shaofengshi pushed a change to branch 2.5.1-release
in repository https://gitbox.apache.org/repos/asf/kylin.git.


  at 24e2452  [maven-release-plugin] prepare release kylin-2.5.1

This branch includes the following new commits:

 new 24e2452  [maven-release-plugin] prepare release kylin-2.5.1

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.

svn commit: r30590 - in /dev/kylin/apache-kylin-2.5.1-rc1: ./ apache-kylin-2.5.1-source-release.zip apache-kylin-2.5.1-source-release.zip.asc apache-kylin-2.5.1-source-release.zip.sha256

2018-11-01 Thread shaofengshi

Author: shaofengshi
Date: Fri Nov  2 05:53:13 2018
New Revision: 30590

Log:
Upload release artifacts to staging

Added:
dev/kylin/apache-kylin-2.5.1-rc1/
dev/kylin/apache-kylin-2.5.1-rc1/apache-kylin-2.5.1-source-release.zip   
(with props)
dev/kylin/apache-kylin-2.5.1-rc1/apache-kylin-2.5.1-source-release.zip.asc

dev/kylin/apache-kylin-2.5.1-rc1/apache-kylin-2.5.1-source-release.zip.sha256

Added: dev/kylin/apache-kylin-2.5.1-rc1/apache-kylin-2.5.1-source-release.zip
==
Binary file - no diff available.

Propchange: 
dev/kylin/apache-kylin-2.5.1-rc1/apache-kylin-2.5.1-source-release.zip
--
svn:mime-type = application/octet-stream

Added: 
dev/kylin/apache-kylin-2.5.1-rc1/apache-kylin-2.5.1-source-release.zip.asc
==
--- dev/kylin/apache-kylin-2.5.1-rc1/apache-kylin-2.5.1-source-release.zip.asc 
(added)
+++ dev/kylin/apache-kylin-2.5.1-rc1/apache-kylin-2.5.1-source-release.zip.asc 
Fri Nov  2 05:53:13 2018
@@ -0,0 +1,17 @@
+-BEGIN PGP SIGNATURE-
+Version: GnuPG v2.0.14 (GNU/Linux)
+
+iQIcBAABCgAGBQJb2+ILAAoJEKvJ+IVvz0cgWnwP/2HnNBvStCUQwArwYb7NGhUn
+80pPnacM0nxZ8zhpV8p94BKsQRTxvi9K9pf+d4jCO2H38B/TWFMNb3oCRKKB1z7U
+K67zmhE5kU+hxLB/t7tVbWVJK3MG4tiJeJFHuVdNXtjE+Wkb34kqYxOUlENvgbCO
+EJ85R+Mk7ZM/3kW7LsNvIF2U89m96Qgc02qvzpv6wYX7YRQQZJ40v0c/C6PKzeqn
+yC/RcAOXpA+2gSTI8kXm92yiRWWBVPSBFV1u+FM9y24wKz16XFTDe75DPIC/+GTQ
+bu8mcN/MJjhn5svZsmay2v6A6labRbgXMubcdsNrj3vuSv1BObz/Z0OcBKqRMPS0
+mwcBdAU3b6a+naqGLm2DgWPklfC7FjO94ZKnvEbmEIpq/j/hpBGfboSPzGg78J6y
+lyrKv267WuVGLFvomk9lDspQ/AmgK29+K2kMjCOEyiQb3pBh44G0N3dQme0ajdDE
+czcmDk/vjlE3qndyW7C6ZyfyTboH0aLm/LkBbglWnMo5TS6owE+0I2xjS9JgDLdS
+sSsX9GgHicWaG6sYnAxpIUXd25qAoC21kIqOkrO2DzOVNN7UtGd5PkiCNKShjNCK
+6j4ffSgMxKhpXIAKa8u6G7ndO2zlVJHUeDbqfUweHc4vKe5FTiIAvUzQ0D/mHSP7
+VEVGzaWNmDACqsJfY5Pc
+=sNhA
+-END PGP SIGNATURE-

Added: 
dev/kylin/apache-kylin-2.5.1-rc1/apache-kylin-2.5.1-source-release.zip.sha256
==
--- 
dev/kylin/apache-kylin-2.5.1-rc1/apache-kylin-2.5.1-source-release.zip.sha256 
(added)
+++ 
dev/kylin/apache-kylin-2.5.1-rc1/apache-kylin-2.5.1-source-release.zip.sha256 
Fri Nov  2 05:53:13 2018
@@ -0,0 +1 @@
+21db5dab4d3900a49237b9083b5d270c8471d1882a5427cddf1cc74873df42f2
\ No newline at end of file

[kylin] branch master updated: KYLIN-3649 fix ci

2018-11-01 Thread shaofengshi

This is an automated email from the ASF dual-hosted git repository.

shaofengshi pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/kylin.git


The following commit(s) were added to refs/heads/master by this push:
 new 37fdf65  KYLIN-3649 fix ci
37fdf65 is described below

commit 37fdf651e5d0d986a177e17e302871400beb0c97
Author: shaofengshi 
AuthorDate: Fri Nov 2 11:37:32 2018 +0800

KYLIN-3649 fix ci
---
 .../src/main/java/org/apache/kylin/rest/job/StorageCleanupJob.java  | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git 
a/server-base/src/main/java/org/apache/kylin/rest/job/StorageCleanupJob.java 
b/server-base/src/main/java/org/apache/kylin/rest/job/StorageCleanupJob.java
index 70a7b8a..b73e916 100755
--- a/server-base/src/main/java/org/apache/kylin/rest/job/StorageCleanupJob.java
+++ b/server-base/src/main/java/org/apache/kylin/rest/job/StorageCleanupJob.java
@@ -154,7 +154,7 @@ public class StorageCleanupJob extends AbstractApplication {
 }
 
 protected void cleanUnusedHBaseTables() throws IOException {
-if ("hbase".equals(config.getStorageUrl().getScheme())) {
+if ("hbase".equals(config.getStorageUrl().getScheme()) && 
!"".equals(config.getMetadataUrl().getScheme())) {
 final int deleteTimeoutMin = 10; // Unit minute
 try {
 // use reflection to isolate NoClassDef errors when HBase is 
not available

[kylin] branch 2.5.x updated: KYLIN-3649 fix ci

2018-11-01 Thread shaofengshi

This is an automated email from the ASF dual-hosted git repository.

shaofengshi pushed a commit to branch 2.5.x
in repository https://gitbox.apache.org/repos/asf/kylin.git


The following commit(s) were added to refs/heads/2.5.x by this push:
 new aa3efa3  KYLIN-3649 fix ci
aa3efa3 is described below

commit aa3efa39f0a7d8cd8c4e4a487ded2ca2ea86d577
Author: shaofengshi 
AuthorDate: Fri Nov 2 11:37:32 2018 +0800

KYLIN-3649 fix ci
---
 .../src/main/java/org/apache/kylin/rest/job/StorageCleanupJob.java  | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git 
a/server-base/src/main/java/org/apache/kylin/rest/job/StorageCleanupJob.java 
b/server-base/src/main/java/org/apache/kylin/rest/job/StorageCleanupJob.java
index 70a7b8a..b73e916 100755
--- a/server-base/src/main/java/org/apache/kylin/rest/job/StorageCleanupJob.java
+++ b/server-base/src/main/java/org/apache/kylin/rest/job/StorageCleanupJob.java
@@ -154,7 +154,7 @@ public class StorageCleanupJob extends AbstractApplication {
 }
 
 protected void cleanUnusedHBaseTables() throws IOException {
-if ("hbase".equals(config.getStorageUrl().getScheme())) {
+if ("hbase".equals(config.getStorageUrl().getScheme()) && 
!"".equals(config.getMetadataUrl().getScheme())) {
 final int deleteTimeoutMin = 10; // Unit minute
 try {
 // use reflection to isolate NoClassDef errors when HBase is 
not available

svn commit: r1845476 - in /kylin/site: ./ cn/docs/howto/ docs/gettingstarted/ docs/howto/

[kylin] branch master updated: KYLIN-3611 upgrade to apache tomcat 7.0.91, update version in pom

[kylin] branch 2.5.x updated: KYLIN-3611 upgrade to apache tomcat 7.0.91, update version in pom

[kylin] branch master updated: KYLIN-3644 Fix SparkFactDistinct step NumberFormatException

[kylin] branch 2.5.x updated: KYLIN-3644 Fix SparkFactDistinct step NumberFormatException

[kylin] branch master updated: KYLIN-3662

[kylin] branch document updated: Update FAQ, update storage and metadata cleanup doc, add the online video section

[kylin] branch 2.5.1-release updated: [maven-release-plugin] prepare for next development iteration

[kylin] 01/01: [maven-release-plugin] prepare release kylin-2.5.1

[kylin] annotated tag kylin-2.5.1 updated (24e2452 -> 3c71bd8)

[kylin] branch 2.5.1-release created (now 24e2452)

svn commit: r30590 - in /dev/kylin/apache-kylin-2.5.1-rc1: ./ apache-kylin-2.5.1-source-release.zip apache-kylin-2.5.1-source-release.zip.asc apache-kylin-2.5.1-source-release.zip.sha256

[kylin] branch master updated: KYLIN-3649 fix ci

[kylin] branch 2.5.x updated: KYLIN-3649 fix ci

14 matches

Site Navigation

Mail list logo

Footer information