Author: lidong Date: Tue Oct 18 07:03:11 2016 New Revision: 1765396 URL: http://svn.apache.org/viewvc?rev=1765396&view=rev Log: Add blog for new streaming
Added: kylin/site/blog/2016/10/ kylin/site/blog/2016/10/18/ kylin/site/blog/2016/10/18/new-nrt-streaming/ kylin/site/blog/2016/10/18/new-nrt-streaming/index.html kylin/site/images/blog/new-streaming.png (with props) kylin/site/images/blog/offset-as-partition-value.png (with props) kylin/site/images/blog/streaming-adapter.png (with props) kylin/site/images/blog/streaming-monitor.png (with props) kylin/site/images/blog/streaming-twitter.png (with props) Modified: kylin/site/blog/index.html kylin/site/feed.xml Added: kylin/site/blog/2016/10/18/new-nrt-streaming/index.html URL: http://svn.apache.org/viewvc/kylin/site/blog/2016/10/18/new-nrt-streaming/index.html?rev=1765396&view=auto ============================================================================== --- kylin/site/blog/2016/10/18/new-nrt-streaming/index.html (added) +++ kylin/site/blog/2016/10/18/new-nrt-streaming/index.html Tue Oct 18 07:03:11 2016 @@ -0,0 +1,309 @@ +<!-- +* Licensed to the Apache Software Foundation (ASF) under one +* or more contributor license agreements. See the NOTICE file +* distributed with this work for additional information +* regarding copyright ownership. The ASF licenses this file +* to you under the Apache License, Version 2.0 (the +* "License"); you may not use this file except in compliance +* with the License. You may obtain a copy of the License at +* +* http://www.apache.org/licenses/LICENSE-2.0 +* +* Unless required by applicable law or agreed to in writing, software +* distributed under the License is distributed on an "AS IS" BASIS, +* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +* See the License for the specific language governing permissions and +* limitations under the License. +--> +<!doctype html> +<html> + <!-- +* Licensed to the Apache Software Foundation (ASF) under one +* or more contributor license agreements. See the NOTICE file +* distributed with this work for additional information +* regarding copyright ownership. The ASF licenses this file +* to you under the Apache License, Version 2.0 (the +* "License"); you may not use this file except in compliance +* with the License. You may obtain a copy of the License at +* +* http://www.apache.org/licenses/LICENSE-2.0 +* +* Unless required by applicable law or agreed to in writing, software +* distributed under the License is distributed on an "AS IS" BASIS, +* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +* See the License for the specific language governing permissions and +* limitations under the License. +--> + +<head> + <meta charset="utf-8"> + <meta http-equiv="X-UA-Compatible" content="IE=edge"> + <meta name="viewport" content="width=device-width, initial-scale=1"> + + <title>Apache Kylin | New NRT Streaming in Apache Kylin</title> + <meta name="description" content="In 1.5.0 Apache Kylin introduces the Streaming Cubing feature, which can consume data from Kafka topic directly. This blog introduces how that be implemented..."> + <meta name="author" content="Apache Kylin"> + <link rel="shortcut icon" href="fav.png" type="image/png"> + + + +<link rel="stylesheet" href="/assets/css/animate.css"> +<!-- Bootstrap --> +<link rel="stylesheet" href="/assets/css/bootstrap.min.css"> + +<!-- Fonts --> +<!-- <link rel="stylesheet" href="http://fonts.googleapis.com/css?family=Alice|Open+Sans:400,300,700"> --> + +<!-- Icons --> +<link rel="stylesheet" href="/assets/css/font-awesome.min.css"> + + <!-- Custom styles --> + <link rel="stylesheet" href="/assets/css/styles.css"> + <link rel="stylesheet" href="/assets/css/docs.css"> + <link rel="stylesheet" href="/assets/css/pygments.css"> + + <link rel="canonical" href="http://kylin.apache.org/blog/2016/10/18/new-nrt-streaming/"> + <link rel="alternate" type="application/rss+xml" title="Apache Kylin" href="http://kylin.apache.org/feed.xml" /> + +<!--[if lt IE 9]> <script src="assets/js/html5shiv.js"></script> <![endif]--> +<script> + (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){ + (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o), + m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m) + })(window,document,'script','//www.google-analytics.com/analytics.js','ga'); + + //oringal tracker for kylin.io + ga('create', 'UA-55534813-1', 'auto'); + //new tracker for kylin.apache.org + ga('create', 'UA-55534813-2', 'auto', {'name':'toplevel'}); + + ga('send', 'pageview'); + ga('toplevel.send', 'pageview'); + + +</script> +<script type="text/javascript" src="/assets/js/jquery-1.9.1.min.js"></script> +<script type="text/javascript" src="/assets/js/nside.js"></script> </script> +<script type="text/javascript" src="/assets/js/nnav.js"></script> </script> +</head> + + <body> + <!-- +* Licensed to the Apache Software Foundation (ASF) under one +* or more contributor license agreements. See the NOTICE file +* distributed with this work for additional information +* regarding copyright ownership. The ASF licenses this file +* to you under the Apache License, Version 2.0 (the +* "License"); you may not use this file except in compliance +* with the License. You may obtain a copy of the License at +* +* http://www.apache.org/licenses/LICENSE-2.0 +* +* Unless required by applicable law or agreed to in writing, software +* distributed under the License is distributed on an "AS IS" BASIS, +* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +* See the License for the specific language governing permissions and +* limitations under the License. +--> + +<header id="header" > + + <div id="head" class="parallax" parallax-speed="3" > + <div id="logo" class="text-center"> <img class="img-circle" id="circlelogo" src="/assets/images/kylin_logo.jpg"> <span class="title" >Apache Kylinâ¢</span> <span class="tagline">Extreme OLAP Engine for Big Data</span> + </div> + </div> + + + <!-- Main Menu --> + <nav class="navbar navbar-default" role="navigation" id="nav-wrapper"> + <div class="container-fluid" id="nav"> + <!-- + <img class="img-circle" width="40px" height="40px" id="circlelogo" src="/assets/images/kylin_logo.jpg"> + --> + <!-- Brand and toggle get grouped for better mobile display --> + <div class="navbar-header"> + <button type="button" class="navbar-toggle collapsed" data-toggle="collapse" data-target="#bs-example-navbar-collapse-1"> + <span class="sr-only">Toggle navigation</span> + <span class="icon-bar"></span> + <span class="icon-bar"></span> + <span class="icon-bar"></span> + </button> + + </div> + + <!-- Collect the nav links, forms, and other content for toggling --> + <div class="collapse navbar-collapse" id="bs-example-navbar-collapse-1"> + <ul class="nav navbar-nav"> + <li><a href="/">Home</a></li> + <li><a href="/docs15" >Docs</a></li> + <li><a href="/download">Download</li> + <li><a href="/community" >Community</a></li> + <li><a href="/development" >Development</a></li> + <li><a href="/blog">Blog</li> + <li><a href="/cn" >ä¸æç</a></li> + <li><a href="https://twitter.com/apachekylin" target="_blank" class="fa fa-twitter fa-lg" title="Twitter: @ApacheKylin" ></a></li> + <li><a href="https://github.com/apache/kylin" target="_blank" class="fa fa-github-alt fa-lg" title="Github: apache/kylin" ></a></li> + <li><a href="https://www.facebook.com/kylinio" target="_blank" class="fa fa-facebook fa-lg" title="Facebook: kylin.io" ></a></li> + </ul> + </div><!-- /.navbar-collapse --> + </div><!-- /.container-fluid --> +</nav> + </header> + + <div class="page-content"> + <header style=" padding:2em 0 0 0"> + <div class="container" > + <h4 class="section-title"><span>Apache Kylin⢠Technical Blog</span></h4> + </div> + </div> + + <div class="container"> + <div> + <article class="post-content" > + <!-- +* Licensed to the Apache Software Foundation (ASF) under one +* or more contributor license agreements. See the NOTICE file +* distributed with this work for additional information +* regarding copyright ownership. The ASF licenses this file +* to you under the Apache License, Version 2.0 (the +* "License"); you may not use this file except in compliance +* with the License. You may obtain a copy of the License at +* +* http://www.apache.org/licenses/LICENSE-2.0 +* +* Unless required by applicable law or agreed to in writing, software +* distributed under the License is distributed on an "AS IS" BASIS, +* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +* See the License for the specific language governing permissions and +* limitations under the License. +--> + +<div class="post" style=" padding:2em 4em 4em 4em"> + + <header class="post-header"> + <h1 class="post-title">New NRT Streaming in Apache Kylin</h1> + <p class="post-meta" >Oct 18, 2016 ⢠Shaofeng Shi</p> + </header> + + <article class="post-content" > + <p>In 1.5.0 Apache Kylin introduces the Streaming Cubing feature, which can consume data from Kafka topic directly. This <a href="/blog/2016/02/03/streaming-cubing/">blog</a> introduces how that be implemented, and this <a href="/docs15/tutorial/cube_streaming.html">tutorial</a> introduces how to use it.</p> + +<p>While, that implementation was marked as âexperimentalâ because it has the following limitations:</p> + +<ul> + <li> + <p>Not scalable: it starts a Java process for a micro-batch cube building, instead of leveraging any computing framework; If too many messages arrive at one time, the build may fail with OutOfMemory error;</p> + </li> + <li> + <p>May loss data: it uses a time window to seek the approximate start/end offsets on Kafka topic, which means too late/early arrived messages will be skipped; Then the query couldnât ensure 100% accuracy.</p> + </li> + <li> + <p>Difficult to monitor: the streaming cubing is out of the Job engineâs scope, user can not monitor the jobs with Web GUI or REST API.</p> + </li> + <li> + <p>Others: hard to recover from accident, difficult to maintain the code, etc.</p> + </li> +</ul> + +<p>To overcome these limitations, the Apache Kylin team developed the new streaming (<a href="https://issues.apache.org/jira/browse/KYLIN-1726">KYLIN-1726</a>) with Kafka 0.10 API, it has been tested internally for some time, will release to public soon.</p> + +<p>The new design is a perfect implementation under Kylin 1.5âs âPlug-inâ architecture: treat Kafka topic as a âData Sourceâ like Hive table, using an adapter to extract the data to HDFS; the next steps are almost the same as from Hive. Figure 1 is a high level architecture of the new design.</p> + +<p><img src="/images/blog/new-streaming.png" alt="Kylin New Streaming Framework Architecture" /></p> + +<p>The adapter to read Kafka messages is modified from <a href="https://github.com/amient/kafka-hadoop-loader">kafka-hadoop-loader</a>, which is open sourced under Apache License V2.0; it starts a mapper for each Kafka partition, reading and then saving the messages to HDFS; in next steps Kylin will be able to leverage existing framework like MR to do the processing, this makes the solution scalable and fault-tolerant.</p> + +<p>To overcome the âdata lossâ problem, Kylin adds the start/end offset information on each Cube segment, and then use the offsets as the partition value (no overlap is allowed); this ensures no data be lost and 1 message be consumed at most once. To let the late/early message can be queried, Cube segments allow overlap for the partition time dimension: Kylin will scan all segments which include the queried time. Figure 2 illurates this.</p> + +<p><img src="/images/blog/offset-as-partition-value.png" alt="Use Offset to Cut Segments" /></p> + +<p>Other changes/enhancements are made in the new streaming:</p> + +<ul> + <li>Allow multiple segments being built/merged concurrently</li> + <li>Automatically seek start/end offsets (if user doesnât specify) from previous segment or Kafka</li> + <li>Support embeded properties in JSON message</li> + <li>Add REST API to trigger streaming cubeâs building</li> + <li>Add REST API to check and fill the segment holes</li> +</ul> + +<p>The integration test result shows big improvements than the previous version:</p> + +<ul> + <li>Scalability: it can easily process up to hundreds of million records in one build;</li> + <li>Flexibility: trigger the build at any time with the frequency you want, e.g: every 5 minutes in day and every hour in night; Kylin manages the offsets so it can resume from the last position;</li> + <li>Stability: pretty stable, no OutOfMemory error;</li> + <li>Management: user can check all jobsâ status through Kylinâs âMonitorâ page or REST API;</li> + <li>Build Performance: in a testing cluster (8 AWS instances to consume Twitter streams), 10 thousands arrives per second, define a 9-dimension cube with 3 measures; when build interval is 2 mintues, the job finishes in around 3 minutes; if change interval to 5 mintues, build finishes in around 4 minutes;</li> +</ul> + +<p>Here are a couple of screenshots in this test:<br /> +<img src="/images/blog/streaming-monitor.png" alt="Streaming Job Monitoring" /></p> + +<p><img src="/images/blog/streaming-adapter.png" alt="Streaming Adapter" /></p> + +<p><img src="/images/blog/streaming-twitter.png" alt="Streaming Twitter Sample" /></p> + + </article> + +</div> + + + + + + </article> + </div> + </div> + <!-- +* Licensed to the Apache Software Foundation (ASF) under one +* or more contributor license agreements. See the NOTICE file +* distributed with this work for additional information +* regarding copyright ownership. The ASF licenses this file +* to you under the Apache License, Version 2.0 (the +* "License"); you may not use this file except in compliance +* with the License. You may obtain a copy of the License at +* +* http://www.apache.org/licenses/LICENSE-2.0 +* +* Unless required by applicable law or agreed to in writing, software +* distributed under the License is distributed on an "AS IS" BASIS, +* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +* See the License for the specific language governing permissions and +* limitations under the License. +--> + +<footer id="underfooter"> + <div class="container"> + <div class="row"> + <div class="col-md-12 widget"> + <div class="widget-body" style="text-align:center"> + <a href="http://www.apache.org"> + <img id="asf-logo" alt="Apache Software Foundation" src="/assets/images/feather-small.gif"> + </a> + + <div> + The contents of this website are © 2015 Apache Software Foundation under the terms of the <a + href="http://www.apache.org/licenses/LICENSE-2.0"> Apache License v2 </a>. Apache Kylin and + its logo are trademarks of the Apache Software Foundation. + </div> + + </div> + </div> + </div> + <!-- /row of widgets --> + + </div> + <div></div> + +</footer> + + <script src="/assets/js/jquery-1.9.1.min.js"></script> + <script src="/assets/js/bootstrap.min.js"></script> + <script src="/assets/js/main.js"></script> + </body> +</html> + + + + Modified: kylin/site/blog/index.html URL: http://svn.apache.org/viewvc/kylin/site/blog/index.html?rev=1765396&r1=1765395&r2=1765396&view=diff ============================================================================== --- kylin/site/blog/index.html (original) +++ kylin/site/blog/index.html Tue Oct 18 07:03:11 2016 @@ -174,6 +174,12 @@ <li> <h2 align="left" style="margin:0px"> + <a class="post-link" href="/blog/2016/10/18/new-nrt-streaming/">New NRT Streaming in Apache Kylin</a></h2><div align="left" class="post-meta">posted: Oct 18, 2016</div> + + </li> + + <li> + <h2 align="left" style="margin:0px"> <a class="post-link" href="/blog/2016/08/27/query-metrics-in-kylin/">Query Metrics in Apache Kylin</a></h2><div align="left" class="post-meta">posted: Aug 27, 2016</div> </li> @@ -216,13 +222,13 @@ <li> <h2 align="left" style="margin:0px"> - <a class="post-link" href="/blog/2016/05/26/release-v1.5.2/">Apache Kylin v1.5.2 Release Announcement</a></h2><div align="left" class="post-meta">posted: May 26, 2016</div> + <a class="post-link" href="/cn/blog/2016/05/26/release-v1.5.2/">Apache Kylin v1.5.2 æ£å¼åå¸</a></h2><div align="left" class="post-meta">posted: May 26, 2016</div> </li> <li> <h2 align="left" style="margin:0px"> - <a class="post-link" href="/cn/blog/2016/05/26/release-v1.5.2/">Apache Kylin v1.5.2 æ£å¼åå¸</a></h2><div align="left" class="post-meta">posted: May 26, 2016</div> + <a class="post-link" href="/blog/2016/05/26/release-v1.5.2/">Apache Kylin v1.5.2 Release Announcement</a></h2><div align="left" class="post-meta">posted: May 26, 2016</div> </li> @@ -258,13 +264,13 @@ <li> <h2 align="left" style="margin:0px"> - <a class="post-link" href="/cn/blog/2016/03/16/release-v1.3.0/">Apache Kylin v1.3.0 æ£å¼åå¸</a></h2><div align="left" class="post-meta">posted: Mar 16, 2016</div> + <a class="post-link" href="/blog/2016/03/16/release-v1.3.0/">Apache Kylin v1.3.0 Release Announcement</a></h2><div align="left" class="post-meta">posted: Mar 16, 2016</div> </li> <li> <h2 align="left" style="margin:0px"> - <a class="post-link" href="/blog/2016/03/16/release-v1.3.0/">Apache Kylin v1.3.0 Release Announcement</a></h2><div align="left" class="post-meta">posted: Mar 16, 2016</div> + <a class="post-link" href="/cn/blog/2016/03/16/release-v1.3.0/">Apache Kylin v1.3.0 æ£å¼åå¸</a></h2><div align="left" class="post-meta">posted: Mar 16, 2016</div> </li> @@ -282,37 +288,37 @@ <li> <h2 align="left" style="margin:0px"> - <a class="post-link" href="/blog/2015/12/25/support-powerbi-tableau9/">Apache Kylin supports Tableau 9 and MS Excel, Power BI now</a></h2><div align="left" class="post-meta">posted: Dec 25, 2015</div> + <a class="post-link" href="/cn/blog/2015/12/25/support-powerbi-tableau9/">Apache Kylinå¢å 对Tableau 9å微软Excel, Power BIçæ¯æ</a></h2><div align="left" class="post-meta">posted: Dec 25, 2015</div> </li> <li> <h2 align="left" style="margin:0px"> - <a class="post-link" href="/cn/blog/2015/12/25/support-powerbi-tableau9/">Apache Kylinå¢å 对Tableau 9å微软Excel, Power BIçæ¯æ</a></h2><div align="left" class="post-meta">posted: Dec 25, 2015</div> + <a class="post-link" href="/blog/2015/12/25/support-powerbi-tableau9/">Apache Kylin supports Tableau 9 and MS Excel, Power BI now</a></h2><div align="left" class="post-meta">posted: Dec 25, 2015</div> </li> <li> <h2 align="left" style="margin:0px"> - <a class="post-link" href="/cn/blog/2015/12/23/release-v1.2/">Apache Kylin v1.2 æ£å¼åå¸</a></h2><div align="left" class="post-meta">posted: Dec 23, 2015</div> + <a class="post-link" href="/blog/2015/12/23/release-v1.2/">Apache Kylin v1.2 Release Announcement</a></h2><div align="left" class="post-meta">posted: Dec 23, 2015</div> </li> <li> <h2 align="left" style="margin:0px"> - <a class="post-link" href="/blog/2015/12/23/release-v1.2/">Apache Kylin v1.2 Release Announcement</a></h2><div align="left" class="post-meta">posted: Dec 23, 2015</div> + <a class="post-link" href="/cn/blog/2015/12/23/release-v1.2/">Apache Kylin v1.2 æ£å¼åå¸</a></h2><div align="left" class="post-meta">posted: Dec 23, 2015</div> </li> <li> <h2 align="left" style="margin:0px"> - <a class="post-link" href="/cn/blog/2015/10/25/release-v1.1-incubating/">Apache Kylin v1.1 (incubating) æ£å¼åå¸</a></h2><div align="left" class="post-meta">posted: Oct 25, 2015</div> + <a class="post-link" href="/blog/2015/10/25/release-v1.1-incubating/">Apache Kylin v1.1 (incubating) Release Announcement</a></h2><div align="left" class="post-meta">posted: Oct 25, 2015</div> </li> <li> <h2 align="left" style="margin:0px"> - <a class="post-link" href="/blog/2015/10/25/release-v1.1-incubating/">Apache Kylin v1.1 (incubating) Release Announcement</a></h2><div align="left" class="post-meta">posted: Oct 25, 2015</div> + <a class="post-link" href="/cn/blog/2015/10/25/release-v1.1-incubating/">Apache Kylin v1.1 (incubating) æ£å¼åå¸</a></h2><div align="left" class="post-meta">posted: Oct 25, 2015</div> </li> Modified: kylin/site/feed.xml URL: http://svn.apache.org/viewvc/kylin/site/feed.xml?rev=1765396&r1=1765395&r2=1765396&view=diff ============================================================================== --- kylin/site/feed.xml (original) +++ kylin/site/feed.xml Tue Oct 18 07:03:11 2016 @@ -19,11 +19,80 @@ <description>Apache Kylin Home</description> <link>http://kylin.apache.org/</link> <atom:link href="http://kylin.apache.org/feed.xml" rel="self" type="application/rss+xml"/> - <pubDate>Tue, 18 Oct 2016 06:59:12 -0700</pubDate> - <lastBuildDate>Tue, 18 Oct 2016 06:59:12 -0700</lastBuildDate> + <pubDate>Tue, 18 Oct 2016 07:59:25 -0700</pubDate> + <lastBuildDate>Tue, 18 Oct 2016 07:59:25 -0700</lastBuildDate> <generator>Jekyll v2.5.3</generator> <item> + <title>New NRT Streaming in Apache Kylin</title> + <description><p>In 1.5.0 Apache Kylin introduces the Streaming Cubing feature, which can consume data from Kafka topic directly. This <a href="/blog/2016/02/03/streaming-cubing/">blog</a> introduces how that be implemented, and this <a href="/docs15/tutorial/cube_streaming.html">tutorial</a> introduces how to use it.</p> + +<p>While, that implementation was marked as âexperimentalâ because it has the following limitations:</p> + +<ul> + <li> + <p>Not scalable: it starts a Java process for a micro-batch cube building, instead of leveraging any computing framework; If too many messages arrive at one time, the build may fail with OutOfMemory error;</p> + </li> + <li> + <p>May loss data: it uses a time window to seek the approximate start/end offsets on Kafka topic, which means too late/early arrived messages will be skipped; Then the query couldnât ensure 100% accuracy.</p> + </li> + <li> + <p>Difficult to monitor: the streaming cubing is out of the Job engineâs scope, user can not monitor the jobs with Web GUI or REST API.</p> + </li> + <li> + <p>Others: hard to recover from accident, difficult to maintain the code, etc.</p> + </li> +</ul> + +<p>To overcome these limitations, the Apache Kylin team developed the new streaming (<a href="https://issues.apache.org/jira/browse/KYLIN-1726">KYLIN-1726</a>) with Kafka 0.10 API, it has been tested internally for some time, will release to public soon.</p> + +<p>The new design is a perfect implementation under Kylin 1.5âs âPlug-inâ architecture: treat Kafka topic as a âData Sourceâ like Hive table, using an adapter to extract the data to HDFS; the next steps are almost the same as from Hive. Figure 1 is a high level architecture of the new design.</p> + +<p><img src="/images/blog/new-streaming.png" alt="Kylin New Streaming Framework Architecture" /></p> + +<p>The adapter to read Kafka messages is modified from <a href="https://github.com/amient/kafka-hadoop-loader">kafka-hadoop-loader</a>, which is open sourced under Apache License V2.0; it starts a mapper for each Kafka partition, reading and then saving the messages to HDFS; in next steps Kylin will be able to leverage existing framework like MR to do the processing, this makes the solution scalable and fault-tolerant.</p> + +<p>To overcome the âdata lossâ problem, Kylin adds the start/end offset information on each Cube segment, and then use the offsets as the partition value (no overlap is allowed); this ensures no data be lost and 1 message be consumed at most once. To let the late/early message can be queried, Cube segments allow overlap for the partition time dimension: Kylin will scan all segments which include the queried time. Figure 2 illurates this.</p> + +<p><img src="/images/blog/offset-as-partition-value.png" alt="Use Offset to Cut Segments" /></p> + +<p>Other changes/enhancements are made in the new streaming:</p> + +<ul> + <li>Allow multiple segments being built/merged concurrently</li> + <li>Automatically seek start/end offsets (if user doesnât specify) from previous segment or Kafka</li> + <li>Support embeded properties in JSON message</li> + <li>Add REST API to trigger streaming cubeâs building</li> + <li>Add REST API to check and fill the segment holes</li> +</ul> + +<p>The integration test result shows big improvements than the previous version:</p> + +<ul> + <li>Scalability: it can easily process up to hundreds of million records in one build;</li> + <li>Flexibility: trigger the build at any time with the frequency you want, e.g: every 5 minutes in day and every hour in night; Kylin manages the offsets so it can resume from the last position;</li> + <li>Stability: pretty stable, no OutOfMemory error;</li> + <li>Management: user can check all jobsâ status through Kylinâs âMonitorâ page or REST API;</li> + <li>Build Performance: in a testing cluster (8 AWS instances to consume Twitter streams), 10 thousands arrives per second, define a 9-dimension cube with 3 measures; when build interval is 2 mintues, the job finishes in around 3 minutes; if change interval to 5 mintues, build finishes in around 4 minutes;</li> +</ul> + +<p>Here are a couple of screenshots in this test:<br /> +<img src="/images/blog/streaming-monitor.png" alt="Streaming Job Monitoring" /></p> + +<p><img src="/images/blog/streaming-adapter.png" alt="Streaming Adapter" /></p> + +<p><img src="/images/blog/streaming-twitter.png" alt="Streaming Twitter Sample" /></p> +</description> + <pubDate>Tue, 18 Oct 2016 10:30:00 -0700</pubDate> + <link>http://kylin.apache.org/blog/2016/10/18/new-nrt-streaming/</link> + <guid isPermaLink="true">http://kylin.apache.org/blog/2016/10/18/new-nrt-streaming/</guid> + + + <category>blog</category> + + </item> + + <item> <title>Query Metrics in Apache Kylin</title> <description><p>Apache Kylin support query metrics since 1.5.4. This blog will introduce why Kylin need query metrics, the concrete contents and meaning of query metrics, the daily function of query metrics and how to collect query metrics.</p> @@ -553,131 +622,6 @@ Check the regionserver log, there should </item> <item> - <title>Apache Kylin v1.5.2 æ£å¼åå¸</title> - <description><p>Apache Kylin社åºé常é«å ´å®£å¸Apache Kylin v1.5.2æ£å¼åå¸ã</p> - -<p>Apache Kylinæ¯ä¸ä¸ªå¼æºçåå¸å¼åæå¼æï¼æä¾Hadoopä¹ä¸çSQLæ¥è¯¢æ¥å£åå¤ç»´åæï¼OLAPï¼è½å以æ¯æè¶ å¤§è§æ¨¡æ°æ®ï¼æåç±eBay Inc. å¼å并贡ç®è³å¼æºç¤¾åºã</p> - -<p>ä¸è½½Apache Kylin v1.5.2æºä»£ç åäºè¿å¶å®è£ å ï¼<br /> -请访é®<a href="http://kylin.apache.org/cn/download/">ä¸è½½</a>页é¢.</p> - -<p>è¿æ¯ä¸ä¸ªä¸»è¦ççæ¬åå¸å¸¦æ¥äºæ´ç¨³å®ï¼å¥å£®åæ´å¥½ç®¡çççæ¬ï¼Apache Kylin社åºè§£å³äº75个issueï¼å æ¬Bugä¿®å¤ï¼åè½å¢å¼ºåä¸äºæ°ç¹æ§çã</p> - -<h2 id="section">主è¦åå</h2> - -<p><strong>æ°åè½</strong></p> - -<ul> - <li>Count Distinctå¯ä»¥å·¥ä½å¨æªå®ç°å®ä¹çç»´åº¦ä¸ <a href="https://issues.apache.org/jira/browse/KYLIN-1016">KYLIN-1016</a></li> - <li>æ¯æHiveè§å¾ä½ä¸ºLookup表 <a href="https://issues.apache.org/jira/browse/KYLIN-1077">KYLIN-1077</a></li> - <li>使Kylinè¿è¡å¨MapRç¯å¢ä¸ <a href="https://issues.apache.org/jira/browse/KYLIN-1515">KYLIN-1515</a></li> - <li>éè¿GUIä¸è½½è¯æzipå <a href="https://issues.apache.org/jira/browse/KYLIN-1600">KYLIN-1600</a></li> - <li>使Kylinæ¯æcdh5.7 <a href="https://issues.apache.org/jira/browse/KYLIN-1672">KYLIN-1672</a></li> - <li>æ¾ç¤ºæ¶é´åä½ä¸ºååºå <a href="https://issues.apache.org/jira/browse/KYLIN-1441">KYLIN-1441</a></li> -</ul> - -<p><strong>æ¹è¿</strong></p> - -<ul> - <li>å¢å¼ºé®ä»¶éç¥ <a href="https://issues.apache.org/jira/browse/KYLIN-869">KYLIN-869</a></li> - <li>HiveColumnCardinalityJobåºè¯¥ä½¿ç¨conf/kylin_job_conf.xmlä¸çé ç½® <a href="https://issues.apache.org/jira/browse/KYLIN-955">KYLIN-955</a></li> - <li>å¨éPK/FKä¸æ¯æ继æ¿ç维度 <a href="https://issues.apache.org/jira/browse/KYLIN-1313">KYLIN-1313</a></li> - <li>å¢å¼ºè½¬æ¢æ°æ®å°HFileé¶æ®µçæ§è½ <a href="https://issues.apache.org/jira/browse/KYLIN-1323">KYLIN-1323</a></li> - <li>æ½åcube/hybrid/projectç¸å ³å æ°æ®ä¿¡æ¯ä»¥ä¾¿äºè¯æ/è°è¯/å享çç¨é <a href="https://issues.apache.org/jira/browse/KYLIN-1340">KYLIN-1340</a></li> - <li>æRealizationCapacityä»3å¥é ç½®æ¹æç¹å®æ°å <a href="https://issues.apache.org/jira/browse/KYLIN-1381">KYLIN-1381</a></li> - <li>æ´å¿«æ´å¥½çååºä»¥åºå¯¹v2åå¨å¼æä¸çrpcè¶ æ¶å¼å¸¸ <a href="https://issues.apache.org/jira/browse/KYLIN-1391">KYLIN-1391</a></li> - <li>å åéæ±è¾å¤§çCubeåºè¯¥æ´æºè½å°éæ©LAYERè¿æ¯INMEMæ建ç®æ³ <a href="https://issues.apache.org/jira/browse/KYLIN-1418">KYLIN-1418</a></li> - <li>å¨GUIä¸ï¼ç»ååºæ¶é´åæ·»å ä¸ä¸ªâyyyy-MM-dd HH:MM:ssâé项 <a href="https://issues.apache.org/jira/browse/KYLIN-1432">KYLIN-1432</a></li> - <li>åºäºç¹å®åè¿è¡Cuboidåç <a href="https://issues.apache.org/jira/browse/KYLIN-1453">KYLIN-1453</a></li> - <li>æ·»å è¶ é¾æ¥ä»ç»æ°çAggregation Group <a href="https://issues.apache.org/jira/browse/KYLIN-1487">KYLIN-1487</a></li> - <li>ææ¥è¯¢ç¼åè°æ´å°æ¥è¯¢æ§å¶å¨çº§å« <a href="https://issues.apache.org/jira/browse/KYLIN-1526">KYLIN-1526</a></li> - <li>Hfileææè ä¸æ¯hbase <a href="https://issues.apache.org/jira/browse/KYLIN-1542">KYLIN-1542</a></li> - <li>使hbaseç¼ç åblock sizeåhbaseå缩ä¸æ ·å¯é ç½® <a href="https://issues.apache.org/jira/browse/KYLIN-1544">KYLIN-1544</a></li> - <li>éæv2åå¨å¼æ使ä¹å¯¹æ©å±æ´å å好 <a href="https://issues.apache.org/jira/browse/KYLIN-1561">KYLIN-1561</a></li> - <li>为in-memoryæ建任å¡æ·»å 并使ç¨ä¸ä¸ªåç¬kylin_job_conf.xml <a href="https://issues.apache.org/jira/browse/KYLIN-1566">KYLIN-1566</a></li> - <li>KYLIN-1557åç«¯å·¥ä½ <a href="https://issues.apache.org/jira/browse/KYLIN-1567">KYLIN-1567</a></li> - <li>åå©å©å¨çº¿ç¨å¨è¶ æ¶åèªå¨åæ¢ <a href="https://issues.apache.org/jira/browse/KYLIN-1578">KYLIN-1578</a></li> - <li>ITæµè¯å¦BuildCubeWithEngineççåå¤é¶æ®µåºè¯¥å¨åºç°å¼å¸¸åæ¥ééåº <a href="https://issues.apache.org/jira/browse/KYLIN-1579">KYLIN-1579</a></li> - <li>å¨Fact distinctçMRä»»å¡ä¸ç¨1个åè代æ¿8åèä½ä¸ºåæ è¯ç¬¦ <a href="https://issues.apache.org/jira/browse/KYLIN-1580">KYLIN-1580</a></li> - <li>å¨Cubedescä¸æå®Regionååsize并使modelä¸çRealizationCapacityä» ä» ä½ä¸ºæ示 <a href="https://issues.apache.org/jira/browse/KYLIN-1584">KYLIN-1584</a></li> - <li>使MAX_HBASE_FUZZY_KEYSå¨GTScanRangePlannerä¸åå¾å¯é ç½®<a href="https://issues.apache.org/jira/browse/KYLIN-1585">KYLIN-1585</a></li> - <li>å¨CubeDesigneræ¾ç¤ºCube级å«çé ç½®è¦ç <a href="https://issues.apache.org/jira/browse/KYLIN-1587">KYLIN-1587</a></li> - <li>对äºå°çåæå¯ä»¥ä½¿ç¨ä¸åçblock size <a href="https://issues.apache.org/jira/browse/KYLIN-1591">KYLIN-1591</a></li> - <li>å¨Rowkeyé¢æ¿æ·»å âisShardByâæ å¿ <a href="https://issues.apache.org/jira/browse/KYLIN-1599">KYLIN-1599</a></li> - <li>å¨hbaseè¡å¾å¤§çæ¶åä¸éè¦ç¼©å°æ«æç¼å <a href="https://issues.apache.org/jira/browse/KYLIN-1601">KYLIN-1601</a></li> - <li>ç¨æ·åºè¯¥å¯ä»¥å°å¤hbase使ç¨æ åµåå©è¯æ <a href="https://issues.apache.org/jira/browse/KYLIN-1602">KYLIN-1602</a></li> - <li>为è¯æå·¥å ·æ·»å æ´å¤ä¿¡æ¯ <a href="https://issues.apache.org/jira/browse/KYLIN-1614">KYLIN-1614</a></li> - <li>å¨åå¤çå¨ä¸ä½¿ç¨1级deflateå缩 <a href="https://issues.apache.org/jira/browse/KYLIN-1621">KYLIN-1621</a></li> - <li>使æ°æ®éæ ·æ¶hll精度å¯é ç½® <a href="https://issues.apache.org/jira/browse/KYLIN-1623">KYLIN-1623</a></li> - <li>å½æå亿æ°æ®è§æ¨¡æ¶HyperLogLogPlusCounterä¼åå¾ä¸ç²¾ç¡® <a href="https://issues.apache.org/jira/browse/KYLIN-1624">KYLIN-1624</a></li> - <li>GCæ¥å¿å¨éå¯åè¦çèæ件 <a href="https://issues.apache.org/jira/browse/KYLIN-1625">KYLIN-1625</a></li> - <li>æ·»å è°è¯æ¥å£ä»¥å¯¼åºäºè¿å¶cubeåå¨æ åµä»¥å©äºæªæ¥åæ <a href="https://issues.apache.org/jira/browse/KYLIN-1627">KYLIN-1627</a></li> -</ul> - -<p><strong>Bug</strong></p> - -<ul> - <li>æ¶é´æ³å段çåå®½å¤ªå° <a href="https://issues.apache.org/jira/browse/KYLIN-989">KYLIN-989</a></li> - <li>cubeæ°æ®å¨purgeå没ææ´æ° <a href="https://issues.apache.org/jira/browse/KYLIN-1197">KYLIN-1197</a></li> - <li>ä¸è½å¨é ç½®ä¸è·åè¶ è¿ä¸ä¸ªçç³»ç»ç®¡çåé®ç®± <a href="https://issues.apache.org/jira/browse/KYLIN-1305">KYLIN-1305</a></li> - <li>åºè¯¥æ£æ¥å¹¶ç¡®ä¿topn度éå¿ é¡»æå®ä¸¤ä¸ªåæ° <a href="https://issues.apache.org/jira/browse/KYLIN-1551">KYLIN-1551</a></li> - <li>HybridInstance#init()ä¸è¿è¡éå®å ¨æ§çåå§å <a href="https://issues.apache.org/jira/browse/KYLIN-1563">KYLIN-1563</a></li> - <li>å¨GUIä¸æ·»å ä¸ä¸ªèªå®ä¹èåæ¶éæ©ä¸ä¸ªå <a href="https://issues.apache.org/jira/browse/KYLIN-1569">KYLIN-1569</a></li> - <li>QueryService#getMetadata()åå¨æ²¡æå ³éçResultSet <a href="https://issues.apache.org/jira/browse/KYLIN-1574">KYLIN-1574</a></li> - <li>å¨Job engineä¸æ§è¡MRä»»å¡æ¶æ¥åºNPE <a href="https://issues.apache.org/jira/browse/KYLIN-1581">KYLIN-1581</a></li> - <li>å½ç¼è¾Cubeæ¶èåç»ä¿¡æ¯ä¼å空 <a href="https://issues.apache.org/jira/browse/KYLIN-1593">KYLIN-1593</a></li> - <li>度éåå¯ä»¥åºç°å¨filter/groupbyä¸ <a href="https://issues.apache.org/jira/browse/KYLIN-1595">KYLIN-1595</a></li> - <li>å符串ç¼ç ä¸ä¸è´å¯¼è´UT失败 <a href="https://issues.apache.org/jira/browse/KYLIN-1596">KYLIN-1596</a></li> - <li>å¨windowså¼åæºä¸è½å®æ´æ§è¡åå æµè¯ <a href="https://issues.apache.org/jira/browse/KYLIN-1598">KYLIN-1598</a></li> - <li>é¨ç½²åå¤çå¨æ¶ä¼åºç°hdfs并ååé®é¢ <a href="https://issues.apache.org/jira/browse/KYLIN-1604">KYLIN-1604</a></li> - <li>Cubeå·²ç»å°±ç»ªä½æ¯insightä¸ç表没æè®°å½ <a href="https://issues.apache.org/jira/browse/KYLIN-1612">KYLIN-1612</a></li> - <li>åå æµè¯âHiveCmdBuilderTestâå¨âtestBeelineâ失败 <a href="https://issues.apache.org/jira/browse/KYLIN-1615">KYLIN-1615</a></li> - <li>å topn度éå¼èµ·çæ¾ä¸å°realization <a href="https://issues.apache.org/jira/browse/KYLIN-1619">KYLIN-1619</a></li> - <li>sqlæ æ³æ§è¡å¹¶æ¥åºtopné误 <a href="https://issues.apache.org/jira/browse/KYLIN-1622">KYLIN-1622</a></li> - <li>TopNçé¢,âgroup byâåä¸è½ä½¿ç¨ä¸ä¸ªç»´åº¦å <a href="https://issues.apache.org/jira/browse/KYLIN-1631">KYLIN-1631</a></li> - <li>SSHClient#scpFileToLocal()ææªå ³éçOutputStream <a href="https://issues.apache.org/jira/browse/KYLIN-1634">KYLIN-1634</a></li> - <li>æ ·ä¾Cubeæ建åºé <a href="https://issues.apache.org/jira/browse/KYLIN-1637">KYLIN-1637</a></li> - <li>ToolUtil#getHBaseMetaStoreId()ä¸ææªå ³éçHBaseAdmin <a href="https://issues.apache.org/jira/browse/KYLIN-1638">KYLIN-1638</a></li> - <li>MapReduceExecutable.javaä¸ä½¿ç¨äºé误çæ¥å¿è®°å½ <a href="https://issues.apache.org/jira/browse/KYLIN-1639">KYLIN-1639</a></li> - <li>Kylinçhll计æ°å¨ænullå½åä¸ä¸ªææå¼ <a href="https://issues.apache.org/jira/browse/KYLIN-1643">KYLIN-1643</a></li> - <li>Purgeä¸ä¸ªcube并å次æ建ï¼èµ·å§æ¥æ没æ被æ´æ° <a href="https://issues.apache.org/jira/browse/KYLIN-1647">KYLIN-1647</a></li> - <li>java.io.IOException: Filesystem closed - å¨Cubeæ建第äºæ¥(MapR) <a href="https://issues.apache.org/jira/browse/KYLIN-1650">KYLIN-1650</a></li> - <li>å½æ°åâgetKylinPropertiesAsInputSteamâæ¼åé误 <a href="https://issues.apache.org/jira/browse/KYLIN-1655">KYLIN-1655</a></li> - <li>Streaming/kafkaé ç½®å表åä¸å¹é <a href="https://issues.apache.org/jira/browse/KYLIN-1660">KYLIN-1660</a></li> - <li>表åå¨å/tables/tableNameå请æ±æ å°æ¶è¢«æªæ <a href="https://issues.apache.org/jira/browse/KYLIN-1662">KYLIN-1662</a></li> - <li>å¨æ·»å steam表æ¶åºè¯¥æ£æ¥projectéæ© <a href="https://issues.apache.org/jira/browse/KYLIN-1666">KYLIN-1666</a></li> - <li>Streaming表ååºè¯¥éµä»âDB.TABLEâæ ¼å¼ <a href="https://issues.apache.org/jira/browse/KYLIN-1667">KYLIN-1667</a></li> - <li>ç¡®ä¿1.5.2å1.5.1çå æ°æ®å ¼å®¹ <a href="https://issues.apache.org/jira/browse/KYLIN-1673">KYLIN-1673</a></li> - <li>å æ°æ®æ¸ çå·¥å ·åªæ¸ çäºFINISHEDåDISCARDçä»»å¡ï¼ä½æ¯ä¸ä¸ªä»»å¡çæ£ç¡®ç¶ææ¯SUCCEED <a href="https://issues.apache.org/jira/browse/KYLIN-1678">KYLIN-1678</a></li> - <li>å½ä½¿ç¨Statementæ¶sqlä¸å å«é®å·ä¼æ¥é <a href="https://issues.apache.org/jira/browse/KYLIN-1685">KYLIN-1685</a></li> - <li>ç»ææ¾ç¤ºè¡¨æ ¼ä¸æéæ³å符 <a href="https://issues.apache.org/jira/browse/KYLIN-1688">KYLIN-1688</a></li> - <li>KylinConfigExtå¨åå¨å°æ件æ¶ä¸¢å¤±åºæ¬ä¿¡æ¯ <a href="https://issues.apache.org/jira/browse/KYLIN-1721">KYLIN-1721</a></li> - <li>IntegerDimEncå¨åå¤çå¨ä¸æåºååå¼å¸¸ <a href="https://issues.apache.org/jira/browse/KYLIN-1722">KYLIN-1722</a></li> -</ul> - -<p><strong>å级</strong></p> - -<p>该çæ¬çæ°æ®ä¸å æ°æ®ä¸v1.5.1å®å ¨å ¼å®¹ï¼ä½ä¹è®¸éè¦<a href="/docs15/howto/howto_update_coprocessor.html">æ´æ°HBaseåå¤çå¨</a>.</p> - -<p><strong>æ¯æ</strong></p> - -<p>å级å使ç¨è¿ç¨ä¸æä»»ä½é®é¢ï¼è¯·ï¼<br /> -æ交è³KylinçJIRA: <a href="https://issues.apache.org/jira/browse/KYLIN/">https://issues.apache.org/jira/browse/KYLIN/</a><br /> -æè <br /> -åéé®ä»¶å°Apache Kyliné®ä»¶å表: <a href="&#109;&#097;&#105;&#108;&#116;&#111;:&#100;&#101;&#118;&#064;&#107;&#121;&#108;&#105;&#110;&#046;&#097;&#112;&#097;&#099;&#104;&#101;&#046;&#111;&#114;&#103;">&#100;&#101;&#118;&#064;&#107;&#121;&#108;&#105;&#110;&#046;&#097;&#112;&#097;&#099;&#104;&#101;&#046;&#111;&#114;&#103;</a></p> - -<p><em>æè°¢æ¯ä¸ä½æåçåä¸åè´¡ç®!</em></p> -</description> - <pubDate>Thu, 26 May 2016 08:00:00 -0700</pubDate> - <link>http://kylin.apache.org/cn/blog/2016/05/26/release-v1.5.2/</link> - <guid isPermaLink="true">http://kylin.apache.org/cn/blog/2016/05/26/release-v1.5.2/</guid> - - - <category>blog</category> - - </item> - - <item> <title>Apache Kylin v1.5.2 Release Announcement</title> <description><p>The Apache Kylin community is pleased to announce the release of Apache Kylin v1.5.2.</p> @@ -803,74 +747,111 @@ send mail to Apache Kylin dev mailing li </item> <item> - <title>Apache Kylin v1.5.1 æ£å¼åå¸</title> - <description><p>Apache Kylin社åºé常é«å ´å®£å¸Apache Kylin v1.5.1æ£å¼åå¸ã</p> + <title>Apache Kylin v1.5.2 æ£å¼åå¸</title> + <description><p>Apache Kylin社åºé常é«å ´å®£å¸Apache Kylin v1.5.2æ£å¼åå¸ã</p> <p>Apache Kylinæ¯ä¸ä¸ªå¼æºçåå¸å¼åæå¼æï¼æä¾Hadoopä¹ä¸çSQLæ¥è¯¢æ¥å£åå¤ç»´åæï¼OLAPï¼è½å以æ¯æè¶ å¤§è§æ¨¡æ°æ®ï¼æåç±eBay Inc. å¼å并贡ç®è³å¼æºç¤¾åºã</p> -<p>ä¸è½½Apache Kylin v1.5.1æºä»£ç åäºè¿å¶å®è£ å ï¼<br /> +<p>ä¸è½½Apache Kylin v1.5.2æºä»£ç åäºè¿å¶å®è£ å ï¼<br /> 请访é®<a href="http://kylin.apache.org/cn/download/">ä¸è½½</a>页é¢.</p> -<p>è¿æ¯ä¸ä¸ªä¸»è¦ççæ¬åå¸å¸¦æ¥äºæ´ç¨³å®ï¼å¥å£®åæ´å¥½ç®¡çççæ¬ï¼Apache Kylin社åºè§£å³äº37个issueï¼å æ¬Bugä¿®å¤ï¼åè½å¢å¼ºåä¸äºæ°ç¹æ§çã</p> +<p>è¿æ¯ä¸ä¸ªä¸»è¦ççæ¬åå¸å¸¦æ¥äºæ´ç¨³å®ï¼å¥å£®åæ´å¥½ç®¡çççæ¬ï¼Apache Kylin社åºè§£å³äº75个issueï¼å æ¬Bugä¿®å¤ï¼åè½å¢å¼ºåä¸äºæ°ç¹æ§çã</p> <h2 id="section">主è¦åå</h2> <p><strong>æ°åè½</strong></p> <ul> - <li>æ¯æfact表æç»æ°æ®æ¥è¯¢ <a href="https://issues.apache.org/jira/browse/KYLIN-1122">KYLIN-1122</a></li> - <li>为TopN度éæ·»å 交äºçé¢ <a href="https://issues.apache.org/jira/browse/KYLIN-1378">KYLIN-1378</a></li> - <li>èªå®ä¹ç»´åº¦ç¼ç <a href="https://issues.apache.org/jira/browse/KYLIN-1492">KYLIN-1492</a></li> - <li>å æ°æ®ä»1.0~1.3å级å°1.5çæ ¡å¯¹ä¸ç¸å ³å·¥å · <a href="https://issues.apache.org/jira/browse/KYLIN-1495">KYLIN-1495</a></li> - <li>å¨Kylinæå¡å¯å¨æ¶æ§è¡æäºç±» <a href="https://issues.apache.org/jira/browse/KYLIN-1501">KYLIN-1501</a></li> - <li>éè¿Kylin.shæå°çæ¬ä¿¡æ¯ <a href="https://issues.apache.org/jira/browse/KYLIN-1503">KYLIN-1503</a></li> - <li>å¢å åçæµè¯èæ¬ <a href="https://issues.apache.org/jira/browse/KYLIN-1531">KYLIN-1531</a></li> - <li>Cubeç¹å®é 置以è¦çå ¨å±kylin.propertiesé ç½® <a href="https://issues.apache.org/jira/browse/KYLIN-1534">KYLIN-1534</a></li> - <li>å é¤SegmentçRESTæ¥å£ <a href="https://issues.apache.org/jira/browse/KYLIN-1540">KYLIN-1540</a></li> - <li>IntegerDimEnc, 为æ´æ°ç±»åå®ç°çèªå®ä¹ç»´åº¦ç¼ç <a href="https://issues.apache.org/jira/browse/KYLIN-1541">KYLIN-1541</a></li> - <li>æå°è¯æä¿¡æ¯çå·¥å · <a href="https://issues.apache.org/jira/browse/KYLIN-1546">KYLIN-1546</a></li> - <li>æä¹ åæè¿å æ¡ä¸è¯æ¥è¯¢ <a href="https://issues.apache.org/jira/browse/KYLIN-1550">KYLIN-1550</a></li> + <li>Count Distinctå¯ä»¥å·¥ä½å¨æªå®ç°å®ä¹çç»´åº¦ä¸ <a href="https://issues.apache.org/jira/browse/KYLIN-1016">KYLIN-1016</a></li> + <li>æ¯æHiveè§å¾ä½ä¸ºLookup表 <a href="https://issues.apache.org/jira/browse/KYLIN-1077">KYLIN-1077</a></li> + <li>使Kylinè¿è¡å¨MapRç¯å¢ä¸ <a href="https://issues.apache.org/jira/browse/KYLIN-1515">KYLIN-1515</a></li> + <li>éè¿GUIä¸è½½è¯æzipå <a href="https://issues.apache.org/jira/browse/KYLIN-1600">KYLIN-1600</a></li> + <li>使Kylinæ¯æcdh5.7 <a href="https://issues.apache.org/jira/browse/KYLIN-1672">KYLIN-1672</a></li> + <li>æ¾ç¤ºæ¶é´åä½ä¸ºååºå <a href="https://issues.apache.org/jira/browse/KYLIN-1441">KYLIN-1441</a></li> </ul> <p><strong>æ¹è¿</strong></p> <ul> - <li>使ç¨InstallShield 2015产çODBC驱å¨å®è£ å <a href="https://issues.apache.org/jira/browse/KYLIN-1490">KYLIN-1490</a></li> - <li>Cubeæè¿°åç¾å没æ被æ£ç¡®è®¡ç® <a href="https://issues.apache.org/jira/browse/KYLIN-1498">KYLIN-1498</a></li> - <li>streaming_fillgapå¼èµ·å åæº¢åº <a href="https://issues.apache.org/jira/browse/KYLIN-1500">KYLIN-1500</a></li> - <li>å½Cubeé空ï¼åªæç¾åä¸è´çCubeæè¿°åå¯ä»¥è¢«æ´æ° <a href="https://issues.apache.org/jira/browse/KYLIN-1502">KYLIN-1502</a></li> - <li>使ç¨NavigableSetåå¨Rowkey并使ç¨åç¼è¿æ»¤å¨æ£æ¥èµæºè·¯å¾ï¼ä»¥æ¿ä»£å¨tomcaté使ç¨å符串æ¯è¾ <a href="https://issues.apache.org/jira/browse/KYLIN-1504">KYLIN-1504</a></li> - <li>ç»åguava filtersåPredicates.andå½æ° <a href="https://issues.apache.org/jira/browse/KYLIN-1505">KYLIN-1505</a></li> - <li>GTFilterScanneræ§è½è°ä¼ <a href="https://issues.apache.org/jira/browse/KYLIN-1543">KYLIN-1543</a></li> - <li>å¢å¼ºaggregation group维度æ°éçæ£æ¥ <a href="https://issues.apache.org/jira/browse/KYLIN-1557">KYLIN-1557</a></li> -</ul> - -<p><strong>Bugä¿®å¤</strong></p> - -<ul> - <li>å¨æ¥è¯¢é¡µé¢éè¦ç¼ç è¾åºæ¥è¯¢çURLæ¥å¾å°æ£ç¡®ç»æ <a href="https://issues.apache.org/jira/browse/KYLIN-1373">KYLIN-1373</a></li> - <li>Kylinä»»å¡çæ§æ¥å£/kylin/api/jobså¨å¤§è§æ¨¡kyliné¨ç½²ç¯å¢ä¸è¾æ ¢ <a href="https://issues.apache.org/jira/browse/KYLIN-1434">KYLIN-1434</a></li> - <li>å½SQLä¸æå å·æ¶å¯¼åºcsvæ¥é <a href="https://issues.apache.org/jira/browse/KYLIN-1472">KYLIN-1472</a></li> - <li>java.lang.IllegalArgumentException: NumberDictionaryä¸æ太å¤ä½ <a href="https://issues.apache.org/jira/browse/KYLIN-1486">KYLIN-1486</a></li> - <li>å½æ²¡æaggregation groupå¹é æ¶åºè¿åbase cuboid <a href="https://issues.apache.org/jira/browse/KYLIN-1491">KYLIN-1491</a></li> - <li>使ExecutableManager.getInstance线ç¨å®å ¨ <a href="https://issues.apache.org/jira/browse/KYLIN-1493">KYLIN-1493</a></li> - <li>使ä¸ä¸ªclass.getInstance线ç¨å®å ¨ <a href="https://issues.apache.org/jira/browse/KYLIN-1497">KYLIN-1497</a></li> - <li>å¨CDNçæäºå¹³å°ä¸ä¸è½æ£ç¡®æ¾å°hiveä¾èµ <a href="https://issues.apache.org/jira/browse/KYLIN-1507">KYLIN-1507</a></li> - <li>跨天çæ¶é´ååºå·¥ä½ä¸æ£å¸¸ <a href="https://issues.apache.org/jira/browse/KYLIN-1513">KYLIN-1513</a></li> - <li>æå æ¶TomcatçMD5æ ¡éªå·¥ä½ä¸æ£å¸¸ <a href="https://issues.apache.org/jira/browse/KYLIN-1514">KYLIN-1514</a></li> - <li>å½Cube段çèµ·å§æ¶é´å¨1970-01-01åä¸è½å·æ°è¯¥æ®µ <a href="https://issues.apache.org/jira/browse/KYLIN-1521">KYLIN-1521</a></li> - <li>å½ä»cache读åæ¶HLLCç»æå¼å¸¸ <a href="https://issues.apache.org/jira/browse/KYLIN-1522">KYLIN-1522</a></li> - <li>å½Top-N度éç±»åæ¯BigIntæ¶æ¥éâjava.lang.Double cannot be cast to java.lang.Longâ <a href="https://issues.apache.org/jira/browse/KYLIN-1524">KYLIN-1524</a></li> - <li>å ¨é¨æ¯NULLçåä¸è½è¢«æ¥è¯¢ <a href="https://issues.apache.org/jira/browse/KYLIN-1527">KYLIN-1527</a></li> - <li>å½ååè¿é¿æ¶ï¼ä¸è½çæhive平表 <a href="https://issues.apache.org/jira/browse/KYLIN-1537">KYLIN-1537</a></li> - <li>DoubleDeltaSerializerå¨åºåååååºåååä¼åºé <a href="https://issues.apache.org/jira/browse/KYLIN-1538">KYLIN-1538</a></li> - <li>å¨Cubeæè¿°åä¸æ¾ä¸å°RowkeyåâCOL_NAMEâ <a href="https://issues.apache.org/jira/browse/KYLIN-1553">KYLIN-1553</a></li> - <li>Tableå¨BuildCubeWithEngine#checkHFilesInHBase()æªå ³é <a href="https://issues.apache.org/jira/browse/KYLIN-1564">KYLIN-1564</a></li> + <li>å¢å¼ºé®ä»¶éç¥ <a href="https://issues.apache.org/jira/browse/KYLIN-869">KYLIN-869</a></li> + <li>HiveColumnCardinalityJobåºè¯¥ä½¿ç¨conf/kylin_job_conf.xmlä¸çé ç½® <a href="https://issues.apache.org/jira/browse/KYLIN-955">KYLIN-955</a></li> + <li>å¨éPK/FKä¸æ¯æ继æ¿ç维度 <a href="https://issues.apache.org/jira/browse/KYLIN-1313">KYLIN-1313</a></li> + <li>å¢å¼ºè½¬æ¢æ°æ®å°HFileé¶æ®µçæ§è½ <a href="https://issues.apache.org/jira/browse/KYLIN-1323">KYLIN-1323</a></li> + <li>æ½åcube/hybrid/projectç¸å ³å æ°æ®ä¿¡æ¯ä»¥ä¾¿äºè¯æ/è°è¯/å享çç¨é <a href="https://issues.apache.org/jira/browse/KYLIN-1340">KYLIN-1340</a></li> + <li>æRealizationCapacityä»3å¥é ç½®æ¹æç¹å®æ°å <a href="https://issues.apache.org/jira/browse/KYLIN-1381">KYLIN-1381</a></li> + <li>æ´å¿«æ´å¥½çååºä»¥åºå¯¹v2åå¨å¼æä¸çrpcè¶ æ¶å¼å¸¸ <a href="https://issues.apache.org/jira/browse/KYLIN-1391">KYLIN-1391</a></li> + <li>å åéæ±è¾å¤§çCubeåºè¯¥æ´æºè½å°éæ©LAYERè¿æ¯INMEMæ建ç®æ³ <a href="https://issues.apache.org/jira/browse/KYLIN-1418">KYLIN-1418</a></li> + <li>å¨GUIä¸ï¼ç»ååºæ¶é´åæ·»å ä¸ä¸ªâyyyy-MM-dd HH:MM:ssâé项 <a href="https://issues.apache.org/jira/browse/KYLIN-1432">KYLIN-1432</a></li> + <li>åºäºç¹å®åè¿è¡Cuboidåç <a href="https://issues.apache.org/jira/browse/KYLIN-1453">KYLIN-1453</a></li> + <li>æ·»å è¶ é¾æ¥ä»ç»æ°çAggregation Group <a href="https://issues.apache.org/jira/browse/KYLIN-1487">KYLIN-1487</a></li> + <li>ææ¥è¯¢ç¼åè°æ´å°æ¥è¯¢æ§å¶å¨çº§å« <a href="https://issues.apache.org/jira/browse/KYLIN-1526">KYLIN-1526</a></li> + <li>Hfileææè ä¸æ¯hbase <a href="https://issues.apache.org/jira/browse/KYLIN-1542">KYLIN-1542</a></li> + <li>使hbaseç¼ç åblock sizeåhbaseå缩ä¸æ ·å¯é ç½® <a href="https://issues.apache.org/jira/browse/KYLIN-1544">KYLIN-1544</a></li> + <li>éæv2åå¨å¼æ使ä¹å¯¹æ©å±æ´å å好 <a href="https://issues.apache.org/jira/browse/KYLIN-1561">KYLIN-1561</a></li> + <li>为in-memoryæ建任å¡æ·»å 并使ç¨ä¸ä¸ªåç¬kylin_job_conf.xml <a href="https://issues.apache.org/jira/browse/KYLIN-1566">KYLIN-1566</a></li> + <li>KYLIN-1557åç«¯å·¥ä½ <a href="https://issues.apache.org/jira/browse/KYLIN-1567">KYLIN-1567</a></li> + <li>åå©å©å¨çº¿ç¨å¨è¶ æ¶åèªå¨åæ¢ <a href="https://issues.apache.org/jira/browse/KYLIN-1578">KYLIN-1578</a></li> + <li>ITæµè¯å¦BuildCubeWithEngineççåå¤é¶æ®µåºè¯¥å¨åºç°å¼å¸¸åæ¥ééåº <a href="https://issues.apache.org/jira/browse/KYLIN-1579">KYLIN-1579</a></li> + <li>å¨Fact distinctçMRä»»å¡ä¸ç¨1个åè代æ¿8åèä½ä¸ºåæ è¯ç¬¦ <a href="https://issues.apache.org/jira/browse/KYLIN-1580">KYLIN-1580</a></li> + <li>å¨Cubedescä¸æå®Regionååsize并使modelä¸çRealizationCapacityä» ä» ä½ä¸ºæ示 <a href="https://issues.apache.org/jira/browse/KYLIN-1584">KYLIN-1584</a></li> + <li>使MAX_HBASE_FUZZY_KEYSå¨GTScanRangePlannerä¸åå¾å¯é ç½®<a href="https://issues.apache.org/jira/browse/KYLIN-1585">KYLIN-1585</a></li> + <li>å¨CubeDesigneræ¾ç¤ºCube级å«çé ç½®è¦ç <a href="https://issues.apache.org/jira/browse/KYLIN-1587">KYLIN-1587</a></li> + <li>对äºå°çåæå¯ä»¥ä½¿ç¨ä¸åçblock size <a href="https://issues.apache.org/jira/browse/KYLIN-1591">KYLIN-1591</a></li> + <li>å¨Rowkeyé¢æ¿æ·»å âisShardByâæ å¿ <a href="https://issues.apache.org/jira/browse/KYLIN-1599">KYLIN-1599</a></li> + <li>å¨hbaseè¡å¾å¤§çæ¶åä¸éè¦ç¼©å°æ«æç¼å <a href="https://issues.apache.org/jira/browse/KYLIN-1601">KYLIN-1601</a></li> + <li>ç¨æ·åºè¯¥å¯ä»¥å°å¤hbase使ç¨æ åµåå©è¯æ <a href="https://issues.apache.org/jira/browse/KYLIN-1602">KYLIN-1602</a></li> + <li>为è¯æå·¥å ·æ·»å æ´å¤ä¿¡æ¯ <a href="https://issues.apache.org/jira/browse/KYLIN-1614">KYLIN-1614</a></li> + <li>å¨åå¤çå¨ä¸ä½¿ç¨1级deflateå缩 <a href="https://issues.apache.org/jira/browse/KYLIN-1621">KYLIN-1621</a></li> + <li>使æ°æ®éæ ·æ¶hll精度å¯é ç½® <a href="https://issues.apache.org/jira/browse/KYLIN-1623">KYLIN-1623</a></li> + <li>å½æå亿æ°æ®è§æ¨¡æ¶HyperLogLogPlusCounterä¼åå¾ä¸ç²¾ç¡® <a href="https://issues.apache.org/jira/browse/KYLIN-1624">KYLIN-1624</a></li> + <li>GCæ¥å¿å¨éå¯åè¦çèæ件 <a href="https://issues.apache.org/jira/browse/KYLIN-1625">KYLIN-1625</a></li> + <li>æ·»å è°è¯æ¥å£ä»¥å¯¼åºäºè¿å¶cubeåå¨æ åµä»¥å©äºæªæ¥åæ <a href="https://issues.apache.org/jira/browse/KYLIN-1627">KYLIN-1627</a></li> +</ul> + +<p><strong>Bug</strong></p> + +<ul> + <li>æ¶é´æ³å段çåå®½å¤ªå° <a href="https://issues.apache.org/jira/browse/KYLIN-989">KYLIN-989</a></li> + <li>cubeæ°æ®å¨purgeå没ææ´æ° <a href="https://issues.apache.org/jira/browse/KYLIN-1197">KYLIN-1197</a></li> + <li>ä¸è½å¨é ç½®ä¸è·åè¶ è¿ä¸ä¸ªçç³»ç»ç®¡çåé®ç®± <a href="https://issues.apache.org/jira/browse/KYLIN-1305">KYLIN-1305</a></li> + <li>åºè¯¥æ£æ¥å¹¶ç¡®ä¿topn度éå¿ é¡»æå®ä¸¤ä¸ªåæ° <a href="https://issues.apache.org/jira/browse/KYLIN-1551">KYLIN-1551</a></li> + <li>HybridInstance#init()ä¸è¿è¡éå®å ¨æ§çåå§å <a href="https://issues.apache.org/jira/browse/KYLIN-1563">KYLIN-1563</a></li> + <li>å¨GUIä¸æ·»å ä¸ä¸ªèªå®ä¹èåæ¶éæ©ä¸ä¸ªå <a href="https://issues.apache.org/jira/browse/KYLIN-1569">KYLIN-1569</a></li> + <li>QueryService#getMetadata()åå¨æ²¡æå ³éçResultSet <a href="https://issues.apache.org/jira/browse/KYLIN-1574">KYLIN-1574</a></li> + <li>å¨Job engineä¸æ§è¡MRä»»å¡æ¶æ¥åºNPE <a href="https://issues.apache.org/jira/browse/KYLIN-1581">KYLIN-1581</a></li> + <li>å½ç¼è¾Cubeæ¶èåç»ä¿¡æ¯ä¼å空 <a href="https://issues.apache.org/jira/browse/KYLIN-1593">KYLIN-1593</a></li> + <li>度éåå¯ä»¥åºç°å¨filter/groupbyä¸ <a href="https://issues.apache.org/jira/browse/KYLIN-1595">KYLIN-1595</a></li> + <li>å符串ç¼ç ä¸ä¸è´å¯¼è´UT失败 <a href="https://issues.apache.org/jira/browse/KYLIN-1596">KYLIN-1596</a></li> + <li>å¨windowså¼åæºä¸è½å®æ´æ§è¡åå æµè¯ <a href="https://issues.apache.org/jira/browse/KYLIN-1598">KYLIN-1598</a></li> + <li>é¨ç½²åå¤çå¨æ¶ä¼åºç°hdfs并ååé®é¢ <a href="https://issues.apache.org/jira/browse/KYLIN-1604">KYLIN-1604</a></li> + <li>Cubeå·²ç»å°±ç»ªä½æ¯insightä¸ç表没æè®°å½ <a href="https://issues.apache.org/jira/browse/KYLIN-1612">KYLIN-1612</a></li> + <li>åå æµè¯âHiveCmdBuilderTestâå¨âtestBeelineâ失败 <a href="https://issues.apache.org/jira/browse/KYLIN-1615">KYLIN-1615</a></li> + <li>å topn度éå¼èµ·çæ¾ä¸å°realization <a href="https://issues.apache.org/jira/browse/KYLIN-1619">KYLIN-1619</a></li> + <li>sqlæ æ³æ§è¡å¹¶æ¥åºtopné误 <a href="https://issues.apache.org/jira/browse/KYLIN-1622">KYLIN-1622</a></li> + <li>TopNçé¢,âgroup byâåä¸è½ä½¿ç¨ä¸ä¸ªç»´åº¦å <a href="https://issues.apache.org/jira/browse/KYLIN-1631">KYLIN-1631</a></li> + <li>SSHClient#scpFileToLocal()ææªå ³éçOutputStream <a href="https://issues.apache.org/jira/browse/KYLIN-1634">KYLIN-1634</a></li> + <li>æ ·ä¾Cubeæ建åºé <a href="https://issues.apache.org/jira/browse/KYLIN-1637">KYLIN-1637</a></li> + <li>ToolUtil#getHBaseMetaStoreId()ä¸ææªå ³éçHBaseAdmin <a href="https://issues.apache.org/jira/browse/KYLIN-1638">KYLIN-1638</a></li> + <li>MapReduceExecutable.javaä¸ä½¿ç¨äºé误çæ¥å¿è®°å½ <a href="https://issues.apache.org/jira/browse/KYLIN-1639">KYLIN-1639</a></li> + <li>Kylinçhll计æ°å¨ænullå½åä¸ä¸ªææå¼ <a href="https://issues.apache.org/jira/browse/KYLIN-1643">KYLIN-1643</a></li> + <li>Purgeä¸ä¸ªcube并å次æ建ï¼èµ·å§æ¥æ没æ被æ´æ° <a href="https://issues.apache.org/jira/browse/KYLIN-1647">KYLIN-1647</a></li> + <li>java.io.IOException: Filesystem closed - å¨Cubeæ建第äºæ¥(MapR) <a href="https://issues.apache.org/jira/browse/KYLIN-1650">KYLIN-1650</a></li> + <li>å½æ°åâgetKylinPropertiesAsInputSteamâæ¼åé误 <a href="https://issues.apache.org/jira/browse/KYLIN-1655">KYLIN-1655</a></li> + <li>Streaming/kafkaé ç½®å表åä¸å¹é <a href="https://issues.apache.org/jira/browse/KYLIN-1660">KYLIN-1660</a></li> + <li>表åå¨å/tables/tableNameå请æ±æ å°æ¶è¢«æªæ <a href="https://issues.apache.org/jira/browse/KYLIN-1662">KYLIN-1662</a></li> + <li>å¨æ·»å steam表æ¶åºè¯¥æ£æ¥projectéæ© <a href="https://issues.apache.org/jira/browse/KYLIN-1666">KYLIN-1666</a></li> + <li>Streaming表ååºè¯¥éµä»âDB.TABLEâæ ¼å¼ <a href="https://issues.apache.org/jira/browse/KYLIN-1667">KYLIN-1667</a></li> + <li>ç¡®ä¿1.5.2å1.5.1çå æ°æ®å ¼å®¹ <a href="https://issues.apache.org/jira/browse/KYLIN-1673">KYLIN-1673</a></li> + <li>å æ°æ®æ¸ çå·¥å ·åªæ¸ çäºFINISHEDåDISCARDçä»»å¡ï¼ä½æ¯ä¸ä¸ªä»»å¡çæ£ç¡®ç¶ææ¯SUCCEED <a href="https://issues.apache.org/jira/browse/KYLIN-1678">KYLIN-1678</a></li> + <li>å½ä½¿ç¨Statementæ¶sqlä¸å å«é®å·ä¼æ¥é <a href="https://issues.apache.org/jira/browse/KYLIN-1685">KYLIN-1685</a></li> + <li>ç»ææ¾ç¤ºè¡¨æ ¼ä¸æéæ³å符 <a href="https://issues.apache.org/jira/browse/KYLIN-1688">KYLIN-1688</a></li> + <li>KylinConfigExtå¨åå¨å°æ件æ¶ä¸¢å¤±åºæ¬ä¿¡æ¯ <a href="https://issues.apache.org/jira/browse/KYLIN-1721">KYLIN-1721</a></li> + <li>IntegerDimEncå¨åå¤çå¨ä¸æåºååå¼å¸¸ <a href="https://issues.apache.org/jira/browse/KYLIN-1722">KYLIN-1722</a></li> </ul> <p><strong>å级</strong></p> -<p>该çæ¬çæ°æ®ä¸å æ°æ®ä¸v1.5.0å®å ¨å ¼å®¹ï¼ä½ä¹è®¸éè¦<a href="/docs15/howto/howto_update_coprocessor.html">æ´æ°HBaseåå¤çå¨</a>.</p> +<p>该çæ¬çæ°æ®ä¸å æ°æ®ä¸v1.5.1å®å ¨å ¼å®¹ï¼ä½ä¹è®¸éè¦<a href="/docs15/howto/howto_update_coprocessor.html">æ´æ°HBaseåå¤çå¨</a>.</p> <p><strong>æ¯æ</strong></p> @@ -881,9 +862,9 @@ send mail to Apache Kylin dev mailing li <p><em>æè°¢æ¯ä¸ä½æåçåä¸åè´¡ç®!</em></p> </description> - <pubDate>Tue, 12 Apr 2016 04:00:00 -0700</pubDate> - <link>http://kylin.apache.org/cn/blog/2016/04/12/release-v1.5.1/</link> - <guid isPermaLink="true">http://kylin.apache.org/cn/blog/2016/04/12/release-v1.5.1/</guid> + <pubDate>Thu, 26 May 2016 08:00:00 -0700</pubDate> + <link>http://kylin.apache.org/cn/blog/2016/05/26/release-v1.5.2/</link> + <guid isPermaLink="true">http://kylin.apache.org/cn/blog/2016/05/26/release-v1.5.2/</guid> <category>blog</category> Added: kylin/site/images/blog/new-streaming.png URL: http://svn.apache.org/viewvc/kylin/site/images/blog/new-streaming.png?rev=1765396&view=auto ============================================================================== Binary file - no diff available. Propchange: kylin/site/images/blog/new-streaming.png ------------------------------------------------------------------------------ svn:mime-type = application/octet-stream Added: kylin/site/images/blog/offset-as-partition-value.png URL: http://svn.apache.org/viewvc/kylin/site/images/blog/offset-as-partition-value.png?rev=1765396&view=auto ============================================================================== Binary file - no diff available. Propchange: kylin/site/images/blog/offset-as-partition-value.png ------------------------------------------------------------------------------ svn:mime-type = application/octet-stream Added: kylin/site/images/blog/streaming-adapter.png URL: http://svn.apache.org/viewvc/kylin/site/images/blog/streaming-adapter.png?rev=1765396&view=auto ============================================================================== Binary file - no diff available. Propchange: kylin/site/images/blog/streaming-adapter.png ------------------------------------------------------------------------------ svn:mime-type = application/octet-stream Added: kylin/site/images/blog/streaming-monitor.png URL: http://svn.apache.org/viewvc/kylin/site/images/blog/streaming-monitor.png?rev=1765396&view=auto ============================================================================== Binary file - no diff available. Propchange: kylin/site/images/blog/streaming-monitor.png ------------------------------------------------------------------------------ svn:mime-type = application/octet-stream Added: kylin/site/images/blog/streaming-twitter.png URL: http://svn.apache.org/viewvc/kylin/site/images/blog/streaming-twitter.png?rev=1765396&view=auto ============================================================================== Binary file - no diff available. Propchange: kylin/site/images/blog/streaming-twitter.png ------------------------------------------------------------------------------ svn:mime-type = application/octet-stream