Author: mahongbin
Date: Wed Feb 3 08:35:20 2016
New Revision: 1728270
URL: http://svn.apache.org/viewvc?rev=1728270&view=rev
Log:
UPDATE MESSAGE
Added:
kylin/site/blog/2016/
kylin/site/blog/2016/02/
kylin/site/blog/2016/02/03/
kylin/site/blog/2016/02/03/streaming-cubing/
kylin/site/blog/2016/02/03/streaming-cubing/index.html
Modified:
kylin/site/blog/index.html
kylin/site/development/about_temp_files.html
kylin/site/development/dev_env.html
kylin/site/development/howto_contribute.html
kylin/site/development/howto_docs.html
kylin/site/development/howto_package.html
kylin/site/development/howto_release.html
kylin/site/development/howto_test.html
kylin/site/development/index.html
kylin/site/development/new_metadata.html
kylin/site/development/web_tech.html
kylin/site/feed.xml
Added: kylin/site/blog/2016/02/03/streaming-cubing/index.html
URL:
http://svn.apache.org/viewvc/kylin/site/blog/2016/02/03/streaming-cubing/index.html?rev=1728270&view=auto
==============================================================================
--- kylin/site/blog/2016/02/03/streaming-cubing/index.html (added)
+++ kylin/site/blog/2016/02/03/streaming-cubing/index.html Wed Feb 3 08:35:20
2016
@@ -0,0 +1,272 @@
+<!--
+* Licensed to the Apache Software Foundation (ASF) under one
+* or more contributor license agreements. See the NOTICE file
+* distributed with this work for additional information
+* regarding copyright ownership. The ASF licenses this file
+* to you under the Apache License, Version 2.0 (the
+* "License"); you may not use this file except in compliance
+* with the License. You may obtain a copy of the License at
+*
+* http://www.apache.org/licenses/LICENSE-2.0
+*
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+-->
+<!doctype html>
+<html>
+ <!--
+* Licensed to the Apache Software Foundation (ASF) under one
+* or more contributor license agreements. See the NOTICE file
+* distributed with this work for additional information
+* regarding copyright ownership. The ASF licenses this file
+* to you under the Apache License, Version 2.0 (the
+* "License"); you may not use this file except in compliance
+* with the License. You may obtain a copy of the License at
+*
+* http://www.apache.org/licenses/LICENSE-2.0
+*
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+-->
+
+<head>
+ <meta charset="utf-8">
+ <meta http-equiv="X-UA-Compatible" content="IE=edge">
+ <meta name="viewport" content="width=device-width, initial-scale=1">
+
+ <title>Apache Kylin | Streaming cubing (Prototype)</title>
+ <meta name="description" content="One of the most important features in 2.x
branches is streaming cubing which enables OLAP analysis on streaming data.
Streaming cubing delivers faster insigh...">
+ <meta name="author" content="Apache Kylin">
+ <link rel="shortcut icon" href="fav.png" type="image/png">
+
+
+
+<link rel="stylesheet" href="/assets/css/animate.css">
+<!-- Bootstrap -->
+<link rel="stylesheet" href="/assets/css/bootstrap.min.css">
+
+<!-- Fonts -->
+<!-- <link rel="stylesheet"
href="http://fonts.googleapis.com/css?family=Alice|Open+Sans:400,300,700"> -->
+
+<!-- Icons -->
+<link rel="stylesheet" href="/assets/css/font-awesome.min.css">
+
+ <!-- Custom styles -->
+ <link rel="stylesheet" href="/assets/css/styles.css">
+ <link rel="stylesheet" href="/assets/css/docs.css">
+ <link rel="stylesheet" href="/assets/css/pygments.css">
+
+ <link rel="canonical"
href="http://kylin.apache.org/blog/2016/02/03/streaming-cubing/";>
+ <link rel="alternate" type="application/rss+xml" title="Apache Kylin"
href="http://kylin.apache.org/feed.xml"; />
+
+<!--[if lt IE 9]> <script src="assets/js/html5shiv.js"></script> <![endif]-->
+<script>
+ (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
+ (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new
Date();a=s.createElement(o),
+
m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
+ })(window,document,'script','//www.google-analytics.com/analytics.js','ga');
+
+ //oringal tracker for kylin.io
+ ga('create', 'UA-55534813-1', 'auto');
+ //new tracker for kylin.apache.org
+ ga('create', 'UA-55534813-2', 'auto', {'name':'toplevel'});
+
+ ga('send', 'pageview');
+ ga('toplevel.send', 'pageview');
+
+
+</script>
+<script type="text/javascript" src="/assets/js/jquery-1.9.1.min.js"></script>
+<script type="text/javascript" src="/assets/js/nside.js"></script> </script>
+<script type="text/javascript" src="/assets/js/nnav.js"></script> </script>
+</head>
+
+ <body>
+ <!--
+* Licensed to the Apache Software Foundation (ASF) under one
+* or more contributor license agreements. See the NOTICE file
+* distributed with this work for additional information
+* regarding copyright ownership. The ASF licenses this file
+* to you under the Apache License, Version 2.0 (the
+* "License"); you may not use this file except in compliance
+* with the License. You may obtain a copy of the License at
+*
+* http://www.apache.org/licenses/LICENSE-2.0
+*
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+-->
+
+<header id="header" >
+
+ <div id="head" class="parallax" parallax-speed="3" >
+ <div id="logo" class="text-center"> <img class="img-circle"
id="circlelogo" src="/assets/images/kylin_logo.jpg"> <span class="title"
>Apache Kylinâ¢</span> <span class="tagline">Extreme OLAP Engine for Big
Data</span>
+ </div>
+ </div>
+
+
+ <!-- Main Menu -->
+ <nav class="navbar navbar-default" role="navigation" id="nav-wrapper">
+ <div class="container-fluid" id="nav">
+ <!--
+ <img class="img-circle" width="40px" height="40px" id="circlelogo"
src="/assets/images/kylin_logo.jpg">
+ -->
+ <!-- Brand and toggle get grouped for better mobile display -->
+ <div class="navbar-header">
+ <button type="button" class="navbar-toggle collapsed"
data-toggle="collapse" data-target="#bs-example-navbar-collapse-1">
+ <span class="sr-only">Toggle navigation</span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ </button>
+
+ </div>
+
+ <!-- Collect the nav links, forms, and other content for toggling -->
+ <div class="collapse navbar-collapse" id="bs-example-navbar-collapse-1">
+ <ul class="nav navbar-nav">
+ <li><a href="/">Home</a></li>
+ <li><a href="/docs" >Docs</a></li>
+ <li><a href="/download">Download</li>
+ <li><a href="/community" >Community</a></li>
+ <li><a href="/development" >Development</a></li>
+ <li><a href="/blog">Blog</li>
+ <li><a href="/cn" >ä¸æç</a></li>
+ <li><a href="https://twitter.com/apachekylin"; target="_blank"
class="fa fa-twitter fa-lg" title="Twitter: @ApacheKylin" ></a></li>
+ <li><a href="https://github.com/apache/kylin"; target="_blank"
class="fa fa-github-alt fa-lg" title="Github: apache/kylin" ></a></li>
+ <li><a href="https://www.facebook.com/kylinio"; target="_blank"
class="fa fa-facebook fa-lg" title="Facebook: kylin.io" ></a></li>
+ </ul>
+ </div><!-- /.navbar-collapse -->
+ </div><!-- /.container-fluid -->
+</nav>
+ </header>
+
+ <div class="page-content">
+ <header style=" padding:2em 0 0 0">
+ <div class="container" >
+ <h4 class="section-title"><span>Apache Kylinâ¢
Technical Blog</span></h4>
+ </div>
+ </div>
+
+ <div class="container">
+ <div>
+ <article class="post-content" >
+ <!--
+* Licensed to the Apache Software Foundation (ASF) under one
+* or more contributor license agreements. See the NOTICE file
+* distributed with this work for additional information
+* regarding copyright ownership. The ASF licenses this file
+* to you under the Apache License, Version 2.0 (the
+* "License"); you may not use this file except in compliance
+* with the License. You may obtain a copy of the License at
+*
+* http://www.apache.org/licenses/LICENSE-2.0
+*
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+-->
+
+<div class="post" style=" padding:2em 4em 4em 4em">
+
+ <header class="post-header">
+ <h1 class="post-title">Streaming cubing (Prototype)</h1>
+ <p class="post-meta" >Feb 3, 2016 ⢠Hongbin Ma</p>
+ </header>
+
+ <article class="post-content" >
+ <p>One of the most important features in 2.x branches is streaming cubing
which enables OLAP analysis on streaming data. Streaming cubing delivers faster
insights on the data to help more promptly business decisions. Even though
there are already many real time analysis tools in open source community, Kylin
Streaming cubing still differs from them in multiple angles:</p>
+
+<p>Firstly, Kylin Streaming Cubing aligns with Kylin traditional cubing to
provided unified, ANSI SQL interface. Actually Kylin Streaming shares the
storage engine and query engine with traditional Kylin cubes, so in theory all
of the optimization techniques to save storage and speed up query performance
can also be applied on streaming cubes. Besides, all the supported
aggregations/filters/UDFs still work for streaming cubes. By unifying the
storage engine and query engine we also get freed from double amount of
maintaince work.</p>
+
+<p>Secondly, Kylin Streaming Cubing does not require large amount of memory to
store real time data, nor does it attempts to provide truly âreal timeâ
analysis. By our customer survey we found that minutes of visualization latency
is acceptable for OLAP analysts. So our streaming cubing adopts the micro batch
approach. Incoming streaming data are partitioned into different time windows
and we build a micro batch for each time window. The cube output for each micro
batch is directly saved to HBase. The query engine goes to HBase for data
retrieving instead of the data ingestion server. The benefit of such design is
that we donât have to maintain large amount of in-memory index which could
easily require tens of gigabytes of memory. In the future Kylin might need to
consider truly âreal timeâ support, too.</p>
+
+<p>Thirdly, Kylin Streaming Cubing data will be persistent and gradually be
converted to traditional cubes, thus customers can still query âcold dataâ
without any compromise on performance. As discussed above the output of
streaming cubing is directly saved to HBase as a new segment. The traditional
job engine will be notified of the new segment and take over to schedule merge
jobs when then segments accumulates. Day after day the segments of the
streaming cube got merged and become a very large traditional cube.</p>
+
+<p><img src="/images/develop/streaming.png" alt="Kylin Streaming Framework
Architecture" /></p>
+
+<p>With the major difference in mind we will introduce the modules for Kylin
Streaming cubing. Kylin Streaming cubing consist of three major parts:</p>
+
+<ul>
+ <li>Streaming Input to retrieve data from a replayable data queue (currently
it is Kafka) within given time window. Streaming Input is also responsible for
primary data cleaning and normalization. By default Kylin Streaming provides a
default implementation to parse the messages from the source queue. Customers
can choose to configure the parser or provide a brand new one based on their
requirements.</li>
+ <li>Streaming Batch Ingestion to ingest the incoming data batch and
transform it into a micro cube. Thanks to the latest Kylin In-memory cubing
technology, this step is now times faster and space-saving than previous. The
micro cube is directly saved to HBase.</li>
+ <li>Job Scheduling Module to trigger Streaming Batch Ingestion. Kylin does
not put too much efforts in job scheduling, streaming cubing is not a
exception. Currently we provided a simple implementation based on Linux
Crontab.</li>
+</ul>
+
+<p>Weâll publish more detailed documents on how to use Kylin Streaming soon.
In latest 2.x branch we are also working on more complicated load balancing
schemes for streaming cubing. Please stay tuned.</p>
+
+
+ </article>
+
+</div>
+
+
+
+
+
+ </article>
+ </div>
+ </div>
+ <!--
+* Licensed to the Apache Software Foundation (ASF) under one
+* or more contributor license agreements. See the NOTICE file
+* distributed with this work for additional information
+* regarding copyright ownership. The ASF licenses this file
+* to you under the Apache License, Version 2.0 (the
+* "License"); you may not use this file except in compliance
+* with the License. You may obtain a copy of the License at
+*
+* http://www.apache.org/licenses/LICENSE-2.0
+*
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+-->
+
+<footer id="underfooter">
+ <div class="container">
+ <div class="row">
+ <div class="col-md-12 widget">
+ <div class="widget-body" style="text-align:center">
+ <a href="http://www.apache.org";>
+ <img id="asf-logo" alt="Apache Software Foundation"
src="/assets/images/feather-small.gif">
+ </a>
+
+ <div>
+ The contents of this website are © 2015 Apache
Software Foundation under the terms of the <a
+ href="http://www.apache.org/licenses/LICENSE-2.0";>
Apache License v2 </a>. Apache Kylin and
+ its logo are trademarks of the Apache Software
Foundation.
+ </div>
+
+ </div>
+ </div>
+ </div>
+ <!-- /row of widgets -->
+
+ </div>
+ <div></div>
+
+</footer>
+
+ <script src="/assets/js/jquery-1.9.1.min.js"></script>
+ <script src="/assets/js/bootstrap.min.js"></script>
+ <script src="/assets/js/main.js"></script>
+ </body>
+</html>
+
+
+
+
Modified: kylin/site/blog/index.html
URL:
http://svn.apache.org/viewvc/kylin/site/blog/index.html?rev=1728270&r1=1728269&r2=1728270&view=diff
==============================================================================
--- kylin/site/blog/index.html (original)
+++ kylin/site/blog/index.html Wed Feb 3 08:35:20 2016
@@ -174,6 +174,12 @@
<li>
<h2 align="left" style="margin:0px">
+ <a class="post-link"
href="/blog/2016/02/03/streaming-cubing/">Streaming cubing
(Prototype)</a></h2><div align="left" class="post-meta">posted: Feb 3,
2016</div>
+
+ </li>
+
+ <li>
+ <h2 align="left" style="margin:0px">
<a class="post-link"
href="/blog/2015/12/25/support-powerbi-tableau9/">Apache Kylin supports Tableau
9 and MS Excel, Power BI now</a></h2><div align="left"
class="post-meta">posted: Dec 25, 2015</div>
</li>
Modified: kylin/site/development/about_temp_files.html
URL:
http://svn.apache.org/viewvc/kylin/site/development/about_temp_files.html?rev=1728270&r1=1728269&r2=1728270&view=diff
==============================================================================
--- kylin/site/development/about_temp_files.html (original)
+++ kylin/site/development/about_temp_files.html Wed Feb 3 08:35:20 2016
@@ -405,8 +405,6 @@
-
-
<li><a href="/development/web_tech.html" class="list-group-item-lay
pjaxlink" id="navlist">Kylin Web Summary</a></li>
@@ -426,39 +424,6 @@
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
- <li><a href="/development/streaming_cubing.html"
class="list-group-item-lay pjaxlink" id="navlist">Streaming cubing
(Prototype)</a></li>
-
-
-
Modified: kylin/site/development/dev_env.html
URL:
http://svn.apache.org/viewvc/kylin/site/development/dev_env.html?rev=1728270&r1=1728269&r2=1728270&view=diff
==============================================================================
--- kylin/site/development/dev_env.html (original)
+++ kylin/site/development/dev_env.html Wed Feb 3 08:35:20 2016
@@ -405,8 +405,6 @@
-
-
<li><a href="/development/web_tech.html" class="list-group-item-lay
pjaxlink" id="navlist">Kylin Web Summary</a></li>
@@ -426,39 +424,6 @@
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
- <li><a href="/development/streaming_cubing.html"
class="list-group-item-lay pjaxlink" id="navlist">Streaming cubing
(Prototype)</a></li>
-
-
-
Modified: kylin/site/development/howto_contribute.html
URL:
http://svn.apache.org/viewvc/kylin/site/development/howto_contribute.html?rev=1728270&r1=1728269&r2=1728270&view=diff
==============================================================================
--- kylin/site/development/howto_contribute.html (original)
+++ kylin/site/development/howto_contribute.html Wed Feb 3 08:35:20 2016
@@ -405,8 +405,6 @@
-
-
<li><a href="/development/web_tech.html" class="list-group-item-lay
pjaxlink" id="navlist">Kylin Web Summary</a></li>
@@ -426,39 +424,6 @@
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
- <li><a href="/development/streaming_cubing.html"
class="list-group-item-lay pjaxlink" id="navlist">Streaming cubing
(Prototype)</a></li>
-
-
-
Modified: kylin/site/development/howto_docs.html
URL:
http://svn.apache.org/viewvc/kylin/site/development/howto_docs.html?rev=1728270&r1=1728269&r2=1728270&view=diff
==============================================================================
--- kylin/site/development/howto_docs.html (original)
+++ kylin/site/development/howto_docs.html Wed Feb 3 08:35:20 2016
@@ -405,8 +405,6 @@
-
-
<li><a href="/development/web_tech.html" class="list-group-item-lay
pjaxlink" id="navlist">Kylin Web Summary</a></li>
@@ -426,39 +424,6 @@
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
- <li><a href="/development/streaming_cubing.html"
class="list-group-item-lay pjaxlink" id="navlist">Streaming cubing
(Prototype)</a></li>
-
-
-
Modified: kylin/site/development/howto_package.html
URL:
http://svn.apache.org/viewvc/kylin/site/development/howto_package.html?rev=1728270&r1=1728269&r2=1728270&view=diff
==============================================================================
--- kylin/site/development/howto_package.html (original)
+++ kylin/site/development/howto_package.html Wed Feb 3 08:35:20 2016
@@ -405,8 +405,6 @@
-
-
<li><a href="/development/web_tech.html" class="list-group-item-lay
pjaxlink" id="navlist">Kylin Web Summary</a></li>
@@ -426,39 +424,6 @@
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
- <li><a href="/development/streaming_cubing.html"
class="list-group-item-lay pjaxlink" id="navlist">Streaming cubing
(Prototype)</a></li>
-
-
-
Modified: kylin/site/development/howto_release.html
URL:
http://svn.apache.org/viewvc/kylin/site/development/howto_release.html?rev=1728270&r1=1728269&r2=1728270&view=diff
==============================================================================
--- kylin/site/development/howto_release.html (original)
+++ kylin/site/development/howto_release.html Wed Feb 3 08:35:20 2016
@@ -405,8 +405,6 @@
-
-
<li><a href="/development/web_tech.html" class="list-group-item-lay
pjaxlink" id="navlist">Kylin Web Summary</a></li>
@@ -426,39 +424,6 @@
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
- <li><a href="/development/streaming_cubing.html"
class="list-group-item-lay pjaxlink" id="navlist">Streaming cubing
(Prototype)</a></li>
-
-
-
Modified: kylin/site/development/howto_test.html
URL:
http://svn.apache.org/viewvc/kylin/site/development/howto_test.html?rev=1728270&r1=1728269&r2=1728270&view=diff
==============================================================================
--- kylin/site/development/howto_test.html (original)
+++ kylin/site/development/howto_test.html Wed Feb 3 08:35:20 2016
@@ -405,8 +405,6 @@
-
-
<li><a href="/development/web_tech.html" class="list-group-item-lay
pjaxlink" id="navlist">Kylin Web Summary</a></li>
@@ -426,39 +424,6 @@
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
- <li><a href="/development/streaming_cubing.html"
class="list-group-item-lay pjaxlink" id="navlist">Streaming cubing
(Prototype)</a></li>
-
-
-
Modified: kylin/site/development/index.html
URL:
http://svn.apache.org/viewvc/kylin/site/development/index.html?rev=1728270&r1=1728269&r2=1728270&view=diff
==============================================================================
--- kylin/site/development/index.html (original)
+++ kylin/site/development/index.html Wed Feb 3 08:35:20 2016
@@ -405,8 +405,6 @@
-
-
<li><a href="/development/web_tech.html" class="list-group-item-lay
pjaxlink" id="navlist">Kylin Web Summary</a></li>
@@ -426,39 +424,6 @@
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
- <li><a href="/development/streaming_cubing.html"
class="list-group-item-lay pjaxlink" id="navlist">Streaming cubing
(Prototype)</a></li>
-
-
-
Modified: kylin/site/development/new_metadata.html
URL:
http://svn.apache.org/viewvc/kylin/site/development/new_metadata.html?rev=1728270&r1=1728269&r2=1728270&view=diff
==============================================================================
--- kylin/site/development/new_metadata.html (original)
+++ kylin/site/development/new_metadata.html Wed Feb 3 08:35:20 2016
@@ -405,8 +405,6 @@
-
-
<li><a href="/development/web_tech.html" class="list-group-item-lay
pjaxlink" id="navlist">Kylin Web Summary</a></li>
@@ -426,39 +424,6 @@
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
- <li><a href="/development/streaming_cubing.html"
class="list-group-item-lay pjaxlink" id="navlist">Streaming cubing
(Prototype)</a></li>
-
-
-
Modified: kylin/site/development/web_tech.html
URL:
http://svn.apache.org/viewvc/kylin/site/development/web_tech.html?rev=1728270&r1=1728269&r2=1728270&view=diff
==============================================================================
--- kylin/site/development/web_tech.html (original)
+++ kylin/site/development/web_tech.html Wed Feb 3 08:35:20 2016
@@ -405,8 +405,6 @@
-
-
<li><a href="/development/web_tech.html" class="list-group-item-lay
pjaxlink" id="navlist">Kylin Web Summary</a></li>
@@ -426,39 +424,6 @@
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
- <li><a href="/development/streaming_cubing.html"
class="list-group-item-lay pjaxlink" id="navlist">Streaming cubing
(Prototype)</a></li>
-
-
-
Modified: kylin/site/feed.xml
URL:
http://svn.apache.org/viewvc/kylin/site/feed.xml?rev=1728270&r1=1728269&r2=1728270&view=diff
==============================================================================
--- kylin/site/feed.xml (original)
+++ kylin/site/feed.xml Wed Feb 3 08:35:20 2016
@@ -19,11 +19,43 @@
<description>Apache Kylin Home</description>
<link>http://kylin.apache.org/</link>
<atom:link href="http://kylin.apache.org/feed.xml"; rel="self"
type="application/rss+xml"/>
- <pubDate>Wed, 03 Feb 2016 00:25:14 -0800</pubDate>
- <lastBuildDate>Wed, 03 Feb 2016 00:25:14 -0800</lastBuildDate>
+ <pubDate>Wed, 03 Feb 2016 00:34:04 -0800</pubDate>
+ <lastBuildDate>Wed, 03 Feb 2016 00:34:04 -0800</lastBuildDate>
<generator>Jekyll v2.5.3</generator>
<item>
+ <title>Streaming cubing (Prototype)</title>
+ <description><p>One of the most important features in 2.x
branches is streaming cubing which enables OLAP analysis on streaming data.
Streaming cubing delivers faster insights on the data to help more promptly
business decisions. Even though there are already many real time analysis tools
in open source community, Kylin Streaming cubing still differs from them in
multiple angles:</p>
+
+<p>Firstly, Kylin Streaming Cubing aligns with Kylin traditional cubing
to provided unified, ANSI SQL interface. Actually Kylin Streaming shares the
storage engine and query engine with traditional Kylin cubes, so in theory all
of the optimization techniques to save storage and speed up query performance
can also be applied on streaming cubes. Besides, all the supported
aggregations/filters/UDFs still work for streaming cubes. By unifying the
storage engine and query engine we also get freed from double amount of
maintaince work.</p>
+
+<p>Secondly, Kylin Streaming Cubing does not require large amount of
memory to store real time data, nor does it attempts to provide truly âreal
timeâ analysis. By our customer survey we found that minutes of visualization
latency is acceptable for OLAP analysts. So our streaming cubing adopts the
micro batch approach. Incoming streaming data are partitioned into different
time windows and we build a micro batch for each time window. The cube output
for each micro batch is directly saved to HBase. The query engine goes to HBase
for data retrieving instead of the data ingestion server. The benefit of such
design is that we donât have to maintain large amount of in-memory index
which could easily require tens of gigabytes of memory. In the future Kylin
might need to consider truly âreal timeâ support, too.</p>
+
+<p>Thirdly, Kylin Streaming Cubing data will be persistent and gradually
be converted to traditional cubes, thus customers can still query âcold
dataâ without any compromise on performance. As discussed above the output of
streaming cubing is directly saved to HBase as a new segment. The traditional
job engine will be notified of the new segment and take over to schedule merge
jobs when then segments accumulates. Day after day the segments of the
streaming cube got merged and become a very large traditional cube.</p>
+
+<p><img src="/images/develop/streaming.png" alt="Kylin
Streaming Framework Architecture" /></p>
+
+<p>With the major difference in mind we will introduce the modules for
Kylin Streaming cubing. Kylin Streaming cubing consist of three major
parts:</p>
+
+<ul>
+ <li>Streaming Input to retrieve data from a replayable data queue
(currently it is Kafka) within given time window. Streaming Input is also
responsible for primary data cleaning and normalization. By default Kylin
Streaming provides a default implementation to parse the messages from the
source queue. Customers can choose to configure the parser or provide a brand
new one based on their requirements.</li>
+ <li>Streaming Batch Ingestion to ingest the incoming data batch and
transform it into a micro cube. Thanks to the latest Kylin In-memory cubing
technology, this step is now times faster and space-saving than previous. The
micro cube is directly saved to HBase.</li>
+ <li>Job Scheduling Module to trigger Streaming Batch Ingestion. Kylin
does not put too much efforts in job scheduling, streaming cubing is not a
exception. Currently we provided a simple implementation based on Linux
Crontab.</li>
+</ul>
+
+<p>Weâll publish more detailed documents on how to use Kylin Streaming
soon. In latest 2.x branch we are also working on more complicated load
balancing schemes for streaming cubing. Please stay tuned.</p>
+
+</description>
+ <pubDate>Wed, 03 Feb 2016 08:30:00 -0800</pubDate>
+ <link>http://kylin.apache.org/blog/2016/02/03/streaming-cubing/</link>
+ <guid
isPermaLink="true">http://kylin.apache.org/blog/2016/02/03/streaming-cubing/</guid>
+
+
+ <category>blog</category>
+
+ </item>
+
+ <item>
<title>Apache Kylin supports Tableau 9 and MS Excel, Power BI
now</title>
<description><p>Apache Kylin Community has rolled out an updated
ODBC Driver (v1.2), which enforced more support for Business Intelligence
clients, including Microsoft Excel, Microsoft Power BI and Tableau 9. This
update did bring more convenience and selectivity for analysts and end
users.</p>
@@ -88,67 +120,6 @@ With sub-seconds query latency feature o
</item>
<item>
- <title>Apache Kylin v1.2 Release Announcement</title>
- <description><p>The Apache Kylin community is pleased to
announce the release of Apache Kylin v1.2, the first release after
graduation.</p>
-
-<p>Apache Kylin is an open source Distributed Analytics Engine designed
to provide SQL interface and multi-dimensional analysis (OLAP) on Hadoop
supporting extremely large datasets, original contributed from eBay
Inc.</p>
-
-<p>To download Apache Kylin v1.2 source code or binary package: <br
/>
-please visit the <a
href="http://kylin.apache.org/download">download</a>
page.</p>
-
-<p>This is a major release which brings more stable, robust and well
management version, Apache Kylin community resolved about 44 issues including
bug fixes, improvements, and few new features.</p>
-
-<h2 id="change-highlights">Change Highlights</h2>
-
-<p><strong>Kylin Core Improvement</strong></p>
-
-<ul>
- <li>Support Excel, Power BI and Tableau 9.1 <a
href="https://issues.apache.org/jira/browse/KYLIN-596">KYLIN-596</a>,<a
href="https://issues.apache.org/jira/browse/KYLIN-1065">KYLIN-1065</a></li>
- <li>Improve small file management on HDFS <a
href="https://issues.apache.org/jira/browse/KYLIN-702">KYLIN-702</a></li>
- <li>Env shell script enhance for Hive HCatalog <a
href="https://issues.apache.org/jira/browse/KYLIN-1081">KYLIN-1081</a>,
<a
href="https://issues.apache.org/jira/browse/KYLIN-1119">KYLIN-1119</a></li>
- <li>Dimenion column supports high cardinality over 10 million <a
href="https://issues.apache.org/jira/browse/KYLIN-1099">KYLIN-1099</a></li>
- <li>Enhance job page loading performance <a
href="https://issues.apache.org/jira/browse/KYLIN-1154">KYLIN-1154</a></li>
- <li>Make memory budget per query configurable <a
href="https://issues.apache.org/jira/browse/KYLIN-1190">KYLIN-1190</a></li>
-</ul>
-
-<p><strong>Main Bug Fixes</strong></p>
-
-<ul>
- <li>Save cube issue in edit model <a
href="https://issues.apache.org/jira/browse/KYLIN-1168">KYLIN-1168</a></li>
- <li>Couldnât change a cubeâs name after it be created <a
href="https://issues.apache.org/jira/browse/KYLIN-693">KYLIN-693</a></li>
- <li>Cube list missing under project <a
href="https://issues.apache.org/jira/browse/KYLIN-930">KYLIN-930</a></li>
- <li>Error when join two sub-query <a
href="https://issues.apache.org/jira/browse/KYLIN-1033">KYLIN-1033</a></li>
- <li>Filter like (A or false) yields wrong result <a
href="https://issues.apache.org/jira/browse/KYLIN-1039">KYLIN-1039</a></li>
- <li>Support get MapReduce Job status for ResourceManager HA Env <a
href="https://issues.apache.org/jira/browse/KYLIN-1067">KYLIN-1067</a></li>
- <li>Can not send email caused by Build Base Cuboid Data step failed
<a
href="https://issues.apache.org/jira/browse/KYLIN-1106">KYLIN-1106</a></li>
- <li>ResourceTool download/upload does not work in binary package <a
href="https://issues.apache.org/jira/browse/KYLIN-1121">KYLIN-1121</a></li>
- <li>Kylinâs sample cube âkylin_sales_cubeâ couldnât be saved
<a
href="https://issues.apache.org/jira/browse/KYLIN-1140">KYLIN-1140</a></li>
- <li>Unit test with minicluster doesnât work on 1.x <a
href="https://issues.apache.org/jira/browse/KYLIN-1155">KYLIN-1155</a></li>
- <li>Canât parse DateFormat like âYYYYMMDDâ correctly in query
<a
href="https://issues.apache.org/jira/browse/KYLIN-1216">KYLIN-1216</a></li>
-</ul>
-
-<p><strong>Upgrade</strong> <br />
-We recommend to upgrade to this version for better performance, stability and
bug fixes.<br />
-Also to keep up to date with community with latest features and
supports.</p>
-
-<p><strong>Support</strong> <br />
-Any issue or question during upgrade, please <br />
-open JIRA to Kylin project: <a
href="https://issues.apache.org/jira/browse/KYLIN/">https://issues.apache.org/jira/browse/KYLIN/</a>
<br />
-or <br />
-send mail to Apache Kylin dev mailing list: <a
href="&#109;&#097;&#105;&#108;&#116;&#111;:&#100;&#101;&#118;&#064;&#107;&#121;&#108;&#105;&#110;&#046;&#097;&#112;&#097;&#099;&#104;&#101;&#046;&#111;&#114;&#103;">&#100;&#101;&#118;&#064;&#107;&#121;&#108;&#105;&#110;&#046;&#097;&#112;&#097;&#099;&#104;&#101;&#046;&#111;&#114;&#103;</a></p>
-
-<p><em>Great thanks to everyone who
contributed!</em></p>
-</description>
- <pubDate>Wed, 23 Dec 2015 14:28:00 -0800</pubDate>
- <link>http://kylin.apache.org/blog/2015/12/23/release-v1.2/</link>
- <guid
isPermaLink="true">http://kylin.apache.org/blog/2015/12/23/release-v1.2/</guid>
-
-
- <category>blog</category>
-
- </item>
-
- <item>
<title>Apache Kylin v1.2 æ£å¼åå¸</title>
<description><p>Apache Kylin社åºé常é«å
´å®£å¸Apache Kylin
v1.2æ£å¼åå¸ï¼è¿æ¯èªé¡ºå©æ¯ä¸æApache顶级项ç®åç第ä¸ä¸ªåå¸çæ¬ã</p>
@@ -210,6 +181,67 @@ send mail to Apache Kylin dev mailing li
</item>
<item>
+ <title>Apache Kylin v1.2 Release Announcement</title>
+ <description><p>The Apache Kylin community is pleased to
announce the release of Apache Kylin v1.2, the first release after
graduation.</p>
+
+<p>Apache Kylin is an open source Distributed Analytics Engine designed
to provide SQL interface and multi-dimensional analysis (OLAP) on Hadoop
supporting extremely large datasets, original contributed from eBay
Inc.</p>
+
+<p>To download Apache Kylin v1.2 source code or binary package: <br
/>
+please visit the <a
href="http://kylin.apache.org/download">download</a>
page.</p>
+
+<p>This is a major release which brings more stable, robust and well
management version, Apache Kylin community resolved about 44 issues including
bug fixes, improvements, and few new features.</p>
+
+<h2 id="change-highlights">Change Highlights</h2>
+
+<p><strong>Kylin Core Improvement</strong></p>
+
+<ul>
+ <li>Support Excel, Power BI and Tableau 9.1 <a
href="https://issues.apache.org/jira/browse/KYLIN-596">KYLIN-596</a>,<a
href="https://issues.apache.org/jira/browse/KYLIN-1065">KYLIN-1065</a></li>
+ <li>Improve small file management on HDFS <a
href="https://issues.apache.org/jira/browse/KYLIN-702">KYLIN-702</a></li>
+ <li>Env shell script enhance for Hive HCatalog <a
href="https://issues.apache.org/jira/browse/KYLIN-1081">KYLIN-1081</a>,
<a
href="https://issues.apache.org/jira/browse/KYLIN-1119">KYLIN-1119</a></li>
+ <li>Dimenion column supports high cardinality over 10 million <a
href="https://issues.apache.org/jira/browse/KYLIN-1099">KYLIN-1099</a></li>
+ <li>Enhance job page loading performance <a
href="https://issues.apache.org/jira/browse/KYLIN-1154">KYLIN-1154</a></li>
+ <li>Make memory budget per query configurable <a
href="https://issues.apache.org/jira/browse/KYLIN-1190">KYLIN-1190</a></li>
+</ul>
+
+<p><strong>Main Bug Fixes</strong></p>
+
+<ul>
+ <li>Save cube issue in edit model <a
href="https://issues.apache.org/jira/browse/KYLIN-1168">KYLIN-1168</a></li>
+ <li>Couldnât change a cubeâs name after it be created <a
href="https://issues.apache.org/jira/browse/KYLIN-693">KYLIN-693</a></li>
+ <li>Cube list missing under project <a
href="https://issues.apache.org/jira/browse/KYLIN-930">KYLIN-930</a></li>
+ <li>Error when join two sub-query <a
href="https://issues.apache.org/jira/browse/KYLIN-1033">KYLIN-1033</a></li>
+ <li>Filter like (A or false) yields wrong result <a
href="https://issues.apache.org/jira/browse/KYLIN-1039">KYLIN-1039</a></li>
+ <li>Support get MapReduce Job status for ResourceManager HA Env <a
href="https://issues.apache.org/jira/browse/KYLIN-1067">KYLIN-1067</a></li>
+ <li>Can not send email caused by Build Base Cuboid Data step failed
<a
href="https://issues.apache.org/jira/browse/KYLIN-1106">KYLIN-1106</a></li>
+ <li>ResourceTool download/upload does not work in binary package <a
href="https://issues.apache.org/jira/browse/KYLIN-1121">KYLIN-1121</a></li>
+ <li>Kylinâs sample cube âkylin_sales_cubeâ couldnât be saved
<a
href="https://issues.apache.org/jira/browse/KYLIN-1140">KYLIN-1140</a></li>
+ <li>Unit test with minicluster doesnât work on 1.x <a
href="https://issues.apache.org/jira/browse/KYLIN-1155">KYLIN-1155</a></li>
+ <li>Canât parse DateFormat like âYYYYMMDDâ correctly in query
<a
href="https://issues.apache.org/jira/browse/KYLIN-1216">KYLIN-1216</a></li>
+</ul>
+
+<p><strong>Upgrade</strong> <br />
+We recommend to upgrade to this version for better performance, stability and
bug fixes.<br />
+Also to keep up to date with community with latest features and
supports.</p>
+
+<p><strong>Support</strong> <br />
+Any issue or question during upgrade, please <br />
+open JIRA to Kylin project: <a
href="https://issues.apache.org/jira/browse/KYLIN/">https://issues.apache.org/jira/browse/KYLIN/</a>
<br />
+or <br />
+send mail to Apache Kylin dev mailing list: <a
href="&#109;&#097;&#105;&#108;&#116;&#111;:&#100;&#101;&#118;&#064;&#107;&#121;&#108;&#105;&#110;&#046;&#097;&#112;&#097;&#099;&#104;&#101;&#046;&#111;&#114;&#103;">&#100;&#101;&#118;&#064;&#107;&#121;&#108;&#105;&#110;&#046;&#097;&#112;&#097;&#099;&#104;&#101;&#046;&#111;&#114;&#103;</a></p>
+
+<p><em>Great thanks to everyone who
contributed!</em></p>
+</description>
+ <pubDate>Wed, 23 Dec 2015 14:28:00 -0800</pubDate>
+ <link>http://kylin.apache.org/blog/2015/12/23/release-v1.2/</link>
+ <guid
isPermaLink="true">http://kylin.apache.org/blog/2015/12/23/release-v1.2/</guid>
+
+
+ <category>blog</category>
+
+ </item>
+
+ <item>
<title>Apache Kylin v1.1 (incubating) Release Announcement</title>
<description><p>The Apache Kylin community is pleased to
announce the release of Apache Kylin v1.1 (incubating).</p>
@@ -652,58 +684,6 @@ So the environment setup is as below:<
<category>blog</category>
-
- </item>
-
- <item>
- <title>Apache Kylin v1.0 (incubating) Release Announcement</title>
- <description><p>The Apache Kylin team is pleased to announce the
release of Apache Kylin v1.0 (incubating). Apache Kylin is an open source
Distributed Analytics Engine designed to provide SQL interface and
multi-dimensional analysis (OLAP) on Hadoop supporting extremely large
datasets.</p>
-
-<p>To download Apache Kylin v1.0 (incubating) visit the <a
href="http://kylin.apache.org/download">download</a>
page.</p>
-
-<p>This is a major release which brings more stable, robust and well
management version, Apache Kylin team resolved about 47 issues including bug
fixes, improvements, and few new features.</p>
-
-<h2 id="change-highlights">Change Highlights</h2>
-
-<p><strong>Kylin Core Improvement</strong></p>
-
-<ul>
- <li>Dynamic Data Model has been added to supporting adding or removing
column in data model without rebuild cube from the beginning <a
href="https://issues.apache.org/jira/browse/KYLIN-867">KYLIN-867</a></li>
- <li>Upgraded Apache Calcite to 1.3 for more bug fixes and new SQL
functions <a
href="https://issues.apache.org/jira/browse/KYLIN-881">KYLIN-881</a></li>
- <li>Cleanup job enhanced to make sure thereâs no garbage files left
in OS and HDFS/HBase after job build <a
href="https://issues.apache.org/jira/browse/KYLIN-926">KYLIN-926</a></li>
- <li>Added setting option for Hive intermediate tables created by Kylin
<a
href="https://issues.apache.org/jira/browse/KYLIN-883">KYLIN-883</a></li>
- <li>HBase coprocessor enhanced to imrpove query performance <a
href="https://issues.apache.org/jira/browse/KYLIN-857">KYLIN-857</a></li>
- <li>Kylin System Dashboard for usage, storage, performance <a
href="https://issues.apache.org/jira/browse/KYLIN-792">KYLIN-792</a></li>
-</ul>
-
-<p><strong>Main Bug Fixes</strong></p>
-
-<ul>
- <li>Canât get source record size, especially when using other Hadoop
distribution rather than HDP, like Cloudera and MapR <a
href="https://issues.apache.org/jira/browse/KYLIN-404">KYLIN-404</a></li>
- <li>Front-end cache cleanup issue <a
href="https://issues.apache.org/jira/browse/KYLIN-757">KYLIN-757</a></li>
- <li>Useless hive intermediate tables and HBase tables will be dropped
after cube build/merge <a
href="https://issues.apache.org/jira/browse/KYLIN-805">KYLIN-805</a></li>
- <li>Support More than one HDFS files of lookup table, especially for
large lookup table <a
href="https://issues.apache.org/jira/browse/KYLIN-889">KYLIN-889</a></li>
- <li>JDBC driver bug fixes <a
href="https://issues.apache.org/jira/browse/KYLIN-945">KYLIN-945</a>,
<a
href="https://issues.apache.org/jira/browse/KYLIN-626">KYLIN-626</a></li>
- <li>UI Bug fixes <a
href="https://issues.apache.org/jira/browse/KYLIN-946">KYLIN-946</a>,
<a
href="https://issues.apache.org/jira/browse/KYLIN-935">KYLIN-935</a></li>
-</ul>
-
-<p><strong>Zeppelin Integration</strong></p>
-
-<p><a
href="http://zeppelin.incubator.apache.org/">Apache
Zeppelin</a> is a web-based notebook that enables interactive data
analytics. The Apache Kylin team has contributed Kylin Interpreter which
enables Zeppelin interaction with Kylin from notebook using ANSI SQL, this
interpreter could be found from Zeppelin master code repo <a
href="https://github.com/apache/incubator-zeppelin/tree/master/kylin">here</a>.</p>
-
-<p><strong>Upgrade</strong></p>
-
-<p>We recommend to upgrade to this version from v0.7.x or even more
early version for better performance, stablility and clear one (most of the
intermediate files will be cleaned up automatically). Also to keep up to date
with community with latest features and supports.<br />
-Any issue or question during upgrade, please send to Apache Kylin dev mailing
list: <a
href="&#109;&#097;&#105;&#108;&#116;&#111;:&#100;&#101;&#118;&#064;&#107;&#121;&#108;&#105;&#110;&#046;&#097;&#112;&#097;&#099;&#104;&#101;&#046;&#111;&#114;&#103;">&#100;&#101;&#118;&#064;&#107;&#121;&#108;&#105;&#110;&#046;&#097;&#112;&#097;&#099;&#104;&#101;&#046;&#111;&#114;&#103;</a></p>
-
-<p><em>Great thanks to everyone who
contributed!</em></p>
-</description>
- <pubDate>Sun, 06 Sep 2015 10:28:00 -0700</pubDate>
-
<link>http://kylin.apache.org/blog/2015/09/06/release-v1.0-incubating/</link>
- <guid
isPermaLink="true">http://kylin.apache.org/blog/2015/09/06/release-v1.0-incubating/</guid>
-
-
- <category>blog</category>
</item>
