http://git-wip-us.apache.org/repos/asf/kudu-site/blob/bc4e6c33/2016/08/31/intro-flume-kudu-sink.html ---------------------------------------------------------------------- diff --git a/2016/08/31/intro-flume-kudu-sink.html b/2016/08/31/intro-flume-kudu-sink.html index d9ca72d..06c3d76 100644 --- a/2016/08/31/intro-flume-kudu-sink.html +++ b/2016/08/31/intro-flume-kudu-sink.html @@ -219,7 +219,7 @@ release and the source code can be found <a href="https://github.com/apache/kudu <p>Here is a sample flume configuration file:</p> -<div class="highlighter-rouge"><pre class="highlight"><code>agent1.sources = source1 +<pre><code>agent1.sources = source1 agent1.channels = channel1 agent1.sinks = sink1 @@ -238,25 +238,24 @@ agent1.sinks.sink1.channel = channel1 agent1.sinks.sink1.batchSize = 50 agent1.sinks.sink1.producer = org.apache.kudu.flume.sink.SimpleKuduEventProducer </code></pre> -</div> -<p>We define a source called <code class="highlighter-rouge">source1</code> which simply executes a <code class="highlighter-rouge">vmstat</code> command to continuously generate -virtual memory statistics for the machine and queue events into an in-memory <code class="highlighter-rouge">channel1</code> channel, -which in turn is used for writing these events to a Kudu table called <code class="highlighter-rouge">stats</code>. We are using -<code class="highlighter-rouge">org.apache.kudu.flume.sink.SimpleKuduEventProducer</code> as the producer. <code class="highlighter-rouge">SimpleKuduEventProducer</code> is +<p>We define a source called <code>source1</code> which simply executes a <code>vmstat</code> command to continuously generate +virtual memory statistics for the machine and queue events into an in-memory <code>channel1</code> channel, +which in turn is used for writing these events to a Kudu table called <code>stats</code>. We are using +<code>org.apache.kudu.flume.sink.SimpleKuduEventProducer</code> as the producer. <code>SimpleKuduEventProducer</code> is the built-in and default producer, but itâs implemented as a showcase for how to write Flume events into Kudu tables. For any serious functionality weâd have to write a custom producer. We -need to make this producer and the <code class="highlighter-rouge">KuduSink</code> class available to Flume. We can do that by simply -copying the <code class="highlighter-rouge">kudu-flume-sink-<VERSION>.jar</code> jar file from the Kudu distribution to the -<code class="highlighter-rouge">$FLUME_HOME/plugins.d/kudu-sink/lib</code> directory in the Flume installation. The jar file contains -<code class="highlighter-rouge">KuduSink</code> and all of its dependencies (including Kudu java client classes).</p> +need to make this producer and the <code>KuduSink</code> class available to Flume. We can do that by simply +copying the <code>kudu-flume-sink-<VERSION>.jar</code> jar file from the Kudu distribution to the +<code>$FLUME_HOME/plugins.d/kudu-sink/lib</code> directory in the Flume installation. The jar file contains +<code>KuduSink</code> and all of its dependencies (including Kudu java client classes).</p> <p>At a minimum, the Kudu Flume Sink needs to know where the Kudu masters are -(<code class="highlighter-rouge">agent1.sinks.sink1.masterAddresses = localhost</code>) and which Kudu table should be used for writing -Flume events to (<code class="highlighter-rouge">agent1.sinks.sink1.tableName = stats</code>). The Kudu Flume Sink doesnât create this +(<code>agent1.sinks.sink1.masterAddresses = localhost</code>) and which Kudu table should be used for writing +Flume events to (<code>agent1.sinks.sink1.tableName = stats</code>). The Kudu Flume Sink doesnât create this table, it has to be created before the Kudu Flume Sink is started.</p> -<p>You may also notice the <code class="highlighter-rouge">batchSize</code> parameter. Batch size is used for batching up to that many +<p>You may also notice the <code>batchSize</code> parameter. Batch size is used for batching up to that many Flume events and flushing the entire batch in one shot. Tuning batchSize properly can have a huge impact on ingest performance of the Kudu cluster.</p> @@ -306,91 +305,89 @@ impact on ingest performance of the Kudu cluster.</p> <p>Letâs take a look at the source code for the built-in producer class:</p> -<div class="highlighter-rouge"><pre class="highlight"><code><span class="kd">public</span> <span class="kd">class</span> <span class="nc">SimpleKuduEventProducer</span> <span class="kd">implements</span> <span class="n">KuduEventProducer</span> <span class="o">{</span> - <span class="kd">private</span> <span class="kt">byte</span><span class="o">[]</span> <span class="n">payload</span><span class="o">;</span> - <span class="kd">private</span> <span class="n">KuduTable</span> <span class="n">table</span><span class="o">;</span> - <span class="kd">private</span> <span class="n">String</span> <span class="n">payloadColumn</span><span class="o">;</span> - - <span class="kd">public</span> <span class="n">SimpleKuduEventProducer</span><span class="o">(){</span> - <span class="o">}</span> - - <span class="nd">@Override</span> - <span class="kd">public</span> <span class="kt">void</span> <span class="n">configure</span><span class="o">(</span><span class="n">Context</span> <span class="n">context</span><span class="o">)</span> <span class="o">{</span> - <span class="n">payloadColumn</span> <span class="o">=</span> <span class="n">context</span><span class="o">.</span><span class="na">getString</span><span class="o">(</span><span class="s">"payloadColumn"</span><span class="o">,</span><span class="s">"payload"</span><span class="o">);</span> - <span class="o">}</span> - - <span class="nd">@Override</span> - <span class="kd">public</span> <span class="kt">void</span> <span class="n">configure</span><span class="o">(</span><span class="n">ComponentConfiguration</span> <span class="n">conf</span><span class="o">)</span> <span class="o">{</span> - <span class="o">}</span> - - <span class="nd">@Override</span> - <span class="kd">public</span> <span class="kt">void</span> <span class="n">initialize</span><span class="o">(</span><span class="n">Event</span> <span class="n">event</span><span class="o">,</span> <span class="n">KuduTable</span> <span class="n">table</span><span class="o">)</span> <span class="o">{</span> - <span class="k">this</span><span class="o">.</span><span class="na">payload</span> <span class="o">=</span> <span class="n">event</span><span class="o">.</span><span class="na">getBody</span><span class="o">();</span> - <span class="k">this</span><span class="o">.</span><span class="na">table</span> <span class="o">=</span> <span class="n">table</span><span class="o">;</span> - <span class="o">}</span> - - <span class="nd">@Override</span> - <span class="kd">public</span> <span class="n">List</span><span class="o"><</span><span class="n">Operation</span><span class="o">></span> <span class="n">getOperations</span><span class="o">()</span> <span class="kd">throws</span> <span class="n">FlumeException</span> <span class="o">{</span> - <span class="k">try</span> <span class="o">{</span> - <span class="n">Insert</span> <span class="n">insert</span> <span class="o">=</span> <span class="n">table</span><span class="o">.</span><span class="na">newInsert</span><span class="o">();</span> - <span class="n">PartialRow</span> <span class="n">row</span> <span class="o">=</span> <span class="n">insert</span><span class="o">.</span><span class="na">getRow</span><span class="o">();</span> - <span class="n">row</span><span class="o">.</span><span class="na">addBinary</span><span class="o">(</span><span class="n">payloadColumn</span><span class="o">,</span> <span class="n">payload</span><span class="o">);</span> - - <span class="k">return</span> <span class="n">Collections</span><span class="o">.</span><span class="na">singletonList</span><span class="o">((</span><span class="n">Operation</span><span class="o">)</span> <span class="n">insert</span><span class="o">);</span> - <span class="o">}</span> <span class="k">catch</span> <span class="o">(</span><span class="n">Exception</span> <span class="n">e</span><span class="o">){</span> - <span class="k">throw</span> <span class="k">new</span> <span class="n">FlumeException</span><span class="o">(</span><span class="s">"Failed to create Kudu Insert object!"</span><span class="o">,</span> <span class="n">e</span><span class="o">);</span> - <span class="o">}</span> - <span class="o">}</span> - - <span class="nd">@Override</span> - <span class="kd">public</span> <span class="kt">void</span> <span class="n">close</span><span class="o">()</span> <span class="o">{</span> - <span class="o">}</span> -<span class="o">}</span> +<pre><code class="language-java">public class SimpleKuduEventProducer implements KuduEventProducer { + private byte[] payload; + private KuduTable table; + private String payloadColumn; + + public SimpleKuduEventProducer(){ + } + + @Override + public void configure(Context context) { + payloadColumn = context.getString("payloadColumn","payload"); + } + + @Override + public void configure(ComponentConfiguration conf) { + } + + @Override + public void initialize(Event event, KuduTable table) { + this.payload = event.getBody(); + this.table = table; + } + + @Override + public List<Operation> getOperations() throws FlumeException { + try { + Insert insert = table.newInsert(); + PartialRow row = insert.getRow(); + row.addBinary(payloadColumn, payload); + + return Collections.singletonList((Operation) insert); + } catch (Exception e){ + throw new FlumeException("Failed to create Kudu Insert object!", e); + } + } + + @Override + public void close() { + } +} </code></pre> -</div> -<p><code class="highlighter-rouge">SimpleKuduEventProducer</code> implements the <code class="highlighter-rouge">org.apache.kudu.flume.sink.KuduEventProducer</code> interface, +<p><code>SimpleKuduEventProducer</code> implements the <code>org.apache.kudu.flume.sink.KuduEventProducer</code> interface, which itself looks like this:</p> -<div class="highlighter-rouge"><pre class="highlight"><code><span class="kd">public</span> <span class="kd">interface</span> <span class="nc">KuduEventProducer</span> <span class="kd">extends</span> <span class="n">Configurable</span><span class="o">,</span> <span class="n">ConfigurableComponent</span> <span class="o">{</span> - <span class="cm">/** +<pre><code class="language-java">public interface KuduEventProducer extends Configurable, ConfigurableComponent { + /** * Initialize the event producer. * @param event to be written to Kudu * @param table the KuduTable object used for creating Kudu Operation objects - */</span> - <span class="kt">void</span> <span class="n">initialize</span><span class="o">(</span><span class="n">Event</span> <span class="n">event</span><span class="o">,</span> <span class="n">KuduTable</span> <span class="n">table</span><span class="o">);</span> + */ + void initialize(Event event, KuduTable table); - <span class="cm">/** + /** * Get the operations that should be written out to Kudu as a result of this * event. This list is written to Kudu using the Kudu client API. * @return List of {@link org.kududb.client.Operation} which * are written as such to Kudu - */</span> - <span class="n">List</span><span class="o"><</span><span class="n">Operation</span><span class="o">></span> <span class="n">getOperations</span><span class="o">();</span> + */ + List<Operation> getOperations(); - <span class="cm">/* + /* * Clean up any state. This will be called when the sink is being stopped. - */</span> - <span class="kt">void</span> <span class="n">close</span><span class="o">();</span> -<span class="o">}</span> + */ + void close(); +} </code></pre> -</div> -<p><code class="highlighter-rouge">public void configure(Context context)</code> is called when an instance of our producer is instantiated +<p><code>public void configure(Context context)</code> is called when an instance of our producer is instantiated by the KuduSink. SimpleKuduEventProducerâs implementation looks for a producer parameter named -<code class="highlighter-rouge">payloadColumn</code> and uses its value (âpayloadâ if not overridden in Flume configuration file) as the +<code>payloadColumn</code> and uses its value (âpayloadâ if not overridden in Flume configuration file) as the column which will hold the value of the Flume event payload. If you recall from above, we had -configured the KuduSink to listen for events generated from the <code class="highlighter-rouge">vmstat</code> command. Each output row -from that command will be stored as a new row containing a <code class="highlighter-rouge">payload</code> column in the <code class="highlighter-rouge">stats</code> table. -<code class="highlighter-rouge">SimpleKuduEventProducer</code> does not have any configuration parameters, but if it had any we would -define them by prefixing it with <code class="highlighter-rouge">producer.</code> (<code class="highlighter-rouge">agent1.sinks.sink1.producer.parameter1</code> for +configured the KuduSink to listen for events generated from the <code>vmstat</code> command. Each output row +from that command will be stored as a new row containing a <code>payload</code> column in the <code>stats</code> table. +<code>SimpleKuduEventProducer</code> does not have any configuration parameters, but if it had any we would +define them by prefixing it with <code>producer.</code> (<code>agent1.sinks.sink1.producer.parameter1</code> for example).</p> -<p>The main producer logic resides in the <code class="highlighter-rouge">public List<Operation> getOperations()</code> method. In +<p>The main producer logic resides in the <code>public List<Operation> getOperations()</code> method. In SimpleKuduEventProducerâs implementation we simply insert the binary body of the Flume event into -the Kudu table. Here we call Kuduâs <code class="highlighter-rouge">newInsert()</code> to initiate an insert, but could have used -<code class="highlighter-rouge">Upsert</code> if updating an existing row was also an option, in fact thereâs another producer -implementation available for doing just that: <code class="highlighter-rouge">SimpleKeyedKuduEventProducer</code>. Most probably you +the Kudu table. Here we call Kuduâs <code>newInsert()</code> to initiate an insert, but could have used +<code>Upsert</code> if updating an existing row was also an option, in fact thereâs another producer +implementation available for doing just that: <code>SimpleKeyedKuduEventProducer</code>. Most probably you will need to write your own custom producer in the real world, but you can base your implementation on the built-in ones.</p> @@ -420,6 +417,8 @@ is included in the Kudu distribution. You can follow him on Twitter at <h3>Recent posts</h3> <ul> + <li> <a href="/2016/09/16/predicate-pushdown.html">Pushing Down Predicate Evaluation in Apache Kudu</a> </li> + <li> <a href="/2016/08/31/intro-flume-kudu-sink.html">An Introduction to the Flume Kudu Sink</a> </li> <li> <a href="/2016/08/23/new-range-partitioning-features.html">New Range Partitioning Features in Kudu 0.10</a> </li> @@ -448,8 +447,6 @@ is included in the Kudu distribution. You can follow him on Twitter at <li> <a href="/2016/06/17/raft-consensus-single-node.html">Using Raft Consensus on a Single Node</a> </li> - <li> <a href="/2016/06/13/weekly-update.html">Apache Kudu (incubating) Weekly Update June 13, 2016</a> </li> - </ul> </div> </div>
http://git-wip-us.apache.org/repos/asf/kudu-site/blob/bc4e6c33/2016/09/16/predicate-pushdown.html ---------------------------------------------------------------------- diff --git a/2016/09/16/predicate-pushdown.html b/2016/09/16/predicate-pushdown.html new file mode 100644 index 0000000..ffa0cf8 --- /dev/null +++ b/2016/09/16/predicate-pushdown.html @@ -0,0 +1,339 @@ +<!DOCTYPE html> +<html lang="en"> + <head> + <meta charset="utf-8" /> + <meta http-equiv="X-UA-Compatible" content="IE=edge" /> + <meta name="viewport" content="width=device-width, initial-scale=1" /> + <!-- The above 3 meta tags *must* come first in the head; any other head content must come *after* these tags --> + <meta name="description" content="A new open source Apache Hadoop ecosystem project, Apache Kudu completes Hadoop's storage layer to enable fast analytics on fast data" /> + <meta name="author" content="Cloudera" /> + <title>Apache Kudu - Pushing Down Predicate Evaluation in Apache Kudu</title> + <!-- Bootstrap core CSS --> + <link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.6/css/bootstrap.min.css"; + integrity="sha384-1q8mTJOASx8j1Au+a5WDVnPi2lkFfwwEAa8hDDdjZlpLegxhjVME1fgjWPGmkzs7" + crossorigin="anonymous"> + + <!-- Custom styles for this template --> + <link href="/css/kudu.css" rel="stylesheet"/> + <link href="/css/asciidoc.css" rel="stylesheet"/> + <link rel="shortcut icon" href="/img/logo-favicon.ico" /> + <link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/font-awesome/4.6.1/css/font-awesome.min.css"; /> + + + <link rel="alternate" type="application/atom+xml" + title="RSS Feed for Apache Kudu blog" + href="/feed.xml" /> + + + <!-- HTML5 shim and Respond.js for IE8 support of HTML5 elements and media queries --> + <!--[if lt IE 9]> + <script src="https://oss.maxcdn.com/html5shiv/3.7.2/html5shiv.min.js";></script> + <script src="https://oss.maxcdn.com/respond/1.4.2/respond.min.js";></script> + <![endif]--> + </head> + <body> + <div class="kudu-site container-fluid"> + <!-- Static navbar --> + <nav class="navbar navbar-default"> + <div class="container-fluid"> + <div class="navbar-header"> + <button type="button" class="navbar-toggle collapsed" data-toggle="collapse" data-target="#navbar" aria-expanded="false" aria-controls="navbar"> + <span class="sr-only">Toggle navigation</span> + <span class="icon-bar"></span> + <span class="icon-bar"></span> + <span class="icon-bar"></span> + </button> + + <a class="logo" href="/"><img + src="//d3dr9sfxru4sde.cloudfront.net/i/k/apachekudu_logo_0716_80px.png" + srcset="//d3dr9sfxru4sde.cloudfront.net/i/k/apachekudu_logo_0716_80px.png 1x, //d3dr9sfxru4sde.cloudfront.net/i/k/apachekudu_logo_0716_160px.png 2x" + alt="Apache Kudu"/></a> + + </div> + <div id="navbar" class="collapse navbar-collapse"> + <ul class="nav navbar-nav navbar-right"> + <li > + <a href="/">Home</a> + </li> + <li > + <a href="/overview.html">Overview</a> + </li> + <li > + <a href="/docs/">Documentation</a> + </li> + <li > + <a href="/releases/">Download</a> + </li> + <li class="active"> + <a href="/blog/">Blog</a> + </li> + <!-- NOTE: this dropdown menu does not appear on Mobile, so don't add anything here + that doesn't also appear elsewhere on the site. --> + <li class="dropdown"> + <a href="/community.html" role="button" aria-haspopup="true" aria-expanded="false">Community <span class="caret"></span></a> + <ul class="dropdown-menu"> + <li class="dropdown-header">GET IN TOUCH</li> + <li><a class="icon email" href="/community.html">Mailing Lists</a></li> + <li><a class="icon slack" href="https://getkudu-slack.herokuapp.com/";>Slack Channel</a></li> + <li role="separator" class="divider"></li> + <li><a href="/community.html#meetups-user-groups-and-conference-presentations">Events and Meetups</a></li> + <li><a href="/committers.html">Project Committers</a></li> + <!--<li><a href="/roadmap.html">Roadmap</a></li>--> + <li><a href="/community.html#contributions">How to Contribute</a></li> + <li role="separator" class="divider"></li> + <li class="dropdown-header">DEVELOPER RESOURCES</li> + <li><a class="icon github" href="https://github.com/apache/incubator-kudu";>GitHub</a></li> + <li><a class="icon gerrit" href="http://gerrit.cloudera.org:8080/#/q/status:open+project:kudu";>Gerrit Code Review</a></li> + <li><a class="icon jira" href="https://issues.apache.org/jira/browse/KUDU";>JIRA Issue Tracker</a></li> + <li role="separator" class="divider"></li> + <li class="dropdown-header">SOCIAL MEDIA</li> + <li><a class="icon twitter" href="https://twitter.com/ApacheKudu";>Twitter</a></li> + </ul> + </li> + <li > + <a href="/faq.html">FAQ</a> + </li> + </ul><!-- /.nav --> + </div><!-- /#navbar --> + </div><!-- /.container-fluid --> + </nav> + +<div class="row header"> + <div class="col-lg-12"> + <h2><a href="/blog">Apache Kudu Blog</a></h2> + </div> +</div> + +<div class="row-fluid"> + <div class="col-lg-9"> + <article> + <header> + <h1 class="entry-title">Pushing Down Predicate Evaluation in Apache Kudu</h1> + <p class="meta">Posted 16 Sep 2016 by Andrew Wong</p> + </header> + <div class="entry-content"> + <p>I had the pleasure of interning with the Apache Kudu team at Cloudera this +summer. This project was my summer contribution to Kudu: a restructuring of the +scan path to speed up queries.</p> + +<!--more--> + +<h2 id="introduction">Introduction</h2> + +<p>In Kudu, <em>predicate pushdown</em> refers to the way in which predicates are +handled. When a scan is requested, its predicates are passed through the +different layers of Kuduâs storage hierarchy, allowing for pruning and other +optimizations to happen at each level before reaching the underlying data.</p> + +<p>While predicates are pushed down, predicate evaluation itself occurs at a fairly +high level, precluding the evaluation process from certain data-specific +optimizations. These optimizations can make tablet scans an order of magnitude +faster, if not more.</p> + +<h2 id="a-day-in-the-life-of-a-query">A Day in the Life of a Query</h2> + +<p>Because Kudu is a columnar storage engine, its scan path has a number of +optimizations to avoid extraneous reads, copies, and computation. When a query +is sent to a tablet server, the server prunes tablets based on the +primary key, directing the request to only the tablets that contain the key +range of interest. Once at a tablet, only the columns relevant to the query are +scanned. Further pruning is done over the primary key, and if the query is +predicated on non-key columns, the entire column is scanned. The columns in a +tablet are stored as <em>cfiles</em>, which are split into encoded <em>blocks</em>. Once the +relevant cfiles are determined, the data are materialized by the block +decoders, i.e. their underlying data are decoded and copied into a buffer, +which is passed back to the tablet layer. The tablet can then evaluate the +predicate on the batch of data and mark which rows should be returned to the +client.</p> + +<p>One of the encoding types I worked very closely with is <em>dictionary encoding</em>, +an encoding type for strings that performs particularly well for cfiles that +have repeating values. Rather than storing every rowâs string, each unique +string is assigned a numeric codeword, and the rows are stored numerically on +disk. When materializing a dictionary block, all of the numeric data are scanned +and all of the corresponding strings are copied and buffered for evaluation. +When the vocabulary of a dictionary-encoded cfile gets too large, the blocks +begin switching to <em>plain encoding mode</em> to act like <em>plain-encoded</em> blocks.</p> + +<p>In a plain-encoded block, strings are stored contiguously and the character +offsets to the start of each string are stored as a list of integers. When +materializing, all of the strings are copied to a buffer for evaluation.</p> + +<p>Therein lies room for improvement: this predicate evaluation path is the same +for all data types and encoding types. Within the tablet, the correct cfiles +are determined, the cfilesâ decoders are opened, all of the data are copied to +a buffer, and the predicates are evaluated on this buffered data via +type-specific comparators. This path is extremely flexible, but because it was +designed to be encoding-independent, there is room for improvement.</p> + +<h2 id="trimming-the-fat">Trimming the Fat</h2> + +<p>The first step is to allow the decoders access to the predicate. In doing so, +each encoding type can specialize its evaluation. Additionally, this puts the +decoder in a position where it can determine whether a given row satisfies the +query, which in turn, allows the decoders to determine what data gets copied +instead of eagerly copying all of its data to get evaluated.</p> + +<p>Take the case of dictionary-encoded strings as an example. With the existing +scan path, not only are all of the strings in a column copied into a buffer, but +string comparisons are done on every row. By taking advantage of the fact that +the data can be represented as integers, the cost of determining the query +results can be greatly reduced. The string comparisons can be swapped out with +evaluation based on the codewords, in which case the room for improvement boils +down to how to most quickly determine whether or not a given codeword +corresponds to a string that satisfies the predicate. Dictionary columns will +now use a bitset to store the codewords that match the predicates. It will then +scan through the integer-valued data and checks the bitset to determine whether +it should copy the corresponding string over.</p> + +<p>This is great in the best case scenario where a cfileâs vocabulary is small, +but when the vocabulary gets too large and the dictionary blocks switch to plain +encoding mode, performance is hampered. In this mode, the blocks donât utilize +any dictionary metadata and end up wasting the codeword bitset. That isnât to +say all is lost: the decoders can still evaluate a predicate via string +comparison, and the fact that evaluation can still occur at the decoder-level +means the eager buffering can still be avoided.</p> + +<p>Dictionary encoding is a perfect storm in that the decoders can completely +evaluate the predicates. This is not the case for most other encoding types, +but having decoders support evaluation leaves the door open for other encoding +types to extend this idea.</p> + +<h2 id="performance">Performance</h2> +<p>Depending on the dataset and query, predicate pushdown can lead to significant +improvements. Tablet scans were timed with datasets consisting of repeated +string patterns of tunable length and tunable cardinality.</p> + +<p><img src="/img/predicate-pushdown/pushdown-10.png" alt="png" class="img-responsive" /> +<img src="/img/predicate-pushdown/pushdown-10M.png" alt="png" class="img-responsive" /></p> + +<p>The above plots show the time taken to completely scan a single tablet, recorded +using a dataset of ten million rows of strings with length ten. Predicates were +designed to select values out of bounds (Empty), select a single value (Equal, +i.e. for cardinality <em>k</em>, this would select 1/<em>k</em> of the dataset), select half +of the full range (Half), and select the full range of values (All).</p> + +<p>With the original evaluation implementation, the tablet must copy and scan +through the tablet to determine whether any values match. This means that even +when the result set is small, the full column is still copied. This is avoided +by pushing down predicates, which only copies as needed, and can be seen in the +above queries: those with near-empty result sets (Empty and Equal) have shorter +scan times than those with larger result sets (Half and All).</p> + +<p>Note that for dictionary encoding, given a low cardinality, Kudu can completely +rely on the dictionary codewords to evaluate, making the query significantly +faster. At higher cardinalities, the dictionaries completely fill up and the +blocks fall back on plain encoding. The slower, albeit still improved, +performance on the dataset containing 10M unique values reflects this.</p> + +<p><img src="/img/predicate-pushdown/pushdown-tpch.png" alt="png" class="img-responsive" /></p> + +<p>Similar predicates were run with the TPC-H dataset, querying on the shipdate +column. The full path of a query includes not only the tablet scanning itself, +but also RPCs and batched data transfer to the caller as the scan progresses. +As such, the times plotted above refer to the average end-to-end time required +to scan and return a batch of rows. Regardless of this additional overhead, +significant improvements on the scan path still yield substantial improvements +to the query performance as a whole.</p> + +<h2 id="conclusion">Conclusion</h2> + +<p>Pushing down predicate evaluation in Kudu yielded substantial improvements to +the scan path. For dictionary encoding, pushdown can be particularly powerful, +and other encoding types are either unaffected or also improved. This change has +been pushed to the main branch of Kudu, and relevant commits can be found +<a href="https://github.com/cloudera/kudu/commit/c0f37278cb09a7781d9073279ea54b08db6e2010";>here</a> +and +<a href="https://github.com/cloudera/kudu/commit/ec80fdb37be44d380046a823b5e6d8e2241ec3da";>here</a>.</p> + +<p>This summer has been a phenomenal learning experience for me, in terms of the +tools, the workflow, the datasets, the thought-processes that go into building +something at Kuduâs scale. I am extremely thankful for all of the mentoring and +support I received, and that I got to be a part of Kuduâs journey from +incubating to a Top Level Apache project. I canât express enough how grateful I +am for the amount of support I got from the Kudu team, from the intern +coordinators, and from the Cloudera community as a whole.</p> + + </div> +</article> + + + </div> + <div class="col-lg-3 recent-posts"> + <h3>Recent posts</h3> + <ul> + + <li> <a href="/2016/09/16/predicate-pushdown.html">Pushing Down Predicate Evaluation in Apache Kudu</a> </li> + + <li> <a href="/2016/08/31/intro-flume-kudu-sink.html">An Introduction to the Flume Kudu Sink</a> </li> + + <li> <a href="/2016/08/23/new-range-partitioning-features.html">New Range Partitioning Features in Kudu 0.10</a> </li> + + <li> <a href="/2016/08/23/apache-kudu-0-10-0-released.html">Apache Kudu 0.10.0 released</a> </li> + + <li> <a href="/2016/08/16/weekly-update.html">Apache Kudu Weekly Update August 16th, 2016</a> </li> + + <li> <a href="/2016/08/08/weekly-update.html">Apache Kudu Weekly Update August 8th, 2016</a> </li> + + <li> <a href="/2016/07/26/weekly-update.html">Apache Kudu Weekly Update July 26, 2016</a> </li> + + <li> <a href="/2016/07/25/asf-graduation.html">The Apache Software Foundation Announces Apache® Kudu™ as a Top-Level Project</a> </li> + + <li> <a href="/2016/07/18/weekly-update.html">Apache Kudu (incubating) Weekly Update July 18, 2016</a> </li> + + <li> <a href="/2016/07/11/weekly-update.html">Apache Kudu (incubating) Weekly Update July 11, 2016</a> </li> + + <li> <a href="/2016/07/01/apache-kudu-0-9-1-released.html">Apache Kudu (incubating) 0.9.1 released</a> </li> + + <li> <a href="/2016/06/27/weekly-update.html">Apache Kudu (incubating) Weekly Update June 27, 2016</a> </li> + + <li> <a href="/2016/06/24/multi-master-1-0-0.html">Master fault tolerance in Kudu 1.0</a> </li> + + <li> <a href="/2016/06/21/weekly-update.html">Apache Kudu (incubating) Weekly Update June 21, 2016</a> </li> + + <li> <a href="/2016/06/17/raft-consensus-single-node.html">Using Raft Consensus on a Single Node</a> </li> + + </ul> + </div> +</div> + + <footer class="footer"> + <p class="small"> + Copyright © 2016 The Apache Software Foundation. + </p> + </footer> + </div> + <script src="https://cdnjs.cloudflare.com/ajax/libs/jquery/1.11.3/jquery.min.js";></script> + <script> + // Try to detect touch-screen devices. Note: Many laptops have touch screens. + $(document).ready(function() { + if ("ontouchstart" in document.documentElement) { + $(document.documentElement).addClass("touch"); + } else { + $(document.documentElement).addClass("no-touch"); + } + }); + </script> + <script src="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.6/js/bootstrap.min.js"; + integrity="sha384-0mSbJDEHialfmuBBQP6A4Qrprq5OVfW37PRR3j5ELqxss1yVqOtnepnHVP9aJ7xS" + crossorigin="anonymous"></script> + <script> + (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){ + (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o), + m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m) + })(window,document,'script','//www.google-analytics.com/analytics.js','ga'); + + ga('create', 'UA-68448017-1', 'auto'); + ga('send', 'pageview'); + </script> + <script src="https://cdnjs.cloudflare.com/ajax/libs/anchor-js/3.1.0/anchor.js";></script> + <script> + anchors.options = { + placement: 'right', + visible: 'touch', + }; + anchors.add(); + </script> + </body> +</html> + http://git-wip-us.apache.org/repos/asf/kudu-site/blob/bc4e6c33/blog/index.html ---------------------------------------------------------------------- diff --git a/blog/index.html b/blog/index.html index 19e2dd7..ccd79c9 100644 --- a/blog/index.html +++ b/blog/index.html @@ -111,17 +111,39 @@ <!-- Articles --> <article> <header> + <h1 class="entry-title"><a href="/2016/09/16/predicate-pushdown.html">Pushing Down Predicate Evaluation in Apache Kudu</a></h1> + <p class="meta">Posted 16 Sep 2016 by Andrew Wong</p> + </header> + <div class="entry-content"> + + <p>I had the pleasure of interning with the Apache Kudu team at Cloudera this +summer. This project was my summer contribution to Kudu: a restructuring of the +scan path to speed up queries.</p> + + + + </div> + <div class="read-full"> + <a class="btn btn-info" href="/2016/09/16/predicate-pushdown.html">Read full post...</a> + </div> +</article> + + + +<!-- Articles --> +<article> + <header> <h1 class="entry-title"><a href="/2016/08/31/intro-flume-kudu-sink.html">An Introduction to the Flume Kudu Sink</a></h1> <p class="meta">Posted 31 Aug 2016 by Ara Abrahamian</p> </header> <div class="entry-content"> - <p>This post discusses the Kudu Flume Sink. First, Iâll give some background on why we considered + <p>This post discusses the Kudu Flume Sink. First, I’ll give some background on why we considered using Kudu, what Flume does for us, and how Flume fits with Kudu in our project.</p> <h2 id="why-kudu">Why Kudu</h2> -<p>Traditionally in the Hadoop ecosystem weâve dealt with various <em>batch processing</em> technologies such +<p>Traditionally in the Hadoop ecosystem we’ve dealt with various <em>batch processing</em> technologies such as MapReduce and the many libraries and tools built on top of it in various languages (Apache Pig, Apache Hive, Apache Oozie and many others). The main problem with this approach is that it needs to process the whole data set in batches, again and again, as soon as new data gets added. Things get @@ -161,14 +183,14 @@ queries and processes need to be carefully planned and implemented.</p> <ul> <li>flexible and expressive, thanks to SQL support via Apache Impala (incubating)</li> <li>a table-oriented, mutable data store that feels like a traditional relational database</li> - <li>very easy to program, you can even pretend itâs good old MySQL</li> + <li>very easy to program, you can even pretend it’s good old MySQL</li> <li>low-latency and relatively high throughput, both for ingest and query</li> </ul> -<p>At Argyle Data, weâre dealing with complex fraud detection scenarios. We need to ingest massive +<p>At Argyle Data, we’re dealing with complex fraud detection scenarios. We need to ingest massive amounts of data, run machine learning algorithms and generate reports. When we created our current architecture two years ago we decided to opt for a database as the backbone of our system. That -database is Apache Accumulo. Itâs a key-value based database which runs on top of Hadoop HDFS, +database is Apache Accumulo. It’s a key-value based database which runs on top of Hadoop HDFS, quite similar to HBase but with some important improvements such as cell level security and ease of deployment and management. To enable querying of this data for quite complex reporting and analytics, we used Presto, a distributed query engine with a pluggable architecture open-sourced @@ -181,12 +203,12 @@ architecture has served us well, but there were a few problems:</p> <li>we need to support ad-hoc queries, plus long-term data warehouse functionality</li> </ul> -<p>So, weâve started gradually moving the core machine-learning pipeline to a streaming based +<p>So, we’ve started gradually moving the core machine-learning pipeline to a streaming based solution. This way we can ingest and process larger data-sets faster in the real-time. But then how would we take care of ad-hoc queries and long-term persistence? This is where Kudu comes in. While the machine learning pipeline ingests and processes real-time data, we store a copy of the same ingested data in Kudu for long-term access and ad-hoc queries. Kudu is our <em>data warehouse</em>. By -using Kudu and Impala, we can retire our in-house Presto connector and rely on Impalaâs +using Kudu and Impala, we can retire our in-house Presto connector and rely on Impala’s super-fast query engine.</p> <p>But how would we make sure data is reliably ingested into the streaming pipeline <em>and</em> the @@ -194,10 +216,10 @@ Kudu-based data warehouse? This is where Apache Flume comes in.</p> <h2 id="why-flume">Why Flume</h2> -<p>According to their <a href="http://flume.apache.org/";>website</a> âFlume is a distributed, reliable, and +<p>According to their <a href="http://flume.apache.org/";>website</a> “Flume is a distributed, reliable, and available service for efficiently collecting, aggregating, and moving large amounts of log data. It has a simple and flexible architecture based on streaming data flows. It is robust and fault -tolerant with tunable reliability mechanisms and many failover and recovery mechanisms.â As you +tolerant with tunable reliability mechanisms and many failover and recovery mechanisms.” As you can see, nowhere is Hadoop mentioned but Flume is typically used for ingesting data to Hadoop clusters.</p> @@ -215,7 +237,7 @@ File-based channels are also provided. As for the sources, Avro, JMS, Thrift, sp source are some of the built-in ones. Flume also ships with many sinks, including sinks for writing data to HDFS, HBase, Hive, Kafka, as well as to other Flume agents.</p> -<p>In the rest of this post Iâll go over the Kudu Flume sink and show you how to configure Flume to +<p>In the rest of this post I’ll go over the Kudu Flume sink and show you how to configure Flume to write ingested data to a Kudu table. The sink has been part of the Kudu distribution since the 0.8 release and the source code can be found <a href="https://github.com/apache/kudu/tree/master/java/kudu-flume-sink";>here</a>.</p> @@ -223,7 +245,7 @@ release and the source code can be found <a href="https://github.com/apache/kudu <p>Here is a sample flume configuration file:</p> -<div class="highlighter-rouge"><pre class="highlight"><code>agent1.sources = source1 +<pre><code>agent1.sources = source1 agent1.channels = channel1 agent1.sinks = sink1 @@ -242,25 +264,24 @@ agent1.sinks.sink1.channel = channel1 agent1.sinks.sink1.batchSize = 50 agent1.sinks.sink1.producer = org.apache.kudu.flume.sink.SimpleKuduEventProducer </code></pre> -</div> -<p>We define a source called <code class="highlighter-rouge">source1</code> which simply executes a <code class="highlighter-rouge">vmstat</code> command to continuously generate -virtual memory statistics for the machine and queue events into an in-memory <code class="highlighter-rouge">channel1</code> channel, -which in turn is used for writing these events to a Kudu table called <code class="highlighter-rouge">stats</code>. We are using -<code class="highlighter-rouge">org.apache.kudu.flume.sink.SimpleKuduEventProducer</code> as the producer. <code class="highlighter-rouge">SimpleKuduEventProducer</code> is -the built-in and default producer, but itâs implemented as a showcase for how to write Flume -events into Kudu tables. For any serious functionality weâd have to write a custom producer. We -need to make this producer and the <code class="highlighter-rouge">KuduSink</code> class available to Flume. We can do that by simply -copying the <code class="highlighter-rouge">kudu-flume-sink-<VERSION>.jar</code> jar file from the Kudu distribution to the -<code class="highlighter-rouge">$FLUME_HOME/plugins.d/kudu-sink/lib</code> directory in the Flume installation. The jar file contains -<code class="highlighter-rouge">KuduSink</code> and all of its dependencies (including Kudu java client classes).</p> +<p>We define a source called <code>source1</code> which simply executes a <code>vmstat</code> command to continuously generate +virtual memory statistics for the machine and queue events into an in-memory <code>channel1</code> channel, +which in turn is used for writing these events to a Kudu table called <code>stats</code>. We are using +<code>org.apache.kudu.flume.sink.SimpleKuduEventProducer</code> as the producer. <code>SimpleKuduEventProducer</code> is +the built-in and default producer, but it’s implemented as a showcase for how to write Flume +events into Kudu tables. For any serious functionality we’d have to write a custom producer. We +need to make this producer and the <code>KuduSink</code> class available to Flume. We can do that by simply +copying the <code>kudu-flume-sink-<VERSION>.jar</code> jar file from the Kudu distribution to the +<code>$FLUME_HOME/plugins.d/kudu-sink/lib</code> directory in the Flume installation. The jar file contains +<code>KuduSink</code> and all of its dependencies (including Kudu java client classes).</p> <p>At a minimum, the Kudu Flume Sink needs to know where the Kudu masters are -(<code class="highlighter-rouge">agent1.sinks.sink1.masterAddresses = localhost</code>) and which Kudu table should be used for writing -Flume events to (<code class="highlighter-rouge">agent1.sinks.sink1.tableName = stats</code>). The Kudu Flume Sink doesnât create this +(<code>agent1.sinks.sink1.masterAddresses = localhost</code>) and which Kudu table should be used for writing +Flume events to (<code>agent1.sinks.sink1.tableName = stats</code>). The Kudu Flume Sink doesn’t create this table, it has to be created before the Kudu Flume Sink is started.</p> -<p>You may also notice the <code class="highlighter-rouge">batchSize</code> parameter. Batch size is used for batching up to that many +<p>You may also notice the <code>batchSize</code> parameter. Batch size is used for batching up to that many Flume events and flushing the entire batch in one shot. Tuning batchSize properly can have a huge impact on ingest performance of the Kudu cluster.</p> @@ -278,7 +299,7 @@ impact on ingest performance of the Kudu cluster.</p> <tr> <td>masterAddresses</td> <td>N/A</td> - <td>Comma-separated list of âhost:portâ pairs of the masters (port optional)</td> + <td>Comma-separated list of “host:port” pairs of the masters (port optional)</td> </tr> <tr> <td>tableName</td> @@ -308,93 +329,91 @@ impact on ingest performance of the Kudu cluster.</p> </tbody> </table> -<p>Letâs take a look at the source code for the built-in producer class:</p> - -<div class="highlighter-rouge"><pre class="highlight"><code><span class="kd">public</span> <span class="kd">class</span> <span class="nc">SimpleKuduEventProducer</span> <span class="kd">implements</span> <span class="n">KuduEventProducer</span> <span class="o">{</span> - <span class="kd">private</span> <span class="kt">byte</span><span class="o">[]</span> <span class="n">payload</span><span class="o">;</span> - <span class="kd">private</span> <span class="n">KuduTable</span> <span class="n">table</span><span class="o">;</span> - <span class="kd">private</span> <span class="n">String</span> <span class="n">payloadColumn</span><span class="o">;</span> - - <span class="kd">public</span> <span class="n">SimpleKuduEventProducer</span><span class="o">(){</span> - <span class="o">}</span> - - <span class="nd">@Override</span> - <span class="kd">public</span> <span class="kt">void</span> <span class="n">configure</span><span class="o">(</span><span class="n">Context</span> <span class="n">context</span><span class="o">)</span> <span class="o">{</span> - <span class="n">payloadColumn</span> <span class="o">=</span> <span class="n">context</span><span class="o">.</span><span class="na">getString</span><span class="o">(</span><span class="s">"payloadColumn"</span><span class="o">,</span><span class="s">"payload"</span><span class="o">);</span> - <span class="o">}</span> - - <span class="nd">@Override</span> - <span class="kd">public</span> <span class="kt">void</span> <span class="n">configure</span><span class="o">(</span><span class="n">ComponentConfiguration</span> <span class="n">conf</span><span class="o">)</span> <span class="o">{</span> - <span class="o">}</span> - - <span class="nd">@Override</span> - <span class="kd">public</span> <span class="kt">void</span> <span class="n">initialize</span><span class="o">(</span><span class="n">Event</span> <span class="n">event</span><span class="o">,</span> <span class="n">KuduTable</span> <span class="n">table</span><span class="o">)</span> <span class="o">{</span> - <span class="k">this</span><span class="o">.</span><span class="na">payload</span> <span class="o">=</span> <span class="n">event</span><span class="o">.</span><span class="na">getBody</span><span class="o">();</span> - <span class="k">this</span><span class="o">.</span><span class="na">table</span> <span class="o">=</span> <span class="n">table</span><span class="o">;</span> - <span class="o">}</span> - - <span class="nd">@Override</span> - <span class="kd">public</span> <span class="n">List</span><span class="o"><</span><span class="n">Operation</span><span class="o">></span> <span class="n">getOperations</span><span class="o">()</span> <span class="kd">throws</span> <span class="n">FlumeException</span> <span class="o">{</span> - <span class="k">try</span> <span class="o">{</span> - <span class="n">Insert</span> <span class="n">insert</span> <span class="o">=</span> <span class="n">table</span><span class="o">.</span><span class="na">newInsert</span><span class="o">();</span> - <span class="n">PartialRow</span> <span class="n">row</span> <span class="o">=</span> <span class="n">insert</span><span class="o">.</span><span class="na">getRow</span><span class="o">();</span> - <span class="n">row</span><span class="o">.</span><span class="na">addBinary</span><span class="o">(</span><span class="n">payloadColumn</span><span class="o">,</span> <span class="n">payload</span><span class="o">);</span> - - <span class="k">return</span> <span class="n">Collections</span><span class="o">.</span><span class="na">singletonList</span><span class="o">((</span><span class="n">Operation</span><span class="o">)</span> <span class="n">insert</span><span class="o">);</span> - <span class="o">}</span> <span class="k">catch</span> <span class="o">(</span><span class="n">Exception</span> <span class="n">e</span><span class="o">){</span> - <span class="k">throw</span> <span class="k">new</span> <span class="n">FlumeException</span><span class="o">(</span><span class="s">"Failed to create Kudu Insert object!"</span><span class="o">,</span> <span class="n">e</span><span class="o">);</span> - <span class="o">}</span> - <span class="o">}</span> - - <span class="nd">@Override</span> - <span class="kd">public</span> <span class="kt">void</span> <span class="n">close</span><span class="o">()</span> <span class="o">{</span> - <span class="o">}</span> -<span class="o">}</span> +<p>Let’s take a look at the source code for the built-in producer class:</p> + +<pre><code class="language-java">public class SimpleKuduEventProducer implements KuduEventProducer { + private byte[] payload; + private KuduTable table; + private String payloadColumn; + + public SimpleKuduEventProducer(){ + } + + @Override + public void configure(Context context) { + payloadColumn = context.getString("payloadColumn","payload"); + } + + @Override + public void configure(ComponentConfiguration conf) { + } + + @Override + public void initialize(Event event, KuduTable table) { + this.payload = event.getBody(); + this.table = table; + } + + @Override + public List<Operation> getOperations() throws FlumeException { + try { + Insert insert = table.newInsert(); + PartialRow row = insert.getRow(); + row.addBinary(payloadColumn, payload); + + return Collections.singletonList((Operation) insert); + } catch (Exception e){ + throw new FlumeException("Failed to create Kudu Insert object!", e); + } + } + + @Override + public void close() { + } +} </code></pre> -</div> -<p><code class="highlighter-rouge">SimpleKuduEventProducer</code> implements the <code class="highlighter-rouge">org.apache.kudu.flume.sink.KuduEventProducer</code> interface, +<p><code>SimpleKuduEventProducer</code> implements the <code>org.apache.kudu.flume.sink.KuduEventProducer</code> interface, which itself looks like this:</p> -<div class="highlighter-rouge"><pre class="highlight"><code><span class="kd">public</span> <span class="kd">interface</span> <span class="nc">KuduEventProducer</span> <span class="kd">extends</span> <span class="n">Configurable</span><span class="o">,</span> <span class="n">ConfigurableComponent</span> <span class="o">{</span> - <span class="cm">/** +<pre><code class="language-java">public interface KuduEventProducer extends Configurable, ConfigurableComponent { + /** * Initialize the event producer. * @param event to be written to Kudu * @param table the KuduTable object used for creating Kudu Operation objects - */</span> - <span class="kt">void</span> <span class="n">initialize</span><span class="o">(</span><span class="n">Event</span> <span class="n">event</span><span class="o">,</span> <span class="n">KuduTable</span> <span class="n">table</span><span class="o">);</span> + */ + void initialize(Event event, KuduTable table); - <span class="cm">/** + /** * Get the operations that should be written out to Kudu as a result of this * event. This list is written to Kudu using the Kudu client API. * @return List of {@link org.kududb.client.Operation} which * are written as such to Kudu - */</span> - <span class="n">List</span><span class="o"><</span><span class="n">Operation</span><span class="o">></span> <span class="n">getOperations</span><span class="o">();</span> + */ + List<Operation> getOperations(); - <span class="cm">/* + /* * Clean up any state. This will be called when the sink is being stopped. - */</span> - <span class="kt">void</span> <span class="n">close</span><span class="o">();</span> -<span class="o">}</span> + */ + void close(); +} </code></pre> -</div> -<p><code class="highlighter-rouge">public void configure(Context context)</code> is called when an instance of our producer is instantiated -by the KuduSink. SimpleKuduEventProducerâs implementation looks for a producer parameter named -<code class="highlighter-rouge">payloadColumn</code> and uses its value (âpayloadâ if not overridden in Flume configuration file) as the +<p><code>public void configure(Context context)</code> is called when an instance of our producer is instantiated +by the KuduSink. SimpleKuduEventProducer’s implementation looks for a producer parameter named +<code>payloadColumn</code> and uses its value (“payload” if not overridden in Flume configuration file) as the column which will hold the value of the Flume event payload. If you recall from above, we had -configured the KuduSink to listen for events generated from the <code class="highlighter-rouge">vmstat</code> command. Each output row -from that command will be stored as a new row containing a <code class="highlighter-rouge">payload</code> column in the <code class="highlighter-rouge">stats</code> table. -<code class="highlighter-rouge">SimpleKuduEventProducer</code> does not have any configuration parameters, but if it had any we would -define them by prefixing it with <code class="highlighter-rouge">producer.</code> (<code class="highlighter-rouge">agent1.sinks.sink1.producer.parameter1</code> for +configured the KuduSink to listen for events generated from the <code>vmstat</code> command. Each output row +from that command will be stored as a new row containing a <code>payload</code> column in the <code>stats</code> table. +<code>SimpleKuduEventProducer</code> does not have any configuration parameters, but if it had any we would +define them by prefixing it with <code>producer.</code> (<code>agent1.sinks.sink1.producer.parameter1</code> for example).</p> -<p>The main producer logic resides in the <code class="highlighter-rouge">public List<Operation> getOperations()</code> method. In -SimpleKuduEventProducerâs implementation we simply insert the binary body of the Flume event into -the Kudu table. Here we call Kuduâs <code class="highlighter-rouge">newInsert()</code> to initiate an insert, but could have used -<code class="highlighter-rouge">Upsert</code> if updating an existing row was also an option, in fact thereâs another producer -implementation available for doing just that: <code class="highlighter-rouge">SimpleKeyedKuduEventProducer</code>. Most probably you +<p>The main producer logic resides in the <code>public List<Operation> getOperations()</code> method. In +SimpleKuduEventProducer’s implementation we simply insert the binary body of the Flume event into +the Kudu table. Here we call Kudu’s <code>newInsert()</code> to initiate an insert, but could have used +<code>Upsert</code> if updating an existing row was also an option, in fact there’s another producer +implementation available for doing just that: <code>SimpleKeyedKuduEventProducer</code>. Most probably you will need to write your own custom producer in the real world, but you can base your implementation on the built-in ones.</p> @@ -491,27 +510,6 @@ covers ongoing development and news in the Apache Kudu project.</p> -<!-- Articles --> -<article> - <header> - <h1 class="entry-title"><a href="/2016/08/08/weekly-update.html">Apache Kudu Weekly Update August 8th, 2016</a></h1> - <p class="meta">Posted 08 Aug 2016 by Todd Lipcon</p> - </header> - <div class="entry-content"> - - <p>Welcome to the nineteenth edition of the Kudu Weekly Update. This weekly blog post -covers ongoing development and news in the Apache Kudu project.</p> - - - - </div> - <div class="read-full"> - <a class="btn btn-info" href="/2016/08/08/weekly-update.html">Read full post...</a> - </div> -</article> - - - <!-- Pagination links --> <nav> @@ -530,6 +528,8 @@ covers ongoing development and news in the Apache Kudu project.</p> <h3>Recent posts</h3> <ul> + <li> <a href="/2016/09/16/predicate-pushdown.html">Pushing Down Predicate Evaluation in Apache Kudu</a> </li> + <li> <a href="/2016/08/31/intro-flume-kudu-sink.html">An Introduction to the Flume Kudu Sink</a> </li> <li> <a href="/2016/08/23/new-range-partitioning-features.html">New Range Partitioning Features in Kudu 0.10</a> </li> @@ -558,8 +558,6 @@ covers ongoing development and news in the Apache Kudu project.</p> <li> <a href="/2016/06/17/raft-consensus-single-node.html">Using Raft Consensus on a Single Node</a> </li> - <li> <a href="/2016/06/13/weekly-update.html">Apache Kudu (incubating) Weekly Update June 13, 2016</a> </li> - </ul> </div> </div> http://git-wip-us.apache.org/repos/asf/kudu-site/blob/bc4e6c33/blog/page/2/index.html ---------------------------------------------------------------------- diff --git a/blog/page/2/index.html b/blog/page/2/index.html index 1d359de..af9fb5b 100644 --- a/blog/page/2/index.html +++ b/blog/page/2/index.html @@ -111,6 +111,27 @@ <!-- Articles --> <article> <header> + <h1 class="entry-title"><a href="/2016/08/08/weekly-update.html">Apache Kudu Weekly Update August 8th, 2016</a></h1> + <p class="meta">Posted 08 Aug 2016 by Todd Lipcon</p> + </header> + <div class="entry-content"> + + <p>Welcome to the nineteenth edition of the Kudu Weekly Update. This weekly blog post +covers ongoing development and news in the Apache Kudu project.</p> + + + + </div> + <div class="read-full"> + <a class="btn btn-info" href="/2016/08/08/weekly-update.html">Read full post...</a> + </div> +</article> + + + +<!-- Articles --> +<article> + <header> <h1 class="entry-title"><a href="/2016/07/26/weekly-update.html">Apache Kudu Weekly Update July 26, 2016</a></h1> <p class="meta">Posted 26 Jul 2016 by Jean-Daniel Cryans</p> </header> @@ -195,35 +216,6 @@ covers ongoing development and news in the Apache Kudu (incubating) project.</p> -<!-- Articles --> -<article> - <header> - <h1 class="entry-title"><a href="/2016/07/01/apache-kudu-0-9-1-released.html">Apache Kudu (incubating) 0.9.1 released</a></h1> - <p class="meta">Posted 01 Jul 2016 by Todd Lipcon</p> - </header> - <div class="entry-content"> - - <p>The Apache Kudu (incubating) team is happy to announce the release of Kudu -0.9.1!</p> - -<p>This release fixes a few issues found in the previous 0.9.0 release. All users -of 0.9.0 are encouraged to update to the new version at their earliest convenience.</p> - -<ul> - <li>Read the detailed <a href="http://kudu.apache.org/releases/0.9.1/docs/release_notes.html";>Kudu 0.9.1 release notes</a></li> - <li>Download the <a href="http://kudu.apache.org/releases/0.9.1/";>Kudu 0.9.1 source release</a></li> -</ul> - - - - </div> - <div class="read-full"> - <a class="btn btn-info" href="/2016/07/01/apache-kudu-0-9-1-released.html">Read full post...</a> - </div> -</article> - - - <!-- Pagination links --> <nav> @@ -244,6 +236,8 @@ of 0.9.0 are encouraged to update to the new version at their earliest convenien <h3>Recent posts</h3> <ul> + <li> <a href="/2016/09/16/predicate-pushdown.html">Pushing Down Predicate Evaluation in Apache Kudu</a> </li> + <li> <a href="/2016/08/31/intro-flume-kudu-sink.html">An Introduction to the Flume Kudu Sink</a> </li> <li> <a href="/2016/08/23/new-range-partitioning-features.html">New Range Partitioning Features in Kudu 0.10</a> </li> @@ -272,8 +266,6 @@ of 0.9.0 are encouraged to update to the new version at their earliest convenien <li> <a href="/2016/06/17/raft-consensus-single-node.html">Using Raft Consensus on a Single Node</a> </li> - <li> <a href="/2016/06/13/weekly-update.html">Apache Kudu (incubating) Weekly Update June 13, 2016</a> </li> - </ul> </div> </div> http://git-wip-us.apache.org/repos/asf/kudu-site/blob/bc4e6c33/blog/page/3/index.html ---------------------------------------------------------------------- diff --git a/blog/page/3/index.html b/blog/page/3/index.html index 7d5a7db..8d8023d 100644 --- a/blog/page/3/index.html +++ b/blog/page/3/index.html @@ -111,6 +111,35 @@ <!-- Articles --> <article> <header> + <h1 class="entry-title"><a href="/2016/07/01/apache-kudu-0-9-1-released.html">Apache Kudu (incubating) 0.9.1 released</a></h1> + <p class="meta">Posted 01 Jul 2016 by Todd Lipcon</p> + </header> + <div class="entry-content"> + + <p>The Apache Kudu (incubating) team is happy to announce the release of Kudu +0.9.1!</p> + +<p>This release fixes a few issues found in the previous 0.9.0 release. All users +of 0.9.0 are encouraged to update to the new version at their earliest convenience.</p> + +<ul> + <li>Read the detailed <a href="http://kudu.apache.org/releases/0.9.1/docs/release_notes.html";>Kudu 0.9.1 release notes</a></li> + <li>Download the <a href="http://kudu.apache.org/releases/0.9.1/";>Kudu 0.9.1 source release</a></li> +</ul> + + + + </div> + <div class="read-full"> + <a class="btn btn-info" href="/2016/07/01/apache-kudu-0-9-1-released.html">Read full post...</a> + </div> +</article> + + + +<!-- Articles --> +<article> + <header> <h1 class="entry-title"><a href="/2016/06/27/weekly-update.html">Apache Kudu (incubating) Weekly Update June 27, 2016</a></h1> <p class="meta">Posted 27 Jun 2016 by Todd Lipcon</p> </header> @@ -138,7 +167,7 @@ covers ongoing development and news in the Apache Kudu (incubating) project.</p> <div class="entry-content"> <p>This blog post describes how the 1.0 release of Apache Kudu (incubating) will -support fault tolerance for the Kudu master, finally eliminating Kuduâs last +support fault tolerance for the Kudu master, finally eliminating Kudu’s last single point of failure.</p> @@ -196,27 +225,6 @@ replication factor of 1.</p> -<!-- Articles --> -<article> - <header> - <h1 class="entry-title"><a href="/2016/06/13/weekly-update.html">Apache Kudu (incubating) Weekly Update June 13, 2016</a></h1> - <p class="meta">Posted 13 Jun 2016 by Jean-Daniel Cryans</p> - </header> - <div class="entry-content"> - - <p>Welcome to the thirteenth edition of the Kudu Weekly Update. This weekly blog post -covers ongoing development and news in the Apache Kudu (incubating) project.</p> - - - - </div> - <div class="read-full"> - <a class="btn btn-info" href="/2016/06/13/weekly-update.html">Read full post...</a> - </div> -</article> - - - <!-- Pagination links --> <nav> @@ -237,6 +245,8 @@ covers ongoing development and news in the Apache Kudu (incubating) project.</p> <h3>Recent posts</h3> <ul> + <li> <a href="/2016/09/16/predicate-pushdown.html">Pushing Down Predicate Evaluation in Apache Kudu</a> </li> + <li> <a href="/2016/08/31/intro-flume-kudu-sink.html">An Introduction to the Flume Kudu Sink</a> </li> <li> <a href="/2016/08/23/new-range-partitioning-features.html">New Range Partitioning Features in Kudu 0.10</a> </li> @@ -265,8 +275,6 @@ covers ongoing development and news in the Apache Kudu (incubating) project.</p> <li> <a href="/2016/06/17/raft-consensus-single-node.html">Using Raft Consensus on a Single Node</a> </li> - <li> <a href="/2016/06/13/weekly-update.html">Apache Kudu (incubating) Weekly Update June 13, 2016</a> </li> - </ul> </div> </div> http://git-wip-us.apache.org/repos/asf/kudu-site/blob/bc4e6c33/blog/page/4/index.html ---------------------------------------------------------------------- diff --git a/blog/page/4/index.html b/blog/page/4/index.html index bcd2a93..9b32932 100644 --- a/blog/page/4/index.html +++ b/blog/page/4/index.html @@ -111,6 +111,27 @@ <!-- Articles --> <article> <header> + <h1 class="entry-title"><a href="/2016/06/13/weekly-update.html">Apache Kudu (incubating) Weekly Update June 13, 2016</a></h1> + <p class="meta">Posted 13 Jun 2016 by Jean-Daniel Cryans</p> + </header> + <div class="entry-content"> + + <p>Welcome to the thirteenth edition of the Kudu Weekly Update. This weekly blog post +covers ongoing development and news in the Apache Kudu (incubating) project.</p> + + + + </div> + <div class="read-full"> + <a class="btn btn-info" href="/2016/06/13/weekly-update.html">Read full post...</a> + </div> +</article> + + + +<!-- Articles --> +<article> + <header> <h1 class="entry-title"><a href="/2016/06/10/apache-kudu-0-9-0-released.html">Apache Kudu (incubating) 0.9.0 released</a></h1> <p class="meta">Posted 10 Jun 2016 by Jean-Daniel Cryans</p> </header> @@ -120,7 +141,7 @@ 0.9.0!</p> <p>This latest version adds basic UPSERT functionality and an improved Apache Spark Data Source -that doesnât rely on the MapReduce I/O formats. It also improves Tablet Server +that doesn’t rely on the MapReduce I/O formats. It also improves Tablet Server restart time as well as write performance under high load. Finally, Kudu now enforces the specification of a partitioning scheme for new tables.</p> @@ -204,27 +225,6 @@ covers ongoing development and news in the Apache Kudu (incubating) project.</p> -<!-- Articles --> -<article> - <header> - <h1 class="entry-title"><a href="/2016/05/23/weekly-update.html">Apache Kudu (incubating) Weekly Update May 23, 2016</a></h1> - <p class="meta">Posted 23 May 2016 by Todd Lipcon</p> - </header> - <div class="entry-content"> - - <p>Welcome to the tenth edition of the Kudu Weekly Update. This weekly blog post -covers ongoing development and news in the Apache Kudu (incubating) project.</p> - - - - </div> - <div class="read-full"> - <a class="btn btn-info" href="/2016/05/23/weekly-update.html">Read full post...</a> - </div> -</article> - - - <!-- Pagination links --> <nav> @@ -245,6 +245,8 @@ covers ongoing development and news in the Apache Kudu (incubating) project.</p> <h3>Recent posts</h3> <ul> + <li> <a href="/2016/09/16/predicate-pushdown.html">Pushing Down Predicate Evaluation in Apache Kudu</a> </li> + <li> <a href="/2016/08/31/intro-flume-kudu-sink.html">An Introduction to the Flume Kudu Sink</a> </li> <li> <a href="/2016/08/23/new-range-partitioning-features.html">New Range Partitioning Features in Kudu 0.10</a> </li> @@ -273,8 +275,6 @@ covers ongoing development and news in the Apache Kudu (incubating) project.</p> <li> <a href="/2016/06/17/raft-consensus-single-node.html">Using Raft Consensus on a Single Node</a> </li> - <li> <a href="/2016/06/13/weekly-update.html">Apache Kudu (incubating) Weekly Update June 13, 2016</a> </li> - </ul> </div> </div> http://git-wip-us.apache.org/repos/asf/kudu-site/blob/bc4e6c33/blog/page/5/index.html ---------------------------------------------------------------------- diff --git a/blog/page/5/index.html b/blog/page/5/index.html index 79a8f83..4d1b220 100644 --- a/blog/page/5/index.html +++ b/blog/page/5/index.html @@ -111,19 +111,19 @@ <!-- Articles --> <article> <header> - <h1 class="entry-title"><a href="/2016/05/16/weekly-update.html">Apache Kudu (incubating) Weekly Update May 16, 2016</a></h1> - <p class="meta">Posted 16 May 2016 by Todd Lipcon</p> + <h1 class="entry-title"><a href="/2016/05/23/weekly-update.html">Apache Kudu (incubating) Weekly Update May 23, 2016</a></h1> + <p class="meta">Posted 23 May 2016 by Todd Lipcon</p> </header> <div class="entry-content"> - <p>Welcome to the ninth edition of the Kudu Weekly Update. This weekly blog post + <p>Welcome to the tenth edition of the Kudu Weekly Update. This weekly blog post covers ongoing development and news in the Apache Kudu (incubating) project.</p> </div> <div class="read-full"> - <a class="btn btn-info" href="/2016/05/16/weekly-update.html">Read full post...</a> + <a class="btn btn-info" href="/2016/05/23/weekly-update.html">Read full post...</a> </div> </article> @@ -132,19 +132,19 @@ covers ongoing development and news in the Apache Kudu (incubating) project.</p> <!-- Articles --> <article> <header> - <h1 class="entry-title"><a href="/2016/05/09/weekly-update.html">Apache Kudu (incubating) Weekly Update May 9, 2016</a></h1> - <p class="meta">Posted 09 May 2016 by Jean-Daniel Cryans</p> + <h1 class="entry-title"><a href="/2016/05/16/weekly-update.html">Apache Kudu (incubating) Weekly Update May 16, 2016</a></h1> + <p class="meta">Posted 16 May 2016 by Todd Lipcon</p> </header> <div class="entry-content"> - <p>Welcome to the eighth edition of the Kudu Weekly Update. This weekly blog post + <p>Welcome to the ninth edition of the Kudu Weekly Update. This weekly blog post covers ongoing development and news in the Apache Kudu (incubating) project.</p> </div> <div class="read-full"> - <a class="btn btn-info" href="/2016/05/09/weekly-update.html">Read full post...</a> + <a class="btn btn-info" href="/2016/05/16/weekly-update.html">Read full post...</a> </div> </article> @@ -153,19 +153,19 @@ covers ongoing development and news in the Apache Kudu (incubating) project.</p> <!-- Articles --> <article> <header> - <h1 class="entry-title"><a href="/2016/05/03/weekly-update.html">Apache Kudu (incubating) Weekly Update May 3, 2016</a></h1> - <p class="meta">Posted 03 May 2016 by Todd Lipcon</p> + <h1 class="entry-title"><a href="/2016/05/09/weekly-update.html">Apache Kudu (incubating) Weekly Update May 9, 2016</a></h1> + <p class="meta">Posted 09 May 2016 by Jean-Daniel Cryans</p> </header> <div class="entry-content"> - <p>Welcome to the seventh edition of the Kudu Weekly Update. This weekly blog post + <p>Welcome to the eighth edition of the Kudu Weekly Update. This weekly blog post covers ongoing development and news in the Apache Kudu (incubating) project.</p> </div> <div class="read-full"> - <a class="btn btn-info" href="/2016/05/03/weekly-update.html">Read full post...</a> + <a class="btn btn-info" href="/2016/05/09/weekly-update.html">Read full post...</a> </div> </article> @@ -174,18 +174,19 @@ covers ongoing development and news in the Apache Kudu (incubating) project.</p> <!-- Articles --> <article> <header> - <h1 class="entry-title"><a href="/2016/04/26/ycsb.html">Benchmarking and Improving Kudu Insert Performance with YCSB</a></h1> - <p class="meta">Posted 26 Apr 2016 by Todd Lipcon</p> + <h1 class="entry-title"><a href="/2016/05/03/weekly-update.html">Apache Kudu (incubating) Weekly Update May 3, 2016</a></h1> + <p class="meta">Posted 03 May 2016 by Todd Lipcon</p> </header> <div class="entry-content"> - <p>Recently, I wanted to stress-test and benchmark some changes to the Kudu RPC server, and decided to use YCSB as a way to generate reasonable load. While running YCSB, I noticed interesting results, and what started as an unrelated testing exercise eventually yielded some new insights into Kuduâs behavior. These insights will motivate changes to default Kudu settings and code in upcoming versions. This post details the benchmark setup, analysis, and conclusions.</p> + <p>Welcome to the seventh edition of the Kudu Weekly Update. This weekly blog post +covers ongoing development and news in the Apache Kudu (incubating) project.</p> </div> <div class="read-full"> - <a class="btn btn-info" href="/2016/04/26/ycsb.html">Read full post...</a> + <a class="btn btn-info" href="/2016/05/03/weekly-update.html">Read full post...</a> </div> </article> @@ -194,19 +195,18 @@ covers ongoing development and news in the Apache Kudu (incubating) project.</p> <!-- Articles --> <article> <header> - <h1 class="entry-title"><a href="/2016/04/25/weekly-update.html">Apache Kudu (incubating) Weekly Update April 25, 2016</a></h1> - <p class="meta">Posted 25 Apr 2016 by Todd Lipcon</p> + <h1 class="entry-title"><a href="/2016/04/26/ycsb.html">Benchmarking and Improving Kudu Insert Performance with YCSB</a></h1> + <p class="meta">Posted 26 Apr 2016 by Todd Lipcon</p> </header> <div class="entry-content"> - <p>Welcome to the sixth edition of the Kudu Weekly Update. This weekly blog post -covers ongoing development and news in the Apache Kudu (incubating) project.</p> + <p>Recently, I wanted to stress-test and benchmark some changes to the Kudu RPC server, and decided to use YCSB as a way to generate reasonable load. While running YCSB, I noticed interesting results, and what started as an unrelated testing exercise eventually yielded some new insights into Kudu’s behavior. These insights will motivate changes to default Kudu settings and code in upcoming versions. This post details the benchmark setup, analysis, and conclusions.</p> </div> <div class="read-full"> - <a class="btn btn-info" href="/2016/04/25/weekly-update.html">Read full post...</a> + <a class="btn btn-info" href="/2016/04/26/ycsb.html">Read full post...</a> </div> </article> @@ -232,6 +232,8 @@ covers ongoing development and news in the Apache Kudu (incubating) project.</p> <h3>Recent posts</h3> <ul> + <li> <a href="/2016/09/16/predicate-pushdown.html">Pushing Down Predicate Evaluation in Apache Kudu</a> </li> + <li> <a href="/2016/08/31/intro-flume-kudu-sink.html">An Introduction to the Flume Kudu Sink</a> </li> <li> <a href="/2016/08/23/new-range-partitioning-features.html">New Range Partitioning Features in Kudu 0.10</a> </li> @@ -260,8 +262,6 @@ covers ongoing development and news in the Apache Kudu (incubating) project.</p> <li> <a href="/2016/06/17/raft-consensus-single-node.html">Using Raft Consensus on a Single Node</a> </li> - <li> <a href="/2016/06/13/weekly-update.html">Apache Kudu (incubating) Weekly Update June 13, 2016</a> </li> - </ul> </div> </div> http://git-wip-us.apache.org/repos/asf/kudu-site/blob/bc4e6c33/blog/page/6/index.html ---------------------------------------------------------------------- diff --git a/blog/page/6/index.html b/blog/page/6/index.html index 16c39eb..70caca7 100644 --- a/blog/page/6/index.html +++ b/blog/page/6/index.html @@ -111,6 +111,27 @@ <!-- Articles --> <article> <header> + <h1 class="entry-title"><a href="/2016/04/25/weekly-update.html">Apache Kudu (incubating) Weekly Update April 25, 2016</a></h1> + <p class="meta">Posted 25 Apr 2016 by Todd Lipcon</p> + </header> + <div class="entry-content"> + + <p>Welcome to the sixth edition of the Kudu Weekly Update. This weekly blog post +covers ongoing development and news in the Apache Kudu (incubating) project.</p> + + + + </div> + <div class="read-full"> + <a class="btn btn-info" href="/2016/04/25/weekly-update.html">Read full post...</a> + </div> +</article> + + + +<!-- Articles --> +<article> + <header> <h1 class="entry-title"><a href="/2016/04/19/kudu-0-8-0-predicate-improvements.html">Predicate Improvements in Kudu 0.8</a></h1> <p class="meta">Posted 19 Apr 2016 by Dan Burkert</p> </header> @@ -213,27 +234,6 @@ client, plus many other improvements and bug fixes.</p> -<!-- Articles --> -<article> - <header> - <h1 class="entry-title"><a href="/2016/04/11/weekly-update.html">Apache Kudu (incubating) Weekly Update April 11, 2016</a></h1> - <p class="meta">Posted 11 Apr 2016 by Todd Lipcon</p> - </header> - <div class="entry-content"> - - <p>Welcome to the fourth edition of the Kudu Weekly Update. This weekly blog post -covers ongoing development and news in the Apache Kudu (incubating) project.</p> - - - - </div> - <div class="read-full"> - <a class="btn btn-info" href="/2016/04/11/weekly-update.html">Read full post...</a> - </div> -</article> - - - <!-- Pagination links --> <nav> @@ -254,6 +254,8 @@ covers ongoing development and news in the Apache Kudu (incubating) project.</p> <h3>Recent posts</h3> <ul> + <li> <a href="/2016/09/16/predicate-pushdown.html">Pushing Down Predicate Evaluation in Apache Kudu</a> </li> + <li> <a href="/2016/08/31/intro-flume-kudu-sink.html">An Introduction to the Flume Kudu Sink</a> </li> <li> <a href="/2016/08/23/new-range-partitioning-features.html">New Range Partitioning Features in Kudu 0.10</a> </li> @@ -282,8 +284,6 @@ covers ongoing development and news in the Apache Kudu (incubating) project.</p> <li> <a href="/2016/06/17/raft-consensus-single-node.html">Using Raft Consensus on a Single Node</a> </li> - <li> <a href="/2016/06/13/weekly-update.html">Apache Kudu (incubating) Weekly Update June 13, 2016</a> </li> - </ul> </div> </div> http://git-wip-us.apache.org/repos/asf/kudu-site/blob/bc4e6c33/blog/page/7/index.html ---------------------------------------------------------------------- diff --git a/blog/page/7/index.html b/blog/page/7/index.html index f39d745..82cdd33 100644 --- a/blog/page/7/index.html +++ b/blog/page/7/index.html @@ -111,6 +111,27 @@ <!-- Articles --> <article> <header> + <h1 class="entry-title"><a href="/2016/04/11/weekly-update.html">Apache Kudu (incubating) Weekly Update April 11, 2016</a></h1> + <p class="meta">Posted 11 Apr 2016 by Todd Lipcon</p> + </header> + <div class="entry-content"> + + <p>Welcome to the fourth edition of the Kudu Weekly Update. This weekly blog post +covers ongoing development and news in the Apache Kudu (incubating) project.</p> + + + + </div> + <div class="read-full"> + <a class="btn btn-info" href="/2016/04/11/weekly-update.html">Read full post...</a> + </div> +</article> + + + +<!-- Articles --> +<article> + <header> <h1 class="entry-title"><a href="/2016/04/04/weekly-update.html">Apache Kudu (incubating) Weekly Update April 4, 2016</a></h1> <p class="meta">Posted 04 Apr 2016 by Todd Lipcon</p> </header> @@ -137,8 +158,8 @@ covers ongoing development and news in the Apache Kudu (incubating) project.</p> </header> <div class="entry-content"> - <p>Welcome to the second edition of the Kudu Weekly Update. As with last weekâs -inaugural post, weâll cover ongoing development and news in the Apache Kudu + <p>Welcome to the second edition of the Kudu Weekly Update. As with last week’s +inaugural post, we’ll cover ongoing development and news in the Apache Kudu project on a weekly basis.</p> @@ -159,13 +180,13 @@ project on a weekly basis.</p> </header> <div class="entry-content"> - <p>Kudu is a fast-moving young open source project, and weâve heard from a few -members of the community that it can be difficult to keep track of whatâs + <p>Kudu is a fast-moving young open source project, and we’ve heard from a few +members of the community that it can be difficult to keep track of what’s going on day-to-day. A typical month comprises 80-100 individual patches committed and hundreds of code review and discussion emails. So, inspired by similar weekly newsletters like -<a href="http://llvmweekly.org/";>LLVM Weekly</a> and <a href="http://lwn.net/Kernel/";>LWNâs weekly kernel coverage</a> -weâre going to experiment with our own weekly newsletter covering +<a href="http://llvmweekly.org/";>LLVM Weekly</a> and <a href="http://lwn.net/Kernel/";>LWN’s weekly kernel coverage</a> +we’re going to experiment with our own weekly newsletter covering recent development and Kudu-related news.</p> @@ -203,29 +224,6 @@ bugs fixed in this release:</p> -<!-- Articles --> -<article> - <header> - <h1 class="entry-title"><a href="/2016/02/26/apache-kudu-0-7-0-released.html">Apache Kudu (incubating) 0.7.0 released</a></h1> - <p class="meta">Posted 26 Feb 2016 by Todd Lipcon</p> - </header> - <div class="entry-content"> - - <p>The Apache Kudu (incubating) team is happy to announce its first release as -part of the ASF Incubator, version 0.7.0!</p> - -<p>This latest version has a number of improvements since 0.6.0:</p> - - - - </div> - <div class="read-full"> - <a class="btn btn-info" href="/2016/02/26/apache-kudu-0-7-0-released.html">Read full post...</a> - </div> -</article> - - - <!-- Pagination links --> <nav> @@ -235,6 +233,8 @@ part of the ASF Incubator, version 0.7.0!</p> <li class="previous"><a href="/blog/page/6"><span aria-hidden="true">←</span> Newer posts</a></li> + <li class="next"><a href="/blog/page/8">Older posts <span aria-hidden="true">→</span></a></li> + </ul> </nav> @@ -244,6 +244,8 @@ part of the ASF Incubator, version 0.7.0!</p> <h3>Recent posts</h3> <ul> + <li> <a href="/2016/09/16/predicate-pushdown.html">Pushing Down Predicate Evaluation in Apache Kudu</a> </li> + <li> <a href="/2016/08/31/intro-flume-kudu-sink.html">An Introduction to the Flume Kudu Sink</a> </li> <li> <a href="/2016/08/23/new-range-partitioning-features.html">New Range Partitioning Features in Kudu 0.10</a> </li> @@ -272,8 +274,6 @@ part of the ASF Incubator, version 0.7.0!</p> <li> <a href="/2016/06/17/raft-consensus-single-node.html">Using Raft Consensus on a Single Node</a> </li> - <li> <a href="/2016/06/13/weekly-update.html">Apache Kudu (incubating) Weekly Update June 13, 2016</a> </li> - </ul> </div> </div>
