http://git-wip-us.apache.org/repos/asf/incubator-crail-website/blob/004e8a6d/content/blog/2017/11/floss.html ---------------------------------------------------------------------- diff --git a/content/blog/2017/11/floss.html b/content/blog/2017/11/floss.html deleted file mode 100644 index 7bdd6cc..0000000 --- a/content/blog/2017/11/floss.html +++ /dev/null @@ -1,93 +0,0 @@ -<!DOCTYPE html> -<html> - <head> - <meta charset="utf-8"> - <title>The Apache Crail (Incubating) Project: Floss</title> - <meta name="viewport" content="width=device-width, initial-scale=1.0"> - <link href="//crail.incubator.apache.org/css/bootstrap.min.css" rel="stylesheet"> - <link href="//crail.incubator.apache.org/css/group.css" rel="stylesheet"> - <link rel="alternate" type="application/atom+xml" title="Atom" - href="//crail.incubator.apache.org/blog/blog.xml"> - - <meta property="og:image" content="//crail.incubator.apache.org/img/blog/preview/floss-summary.png" /> - <meta property="og:image:secure_url" content="//crail.incubator.apache.org/img/blog/preview/floss-summary.png" /> - </head> - - <body> - <div class="container"> - <div class="header"> - <ul class="nav nav-pills pull-right"> - - - - <li > - <a href="//crail.incubator.apache.org/"> - Home - </a> - </li> - - - <li > - <a href="//crail.incubator.apache.org/overview/"> - Overview - </a> - </li> - - - <li > - <a href="//crail.incubator.apache.org/download/"> - Downloads - </a> - </li> - - - <li > - <a href="//crail.incubator.apache.org/blog/"> - Blog - </a> - </li> - - - <li > - <a href="//crail.incubator.apache.org/community/"> - Community - </a> - </li> - - - <li > - <a href="//crail.incubator.apache.org/documentation/"> - Documentation - </a> - </li> - - </ul> - <a href="//crail.incubator.apache.org/"> - <img src="//crail.incubator.apache.org/img/crail_logo.png" - srcset="//crail.incubator.apache.org/img/crail_logo.png" - alt="Crail" id="logo"> - </a> - </div> - - - - <h2>Floss</h2> - - - <p>Crail features in the <a href="https://twit.tv/shows/floss-weekly/episodes/458?autostart=false";>FLOSS weekly podcast</a></p> - - - <br> - <br> - <div class="footer"> - <p>Apache Crail is an effort undergoing <a href="https://incubator.apache.org/";>incubation</a> at <a href="https://www.apache.org/";>The Apache Software Foundation (ASF)</a>, sponsored by the Apache Incubator PMC. Incubation is required of all newly accepted projects until a further review indicates that the infrastructure, communications, and decision making process have stabilized in a manner consistent with other successful ASF projects. While incubation status is not necessarily a reflection of the completeness or stability of the code, it does indicate that the project has yet to be fully endorsed by the ASF. - </p> - </div> - - </div> <!-- /container --> - - <!-- Support retina images. --> - <script type="text/javascript" - src="//crail.incubator.apache.org/js/srcset-polyfill.js"></script> - </body> -</html>
http://git-wip-us.apache.org/repos/asf/incubator-crail-website/blob/004e8a6d/content/blog/2017/11/iops.html ---------------------------------------------------------------------- diff --git a/content/blog/2017/11/iops.html b/content/blog/2017/11/iops.html deleted file mode 100644 index 5b91957..0000000 --- a/content/blog/2017/11/iops.html +++ /dev/null @@ -1,93 +0,0 @@ -<!DOCTYPE html> -<html> - <head> - <meta charset="utf-8"> - <title>The Apache Crail (Incubating) Project: Iops</title> - <meta name="viewport" content="width=device-width, initial-scale=1.0"> - <link href="//crail.incubator.apache.org/css/bootstrap.min.css" rel="stylesheet"> - <link href="//crail.incubator.apache.org/css/group.css" rel="stylesheet"> - <link rel="alternate" type="application/atom+xml" title="Atom" - href="//crail.incubator.apache.org/blog/blog.xml"> - - <meta property="og:image" content="//crail.incubator.apache.org/img/blog/preview/iops-summary.png" /> - <meta property="og:image:secure_url" content="//crail.incubator.apache.org/img/blog/preview/iops-summary.png" /> - </head> - - <body> - <div class="container"> - <div class="header"> - <ul class="nav nav-pills pull-right"> - - - - <li > - <a href="//crail.incubator.apache.org/"> - Home - </a> - </li> - - - <li > - <a href="//crail.incubator.apache.org/overview/"> - Overview - </a> - </li> - - - <li > - <a href="//crail.incubator.apache.org/download/"> - Downloads - </a> - </li> - - - <li > - <a href="//crail.incubator.apache.org/blog/"> - Blog - </a> - </li> - - - <li > - <a href="//crail.incubator.apache.org/community/"> - Community - </a> - </li> - - - <li > - <a href="//crail.incubator.apache.org/documentation/"> - Documentation - </a> - </li> - - </ul> - <a href="//crail.incubator.apache.org/"> - <img src="//crail.incubator.apache.org/img/crail_logo.png" - srcset="//crail.incubator.apache.org/img/crail_logo.png" - alt="Crail" id="logo"> - </a> - </div> - - - - <h2>Iops</h2> - - - <p>New blog <a href="//crail.incubator.apache.org/blog/2017/11/crail-metadata.html">post</a> about Crailâs metadata performance and scalability</p> - - - <br> - <br> - <div class="footer"> - <p>Apache Crail is an effort undergoing <a href="https://incubator.apache.org/";>incubation</a> at <a href="https://www.apache.org/";>The Apache Software Foundation (ASF)</a>, sponsored by the Apache Incubator PMC. Incubation is required of all newly accepted projects until a further review indicates that the infrastructure, communications, and decision making process have stabilized in a manner consistent with other successful ASF projects. While incubation status is not necessarily a reflection of the completeness or stability of the code, it does indicate that the project has yet to be fully endorsed by the ASF. - </p> - </div> - - </div> <!-- /container --> - - <!-- Support retina images. --> - <script type="text/javascript" - src="//crail.incubator.apache.org/js/srcset-polyfill.js"></script> - </body> -</html> http://git-wip-us.apache.org/repos/asf/incubator-crail-website/blob/004e8a6d/content/blog/2017/11/rdmashuffle.html ---------------------------------------------------------------------- diff --git a/content/blog/2017/11/rdmashuffle.html b/content/blog/2017/11/rdmashuffle.html deleted file mode 100644 index 816139f..0000000 --- a/content/blog/2017/11/rdmashuffle.html +++ /dev/null @@ -1,181 +0,0 @@ -<!DOCTYPE html> -<html> - <head> - <meta charset="utf-8"> - <title>The Apache Crail (Incubating) Project: Spark Shuffle: SparkRDMA vs Crail</title> - <meta name="viewport" content="width=device-width, initial-scale=1.0"> - <link href="//crail.incubator.apache.org/css/bootstrap.min.css" rel="stylesheet"> - <link href="//crail.incubator.apache.org/css/group.css" rel="stylesheet"> - <link rel="alternate" type="application/atom+xml" title="Atom" - href="//crail.incubator.apache.org/blog/blog.xml"> - - <meta property="og:image" content="//crail.incubator.apache.org/img/blog/preview/rdmashuffle-summary.png" /> - <meta property="og:image:secure_url" content="//crail.incubator.apache.org/img/blog/preview/rdmashuffle-summary.png" /> - </head> - - <body> - <div class="container"> - <div class="header"> - <ul class="nav nav-pills pull-right"> - - - - <li > - <a href="//crail.incubator.apache.org/"> - Home - </a> - </li> - - - <li > - <a href="//crail.incubator.apache.org/overview/"> - Overview - </a> - </li> - - - <li > - <a href="//crail.incubator.apache.org/download/"> - Downloads - </a> - </li> - - - <li > - <a href="//crail.incubator.apache.org/blog/"> - Blog - </a> - </li> - - - <li > - <a href="//crail.incubator.apache.org/community/"> - Community - </a> - </li> - - - <li > - <a href="//crail.incubator.apache.org/documentation/"> - Documentation - </a> - </li> - - </ul> - <a href="//crail.incubator.apache.org/"> - <img src="//crail.incubator.apache.org/img/crail_logo.png" - srcset="//crail.incubator.apache.org/img/crail_logo.png" - alt="Crail" id="logo"> - </a> - </div> - - - - <h2>Spark Shuffle: SparkRDMA vs Crail</h2> - - - <p class="meta">17 Nov 2017, <mark>this is a blog post from a user of the Crail project.</mark> </p> - -<div class="post"> -<div style="text-align: justify"> -<p> -This blog is comparing the shuffle performance of Crail with SparkRDMA, an alternative RDMA-based shuffle plugin for Spark. -</p> -</div> - -<h3 id="hardware-configuration">Hardware Configuration</h3> - -<p>The specific cluster configuration used for the experiments in this blog:</p> - -<ul> - <li>Cluster - <ul> - <li>8 compute + 1 management node x86_64 cluster</li> - </ul> - </li> - <li>Node configuration - <ul> - <li>CPU: 2 x Intel(R) Xeon(R) CPU E5-2690 0 @ 2.90GHz</li> - <li>DRAM: 96GB DDR3</li> - <li>Network: 1x100Gbit/s Mellanox ConnectX-5</li> - </ul> - </li> - <li>Software - <ul> - <li>Ubuntu 16.04.3 LTS (Xenial Xerus) with Linux kernel version 4.10.0-33-generic</li> - <li><a href="https://github.com/zrlio/crail";>Crail 1.0</a>, commit a45c8382050f471e9342e1c6cf25f9f2001af6b5</li> - <li><a href="">Crail Shuffle plugin</a>, commit 2273b5dd53405cab3389f5c1fc2ee4cd30f02ae6</li> - <li><a href="https://github.com/Mellanox/SparkRDMA";>SparkRDMA</a>, commit d95ce3e370a8e3b5146f4e0ab5e67a19c6f405a5 (latest master on 8th of November 2017)</li> - </ul> - </li> -</ul> - -<h3 id="overview">Overview</h3> -<div style="text-align: justify"> -<p> -Lately there has been an increasing interest in the community to include RDMA networking into data processing frameworks like Spark and Hadoop. One natural spot to integrate RDMA is in the shuffle operation that involves all-to-all network communication pattern. Naturally, due to its performance requirements the shuffle operation is of interest to us as well, and we have developed a Spark plugin for shuffle. In our previous blog posts, we have already shown that the Crail Shuffler achieves great workload-level speedups compared to vanilla Spark. In this blog post, we take a look at another recently proposed design called <a href="https://github.com/Mellanox/SparkRDMA";>SparkRDMA</a> (<a href="https://issues.apache.org/jira/browse/SPARK-22229";>SPARK-22229 JIRA</a>). SparkRDMA proposes to improve the shuffle performance of Spark by performing data transfers over RDMA. For this, the code manages its own off-heap memory which needs to be registered with the NIC for RDMA use. It supports two ways to store shuffle data between the stages: (1) shuffle data is stored in regular files (just like vanilla Spark) but the data transfer is implemented via RDMA, (2) data is stored in memory (allocated and registered for RDMA transfer) and the data transfer is implemented via RDMA. We call it the "last-mile" approach where just the networking operations are replaced by the RDMA operations. -</p> -<p> -In contrast, the Crail shuffler plugin takes a more holistic approach and leverages the high performance of Crail distributed data store to deliver gains. It uses Crail store to efficiently manage I/O resources, storage and networking devices, memory registrations, client sessions, data distribution, etc. Consequently, the shuffle operation becomes as simple as writing and reading files. And recall that Crail store is designed as a fast data bus for the intermediate data. The shuffle operation is just one of many operations that can be accelerated using Crail store. Beyond these operations, the modular architecture of Crail store enables us to seamlessly leverage different storage types (DRAM, NVMe, and more), perform tiering, support disaggregation, share inter-job data, jointly optimize I/O resources for various workloads, etc. These capabilities and performance gains give us confidence in the design choices we made for the Crail project. -</p> -</div> - -<h3 id="performance-comparison">Performance comparison</h3> -<div style="text-align: justify"> -<p>Lets start by quantitatively assessing performance gains from the Crail shuffle plugin and SparkRDMA. As described above, SparkRDMA can be operated in two different modes. Users decide which mode to use by selecting a particular type of shuffle writer (spark.shuffle.rdma.shuffleWriterMethod). The Wrapper shuffle writer writes shuffle data to files between the stages, the Chunked shuffle writer stores shuffle data in memory. We evaluate both writer methods for terasort and SQL equijoin. -</p> -</div> -<div style="text-align:center"><img src="//crail.incubator.apache.org/img/blog/rdma-shuffle/terasort.svg" width="550" /></div> -<p><br /></p> -<div style="text-align: justify"> -<p> -First we run <a href="https://github.com/zrlio/crail-spark-terasort";>terasort</a> on our 8+1 machine cluster (see above). We sort 200GB, thus, each node gets 25GB of data (equal distribution). We further did a basic search of the parameter space for each of the systems to find the best possible configuration. In all the experiments we use 8 executors with 12 cores each. Note that in a typical Spark run more CPU cores than assigned are engaged because of garbabge collection, etc. In our test runs assigning 12 cores lead to the best performance. -</p> -<p> -The plot above shows runtimes of the various configuration we run with terasort. SparkRDMA with the Wrapper shuffle writer performance slightly better (3-4%) than vanilla Spark whereas the Chunked shuffle writer shows a 30% overhead. On a quick inspection we found that this overhead stems from memory allocation and registration for the shuffle data that is kept in memory between the stages. Compared to vanilla Spark, Crail's shuffle plugin shows performance improvement of around 235%. -</p> -</div> -<div style="text-align:center"><img src="//crail.incubator.apache.org/img/blog/rdma-shuffle/sql.svg" width="550" /></div> -<p><br /></p> - -<div style="text-align: justify"> -<p> -For our second workload we choose the <a href="https://github.com/zrlio/sql-benchmarks";>SQL equijoin</a> with a <a href="https://github.com/zrlio/spark-nullio-fileformat";>special fileformat</a> that allows data to be generated on the fly. By generating data on the fly we eliminate any costs for reading data from storage and focus entirely on the shuffle performance. The shuffle data size is around 148GB. Here the Wrapper shuffle writer is slightly slower than vanilla Spark but instead the Chunked shuffle writer is roughly the same amount faster. The Crail shuffle plugin again delivers a great performance increase over vanilla Spark. -</p> -</div> - -<div style="text-align: justify"> -<p>Please let us know if your have recommendations about how these experiments can be improved.</p> -</div> - -<h3 id="summary">Summary</h3> - -<div style="text-align: justify"> -<p> -These benchmarks validate our belief that a "last-mile" integration cannot deliver the same performance gains as a holistic approach, i.e. one has to look at the whole picture in how to integrate RDMA into Spark applications (and for that matter any framework or application). Only replacing the data transfer alone does not lead to the anticipated performance increase. We learned this the hard way when we intially started working on Crail. -</p> - -</div> - - -</div> - -<!-- - ---> - - - <br> - <br> - <div class="footer"> - <p>Apache Crail is an effort undergoing <a href="https://incubator.apache.org/";>incubation</a> at <a href="https://www.apache.org/";>The Apache Software Foundation (ASF)</a>, sponsored by the Apache Incubator PMC. Incubation is required of all newly accepted projects until a further review indicates that the infrastructure, communications, and decision making process have stabilized in a manner consistent with other successful ASF projects. While incubation status is not necessarily a reflection of the completeness or stability of the code, it does indicate that the project has yet to be fully endorsed by the ASF. - </p> - </div> - - </div> <!-- /container --> - - <!-- Support retina images. --> - <script type="text/javascript" - src="//crail.incubator.apache.org/js/srcset-polyfill.js"></script> - </body> -</html> http://git-wip-us.apache.org/repos/asf/incubator-crail-website/blob/004e8a6d/content/blog/2018/01/apache.html ---------------------------------------------------------------------- diff --git a/content/blog/2018/01/apache.html b/content/blog/2018/01/apache.html deleted file mode 100644 index e3abaf2..0000000 --- a/content/blog/2018/01/apache.html +++ /dev/null @@ -1,93 +0,0 @@ -<!DOCTYPE html> -<html> - <head> - <meta charset="utf-8"> - <title>The Apache Crail (Incubating) Project: Apache</title> - <meta name="viewport" content="width=device-width, initial-scale=1.0"> - <link href="//crail.incubator.apache.org/css/bootstrap.min.css" rel="stylesheet"> - <link href="//crail.incubator.apache.org/css/group.css" rel="stylesheet"> - <link rel="alternate" type="application/atom+xml" title="Atom" - href="//crail.incubator.apache.org/blog/blog.xml"> - - <meta property="og:image" content="//crail.incubator.apache.org/img/blog/preview/apache-summary.png" /> - <meta property="og:image:secure_url" content="//crail.incubator.apache.org/img/blog/preview/apache-summary.png" /> - </head> - - <body> - <div class="container"> - <div class="header"> - <ul class="nav nav-pills pull-right"> - - - - <li > - <a href="//crail.incubator.apache.org/"> - Home - </a> - </li> - - - <li > - <a href="//crail.incubator.apache.org/overview/"> - Overview - </a> - </li> - - - <li > - <a href="//crail.incubator.apache.org/download/"> - Downloads - </a> - </li> - - - <li > - <a href="//crail.incubator.apache.org/blog/"> - Blog - </a> - </li> - - - <li > - <a href="//crail.incubator.apache.org/community/"> - Community - </a> - </li> - - - <li > - <a href="//crail.incubator.apache.org/documentation/"> - Documentation - </a> - </li> - - </ul> - <a href="//crail.incubator.apache.org/"> - <img src="//crail.incubator.apache.org/img/crail_logo.png" - srcset="//crail.incubator.apache.org/img/crail_logo.png" - alt="Crail" id="logo"> - </a> - </div> - - - - <h2>Apache</h2> - - - <p>Crail is now an Apache Incubator project!</p> - - - <br> - <br> - <div class="footer"> - <p>Apache Crail is an effort undergoing <a href="https://incubator.apache.org/";>incubation</a> at <a href="https://www.apache.org/";>The Apache Software Foundation (ASF)</a>, sponsored by the Apache Incubator PMC. Incubation is required of all newly accepted projects until a further review indicates that the infrastructure, communications, and decision making process have stabilized in a manner consistent with other successful ASF projects. While incubation status is not necessarily a reflection of the completeness or stability of the code, it does indicate that the project has yet to be fully endorsed by the ASF. - </p> - </div> - - </div> <!-- /container --> - - <!-- Support retina images. --> - <script type="text/javascript" - src="//crail.incubator.apache.org/js/srcset-polyfill.js"></script> - </body> -</html> http://git-wip-us.apache.org/repos/asf/incubator-crail-website/blob/004e8a6d/content/blog/2018/06/apache-release.html ---------------------------------------------------------------------- diff --git a/content/blog/2018/06/apache-release.html b/content/blog/2018/06/apache-release.html deleted file mode 100644 index 23f40a2..0000000 --- a/content/blog/2018/06/apache-release.html +++ /dev/null @@ -1,93 +0,0 @@ -<!DOCTYPE html> -<html> - <head> - <meta charset="utf-8"> - <title>The Apache Crail (Incubating) Project: Apache Release</title> - <meta name="viewport" content="width=device-width, initial-scale=1.0"> - <link href="//crail.incubator.apache.org/css/bootstrap.min.css" rel="stylesheet"> - <link href="//crail.incubator.apache.org/css/group.css" rel="stylesheet"> - <link rel="alternate" type="application/atom+xml" title="Atom" - href="//crail.incubator.apache.org/blog/blog.xml"> - - <meta property="og:image" content="//crail.incubator.apache.org/img/blog/preview/apache-release-summary.png" /> - <meta property="og:image:secure_url" content="//crail.incubator.apache.org/img/blog/preview/apache-release-summary.png" /> - </head> - - <body> - <div class="container"> - <div class="header"> - <ul class="nav nav-pills pull-right"> - - - - <li > - <a href="//crail.incubator.apache.org/"> - Home - </a> - </li> - - - <li > - <a href="//crail.incubator.apache.org/overview/"> - Overview - </a> - </li> - - - <li > - <a href="//crail.incubator.apache.org/download/"> - Downloads - </a> - </li> - - - <li > - <a href="//crail.incubator.apache.org/blog/"> - Blog - </a> - </li> - - - <li > - <a href="//crail.incubator.apache.org/community/"> - Community - </a> - </li> - - - <li > - <a href="//crail.incubator.apache.org/documentation/"> - Documentation - </a> - </li> - - </ul> - <a href="//crail.incubator.apache.org/"> - <img src="//crail.incubator.apache.org/img/crail_logo.png" - srcset="//crail.incubator.apache.org/img/crail_logo.png" - alt="Crail" id="logo"> - </a> - </div> - - - - <h2>Apache Release</h2> - - - <p>Apache Crail 1.0 incubator <a href="//crail.incubator.apache.org/download">release</a></p> - - - <br> - <br> - <div class="footer"> - <p>Apache Crail is an effort undergoing <a href="https://incubator.apache.org/";>incubation</a> at <a href="https://www.apache.org/";>The Apache Software Foundation (ASF)</a>, sponsored by the Apache Incubator PMC. Incubation is required of all newly accepted projects until a further review indicates that the infrastructure, communications, and decision making process have stabilized in a manner consistent with other successful ASF projects. While incubation status is not necessarily a reflection of the completeness or stability of the code, it does indicate that the project has yet to be fully endorsed by the ASF. - </p> - </div> - - </div> <!-- /container --> - - <!-- Support retina images. --> - <script type="text/javascript" - src="//crail.incubator.apache.org/js/srcset-polyfill.js"></script> - </body> -</html> http://git-wip-us.apache.org/repos/asf/incubator-crail-website/blob/004e8a6d/content/blog/2018/06/dataworks.html ---------------------------------------------------------------------- diff --git a/content/blog/2018/06/dataworks.html b/content/blog/2018/06/dataworks.html deleted file mode 100644 index c278119..0000000 --- a/content/blog/2018/06/dataworks.html +++ /dev/null @@ -1,93 +0,0 @@ -<!DOCTYPE html> -<html> - <head> - <meta charset="utf-8"> - <title>The Apache Crail (Incubating) Project: Dataworks</title> - <meta name="viewport" content="width=device-width, initial-scale=1.0"> - <link href="//crail.incubator.apache.org/css/bootstrap.min.css" rel="stylesheet"> - <link href="//crail.incubator.apache.org/css/group.css" rel="stylesheet"> - <link rel="alternate" type="application/atom+xml" title="Atom" - href="//crail.incubator.apache.org/blog/blog.xml"> - - <meta property="og:image" content="//crail.incubator.apache.org/img/blog/preview/dataworks-summary.png" /> - <meta property="og:image:secure_url" content="//crail.incubator.apache.org/img/blog/preview/dataworks-summary.png" /> - </head> - - <body> - <div class="container"> - <div class="header"> - <ul class="nav nav-pills pull-right"> - - - - <li > - <a href="//crail.incubator.apache.org/"> - Home - </a> - </li> - - - <li > - <a href="//crail.incubator.apache.org/overview/"> - Overview - </a> - </li> - - - <li > - <a href="//crail.incubator.apache.org/download/"> - Downloads - </a> - </li> - - - <li > - <a href="//crail.incubator.apache.org/blog/"> - Blog - </a> - </li> - - - <li > - <a href="//crail.incubator.apache.org/community/"> - Community - </a> - </li> - - - <li > - <a href="//crail.incubator.apache.org/documentation/"> - Documentation - </a> - </li> - - </ul> - <a href="//crail.incubator.apache.org/"> - <img src="//crail.incubator.apache.org/img/crail_logo.png" - srcset="//crail.incubator.apache.org/img/crail_logo.png" - alt="Crail" id="logo"> - </a> - </div> - - - - <h2>Dataworks</h2> - - - <p>Apache Crail (incubating) to feature in the <a href="https://dataworkssummit.com/san-jose-2018/session/data-processing-at-the-speed-of-100-gbpsapache-crail-incubating/";>DataWorks Summit</a> on June 21st</p> - - - <br> - <br> - <div class="footer"> - <p>Apache Crail is an effort undergoing <a href="https://incubator.apache.org/";>incubation</a> at <a href="https://www.apache.org/";>The Apache Software Foundation (ASF)</a>, sponsored by the Apache Incubator PMC. Incubation is required of all newly accepted projects until a further review indicates that the infrastructure, communications, and decision making process have stabilized in a manner consistent with other successful ASF projects. While incubation status is not necessarily a reflection of the completeness or stability of the code, it does indicate that the project has yet to be fully endorsed by the ASF. - </p> - </div> - - </div> <!-- /container --> - - <!-- Support retina images. --> - <script type="text/javascript" - src="//crail.incubator.apache.org/js/srcset-polyfill.js"></script> - </body> -</html> http://git-wip-us.apache.org/repos/asf/incubator-crail-website/blob/004e8a6d/content/blog/2018/06/sparksummit.html ---------------------------------------------------------------------- diff --git a/content/blog/2018/06/sparksummit.html b/content/blog/2018/06/sparksummit.html deleted file mode 100644 index d6afa64..0000000 --- a/content/blog/2018/06/sparksummit.html +++ /dev/null @@ -1,93 +0,0 @@ -<!DOCTYPE html> -<html> - <head> - <meta charset="utf-8"> - <title>The Apache Crail (Incubating) Project: Sparksummit</title> - <meta name="viewport" content="width=device-width, initial-scale=1.0"> - <link href="//crail.incubator.apache.org/css/bootstrap.min.css" rel="stylesheet"> - <link href="//crail.incubator.apache.org/css/group.css" rel="stylesheet"> - <link rel="alternate" type="application/atom+xml" title="Atom" - href="//crail.incubator.apache.org/blog/blog.xml"> - - <meta property="og:image" content="//crail.incubator.apache.org/img/blog/preview/sparksummit-summary.png" /> - <meta property="og:image:secure_url" content="//crail.incubator.apache.org/img/blog/preview/sparksummit-summary.png" /> - </head> - - <body> - <div class="container"> - <div class="header"> - <ul class="nav nav-pills pull-right"> - - - - <li > - <a href="//crail.incubator.apache.org/"> - Home - </a> - </li> - - - <li > - <a href="//crail.incubator.apache.org/overview/"> - Overview - </a> - </li> - - - <li > - <a href="//crail.incubator.apache.org/download/"> - Downloads - </a> - </li> - - - <li > - <a href="//crail.incubator.apache.org/blog/"> - Blog - </a> - </li> - - - <li > - <a href="//crail.incubator.apache.org/community/"> - Community - </a> - </li> - - - <li > - <a href="//crail.incubator.apache.org/documentation/"> - Documentation - </a> - </li> - - </ul> - <a href="//crail.incubator.apache.org/"> - <img src="//crail.incubator.apache.org/img/crail_logo.png" - srcset="//crail.incubator.apache.org/img/crail_logo.png" - alt="Crail" id="logo"> - </a> - </div> - - - - <h2>Sparksummit</h2> - - - <p>A Spark serverless architecture powered by Crail will be presented today at the <a href="https://databricks.com/session/serverless-machine-learning-on-modern-hardware-using-apache-spark";>Spark Summit</a></p> - - - <br> - <br> - <div class="footer"> - <p>Apache Crail is an effort undergoing <a href="https://incubator.apache.org/";>incubation</a> at <a href="https://www.apache.org/";>The Apache Software Foundation (ASF)</a>, sponsored by the Apache Incubator PMC. Incubation is required of all newly accepted projects until a further review indicates that the infrastructure, communications, and decision making process have stabilized in a manner consistent with other successful ASF projects. While incubation status is not necessarily a reflection of the completeness or stability of the code, it does indicate that the project has yet to be fully endorsed by the ASF. - </p> - </div> - - </div> <!-- /container --> - - <!-- Support retina images. --> - <script type="text/javascript" - src="//crail.incubator.apache.org/js/srcset-polyfill.js"></script> - </body> -</html> http://git-wip-us.apache.org/repos/asf/incubator-crail-website/blob/004e8a6d/content/blog/2018/08/sql-p1-news.html ---------------------------------------------------------------------- diff --git a/content/blog/2018/08/sql-p1-news.html b/content/blog/2018/08/sql-p1-news.html deleted file mode 100644 index a3b81cc..0000000 --- a/content/blog/2018/08/sql-p1-news.html +++ /dev/null @@ -1,93 +0,0 @@ -<!DOCTYPE html> -<html> - <head> - <meta charset="utf-8"> - <title>The Apache Crail (Incubating) Project: Sql P1 News</title> - <meta name="viewport" content="width=device-width, initial-scale=1.0"> - <link href="//crail.incubator.apache.org/css/bootstrap.min.css" rel="stylesheet"> - <link href="//crail.incubator.apache.org/css/group.css" rel="stylesheet"> - <link rel="alternate" type="application/atom+xml" title="Atom" - href="//crail.incubator.apache.org/blog/blog.xml"> - - <meta property="og:image" content="//crail.incubator.apache.org/img/blog/preview/sql-p1-news-summary.png" /> - <meta property="og:image:secure_url" content="//crail.incubator.apache.org/img/blog/preview/sql-p1-news-summary.png" /> - </head> - - <body> - <div class="container"> - <div class="header"> - <ul class="nav nav-pills pull-right"> - - - - <li > - <a href="//crail.incubator.apache.org/"> - Home - </a> - </li> - - - <li > - <a href="//crail.incubator.apache.org/overview/"> - Overview - </a> - </li> - - - <li > - <a href="//crail.incubator.apache.org/download/"> - Downloads - </a> - </li> - - - <li > - <a href="//crail.incubator.apache.org/blog/"> - Blog - </a> - </li> - - - <li > - <a href="//crail.incubator.apache.org/community/"> - Community - </a> - </li> - - - <li > - <a href="//crail.incubator.apache.org/documentation/"> - Documentation - </a> - </li> - - </ul> - <a href="//crail.incubator.apache.org/"> - <img src="//crail.incubator.apache.org/img/crail_logo.png" - srcset="//crail.incubator.apache.org/img/crail_logo.png" - alt="Crail" id="logo"> - </a> - </div> - - - - <h2>Sql P1 News</h2> - - - <p>A new blog <a href="//crail.incubator.apache.org/blog/2018/08/sql-p1.html">post</a> discussing file formats performance is now online</p> - - - <br> - <br> - <div class="footer"> - <p>Apache Crail is an effort undergoing <a href="https://incubator.apache.org/";>incubation</a> at <a href="https://www.apache.org/";>The Apache Software Foundation (ASF)</a>, sponsored by the Apache Incubator PMC. Incubation is required of all newly accepted projects until a further review indicates that the infrastructure, communications, and decision making process have stabilized in a manner consistent with other successful ASF projects. While incubation status is not necessarily a reflection of the completeness or stability of the code, it does indicate that the project has yet to be fully endorsed by the ASF. - </p> - </div> - - </div> <!-- /container --> - - <!-- Support retina images. --> - <script type="text/javascript" - src="//crail.incubator.apache.org/js/srcset-polyfill.js"></script> - </body> -</html> http://git-wip-us.apache.org/repos/asf/incubator-crail-website/blob/004e8a6d/content/blog/2018/08/sql-p1.html ---------------------------------------------------------------------- diff --git a/content/blog/2018/08/sql-p1.html b/content/blog/2018/08/sql-p1.html deleted file mode 100644 index 40cea88..0000000 --- a/content/blog/2018/08/sql-p1.html +++ /dev/null @@ -1,232 +0,0 @@ -<!DOCTYPE html> -<html> - <head> - <meta charset="utf-8"> - <title>The Apache Crail (Incubating) Project: SQL Performance: Part 1 - Input File Formats</title> - <meta name="viewport" content="width=device-width, initial-scale=1.0"> - <link href="//crail.incubator.apache.org/css/bootstrap.min.css" rel="stylesheet"> - <link href="//crail.incubator.apache.org/css/group.css" rel="stylesheet"> - <link rel="alternate" type="application/atom+xml" title="Atom" - href="//crail.incubator.apache.org/blog/blog.xml"> - - <meta property="og:image" content="//crail.incubator.apache.org/img/blog/preview/sql-p1-summary.png" /> - <meta property="og:image:secure_url" content="//crail.incubator.apache.org/img/blog/preview/sql-p1-summary.png" /> - </head> - - <body> - <div class="container"> - <div class="header"> - <ul class="nav nav-pills pull-right"> - - - - <li > - <a href="//crail.incubator.apache.org/"> - Home - </a> - </li> - - - <li > - <a href="//crail.incubator.apache.org/overview/"> - Overview - </a> - </li> - - - <li > - <a href="//crail.incubator.apache.org/download/"> - Downloads - </a> - </li> - - - <li > - <a href="//crail.incubator.apache.org/blog/"> - Blog - </a> - </li> - - - <li > - <a href="//crail.incubator.apache.org/community/"> - Community - </a> - </li> - - - <li > - <a href="//crail.incubator.apache.org/documentation/"> - Documentation - </a> - </li> - - </ul> - <a href="//crail.incubator.apache.org/"> - <img src="//crail.incubator.apache.org/img/crail_logo.png" - srcset="//crail.incubator.apache.org/img/crail_logo.png" - alt="Crail" id="logo"> - </a> - </div> - - - - <h2>SQL Performance: Part 1 - Input File Formats</h2> - - - <p class="meta">08 Aug 2018, <mark>this is a blog post from a user of the Crail project.</mark> </p> - -<div class="post"> -<div style="text-align: justify"> -<p> -This is the first user blog post in a multi-part series where we will focus on relational data processing performance (e.g., SQL) in presence of high-performance network and storage devices - the kind of devices that Crail targets. Relational data processing is one of the most popular and versatile workloads people run in the cloud. The general idea is that data is stored in tables with a schema, and is processed using a domain specific language like SQL. Examples of some popular systems that support such relational data analytics in the cloud are <a href="https://spark.apache.org/sql/";>Apache Spark/SQL</a>, <a href="https://hive.apache.org/";>Apache Hive</a>, <a href="https://impala.apache.org/";>Apache Impala</a>, etc. In this post, we discuss the important first step in relational data processing, which is the reading of input data tables. -</p> -</div> - -<h3 id="hardware-and-software-configuration">Hardware and Software Configuration</h3> - -<p>The specific cluster configuration used for the experiments in this blog:</p> - -<ul> - <li>Cluster - <ul> - <li>4 compute + 1 management node x86_64 cluster</li> - </ul> - </li> - <li>Node configuration - <ul> - <li>CPU: 2 x Intel(R) Xeon(R) CPU E5-2690 0 @ 2.90GHz</li> - <li>DRAM: 256 GB DDR3</li> - <li>Network: 1x100Gbit/s Mellanox ConnectX-5</li> - </ul> - </li> - <li>Software - <ul> - <li>Ubuntu 16.04.3 LTS (Xenial Xerus) with Linux kernel version 4.10.0-33-generic</li> - <li>Apache HDFS (2.7.3)</li> - <li>Apache Paruqet (1.8), Apache ORC (1.4), Apache Arrow (0.8), Apache Avro (1.4)</li> - <li><a href="https://github.com/apache/incubator-crail/";>Apache Crail (incubating) with NVMeF support</a>, commit 64e635e5ce9411041bf47fac5d7fadcb83a84355 (since then Crail has a stable source release v1.0 with a newer NVMeF code-base)</li> - </ul> - </li> -</ul> - -<h3 id="overview">Overview</h3> - -<p>In a typical cloud-based relational data processing setup, the input data is stored on an external data storage solution like HDFS or AWS S3. Data tables and their associated schema are converted into a storage-friendly format for optimal performance. Examples of some popular and familiar file formats are <a href="https://parquet.apache.org/";>Apache Parquet</a>, <a href="https://orc.apache.org/";>Apache ORC</a>, <a href="https://avro.apache.org/";>Apache Avro</a>, <a href="https://en.wikipedia.org/wiki/JSON";>JSON</a>, etc. More recently, <a href="https://arrow.apache.org/";>Apache Arrow</a> has been introduced to standardize the in-memory columnar data representation between multiple frameworks. To be precise, Arrow is not a storage format but it defines an <a href="https://github.com/apache/arrow/blob/master/format/IPC.md";>interprocess communication (IPC) format</a> that can be used to store data in a stroage system (our binding for reading Arrow IPC messages from HDFS is available <a href="https://github.com/zrlio/fileformat-benchmarks/blob/master/src/main/java/com/github/animeshtrivedi/FileBench/HdfsSeekableByteChannel.java";>here</a>). There is no one size fits all as all these formats have their own strengths, weaknesses, and features. In this blog, we are specifically interested in the performance of these formats on modern high-performance networking and storage devices.</p> - -<figure><div style="text-align:center"><img src="//crail.incubator.apache.org/img/blog/sql-p1/outline.svg" width="550" /><figcaption>Figure 1: The benchmarking setup with HDFS and file formats on a 100 Gbps network with NVMe flash devices. All formats contains routines for compression, encoding, and value materialization with associated I/O buffer management and data copies routines.<p></p></figcaption></div></figure> - -<p>To benchmark the performance of file formats, we wrote a set of micro-benchmarks which are available at <a href="https://github.com/zrlio/fileformat-benchmarks";>https://github.com/zrlio/fileformat-benchmarks</a>. We cannot use typical SQL micro-benchmarks because every SQL engine has its own favorite file format, on which it performs the best. Hence, in order to ensure parity, we decoupled the performance of reading the input file format from the SQL query processing by writing simple table reading micro-benchmarks. Our benchmark reads in the store_sales table from the TPC-DS dataset (scale factor 100), and calculates a sum of values present in the table. The table contains 23 columns of integers, doubles, and longs.</p> - -<figure><div style="text-align:center"><img src="//crail.incubator.apache.org/img/blog/sql-p1/performance-all.svg" width="550" /><figcaption>Figure 2: Performance of JSON, Avro, Parquet, ORC, and Arrow on NVMe devices over a 100 Gbps network.<p></p></figcaption></div></figure> - -<p>We evaluate the performance of the benchmark on a 3 node HDFS cluster connected using 100 Gbps RoCE. One datanode in HDFS contains 4 NVMe devices with a collective aggregate bandwidth of 12.5 GB/sec (equals to 100 Gbps, hence, we have a balanced network and storage performance). Figure 2 shows our results where none of the file formats is able to deliver the full hardware performance for reading input files. One third of the performance is already lost in HDFS (maximum throughput 74.9 Gbps out of possible 100 Gbps). The rest of the performance is lost inside the file format implementation, which needs to deal with encoding, buffer and I/O management, compression, etc. The best performer is Apache Arrow which is designed for in-memory columnar datasets. The performance of these file formats are bounded by the performance of the CPU, which is 100% loaded during the experiment. For a detailed analysis of the file formats, please refer to our paper - <a href="https://www.usenix.org/c onference/atc18/presentation/trivedi">Albis: High-Performance File Format for Big Data Systems (USENIX, ATCâ18)</a>. As a side-note on the Arrow performance - we have evaluated the performance of <em>implementation of Arrowâs Java library</em>. As this library has been focused on interactions with off-heap memory, there is a head room for optimizing the HDFS/on-heap reading path of Arrowâs Java library.</p> - -<h3 id="albis-high-performance-file-format-for-big-data-systems">Albis: High-Performance File Format for Big Data Systems</h3> - -<p>Based on these findings, we have developed a new file format called Albis. Albis is built on similar design choices as Crail. The top-level idea is to leverage the performance of modern networking and storage devices without being bottleneck by the CPU. While designing Albis we revisited many outdated assumptions about the nature of I/O in a distributed setting, and came up with the following ideas:</p> - -<ul> - <li>No compression or encoding: Modern network and storage devices are fast. Hence, there is no need to trade CPU cycles for performance. A 4 byte integer should be stored as a 4 byte value.</li> - <li>Keep the data/metadata management simple: Albis splits a table into row and column groups, which are stored in hierarchical files and directories on the underlying file system (e.g., HDFS or Crail).</li> - <li>Careful object materialization using a binary API: To optimize the runtime representation in managed runtimes like the JVM, only objects which are necessary for SQL processing are materialized. Otherwise, a 4 byte integer can be passed around as a byte array (using the binary API of Albis).</li> -</ul> - -<figure><div style="text-align:center"><img src="//crail.incubator.apache.org/img/blog/sql-p1/core-scalability.svg" width="550" /><figcaption>Figure 3: Core scalability of JSON, Avro, Parquet, ORC, Arrow, and Albis on HDFS/NVMe.<p></p></figcaption></div></figure> - -<p>Using the Albis format, we revise our previous experiment where we read the input store_sales table from HDFS. In the figure above, we show the performance of Albis and other file formats with number of CPU cores involved. At the right hand of the x-axis, we have performance with all 16 cores engaged, hence, representing the peak possible performance. As evident, Albis delivers 59.9 Gbps out of 74.9 Gbps possible bandwidth with HDFS over NVMe. Albis performance is 1.9 - 21.4x better than other file formats. To give an impression where the performance is coming from, in the table below we show some micro-architectural features for Parquet, ORC, Arrow, and Albis. Our previously discussed design ideas in Albis result in a shorter code path (shown as less instructions required for each row), better cache performance (shows as lower cache misses per row), and clearly better performance (shown as nanoseconds required per row for processing). For a detailed evaluation of Albis please re fer to our paper.</p> - -<table style="width:100%"> - <caption> Table 1: Micro-architectural analysis for Parquet, ORC, Arrow, and Albis on a 16-core Xeon machine.<p></p></caption> - <tr> - <th></th> - <th>Parquet</th> - <th>ORC</th> - <th>Arrow</th> - <th>Albis</th> - </tr> - <tr> - <th>Instructions/row</th> - <td>6.6K</td> - <td>4.9K</td> - <td>1.9K</td> - <td>1.6K</td> - </tr> - <tr> - <th>Cache misses/row</th> - <td>9.2</td> - <td>4.6</td> - <td>5.1</td> - <td>3.0</td> - </tr> - <tr> - <th>Nanoseconds/row</th> - <td>105.3</td> - <td>63.0</td> - <td>31.2</td> - <td>20.8</td> - </tr> -</table> -<p></p> - -<h3 id="apache-crail-incubating-with-albis">Apache Crail (Incubating) with Albis</h3> - -<p>For our final experiment, we try to answer the question what it would take to deliver the full 100 Gbps bandwidth for Albis. Certainly, the first bottleneck is to improve the base storage layer performance. Here we use Apache Crail (Incubating) with its <a href="https://en.wikipedia.org/wiki/NVM_Express#NVMeOF";>NVMeF</a> storage tier. This tier uses <a href="https://github.com/zrlio/jNVMf";>jNVMf library</a> to implement NVMeF stack in Java. As we have shown in a previous blog <a href="//crail.incubator.apache.org/blog/2017/08/crail-nvme-fabrics-v1.html">post</a> that Crailâs NVMeF tier can deliver performance (97.8 Gbps) very close to the hardware limits. Hence, Albis with Crail is a perfect setup to evaluate on high-performance NVMe and RDMA devices. Before we get there, letâs get some calculations right. The store_sales table in the TPC-DS dataset has a data density of 93.9% (out of 100 bytes, only 93.9 is data, others are null values). As we measure the goodput, the expect ed performance of Albis on Crail is 93.9% of 97.8 Gbps, which calculates to 91.8 Gbps. In our experiments, Albis on Crail delivers 85.5 Gbps. Figure 4 shows more detailed results.</p> - -<figure><div style="text-align:center"><img src="//crail.incubator.apache.org/img/blog/sql-p1/albis-crail.svg" width="550" /><figcaption>Figure 4: Performance of Albis on Crail.<p></p></figcaption></div></figure> - -<p>The left half of the figure shows the performance scalability of Albis on Crail in a setup with 1 core (8.9 Gbps) to 16 cores (85.5 Gbps). In comparison, the right half of the figure shows the performance of Crail on HDFS/NVMe at 59.9 Gbps, and on Crail/NVMe at 85.5 Gbps. The last bar shows the performance of Albis if the benchmark does not materialize Java object values. In this configuration, Albis on Crail delivers 91.3 Gbps, which is very close to the expected peak of 91.8 Gbps.</p> - -<h3 id="summary">Summary</h3> -<div style="text-align: justify"> -<p> -In this first blog of a multipart series, we have looked at the data ingestion performance of file formats on high-performance networking and storage devices. We found that popular file formats are in need for a performance revision. Based on our analysis, we designed and implemented Albis - a new file format for storing relational data. Albis and Crail share many design choices. Their combined performance of 85+ Gbps on a 100 Gbps network, gives us confidence in our approach and underlying software philosophy for both, Crail and Albis. -</p> - -<p> -Stay tuned for the next part where we look at workload-level performance in Spark/SQL on modern high-performance networking and storage devices. Meanwhile let us know if you have any feedback or comments. -</p> -</div> - -</div> - -<!-- - -<div id="disqus_thread"></div> -<script> - -/** -* RECOMMENDED CONFIGURATION VARIABLES: EDIT AND UNCOMMENT THE SECTION BELOW TO INSERT DYNAMIC VALUES FROM YOUR PLATFORM OR CMS. -* LEARN WHY DEFINING THESE VARIABLES IS IMPORTANT: https://disqus.com/admin/universalcode/#configuration-variables*/ -/* -var disqus_config = function () { -this.page.url = PAGE_URL; // Replace PAGE_URL with your page's canonical URL variable -this.page.identifier = PAGE_IDENTIFIER; // Replace PAGE_IDENTIFIER with your page's unique identifier variable -}; -*/ -(function() { // DON'T EDIT BELOW THIS LINE -var d = document, s = d.createElement('script'); -s.src = '//crail-io.disqus.com/embed.js'; -s.setAttribute('data-timestamp', +new Date()); -(d.head || d.body).appendChild(s); -})(); -</script> -<noscript>Please enable JavaScript to view the <a href="https://disqus.com/?ref_noscript";>comments powered by Disqus.</a></noscript> - ---> - - - <br> - <br> - <div class="footer"> - <p>Apache Crail is an effort undergoing <a href="https://incubator.apache.org/";>incubation</a> at <a href="https://www.apache.org/";>The Apache Software Foundation (ASF)</a>, sponsored by the Apache Incubator PMC. Incubation is required of all newly accepted projects until a further review indicates that the infrastructure, communications, and decision making process have stabilized in a manner consistent with other successful ASF projects. While incubation status is not necessarily a reflection of the completeness or stability of the code, it does indicate that the project has yet to be fully endorsed by the ASF. - </p> - </div> - - </div> <!-- /container --> - - <!-- Support retina images. --> - <script type="text/javascript" - src="//crail.incubator.apache.org/js/srcset-polyfill.js"></script> - </body> -</html> http://git-wip-us.apache.org/repos/asf/incubator-crail-website/blob/004e8a6d/content/blog/index.html ---------------------------------------------------------------------- diff --git a/content/blog/index.html b/content/blog/index.html deleted file mode 100644 index 0efa9ee..0000000 --- a/content/blog/index.html +++ /dev/null @@ -1,184 +0,0 @@ -<!DOCTYPE html> -<html> - <head> - <meta charset="utf-8"> - <title>The Apache Crail (Incubating) Project: Latest posts</title> - <meta name="viewport" content="width=device-width, initial-scale=1.0"> - <link href="//crail.incubator.apache.org/css/bootstrap.min.css" rel="stylesheet"> - <link href="//crail.incubator.apache.org/css/group.css" rel="stylesheet"> - <link rel="alternate" type="application/atom+xml" title="Atom" - href="//crail.incubator.apache.org/blog/blog.xml"> - - <meta property="og:image" content="//crail.incubator.apache.org/img/blog/preview/blog-summary.png" /> - <meta property="og:image:secure_url" content="//crail.incubator.apache.org/img/blog/preview/blog-summary.png" /> - </head> - - <body> - <div class="container"> - <div class="header"> - <ul class="nav nav-pills pull-right"> - - - - <li > - <a href="//crail.incubator.apache.org/"> - Home - </a> - </li> - - - <li > - <a href="//crail.incubator.apache.org/overview/"> - Overview - </a> - </li> - - - <li > - <a href="//crail.incubator.apache.org/download/"> - Downloads - </a> - </li> - - - <li class="active"> - <a href="//crail.incubator.apache.org/blog/"> - Blog - </a> - </li> - - - <li > - <a href="//crail.incubator.apache.org/community/"> - Community - </a> - </li> - - - <li > - <a href="//crail.incubator.apache.org/documentation/"> - Documentation - </a> - </li> - - </ul> - <a href="//crail.incubator.apache.org/"> - <img src="//crail.incubator.apache.org/img/crail_logo.png" - srcset="//crail.incubator.apache.org/img/crail_logo.png" - alt="Crail" id="logo"> - </a> - </div> - - - - <h2>Latest posts</h2> - - - -<div class="post-preview"> - <a href="/blog/2018/08/sql-p1.html"> - <h2 class="post-title"> - SQL Performance: Part 1 - Input File Formats - </h2> - - </a> - - <p class="post-meta"><b>User Post</b> by Animesh Trivedi on August 8, 2018</p> - -</div> -<hr> - -<div class="post-preview"> - <a href="/blog/2017/11/crail-metadata.html"> - <h2 class="post-title"> - Crail Storage Performance -- Part III: Metadata - </h2> - - </a> - - <p class="post-meta"><b>Developer Post</b> by Adrian Schuepbach and Patrick Stuedi on November 21, 2017</p> - -</div> -<hr> - -<div class="post-preview"> - <a href="/blog/2017/11/rdmashuffle.html"> - <h2 class="post-title"> - Spark Shuffle: SparkRDMA vs Crail - </h2> - - </a> - - <p class="post-meta"><b>User Post</b> by Jonas Pfefferle, Patrick Stuedi, Animesh Trivedi, Bernard Metzler, Adrian Schuepbach on November 17, 2017</p> - -</div> -<hr> - -<div class="post-preview"> - <a href="/blog/2017/08/crail-nvme-fabrics-v1.html"> - <h2 class="post-title"> - Crail Storage Performance -- Part II: NVMf - </h2> - - </a> - - <p class="post-meta"><b>Developer Post</b> by Jonas Pfefferle on August 22, 2017</p> - -</div> -<hr> - -<div class="post-preview"> - <a href="/blog/2017/08/crail-memory.html"> - <h2 class="post-title"> - Crail Storage Performance -- Part I: DRAM - </h2> - - </a> - - <p class="post-meta"><b>Developer Post</b> by Patrick Stuedi on August 18, 2017</p> - -</div> -<hr> - -<div class="post-preview"> - <a href="/blog/2017/01/sorting.html"> - <h2 class="post-title"> - Sorting on a 100Gbit/s Cluster using Spark/Crail - </h2> - - </a> - - <p class="post-meta"><b>User Post</b> by Patrick Stuedi on January 17, 2017</p> - -</div> -<hr> - - -<!-- Pager --> - -<ul class="pager"> - - - <li class="next"> - <a href="/blog/page2">Older Posts →</a> - </li> - -</ul> - - - - - <br> - <br> - <div class="footer"> - <p>Apache Crail is an effort undergoing <a href="https://incubator.apache.org/";>incubation</a> at <a href="https://www.apache.org/";>The Apache Software Foundation (ASF)</a>, sponsored by the Apache Incubator PMC. Incubation is required of all newly accepted projects until a further review indicates that the infrastructure, communications, and decision making process have stabilized in a manner consistent with other successful ASF projects. While incubation status is not necessarily a reflection of the completeness or stability of the code, it does indicate that the project has yet to be fully endorsed by the ASF. - </p> - </div> - - </div> <!-- /container --> - - <!-- Support retina images. --> - <script type="text/javascript" - src="//crail.incubator.apache.org/js/srcset-polyfill.js"></script> - </body> -</html> http://git-wip-us.apache.org/repos/asf/incubator-crail-website/blob/004e8a6d/content/blog/page2/index.html ---------------------------------------------------------------------- diff --git a/content/blog/page2/index.html b/content/blog/page2/index.html deleted file mode 100644 index 4154bc4..0000000 --- a/content/blog/page2/index.html +++ /dev/null @@ -1,188 +0,0 @@ -<!DOCTYPE html> -<html> - <head> - <meta charset="utf-8"> - <title>The Apache Crail (Incubating) Project: Latest posts</title> - <meta name="viewport" content="width=device-width, initial-scale=1.0"> - <link href="//crail.incubator.apache.org/css/bootstrap.min.css" rel="stylesheet"> - <link href="//crail.incubator.apache.org/css/group.css" rel="stylesheet"> - <link rel="alternate" type="application/atom+xml" title="Atom" - href="//crail.incubator.apache.org/blog/blog.xml"> - - <meta property="og:image" content="//crail.incubator.apache.org/img/blog/preview/page2-summary.png" /> - <meta property="og:image:secure_url" content="//crail.incubator.apache.org/img/blog/preview/page2-summary.png" /> - </head> - - <body> - <div class="container"> - <div class="header"> - <ul class="nav nav-pills pull-right"> - - - - <li > - <a href="//crail.incubator.apache.org/"> - Home - </a> - </li> - - - <li > - <a href="//crail.incubator.apache.org/overview/"> - Overview - </a> - </li> - - - <li > - <a href="//crail.incubator.apache.org/download/"> - Downloads - </a> - </li> - - - <li > - <a href="//crail.incubator.apache.org/blog/"> - Blog - </a> - </li> - - - <li > - <a href="//crail.incubator.apache.org/community/"> - Community - </a> - </li> - - - <li > - <a href="//crail.incubator.apache.org/documentation/"> - Documentation - </a> - </li> - - </ul> - <a href="//crail.incubator.apache.org/"> - <img src="//crail.incubator.apache.org/img/crail_logo.png" - srcset="//crail.incubator.apache.org/img/crail_logo.png" - alt="Crail" id="logo"> - </a> - </div> - - - - <h2>Latest posts</h2> - - - -<div class="post-preview"> - <a href="/blog/2018/08/sql-p1.html"> - <h2 class="post-title"> - SQL Performance: Part 1 - Input File Formats - </h2> - - </a> - - <p class="post-meta"><b>User Post</b> by Animesh Trivedi on August 8, 2018</p> - -</div> -<hr> - -<div class="post-preview"> - <a href="/blog/2017/11/crail-metadata.html"> - <h2 class="post-title"> - Crail Storage Performance -- Part III: Metadata - </h2> - - </a> - - <p class="post-meta"><b>Developer Post</b> by Adrian Schuepbach and Patrick Stuedi on November 21, 2017</p> - -</div> -<hr> - -<div class="post-preview"> - <a href="/blog/2017/11/rdmashuffle.html"> - <h2 class="post-title"> - Spark Shuffle: SparkRDMA vs Crail - </h2> - - </a> - - <p class="post-meta"><b>User Post</b> by Jonas Pfefferle, Patrick Stuedi, Animesh Trivedi, Bernard Metzler, Adrian Schuepbach on November 17, 2017</p> - -</div> -<hr> - -<div class="post-preview"> - <a href="/blog/2017/08/crail-nvme-fabrics-v1.html"> - <h2 class="post-title"> - Crail Storage Performance -- Part II: NVMf - </h2> - - </a> - - <p class="post-meta"><b>Developer Post</b> by Jonas Pfefferle on August 22, 2017</p> - -</div> -<hr> - -<div class="post-preview"> - <a href="/blog/2017/08/crail-memory.html"> - <h2 class="post-title"> - Crail Storage Performance -- Part I: DRAM - </h2> - - </a> - - <p class="post-meta"><b>Developer Post</b> by Patrick Stuedi on August 18, 2017</p> - -</div> -<hr> - -<div class="post-preview"> - <a href="/blog/2017/01/sorting.html"> - <h2 class="post-title"> - Sorting on a 100Gbit/s Cluster using Spark/Crail - </h2> - - </a> - - <p class="post-meta"><b>User Post</b> by Patrick Stuedi on January 17, 2017</p> - -</div> -<hr> - - -<!-- Pager --> - -<ul class="pager"> - - <li class="previous"> - <a href="/blog/">← Newer Posts</a> - </li> - - - <li class="next"> - <a href="/blog/page3">Older Posts →</a> - </li> - -</ul> - - - - - <br> - <br> - <div class="footer"> - <p>Apache Crail is an effort undergoing <a href="https://incubator.apache.org/";>incubation</a> at <a href="https://www.apache.org/";>The Apache Software Foundation (ASF)</a>, sponsored by the Apache Incubator PMC. Incubation is required of all newly accepted projects until a further review indicates that the infrastructure, communications, and decision making process have stabilized in a manner consistent with other successful ASF projects. While incubation status is not necessarily a reflection of the completeness or stability of the code, it does indicate that the project has yet to be fully endorsed by the ASF. - </p> - </div> - - </div> <!-- /container --> - - <!-- Support retina images. --> - <script type="text/javascript" - src="//crail.incubator.apache.org/js/srcset-polyfill.js"></script> - </body> -</html> http://git-wip-us.apache.org/repos/asf/incubator-crail-website/blob/004e8a6d/content/blog/page3/index.html ---------------------------------------------------------------------- diff --git a/content/blog/page3/index.html b/content/blog/page3/index.html deleted file mode 100644 index 3953e16..0000000 --- a/content/blog/page3/index.html +++ /dev/null @@ -1,188 +0,0 @@ -<!DOCTYPE html> -<html> - <head> - <meta charset="utf-8"> - <title>The Apache Crail (Incubating) Project: Latest posts</title> - <meta name="viewport" content="width=device-width, initial-scale=1.0"> - <link href="//crail.incubator.apache.org/css/bootstrap.min.css" rel="stylesheet"> - <link href="//crail.incubator.apache.org/css/group.css" rel="stylesheet"> - <link rel="alternate" type="application/atom+xml" title="Atom" - href="//crail.incubator.apache.org/blog/blog.xml"> - - <meta property="og:image" content="//crail.incubator.apache.org/img/blog/preview/page3-summary.png" /> - <meta property="og:image:secure_url" content="//crail.incubator.apache.org/img/blog/preview/page3-summary.png" /> - </head> - - <body> - <div class="container"> - <div class="header"> - <ul class="nav nav-pills pull-right"> - - - - <li > - <a href="//crail.incubator.apache.org/"> - Home - </a> - </li> - - - <li > - <a href="//crail.incubator.apache.org/overview/"> - Overview - </a> - </li> - - - <li > - <a href="//crail.incubator.apache.org/download/"> - Downloads - </a> - </li> - - - <li > - <a href="//crail.incubator.apache.org/blog/"> - Blog - </a> - </li> - - - <li > - <a href="//crail.incubator.apache.org/community/"> - Community - </a> - </li> - - - <li > - <a href="//crail.incubator.apache.org/documentation/"> - Documentation - </a> - </li> - - </ul> - <a href="//crail.incubator.apache.org/"> - <img src="//crail.incubator.apache.org/img/crail_logo.png" - srcset="//crail.incubator.apache.org/img/crail_logo.png" - alt="Crail" id="logo"> - </a> - </div> - - - - <h2>Latest posts</h2> - - - -<div class="post-preview"> - <a href="/blog/2018/08/sql-p1.html"> - <h2 class="post-title"> - SQL Performance: Part 1 - Input File Formats - </h2> - - </a> - - <p class="post-meta"><b>User Post</b> by Animesh Trivedi on August 8, 2018</p> - -</div> -<hr> - -<div class="post-preview"> - <a href="/blog/2017/11/crail-metadata.html"> - <h2 class="post-title"> - Crail Storage Performance -- Part III: Metadata - </h2> - - </a> - - <p class="post-meta"><b>Developer Post</b> by Adrian Schuepbach and Patrick Stuedi on November 21, 2017</p> - -</div> -<hr> - -<div class="post-preview"> - <a href="/blog/2017/11/rdmashuffle.html"> - <h2 class="post-title"> - Spark Shuffle: SparkRDMA vs Crail - </h2> - - </a> - - <p class="post-meta"><b>User Post</b> by Jonas Pfefferle, Patrick Stuedi, Animesh Trivedi, Bernard Metzler, Adrian Schuepbach on November 17, 2017</p> - -</div> -<hr> - -<div class="post-preview"> - <a href="/blog/2017/08/crail-nvme-fabrics-v1.html"> - <h2 class="post-title"> - Crail Storage Performance -- Part II: NVMf - </h2> - - </a> - - <p class="post-meta"><b>Developer Post</b> by Jonas Pfefferle on August 22, 2017</p> - -</div> -<hr> - -<div class="post-preview"> - <a href="/blog/2017/08/crail-memory.html"> - <h2 class="post-title"> - Crail Storage Performance -- Part I: DRAM - </h2> - - </a> - - <p class="post-meta"><b>Developer Post</b> by Patrick Stuedi on August 18, 2017</p> - -</div> -<hr> - -<div class="post-preview"> - <a href="/blog/2017/01/sorting.html"> - <h2 class="post-title"> - Sorting on a 100Gbit/s Cluster using Spark/Crail - </h2> - - </a> - - <p class="post-meta"><b>User Post</b> by Patrick Stuedi on January 17, 2017</p> - -</div> -<hr> - - -<!-- Pager --> - -<ul class="pager"> - - <li class="previous"> - <a href="/blog/page2">← Newer Posts</a> - </li> - - - <li class="next"> - <a href="/blog/page4">Older Posts →</a> - </li> - -</ul> - - - - - <br> - <br> - <div class="footer"> - <p>Apache Crail is an effort undergoing <a href="https://incubator.apache.org/";>incubation</a> at <a href="https://www.apache.org/";>The Apache Software Foundation (ASF)</a>, sponsored by the Apache Incubator PMC. Incubation is required of all newly accepted projects until a further review indicates that the infrastructure, communications, and decision making process have stabilized in a manner consistent with other successful ASF projects. While incubation status is not necessarily a reflection of the completeness or stability of the code, it does indicate that the project has yet to be fully endorsed by the ASF. - </p> - </div> - - </div> <!-- /container --> - - <!-- Support retina images. --> - <script type="text/javascript" - src="//crail.incubator.apache.org/js/srcset-polyfill.js"></script> - </body> -</html> http://git-wip-us.apache.org/repos/asf/incubator-crail-website/blob/004e8a6d/content/blog/page4/index.html ---------------------------------------------------------------------- diff --git a/content/blog/page4/index.html b/content/blog/page4/index.html deleted file mode 100644 index 62129bb..0000000 --- a/content/blog/page4/index.html +++ /dev/null @@ -1,188 +0,0 @@ -<!DOCTYPE html> -<html> - <head> - <meta charset="utf-8"> - <title>The Apache Crail (Incubating) Project: Latest posts</title> - <meta name="viewport" content="width=device-width, initial-scale=1.0"> - <link href="//crail.incubator.apache.org/css/bootstrap.min.css" rel="stylesheet"> - <link href="//crail.incubator.apache.org/css/group.css" rel="stylesheet"> - <link rel="alternate" type="application/atom+xml" title="Atom" - href="//crail.incubator.apache.org/blog/blog.xml"> - - <meta property="og:image" content="//crail.incubator.apache.org/img/blog/preview/page4-summary.png" /> - <meta property="og:image:secure_url" content="//crail.incubator.apache.org/img/blog/preview/page4-summary.png" /> - </head> - - <body> - <div class="container"> - <div class="header"> - <ul class="nav nav-pills pull-right"> - - - - <li > - <a href="//crail.incubator.apache.org/"> - Home - </a> - </li> - - - <li > - <a href="//crail.incubator.apache.org/overview/"> - Overview - </a> - </li> - - - <li > - <a href="//crail.incubator.apache.org/download/"> - Downloads - </a> - </li> - - - <li > - <a href="//crail.incubator.apache.org/blog/"> - Blog - </a> - </li> - - - <li > - <a href="//crail.incubator.apache.org/community/"> - Community - </a> - </li> - - - <li > - <a href="//crail.incubator.apache.org/documentation/"> - Documentation - </a> - </li> - - </ul> - <a href="//crail.incubator.apache.org/"> - <img src="//crail.incubator.apache.org/img/crail_logo.png" - srcset="//crail.incubator.apache.org/img/crail_logo.png" - alt="Crail" id="logo"> - </a> - </div> - - - - <h2>Latest posts</h2> - - - -<div class="post-preview"> - <a href="/blog/2018/08/sql-p1.html"> - <h2 class="post-title"> - SQL Performance: Part 1 - Input File Formats - </h2> - - </a> - - <p class="post-meta"><b>User Post</b> by Animesh Trivedi on August 8, 2018</p> - -</div> -<hr> - -<div class="post-preview"> - <a href="/blog/2017/11/crail-metadata.html"> - <h2 class="post-title"> - Crail Storage Performance -- Part III: Metadata - </h2> - - </a> - - <p class="post-meta"><b>Developer Post</b> by Adrian Schuepbach and Patrick Stuedi on November 21, 2017</p> - -</div> -<hr> - -<div class="post-preview"> - <a href="/blog/2017/11/rdmashuffle.html"> - <h2 class="post-title"> - Spark Shuffle: SparkRDMA vs Crail - </h2> - - </a> - - <p class="post-meta"><b>User Post</b> by Jonas Pfefferle, Patrick Stuedi, Animesh Trivedi, Bernard Metzler, Adrian Schuepbach on November 17, 2017</p> - -</div> -<hr> - -<div class="post-preview"> - <a href="/blog/2017/08/crail-nvme-fabrics-v1.html"> - <h2 class="post-title"> - Crail Storage Performance -- Part II: NVMf - </h2> - - </a> - - <p class="post-meta"><b>Developer Post</b> by Jonas Pfefferle on August 22, 2017</p> - -</div> -<hr> - -<div class="post-preview"> - <a href="/blog/2017/08/crail-memory.html"> - <h2 class="post-title"> - Crail Storage Performance -- Part I: DRAM - </h2> - - </a> - - <p class="post-meta"><b>Developer Post</b> by Patrick Stuedi on August 18, 2017</p> - -</div> -<hr> - -<div class="post-preview"> - <a href="/blog/2017/01/sorting.html"> - <h2 class="post-title"> - Sorting on a 100Gbit/s Cluster using Spark/Crail - </h2> - - </a> - - <p class="post-meta"><b>User Post</b> by Patrick Stuedi on January 17, 2017</p> - -</div> -<hr> - - -<!-- Pager --> - -<ul class="pager"> - - <li class="previous"> - <a href="/blog/page3">← Newer Posts</a> - </li> - - - <li class="next"> - <a href="/blog/page5">Older Posts →</a> - </li> - -</ul> - - - - - <br> - <br> - <div class="footer"> - <p>Apache Crail is an effort undergoing <a href="https://incubator.apache.org/";>incubation</a> at <a href="https://www.apache.org/";>The Apache Software Foundation (ASF)</a>, sponsored by the Apache Incubator PMC. Incubation is required of all newly accepted projects until a further review indicates that the infrastructure, communications, and decision making process have stabilized in a manner consistent with other successful ASF projects. While incubation status is not necessarily a reflection of the completeness or stability of the code, it does indicate that the project has yet to be fully endorsed by the ASF. - </p> - </div> - - </div> <!-- /container --> - - <!-- Support retina images. --> - <script type="text/javascript" - src="//crail.incubator.apache.org/js/srcset-polyfill.js"></script> - </body> -</html> http://git-wip-us.apache.org/repos/asf/incubator-crail-website/blob/004e8a6d/content/blog/page5/index.html ---------------------------------------------------------------------- diff --git a/content/blog/page5/index.html b/content/blog/page5/index.html deleted file mode 100644 index bb9ecdd..0000000 --- a/content/blog/page5/index.html +++ /dev/null @@ -1,184 +0,0 @@ -<!DOCTYPE html> -<html> - <head> - <meta charset="utf-8"> - <title>The Apache Crail (Incubating) Project: Latest posts</title> - <meta name="viewport" content="width=device-width, initial-scale=1.0"> - <link href="//crail.incubator.apache.org/css/bootstrap.min.css" rel="stylesheet"> - <link href="//crail.incubator.apache.org/css/group.css" rel="stylesheet"> - <link rel="alternate" type="application/atom+xml" title="Atom" - href="//crail.incubator.apache.org/blog/blog.xml"> - - <meta property="og:image" content="//crail.incubator.apache.org/img/blog/preview/page5-summary.png" /> - <meta property="og:image:secure_url" content="//crail.incubator.apache.org/img/blog/preview/page5-summary.png" /> - </head> - - <body> - <div class="container"> - <div class="header"> - <ul class="nav nav-pills pull-right"> - - - - <li > - <a href="//crail.incubator.apache.org/"> - Home - </a> - </li> - - - <li > - <a href="//crail.incubator.apache.org/overview/"> - Overview - </a> - </li> - - - <li > - <a href="//crail.incubator.apache.org/download/"> - Downloads - </a> - </li> - - - <li > - <a href="//crail.incubator.apache.org/blog/"> - Blog - </a> - </li> - - - <li > - <a href="//crail.incubator.apache.org/community/"> - Community - </a> - </li> - - - <li > - <a href="//crail.incubator.apache.org/documentation/"> - Documentation - </a> - </li> - - </ul> - <a href="//crail.incubator.apache.org/"> - <img src="//crail.incubator.apache.org/img/crail_logo.png" - srcset="//crail.incubator.apache.org/img/crail_logo.png" - alt="Crail" id="logo"> - </a> - </div> - - - - <h2>Latest posts</h2> - - - -<div class="post-preview"> - <a href="/blog/2018/08/sql-p1.html"> - <h2 class="post-title"> - SQL Performance: Part 1 - Input File Formats - </h2> - - </a> - - <p class="post-meta"><b>User Post</b> by Animesh Trivedi on August 8, 2018</p> - -</div> -<hr> - -<div class="post-preview"> - <a href="/blog/2017/11/crail-metadata.html"> - <h2 class="post-title"> - Crail Storage Performance -- Part III: Metadata - </h2> - - </a> - - <p class="post-meta"><b>Developer Post</b> by Adrian Schuepbach and Patrick Stuedi on November 21, 2017</p> - -</div> -<hr> - -<div class="post-preview"> - <a href="/blog/2017/11/rdmashuffle.html"> - <h2 class="post-title"> - Spark Shuffle: SparkRDMA vs Crail - </h2> - - </a> - - <p class="post-meta"><b>User Post</b> by Jonas Pfefferle, Patrick Stuedi, Animesh Trivedi, Bernard Metzler, Adrian Schuepbach on November 17, 2017</p> - -</div> -<hr> - -<div class="post-preview"> - <a href="/blog/2017/08/crail-nvme-fabrics-v1.html"> - <h2 class="post-title"> - Crail Storage Performance -- Part II: NVMf - </h2> - - </a> - - <p class="post-meta"><b>Developer Post</b> by Jonas Pfefferle on August 22, 2017</p> - -</div> -<hr> - -<div class="post-preview"> - <a href="/blog/2017/08/crail-memory.html"> - <h2 class="post-title"> - Crail Storage Performance -- Part I: DRAM - </h2> - - </a> - - <p class="post-meta"><b>Developer Post</b> by Patrick Stuedi on August 18, 2017</p> - -</div> -<hr> - -<div class="post-preview"> - <a href="/blog/2017/01/sorting.html"> - <h2 class="post-title"> - Sorting on a 100Gbit/s Cluster using Spark/Crail - </h2> - - </a> - - <p class="post-meta"><b>User Post</b> by Patrick Stuedi on January 17, 2017</p> - -</div> -<hr> - - -<!-- Pager --> - -<ul class="pager"> - - <li class="previous"> - <a href="/blog/page4">← Newer Posts</a> - </li> - - -</ul> - - - - - <br> - <br> - <div class="footer"> - <p>Apache Crail is an effort undergoing <a href="https://incubator.apache.org/";>incubation</a> at <a href="https://www.apache.org/";>The Apache Software Foundation (ASF)</a>, sponsored by the Apache Incubator PMC. Incubation is required of all newly accepted projects until a further review indicates that the infrastructure, communications, and decision making process have stabilized in a manner consistent with other successful ASF projects. While incubation status is not necessarily a reflection of the completeness or stability of the code, it does indicate that the project has yet to be fully endorsed by the ASF. - </p> - </div> - - </div> <!-- /container --> - - <!-- Support retina images. --> - <script type="text/javascript" - src="//crail.incubator.apache.org/js/srcset-polyfill.js"></script> - </body> -</html>
