http://git-wip-us.apache.org/repos/asf/drill-site/blob/bbdefcb2/_site/blog/2014/12/02/drill-top-level-project/index.html ---------------------------------------------------------------------- diff --git a/_site/blog/2014/12/02/drill-top-level-project/index.html b/_site/blog/2014/12/02/drill-top-level-project/index.html new file mode 100644 index 0000000..9407273 --- /dev/null +++ b/_site/blog/2014/12/02/drill-top-level-project/index.html @@ -0,0 +1,210 @@ +<!DOCTYPE html> +<html> + +<head> + +<meta charset="UTF-8"> +<meta name=viewport content="width=device-width, initial-scale=1"> +<meta name="robots" content="noindex"> + +<title>Apache Drill Graduates to a Top-Level Project - Apache Drill</title> + +<link href="//maxcdn.bootstrapcdn.com/font-awesome/4.3.0/css/font-awesome.min.css" rel="stylesheet" type="text/css"/> +<link href='//fonts.googleapis.com/css?family=PT+Sans' rel='stylesheet' type='text/css'/> +<link href="/drill/css/site.css" rel="stylesheet" type="text/css"/> + +<link rel="shortcut icon" href="/drill/favicon.ico" type="image/x-icon"/> +<link rel="icon" href="/drill/favicon.ico" type="image/x-icon"/> + +<script src="//ajax.googleapis.com/ajax/libs/jquery/1.11.1/jquery.min.js" language="javascript" type="text/javascript"></script> +<script src="//cdnjs.cloudflare.com/ajax/libs/jquery-easing/1.3/jquery.easing.min.js" language="javascript" type="text/javascript"></script> +<script language="javascript" type="text/javascript" src="/drill/js/modernizr.custom.js"></script> +<script language="javascript" type="text/javascript" src="/drill/js/script.js"></script> +<script language="javascript" type="text/javascript" src="/drill/js/drill.js"></script> + +</head> + + +<body onResize="resized();"> + <div class="page-wrap"> + <div class="bui"></div> + +<div id="menu" class="mw"> +<ul> + <li class='toc-categories'> + <a class="expand-toc-icon" href="javascript:void(0);"><i class="fa fa-bars"></i></a> + </li> + <li class="logo"><a href="/drill/"></a></li> + <li class='expand-menu'> + <a href="javascript:void(0);"><span class='menu-text'>Menu</span><span class='expand-icon'><i class="fa fa-bars"></i></span></a> + </li> + <li class='clear-float'></li> + <li class="documentation-menu"> + <a href="/drill/docs/">Documentation</a> + <ul> + + <li><a href="/drill/docs/getting-started/">Getting Started</a></li> + + <li><a href="/drill/docs/architecture/">Architecture</a></li> + + <li><a href="/drill/docs/tutorials/">Tutorials</a></li> + + <li><a href="/drill/docs/install-drill/">Install Drill</a></li> + + <li><a href="/drill/docs/configure-drill/">Configure Drill</a></li> + + <li><a href="/drill/docs/connect-a-data-source/">Connect a Data Source</a></li> + + <li><a href="/drill/docs/odbc-jdbc-interfaces/">ODBC/JDBC Interfaces</a></li> + + <li><a href="/drill/docs/query-data/">Query Data</a></li> + + <li><a href="/drill/docs/performance-tuning/">Performance Tuning</a></li> + + <li><a href="/drill/docs/log-and-debug/">Log and Debug</a></li> + + <li><a href="/drill/docs/sql-reference/">SQL Reference</a></li> + + <li><a href="/drill/docs/data-sources-and-file-formats/">Data Sources and File Formats</a></li> + + <li><a href="/drill/docs/develop-custom-functions/">Develop Custom Functions</a></li> + + <li><a href="/drill/docs/troubleshooting/">Troubleshooting</a></li> + + <li><a href="/drill/docs/developer-information/">Developer Information</a></li> + + <li><a href="/drill/docs/release-notes/">Release Notes</a></li> + + <li><a href="/drill/docs/sample-datasets/">Sample Datasets</a></li> + + <li><a href="/drill/docs/project-bylaws/">Project Bylaws</a></li> + + </ul> + </li> + <li class='nav'> + <a href="/drill/community-resources/">Community</a> + <ul> + <li><a href="/drill/team/">Team</a></li> + <li><a href="/drill/mailinglists/">Mailing Lists</a></li> + <li><a href="/drill/community-resources/">Community Resources</a></li> + </ul> + </li> + <li class='nav'><a href="/drill/faq/">FAQ</a></li> + <li class='nav'><a href="/drill/blog/">Blog</a></li> + <li id="twitter-menu-item"><a href="https://twitter.com/apachedrill"; title="apachedrill on twitter" target="_blank"><img src="/drill/images/twitter_32_26_white.png" alt="twitter logo" align="center"></a> </li> + <li class='search-bar'> + <form id="drill-search-form"> + <input type="text" placeholder="Search Apache Drill" id="drill-search-term" /> + <button type="submit"> + <i class="fa fa-search"></i> + </button> + </form> + </li> + <li class="d"> + <a href="/drill/download/"> + <i class="fa fa-cloud-download"></i> Download + </a> + </li> +</ul> +</div> + + <link href="/drill/css/content.css" rel="stylesheet" type="text/css"> + +<div class="post int_text"> + <header class="post-header"> + <div class="int_title"> + <h1 class="post-title">Apache Drill Graduates to a Top-Level Project</h1> + </div> + <p class="post-meta"> + + + + <strong>Author:</strong> Tomer Shiran (Founder, PMC Member and Committer, Apache Drill)<br /> + +<strong>Date:</strong> Dec 2, 2014 +</p> + </header> + <div class="addthis_sharing_toolbox"></div> + + <article class="post-content"> + <p>The Apache Software Foundation has just announced that it has promoted Drill to a top-level project at Apache, similar to other well-known projects like Apache Hadoop and httpd (the world's most popular Web server). This marks a significant accomplishment for the Drill community, and I wanted to personally thank everyone who has contributed to the project. It takes many people, and countless hours, to develop something as complex and innovative as Drill.</p> + +<p>In this post I wanted to reflect on the past and future of Drill.</p> + +<h2 id="why-we-started-drill">Why We Started Drill</h2> + +<h3 id="the-evolution-of-application-development-and-data">The Evolution of Application Development and Data</h3> + +<p>Over the last decade, organizations have been striving to become more agile and data-driven, seeking to gain competitive advantage in their markets. This trend has led to dramatic changes in the way applications are built and delivered, and in the type and volume of data that is being leveraged.</p> + +<p><strong>Applications</strong>: In previous decades, software development was a carefully orchestrated and planned process. The release cycles were often measured in years, and upgrades were infrequent. Today, Web and mobile applications are developed in a much more iterative fashion. The release cycles are measured in days or weeks, and upgrades are a non-issue. (What version of Salesforce.com or Google Maps are you using?)</p> + +<p><strong>Data</strong>: In previous decades, data was measured in MBs or GBs, and it was highly structured and denormalized. Today's data is often measured in TBs or PBs, and it tends to be multi-structured â a combination of unstructured, semi-structured and structured. The data comes from many different sources, including a variety of applications, devices and services, and its structure changes much more frequently.</p> + +<h3 id="a-new-generation-of-datastores">A New Generation of Datastores</h3> + +<p>The relational database, which was invented in 1970, was not designed for these new processes and data volumes and structures. As a result, a new generation of datastores has emerged, including HDFS, NoSQL (HBase, MongoDB, etc.) and search (Elasticsearch, Solr). These systems are schema-free (also known as "dynamic schema"). Applications, as opposed to DBAs, control the data structure, enabling more agility and flexibility. For example, an application developer can independently evolve the data structure with each application release (which could be daily or weekly) without filing a ticket with IT and waiting for the schema of the databae to be modified.</p> + +<h2 id="the-need-for-a-new-query-engine">The Need for a New Query Engine</h2> + +<p>With data increasingly being stored in schema-free datastores (HDFS, HBase, MongoDB, etc.) and a variety of cloud services, users need a way to explore and analyze this data, and a way to visualize it with BI tools (reports, dashboards, etc.). In 2012 we decided to embark on a journey to create the world's next-generation SQL engine. We had several high-level requirements in mind:</p> + +<ul> +<li><strong>A schema-free data model.</strong> Schema-free datastores (HDFS, NoSQL, search) need a schema-free SQL engine. These datastores became popular for a reason, and we shouldn't expect organizations to sacrifice those advantages in order to enjoy SQL-based analytics and BI. Today's organizations need agility and flexibility to cope with the volume, variety and velocity associated with modern applications and data.<br></li> +<li><strong>A standalone query engine that supports multiple data sources.</strong> Most companies now use a variety of best-of-breed datastores and services to store data. This is true not just for large Global 2000 companies, but also for small startups. For example, it is not uncommon for a startup to have data in MySQL, MongoDB, HBase and HDFS, as well as a variety of online services. ETL was hard even 10 years ago when data was static and 100x smaller than it is today, and in today's era of Big Data it's often impractical or impossible to ingest all the data into a single system.</li> +<li><strong>Ease of use.</strong> The SQL engine can't be hard to setup and use. Analysts and developers should be able to download and use it without deploying any complex infrastructure such as Hadoop.</li> +<li><strong>Scalability and performance.</strong> The SQL engine must support interactive queries. It can't be batch-oriented like Hive. In addition, it must be able to scale linearly from a small laptop or virtual machine to a large cluster with hundreds or thousands of powerful servers.</li> +</ul> + +<p>With these requirements in mind, we decided to incubate a new project in 2012 in the Apache Software Foundation so that a community of vendors and developers could come together and develop the technology. (One little known fact is that the name "Drill" was actually suggested by Google engineers due to its inspiration from Google's Dremel execution engine.)</p> + +<p>After almost two years of research and development, we released Drill 0.4 in August, and continued with monthly releases since then.</p> + +<h2 id="what's-next">What's Next</h2> + +<p>Graduating to a top-level project is a significant milestone, but it's really just the beginning of the journey. In fact, we're currently wrapping up Drill 0.7, which includes hundreds of fixes and enhancements, and we expect to release that in the next couple weeks.</p> + +<p>Drill is currently being used by dozens of organizations, ranging from small startups to some of the largest Fortune 100s. These organizations are already gaining tremendous business value with Drill. As we march towards a 1.0 release early next year, these organizations are helping us shape the project and ensure that it meets the needs of a broad range of organizations as well as users (business analysts, technical analysts, data scientists and application developers). I would like to encourage you to join the ride today by <a href="http://drill.apache.org/download/";>downloading Drill</a> and <a href="mailto:[email protected]";>letting us know</a> what you think.</p> + +<p>Happy Drilling!<br> +Tomer Shiran</p> + + </article> + <div id="disqus_thread"></div> + <script type="text/javascript"> + /* * * CONFIGURATION VARIABLES: EDIT BEFORE PASTING INTO YOUR WEBPAGE * * */ + var disqus_shortname = 'drill'; // required: replace example with your forum shortname + + /* * * DON'T EDIT BELOW THIS LINE * * */ + (function() { + var dsq = document.createElement('script'); dsq.type = 'text/javascript'; dsq.async = true; + dsq.src = '//' + disqus_shortname + '.disqus.com/embed.js'; + (document.getElementsByTagName('head')[0] || document.getElementsByTagName('body')[0]).appendChild(dsq); + })(); + </script> + <noscript>Please enable JavaScript to view the <a href="http://disqus.com/?ref_noscript";>comments powered by Disqus.</a></noscript> + +</div> +<script type="text/javascript" src="//s7.addthis.com/js/300/addthis_widget.js#pubid=ra-548b2caa33765e8d" async="async"></script> + + </div> + <p class="push"></p> +<div id="footer" class="mw"> +<div class="wrapper"> +Copyright © 2012-2014 The Apache Software Foundation, licensed under the Apache License, Version 2.0.<br> +Apache and the Apache feather logo are trademarks of The Apache Software Foundation. Other names appearing on the site may be trademarks of their respective owners.<br/><br/> +</div> +</div> + + <script> +(function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){ +(i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o), +m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m) +})(window,document,'script','//www.google-analytics.com/analytics.js','ga'); + +ga('create', 'UA-53379651-1', 'auto'); +ga('send', 'pageview'); +</script> +<script type="text/javascript" src="//s7.addthis.com/js/300/addthis_widget.js#pubid=ra-548b2caa33765e8d" async="async"></script> +</body> +</html>
http://git-wip-us.apache.org/repos/asf/drill-site/blob/bbdefcb2/_site/blog/2014/12/09/running-sql-queries-on-amazon-s3/index.html ---------------------------------------------------------------------- diff --git a/_site/blog/2014/12/09/running-sql-queries-on-amazon-s3/index.html b/_site/blog/2014/12/09/running-sql-queries-on-amazon-s3/index.html new file mode 100644 index 0000000..b2c7df7 --- /dev/null +++ b/_site/blog/2014/12/09/running-sql-queries-on-amazon-s3/index.html @@ -0,0 +1,240 @@ +<!DOCTYPE html> +<html> + +<head> + +<meta charset="UTF-8"> +<meta name=viewport content="width=device-width, initial-scale=1"> +<meta name="robots" content="noindex"> + +<title>Running SQL Queries on Amazon S3 - Apache Drill</title> + +<link href="//maxcdn.bootstrapcdn.com/font-awesome/4.3.0/css/font-awesome.min.css" rel="stylesheet" type="text/css"/> +<link href='//fonts.googleapis.com/css?family=PT+Sans' rel='stylesheet' type='text/css'/> +<link href="/drill/css/site.css" rel="stylesheet" type="text/css"/> + +<link rel="shortcut icon" href="/drill/favicon.ico" type="image/x-icon"/> +<link rel="icon" href="/drill/favicon.ico" type="image/x-icon"/> + +<script src="//ajax.googleapis.com/ajax/libs/jquery/1.11.1/jquery.min.js" language="javascript" type="text/javascript"></script> +<script src="//cdnjs.cloudflare.com/ajax/libs/jquery-easing/1.3/jquery.easing.min.js" language="javascript" type="text/javascript"></script> +<script language="javascript" type="text/javascript" src="/drill/js/modernizr.custom.js"></script> +<script language="javascript" type="text/javascript" src="/drill/js/script.js"></script> +<script language="javascript" type="text/javascript" src="/drill/js/drill.js"></script> + +</head> + + +<body onResize="resized();"> + <div class="page-wrap"> + <div class="bui"></div> + +<div id="menu" class="mw"> +<ul> + <li class='toc-categories'> + <a class="expand-toc-icon" href="javascript:void(0);"><i class="fa fa-bars"></i></a> + </li> + <li class="logo"><a href="/drill/"></a></li> + <li class='expand-menu'> + <a href="javascript:void(0);"><span class='menu-text'>Menu</span><span class='expand-icon'><i class="fa fa-bars"></i></span></a> + </li> + <li class='clear-float'></li> + <li class="documentation-menu"> + <a href="/drill/docs/">Documentation</a> + <ul> + + <li><a href="/drill/docs/getting-started/">Getting Started</a></li> + + <li><a href="/drill/docs/architecture/">Architecture</a></li> + + <li><a href="/drill/docs/tutorials/">Tutorials</a></li> + + <li><a href="/drill/docs/install-drill/">Install Drill</a></li> + + <li><a href="/drill/docs/configure-drill/">Configure Drill</a></li> + + <li><a href="/drill/docs/connect-a-data-source/">Connect a Data Source</a></li> + + <li><a href="/drill/docs/odbc-jdbc-interfaces/">ODBC/JDBC Interfaces</a></li> + + <li><a href="/drill/docs/query-data/">Query Data</a></li> + + <li><a href="/drill/docs/performance-tuning/">Performance Tuning</a></li> + + <li><a href="/drill/docs/log-and-debug/">Log and Debug</a></li> + + <li><a href="/drill/docs/sql-reference/">SQL Reference</a></li> + + <li><a href="/drill/docs/data-sources-and-file-formats/">Data Sources and File Formats</a></li> + + <li><a href="/drill/docs/develop-custom-functions/">Develop Custom Functions</a></li> + + <li><a href="/drill/docs/troubleshooting/">Troubleshooting</a></li> + + <li><a href="/drill/docs/developer-information/">Developer Information</a></li> + + <li><a href="/drill/docs/release-notes/">Release Notes</a></li> + + <li><a href="/drill/docs/sample-datasets/">Sample Datasets</a></li> + + <li><a href="/drill/docs/project-bylaws/">Project Bylaws</a></li> + + </ul> + </li> + <li class='nav'> + <a href="/drill/community-resources/">Community</a> + <ul> + <li><a href="/drill/team/">Team</a></li> + <li><a href="/drill/mailinglists/">Mailing Lists</a></li> + <li><a href="/drill/community-resources/">Community Resources</a></li> + </ul> + </li> + <li class='nav'><a href="/drill/faq/">FAQ</a></li> + <li class='nav'><a href="/drill/blog/">Blog</a></li> + <li id="twitter-menu-item"><a href="https://twitter.com/apachedrill"; title="apachedrill on twitter" target="_blank"><img src="/drill/images/twitter_32_26_white.png" alt="twitter logo" align="center"></a> </li> + <li class='search-bar'> + <form id="drill-search-form"> + <input type="text" placeholder="Search Apache Drill" id="drill-search-term" /> + <button type="submit"> + <i class="fa fa-search"></i> + </button> + </form> + </li> + <li class="d"> + <a href="/drill/download/"> + <i class="fa fa-cloud-download"></i> Download + </a> + </li> +</ul> +</div> + + <link href="/drill/css/content.css" rel="stylesheet" type="text/css"> + +<div class="post int_text"> + <header class="post-header"> + <div class="int_title"> + <h1 class="post-title">Running SQL Queries on Amazon S3</h1> + </div> + <p class="post-meta"> + + + + <strong>Author:</strong> Nick Amato (Director, Technical Marketing, MapR Technologies)<br /> + +<strong>Date:</strong> Dec 9, 2014 +</p> + </header> + <div class="addthis_sharing_toolbox"></div> + + <article class="post-content"> + <p>The functionality and sheer usefulness of Drill is growing fast. If you're a user of some of the popular BI tools out there like Tableau or SAP Lumira, now is a good time to take a look at how Drill can make your life easier, especially if you're faced with the task of quickly getting a handle on large sets of unstructured data. With schema generated on the fly, you can save a lot of time and headaches by running SQL queries on the data where it rests without knowing much about columns or formats. There's even more good news: Drill also works with data stored in the cloud. With a few simple steps, you can configure the S3 storage plugin for Drill and be off to the races running queries. In this post we'll look at how to configure Drill to access data stored in an S3 bucket.</p> + +<p>If you're more of a visual person, you can skip this article entirely and <a href="https://www.youtube.com/watch?v=w8gZ2nn_ZUQ";>go straight to a video</a> I put together that walks through an end-to-end example with Tableau. This example is easily extended to other BI tools, as the steps are identical on the Drill side.</p> + +<p>At a high level, configuring Drill to access S3 bucket data is accomplished with the following steps on each node running a drillbit.</p> + +<ul> +<li>Download and install the <a href="http://www.jets3t.org/";>JetS3t</a> JAR files and enable them.</li> +<li>Add your S3 credentials in the relevant XML configuration file.</li> +<li>Configure and enable the S3 storage plugin through the Drill web interface.</li> +<li>Connect your BI tool of choice and query away.</li> +</ul> + +<p>Consult the <a href="https://cwiki.apache.org/confluence/display/DRILL/Architectural+Overview";>Architectural Overview</a> for a refresher on the architecture of Drill.</p> + +<h3 id="prerequisites">Prerequisites</h3> + +<p>These steps assume you have a <a href="https://cwiki.apache.org/confluence/display/DRILL/Apache+Drill+in+10+Minutes";>typical Drill cluster and ZooKeeper quorum</a> configured and running. To access data in S3, you will need an S3 bucket configured and have the required Amazon security credentials in your possession. An <a href="http://blogs.aws.amazon.com/security/post/Tx1R9KDN9ISZ0HF/Where-s-my-secret-access-key";>Amazon blog post</a> has more information on how to get these from your account.</p> + +<h3 id="configuration-steps">Configuration Steps</h3> + +<p>To connect Drill to S3, all of the drillbit nodes will need to access code in the JetS3t library developed by Amazon. As of this writing, 0.9.2 is the latest version but you might want to check <a href="https://jets3t.s3.amazonaws.com/toolkit/toolkit.html";>the main page</a> to see if anything has been updated. Be sure to get version 0.9.2 or later as earlier versions have a bug relating to reading Parquet data.</p> +<div class="highlight"><pre><code class="language-bash" data-lang="bash">wget http://bitbucket.org/jmurty/jets3t/downloads/jets3t-0.9.2.zip +cp jets3t-0.9.2/jars/jets3t-0.9.2.jar <span class="nv">$DRILL_HOME</span>/jars/3rdparty +</code></pre></div> +<p>Next, enable the plugin by editing the file:</p> +<div class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">$DRILL_HOME</span>/bin/hadoop_excludes.txt +</code></pre></div> +<p>and removing the line <code>jets3t</code>.</p> + +<p>Drill will need to know your S3 credentials in order to access data there. These credentials will need to be placed in the core-site.xml file for your installation. If you already have a core-site.xml file configured for your environment, add the following parameters to it, otherwise create the file from scratch. If you do end up creating it from scratch you will need to wrap these parameters with <code><configuration></code> and <code></configuration></code>.</p> +<div class="highlight"><pre><code class="language-xml" data-lang="xml"><span class="nt"><property></span> + <span class="nt"><name></span>fs.s3.awsAccessKeyId<span class="nt"></name></span> + <span class="nt"><value></span>ID<span class="nt"></value></span> +<span class="nt"></property></span> + +<span class="nt"><property></span> + <span class="nt"><name></span>fs.s3.awsSecretAccessKey<span class="nt"></name></span> + <span class="nt"><value></span>SECRET<span class="nt"></value></span> +<span class="nt"></property></span> + +<span class="nt"><property></span> + <span class="nt"><name></span>fs.s3n.awsAccessKeyId<span class="nt"></name></span> + <span class="nt"><value></span>ID<span class="nt"></value></span> +<span class="nt"></property></span> + +<span class="nt"><property></span> + <span class="nt"><name></span>fs.s3n.awsSecretAccessKey<span class="nt"></name></span> + <span class="nt"><value></span>SECRET<span class="nt"></value></span> +<span class="nt"></property></span> +</code></pre></div> +<p>The steps so far give Drill enough information to connect to the S3 service. Remember, you have to do this on all the nodes running drillbit.</p> + +<p>Next, let's go into the Drill web interface and enable the S3 storage plugin. In this case you only need to connect to <strong>one</strong> of the nodes because Drill's configuration is synchronized across the cluster. Complete the following steps:</p> + +<ol> +<li>Point your browser to <code>http://<host>:8047</code></li> +<li>Select the 'Storage' tab.</li> +<li>A good starting configuration for S3 can be entirely the same as the <code>dfs</code> plugin, except the connection parameter is changed to <code>s3://bucket</code>. So first select the <code>Update</code> button for <code>dfs</code>, then select the text area and copy it into the clipboard (on Windows, ctrl-A, ctrl-C works).</li> +<li>Press <code>Back</code>, then create a new plugin by typing the name into the <code>New Storage Plugin</code>, then press <code>Create</code>. You can choose any name, but a good convention is to use <code>s3-<bucketname></code> so you can easily identify it later.</li> +<li>In the configuration area, paste the configuration you just grabbed from 'dfs'. Change the line <code>connection: "file:///"</code> to <code>connection: "s3://<bucket>"</code>.</li> +<li>Click <code>Update</code>. You should see a message that indicates success.</li> +</ol> + +<p>At this point you can run queries on the data directly and you have a couple of options on how you want to access it. You can use Drill Explorer and create a custom view (based on an SQL query) that you can then access in Tableau or other BI tools, or just use Drill directly from within the tool.</p> + +<p>You may want to check out the <a href="http://www.youtube.com/watch?v=jNUsprJNQUg";>Tableau demo</a>.</p> + +<p>With just a few lines of configuration, you've just opened the vast world of data available in the Amazon cloud and reduced the amount of work you have to do in advance to access data stored there with SQL. There are even some <a href="https://aws.amazon.com/datasets";>public datasets</a> available directly on S3 that are great for experimentation.</p> + +<p>Happy Drilling!</p> + + </article> + <div id="disqus_thread"></div> + <script type="text/javascript"> + /* * * CONFIGURATION VARIABLES: EDIT BEFORE PASTING INTO YOUR WEBPAGE * * */ + var disqus_shortname = 'drill'; // required: replace example with your forum shortname + + /* * * DON'T EDIT BELOW THIS LINE * * */ + (function() { + var dsq = document.createElement('script'); dsq.type = 'text/javascript'; dsq.async = true; + dsq.src = '//' + disqus_shortname + '.disqus.com/embed.js'; + (document.getElementsByTagName('head')[0] || document.getElementsByTagName('body')[0]).appendChild(dsq); + })(); + </script> + <noscript>Please enable JavaScript to view the <a href="http://disqus.com/?ref_noscript";>comments powered by Disqus.</a></noscript> + +</div> +<script type="text/javascript" src="//s7.addthis.com/js/300/addthis_widget.js#pubid=ra-548b2caa33765e8d" async="async"></script> + + </div> + <p class="push"></p> +<div id="footer" class="mw"> +<div class="wrapper"> +Copyright © 2012-2014 The Apache Software Foundation, licensed under the Apache License, Version 2.0.<br> +Apache and the Apache feather logo are trademarks of The Apache Software Foundation. Other names appearing on the site may be trademarks of their respective owners.<br/><br/> +</div> +</div> + + <script> +(function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){ +(i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o), +m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m) +})(window,document,'script','//www.google-analytics.com/analytics.js','ga'); + +ga('create', 'UA-53379651-1', 'auto'); +ga('send', 'pageview'); +</script> +<script type="text/javascript" src="//s7.addthis.com/js/300/addthis_widget.js#pubid=ra-548b2caa33765e8d" async="async"></script> +</body> +</html> http://git-wip-us.apache.org/repos/asf/drill-site/blob/bbdefcb2/_site/blog/2014/12/11/apache-drill-qa-panelist-spotlight/index.html ---------------------------------------------------------------------- diff --git a/_site/blog/2014/12/11/apache-drill-qa-panelist-spotlight/index.html b/_site/blog/2014/12/11/apache-drill-qa-panelist-spotlight/index.html new file mode 100644 index 0000000..e95bce7 --- /dev/null +++ b/_site/blog/2014/12/11/apache-drill-qa-panelist-spotlight/index.html @@ -0,0 +1,214 @@ +<!DOCTYPE html> +<html> + +<head> + +<meta charset="UTF-8"> +<meta name=viewport content="width=device-width, initial-scale=1"> +<meta name="robots" content="noindex"> + +<title>Apache Drill Q&A Panelist Spotlight - Apache Drill</title> + +<link href="//maxcdn.bootstrapcdn.com/font-awesome/4.3.0/css/font-awesome.min.css" rel="stylesheet" type="text/css"/> +<link href='//fonts.googleapis.com/css?family=PT+Sans' rel='stylesheet' type='text/css'/> +<link href="/drill/css/site.css" rel="stylesheet" type="text/css"/> + +<link rel="shortcut icon" href="/drill/favicon.ico" type="image/x-icon"/> +<link rel="icon" href="/drill/favicon.ico" type="image/x-icon"/> + +<script src="//ajax.googleapis.com/ajax/libs/jquery/1.11.1/jquery.min.js" language="javascript" type="text/javascript"></script> +<script src="//cdnjs.cloudflare.com/ajax/libs/jquery-easing/1.3/jquery.easing.min.js" language="javascript" type="text/javascript"></script> +<script language="javascript" type="text/javascript" src="/drill/js/modernizr.custom.js"></script> +<script language="javascript" type="text/javascript" src="/drill/js/script.js"></script> +<script language="javascript" type="text/javascript" src="/drill/js/drill.js"></script> + +</head> + + +<body onResize="resized();"> + <div class="page-wrap"> + <div class="bui"></div> + +<div id="menu" class="mw"> +<ul> + <li class='toc-categories'> + <a class="expand-toc-icon" href="javascript:void(0);"><i class="fa fa-bars"></i></a> + </li> + <li class="logo"><a href="/drill/"></a></li> + <li class='expand-menu'> + <a href="javascript:void(0);"><span class='menu-text'>Menu</span><span class='expand-icon'><i class="fa fa-bars"></i></span></a> + </li> + <li class='clear-float'></li> + <li class="documentation-menu"> + <a href="/drill/docs/">Documentation</a> + <ul> + + <li><a href="/drill/docs/getting-started/">Getting Started</a></li> + + <li><a href="/drill/docs/architecture/">Architecture</a></li> + + <li><a href="/drill/docs/tutorials/">Tutorials</a></li> + + <li><a href="/drill/docs/install-drill/">Install Drill</a></li> + + <li><a href="/drill/docs/configure-drill/">Configure Drill</a></li> + + <li><a href="/drill/docs/connect-a-data-source/">Connect a Data Source</a></li> + + <li><a href="/drill/docs/odbc-jdbc-interfaces/">ODBC/JDBC Interfaces</a></li> + + <li><a href="/drill/docs/query-data/">Query Data</a></li> + + <li><a href="/drill/docs/performance-tuning/">Performance Tuning</a></li> + + <li><a href="/drill/docs/log-and-debug/">Log and Debug</a></li> + + <li><a href="/drill/docs/sql-reference/">SQL Reference</a></li> + + <li><a href="/drill/docs/data-sources-and-file-formats/">Data Sources and File Formats</a></li> + + <li><a href="/drill/docs/develop-custom-functions/">Develop Custom Functions</a></li> + + <li><a href="/drill/docs/troubleshooting/">Troubleshooting</a></li> + + <li><a href="/drill/docs/developer-information/">Developer Information</a></li> + + <li><a href="/drill/docs/release-notes/">Release Notes</a></li> + + <li><a href="/drill/docs/sample-datasets/">Sample Datasets</a></li> + + <li><a href="/drill/docs/project-bylaws/">Project Bylaws</a></li> + + </ul> + </li> + <li class='nav'> + <a href="/drill/community-resources/">Community</a> + <ul> + <li><a href="/drill/team/">Team</a></li> + <li><a href="/drill/mailinglists/">Mailing Lists</a></li> + <li><a href="/drill/community-resources/">Community Resources</a></li> + </ul> + </li> + <li class='nav'><a href="/drill/faq/">FAQ</a></li> + <li class='nav'><a href="/drill/blog/">Blog</a></li> + <li id="twitter-menu-item"><a href="https://twitter.com/apachedrill"; title="apachedrill on twitter" target="_blank"><img src="/drill/images/twitter_32_26_white.png" alt="twitter logo" align="center"></a> </li> + <li class='search-bar'> + <form id="drill-search-form"> + <input type="text" placeholder="Search Apache Drill" id="drill-search-term" /> + <button type="submit"> + <i class="fa fa-search"></i> + </button> + </form> + </li> + <li class="d"> + <a href="/drill/download/"> + <i class="fa fa-cloud-download"></i> Download + </a> + </li> +</ul> +</div> + + <link href="/drill/css/content.css" rel="stylesheet" type="text/css"> + +<div class="post int_text"> + <header class="post-header"> + <div class="int_title"> + <h1 class="post-title">Apache Drill Q&A Panelist Spotlight</h1> + </div> + <p class="post-meta"> + + + + <strong>Author:</strong> Tomer Shiran (Founder, PMC Member and Committer, Apache Drill)<br /> + + +</p> + </header> + <div class="addthis_sharing_toolbox"></div> + + <article class="post-content"> + <script type="text/javascript" src="//addthisevent.com/libs/1.5.8/ate.min.js"></script> + +<p><a href="/drill/blog/2014/12/11/apache-drill-qa-panelist-spotlight/" title="Add to Calendar" class="addthisevent"> + Add to Calendar + <span class="_start">12-17-2014 11:30:00</span> + <span class="_end">12-17-2014 12:30:00</span> + <span class="_zonecode">6</span> + <span class="_summary">Apache Drill - Live Q&A on Twitter</span> + <span class="_description">Join us on Twitter for a one-hour, live SQL-on-Hadoop Q&A. Use the <strong>hashtag #DrillQA</strong> so the panelists can engage with your questions and comments. Apache Drill committers Tomer Shiran, Jacques Nadeau, and Ted Dunning, as well as Tableau Product Manager Jeff Feng and Data Scientist Dr. Kirk Borne will be on hand to answer your questions.</span> + <span class="_location">Twitter: #DrillQA</span> + <span class="_organizer">Tomer Shiran</span> + <span class="_organizer_email">tshiran\@apache.org</span> + <span class="_all_day_event">false</span> + <span class="_date_format">MM-DD-YYYY</span> +</a></p> + +<p>Hadoop has always been a powerful platform, but it is even more so with the release of Apache Drill, a valuable technology for self-service data exploration on big data. For BI users, this is really exciting news. </p> + +<p>With Apache Drill, you can immediately query complex data in native formats, such as schema-less data, nested data, and data with rapidly-evolving schemas. And with analytic tools likes Tableau, you can easily create queries, build dashboards and explore data. </p> + +<p>Want to learn how to leverage Apache Drill in order to get better analytical insights? </p> + +<p><strong>Join us on Twitter</strong> for a one-hour, live SQL-on-Hadoop Q&A, next <strong>Wednesday, December 17th starting at 11:30am PST, 2:30pm EST</strong>. Use the <strong>hashtag #DrillQA</strong> so the panelists can engage with your questions and comments.</p> + +<p>Apache Drill committers Tomer Shiran, Jacques Nadeau, and Ted Dunning, as well as Tableau Product Manager Jeff Feng and Data Scientist Dr. Kirk Borne will be on hand to answer your questions.</p> + +<h4 id="tomer-shiran,-apache-drill-founder-(@tshiran)">Tomer Shiran, Apache Drill Founder (@tshiran)</h4> + +<p>Tomer Shiran is the founder of Apache Drill, and a PMC member and committer on the project. He is VP Product Management at MapR, responsible for product strategy, roadmap and new feature development. Prior to MapR, Tomer held numerous product management and engineering roles at Microsoft, most recently as the product manager for Microsoft Internet Security & Acceleration Server (now Microsoft Forefront). He is the founder of two websites that have served tens of millions of users, and received coverage in prestigious publications such as The New York Times, USA Today and The Times of London. Tomer is also the author of a 900-page programming book. He holds an MS in Computer Engineering from Carnegie Mellon University and a BS in Computer Science from Technion - Israel Institute of Technology.</p> + +<h4 id="jeff-feng,-product-manager-tableau-software-(@jtfeng)">Jeff Feng, Product Manager Tableau Software (@jtfeng)</h4> + +<p>Jeff Feng is a Product Manager at Tableau and leads their Big Data product roadmap & strategic vision. In his role, he focuses on joint technology integration and partnership efforts with a number of Hadoop, NoSQL and web application partners in helping users see and understand their data.</p> + +<h4 id="ted-dunning,-apache-drill-comitter-(@ted_dunning)">Ted Dunning, Apache Drill Comitter (@Ted_Dunning)</h4> + +<p>Ted Dunning is Chief Applications Architect at MapR Technologies and committer and PMC member of the Apache Mahout, Apache ZooKeeper, and Apache Drill projects and mentor for Apache Storm. He contributed to Mahout clustering, classification and matrix decomposition algorithms and helped expand the new version of Mahout Math library. Ted was the chief architect behind the MusicMatch (now Yahoo Music) and Veoh recommendation systems, he built fraud detection systems for ID Analytics (LifeLock) and he has issued 24 patents to date. Ted has a PhD in computing science from University of Sheffield. When heâs not doing data science, he plays guitar and mandolin.</p> + +<h4 id="jacques-nadeau,-vice-president,-apache-drill-(@intjesus)">Jacques Nadeau, Vice President, Apache Drill (@intjesus)</h4> + +<p>Jacques Nadeau leads Apache Drill development efforts at MapR Technologies. He is an industry veteran with over 15 years of big data and analytics experience. Most recently, he was cofounder and CTO of search engine startup YapMap. Before that, he was director of new product engineering with Quigo (contextual advertising, acquired by AOL in 2007). He also built the Avenue A | Razorfish analytics data warehousing system and associated services practice (acquired by Microsoft).</p> + +<h4 id="dr.-kirk-borne,-george-mason-university-(@kirkdborne)">Dr. Kirk Borne, George Mason University (@KirkDBorne)</h4> + +<p>Dr. Kirk Borne is a Transdisciplinary Data Scientist and an Astrophysicist. He is Professor of Astrophysics and Computational Science in the George Mason University School of Physics, Astronomy, and Computational Sciences. He has been at Mason since 2003, where he teaches and advises students in the graduate and undergraduate Computational Science, Informatics, and Data Science programs. Previously, he spent nearly 20 years in positions supporting NASA projects, including an assignment as NASA's Data Archive Project Scientist for the Hubble Space Telescope, and as Project Manager in NASA's Space Science Data Operations Office. He has extensive experience in big data and data science, including expertise in scientific data mining and data systems. He has published over 200 articles (research papers, conference papers, and book chapters), and given over 200 invited talks at conferences and universities worldwide.</p> + + </article> + <div id="disqus_thread"></div> + <script type="text/javascript"> + /* * * CONFIGURATION VARIABLES: EDIT BEFORE PASTING INTO YOUR WEBPAGE * * */ + var disqus_shortname = 'drill'; // required: replace example with your forum shortname + + /* * * DON'T EDIT BELOW THIS LINE * * */ + (function() { + var dsq = document.createElement('script'); dsq.type = 'text/javascript'; dsq.async = true; + dsq.src = '//' + disqus_shortname + '.disqus.com/embed.js'; + (document.getElementsByTagName('head')[0] || document.getElementsByTagName('body')[0]).appendChild(dsq); + })(); + </script> + <noscript>Please enable JavaScript to view the <a href="http://disqus.com/?ref_noscript";>comments powered by Disqus.</a></noscript> + +</div> +<script type="text/javascript" src="//s7.addthis.com/js/300/addthis_widget.js#pubid=ra-548b2caa33765e8d" async="async"></script> + + </div> + <p class="push"></p> +<div id="footer" class="mw"> +<div class="wrapper"> +Copyright © 2012-2014 The Apache Software Foundation, licensed under the Apache License, Version 2.0.<br> +Apache and the Apache feather logo are trademarks of The Apache Software Foundation. Other names appearing on the site may be trademarks of their respective owners.<br/><br/> +</div> +</div> + + <script> +(function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){ +(i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o), +m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m) +})(window,document,'script','//www.google-analytics.com/analytics.js','ga'); + +ga('create', 'UA-53379651-1', 'auto'); +ga('send', 'pageview'); +</script> +<script type="text/javascript" src="//s7.addthis.com/js/300/addthis_widget.js#pubid=ra-548b2caa33765e8d" async="async"></script> +</body> +</html> http://git-wip-us.apache.org/repos/asf/drill-site/blob/bbdefcb2/_site/blog/2014/12/16/whats-coming-in-2015/index.html ---------------------------------------------------------------------- diff --git a/_site/blog/2014/12/16/whats-coming-in-2015/index.html b/_site/blog/2014/12/16/whats-coming-in-2015/index.html new file mode 100644 index 0000000..ab99bca --- /dev/null +++ b/_site/blog/2014/12/16/whats-coming-in-2015/index.html @@ -0,0 +1,293 @@ +<!DOCTYPE html> +<html> + +<head> + +<meta charset="UTF-8"> +<meta name=viewport content="width=device-width, initial-scale=1"> +<meta name="robots" content="noindex"> + +<title>What's Coming in 2015? - Apache Drill</title> + +<link href="//maxcdn.bootstrapcdn.com/font-awesome/4.3.0/css/font-awesome.min.css" rel="stylesheet" type="text/css"/> +<link href='//fonts.googleapis.com/css?family=PT+Sans' rel='stylesheet' type='text/css'/> +<link href="/drill/css/site.css" rel="stylesheet" type="text/css"/> + +<link rel="shortcut icon" href="/drill/favicon.ico" type="image/x-icon"/> +<link rel="icon" href="/drill/favicon.ico" type="image/x-icon"/> + +<script src="//ajax.googleapis.com/ajax/libs/jquery/1.11.1/jquery.min.js" language="javascript" type="text/javascript"></script> +<script src="//cdnjs.cloudflare.com/ajax/libs/jquery-easing/1.3/jquery.easing.min.js" language="javascript" type="text/javascript"></script> +<script language="javascript" type="text/javascript" src="/drill/js/modernizr.custom.js"></script> +<script language="javascript" type="text/javascript" src="/drill/js/script.js"></script> +<script language="javascript" type="text/javascript" src="/drill/js/drill.js"></script> + +</head> + + +<body onResize="resized();"> + <div class="page-wrap"> + <div class="bui"></div> + +<div id="menu" class="mw"> +<ul> + <li class='toc-categories'> + <a class="expand-toc-icon" href="javascript:void(0);"><i class="fa fa-bars"></i></a> + </li> + <li class="logo"><a href="/drill/"></a></li> + <li class='expand-menu'> + <a href="javascript:void(0);"><span class='menu-text'>Menu</span><span class='expand-icon'><i class="fa fa-bars"></i></span></a> + </li> + <li class='clear-float'></li> + <li class="documentation-menu"> + <a href="/drill/docs/">Documentation</a> + <ul> + + <li><a href="/drill/docs/getting-started/">Getting Started</a></li> + + <li><a href="/drill/docs/architecture/">Architecture</a></li> + + <li><a href="/drill/docs/tutorials/">Tutorials</a></li> + + <li><a href="/drill/docs/install-drill/">Install Drill</a></li> + + <li><a href="/drill/docs/configure-drill/">Configure Drill</a></li> + + <li><a href="/drill/docs/connect-a-data-source/">Connect a Data Source</a></li> + + <li><a href="/drill/docs/odbc-jdbc-interfaces/">ODBC/JDBC Interfaces</a></li> + + <li><a href="/drill/docs/query-data/">Query Data</a></li> + + <li><a href="/drill/docs/performance-tuning/">Performance Tuning</a></li> + + <li><a href="/drill/docs/log-and-debug/">Log and Debug</a></li> + + <li><a href="/drill/docs/sql-reference/">SQL Reference</a></li> + + <li><a href="/drill/docs/data-sources-and-file-formats/">Data Sources and File Formats</a></li> + + <li><a href="/drill/docs/develop-custom-functions/">Develop Custom Functions</a></li> + + <li><a href="/drill/docs/troubleshooting/">Troubleshooting</a></li> + + <li><a href="/drill/docs/developer-information/">Developer Information</a></li> + + <li><a href="/drill/docs/release-notes/">Release Notes</a></li> + + <li><a href="/drill/docs/sample-datasets/">Sample Datasets</a></li> + + <li><a href="/drill/docs/project-bylaws/">Project Bylaws</a></li> + + </ul> + </li> + <li class='nav'> + <a href="/drill/community-resources/">Community</a> + <ul> + <li><a href="/drill/team/">Team</a></li> + <li><a href="/drill/mailinglists/">Mailing Lists</a></li> + <li><a href="/drill/community-resources/">Community Resources</a></li> + </ul> + </li> + <li class='nav'><a href="/drill/faq/">FAQ</a></li> + <li class='nav'><a href="/drill/blog/">Blog</a></li> + <li id="twitter-menu-item"><a href="https://twitter.com/apachedrill"; title="apachedrill on twitter" target="_blank"><img src="/drill/images/twitter_32_26_white.png" alt="twitter logo" align="center"></a> </li> + <li class='search-bar'> + <form id="drill-search-form"> + <input type="text" placeholder="Search Apache Drill" id="drill-search-term" /> + <button type="submit"> + <i class="fa fa-search"></i> + </button> + </form> + </li> + <li class="d"> + <a href="/drill/download/"> + <i class="fa fa-cloud-download"></i> Download + </a> + </li> +</ul> +</div> + + <link href="/drill/css/content.css" rel="stylesheet" type="text/css"> + +<div class="post int_text"> + <header class="post-header"> + <div class="int_title"> + <h1 class="post-title">What's Coming in 2015?</h1> + </div> + <p class="post-meta"> + + + + <strong>Author:</strong> Tomer Shiran (Founder, PMC Member and Committer, Apache Drill)<br /> + +<strong>Date:</strong> Dec 16, 2014 +</p> + </header> + <div class="addthis_sharing_toolbox"></div> + + <article class="post-content"> + <p>2014 was an exciting year for the Drill community. In August we made Drill available for downloads, and last week the Apache Software Foundation promoted Drill to a top-level project. Many of you have asked me what's coming next, so I decided to sit down and outline some of the interesting initiatives that the Drill community is currently working on:</p> + +<ul> +<li>Flexible Access Control</li> +<li>JSON in Any Shape or Form</li> +<li>Advanced SQL</li> +<li>New Data Sources</li> +<li>Drill/Spark Integration</li> +<li>Operational Enhancements: Speed, Scalability and Workload Management</li> +</ul> + +<p>This is by no means intended to be an exhaustive list of everything that will be added to Drill in 2015. With Drill's rapidly expanding community, I anticipate that you'll see a whole lot more.</p> + +<h2 id="flexible-access-control">Flexible Access Control</h2> + +<p>Many organizations are now interested in providing Drill as a service to their users, supporting many users, groups and organizations with a single cluster. To do so, they need to be able to control who can access what data. Today's volume and variety of data requires a new approach to access control. For example, it is becoming impractical for organizations to manage a standalone, centralized repository of permissions for every column/row of every table. Drill's virtual datasets (views) provide a more scalable solution to access control:</p> + +<ul> +<li>The user creates a virtual dataset (<code>CREATE VIEW vd AS SELECT ...</code>), selecting the data to be exposed/shared. The virtual dataset is defined as a SQL statement. For example, a virtual dataset may represent only the records that were created in the last 30 days and don't have the <code>restricted</code> flag. It could even mask some columns. Drill's virtual datasets (just the SQL statement) are stored as files in the file system, so users can leverage file system permissions to control who can access the virtual dataset, without granting access to the source data.</li> +<li>A virtual dataset is owned by a specific user and can only "select" data that the owner has access to. The data sources (HDFS, HBase, MongoDB, etc.) are responsible for access control decisions. Users and administrators do not need to define separate permissions inside Drill or utilize yet another centralized permission repository, such as Sentry and Ranger.</li> +</ul> + +<h2 id="json-in-any-shape-or-form">JSON in Any Shape or Form</h2> + +<p>When data is <strong>Big</strong> (as in Big Data), it is painful to copy and transform it. Users should be able to explore the raw data without (or at least prior to) transforming it into another format. Drill is designed to enable in-situ analytics. Just point it at a file or directory and run the queries.</p> + +<p>JSON has emerged as the most common self-describing format, and Drill is able to query JSON files out of the box. Drill currently assumes that the JSON documents (or records) are stored sequentially in a file:</p> +<div class="highlight"><pre><code class="language-json" data-lang="json"><span class="p">{</span> <span class="nt">"name"</span><span class="p">:</span> <span class="s2">"Lee"</span><span class="p">,</span> <span class="nt">"yelping_since"</span><span class="p">:</span> <span class="s2">"2012-02"</span> <span class="p">}</span> +<span class="p">{</span> <span class="nt">"name"</span><span class="p">:</span> <span class="s2">"Matthew"</span><span class="p">,</span> <span class="nt">"yelping_since"</span><span class="p">:</span> <span class="s2">"2011-12"</span> <span class="p">}</span> +<span class="p">{</span> <span class="nt">"name"</span><span class="p">:</span> <span class="s2">"Jasmine"</span><span class="p">,</span> <span class="nt">"yelping_since"</span><span class="p">:</span> <span class="s2">"2010-09"</span> <span class="p">}</span> +</code></pre></div> +<p>However, many JSON-based datasets, ranging from <a href="http://data.gov";>data.gov</a> (government) datasets to Twitter API responses, are not organized as simple sequences of JSON documents. In some cases the actual records are listed as elements of an internal array inside a single JSON document. For example, consider the following file, which technically consists of a single JSON document, but really contains three records (under the <code>data.records</code> field):</p> +<div class="highlight"><pre><code class="language-json" data-lang="json"><span class="p">{</span> + <span class="nt">"metadata"</span><span class="p">:</span> <span class="err">...</span><span class="p">,</span> + <span class="nt">"data"</span><span class="p">:</span> <span class="p">{</span> + <span class="nt">"records"</span><span class="p">:</span> <span class="p">[</span> + <span class="p">{</span> <span class="nt">"name"</span><span class="p">:</span> <span class="s2">"Lee"</span><span class="p">,</span> <span class="nt">"yelping_since"</span><span class="p">:</span> <span class="s2">"2012-02"</span> <span class="p">},</span> + <span class="p">{</span> <span class="nt">"name"</span><span class="p">:</span> <span class="s2">"Matthew"</span><span class="p">,</span> <span class="nt">"yelping_since"</span><span class="p">:</span> <span class="s2">"2011-12"</span> <span class="p">},</span> + <span class="p">{</span> <span class="nt">"name"</span><span class="p">:</span> <span class="s2">"Jasmine"</span><span class="p">,</span> <span class="nt">"yelping_since"</span><span class="p">:</span> <span class="s2">"2010-09"</span> <span class="p">}</span> + <span class="p">]</span> + <span class="p">}</span> +<span class="p">}</span> +</code></pre></div> +<p>The <code>FLATTEN</code> function in Drill 0.7+ takes an array and converts each item into a top-level record:</p> +<div class="highlight"><pre><code class="language-sql" data-lang="sql"><span class="k">SELECT</span> <span class="n">FLATTEN</span><span class="p">(</span><span class="k">data</span><span class="p">.</span><span class="n">records</span><span class="p">)</span> <span class="k">FROM</span> <span class="n">dfs</span><span class="p">.</span><span class="n">tmp</span><span class="p">.</span><span class="o">`</span><span class="n">foo</span><span class="p">.</span><span class="n">json</span><span class="o">`</span><span class="p">;</span> +</code></pre></div> +<p>You can use this as an inner query (or inside a view):</p> +<div class="highlight"><pre><code class="language-sql" data-lang="sql"><span class="o">></span> <span class="k">SELECT</span> <span class="n">t</span><span class="p">.</span><span class="n">record</span><span class="p">.</span><span class="n">name</span> <span class="k">AS</span> <span class="n">name</span> + <span class="k">FROM</span> <span class="p">(</span><span class="k">SELECT</span> <span class="n">FLATTEN</span><span class="p">(</span><span class="k">data</span><span class="p">.</span><span class="n">records</span><span class="p">)</span> <span class="k">AS</span> <span class="n">record</span> <span class="k">FROM</span> <span class="n">dfs</span><span class="p">.</span><span class="n">tmp</span><span class="p">.</span><span class="o">`</span><span class="n">test</span><span class="o">/</span><span class="n">foo</span><span class="p">.</span><span class="n">json</span><span class="o">`</span><span class="p">)</span> <span class="n">t</span><span class="p">;</span> +<span class="o">+</span><span class="c1">------------+</span> +<span class="o">|</span> <span class="n">name</span> <span class="o">|</span> +<span class="o">+</span><span class="c1">------------+</span> +<span class="o">|</span> <span class="n">Lee</span> <span class="o">|</span> +<span class="o">|</span> <span class="n">Matthew</span> <span class="o">|</span> +<span class="o">|</span> <span class="n">Jasmine</span> <span class="o">|</span> +<span class="o">+</span><span class="c1">------------+</span> +</code></pre></div> +<p>While this works today, the dataset is technically a single JSON document, so Drill ends up reading the entire dataset into memory. We're developing a FLATTEN-pushdown mechanism that will enable the JSON reader to emit the individual records into the downstream operators, thereby making this work with datasets of arbitrary size. Once that's implemented, users will be able to explore any JSON-based dataset in-situ (ie, without having to transform it).</p> + +<h2 id="full-sql">Full SQL</h2> + +<p>Unlike the majority of SQL engines for Hadoop and NoSQL databases, which support SQL-like languages (HiveQL, CQL, etc.), Drill is designed from the ground up to be compliant with ANSI SQL. We simply started with a real SQL parser (Apache Calcite, previously known as Optiq). We're currently implementing the remaining SQL constructs, and plan to support the full TPC-DS suite (with no query modifications) in 2015. Full SQL support makes BI tools work better, and enables users who are proficient with SQL to leverage their existing knowledge and skills.</p> + +<h2 id="new-data-sources">New Data Sources</h2> + +<p>Drill is a standalone, distributed SQL engine. It has a pluggable architecture that allows it to support multiple data sources. Drill 0.6 includes storage plugins for:</p> + +<ul> +<li><a href="https://hadoop.apache.org/docs/current/api/org/apache/hadoop/fs/FileSystem.html";>Hadoop File System</a> implementations (local file system, HDFS, MapR-FS, Amazon S3, etc.)</li> +<li>HBase and MapR-DB</li> +<li>MongoDB</li> +<li>Hive Metastore (query any dataset that is registered in Hive Metastore)</li> +</ul> + +<p>A single query can join data from different systems. For example, a query can join user profiles in MongoDB with log files in Hadoop, or datasets in multiple Hadoop clusters.</p> + +<p>I'm eager to see what storage plugins the community develops over the next 12 months. In the last few weeks alone, developers in the community have expressed their desire (on the <a href="mailto:[email protected]";>public list</a>) to develop additional storage plugins for the following data sources:</p> + +<ul> +<li>Cassandra</li> +<li>Solr</li> +<li>JDBC (any RDBMS, including Oracle, MySQL, PostgreSQL and SQL Server)</li> +</ul> + +<p>If you're interested in implementing a new storage plugin, I would encourage you to reach out to the Drill developer community on <a href="mailto:[email protected]";>[email protected]</a>. I'm looking forward to publishing an example of a single-query join across 10 data sources.</p> + +<h2 id="drill/spark-integration">Drill/Spark Integration</h2> + +<p>We're seeing growing interest in Spark as an execution engine for data pipelines, providing an alternative to MapReduce. The Drill community is working on integrating Drill and Spark to address a few new use cases:</p> + +<ul> +<li><p>Use a Drill query (or view) as the input to Spark. Drill is a powerful engine for extracting and pre-processing data from various data sources, thereby reducing development time and effort. Here's an example:</p> +<div class="highlight"><pre><code class="language-scala" data-lang="scala"><span class="k">val</span> <span class="n">sc</span> <span class="k">=</span> <span class="k">new</span> <span class="nc">SparkContext</span><span class="o">(</span><span class="n">conf</span><span class="o">)</span> +<span class="k">val</span> <span class="n">result</span> <span class="k">=</span> <span class="n">sc</span><span class="o">.</span><span class="n">drillRDD</span><span class="o">(</span><span class="s">"SELECT * FROM dfs.root.`path/to/logs` l, mongo.mydb.users u WHERE l.user_id = u.id GROUP BY ..."</span><span class="o">)</span> +<span class="k">val</span> <span class="n">formatted</span> <span class="k">=</span> <span class="n">result</span><span class="o">.</span><span class="n">map</span> <span class="o">{</span> <span class="n">r</span> <span class="k">=></span> + <span class="k">val</span> <span class="o">(</span><span class="n">first</span><span class="o">,</span> <span class="n">last</span><span class="o">,</span> <span class="n">visits</span><span class="o">)</span> <span class="k">=</span> <span class="o">(</span><span class="n">r</span><span class="o">.</span><span class="n">name</span><span class="o">.</span><span class="n">first</span><span class="o">,</span> <span class="n">r</span><span class="o">.</span><span class="n">name</span><span class="o">.</span><span class="n">last</span><span class="o">,</span> <span class="n">r</span><span class="o">.</span><span class="n">visits</span><span class="o">)</span> + <span class="n">s</span><span class="s">"$first $last $visits"</span> +<span class="o">}</span> +</code></pre></div></li> +<li><p>Use Drill to query Spark RDDs. Analysts will be able to use BI tools like MicroStrategy, Spotfire and Tableau to query in-memory data in Spark. In addition, Spark developers will be able to embed Drill execution in a Spark data pipeline, thereby enjoying the power of Drill's schema-free, columnar execution engine.</p></li> +</ul> + +<h2 id="operational-enhancements">Operational Enhancements</h2> + +<p>As we continue with our monthly releases and march towards the 1.0 release early next year, we're focused on improving Drill's speed and scalability. We'll also enhance Drill's multi-tenancy with more advanced workload management.</p> + +<ul> +<li><strong>Speed</strong>: Drill is already extremely fast, and we're going to make it even faster over the next few months. With that said, we think that improving user productivity and time-to-insight is as important as shaving a few milliseconds off a query's runtime.</li> +<li><strong>Scalability</strong>: To date we've focused mainly on clusters of up to a couple hundred nodes. We're currently working to support clusters with thousands of nodes. We're also improving concurrency to better support deployments in which hundreds of analysts or developers are running queries at the same time.</li> +<li><strong>Workload management</strong>: A single cluster is often shared among many users and groups, and everyone expects answers in real-time. Workload management prioritizes the allocation of resources to ensure that the most important workloads get done first so that business demands can be met. Administrators need to be able to assign priorities and quotas at a fine granularity. We're working on enhancing Drill's workload management to provide these capabilities while providing tight integration with YARN and Mesos.</li> +</ul> + +<h2 id="we-would-love-to-hear-from-you!">We Would Love to Hear From You!</h2> + +<p>Are there other features you would like to see in Drill? We would love to hear from you:</p> + +<ul> +<li>Drill users: <a href="mailto:[email protected]";>[email protected]</a></li> +<li>Drill developers: <a href="mailto:[email protected]";>[email protected]</a></li> +<li>Me: <a href="mailto:[email protected]";>[email protected]</a></li> +</ul> + +<p>Happy Drilling!<br> +Tomer Shiran</p> + + </article> + <div id="disqus_thread"></div> + <script type="text/javascript"> + /* * * CONFIGURATION VARIABLES: EDIT BEFORE PASTING INTO YOUR WEBPAGE * * */ + var disqus_shortname = 'drill'; // required: replace example with your forum shortname + + /* * * DON'T EDIT BELOW THIS LINE * * */ + (function() { + var dsq = document.createElement('script'); dsq.type = 'text/javascript'; dsq.async = true; + dsq.src = '//' + disqus_shortname + '.disqus.com/embed.js'; + (document.getElementsByTagName('head')[0] || document.getElementsByTagName('body')[0]).appendChild(dsq); + })(); + </script> + <noscript>Please enable JavaScript to view the <a href="http://disqus.com/?ref_noscript";>comments powered by Disqus.</a></noscript> + +</div> +<script type="text/javascript" src="//s7.addthis.com/js/300/addthis_widget.js#pubid=ra-548b2caa33765e8d" async="async"></script> + + </div> + <p class="push"></p> +<div id="footer" class="mw"> +<div class="wrapper"> +Copyright © 2012-2014 The Apache Software Foundation, licensed under the Apache License, Version 2.0.<br> +Apache and the Apache feather logo are trademarks of The Apache Software Foundation. Other names appearing on the site may be trademarks of their respective owners.<br/><br/> +</div> +</div> + + <script> +(function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){ +(i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o), +m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m) +})(window,document,'script','//www.google-analytics.com/analytics.js','ga'); + +ga('create', 'UA-53379651-1', 'auto'); +ga('send', 'pageview'); +</script> +<script type="text/javascript" src="//s7.addthis.com/js/300/addthis_widget.js#pubid=ra-548b2caa33765e8d" async="async"></script> +</body> +</html> http://git-wip-us.apache.org/repos/asf/drill-site/blob/bbdefcb2/_site/blog/2014/12/23/drill-0.7-released/index.html ---------------------------------------------------------------------- diff --git a/_site/blog/2014/12/23/drill-0.7-released/index.html b/_site/blog/2014/12/23/drill-0.7-released/index.html new file mode 100644 index 0000000..1f69402 --- /dev/null +++ b/_site/blog/2014/12/23/drill-0.7-released/index.html @@ -0,0 +1,186 @@ +<!DOCTYPE html> +<html> + +<head> + +<meta charset="UTF-8"> +<meta name=viewport content="width=device-width, initial-scale=1"> +<meta name="robots" content="noindex"> + +<title>Drill 0.7 Released - Apache Drill</title> + +<link href="//maxcdn.bootstrapcdn.com/font-awesome/4.3.0/css/font-awesome.min.css" rel="stylesheet" type="text/css"/> +<link href='//fonts.googleapis.com/css?family=PT+Sans' rel='stylesheet' type='text/css'/> +<link href="/drill/css/site.css" rel="stylesheet" type="text/css"/> + +<link rel="shortcut icon" href="/drill/favicon.ico" type="image/x-icon"/> +<link rel="icon" href="/drill/favicon.ico" type="image/x-icon"/> + +<script src="//ajax.googleapis.com/ajax/libs/jquery/1.11.1/jquery.min.js" language="javascript" type="text/javascript"></script> +<script src="//cdnjs.cloudflare.com/ajax/libs/jquery-easing/1.3/jquery.easing.min.js" language="javascript" type="text/javascript"></script> +<script language="javascript" type="text/javascript" src="/drill/js/modernizr.custom.js"></script> +<script language="javascript" type="text/javascript" src="/drill/js/script.js"></script> +<script language="javascript" type="text/javascript" src="/drill/js/drill.js"></script> + +</head> + + +<body onResize="resized();"> + <div class="page-wrap"> + <div class="bui"></div> + +<div id="menu" class="mw"> +<ul> + <li class='toc-categories'> + <a class="expand-toc-icon" href="javascript:void(0);"><i class="fa fa-bars"></i></a> + </li> + <li class="logo"><a href="/drill/"></a></li> + <li class='expand-menu'> + <a href="javascript:void(0);"><span class='menu-text'>Menu</span><span class='expand-icon'><i class="fa fa-bars"></i></span></a> + </li> + <li class='clear-float'></li> + <li class="documentation-menu"> + <a href="/drill/docs/">Documentation</a> + <ul> + + <li><a href="/drill/docs/getting-started/">Getting Started</a></li> + + <li><a href="/drill/docs/architecture/">Architecture</a></li> + + <li><a href="/drill/docs/tutorials/">Tutorials</a></li> + + <li><a href="/drill/docs/install-drill/">Install Drill</a></li> + + <li><a href="/drill/docs/configure-drill/">Configure Drill</a></li> + + <li><a href="/drill/docs/connect-a-data-source/">Connect a Data Source</a></li> + + <li><a href="/drill/docs/odbc-jdbc-interfaces/">ODBC/JDBC Interfaces</a></li> + + <li><a href="/drill/docs/query-data/">Query Data</a></li> + + <li><a href="/drill/docs/performance-tuning/">Performance Tuning</a></li> + + <li><a href="/drill/docs/log-and-debug/">Log and Debug</a></li> + + <li><a href="/drill/docs/sql-reference/">SQL Reference</a></li> + + <li><a href="/drill/docs/data-sources-and-file-formats/">Data Sources and File Formats</a></li> + + <li><a href="/drill/docs/develop-custom-functions/">Develop Custom Functions</a></li> + + <li><a href="/drill/docs/troubleshooting/">Troubleshooting</a></li> + + <li><a href="/drill/docs/developer-information/">Developer Information</a></li> + + <li><a href="/drill/docs/release-notes/">Release Notes</a></li> + + <li><a href="/drill/docs/sample-datasets/">Sample Datasets</a></li> + + <li><a href="/drill/docs/project-bylaws/">Project Bylaws</a></li> + + </ul> + </li> + <li class='nav'> + <a href="/drill/community-resources/">Community</a> + <ul> + <li><a href="/drill/team/">Team</a></li> + <li><a href="/drill/mailinglists/">Mailing Lists</a></li> + <li><a href="/drill/community-resources/">Community Resources</a></li> + </ul> + </li> + <li class='nav'><a href="/drill/faq/">FAQ</a></li> + <li class='nav'><a href="/drill/blog/">Blog</a></li> + <li id="twitter-menu-item"><a href="https://twitter.com/apachedrill"; title="apachedrill on twitter" target="_blank"><img src="/drill/images/twitter_32_26_white.png" alt="twitter logo" align="center"></a> </li> + <li class='search-bar'> + <form id="drill-search-form"> + <input type="text" placeholder="Search Apache Drill" id="drill-search-term" /> + <button type="submit"> + <i class="fa fa-search"></i> + </button> + </form> + </li> + <li class="d"> + <a href="/drill/download/"> + <i class="fa fa-cloud-download"></i> Download + </a> + </li> +</ul> +</div> + + <link href="/drill/css/content.css" rel="stylesheet" type="text/css"> + +<div class="post int_text"> + <header class="post-header"> + <div class="int_title"> + <h1 class="post-title">Drill 0.7 Released</h1> + </div> + <p class="post-meta"> + + + + <strong>Author:</strong> Jacques Nadeau (PMC Chair and Committer, Apache Drill)<br /> + +<strong>Date:</strong> Dec 23, 2014 +</p> + </header> + <div class="addthis_sharing_toolbox"></div> + + <article class="post-content"> + <p>I'm excited to announce that the community has just released Drill 0.7, which includes <a href="https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12313820&version=12327473";>228 resolved JIRAs</a> and numerous enhancements such as: </p> + +<ul> +<li>No dependency on UDP multicast. Drill can now work on EC2, as well as clusters with multiple subnets or multihomed configurations</li> +<li><a href="https://cwiki.apache.org/confluence/display/DRILL/Partition+Pruning";>Automatic partition pruning</a> based on directory structures</li> +<li>New nested data functions: <a href="https://cwiki.apache.org/confluence/display/DRILL/KVGEN+Function";>KVGEN</a> and <a href="https://cwiki.apache.org/confluence/display/DRILL/FLATTEN+Function";>FLATTEN</a></li> +<li>Fast "schema" return. This provides a better experience when using BI tools</li> +<li>Hive 0.13 Metastore support</li> +<li>Improved performance for queries on JSON data</li> +</ul> + +<p>You can now <a href="/drill/download/">download Drill 0.7</a>. As always, you may check out the official <a href="https://cwiki.apache.org/confluence/display/DRILL/Release+Notes";>release notes</a> for more details.</p> + +<p>In case you're interested in understanding more about where we're heading, check out Tomer's recent blog post outlining some of the <a href="/drill/blog/2014/12/16/whats-coming-in-2015/">planned initiatives for 2015</a>.</p> + +<p>Happy Drilling!<br> +Jacques Nadeau</p> + + </article> + <div id="disqus_thread"></div> + <script type="text/javascript"> + /* * * CONFIGURATION VARIABLES: EDIT BEFORE PASTING INTO YOUR WEBPAGE * * */ + var disqus_shortname = 'drill'; // required: replace example with your forum shortname + + /* * * DON'T EDIT BELOW THIS LINE * * */ + (function() { + var dsq = document.createElement('script'); dsq.type = 'text/javascript'; dsq.async = true; + dsq.src = '//' + disqus_shortname + '.disqus.com/embed.js'; + (document.getElementsByTagName('head')[0] || document.getElementsByTagName('body')[0]).appendChild(dsq); + })(); + </script> + <noscript>Please enable JavaScript to view the <a href="http://disqus.com/?ref_noscript";>comments powered by Disqus.</a></noscript> + +</div> +<script type="text/javascript" src="//s7.addthis.com/js/300/addthis_widget.js#pubid=ra-548b2caa33765e8d" async="async"></script> + + </div> + <p class="push"></p> +<div id="footer" class="mw"> +<div class="wrapper"> +Copyright © 2012-2014 The Apache Software Foundation, licensed under the Apache License, Version 2.0.<br> +Apache and the Apache feather logo are trademarks of The Apache Software Foundation. Other names appearing on the site may be trademarks of their respective owners.<br/><br/> +</div> +</div> + + <script> +(function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){ +(i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o), +m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m) +})(window,document,'script','//www.google-analytics.com/analytics.js','ga'); + +ga('create', 'UA-53379651-1', 'auto'); +ga('send', 'pageview'); +</script> +<script type="text/javascript" src="//s7.addthis.com/js/300/addthis_widget.js#pubid=ra-548b2caa33765e8d" async="async"></script> +</body> +</html> http://git-wip-us.apache.org/repos/asf/drill-site/blob/bbdefcb2/_site/blog/2015/01/27/schema-free-json-data-infrastructure/index.html ---------------------------------------------------------------------- diff --git a/_site/blog/2015/01/27/schema-free-json-data-infrastructure/index.html b/_site/blog/2015/01/27/schema-free-json-data-infrastructure/index.html new file mode 100644 index 0000000..53659e3 --- /dev/null +++ b/_site/blog/2015/01/27/schema-free-json-data-infrastructure/index.html @@ -0,0 +1,222 @@ +<!DOCTYPE html> +<html> + +<head> + +<meta charset="UTF-8"> +<meta name=viewport content="width=device-width, initial-scale=1"> +<meta name="robots" content="noindex"> + +<title>Schema-free JSON Data Infrastructure - Apache Drill</title> + +<link href="//maxcdn.bootstrapcdn.com/font-awesome/4.3.0/css/font-awesome.min.css" rel="stylesheet" type="text/css"/> +<link href='//fonts.googleapis.com/css?family=PT+Sans' rel='stylesheet' type='text/css'/> +<link href="/drill/css/site.css" rel="stylesheet" type="text/css"/> + +<link rel="shortcut icon" href="/drill/favicon.ico" type="image/x-icon"/> +<link rel="icon" href="/drill/favicon.ico" type="image/x-icon"/> + +<script src="//ajax.googleapis.com/ajax/libs/jquery/1.11.1/jquery.min.js" language="javascript" type="text/javascript"></script> +<script src="//cdnjs.cloudflare.com/ajax/libs/jquery-easing/1.3/jquery.easing.min.js" language="javascript" type="text/javascript"></script> +<script language="javascript" type="text/javascript" src="/drill/js/modernizr.custom.js"></script> +<script language="javascript" type="text/javascript" src="/drill/js/script.js"></script> +<script language="javascript" type="text/javascript" src="/drill/js/drill.js"></script> + +</head> + + +<body onResize="resized();"> + <div class="page-wrap"> + <div class="bui"></div> + +<div id="menu" class="mw"> +<ul> + <li class='toc-categories'> + <a class="expand-toc-icon" href="javascript:void(0);"><i class="fa fa-bars"></i></a> + </li> + <li class="logo"><a href="/drill/"></a></li> + <li class='expand-menu'> + <a href="javascript:void(0);"><span class='menu-text'>Menu</span><span class='expand-icon'><i class="fa fa-bars"></i></span></a> + </li> + <li class='clear-float'></li> + <li class="documentation-menu"> + <a href="/drill/docs/">Documentation</a> + <ul> + + <li><a href="/drill/docs/getting-started/">Getting Started</a></li> + + <li><a href="/drill/docs/architecture/">Architecture</a></li> + + <li><a href="/drill/docs/tutorials/">Tutorials</a></li> + + <li><a href="/drill/docs/install-drill/">Install Drill</a></li> + + <li><a href="/drill/docs/configure-drill/">Configure Drill</a></li> + + <li><a href="/drill/docs/connect-a-data-source/">Connect a Data Source</a></li> + + <li><a href="/drill/docs/odbc-jdbc-interfaces/">ODBC/JDBC Interfaces</a></li> + + <li><a href="/drill/docs/query-data/">Query Data</a></li> + + <li><a href="/drill/docs/performance-tuning/">Performance Tuning</a></li> + + <li><a href="/drill/docs/log-and-debug/">Log and Debug</a></li> + + <li><a href="/drill/docs/sql-reference/">SQL Reference</a></li> + + <li><a href="/drill/docs/data-sources-and-file-formats/">Data Sources and File Formats</a></li> + + <li><a href="/drill/docs/develop-custom-functions/">Develop Custom Functions</a></li> + + <li><a href="/drill/docs/troubleshooting/">Troubleshooting</a></li> + + <li><a href="/drill/docs/developer-information/">Developer Information</a></li> + + <li><a href="/drill/docs/release-notes/">Release Notes</a></li> + + <li><a href="/drill/docs/sample-datasets/">Sample Datasets</a></li> + + <li><a href="/drill/docs/project-bylaws/">Project Bylaws</a></li> + + </ul> + </li> + <li class='nav'> + <a href="/drill/community-resources/">Community</a> + <ul> + <li><a href="/drill/team/">Team</a></li> + <li><a href="/drill/mailinglists/">Mailing Lists</a></li> + <li><a href="/drill/community-resources/">Community Resources</a></li> + </ul> + </li> + <li class='nav'><a href="/drill/faq/">FAQ</a></li> + <li class='nav'><a href="/drill/blog/">Blog</a></li> + <li id="twitter-menu-item"><a href="https://twitter.com/apachedrill"; title="apachedrill on twitter" target="_blank"><img src="/drill/images/twitter_32_26_white.png" alt="twitter logo" align="center"></a> </li> + <li class='search-bar'> + <form id="drill-search-form"> + <input type="text" placeholder="Search Apache Drill" id="drill-search-term" /> + <button type="submit"> + <i class="fa fa-search"></i> + </button> + </form> + </li> + <li class="d"> + <a href="/drill/download/"> + <i class="fa fa-cloud-download"></i> Download + </a> + </li> +</ul> +</div> + + <link href="/drill/css/content.css" rel="stylesheet" type="text/css"> + +<div class="post int_text"> + <header class="post-header"> + <div class="int_title"> + <h1 class="post-title">Schema-free JSON Data Infrastructure</h1> + </div> + <p class="post-meta"> + + + + <strong>Author:</strong> Tomer Shiran (Founder, PMC Member and Committer, Apache Drill)<br /> + +<strong>Date:</strong> Jan 27, 2015 +</p> + </header> + <div class="addthis_sharing_toolbox"></div> + + <article class="post-content"> + <p>JSON has emerged in recent years as the de-facto standard data exchange format. It is being used everywhere. Front-end Web applications use JSON to maintain data and communicate with back-end applications. Web APIs are JSON-based (eg, <a href="https://dev.twitter.com/rest/public";>Twitter REST APIs</a>, <a href="http://developers.marketo.com/documentation/rest/";>Marketo REST APIs</a>, <a href="https://developer.github.com/v3/";>GitHub API</a>). It's the format of choice for public datasets, operational log files and more.</p> + +<h1 id="why-is-json-a-convenient-data-exchange-format?">Why is JSON a Convenient Data Exchange Format?</h1> + +<p>While I won't dive into the historical roots of JSON (JavaScript Object Notation, <a href="http://en.wikipedia.org/wiki/JSON#JavaScript_eval.28.29";><code>eval()</code></a>, etc.), I do want to highlight several attributes of JSON that make it a convenient data exchange format:</p> + +<ul> +<li><strong>JSON is self-describing</strong>. You can look at a JSON document and understand what it represents. The field names are included in the document. You don't need an external schema or definition to interpret JSON-encoded data. This makes life easier for anyone who wants to deal with the data, and it also means that a collection of JSON documents represents what many people call a "schema-less dataset" (where structure can evolve, and different records can have different fields).</li> +<li><strong>JSON is simple</strong>. Other self-describing formats such as XML are much more complicated. A JSON document is made up of arrays and maps (or objects, in JSON terminology), and that's about it.</li> +<li><strong>JSON can naturally represent real-world objects</strong>. Try representing your application's <code>Customer</code> object (with the person's address, order history, etc.) in a CSV file or a relational database. It's hard. In fact, ORM systems were invented to help alleviate this issue.</li> +<li><strong>JSON libraries are available in virtually every programming language</strong>. Take a look at <a href="http://www.json.org/";>the list of supported languages on JSON.org</a>. I counted 15 languages that start with the letters A, B or C.</li> +<li><strong>JSON is idiomatic in loosely typed languages</strong>. Many loosely typed languages, such as Python, Ruby and JavaScript, have data structures that are similar to JSON objects, making it very natural to handle JSON data in those languages. For example, a Python dictionary looks just like a JSON object. This makes it easy for developers to utilize JSON in their applications.</li> +</ul> + +<h1 id="json-data-infrastructure">JSON Data Infrastructure</h1> + +<p>Traditional data infrastructure, such as relational databases, has some features that make it easier to store and process JSON-encoded data. For example, Oracle has <a href="https://docs.oracle.com/database/121/ADXDB/json.htm";>a JSON data type and a set of functions for handling JSON data</a>.</p> + +<p>However, a new class of data infrastructure is providing a much more seamless experience via a full-fledged JSON data model. For example:</p> + +<ul> +<li>Drill is a SQL engine in which each record is conceptually a JSON document.</li> +<li>Elasticsearch is a search engine in which each indexed document is conceptually a JSON document.</li> +<li>MongoDB is an operational database in which each record is conceptually a JSON document.</li> +</ul> + +<p>These systems view JSON as a data model as opposed to one of many data types, realizing that JSON offers a simple way to represent real-world objects.</p> + +<table><thead> +<tr> +<th></th> +<th>Traditional Infrastructure</th> +<th>JSON Infrastructure</th> +</tr> +</thead><tbody> +<tr> +<td><strong>Examples:</strong></td> +<td>Oracle, SQL Server</td> +<td>Drill, Elasticsearch, MongoDB</td> +</tr> +<tr> +<td><strong>Record:</strong></td> +<td>Tuple</td> +<td>JSON document</td> +</tr> +<tr> +<td><strong>Variable schema:</strong></td> +<td>No</td> +<td>Yes</td> +</tr> +</tbody></table> + +<p>If you happen to be in the Bay Area tomorrow, please join Gaurav Gupta (VP Product Management, Elasticsearch), Paul Pedersen (Deputy CTO, MongoDB), Robert Greene (Senior Principal Product Manager, Oracle), Sukanta Ganguly (VP Solutions Architecture, Aerospike) and me for a panel moderated by Gartner's Nick Heudecker on this new world of schema-free JSON. Check out <a href="http://www.meetup.com/SF-Bay-Areas-Big-Data-Think-Tank/";>The Hive Big Data Think Tank</a> for more information.</p> + + </article> + <div id="disqus_thread"></div> + <script type="text/javascript"> + /* * * CONFIGURATION VARIABLES: EDIT BEFORE PASTING INTO YOUR WEBPAGE * * */ + var disqus_shortname = 'drill'; // required: replace example with your forum shortname + + /* * * DON'T EDIT BELOW THIS LINE * * */ + (function() { + var dsq = document.createElement('script'); dsq.type = 'text/javascript'; dsq.async = true; + dsq.src = '//' + disqus_shortname + '.disqus.com/embed.js'; + (document.getElementsByTagName('head')[0] || document.getElementsByTagName('body')[0]).appendChild(dsq); + })(); + </script> + <noscript>Please enable JavaScript to view the <a href="http://disqus.com/?ref_noscript";>comments powered by Disqus.</a></noscript> + +</div> +<script type="text/javascript" src="//s7.addthis.com/js/300/addthis_widget.js#pubid=ra-548b2caa33765e8d" async="async"></script> + + </div> + <p class="push"></p> +<div id="footer" class="mw"> +<div class="wrapper"> +Copyright © 2012-2014 The Apache Software Foundation, licensed under the Apache License, Version 2.0.<br> +Apache and the Apache feather logo are trademarks of The Apache Software Foundation. Other names appearing on the site may be trademarks of their respective owners.<br/><br/> +</div> +</div> + + <script> +(function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){ +(i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o), +m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m) +})(window,document,'script','//www.google-analytics.com/analytics.js','ga'); + +ga('create', 'UA-53379651-1', 'auto'); +ga('send', 'pageview'); +</script> +<script type="text/javascript" src="//s7.addthis.com/js/300/addthis_widget.js#pubid=ra-548b2caa33765e8d" async="async"></script> +</body> +</html>
