drill git commit: add kafka sp doc

2018-02-09 Thread bridgetb 
Repository: drill
Updated Branches:
  refs/heads/gh-pages 159541579 -> 9ce5ae1f0


add kafka sp doc


Project: http://git-wip-us.apache.org/repos/asf/drill/repo
Commit: http://git-wip-us.apache.org/repos/asf/drill/commit/9ce5ae1f
Tree: http://git-wip-us.apache.org/repos/asf/drill/tree/9ce5ae1f
Diff: http://git-wip-us.apache.org/repos/asf/drill/diff/9ce5ae1f

Branch: refs/heads/gh-pages
Commit: 9ce5ae1f06941f86cdc8643ecda523d62972ae08
Parents: 1595415
Author: Bridget Bevens 
Authored: Fri Feb 9 17:02:56 2018 -0800
Committer: Bridget Bevens 
Committed: Fri Feb 9 17:02:56 2018 -0800

--
 _data/docs.json | 71 
 1 file changed, 61 insertions(+), 10 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/drill/blob/9ce5ae1f/_data/docs.json
--
diff --git a/_data/docs.json b/_data/docs.json
index e25ef47..dca27aa 100644
--- a/_data/docs.json
+++ b/_data/docs.json
@@ -2484,14 +2484,31 @@
 }
 ], 
 "children": [], 
-"next_title": "ODBC/JDBC Interfaces", 
-"next_url": "/docs/odbc-jdbc-interfaces/", 
+"next_title": "Kafka Storage Plugin", 
+"next_url": "/docs/kafka-storage-plugin/", 
 "parent": "Connect a Data Source", 
 "previous_title": "HTTPD Storage Plugin", 
 "previous_url": "/docs/httpd-storage-plugin/", 
 "relative_path": 
"_docs/connect-a-data-source/plugins/112-opentsdb-storage-plugin.md", 
 "title": "OpenTSDB Storage Plugin", 
 "url": "/docs/opentsdb-storage-plugin/"
+}, 
+{
+"breadcrumbs": [
+{
+"title": "Connect a Data Source", 
+"url": "/docs/connect-a-data-source/"
+}
+], 
+"children": [], 
+"next_title": "ODBC/JDBC Interfaces", 
+"next_url": "/docs/odbc-jdbc-interfaces/", 
+"parent": "Connect a Data Source", 
+"previous_title": "OpenTSDB Storage Plugin", 
+"previous_url": "/docs/opentsdb-storage-plugin/", 
+"relative_path": 
"_docs/connect-a-data-source/plugins/113-kafka-storage-plugin.md", 
+"title": "Kafka Storage Plugin", 
+"url": "/docs/kafka-storage-plugin/"
 }
 ], 
 "next_title": "Connect a Data Source Introduction", 
@@ -5274,6 +5291,23 @@
 "title": "KVGEN", 
 "url": "/docs/kvgen/"
 }, 
+"Kafka Storage Plugin": {
+"breadcrumbs": [
+{
+"title": "Connect a Data Source", 
+"url": "/docs/connect-a-data-source/"
+}
+], 
+"children": [], 
+"next_title": "ODBC/JDBC Interfaces", 
+"next_url": "/docs/odbc-jdbc-interfaces/", 
+"parent": "Connect a Data Source", 
+"previous_title": "OpenTSDB Storage Plugin", 
+"previous_url": "/docs/opentsdb-storage-plugin/", 
+"relative_path": 
"_docs/connect-a-data-source/plugins/113-kafka-storage-plugin.md", 
+"title": "Kafka Storage Plugin", 
+"url": "/docs/kafka-storage-plugin/"
+}, 
 "LIMIT Clause": {
 "breadcrumbs": [
 {
@@ -6619,8 +6653,8 @@
 "next_title": "Interfaces Introduction", 
 "next_url": "/docs/interfaces-introduction/", 
 "parent": "", 
-"previous_title": "OpenTSDB Storage Plugin", 
-"previous_url": "/docs/opentsdb-storage-plugin/", 
+"previous_title": "Kafka Storage Plugin", 
+"previous_url": "/docs/kafka-storage-plugin/", 
 "relative_path": "_docs/060-odbc-jdbc-interfaces.md", 
 "title": "ODBC/JDBC Interfaces", 
 "url": "/docs/odbc-jdbc-interfaces/"
@@ -6675,8 +6709,8 @@
 }
 ], 
 "children": [], 
-"next_title": "ODBC/JDBC Interfaces", 
-"next_url": "/docs/odbc-jdbc-interfaces/", 
+"next_title": "Kafka Storage Plugin", 
+"next_url": "/docs/kafka-storage-plugin/", 
 "parent": "Connect a Data Source", 
 "previous_title": "HTTPD Storage Plugin", 
 "previous_url": "/docs/httpd-storage-plugin/", 
@@ -15457,14 +15491,31 @@
 }
 ],

drill git commit: add kafka sp doc

2018-02-09 Thread bridgetb 
Repository: drill
Updated Branches:
  refs/heads/gh-pages 1fbd74fe2 -> 159541579


add kafka sp doc


Project: http://git-wip-us.apache.org/repos/asf/drill/repo
Commit: http://git-wip-us.apache.org/repos/asf/drill/commit/15954157
Tree: http://git-wip-us.apache.org/repos/asf/drill/tree/15954157
Diff: http://git-wip-us.apache.org/repos/asf/drill/diff/15954157

Branch: refs/heads/gh-pages
Commit: 159541579cca476aa031c5aed26594768e426899
Parents: 1fbd74f
Author: Bridget Bevens 
Authored: Fri Feb 9 16:51:32 2018 -0800
Committer: Bridget Bevens 
Committed: Fri Feb 9 16:51:32 2018 -0800

--
 .../plugins/113-kafka-storage-plugin.md | 218 +++
 1 file changed, 218 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/drill/blob/15954157/_docs/connect-a-data-source/plugins/113-kafka-storage-plugin.md
--
diff --git a/_docs/connect-a-data-source/plugins/113-kafka-storage-plugin.md 
b/_docs/connect-a-data-source/plugins/113-kafka-storage-plugin.md
new file mode 100644
index 000..81e984b
--- /dev/null
+++ b/_docs/connect-a-data-source/plugins/113-kafka-storage-plugin.md
@@ -0,0 +1,218 @@
+---
+title: "Kafka Storage Plugin"
+date: 2018-02-08 02:32:57 UTC
+parent: "Connect a Data Source"
+---
+
+As of Drill 1.12, Drill provides a storage plugin for Kafka. The Kafka storage 
plugin enables you to run SQL queries on Apache Kafka and perform interactive 
analysis on the data. When you install Drill, a preconfigured Kafka storage 
plugin is available on the Storage page in the Drill Web Console. Once you 
enable and configure the storage plugin, you can query Kafka from Drill. 
+
+The following sections provide information about the Kafka storage plugin, how 
to enable and configure the Kafka storage plugin in Drill, options that you can 
set at the system or session level, and example queries on a Kafka data source. 
You can refer to the 
[README](https://github.com/apache/drill/tree/master/contrib/storage-kafka#drill-kafka-plugin)
 and [Apache Kafka](https://kafka.apache.org/) for additional information.  
+
+## Kafka Support
+
+Currently, the Kafka storage plugin supports: 
+
+- Kafka-0.10 and above
+- Reading Kafka messages of type JSON only
+- The following message reader to read the Kafka messages:  
+
+| **MessageReader** | **Description**   | **Key DeSerializer** 
   | **Value DeSerializer** 
|
+|---|---|-||
+| JsonMessageReader | To read Json messages | 
org.apache.kafka.common.serialization.ByteArrayDeserializer | 
org.apache.kafka.common.serialization.ByteArrayDeserializer| 
+
+## About the Kafka Storage Plugin  
+
+In Drill, each Kafka topic is mapped to an SQL table. When run a query on a 
table, the query scans all the messages from the earliest offset to the latest 
offset of that topic at that point in time. The Kafka storage plugin 
automatically discovers all the topics (tables), so you can perform analysis 
without executing DDL statements.  
+
+The Kafka storage plugin also fetches the following metadata for each message: 
 
+
+- kafkaTopic
+- kafkaPartitionId
+- kafkaMsgOffset
+- kafkaMsgTimestamp
+- kafkaMsgKey, unless it is not null  
+
+The following examples show Kafka topics and message offsets:  
+
+   $bin/kafka-topics --list --zookeeper localhost:2181
+   clicks
+   clickstream
+   clickstream-json-demo
+
+   $ bin/kafka-console-consumer --bootstrap-server localhost:9092 --topic 
clickstream-json-demo --from-beginning | more
+   
{"userID":"055e9af4-8c3c-4834-8482-8e05367a7bef","sessionID":"7badf08e-1e1d-4aeb-b853-7df2df4431ac","pageName":"shoes","refferalUrl":"yelp","ipAddress":"20.44.183.126","userAgent":"Mozilla/5.0
 (iPhone; CPU iPhone OS 10_3_1 like Mac OS X) AppleWebKit/603.1.30 (KHTML, like 
Gecko) Version/10.0 Mobile/14E304 Safari/602.1","client_ts":1509926023099}
+   
{"userID":"a29454b3-642d-481e-9dd8-0e0d7ef32ef5","sessionID":"b4a89204-b98c-4b4b-a1a9-f28f22d5ead3","pageName":"books","refferalUrl":"yelp","ipAddress":"252.252.113.190","userAgent":"Mozilla/5.0
 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) 
Chrome/51.0.2704.106 Safari/537.36 OPR/38.0.2220.41","client_ts":1509926023100}
+   
{"userID":"8c53b1c6-da47-4b5a-989d-61b5594f3a1d","sessionID":"baae3a1d-25b2-4955-8d07-20191f29ab32","pageName":"login","refferalUrl":"yelp","ipAddress":"110.170.214.255","userAgent":"Mozilla/5.0
 (Macintosh; Intel Mac OS X x.y; rv:42.0) Gecko/20100101 
Firefox/42.0","client_ts":1509926023100}
+
+   $ bin/kafka-run-class