This is an automated email from the ASF dual-hosted git repository.
sunithabeeram pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git
The following commit(s) were added to refs/heads/master by this push:
new 9ded2a6 Add more documents in Admin Guide (#3972)
9ded2a6 is described below
commit 9ded2a644485d45041a2470d20d04b460d5e507b
Author: Jialiang Li <[email protected]>
AuthorDate: Thu Mar 14 12:16:40 2019 -0700
Add more documents in Admin Guide (#3972)
---
docs/img/generate-segment.png | Bin 0 -> 218597 bytes
docs/img/list-schemas.png | Bin 0 -> 8952 bytes
docs/img/query-table.png | Bin 0 -> 35914 bytes
docs/img/upload-segment.png | Bin 0 -> 13944 bytes
docs/management_api.rst | 32 ++++++++++++++++---
docs/pinot_hadoop.rst | 71 ++++++++++++++++++++++++++++++++++++++++--
6 files changed, 96 insertions(+), 7 deletions(-)
diff --git a/docs/img/generate-segment.png b/docs/img/generate-segment.png
new file mode 100644
index 0000000..5848781
Binary files /dev/null and b/docs/img/generate-segment.png differ
diff --git a/docs/img/list-schemas.png b/docs/img/list-schemas.png
new file mode 100644
index 0000000..9b00855
Binary files /dev/null and b/docs/img/list-schemas.png differ
diff --git a/docs/img/query-table.png b/docs/img/query-table.png
new file mode 100644
index 0000000..5859f2b
Binary files /dev/null and b/docs/img/query-table.png differ
diff --git a/docs/img/upload-segment.png b/docs/img/upload-segment.png
new file mode 100644
index 0000000..edd348d
Binary files /dev/null and b/docs/img/upload-segment.png differ
diff --git a/docs/management_api.rst b/docs/management_api.rst
index f8483eb..5a7a0e4 100644
--- a/docs/management_api.rst
+++ b/docs/management_api.rst
@@ -17,11 +17,35 @@
.. under the License.
..
-Managing Pinot via REST API on the Controller
-=============================================
+Managing Pinot
+==============
-*TODO* : Remove this section altogether and find a place somewhere for a
pointer to the management API. Maybe in the 'Running pinot in production'
section?
+There are two ways to manage Pinot cluster, i.e. using Pinot management
console and ``pinot-admin.sh`` script.
-There is a REST API which allows management of tables, tenants, segments and
schemas. It can be accessed by going to ``http://[controller host]/help`` which
offers a web UI to do these tasks, as well as document the REST API.
+Pinot Management Console
+------------------------
+
+There is a REST API which allows management of tables, tenants, segments and
schemas. It can be accessed by going to
+``http://[controller host]/help`` which offers a web UI to do these tasks, as
well as document the REST API.
+
+For example, list all the schema within Pinot cluster:
+
+ .. figure:: img/list-schemas.png
+
+Upload a pinot segment:
+
+ .. figure:: img/upload-segment.png
+
+
+Pinot-admin.sh
+--------------
It can be used instead of the ``pinot-admin.sh`` commands to automate the
creation of tables and tenants.
+
+For example, create a pinot segment:
+
+ .. figure:: img/generate-segment.png
+
+Query a table:
+
+ .. figure:: img/query-table.png
diff --git a/docs/pinot_hadoop.rst b/docs/pinot_hadoop.rst
index 1611040..95918d2 100644
--- a/docs/pinot_hadoop.rst
+++ b/docs/pinot_hadoop.rst
@@ -23,7 +23,8 @@ Creating Pinot segments
=======================
Pinot segments can be created offline on Hadoop, or via command line from data
files. Controller REST endpoint
-can then be used to add the segment to the table to which the segment belongs.
+can then be used to add the segment to the table to which the segment belongs.
Pinot segments can also be created by
+ingesting data from realtime resources (such as Kafka).
Creating segments using hadoop
------------------------------
@@ -177,8 +178,8 @@ Sample Schema:
"schemaName" : "mySchema",
}
-Pushing segments to Pinot
-^^^^^^^^^^^^^^^^^^^^^^^^^
+Pushing offline segments to Pinot
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
You can use curl to push a segment to pinot:
@@ -196,3 +197,67 @@ Alternatively you can use the pinot-admin.sh utility to
upload one or more segme
The command uploads all the segments found in ``segmentDirectoryPath``.
The segments could be either tar-compressed (in which case it is a file under
``segmentDirectoryPath``)
or uncompressed (in which case it is a directory under
``segmentDirectoryPath``).
+
+Realtime segment generation
+^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+To consume in realtime, we simply need to create a table that uses the same
schema and points to the Kafka topic to
+consume from, using a table definition such as this one:
+
+.. code-block:: none
+ {
+ "tableName":"flights",
+ "segmentsConfig" : {
+ "retentionTimeUnit":"DAYS",
+ "retentionTimeValue":"7",
+ "segmentPushFrequency":"daily",
+ "segmentPushType":"APPEND",
+ "replication" : "1",
+ "schemaName" : "flights",
+ "timeColumnName" : "daysSinceEpoch",
+ "timeType" : "DAYS",
+ "segmentAssignmentStrategy" : "BalanceNumSegmentAssignmentStrategy"
+ },
+ "tableIndexConfig" : {
+ "invertedIndexColumns" : ["Carrier"],
+ "loadMode" : "HEAP",
+ "lazyLoad" : "false",
+ "streamConfigs": {
+ "streamType": "kafka",
+ "stream.kafka.consumer.type": "highLevel",
+ "stream.kafka.topic.name": "flights-realtime",
+ "stream.kafka.decoder.class.name":
"org.apache.pinot.core.realtime.impl.kafka.KafkaJSONMessageDecoder",
+ "stream.kafka.zk.broker.url": "localhost:2181",
+ "stream.kafka.hlc.zk.connect.string": "localhost:2181"
+ }
+ },
+ "tableType":"REALTIME",
+ "tenants" : {
+ "broker":"DefaultTenant_BROKER",
+ "server":"DefaultTenant_SERVER"
+ },
+ "metadata": {
+ }
+ }
+
+First, we'll start a local instance of Kafka and start streaming data into it:
+
+.. code-block:: none
+
+ bin/pinot-admin.sh StartKafka &
+ bin/pinot-admin.sh StreamAvroIntoKafka -avroFile flights-2014.avro
-kafkaTopic flights-realtime &
+
+This will stream one event per second from the Avro file to the Kafka topic.
Then, we'll create a realtime table, which
+will start consuming from the Kafka topic.
+
+.. code-block:: none
+
+ bin/pinot-admin.sh AddTable -filePath flights-definition-realtime.json
+
+We can then query the table and see the events stream in:
+
+.. code-block:: none
+
+
{"traceInfo":{},"numDocsScanned":17,"aggregationResults":[{"function":"count_star","value":"17"}],"timeUsedMs":27,"segmentStatistics":[],"exceptions":[],"totalDocs":17}
+
+Repeating the query multiple times should show the events slowly being
streamed into the table.
\ No newline at end of file
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
