jackjlli commented on a change in pull request #4086: Update doc for "Creating
Pinot segments" to reflect the current code base
URL: https://github.com/apache/incubator-pinot/pull/4086#discussion_r273236349
##########
File path: docs/pinot_hadoop.rst
##########
@@ -201,64 +183,66 @@ or uncompressed (in which case it is a directory under
``segmentDirectoryPath``)
Realtime segment generation
^^^^^^^^^^^^^^^^^^^^^^^^^^^
-To consume in realtime, we simply need to create a table that uses the same
schema and points to the Kafka topic to
-consume from, using a table definition such as this one:
-
-.. code-block:: none
-
- {
- "tableName":"flights",
- "segmentsConfig" : {
- "retentionTimeUnit":"DAYS",
- "retentionTimeValue":"7",
- "segmentPushFrequency":"daily",
- "segmentPushType":"APPEND",
- "replication" : "1",
- "schemaName" : "flights",
- "timeColumnName" : "daysSinceEpoch",
- "timeType" : "DAYS",
- "segmentAssignmentStrategy" : "BalanceNumSegmentAssignmentStrategy"
- },
- "tableIndexConfig" : {
- "invertedIndexColumns" : ["Carrier"],
- "loadMode" : "HEAP",
- "lazyLoad" : "false",
- "streamConfigs": {
- "streamType": "kafka",
- "stream.kafka.consumer.type": "highLevel",
- "stream.kafka.topic.name": "flights-realtime",
- "stream.kafka.decoder.class.name":
"org.apache.pinot.core.realtime.impl.kafka.KafkaJSONMessageDecoder",
- "stream.kafka.zk.broker.url": "localhost:2181",
- "stream.kafka.hlc.zk.connect.string": "localhost:2181"
- }
- },
- "tableType":"REALTIME",
- "tenants" : {
- "broker":"DefaultTenant_BROKER",
- "server":"DefaultTenant_SERVER"
- },
- "metadata": {
- }
- }
+To consume in realtime, we simply need to create a table with the same name as
the schema and point to the Kafka topic
+to consume from, using a table definition such as this one:
+
+.. code-block:: json
+
+ {
+ "tableName": "flights",
+ "tableType": "REALTIME",
+ "segmentsConfig": {
+ "retentionTimeUnit": "DAYS",
+ "retentionTimeValue": "7",
+ "segmentPushFrequency": "daily",
+ "segmentPushType": "APPEND",
+ "replication": "1",
+ "timeColumnName": "daysSinceEpoch",
+ "timeType": "DAYS",
+ "segmentAssignmentStrategy": "BalanceNumSegmentAssignmentStrategy"
+ },
+ "tableIndexConfig": {
+ "invertedIndexColumns": [
+ "flightNumber",
+ "tags",
+ "daysSinceEpoch"
+ ],
+ "loadMode": "MMAP",
+ "streamConfigs": {
+ "streamType": "kafka",
+ "stream.kafka.consumer.type": "highLevel",
+ "stream.kafka.topic.name": "flights-realtime",
+ "stream.kafka.decoder.class.name":
"org.apache.pinot.core.realtime.impl.kafka.KafkaJSONMessageDecoder",
+ "stream.kafka.zk.broker.url": "localhost:2181",
+ "stream.kafka.hlc.zk.connect.string": "localhost:2181"
+ }
+ },
+ "tenants": {
+ "broker": "brokerTenant",
+ "server": "serverTenant"
+ },
+ "metadata": {
+ }
+ }
First, we'll start a local instance of Kafka and start streaming data into it:
-.. code-block:: none
+.. code-block:: bash
- bin/pinot-admin.sh StartKafka &
- bin/pinot-admin.sh StreamAvroIntoKafka -avroFile flights-2014.avro
-kafkaTopic flights-realtime &
+ bin/pinot-admin.sh StartKafka &
+ bin/pinot-admin.sh StreamAvroIntoKafka -avroFile flights-2014.avro
-kafkaTopic flights-realtime &
This will stream one event per second from the Avro file to the Kafka topic.
Then, we'll create a realtime table, which
will start consuming from the Kafka topic.
-.. code-block:: none
+.. code-block:: bash
- bin/pinot-admin.sh AddTable -filePath flights-definition-realtime.json
+ bin/pinot-admin.sh AddTable -filePath flights-definition-realtime.json
-We can then query the table and see the events stream in:
+We can then query the table with the following query to see the events stream
in:
Review comment:
You can remove the `in:` in this sentence now. :)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
