[GitHub] [incubator-pinot] mcvsubbu commented on a change in pull request #3975: ReadTheDocs documentation for Table Configs, Monitoring, and Deployment

[GitBox]

mcvsubbu commented on a change in pull request #3975: ReadTheDocs documentation 
for Table Configs, Monitoring, and Deployment
URL: https://github.com/apache/incubator-pinot/pull/3975#discussion_r269769855
 
 

 ##
 File path: docs/in_production.rst
 ##
 @@ -17,51 +17,127 @@
 .. under the License.
 ..
 
-Running Pinot in production
+Running Pinot in Production
 ===
 
-Installing Pinot
-
-
 Requirements
 
 
-* Java 8+
-* Several nodes with enough memory
-* A working installation of Zookeeper
+You will need the following in order to run pinot in production:
 
-Recommended environment
-~~~
-
-* Shared storage infrastructure (such as NFS)
-* Regular Zookeeper backups
-* HTTP load balancers (such as nginx/haproxy)
+* Hardware for controller/broker/servers as per your load
+* Working installation of Zookeeper that Pinot can use. We recommend setting 
aside a path within zookpeer and including that path in pinot.controller.zkStr. 
Pinot will create its own cluster under this path (cluster name decided by 
pinot.controller.helixClusterName)
+* Shared storage mounted on controllers (if you plan to have multiple 
controllers for the same cluster). Alternatively, an implementation of PinotFS 
that the Pinot hosts have access to.
+* HTTP load balancers for spraying queries across brokers (or other mechanism 
to balance queries)
+* HTTP load balancers for spraying controller requests (e.g. segment push, or 
other controller APIs) or other mechanisms for distribution of these requests.
 
 Deploying Pinot

+~~~
+
+In general, when deploying Pinot services, it is best to adhere to a specific 
ordering in which the various components should be deployed. This deployment 
order is recommended in case of the scenario that there might be protocol or 
other significant differences, the deployments go out in a predictable order in 
which failure  due to these changes can be avoided.
 
-Direct deployment of Pinot
-~~
+The ordering is as follows:
 
-Deployment of Pinot on Kubernetes
-~
+#. pinot-controller
+#. pinot-broker
+#. pinot-server
+#. pinot-minion
 
 Managing Pinot
---
+~~
 
 Creating tables
-~~~
+---
 
 Updating tables
-~~~
+---
 
 Uploading data
-~~
+--
 
 Configuring realtime data ingestion
-~~~
+---
 
 Monitoring Pinot
 
 
+Pinot exposes several metrics to monitor the service and ensure that pinot 
users are not experiencing issues. In this section we discuss some of the key 
metrics that are useful to monitor. A full list of metrics is available in the 
`Metrics `_ section.
 
 Review comment:
   we dont have a 'full list' so to speak.
   so maybe we should remove this?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

[GitHub] [incubator-pinot] mcvsubbu commented on a change in pull request #3975: ReadTheDocs documentation for Table Configs, Monitoring, and Deployment

[GitBox]

mcvsubbu commented on a change in pull request #3975: ReadTheDocs documentation 
for Table Configs, Monitoring, and Deployment
URL: https://github.com/apache/incubator-pinot/pull/3975#discussion_r269801568
 
 

 ##
 File path: docs/in_production.rst
 ##
 @@ -17,51 +17,127 @@
 .. under the License.
 ..
 
-Running Pinot in production
+Running Pinot in Production
 ===
 
-Installing Pinot
-
-
 Requirements
 
 
-* Java 8+
-* Several nodes with enough memory
-* A working installation of Zookeeper
+You will need the following in order to run pinot in production:
 
-Recommended environment
-~~~
-
-* Shared storage infrastructure (such as NFS)
-* Regular Zookeeper backups
-* HTTP load balancers (such as nginx/haproxy)
+* Hardware for controller/broker/servers as per your load
+* Working installation of Zookeeper that Pinot can use. We recommend setting 
aside a path within zookpeer and including that path in pinot.controller.zkStr. 
Pinot will create its own cluster under this path (cluster name decided by 
pinot.controller.helixClusterName)
+* Shared storage mounted on controllers (if you plan to have multiple 
controllers for the same cluster). Alternatively, an implementation of PinotFS 
that the Pinot hosts have access to.
+* HTTP load balancers for spraying queries across brokers (or other mechanism 
to balance queries)
+* HTTP load balancers for spraying controller requests (e.g. segment push, or 
other controller APIs) or other mechanisms for distribution of these requests.
 
 Deploying Pinot

+~~~
+
+In general, when deploying Pinot services, it is best to adhere to a specific 
ordering in which the various components should be deployed. This deployment 
order is recommended in case of the scenario that there might be protocol or 
other significant differences, the deployments go out in a predictable order in 
which failure  due to these changes can be avoided.
 
-Direct deployment of Pinot
-~~
+The ordering is as follows:
 
-Deployment of Pinot on Kubernetes
-~
+#. pinot-controller
+#. pinot-broker
+#. pinot-server
+#. pinot-minion
 
 Managing Pinot
---
+~~
 
 Creating tables
-~~~
+---
 
 Updating tables
-~~~
+---
 
 Uploading data
-~~
+--
 
 Configuring realtime data ingestion
-~~~
+---
 
 Monitoring Pinot
 
 
+Pinot exposes several metrics to monitor the service and ensure that pinot 
users are not experiencing issues. In this section we discuss some of the key 
metrics that are useful to monitor. A full list of metrics is available in the 
`Metrics `_ section.
 
 Review comment:
   ah i see you have a todo. no problem. your call.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

[GitHub] [incubator-pinot] mcvsubbu commented on a change in pull request #3975: ReadTheDocs documentation for Table Configs, Monitoring, and Deployment

[GitBox]

mcvsubbu commented on a change in pull request #3975: ReadTheDocs documentation 
for Table Configs, Monitoring, and Deployment
URL: https://github.com/apache/incubator-pinot/pull/3975#discussion_r269360962
 
 

 ##
 File path: docs/in_production.rst
 ##
 @@ -40,11 +40,14 @@ Recommended environment
 Deploying Pinot
 ---
 
-Direct deployment of Pinot
-~~
+In general, when deploying Pinot services, it is best to adhere to a specific 
ordering in which the various components should be deployed. This deployment 
order is recommended incase of the scenario that there might be protocol or 
other significant differences, the deployments go out in a predictable order in 
which failure  due to these changes can be avoided.
 
 Review comment:
   Can we add this text under line 26 and remove the "Recommended environemnt" 
segction? thanks


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

[GitHub] [incubator-pinot] mcvsubbu commented on a change in pull request #3975: ReadTheDocs documentation for Table Configs, Monitoring, and Deployment

[GitBox]

mcvsubbu commented on a change in pull request #3975: ReadTheDocs documentation 
for Table Configs, Monitoring, and Deployment
URL: https://github.com/apache/incubator-pinot/pull/3975#discussion_r268340700
 
 

 ##
 File path: docs/in_production.rst
 ##
 @@ -64,4 +67,61 @@ Configuring realtime data ingestion
 Monitoring Pinot
 
 
+Pinot exposes several metrics to monitor the service and ensure that pinot 
users are not experiencing issues. In this section we discuss some of the key 
metrics that are useful to monitor. A full list of metrics is available in the 
`Metrics `_ section.
+
+Pinot Server
+
+
+* Missing Segments - `NUM_MISSING_SEGMENTS 
`_
+
+  * Number of missing segments that the broker queried for (expected to be on 
the server) but the server didn't have. This can be due to retention or stale 
routing table.
+
+* Query latency - `TOTAL_QUERY_TIME 
`_
+
+  * The number of exception which might have occurred during query execution
+
+* Query Execution Exceptions - `QUERY_EXECUTION_EXCEPTIONS 
`_
+
+  * The number of exception which might have occurred during query execution
+
+* Realtime Consumption Status - `LLC_PARTITION_CONSUMING 
`_
+
+  * This gives a binary value based on whether low-level consumption is 
healthy (1) or unhealthy (0). It's important to ensure at least a single 
replica of each partition is consuming
+
+* Realtime Highest Offset Consumed - `HIGHEST_STREAM_OFFSET_CONSUMED 
`_
+
+  * The highest offset which has been consumed so far.
+
+Pinot Broker
+
+
+* Incoming QPS (per broker) - `QUERIES 
`_
+
+  * The rate which an individual broker is receiving queries. Units are in QPS.
+
+* Dropped Requests - `REQUEST_DROPPED_DUE_TO_SEND_ERROR 
`_,
 `REQUEST_DROPPED_DUE_TO_CONNECTION_ERROR 
`_,
 `REQUEST_DROPPED_DUE_TO_ACCESS_ERROR 
`_
+
+  * These multiple metrics will indicate if a query is dropped, ie the 
processing of that query has been forfeited for some reason.
+
+* Partial Responses - `BROKER_RESPONSES_WITH_PARTIAL_SERVERS_RESPONDED 
`_
+
+  * Indicates a count of partial responses. A partial response is when at 
least 1 of the requested servers fails to respond to the query.
+
+* Table QPS quota exceeded - `QUERY_QUOTA_EXCEEDED 
`_
+
+  * Binary metric which will indicate when the configured QPS quota for a 
table is exceeded (1) or if there is capacity remaining (0).
+
+* Table QPS quota usage percent - `QUERY_QUOTA_CAPACITY_UTILIZATION_RATE 
`_
+
+  * Percentage of the configured QPS quota being utilized.
+
+Pinot Controller
 
 Review comment:
   See the ControllerGauge class.
   
   In general, we have classes BrokerGauge, BrokerMeter, ControllerGauge, 
ControllerMeter, ServerGauge. ServerMeter, MinionGauge, MinionMeter that define 
metric names. There are methiods in the base class that add table name if 
necessary, and  methods that add phase timings (e.g. query phase times)


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: 

[GitHub] [incubator-pinot] mcvsubbu commented on a change in pull request #3975: ReadTheDocs documentation for Table Configs, Monitoring, and Deployment

[GitBox]

mcvsubbu commented on a change in pull request #3975: ReadTheDocs documentation 
for Table Configs, Monitoring, and Deployment
URL: https://github.com/apache/incubator-pinot/pull/3975#discussion_r268339141
 
 

 ##
 File path: docs/in_production.rst
 ##
 @@ -40,11 +40,14 @@ Recommended environment
 Deploying Pinot
 ---
 
-Direct deployment of Pinot
-~~
+In general, when deploying Pinot services, it is best to adhere to a specific 
ordering in which the various components should be deployed. This deployment 
order is recommended incase of the scenario that there might be protocol or 
other significant differences, the deployments go out in a predictable order in 
which failure  due to these changes can be avoided.
 
 Review comment:
   Can you update the requirments section to be not as vague as it is now? A 
paragraph would do. Here is a suggestion. We can iterate and modify later, but 
let us have something in place now. Feel free to modify/add to as you please.
   
   You will need the following in order to run pinot in production:
   * Hardware for controller/broker/servers as per your load
   * Working installation of Zookeeper that Pinot can use. We recommend setting 
aside a path within zookpeer and including that path in pinot.controller.zkStr. 
Pinot will create its own cluster under this path (cluster name decided by 
pinot.controller.helixClusterName)
   * Shared storage mounted on controllers (if you plan to have multiple 
controllers for the same cluster). Alternatively, an implementation of PinotFS 
that the Pinot hosts have access to.
   * HTTP load balancers for spraying queries across brokers (or other 
mechanism to balance queries)
   * HTTP load balancers for spraying controller requests (e.g. segment push, 
or other controller APIs) or other mechanisms for distribution of these 
requests.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

[GitHub] [incubator-pinot] mcvsubbu commented on a change in pull request #3975: ReadTheDocs documentation for Table Configs, Monitoring, and Deployment

[GitBox]

mcvsubbu commented on a change in pull request #3975: ReadTheDocs documentation 
for Table Configs, Monitoring, and Deployment
URL: https://github.com/apache/incubator-pinot/pull/3975#discussion_r268336400
 
 

 ##
 File path: docs/tableconfig_schema.rst
 ##
 @@ -0,0 +1,204 @@
+..
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements.  See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership.  The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License.  You may obtain a copy of the License at
+..
+..   http://www.apache.org/licenses/LICENSE-2.0
+..
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied.  See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+..
+
+Table Config
+
+
+Table Config
+-
+
+Introduction
+
+
+Using tables is how Pinot serves and organizes data. There are many settings 
in the table config which will influence how Pinot operates. The first and most 
significant distinction is using an offline versus a realtime table.
+
+An offline table in Pinot is used to host data which might be periodically 
uploaded - daily, weekly, etc. A realtime table, however, is used to consume 
data from incoming data streams and serve this data in a near-realtime manner. 
This might also be referred to as nearline or just plain 'realtime'.
+
+In this section a sample table configuration will be shown and all sections 
will be explained and if applicable have appropriate sections linked to for 
further explanation of those corresponding Pinot features.
+
+Sample table config and descriptions
+
+
+A sample table config is shown below which has sub-sections collasped. The sub 
sections will be described individually in following sections.
+
+The ``tableName`` should only contain alpha-numeric characters, hyphens ('-'), 
or underscores ('_'). Though using a double-underscore ('__') is not allowed 
and reserved for other features within Pinot.
+
+The ``tableType`` will indicate the type of the table, ``OFFLINE`` or 
``REALTIME``. There are some settings specific to each type. This 
differentiation will be called out below as options are explained.
+
+.. code-block:: none
+
+{
+  "tableName": "myPinotTable",
+  "tableType": "REALTIME"
+  "segmentsConfig": {},
+  "tableIndexConfig": {},
+  "tenants": {},
+  "routing": {},
+  "task": {},
+  "metadata": {}
+}
+
+Segments Config Section
+~~~
+
+The ``segmentsConfig`` section has information about configuring the following:
+
+* Segment Retention - with the ``retentionTimeUnit`` and 
``retentionTimeValue`` options.
+
+  * Allowed values:
+
+* ``retentionTimeUnit`` - ``DAYS``
+* ``retentionTimeValue`` - Positive integers
+
+* ``segmentPushFrequency`` - to indicate how frequently segments are uploaded.
+
+  * Allowed values - ``daily``, ``hourly``
+
+* ``segmentPushType`` - Indicates the type of push to the table.
+
+  * Allowed values - ``APPEND`` means new data will be pushed and appended to 
the current data in the table, all realtime tables *must* be ``APPEND``. 
``REFRESH`` will refresh the entire dataset contained within the table.
+
+* ``replication`` - Number of replicas of data in a table, used for offline 
tables only.
+
+  * Allowed values - Positive integers
+
+* ``replicasPerPartition`` - Number of of data in a table, used for offline 
tables only.
+
+  * Allowed values - Positive integers
+
+* ``schemaName`` - Name of the schema that's been uploaded to the controller
+
+  * Allowed values - String
+
+* Time column - using ``timeColumnName`` and ``timeType``, this must match 
what's configured in the preceeding schema
+
+  * Allowed values - String, this must match the ``timeFieldSpec`` section in 
the schema
+
+* Segment assignment strategy - Described more on the page `Customizing Pinot 
`_
+
+
+.. code-block:: none
+
+"segmentsConfig": {
+  "retentionTimeUnit": "DAYS",
+  "retentionTimeValue": "5",
+  "segmentPushFrequency": "daily",
+  "segmentPushType": "APPEND",
+  "replication": "3",
+  "replicasPerPartition": "3",
+  "schemaName": "myPinotSchmea",
+  "timeColumnName": "daysSinceEpoch",
+  "timeType": "DAYS",
+  "segmentAssignmentStrategy": "BalanceNumSegmentAssignmentStrategy"
+},
+
+Table Index Config Section
+~~
+
+The ``tableIndexConfig`` section has information about how to configure:
+
+* ``invertedIndexColumns`` - Indicates a list of real column names as 
specified in the schema to create 

[GitHub] [incubator-pinot] mcvsubbu commented on a change in pull request #3975: ReadTheDocs documentation for Table Configs, Monitoring, and Deployment

[GitBox]

mcvsubbu commented on a change in pull request #3975: ReadTheDocs documentation 
for Table Configs, Monitoring, and Deployment
URL: https://github.com/apache/incubator-pinot/pull/3975#discussion_r267903230
 
 

 ##
 File path: docs/in_production.rst
 ##
 @@ -40,11 +40,14 @@ Recommended environment
 Deploying Pinot
 ---
 
-Direct deployment of Pinot
-~~
+In general, when deploying Pinot services, it is best to adhere to a specific 
ordering in which the various components should be deployed. This deployment 
order is recommended incase of the scenario that there might be protocol or 
other significant differences, the deployments go out in a predictable order in 
which failure  due to these changes can be avoided.
 
 Review comment:
   "in case", not "incase" right?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

[GitHub] [incubator-pinot] mcvsubbu commented on a change in pull request #3975: ReadTheDocs documentation for Table Configs, Monitoring, and Deployment

[GitBox]

mcvsubbu commented on a change in pull request #3975: ReadTheDocs documentation 
for Table Configs, Monitoring, and Deployment
URL: https://github.com/apache/incubator-pinot/pull/3975#discussion_r267905349
 
 

 ##
 File path: docs/in_production.rst
 ##
 @@ -64,4 +67,63 @@ Configuring realtime data ingestion
 Monitoring Pinot
 
 
+In order for Pinot to provide effective service there is a core set of metrics 
which should be monitored to ensure service stability, fault tolerance and 
acceptable response times. In the section following, there are service level 
metrics which are recommended to be monitored.
+
+More info on metrics collection and viewing a complete set of available metric 
is available in the `Metrics `_ section.
+
+Pinot Server
+
+
+* Missing Segments - `NUM_MISSING_SEGMENTS 
`_
 
 Review comment:
   thanks for adding links


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

[GitHub] [incubator-pinot] mcvsubbu commented on a change in pull request #3975: ReadTheDocs documentation for Table Configs, Monitoring, and Deployment

[GitBox]

mcvsubbu commented on a change in pull request #3975: ReadTheDocs documentation 
for Table Configs, Monitoring, and Deployment
URL: https://github.com/apache/incubator-pinot/pull/3975#discussion_r267902633
 
 

 ##
 File path: docs/in_production.rst
 ##
 @@ -40,11 +40,14 @@ Recommended environment
 Deploying Pinot
 ---
 
-Direct deployment of Pinot
-~~
+In general, when deploying Pinot services, it is best to adhere to a specific 
ordering in which the various components should be deployed. This deployment 
order is recommended incase of the scenario that there might be protocol or 
other significant differences, the deployments go out in a predictable order in 
which failure  due to these changes can be avoided.
 
 Review comment:
   The requirement section is very vague. Either expand it or take it out 
altogether (but we need to add zk somewhere I suppose). In the recommended 
section, pls add shared storage env mounted across controllers. If this storage 
is not shared, things will not work. Altnerativelyu, pinotfs can be used. (and 
ou can point to a section for that). Not clear what the load baancers are 
needed for. Queries to broker or rest to controlers or both. 


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

[GitHub] [incubator-pinot] mcvsubbu commented on a change in pull request #3975: ReadTheDocs documentation for Table Configs, Monitoring, and Deployment

[GitBox]

mcvsubbu commented on a change in pull request #3975: ReadTheDocs documentation 
for Table Configs, Monitoring, and Deployment
URL: https://github.com/apache/incubator-pinot/pull/3975#discussion_r267904312
 
 

 ##
 File path: docs/in_production.rst
 ##
 @@ -64,4 +67,63 @@ Configuring realtime data ingestion
 Monitoring Pinot
 
 
+In order for Pinot to provide effective service there is a core set of metrics 
which should be monitored to ensure service stability, fault tolerance and 
acceptable response times. In the section following, there are service level 
metrics which are recommended to be monitored.
 
 Review comment:
   I might word it like: "Pinot exposes several metrics to monitor the service 
and ensure that pinot users are not experiencing issues. In this section we 
discuss some of the key metrics that are useful to monitor. A full list of 
metrics is available in "


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

[GitHub] [incubator-pinot] mcvsubbu commented on a change in pull request #3975: ReadTheDocs documentation for Table Configs, Monitoring, and Deployment

[GitBox]

mcvsubbu commented on a change in pull request #3975: ReadTheDocs documentation 
for Table Configs, Monitoring, and Deployment
URL: https://github.com/apache/incubator-pinot/pull/3975#discussion_r267903566
 
 

 ##
 File path: docs/in_production.rst
 ##
 @@ -64,4 +67,63 @@ Configuring realtime data ingestion
 Monitoring Pinot
 
 
+In order for Pinot to provide effective service there is a core set of metrics 
which should be monitored to ensure service stability, fault tolerance and 
acceptable response times. In the section following, there are service level 
metrics which are recommended to be monitored.
 
 Review comment:
   Are you going to fill in the empty sections above in a future PR?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

[GitHub] [incubator-pinot] mcvsubbu commented on a change in pull request #3975: ReadTheDocs documentation for Table Configs, Monitoring, and Deployment

[GitBox]

mcvsubbu commented on a change in pull request #3975: ReadTheDocs documentation 
for Table Configs, Monitoring, and Deployment
URL: https://github.com/apache/incubator-pinot/pull/3975#discussion_r267892780
 
 

 ##
 File path: docs/tableconfig_schema.rst
 ##
 @@ -0,0 +1,204 @@
+..
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements.  See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership.  The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License.  You may obtain a copy of the License at
+..
+..   http://www.apache.org/licenses/LICENSE-2.0
+..
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied.  See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+..
+
+Table Config
+
+
+Table Config
+-
+
+Introduction
+
+
+Using tables is how Pinot serves and organizes data. There are many settings 
in the table config which will influence how Pinot operates. The first and most 
significant distinction is using an offline versus a realtime table.
+
+An offline table in Pinot is used to host data which might be periodically 
uploaded - daily, weekly, etc. A realtime table, however, is used to consume 
data from incoming data streams and serve this data in a near-realtime manner. 
This might also be referred to as nearline or just plain 'realtime'.
+
+In this section a sample table configuration will be shown and all sections 
will be explained and if applicable have appropriate sections linked to for 
further explanation of those corresponding Pinot features.
+
+Sample table config and descriptions
+
+
+A sample table config is shown below which has sub-sections collasped. The sub 
sections will be described individually in following sections.
+
+The ``tableName`` should only contain alpha-numeric characters, hyphens ('-'), 
or underscores ('_'). Though using a double-underscore ('__') is not allowed 
and reserved for other features within Pinot.
+
+The ``tableType`` will indicate the type of the table, ``OFFLINE`` or 
``REALTIME``. There are some settings specific to each type. This 
differentiation will be called out below as options are explained.
+
+.. code-block:: none
+
+{
+  "tableName": "myPinotTable",
+  "tableType": "REALTIME"
+  "segmentsConfig": {},
+  "tableIndexConfig": {},
+  "tenants": {},
+  "routing": {},
+  "task": {},
+  "metadata": {}
+}
+
+Segments Config Section
+~~~
+
+The ``segmentsConfig`` section has information about configuring the following:
+
+* Segment Retention - with the ``retentionTimeUnit`` and 
``retentionTimeValue`` options.
+
+  * Allowed values:
+
+* ``retentionTimeUnit`` - ``DAYS``
+* ``retentionTimeValue`` - Positive integers
+
+* ``segmentPushFrequency`` - to indicate how frequently segments are uploaded.
+
+  * Allowed values - ``daily``, ``hourly``
+
+* ``segmentPushType`` - Indicates the type of push to the table.
+
+  * Allowed values - ``APPEND`` means new data will be pushed and appended to 
the current data in the table, all realtime tables *must* be ``APPEND``. 
``REFRESH`` will refresh the entire dataset contained within the table.
+
+* ``replication`` - Number of replicas of data in a table, used for offline 
tables only.
+
+  * Allowed values - Positive integers
+
+* ``replicasPerPartition`` - Number of of data in a table, used for offline 
tables only.
+
+  * Allowed values - Positive integers
+
+* ``schemaName`` - Name of the schema that's been uploaded to the controller
+
+  * Allowed values - String
+
+* Time column - using ``timeColumnName`` and ``timeType``, this must match 
what's configured in the preceeding schema
+
+  * Allowed values - String, this must match the ``timeFieldSpec`` section in 
the schema
+
+* Segment assignment strategy - Described more on the page `Customizing Pinot 
`_
+
+
+.. code-block:: none
+
+"segmentsConfig": {
+  "retentionTimeUnit": "DAYS",
+  "retentionTimeValue": "5",
+  "segmentPushFrequency": "daily",
+  "segmentPushType": "APPEND",
+  "replication": "3",
+  "replicasPerPartition": "3",
+  "schemaName": "myPinotSchmea",
+  "timeColumnName": "daysSinceEpoch",
+  "timeType": "DAYS",
+  "segmentAssignmentStrategy": "BalanceNumSegmentAssignmentStrategy"
+},
+
+Table Index Config Section
+~~
+
+The ``tableIndexConfig`` section has information about how to configure:
+
+* ``invertedIndexColumns`` - Indicates a list of real column names as 
specified in the schema to create 

[GitHub] [incubator-pinot] mcvsubbu commented on a change in pull request #3975: ReadTheDocs documentation for Table Configs, Monitoring, and Deployment

[GitBox]

mcvsubbu commented on a change in pull request #3975: ReadTheDocs documentation 
for Table Configs, Monitoring, and Deployment
URL: https://github.com/apache/incubator-pinot/pull/3975#discussion_r267890561
 
 

 ##
 File path: docs/tableconfig_schema.rst
 ##
 @@ -0,0 +1,204 @@
+..
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements.  See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership.  The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License.  You may obtain a copy of the License at
+..
+..   http://www.apache.org/licenses/LICENSE-2.0
+..
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied.  See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+..
+
+Table Config
+
+
+Table Config
+-
+
+Introduction
+
+
+Using tables is how Pinot serves and organizes data. There are many settings 
in the table config which will influence how Pinot operates. The first and most 
significant distinction is using an offline versus a realtime table.
+
+An offline table in Pinot is used to host data which might be periodically 
uploaded - daily, weekly, etc. A realtime table, however, is used to consume 
data from incoming data streams and serve this data in a near-realtime manner. 
This might also be referred to as nearline or just plain 'realtime'.
+
+In this section a sample table configuration will be shown and all sections 
will be explained and if applicable have appropriate sections linked to for 
further explanation of those corresponding Pinot features.
+
+Sample table config and descriptions
+
+
+A sample table config is shown below which has sub-sections collasped. The sub 
sections will be described individually in following sections.
+
+The ``tableName`` should only contain alpha-numeric characters, hyphens ('-'), 
or underscores ('_'). Though using a double-underscore ('__') is not allowed 
and reserved for other features within Pinot.
+
+The ``tableType`` will indicate the type of the table, ``OFFLINE`` or 
``REALTIME``. There are some settings specific to each type. This 
differentiation will be called out below as options are explained.
+
+.. code-block:: none
+
+{
+  "tableName": "myPinotTable",
+  "tableType": "REALTIME"
+  "segmentsConfig": {},
+  "tableIndexConfig": {},
+  "tenants": {},
+  "routing": {},
+  "task": {},
+  "metadata": {}
+}
+
+Segments Config Section
+~~~
+
+The ``segmentsConfig`` section has information about configuring the following:
+
+* Segment Retention - with the ``retentionTimeUnit`` and 
``retentionTimeValue`` options.
+
+  * Allowed values:
+
+* ``retentionTimeUnit`` - ``DAYS``
+* ``retentionTimeValue`` - Positive integers
+
+* ``segmentPushFrequency`` - to indicate how frequently segments are uploaded.
+
+  * Allowed values - ``daily``, ``hourly``
+
+* ``segmentPushType`` - Indicates the type of push to the table.
+
+  * Allowed values - ``APPEND`` means new data will be pushed and appended to 
the current data in the table, all realtime tables *must* be ``APPEND``. 
``REFRESH`` will refresh the entire dataset contained within the table.
+
+* ``replication`` - Number of replicas of data in a table, used for offline 
tables only.
+
+  * Allowed values - Positive integers
+
+* ``replicasPerPartition`` - Number of of data in a table, used for offline 
tables only.
 
 Review comment:
   this is used for realtime tables only, if the realtime tables are LLC


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

[GitHub] [incubator-pinot] mcvsubbu commented on a change in pull request #3975: ReadTheDocs documentation for Table Configs, Monitoring, and Deployment

[GitBox]

mcvsubbu commented on a change in pull request #3975: ReadTheDocs documentation 
for Table Configs, Monitoring, and Deployment
URL: https://github.com/apache/incubator-pinot/pull/3975#discussion_r267891006
 
 

 ##
 File path: docs/tableconfig_schema.rst
 ##
 @@ -0,0 +1,204 @@
+..
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements.  See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership.  The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License.  You may obtain a copy of the License at
+..
+..   http://www.apache.org/licenses/LICENSE-2.0
+..
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied.  See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+..
+
+Table Config
+
+
+Table Config
+-
+
+Introduction
+
+
+Using tables is how Pinot serves and organizes data. There are many settings 
in the table config which will influence how Pinot operates. The first and most 
significant distinction is using an offline versus a realtime table.
+
+An offline table in Pinot is used to host data which might be periodically 
uploaded - daily, weekly, etc. A realtime table, however, is used to consume 
data from incoming data streams and serve this data in a near-realtime manner. 
This might also be referred to as nearline or just plain 'realtime'.
+
+In this section a sample table configuration will be shown and all sections 
will be explained and if applicable have appropriate sections linked to for 
further explanation of those corresponding Pinot features.
+
+Sample table config and descriptions
+
+
+A sample table config is shown below which has sub-sections collasped. The sub 
sections will be described individually in following sections.
+
+The ``tableName`` should only contain alpha-numeric characters, hyphens ('-'), 
or underscores ('_'). Though using a double-underscore ('__') is not allowed 
and reserved for other features within Pinot.
+
+The ``tableType`` will indicate the type of the table, ``OFFLINE`` or 
``REALTIME``. There are some settings specific to each type. This 
differentiation will be called out below as options are explained.
+
+.. code-block:: none
+
+{
+  "tableName": "myPinotTable",
+  "tableType": "REALTIME"
+  "segmentsConfig": {},
+  "tableIndexConfig": {},
+  "tenants": {},
+  "routing": {},
+  "task": {},
+  "metadata": {}
+}
+
+Segments Config Section
+~~~
+
+The ``segmentsConfig`` section has information about configuring the following:
+
+* Segment Retention - with the ``retentionTimeUnit`` and 
``retentionTimeValue`` options.
+
+  * Allowed values:
+
+* ``retentionTimeUnit`` - ``DAYS``
+* ``retentionTimeValue`` - Positive integers
+
+* ``segmentPushFrequency`` - to indicate how frequently segments are uploaded.
+
+  * Allowed values - ``daily``, ``hourly``
+
+* ``segmentPushType`` - Indicates the type of push to the table.
+
+  * Allowed values - ``APPEND`` means new data will be pushed and appended to 
the current data in the table, all realtime tables *must* be ``APPEND``. 
``REFRESH`` will refresh the entire dataset contained within the table.
+
+* ``replication`` - Number of replicas of data in a table, used for offline 
tables only.
+
+  * Allowed values - Positive integers
+
+* ``replicasPerPartition`` - Number of of data in a table, used for offline 
tables only.
+
+  * Allowed values - Positive integers
+
+* ``schemaName`` - Name of the schema that's been uploaded to the controller
 
 Review comment:
   I think we should not document this. 
   
   We want schema name = raw table name. the schema name is for older tables 
that had different names before.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

[GitHub] [incubator-pinot] mcvsubbu commented on a change in pull request #3975: ReadTheDocs documentation for Table Configs, Monitoring, and Deployment

[GitBox]

mcvsubbu commented on a change in pull request #3975: ReadTheDocs documentation 
for Table Configs, Monitoring, and Deployment
URL: https://github.com/apache/incubator-pinot/pull/3975#discussion_r267891543
 
 

 ##
 File path: docs/tableconfig_schema.rst
 ##
 @@ -0,0 +1,204 @@
+..
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements.  See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership.  The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License.  You may obtain a copy of the License at
+..
+..   http://www.apache.org/licenses/LICENSE-2.0
+..
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied.  See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+..
+
+Table Config
+
+
+Table Config
+-
+
+Introduction
+
+
+Using tables is how Pinot serves and organizes data. There are many settings 
in the table config which will influence how Pinot operates. The first and most 
significant distinction is using an offline versus a realtime table.
+
+An offline table in Pinot is used to host data which might be periodically 
uploaded - daily, weekly, etc. A realtime table, however, is used to consume 
data from incoming data streams and serve this data in a near-realtime manner. 
This might also be referred to as nearline or just plain 'realtime'.
+
+In this section a sample table configuration will be shown and all sections 
will be explained and if applicable have appropriate sections linked to for 
further explanation of those corresponding Pinot features.
+
+Sample table config and descriptions
+
+
+A sample table config is shown below which has sub-sections collasped. The sub 
sections will be described individually in following sections.
+
+The ``tableName`` should only contain alpha-numeric characters, hyphens ('-'), 
or underscores ('_'). Though using a double-underscore ('__') is not allowed 
and reserved for other features within Pinot.
+
+The ``tableType`` will indicate the type of the table, ``OFFLINE`` or 
``REALTIME``. There are some settings specific to each type. This 
differentiation will be called out below as options are explained.
+
+.. code-block:: none
+
+{
+  "tableName": "myPinotTable",
+  "tableType": "REALTIME"
+  "segmentsConfig": {},
 
 Review comment:
   maybe useful to denote a collapsed section with a few dots here?
   
   "segmentsConfig" : {...}


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

[GitHub] [incubator-pinot] mcvsubbu commented on a change in pull request #3975: ReadTheDocs documentation for Table Configs, Monitoring, and Deployment

[GitBox]

mcvsubbu commented on a change in pull request #3975: ReadTheDocs documentation 
for Table Configs, Monitoring, and Deployment
URL: https://github.com/apache/incubator-pinot/pull/3975#discussion_r267888179
 
 

 ##
 File path: docs/tableconfig_schema.rst
 ##
 @@ -0,0 +1,204 @@
+..
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements.  See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership.  The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License.  You may obtain a copy of the License at
+..
+..   http://www.apache.org/licenses/LICENSE-2.0
+..
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied.  See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+..
+
+Table Config
+
+
+Table Config
+-
+
+Introduction
+
+
+Using tables is how Pinot serves and organizes data. There are many settings 
in the table config which will influence how Pinot operates. The first and most 
significant distinction is using an offline versus a realtime table.
+
+An offline table in Pinot is used to host data which might be periodically 
uploaded - daily, weekly, etc. A realtime table, however, is used to consume 
data from incoming data streams and serve this data in a near-realtime manner. 
This might also be referred to as nearline or just plain 'realtime'.
 
 Review comment:
   Can you qualify 'This' in the last sentence? Not sure what you mean there


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

[GitHub] [incubator-pinot] mcvsubbu commented on a change in pull request #3975: ReadTheDocs documentation for Table Configs, Monitoring, and Deployment

[GitBox]

mcvsubbu commented on a change in pull request #3975: ReadTheDocs documentation 
for Table Configs, Monitoring, and Deployment
URL: https://github.com/apache/incubator-pinot/pull/3975#discussion_r267887625
 
 

 ##
 File path: docs/tableconfig_schema.rst
 ##
 @@ -0,0 +1,204 @@
+..
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements.  See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership.  The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License.  You may obtain a copy of the License at
+..
+..   http://www.apache.org/licenses/LICENSE-2.0
+..
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied.  See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+..
+
+Table Config
 
 Review comment:
   We have two headings with the same name, and it appears strange.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

[GitHub] [incubator-pinot] mcvsubbu commented on a change in pull request #3975: ReadTheDocs documentation for Table Configs, Monitoring, and Deployment

[GitBox]

mcvsubbu commented on a change in pull request #3975: ReadTheDocs documentation 
for Table Configs, Monitoring, and Deployment
URL: https://github.com/apache/incubator-pinot/pull/3975#discussion_r267899010
 
 

 ##
 File path: docs/tableconfig_schema.rst
 ##
 @@ -0,0 +1,204 @@
+..
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements.  See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership.  The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License.  You may obtain a copy of the License at
+..
+..   http://www.apache.org/licenses/LICENSE-2.0
+..
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied.  See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+..
+
+Table Config
+
+
+Table Config
+-
+
+Introduction
+
+
+Using tables is how Pinot serves and organizes data. There are many settings 
in the table config which will influence how Pinot operates. The first and most 
significant distinction is using an offline versus a realtime table.
+
+An offline table in Pinot is used to host data which might be periodically 
uploaded - daily, weekly, etc. A realtime table, however, is used to consume 
data from incoming data streams and serve this data in a near-realtime manner. 
This might also be referred to as nearline or just plain 'realtime'.
+
+In this section a sample table configuration will be shown and all sections 
will be explained and if applicable have appropriate sections linked to for 
further explanation of those corresponding Pinot features.
+
+Sample table config and descriptions
+
+
+A sample table config is shown below which has sub-sections collasped. The sub 
sections will be described individually in following sections.
+
+The ``tableName`` should only contain alpha-numeric characters, hyphens ('-'), 
or underscores ('_'). Though using a double-underscore ('__') is not allowed 
and reserved for other features within Pinot.
+
+The ``tableType`` will indicate the type of the table, ``OFFLINE`` or 
``REALTIME``. There are some settings specific to each type. This 
differentiation will be called out below as options are explained.
+
+.. code-block:: none
+
+{
+  "tableName": "myPinotTable",
+  "tableType": "REALTIME"
+  "segmentsConfig": {},
+  "tableIndexConfig": {},
+  "tenants": {},
+  "routing": {},
+  "task": {},
+  "metadata": {}
+}
+
+Segments Config Section
+~~~
+
+The ``segmentsConfig`` section has information about configuring the following:
+
+* Segment Retention - with the ``retentionTimeUnit`` and 
``retentionTimeValue`` options.
+
+  * Allowed values:
+
+* ``retentionTimeUnit`` - ``DAYS``
+* ``retentionTimeValue`` - Positive integers
+
+* ``segmentPushFrequency`` - to indicate how frequently segments are uploaded.
+
+  * Allowed values - ``daily``, ``hourly``
+
+* ``segmentPushType`` - Indicates the type of push to the table.
+
+  * Allowed values - ``APPEND`` means new data will be pushed and appended to 
the current data in the table, all realtime tables *must* be ``APPEND``. 
``REFRESH`` will refresh the entire dataset contained within the table.
+
+* ``replication`` - Number of replicas of data in a table, used for offline 
tables only.
+
+  * Allowed values - Positive integers
+
+* ``replicasPerPartition`` - Number of of data in a table, used for offline 
tables only.
+
+  * Allowed values - Positive integers
+
+* ``schemaName`` - Name of the schema that's been uploaded to the controller
+
+  * Allowed values - String
+
+* Time column - using ``timeColumnName`` and ``timeType``, this must match 
what's configured in the preceeding schema
+
+  * Allowed values - String, this must match the ``timeFieldSpec`` section in 
the schema
+
+* Segment assignment strategy - Described more on the page `Customizing Pinot 
`_
+
+
+.. code-block:: none
+
+"segmentsConfig": {
+  "retentionTimeUnit": "DAYS",
+  "retentionTimeValue": "5",
+  "segmentPushFrequency": "daily",
+  "segmentPushType": "APPEND",
+  "replication": "3",
+  "replicasPerPartition": "3",
+  "schemaName": "myPinotSchmea",
+  "timeColumnName": "daysSinceEpoch",
+  "timeType": "DAYS",
+  "segmentAssignmentStrategy": "BalanceNumSegmentAssignmentStrategy"
+},
+
+Table Index Config Section
+~~
+
+The ``tableIndexConfig`` section has information about how to configure:
+
+* ``invertedIndexColumns`` - Indicates a list of real column names as 
specified in the schema to create 

[GitHub] [incubator-pinot] mcvsubbu commented on a change in pull request #3975: ReadTheDocs documentation for Table Configs, Monitoring, and Deployment

[GitBox]

mcvsubbu commented on a change in pull request #3975: ReadTheDocs documentation 
for Table Configs, Monitoring, and Deployment
URL: https://github.com/apache/incubator-pinot/pull/3975#discussion_r267890343
 
 

 ##
 File path: docs/tableconfig_schema.rst
 ##
 @@ -0,0 +1,204 @@
+..
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements.  See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership.  The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License.  You may obtain a copy of the License at
+..
+..   http://www.apache.org/licenses/LICENSE-2.0
+..
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied.  See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+..
+
+Table Config
+
+
+Table Config
+-
+
+Introduction
+
+
+Using tables is how Pinot serves and organizes data. There are many settings 
in the table config which will influence how Pinot operates. The first and most 
significant distinction is using an offline versus a realtime table.
+
+An offline table in Pinot is used to host data which might be periodically 
uploaded - daily, weekly, etc. A realtime table, however, is used to consume 
data from incoming data streams and serve this data in a near-realtime manner. 
This might also be referred to as nearline or just plain 'realtime'.
+
+In this section a sample table configuration will be shown and all sections 
will be explained and if applicable have appropriate sections linked to for 
further explanation of those corresponding Pinot features.
+
+Sample table config and descriptions
+
+
+A sample table config is shown below which has sub-sections collasped. The sub 
sections will be described individually in following sections.
+
+The ``tableName`` should only contain alpha-numeric characters, hyphens ('-'), 
or underscores ('_'). Though using a double-underscore ('__') is not allowed 
and reserved for other features within Pinot.
+
+The ``tableType`` will indicate the type of the table, ``OFFLINE`` or 
``REALTIME``. There are some settings specific to each type. This 
differentiation will be called out below as options are explained.
+
+.. code-block:: none
+
+{
+  "tableName": "myPinotTable",
+  "tableType": "REALTIME"
+  "segmentsConfig": {},
+  "tableIndexConfig": {},
+  "tenants": {},
+  "routing": {},
+  "task": {},
+  "metadata": {}
+}
+
+Segments Config Section
+~~~
+
+The ``segmentsConfig`` section has information about configuring the following:
+
+* Segment Retention - with the ``retentionTimeUnit`` and 
``retentionTimeValue`` options.
+
+  * Allowed values:
+
+* ``retentionTimeUnit`` - ``DAYS``
+* ``retentionTimeValue`` - Positive integers
 
 Review comment:
   useful to note that retention is meaningful only if table type is APPEND.  
Or, in the segment push type, call out that retention setting is ignored if the 
push type is not APPEND


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org