mcvsubbu commented on a change in pull request #3661: Add documentation for the
new config format
URL: https://github.com/apache/incubator-pinot/pull/3661#discussion_r249903387
##########
File path: docs/table_configuration.rst
##########
@@ -0,0 +1,220 @@
+Table configuration format
+==========================
+
+Tables are defined in Pinot using a HOCON-based file format.
+Here is an example of a table configuration in Pinot:
+
+.. code-block:: javascript
+
+ table {
+ name: "myTable"
+ types: [OFFLINE]
+ replication: 3
+ retention: "30 DAYS"
+ schema {
+ dimensions {
+ myColumn {
+ dataType=INT
+ singleValue=true
+ }
+ }
+ }
+ }
+
+This defines a simple table that has a single dimension column called
``myColumn``.
+
+A sample table configuration that contains all the possible configuration
options can be found in the Pinot distribution (TODO).
+
+Useful HOCON features
+---------------------
+
+HOCON supports comments, file inclusion, variable expansion and inheritance,
which can be helpful to manage table configurations across multiple
+Pinot deployments.
+
+Here is an example of these features:
+
+``default-table.conf``
+
+.. code-block:: javascript
+
+ table {
+ replication: 3
+ retention: "30 DAYS"
+ }
+ config.kafkaHost="kafka-host-1234.us-west-4.cloudprovider.com:12345"
+
+``my-table.conf``
+
+.. code-block:: javascript
+
+ #include "default-table.conf"
+ table {
+ name: "myTable"
+ retention: "60 days" // Retention is overridden from the default 30 days
+ // Replication is inherited to be 3
+ schema {
+ // ...
+ }
+ "streamConfigs.stream.kafka.broker.list": "${config.kafkaHost}"
+ }
+
+Configuration profiles
+----------------------
+
+Configuration profiles can also be used to simplify configuration management
across different Pinot deployments. The Pinot
+table configuration deployment tools optionally support passing in a list of
configuration profiles, which are included at the top of the
+table definition file. For example, passing in the ``prod`` and
``prod-us-west-4`` profiles is equivalent to
+including ``profiles/prod.conf`` and ``profiles/prod-us-west-4.conf`` at the
top of the table configuration.
+
+Table definitions can also contain an optional ``profiles`` section which can
be used to control automated table deployments.
+
+``profiles/prod-us-west-4.conf``
+
+.. code-block:: javascript
+
+ config.kafkaHost="kafka-host-prod-1234.us-west-4.cloudprovider.com:12345"
+
+``my-table.conf``
+
+.. code-block:: javascript
+
+ profiles=[prod, prod-us-west-4, test, test-us-east-2]
+ table {
+ // ...
+ "streamConfigs.stream.kafka.broker.list": "${config.kafkaHost}"
+ }
+
+If the Pinot tools are used to deploy this table configuration with the
profiles ``prod`` and ``prod-us-west-4``, this configuration
+will get deployed to that particular Pinot deployment. However, the table
would be skipped if the configuration profiles ``prod``
+and ``prod-tokyo-1`` are passed, due to the ``prod-tokyo-1`` configuration
profile not being part of the ``profiles`` section.
+
Review comment:
will be good to add an explicit note that _all_ profiles in a table
configuration must be included in the Pinot deployment command in order for a
table configuration to be deployed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
