mcvsubbu commented on a change in pull request #3661: Add documentation for the
new config format
URL: https://github.com/apache/incubator-pinot/pull/3661#discussion_r249908824
##########
File path: docs/table_configuration.rst
##########
@@ -0,0 +1,220 @@
+Table configuration format
+==========================
+
+Tables are defined in Pinot using a HOCON-based file format.
+Here is an example of a table configuration in Pinot:
+
+.. code-block:: javascript
+
+ table {
+ name: "myTable"
+ types: [OFFLINE]
+ replication: 3
+ retention: "30 DAYS"
+ schema {
+ dimensions {
+ myColumn {
+ dataType=INT
+ singleValue=true
+ }
+ }
+ }
+ }
+
+This defines a simple table that has a single dimension column called
``myColumn``.
+
+A sample table configuration that contains all the possible configuration
options can be found in the Pinot distribution (TODO).
+
+Useful HOCON features
+---------------------
+
+HOCON supports comments, file inclusion, variable expansion and inheritance,
which can be helpful to manage table configurations across multiple
+Pinot deployments.
+
+Here is an example of these features:
+
+``default-table.conf``
+
+.. code-block:: javascript
+
+ table {
+ replication: 3
+ retention: "30 DAYS"
+ }
+ config.kafkaHost="kafka-host-1234.us-west-4.cloudprovider.com:12345"
+
+``my-table.conf``
+
+.. code-block:: javascript
+
+ #include "default-table.conf"
+ table {
+ name: "myTable"
+ retention: "60 days" // Retention is overridden from the default 30 days
+ // Replication is inherited to be 3
+ schema {
+ // ...
+ }
+ "streamConfigs.stream.kafka.broker.list": "${config.kafkaHost}"
+ }
+
+Configuration profiles
+----------------------
+
+Configuration profiles can also be used to simplify configuration management
across different Pinot deployments. The Pinot
+table configuration deployment tools optionally support passing in a list of
configuration profiles, which are included at the top of the
+table definition file. For example, passing in the ``prod`` and
``prod-us-west-4`` profiles is equivalent to
+including ``profiles/prod.conf`` and ``profiles/prod-us-west-4.conf`` at the
top of the table configuration.
+
+Table definitions can also contain an optional ``profiles`` section which can
be used to control automated table deployments.
+
+``profiles/prod-us-west-4.conf``
+
+.. code-block:: javascript
+
+ config.kafkaHost="kafka-host-prod-1234.us-west-4.cloudprovider.com:12345"
+
+``my-table.conf``
+
+.. code-block:: javascript
+
+ profiles=[prod, prod-us-west-4, test, test-us-east-2]
+ table {
+ // ...
+ "streamConfigs.stream.kafka.broker.list": "${config.kafkaHost}"
+ }
+
+If the Pinot tools are used to deploy this table configuration with the
profiles ``prod`` and ``prod-us-west-4``, this configuration
+will get deployed to that particular Pinot deployment. However, the table
would be skipped if the configuration profiles ``prod``
+and ``prod-tokyo-1`` are passed, due to the ``prod-tokyo-1`` configuration
profile not being part of the ``profiles`` section.
+
+Profile conditionals
+--------------------
+
+Configuration values can be set based on which profiles are enabled. For
example:
+
+.. code-block:: javascript
+
+ table {
+ //...
+ replication___prod = 3
+ replication___testing = 1
+ }
+
+In the example above, the replication for the table will be 3 if the ``prod``
profile is enabled, 1 if the ``testing``
+profile is enabled, and unset otherwise.
+
+There is no conflict resolution between conflicting conditional values, so a
configuration containing simultaneous
+different values for profile conditionals is invalid.
Review comment:
So, the result will be that the ApplyTableConfig command will throw an
error? If so, please mention that.
Also, I suppose there is no implicit assumption that 'prod-us-west-4' is an
extension of 'prod' profile, right? In the previous example, if somehow 'prod'
and 'testing' profiles are enabled simultaneously it would result in the same
error?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
