jfim opened a new pull request #3506: Add configuration merging tool URL: https://github.com/apache/incubator-pinot/pull/3506 Add a configuration merging tool to aid in migration between old-style configurations and the new style configuration. Configurations can be migrated by backing up tables using the config backup tool, then merging them. The merging is only necessary if configurations for multiple configuration profiles are desired. This happens when multiple Pinot clusters are managed using configuration profiles and a single source of truth for configurations is desired. For example, given multiple Pinot clusters in multiple data centers, one might have two staging clusters and two production clusters. Since the configuration backup tool only backs up the configuration for a single Pinot cluster, without merging there would be one set of configurations for each cluster. With config merging, one would end up with identical keys merged across configurations (eg. table.name is the same across all clusters) and different keys across configurations being merged (eg. table.replication___staging=1 and table.replication___prod=3). An optimization process is applied to minimize the number of generated keys in the case where there are heterogenous combinations of clusters with multiple profiles applied to them. For example, the combination table.replication___staging=1 and table.replication___prod=3 is preferred over table.replication___staging-us-west1=1, table.replication___staging-us-east1=1, table.replication___prod-us-west1=3, table.replication___us-east1=3 even though both combinations expand to the same data. Furthermore, a substitution process is applied to replace values by configuration values read from the existing profiles. For example, if a profile exists for "prod-us-east1" that contains "kafka.broker=kafka-us-east1:2181", configuration keys that have the value "kafka-us-east1:2181" and the "prod-us-east1" configuration profile enabled will get replaced with "${kafka.broker}". In the case where multiple replacements are possible, a minimization process is applied to generate the smallest set of generated keys, similar to the configuration optimization for keys that differ across clusters. Thus, the migration process is: 1. Write the per profile configuration files (eg. profile/staging.conf, profile/staging-us-east1.conf", ...) 2. Back up configuration files using the table config backup tool in directories that contain the profile tags to be applied for that cluster. For example, data from the staging-us-west1 cluster would be backed up to configBackup/staging/staging-us-west1 so that configuration files in that directory are understood to have the staging and staging-us-west1 configuration profiles. 3. Invoke the configuration merger with the configBackup directory as input. 4. Review the generated merged files for stylistic preferences, as desired.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
