[
https://issues.apache.org/jira/browse/DRILL-4591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Paul Rogers updated DRILL-4591:
-------------------------------
Description:
Today Drill provides the drill-override.conf file to set Drill properties, and
the drill-env.sh file to provide custom launch properties. Today, most users
seem to have a copy of DRILL_HOME per node, and thus they copy these two files
per-node. The result is that the two files act as both the overall "site"
configuration (for all nodes) and the "per-node" configuration for that one
node.
In addition, some distributions of Drill (such as MapR), modify the "user"
config files with settings for that distribution. Now, the same files hold
settings for the distribution, site and node.
The approach works, but is awkward. Ideally, provide the option to have three
sets of files: for the distribution, site, and node.
The proposal is to extend configuration to provide additional levels:
* Drill defaults (drill default and module conf files, code in drill-config.sh)
* Distribution settings (special JVM settings, say)
* Site settings (standard log or spill file locations)
* Node settings
* Launch settings (environment variables, -Dname=value options)
The improvement becomes more important if a user employs NFS, MapR FS or YARN
to automatically deploy the site-wide files. In that case, the site files
cannot also act as per-node files.
The improvement also simplifies upgrades. Today, users must copy customizations
from and old to a new install. With the revision, Drill files are complely
separated from user files, making upgrades (of software) easier.
For backward compatibility, the site and node directories are optional and
ignored if the environment variables are not set. The site and node config
files should be optional: skip them if they do not exist.
was:
Today Drill provides the drill-override.conf file to set Drill properties, and
the drill-env.sh file to provide custom launch properties. Today, most users
seem to have a copy of DRILL_HOME per node, and thus they copy these two files
per-node. The result is that the two files act as both the overall "site"
configuration (for all nodes) and the "per-node" configuration for that one
node.
In addition, some distributions of Drill (such as MapR), modify the "user"
config files with settings for that distribution. Now, the same files hold
settings for the distribution, site and node.
The approach works, but is awkward. Ideally, provide the option to have three
sets of files: for the distribution, site, and node.
Let's assume each set resides in its own directory. (Merging the three sets
into a single directory would simply shift the sync problems from files to
directories.)
* DRILL_HOME/conf: Distribution files
* DRILL_SITE_DIR/conf: Site-wide files
* DRILL_NODE_DIR/conf: Node-specific files
Each directory might contain its own drill-override.conf, drill-env.sh files.
(The idea extends to site jar files as well by adding a jars directory under
DRILL_SITE_DIR.)
Configuration now provides additional levels:
* Drill defaults (drill default and module conf files, code in drill-config.sh)
* Distribution settings (special JVM settings, say)
* Site settings (standard log or spill file locations)
* Node settings
* Launch settings (environment variables, -Dname=value options)
The improvement becomes more important if a user employs NFS, MapR FS or YARN
to automatically deploy the site-wide files. In that case, the site files
cannot also act as per-node files.
The improvement also simplifies upgrades. Today, users must copy customizations
from and old to a new install. With the new system, Drill files are complely
separated from user files, making upgrades (of software) trivial.
Note that the current version of Drill does allow users to put config files in
/etc/drill/conf, but that location replaces $DRILL_HOME/conf; the user must
still start with the Distribution-specific files, and must merge any new
distribution changes in each new release.
For backward compatibility, the site and node directories are optional and
ignored if the environment variables are not set. The site and node config
files should be optional: skip them if they do not exist (or, for node files,
skip them if DRILL_NODE_CONF_DIR is not set.)
> Extend config system with distrib, site, node property files
> ------------------------------------------------------------
>
> Key: DRILL-4591
> URL: https://issues.apache.org/jira/browse/DRILL-4591
> Project: Apache Drill
> Issue Type: Improvement
> Reporter: Paul Rogers
> Attachments: Drill-on-YARNDirectoryStructures.pdf
>
>
> Today Drill provides the drill-override.conf file to set Drill properties,
> and the drill-env.sh file to provide custom launch properties. Today, most
> users seem to have a copy of DRILL_HOME per node, and thus they copy these
> two files per-node. The result is that the two files act as both the overall
> "site" configuration (for all nodes) and the "per-node" configuration for
> that one node.
> In addition, some distributions of Drill (such as MapR), modify the "user"
> config files with settings for that distribution. Now, the same files hold
> settings for the distribution, site and node.
> The approach works, but is awkward. Ideally, provide the option to have three
> sets of files: for the distribution, site, and node.
> The proposal is to extend configuration to provide additional levels:
> * Drill defaults (drill default and module conf files, code in
> drill-config.sh)
> * Distribution settings (special JVM settings, say)
> * Site settings (standard log or spill file locations)
> * Node settings
> * Launch settings (environment variables, -Dname=value options)
> The improvement becomes more important if a user employs NFS, MapR FS or YARN
> to automatically deploy the site-wide files. In that case, the site files
> cannot also act as per-node files.
> The improvement also simplifies upgrades. Today, users must copy
> customizations from and old to a new install. With the revision, Drill files
> are complely separated from user files, making upgrades (of software) easier.
> For backward compatibility, the site and node directories are optional and
> ignored if the environment variables are not set. The site and node config
> files should be optional: skip them if they do not exist.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
