[jira] [Updated] (DRILL-4591) Extend config system with distrib, site, node property files
[ https://issues.apache.org/jira/browse/DRILL-4591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Paul Rogers updated DRILL-4591: --- Issue Type: Sub-task (was: Improvement) Parent: DRILL-1170 > Extend config system with distrib, site, node property files > > > Key: DRILL-4591 > URL: https://issues.apache.org/jira/browse/DRILL-4591 > Project: Apache Drill > Issue Type: Sub-task >Reporter: Paul Rogers >Assignee: Paul Rogers > Attachments: Drill-on-YARNDirectoryStructures.pdf > > > Today Drill provides the drill-override.conf file to set Drill properties, > and the drill-env.sh file to provide custom launch properties. Today, most > users seem to have a copy of DRILL_HOME per node, and thus they copy these > two files per-node. The result is that the two files act as both the overall > "site" configuration (for all nodes) and the "per-node" configuration for > that one node. > In addition, some distributions of Drill (such as MapR), modify the "user" > config files with settings for that distribution. Now, the same files hold > settings for the distribution, site and node. > The approach works, but is awkward. Ideally, provide the option to have three > sets of files: for the distribution, site, and node. > The proposal is to extend configuration to provide additional levels: > * Drill defaults (drill default and module conf files, code in > drill-config.sh) > * Distribution settings (special JVM settings, say) > * Site settings (standard log or spill file locations) > * Node settings > * Launch settings (environment variables, -Dname=value options) > The improvement becomes more important if a user employs NFS, MapR FS or YARN > to automatically deploy the site-wide files. In that case, the site files > cannot also act as per-node files. > The improvement also simplifies upgrades. Today, users must copy > customizations from and old to a new install. With the revision, Drill files > are complely separated from user files, making upgrades (of software) easier. > For backward compatibility, the site and node directories are optional and > ignored if the environment variables are not set. The site and node config > files should be optional: skip them if they do not exist. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (DRILL-4591) Extend config system with distrib, site, node property files
[ https://issues.apache.org/jira/browse/DRILL-4591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Paul Rogers updated DRILL-4591: --- Attachment: Drill-on-YARNDirectoryStructures.pdf Draft design proposal. Proposal explains suggested Drill changes in the context of the Drill-on-YARN project. > Extend config system with distrib, site, node property files > > > Key: DRILL-4591 > URL: https://issues.apache.org/jira/browse/DRILL-4591 > Project: Apache Drill > Issue Type: Improvement >Reporter: Paul Rogers > Attachments: Drill-on-YARNDirectoryStructures.pdf > > > Today Drill provides the drill-override.conf file to set Drill properties, > and the drill-env.sh file to provide custom launch properties. Today, most > users seem to have a copy of DRILL_HOME per node, and thus they copy these > two files per-node. The result is that the two files act as both the overall > "site" configuration (for all nodes) and the "per-node" configuration for > that one node. > In addition, some distributions of Drill (such as MapR), modify the "user" > config files with settings for that distribution. Now, the same files hold > settings for the distribution, site and node. > The approach works, but is awkward. Ideally, provide the option to have three > sets of files: for the distribution, site, and node. > Let's assume each set resides in its own directory. (Merging the three sets > into a single directory would simply shift the sync problems from files to > directories.) > * DRILL_HOME/conf: Distribution files > * DRILL_SITE_DIR/conf: Site-wide files > * DRILL_NODE_DIR/conf: Node-specific files > Each directory might contain its own drill-override.conf, drill-env.sh files. > (The idea extends to site jar files as well by adding a jars directory under > DRILL_SITE_DIR.) > Configuration now provides additional levels: > * Drill defaults (drill default and module conf files, code in > drill-config.sh) > * Distribution settings (special JVM settings, say) > * Site settings (standard log or spill file locations) > * Node settings > * Launch settings (environment variables, -Dname=value options) > The improvement becomes more important if a user employs NFS, MapR FS or YARN > to automatically deploy the site-wide files. In that case, the site files > cannot also act as per-node files. > The improvement also simplifies upgrades. Today, users must copy > customizations from and old to a new install. With the new system, Drill > files are complely separated from user files, making upgrades (of software) > trivial. > Note that the current version of Drill does allow users to put config files > in /etc/drill/conf, but that location replaces $DRILL_HOME/conf; the user > must still start with the Distribution-specific files, and must merge any new > distribution changes in each new release. > For backward compatibility, the site and node directories are optional and > ignored if the environment variables are not set. The site and node config > files should be optional: skip them if they do not exist (or, for node files, > skip them if DRILL_NODE_CONF_DIR is not set.) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (DRILL-4591) Extend config system with distrib, site, node property files
[ https://issues.apache.org/jira/browse/DRILL-4591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Paul Rogers updated DRILL-4591: --- Priority: Major (was: Minor) > Extend config system with distrib, site, node property files > > > Key: DRILL-4591 > URL: https://issues.apache.org/jira/browse/DRILL-4591 > Project: Apache Drill > Issue Type: Improvement >Reporter: Paul Rogers > > Today Drill provides the drill-override.conf file to set Drill properties, > and the drill-env.sh file to provide custom launch properties. Today, most > users seem to have a copy of DRILL_HOME per node, and thus they copy these > two files per-node. The result is that the two files act as both the overall > "site" configuration (for all nodes) and the "per-node" configuration for > that one node. > In addition, some distributions of Drill (such as MapR), modify the "user" > config files with settings for that distribution. Now, the same files hold > settings for the distribution, site and node. > The approach works, but is awkward. Ideally, provide the option to have three > sets of files: for the distribution, site, and node. > Let's assume each set resides in its own directory. (Merging the three sets > into a single directory would simply shift the sync problems from files to > directories.) > * DRILL_HOME/conf: Distribution files > * DRILL_SITE_DIR/conf: Site-wide files > * DRILL_NODE_DIR/conf: Node-specific files > Each directory might contain its own drill-override.conf, drill-env.sh files. > (The idea extends to site jar files as well by adding a jars directory under > DRILL_SITE_DIR.) > Configuration now provides additional levels: > * Drill defaults (drill default and module conf files, code in > drill-config.sh) > * Distribution settings (special JVM settings, say) > * Site settings (standard log or spill file locations) > * Node settings > * Launch settings (environment variables, -Dname=value options) > The improvement becomes more important if a user employs NFS, MapR FS or YARN > to automatically deploy the site-wide files. In that case, the site files > cannot also act as per-node files. > The improvement also simplifies upgrades. Today, users must copy > customizations from and old to a new install. With the new system, Drill > files are complely separated from user files, making upgrades (of software) > trivial. > Note that the current version of Drill does allow users to put config files > in /etc/drill/conf, but that location replaces $DRILL_HOME/conf; the user > must still start with the Distribution-specific files, and must merge any new > distribution changes in each new release. > For backward compatibility, the site and node directories are optional and > ignored if the environment variables are not set. The site and node config > files should be optional: skip them if they do not exist (or, for node files, > skip them if DRILL_NODE_CONF_DIR is not set.) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (DRILL-4591) Extend config system with distrib, site, node property files
[ https://issues.apache.org/jira/browse/DRILL-4591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Paul Rogers updated DRILL-4591: --- Description: Today Drill provides the drill-override.conf file to set Drill properties, and the drill-env.sh file to provide custom launch properties. Today, most users seem to have a copy of DRILL_HOME per node, and thus they copy these two files per-node. The result is that the two files act as both the overall "site" configuration (for all nodes) and the "per-node" configuration for that one node. In addition, some distributions of Drill (such as MapR), modify the "user" config files with settings for that distribution. Now, the same files hold settings for the distribution, site and node. The approach works, but is awkward. Ideally, provide the option to have three sets of files: for the distribution, site, and node. Let's assume each set resides in its own directory. (Merging the three sets into a single directory would simply shift the sync problems from files to directories.) * DRILL_HOME/conf: Distribution files * DRILL_SITE_DIR/conf: Site-wide files * DRILL_NODE_DIR/conf: Node-specific files Each directory might contain its own drill-override.conf, drill-env.sh files. (The idea extends to site jar files as well by adding a jars directory under DRILL_SITE_DIR.) Configuration now provides additional levels: * Drill defaults (drill default and module conf files, code in drill-config.sh) * Distribution settings (special JVM settings, say) * Site settings (standard log or spill file locations) * Node settings * Launch settings (environment variables, -Dname=value options) The improvement becomes more important if a user employs NFS, MapR FS or YARN to automatically deploy the site-wide files. In that case, the site files cannot also act as per-node files. The improvement also simplifies upgrades. Today, users must copy customizations from and old to a new install. With the new system, Drill files are complely separated from user files, making upgrades (of software) trivial. Note that the current version of Drill does allow users to put config files in /etc/drill/conf, but that location replaces $DRILL_HOME/conf; the user must still start with the Distribution-specific files, and must merge any new distribution changes in each new release. For backward compatibility, the site and node directories are optional and ignored if the environment variables are not set. The site and node config files should be optional: skip them if they do not exist (or, for node files, skip them if DRILL_NODE_CONF_DIR is not set.) was: Today Drill provides the drill-override.conf file to set Drill properties, and the drill-env.sh file to provide custom launch properties. Today, most users seem to have a copy of DRILL_HOME per node, and thus they copy these two files per-node. The result is that the two files act as both the overall "site" configuration (for all nodes) and the "per-node" configuration for that one node. The approach works, but is awkward. Ideally, provide the option to have two files: one for the site, one for the node. drill-override.conf, drill-env.sh: For the site drill-override.conf, drill-env.sh: For the particular node The site-wide site-env.sh might, say, point to custom locations for log, conf, pid or other file locations, or set custom JVM options. Similarly, the per-node drill-node.conf might customize the per-node memory or other resources if one node has a different amount of physical memory or other resources. The improvement becomes more important if a user employs NFS, MapR FS or YARN to automatically deploy the site-wide files. In that case, the site files cannot also act as per-node files. Existing installs have only one each of drill-override.conf, drill-env.sh, copied to each node and possibly modified. For compatibility, the existing files become the site-wide files. When converting to a split-setup, copy the site-wide files as before. But, rather than modifying the site-wide files per-node, create a new per-load location, create new per-node files, and place the per-node modifications in those files. The per-node files need not change on following upgrades, just copy the site-wide files and that's it. We assume that DRILL_CONF_DIR points to the site-wide files. It defaults to $DRILL_HOME/conf or in /etc/drill/conf. To use node-specific files, the admin must set up a new, per-node location and identify that location using a proposed DRILL_NODE_CONF_DIR variable. The config files should be optional: skip them if they do not exist (or, for node files, skip them if DRILL_NODE_CONF_DIR is not set.) When loading, per-node files are loaded after the site-wide files (so that per-node values override site-wide values.) The change should be fully backward compatible. The change will also work with a YARN solution that "localizes"
[jira] [Updated] (DRILL-4591) Extend config system with distrib, site, node property files
[ https://issues.apache.org/jira/browse/DRILL-4591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Paul Rogers updated DRILL-4591: --- Summary: Extend config system with distrib, site, node property files (was: Extend config system with site, node property files) > Extend config system with distrib, site, node property files > > > Key: DRILL-4591 > URL: https://issues.apache.org/jira/browse/DRILL-4591 > Project: Apache Drill > Issue Type: Improvement >Reporter: Paul Rogers >Priority: Minor > > Today Drill provides the drill-override.conf file to set Drill properties, > and the drill-env.sh file to provide custom launch properties. Today, most > users seem to have a copy of DRILL_HOME per node, and thus they copy these > two files per-node. > The result is that the two files act as both the overall "site" configuration > (for all nodes) and the "per-node" configuration for that one node. > The approach works, but is awkward. Ideally, provide the option to have two > files: one for the site, one for the node. > drill-override.conf, drill-env.sh: For the site > drill-override.conf, drill-env.sh: For the particular node > The site-wide site-env.sh might, say, point to custom locations for log, > conf, pid or other file locations, or set custom JVM options. > Similarly, the per-node drill-node.conf might customize the per-node memory > or other resources if one node has a different amount of physical memory or > other resources. > The improvement becomes more important if a user employs NFS, MapR FS or YARN > to automatically deploy the site-wide files. In that case, the site files > cannot also act as per-node files. > Existing installs have only one each of drill-override.conf, drill-env.sh, > copied to each node and possibly modified. For compatibility, the existing > files become the site-wide files. When converting to a split-setup, copy the > site-wide files as before. But, rather than modifying the site-wide files > per-node, create a new per-load location, create new per-node files, and > place the per-node modifications in those files. The per-node files need not > change on following upgrades, just copy the site-wide files and that's it. > We assume that DRILL_CONF_DIR points to the site-wide files. It defaults to > $DRILL_HOME/conf or in /etc/drill/conf. To use node-specific files, the admin > must set up a new, per-node location and identify that location using a > proposed DRILL_NODE_CONF_DIR variable. > The config files should be optional: skip them if they do not exist (or, for > node files, skip them if DRILL_NODE_CONF_DIR is not set.) > When loading, per-node files are loaded after the site-wide files (so that > per-node values override site-wide values.) > The change should be fully backward compatible. The change will also work > with a YARN solution that "localizes" (copies to the local node) the > site-wide files. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
