[
https://issues.apache.org/jira/browse/DRILL-4591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Paul Rogers updated DRILL-4591:
-------------------------------
Description:
Today Drill provides the drill-override.conf file to set Drill properties, and
the drill-env.sh file to provide custom launch properties. Today, most users
seem to have a copy of DRILL_HOME per node, and thus they copy these two files
per-node.
The result is that the two files act as both the overall "site" configuration
(for all nodes) and the "per-node" configuration for that one node.
The approach works, but is awkward. Ideally, provide the option to have two
files: one for the site, one for the node.
drill-override.conf, drill-env.sh: For the site
drill-override.conf, drill-env.sh: For the particular node
The site-wide site-env.sh might, say, point to custom locations for log, conf,
pid or other file locations, or set custom JVM options.
Similarly, the per-node drill-node.conf might customize the per-node memory or
other resources if one node has a different amount of physical memory or other
resources.
The improvement becomes more important if a user employs NFS, MapR FS or YARN
to automatically deploy the site-wide files. In that case, the site files
cannot also act as per-node files.
Existing installs have only one each of drill-override.conf, drill-env.sh,
copied to each node and possibly modified. For compatibility, the existing
files become the site-wide files. When converting to a split-setup, copy the
site-wide files as before. But, rather than modifying the site-wide files
per-node, create a new per-load location, create new per-node files, and place
the per-node modifications in those files. The per-node files need not change
on following upgrades, just copy the site-wide files and that's it.
We assume that DRILL_CONF_DIR points to the site-wide files. It defaults to
$DRILL_HOME/conf or in /etc/drill/conf. To use node-specific files, the admin
must set up a new, per-node location and identify that location using a
proposed DRILL_NODE_CONF_DIR variable.
The config files should be optional: skip them if they do not exist (or, for
node files, skip them if DRILL_NODE_CONF_DIR is not set.)
When loading, per-node files are loaded after the site-wide files (so that
per-node values override site-wide values.)
The change should be fully backward compatible. The change will also work with
a YARN solution that "localizes" (copies to the local node) the site-wide
files.
was:
Today Drill provides the drill-override.conf file to set Drill properties, and
the drill-env.sh file to provide custom launch properties. Today, most users
seem to have a copy of DRILL_HOME per node, and thus they copy these two files
per-node.
The result is that the two files act as both the overall "site" configuration
(for all nodes) and the "per-node" configuration for that one node.
The approach works, but is awkward. Ideally, provide the option to have two
files: one for the site, one for the node.
drill-override.conf, site-env.sh: For the site
drill-node.conf, drill-env.sh: For the particular node
The improvement becomes more important if a user employs NFS, MapR FS or YARN
to automatically deploy the site-wide files. In that case, the site files
cannot also act as per-node files.
Note that, above, site-env.sh is new. The existing file is supposed to be
per-node, so the new file is site-wide.
By contrast, the drill-node.conf per-node configuration is new as the existing
file is supposed to be site-wide.
The site-wide site-env.sh might, say, point to custom locations for log, conf,
pid or other file locations, or set custom JVM options.
Similarly, the per-node drill-node.conf might customize the per-node memory or
other resources if one node has a different amount of physical memory or other
resources.
The existing files are identified with the DRILL_CONF_DIR environment variable.
Add a new DRILL_SITE_CONF_DIR variable to point to the site-wide files.
When loading, per-node files are loaded after the site-wide files (so that
per-node values override site-wide values.)
As described, the change should be fully backward compatible. The change will
also work with a YARN solution that "localizes" (copies to the local node) the
site-wide files.
> Extend config system with site, node property files
> ---------------------------------------------------
>
> Key: DRILL-4591
> URL: https://issues.apache.org/jira/browse/DRILL-4591
> Project: Apache Drill
> Issue Type: Improvement
> Reporter: Paul Rogers
> Priority: Minor
>
> Today Drill provides the drill-override.conf file to set Drill properties,
> and the drill-env.sh file to provide custom launch properties. Today, most
> users seem to have a copy of DRILL_HOME per node, and thus they copy these
> two files per-node.
> The result is that the two files act as both the overall "site" configuration
> (for all nodes) and the "per-node" configuration for that one node.
> The approach works, but is awkward. Ideally, provide the option to have two
> files: one for the site, one for the node.
> drill-override.conf, drill-env.sh: For the site
> drill-override.conf, drill-env.sh: For the particular node
> The site-wide site-env.sh might, say, point to custom locations for log,
> conf, pid or other file locations, or set custom JVM options.
> Similarly, the per-node drill-node.conf might customize the per-node memory
> or other resources if one node has a different amount of physical memory or
> other resources.
> The improvement becomes more important if a user employs NFS, MapR FS or YARN
> to automatically deploy the site-wide files. In that case, the site files
> cannot also act as per-node files.
> Existing installs have only one each of drill-override.conf, drill-env.sh,
> copied to each node and possibly modified. For compatibility, the existing
> files become the site-wide files. When converting to a split-setup, copy the
> site-wide files as before. But, rather than modifying the site-wide files
> per-node, create a new per-load location, create new per-node files, and
> place the per-node modifications in those files. The per-node files need not
> change on following upgrades, just copy the site-wide files and that's it.
> We assume that DRILL_CONF_DIR points to the site-wide files. It defaults to
> $DRILL_HOME/conf or in /etc/drill/conf. To use node-specific files, the admin
> must set up a new, per-node location and identify that location using a
> proposed DRILL_NODE_CONF_DIR variable.
> The config files should be optional: skip them if they do not exist (or, for
> node files, skip them if DRILL_NODE_CONF_DIR is not set.)
> When loading, per-node files are loaded after the site-wide files (so that
> per-node values override site-wide values.)
> The change should be fully backward compatible. The change will also work
> with a YARN solution that "localizes" (copies to the local node) the
> site-wide files.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
