[
https://issues.apache.org/jira/browse/DRILL-4052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14997117#comment-14997117
]
John Omernik commented on DRILL-4052:
-------------------------------------
Awesome, I believe it does... for others looking to do the same, I created
using mapr volumes my spill location, and then used the exported variable
$DRILL_SPILLLOC per the HOCON docs. That said, other than it not throwing an
error when I started the drill bit, I am not sure how to validate that the bits
are each running happily with their own spill location. I tried running a
really big query with a join, but didn't see any files get created. Is there
way to validate in the config, or by running a test that this is working as I
intend? Once I validate, I will close this JIRA.
In drill-override.conf:
sort.external.spill.directories: [ ${?DRILL_SPILLLOC} ],
sort.external.spill.fs: "maprfs:///",
In drill.env.sh:
TOPOROOT="/data/default-rack/"
TOPO="${TOPOROOT}${HOSTNAME}"
NFSROOT="/mapr/mycluster"
SPILLLOC="/var/mapr/local/${HOSTNAME}/drillspill"
export DRILL_SPILLLOC="$SPILLLOC"
VOLNAME="mapr.${HOSTNAME}.local.drillspill"
if [ -d "${NFSROOT}${SPILLLOC}" ]; then
echo "Spill Location exists: ${SPILLLOC}"
else
echo "Need to create SPILL LOCATION: ${SPILLLOC}"
RUNCMD="maprcli volume create -name ${VOLNAME} -path ${SPILLLOC}
-rootdirperms 775 -user mapr:fc,a,dump,restore,m,d
dataadm:fc,a,d,m,restore,dump -minreplication 1 -replication 1 -topology
${TOPO} -mount 1"
echo "$RUNCMD"
$RUNCMD
fi
> Allow variables in drill-override.conf
> --------------------------------------
>
> Key: DRILL-4052
> URL: https://issues.apache.org/jira/browse/DRILL-4052
> Project: Apache Drill
> Issue Type: Improvement
> Components: Tools, Build & Test
> Affects Versions: 1.1.0
> Environment: All
> Reporter: John Omernik
> Labels: features
> Fix For: Future
>
>
> Basic variables, starting with "$HOSTNAME" if that simplified things would be
> a huge help to managing clusters of drill bits, especially in multi-tenant
> environments.
> (Note: Perhaps being able to use all ENV variables through a $ENV:ENVVARNAME
> would address the issue completely, giving ultimate flexibility).
> The use case/story I have is a drill cluster where I am running the
> individual drill bits in Marathon on Mesos. Since I am using MapR-FS
> (although this could work without MapR through Marathon downloading the
> config at run time) I point all drill bits to one config location. This
> allows me to make a change in one place, making for easy changes and
> configuration management.
> For the drill-env.sh, since that is shell script, I found I could easily put
> in items like specifying log locations that allowed me to use one shared log
> location for all drillbits, and then specifying that each logfile had the
> hostname of the drillbit, so I could keep all logs together and thus
> troubleshoot much easier.
> When it came to drill-override.conf, I wanted to specify spill directory
> locations. Specifically, I wanted to use local volumes in MapR-FS. It allows
> me to use a clustered filesystem, but have that data only exit on the node
> that the drill bit is running on. So, Scheme: maprfs works, but what to
> specify for the location? I didn't want to specify the same location,
> /tmp/spill as then it would be shared by drillbits, and not guaranteed to be
> local to the drill bit (giving up on performance). I wanted to create a
> volume that could be mounted /tmp/$hostname/drillspill I could create the
> volume in drill-env.sh, but I could not specify that in the drill-override
> without creating a separate override file for each node.
> Given scaling and multiple users/clusters, I really want to avoid that
> situation (a config for each drill bit), hence creating this JIRA for the
> ability to use variables in the drill-override. Happy to discuss other use
> cases as well.
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
