I’m trying to do some experimentation and set the
drill.exec.sort.external.spill.directories value. Since this option appears as
a ‘boot’ option ( https://drill.apache.org/docs/start-up-options/ ) , I believe
the right way is to set this in drill-override.conf on each node.
I tried doing this via the following:
drill.exec: {
cluster-id: "se1-drillbits",
zk.connect: "10.10.15.10:5181,10.10.15.11:5181,10.10.15.12:5181",
sys.store.provider.zk.blobroot: "maprfs:///user/mapr/profiles",
sort.external.spill.directories: [ "/var/mapr/$hostname/drillspill" ],
sort.external.spill.fs: "maprfs:///",
impersonation: {
enabled: true,
max_chained_user_hops: 3
}
}
I also tried setting via:
sort: {
purge.threshold : 100,
external: {
batch.size : 4000,
spill: {
batch.size : 4000,
group.size : 100,
threshold : 200,
directories : [ "/var/mapr/$hostname/drillspill" ],
fs : “maprfs:///"
}
}
},
But then looking at the sys.boot table after restarting the drill bits, I still
see the default values:
0: jdbc:drill:> select * from sys.boot where name like '%spill%';
+------+------+------+--------+---------+------------+----------+-----------+
| name | kind | type | status | num_val | string_val | bool_val | float_val |
+------+------+------+--------+---------+------------+----------+-----------+
| drill.exec.sort.external.spill.batch.size | LONG | BOOT | BOOT | 4000 | null
| null | null |
| drill.exec.sort.external.spill.directories | STRING | BOOT | BOOT | null | [
#
jar:file:/opt/mapr/drill/drill-1.1.0/jars/drill-java-exec-1.1.0.jar!/drill-module.conf:
145
"/tmp/drill/spill"
] | null | null |
| drill.exec.sort.external.spill.fs | STRING | BOOT | BOOT | null | "file:///"
| null | null |
| drill.exec.sort.external.spill.group.size | LONG | BOOT | BOOT | 40000 | null
| null | null |
| drill.exec.sort.external.spill.threshold | LONG | BOOT | BOOT | 40000 | null
| null | null |
+------+------+------+--------+---------+------------+----------+-----------+
Note that I’ve tried removing the shell ‘$hostname’ variable (in case it causes
issues), no dice.
What’s the right way to set these values?
Andy Pernsteiner
Manager, Field Enablement
ph: 206.228.0737
www.mapr.com
Now Available - Free Hadoop On-Demand Training
