Hi All, The discussion with John and Charles about the drillbit scripts reminded me to get your thoughts on another change we’re working on.
Today, drillbit.sh has the --config option so you can put your config files in a location separate from DRILL_HOME: $DRILL_HOME/bin/drillbit.sh —config /some/path/to/conf start This is handy, but it only holds config files (drill-env.sh, drill-override.sh). If you have custom code, it still must go into $DRILL_HOME/jars/3rdparty. This presents two challenges: * On upgrades, you have to grab your files from the old $DRILL_HOME and copy them into the new one. * With YARN, we have to create an archive of your entire $DRILL_HOME just to grab your “site” files. So, we propose to extend the —config option to include code as well as config. We call this “complete” set of files the “site” directory (using Hadoop terminology.) (See DRILL-4591.) This way: * Upgrade is easy, throw away the old $DRILL_HOME and extract the Drill archive to create the new one. * With YARN, we upload the “stock” drill archive plus your (much smaller) site files. * We can more easily support multiple Drill “clusters” (each with its own site files, including assigned ports.) With YARN, you only need one copy of the DRILL_HOME and site directory; YARN copies (“localizes”) the files to all your worker nodes. Without YARN, you have to do the copy, probably with your favorite system admin tool. So, the question is this: is the site directory a help for those of you that won’t be using YARN? Or, does everyone just copy site files from one DRILL_HOME to the next on upgrade, then push the merged directory to all your worker nodes? Thoughts? Thanks, - Paul
