Since a vast majority of my posts are me struggling with something or breaking something, I thought I'd take the time to craft a story of Myriad success.
Over the weekend, I took the time to run Elastic Search on Yarn using the es-yarn package that elastic search has. This is a beta package, and struggles with some components such as "Storage" for the data. With MapR I've been able to create a place and some scripts to manage the data issue. This combined with ES2's dynamic node allocation, and the Myriad's fine grain scaling made it so I have a powerful way to elasticize elastic search! Basically, I did this through some simple steps. The first was to take the "include" file (esinc.sh) from the distribution and add some items to it and tell the es-yarn framework to use this instead of the included file. This allowed me to set some parameters at start time with environmental variables. Simple steps. Download the es-yarn packages and the elaticsearch zip. (optional: I added https://github.com/royrusso/elasticsearch-HQ as a plugin, basically I unzipped the ES2 zip, ran the plugin script, and the rezipped the package) In a location in MapR I copied the esinc.sh to a location out of the zip (in the root of say my working directory, /mapr/mycluster/mesos/dev/es-yarn/). (see below) Then I created a script that was how I started and scaled up the clusters. (see below). I do have some notes on how it works in the script. For basics this was awesome. I didn't use FGS at first, because of a bug in the es-yarn https://github.com/elastic/elasticsearch-hadoop/issues/598 if the min allocation cpu is 0 there is a divide by 0. I thought this would have broken FGS, but apparently it didn't (I started my nodes with min allocation CPU = 1 and FGS still worked for me... not sure about that, would love feedback there) When I "start" a cluster, I initialize it and eventually my cluster is running with the specified es nodes. If I want to scale up, I run the script again with the same cluster name. Those nodes are added to the ES cluster, and now I am running two yarn applications. I can only scale down by applications. So, if I want to scale down, then I have to kill an application, if I started 3 ES nodes with that application, I'll scale down by 3 nodes. Thus, there is an argument to always scale by one ES node, especially if you are using larger nodes (I wonder what sort of Application manager overhead I'd get on that). Either way, this worked really well. The cool thing was with FGS though. I had one node running in a "small" config and 4 running with zero (even though I set the min allocation size to 1, this still started and seemed to work). When I submitted the request for ES nodes, they got put into mesos tasks for each container and it worked great. When I scaled the application down it too worked great. This provided me huge flexibility in scaling up and down without reserving resources for elastic search clusters. Kudos to Myriad!!!!! My only comment to the Myriad crew would be a wiki article explaining FGS a little bit. I just "did it" and it worked, but a little bit more on how it worked, the challenges, gotchas etc. Would be outstanding. Thanks to everyone's hard work on Myriad, this project has lots of power that it can give Admins/users, and I just wanted to share a win here after all of my "how do I ... " posts John startcluster.sh #!/bin/bash #Perhaps these should be parameters or at least check to see if they are set in the ENV, if not use these? ES_JAR="elasticsearch-yarn-2.2.0-beta1.jar" # Jarfile of es-yarn to use ES_NFSMOUNT="/mapr/mycluster" # Root of NFS Mount in MapR ES_BASELOC="/mesos/dev/es-yarn" # Location of this script, the esinc.sh, and the basis for all things es-yarn ES_DATALOC="/data" # This is the data location, which is $ES_BASELOC$ES_DATALOC it creates directories under that for each clustername, then each node ES_PROVISION_LOC="/tmp/esprovision/" # Where the jar file for es-yarn, and the elastic search zip file is # These are your node names. It needs these to do it's unicast discovery. This may need to be updated (perhaps I can curl the Mesos Master to get the node list) ES_UNICAST_HOSTS="node1.mydomain.com,node2.mydomain.com,node3.mydomain.com" # The elastic search version you are running (the es-yarn jar uses this to pick the right zip file, make sure you have not changed the name of the ES Zip file) ES_VER="2.0.0" # Cluster settings. Name and Port ES_CLUSTERNAME="MYESCLUSTER" ES_TRANSPORT_PORT="59300-59400" ES_HTTP_PORT="59200-59300" # For this run, the number of nodes to add in "this" application. (Each submission is a yarn application) and the node size NUM_NODES="3" NODEMEM="2048" NODECPU="2" # Don't change anything else here: ES_INCLUDE="${ES_NFSMOUNT}${ES_BASELOC}/esinc.sh" ES_YARN_JAR="${ES_NFSMOUNT}${ES_BASELOC}/${ES_JAR}" ES_ENV="env.ES_CLUSTERNAME=$ES_CLUSTERNAME env.ES_TRANSPORT_PORT=$ES_TRANSPORT_PORT env.ES_HTTP_PORT=$ES_HTTP_PORT env.ES_UNICAST_HOSTS=$ES_UNICAST_HOSTS env.ES_NFSMOUNT=$ES_NFSMOUNT env.ES_BASELOC=$ES_BASELOC env.ES_DATALOC=$ES_DATALOC" RUNCMD="hadoop jar $ES_YARN_JAR -start containers=$NUM_NODES hdfs.upload.dir=$ES_PROVISION_LOC es.version=$ES_VER container.mem=$NODEMEM container.vcores=$NODECPU env.ES_INCLUDE=$ES_INCLUDE $ES_ENV" echo "Starting Cluster.... " $RUNCMD echo "Application Submitted, check logs for http endpoints (perhaps we should monitor the logs until we get one and then display it as http://IP:port/_plugin/hq"; Append to your customized esinc.sh: ########################################################### echo "Start Customization of esinc for yarn/myriad/mapr" export ES_NODE_NAME="$CONTAINER_ID" #A Unique value export ES_NETWORK_HOST="_non_loopback_" export ES_PATH_LOGS="${ES_NFSMOUNT}${ES_BASELOC}${ES_DATALOC}/$ES_CLUSTERNAME/$ES_NODE_NAME/logs" export ES_PATH_DATA="${ES_NFSMOUNT}${ES_BASELOC}${ES_DATALOC}/$ES_CLUSTERNAME/$ES_NODE_NAME/data" # These are set here, eventually they should be set by the framework and these just grab the value from above at the framework level #Set log level if needed ES_DEBUG_OPTS="-Des.logger.level=DEBUG" # Cluster Name and Node Name - Think through Nodename here more... Perhaps we could generate a name that if it failed and tried to restart would have the same node name... ES_NAME_OPTS="-Des.cluster.name <http://des.cluster.name/>=$ES_CLUSTERNAME - Des.node.name <http://des.node.name/>=$ES_NODE_NAME" # Path to Log Locations and Datafile locations Todo: Perhaps add labels or create volumes with mapr rest api ES_PATH_OPTS="-Des.path.logs=$ES_PATH_LOGS -Des.path.data=$ES_PATH_DATA" #Networking options. Need to set the other nodes for discovery, the network host so it's listening on the right interfaces, and the ports used for transport and http ES_NETWORK_OPTS="-Des.discovery.zen.ping.unicast.hosts=$ES_UNICAST_HOSTS -Des.network.host=$ES_NETWORK_HOST -Des.transport.tcp.port=$ES_TRANSPORT_PORT -Des.http.port=$ES_HTTP_PORT" export ES_JAVA_OPTS="$ES_DEBUG_OPTS $ES_NAME_OPTS $ES_PATH_OPTS $ES_NETWORK_OPTS" #This is just for debugging env #########################################
