Thanks for trying out these experiments. >I thought this would have >broken FGS, but apparently it didn't (I started my nodes with min >allocation CPU = 1 and FGS still worked for me... not sure about that, >would love feedback there)
I presume you are referring to "yarn.scheduler.minimum-allocation-vcores". The behavior of this variable is that when an app requests for a container with less than the "min allocation vcores", then RM ends up allocating a container with "min allocation vcores". YARN's default value for this is 1 - i.e. if an app wants a container with < 1 cpu vcores, YARN allocates a container with 1 CPU vcore. This also implicitly means that YARN cannot allocate a container unless at least "min allocation vcores" are available on a NM. If a NM has less than "miin allocation vcores", that NM will not get any containers allocated to it, unless existing containers finish (in case of plain YARN and with Myriad CGS) or unless Mesos offers more resources (in case of FGS). In that sense, FGS/CGS do not interfere with "min allocation vcores". Rather FGS and CGS just influence the NM capacities. Hope this helps. Thanks, Santosh On Tue, Nov 24, 2015 at 9:25 AM, John Omernik <[email protected]> wrote: > Since a vast majority of my posts are me struggling with something or > breaking something, I thought I'd take the time to craft a story of Myriad > success. > > Over the weekend, I took the time to run Elastic Search on Yarn using the > es-yarn package that elastic search has. This is a beta package, and > struggles with some components such as "Storage" for the data. > > With MapR I've been able to create a place and some scripts to manage the > data issue. This combined with ES2's dynamic node allocation, and the > Myriad's fine grain scaling made it so I have a powerful way to elasticize > elastic search! > > Basically, I did this through some simple steps. The first was to take the > "include" file (esinc.sh) from the distribution and add some items to it > and tell the es-yarn framework to use this instead of the included file. > This allowed me to set some parameters at start time with environmental > variables. > > Simple steps. > > Download the es-yarn packages and the elaticsearch zip. > > (optional: I added https://github.com/royrusso/elasticsearch-HQ as a > plugin, basically I unzipped the ES2 zip, ran the plugin script, and the > rezipped the package) > > In a location in MapR I copied the esinc.sh to a location out of the zip > (in the root of say my working directory, > /mapr/mycluster/mesos/dev/es-yarn/). (see below) Then I created a script > that was how I started and scaled up the clusters. (see below). I do have > some notes on how it works in the script. > > For basics this was awesome. I didn't use FGS at first, because of a bug in > the es-yarn > https://github.com/elastic/elasticsearch-hadoop/issues/598 if the min > allocation cpu is 0 there is a divide by 0. I thought this would have > broken FGS, but apparently it didn't (I started my nodes with min > allocation CPU = 1 and FGS still worked for me... not sure about that, > would love feedback there) > > When I "start" a cluster, I initialize it and eventually my cluster is > running with the specified es nodes. If I want to scale up, I run the > script again with the same cluster name. Those nodes are added to the ES > cluster, and now I am running two yarn applications. I can only scale down > by applications. So, if I want to scale down, then I have to kill an > application, if I started 3 ES nodes with that application, I'll scale down > by 3 nodes. Thus, there is an argument to always scale by one ES node, > especially if you are using larger nodes (I wonder what sort of Application > manager overhead I'd get on that). > > Either way, this worked really well. > > The cool thing was with FGS though. I had one node running in a "small" > config and 4 running with zero (even though I set the min allocation size > to 1, this still started and seemed to work). When I submitted the request > for ES nodes, they got put into mesos tasks for each container and it > worked great. When I scaled the application down it too worked great. > This provided me huge flexibility in scaling up and down without reserving > resources for elastic search clusters. Kudos to Myriad!!!!! > > > My only comment to the Myriad crew would be a wiki article explaining FGS a > little bit. I just "did it" and it worked, but a little bit more on how it > worked, the challenges, gotchas etc. Would be outstanding. > > Thanks to everyone's hard work on Myriad, this project has lots of power > that it can give Admins/users, and I just wanted to share a win here after > all of my "how do I ... " posts > > John > > > > > startcluster.sh > > #!/bin/bash > > > > > > #Perhaps these should be parameters or at least check to see if they are > set in the ENV, if not use these? > > > > ES_JAR="elasticsearch-yarn-2.2.0-beta1.jar" # Jarfile of es-yarn to use > > ES_NFSMOUNT="/mapr/mycluster" # Root of NFS Mount in MapR > > ES_BASELOC="/mesos/dev/es-yarn" # Location of this script, the > esinc.sh, and the basis for all things es-yarn > > ES_DATALOC="/data" # This is the data location, > which is $ES_BASELOC$ES_DATALOC it creates directories under that for each > clustername, then each node > > ES_PROVISION_LOC="/tmp/esprovision/" # Where the jar file for > es-yarn, and the elastic search zip file is > > > > # These are your node names. It needs these to do it's unicast discovery. > This may need to be updated (perhaps I can curl the Mesos Master to get the > node list) > > ES_UNICAST_HOSTS="node1.mydomain.com,node2.mydomain.com,node3.mydomain.com > " > > > > > # The elastic search version you are running (the es-yarn jar uses this to > pick the right zip file, make sure you have not changed the name of the ES > Zip file) > > ES_VER="2.0.0" > > > > > # Cluster settings. Name and Port > > > ES_CLUSTERNAME="MYESCLUSTER" > > ES_TRANSPORT_PORT="59300-59400" > > ES_HTTP_PORT="59200-59300" > > > > # For this run, the number of nodes to add in "this" application. (Each > submission is a yarn application) and the node size > > NUM_NODES="3" > > NODEMEM="2048" > > NODECPU="2" > > > > > # Don't change anything else here: > > > > ES_INCLUDE="${ES_NFSMOUNT}${ES_BASELOC}/esinc.sh" > > ES_YARN_JAR="${ES_NFSMOUNT}${ES_BASELOC}/${ES_JAR}" > > > > > > ES_ENV="env.ES_CLUSTERNAME=$ES_CLUSTERNAME > env.ES_TRANSPORT_PORT=$ES_TRANSPORT_PORT env.ES_HTTP_PORT=$ES_HTTP_PORT > env.ES_UNICAST_HOSTS=$ES_UNICAST_HOSTS env.ES_NFSMOUNT=$ES_NFSMOUNT > env.ES_BASELOC=$ES_BASELOC env.ES_DATALOC=$ES_DATALOC" > > > > RUNCMD="hadoop jar $ES_YARN_JAR -start containers=$NUM_NODES > hdfs.upload.dir=$ES_PROVISION_LOC es.version=$ES_VER container.mem=$NODEMEM > container.vcores=$NODECPU env.ES_INCLUDE=$ES_INCLUDE $ES_ENV" > > > > > > echo "Starting Cluster.... " > > > > $RUNCMD > > > > echo "Application Submitted, check logs for http endpoints (perhaps we > should monitor the logs until we get one and then display it as > http://IP:port/_plugin/hq"; > > > > > Append to your customized esinc.sh: > > ########################################################### > > echo "Start Customization of esinc for yarn/myriad/mapr" > > > > export ES_NODE_NAME="$CONTAINER_ID" #A Unique value > > export ES_NETWORK_HOST="_non_loopback_" > > export > > ES_PATH_LOGS="${ES_NFSMOUNT}${ES_BASELOC}${ES_DATALOC}/$ES_CLUSTERNAME/$ES_NODE_NAME/logs" > > export > > ES_PATH_DATA="${ES_NFSMOUNT}${ES_BASELOC}${ES_DATALOC}/$ES_CLUSTERNAME/$ES_NODE_NAME/data" > > > > > > # These are set here, eventually they should be set by the framework and > these just grab the value from above at the framework level > > > > #Set log level if needed > > ES_DEBUG_OPTS="-Des.logger.level=DEBUG" > > > > # Cluster Name and Node Name - Think through Nodename here more... Perhaps > we could generate a name that if it failed and tried to restart would have > the same node name... > > ES_NAME_OPTS="-Des.cluster.name <http://des.cluster.name/>=$ES_CLUSTERNAME > - > Des.node.name <http://des.node.name/>=$ES_NODE_NAME" > > > > # Path to Log Locations and Datafile locations Todo: Perhaps add labels or > create volumes with mapr rest api > > ES_PATH_OPTS="-Des.path.logs=$ES_PATH_LOGS -Des.path.data=$ES_PATH_DATA" > > > > #Networking options. Need to set the other nodes for discovery, the network > host so it's listening on the right interfaces, and the ports used for > transport and http > > ES_NETWORK_OPTS="-Des.discovery.zen.ping.unicast.hosts=$ES_UNICAST_HOSTS > -Des.network.host=$ES_NETWORK_HOST > -Des.transport.tcp.port=$ES_TRANSPORT_PORT -Des.http.port=$ES_HTTP_PORT" > > > > > > export ES_JAVA_OPTS="$ES_DEBUG_OPTS $ES_NAME_OPTS $ES_PATH_OPTS > $ES_NETWORK_OPTS" > > > > #This is just for debugging > > env > > > ######################################### >
