Great to hear!!!
Thank you John.
Yuliya
From: John Omernik <[email protected]>
To: [email protected]
Sent: Tuesday, November 24, 2015 9:25 AM
Subject: A Myriad Story
Since a vast majority of my posts are me struggling with something or
breaking something, I thought I'd take the time to craft a story of Myriad
success.
Over the weekend, I took the time to run Elastic Search on Yarn using the
es-yarn package that elastic search has. This is a beta package, and
struggles with some components such as "Storage" for the data.
With MapR I've been able to create a place and some scripts to manage the
data issue. This combined with ES2's dynamic node allocation, and the
Myriad's fine grain scaling made it so I have a powerful way to elasticize
elastic search!
Basically, I did this through some simple steps. The first was to take the
"include" file (esinc.sh) from the distribution and add some items to it
and tell the es-yarn framework to use this instead of the included file.
This allowed me to set some parameters at start time with environmental
variables.
Simple steps.
Download the es-yarn packages and the elaticsearch zip.
(optional: I added https://github.com/royrusso/elasticsearch-HQ as a
plugin, basically I unzipped the ES2 zip, ran the plugin script, and the
rezipped the package)
In a location in MapR I copied the esinc.sh to a location out of the zip
(in the root of say my working directory,
/mapr/mycluster/mesos/dev/es-yarn/). (see below) Then I created a script
that was how I started and scaled up the clusters. (see below). I do have
some notes on how it works in the script.
For basics this was awesome. I didn't use FGS at first, because of a bug in
the es-yarn
https://github.com/elastic/elasticsearch-hadoop/issues/598 if the min
allocation cpu is 0 there is a divide by 0. I thought this would have
broken FGS, but apparently it didn't (I started my nodes with min
allocation CPU = 1 and FGS still worked for me... not sure about that,
would love feedback there)
When I "start" a cluster, I initialize it and eventually my cluster is
running with the specified es nodes. If I want to scale up, I run the
script again with the same cluster name. Those nodes are added to the ES
cluster, and now I am running two yarn applications. I can only scale down
by applications. So, if I want to scale down, then I have to kill an
application, if I started 3 ES nodes with that application, I'll scale down
by 3 nodes. Thus, there is an argument to always scale by one ES node,
especially if you are using larger nodes (I wonder what sort of Application
manager overhead I'd get on that).
Either way, this worked really well.
The cool thing was with FGS though. I had one node running in a "small"
config and 4 running with zero (even though I set the min allocation size
to 1, this still started and seemed to work). When I submitted the request
for ES nodes, they got put into mesos tasks for each container and it
worked great. When I scaled the application down it too worked great.
This provided me huge flexibility in scaling up and down without reserving
resources for elastic search clusters. Kudos to Myriad!!!!!
My only comment to the Myriad crew would be a wiki article explaining FGS a
little bit. I just "did it" and it worked, but a little bit more on how it
worked, the challenges, gotchas etc. Would be outstanding.
Thanks to everyone's hard work on Myriad, this project has lots of power
that it can give Admins/users, and I just wanted to share a win here after
all of my "how do I ... " posts
John
startcluster.sh
#!/bin/bash
#Perhaps these should be parameters or at least check to see if they are
set in the ENV, if not use these?
ES_JAR="elasticsearch-yarn-2.2.0-beta1.jar" # Jarfile of es-yarn to use
ES_NFSMOUNT="/mapr/mycluster" # Root of NFS Mount in MapR
ES_BASELOC="/mesos/dev/es-yarn" # Location of this script, the
esinc.sh, and the basis for all things es-yarn
ES_DATALOC="/data" # This is the data location,
which is $ES_BASELOC$ES_DATALOC it creates directories under that for each
clustername, then each node
ES_PROVISION_LOC="/tmp/esprovision/" # Where the jar file for
es-yarn, and the elastic search zip file is
# These are your node names. It needs these to do it's unicast discovery.
This may need to be updated (perhaps I can curl the Mesos Master to get the
node list)
ES_UNICAST_HOSTS="node1.mydomain.com,node2.mydomain.com,node3.mydomain.com"
# The elastic search version you are running (the es-yarn jar uses this to
pick the right zip file, make sure you have not changed the name of the ES
Zip file)
ES_VER="2.0.0"
# Cluster settings. Name and Port
ES_CLUSTERNAME="MYESCLUSTER"
ES_TRANSPORT_PORT="59300-59400"
ES_HTTP_PORT="59200-59300"
# For this run, the number of nodes to add in "this" application. (Each
submission is a yarn application) and the node size
NUM_NODES="3"
NODEMEM="2048"
NODECPU="2"
# Don't change anything else here:
ES_INCLUDE="${ES_NFSMOUNT}${ES_BASELOC}/esinc.sh"
ES_YARN_JAR="${ES_NFSMOUNT}${ES_BASELOC}/${ES_JAR}"
ES_ENV="env.ES_CLUSTERNAME=$ES_CLUSTERNAME
env.ES_TRANSPORT_PORT=$ES_TRANSPORT_PORT env.ES_HTTP_PORT=$ES_HTTP_PORT
env.ES_UNICAST_HOSTS=$ES_UNICAST_HOSTS env.ES_NFSMOUNT=$ES_NFSMOUNT
env.ES_BASELOC=$ES_BASELOC env.ES_DATALOC=$ES_DATALOC"
RUNCMD="hadoop jar $ES_YARN_JAR -start containers=$NUM_NODES
hdfs.upload.dir=$ES_PROVISION_LOC es.version=$ES_VER container.mem=$NODEMEM
container.vcores=$NODECPU env.ES_INCLUDE=$ES_INCLUDE $ES_ENV"
echo "Starting Cluster.... "
$RUNCMD
echo "Application Submitted, check logs for http endpoints (perhaps we
should monitor the logs until we get one and then display it as
http://IP:port/_plugin/hq";
Append to your customized esinc.sh:
###########################################################
echo "Start Customization of esinc for yarn/myriad/mapr"
export ES_NODE_NAME="$CONTAINER_ID" #A Unique value
export ES_NETWORK_HOST="_non_loopback_"
export
ES_PATH_LOGS="${ES_NFSMOUNT}${ES_BASELOC}${ES_DATALOC}/$ES_CLUSTERNAME/$ES_NODE_NAME/logs"
export
ES_PATH_DATA="${ES_NFSMOUNT}${ES_BASELOC}${ES_DATALOC}/$ES_CLUSTERNAME/$ES_NODE_NAME/data"
# These are set here, eventually they should be set by the framework and
these just grab the value from above at the framework level
#Set log level if needed
ES_DEBUG_OPTS="-Des.logger.level=DEBUG"
# Cluster Name and Node Name - Think through Nodename here more... Perhaps
we could generate a name that if it failed and tried to restart would have
the same node name...
ES_NAME_OPTS="-Des.cluster.name <http://des.cluster.name/>=$ES_CLUSTERNAME -
Des.node.name <http://des.node.name/>=$ES_NODE_NAME"
# Path to Log Locations and Datafile locations Todo: Perhaps add labels or
create volumes with mapr rest api
ES_PATH_OPTS="-Des.path.logs=$ES_PATH_LOGS -Des.path.data=$ES_PATH_DATA"
#Networking options. Need to set the other nodes for discovery, the network
host so it's listening on the right interfaces, and the ports used for
transport and http
ES_NETWORK_OPTS="-Des.discovery.zen.ping.unicast.hosts=$ES_UNICAST_HOSTS
-Des.network.host=$ES_NETWORK_HOST
-Des.transport.tcp.port=$ES_TRANSPORT_PORT -Des.http.port=$ES_HTTP_PORT"
export ES_JAVA_OPTS="$ES_DEBUG_OPTS $ES_NAME_OPTS $ES_PATH_OPTS
$ES_NETWORK_OPTS"
#This is just for debugging
env
#########################################
