Repository: incubator-slider Updated Branches: refs/heads/develop 32329af24 -> 5960226df
SLIDER-890 Bundle Solr as an app-package include with the Slider distribution (Timothy Potter via gourksaha) Project: http://git-wip-us.apache.org/repos/asf/incubator-slider/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-slider/commit/5960226d Tree: http://git-wip-us.apache.org/repos/asf/incubator-slider/tree/5960226d Diff: http://git-wip-us.apache.org/repos/asf/incubator-slider/diff/5960226d Branch: refs/heads/develop Commit: 5960226dfef8c0981b591f1e9ccf8ff27345b873 Parents: 32329af Author: Gour Saha <[email protected]> Authored: Sat May 30 14:57:08 2015 -0700 Committer: Gour Saha <[email protected]> Committed: Sat May 30 14:57:08 2015 -0700 ---------------------------------------------------------------------- app-packages/solr/README.md | 110 +++++++++++++++++++ app-packages/solr/appConfig-default.json | 27 +++++ app-packages/solr/metainfo.xml | 68 ++++++++++++ .../solr/package/files/solr.tgz.REPLACE | 26 +++++ app-packages/solr/package/scripts/params.py | 42 +++++++ app-packages/solr/package/scripts/solr_node.py | 88 +++++++++++++++ app-packages/solr/resources-default.json | 16 +++ 7 files changed, 377 insertions(+) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/5960226d/app-packages/solr/README.md ---------------------------------------------------------------------- diff --git a/app-packages/solr/README.md b/app-packages/solr/README.md new file mode 100644 index 0000000..ef9caa5 --- /dev/null +++ b/app-packages/solr/README.md @@ -0,0 +1,110 @@ +<!--- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> + +Solr on YARN via Slider +======== + +Solr on YARN - Slider package for deploying SolrCloud to a YARN cluster. + +Getting Started +======== + +Follow the instructions for getting started with Slider: +http://slider.incubator.apache.org/docs/getting_started.html + +Be sure to add the `$SLIDER_HOME/bin` directory to your path. + +Also, make sure your `conf/slider-client.xml` file sets the ResourceManager address so you don't have to +include the `--manager` parameter with every slider command. + +``` + <property> + <name>yarn.resourcemanager.address</name> + <value>localhost:8032</value> + </property> +``` + +Throughout these instructions, `$PROJECT_HOME` refers to the directory where you cloned this project. + +**1) Start ZooKeeper 3.4.6+ on your local workstation** + +Running Solr with embedded ZooKeeper is not supported. Thus, you need to deploy at least one standalone ZooKeeper +process, but of course at least 3 servers are required to establish quorum for production environments. +Take note of the ZooKeeper connection string as you'll need to include it in the Solr deployment metadata +below (see step 5). + +**2) Download a Solr distribution archive (tgz)** + +Download the latest Solr distribution from: http://lucene.apache.org/solr/downloads.html + +Once downloaded, move the distribution archive to `$PROJECT_HOME/package/files/solr.tgz` + +The distribution archive must be named `solr.tgz` as the `metainfo.xml` file references this path. + +**3) Create the solr-on-yarn.zip deployment package** + +Create the Slider package using zip: + +``` +zip -r solr-on-yarn.zip metainfo.xml package/ +``` + +**4) Install the package on HDFS** + +``` +slider install-package --replacepkg --name solr --package $PROJECT_HOME/solr-on-yarn.zip +``` + +**5) Configure environment specific settings** + +Edit the `$PROJECT_HOME/appConfig-default.json`. At a minimum, you'll need to update the following settings +to match your environment: + +``` + "java_home": "/Library/Java/JavaVirtualMachines/jdk1.7.0_67.jdk/Contents/Home", + "site.global.app_root": "${AGENT_WORK_ROOT}/app/install/solr-5.2.0", + "site.global.zk_host": "localhost:2181", +``` + +Review the other settings in this file to verify they are correct for your environment. + +**6) Configure the number of Solr nodes to deploy** + +Edit `yarn.component.instances` in `resources-default.json` to set the number of Solr nodes to deploy across your cluster. + +**7) Deploy Solr on YARN** + +``` +slider create solr --template $PROJECT_HOME/appConfig-default.json \ + --resources $PROJECT_HOME/resources-default.json +``` + +where `$YARN_MANAGER` is the host and port of the YARN resource manager, such as localhost:8032. + +**8) Navigate to the YARN ResourceManager Web UI** + +Also, you can get the URL of each Solr instance by consulting the Slider registry +(so you can reach the Solr Web Admin UI) by doing: + +``` +slider registry --name solr --getexp servers +``` + +This will also show the ZooKeeper connection string used by each Solr (of course it should be the same for all nodes). + +NOTE: The registry command requires using Java 7 as there's a bug in Slider that prevents the registry command +from working correctly, see: https://issues.apache.org/jira/browse/SLIDER-878 http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/5960226d/app-packages/solr/appConfig-default.json ---------------------------------------------------------------------- diff --git a/app-packages/solr/appConfig-default.json b/app-packages/solr/appConfig-default.json new file mode 100644 index 0000000..d1ca947 --- /dev/null +++ b/app-packages/solr/appConfig-default.json @@ -0,0 +1,27 @@ +{ + "schema": "http://example.org/specification/v2.0.0";, + "metadata": { + }, + "global": { + "application.def": ".slider/package/solr/solr-on-yarn.zip", + "java_home": "/Library/Java/JavaVirtualMachines/jdk1.7.0_67.jdk/Contents/Home", + "site.global.app_root": "${AGENT_WORK_ROOT}/app/install/solr-5.2.0-SNAPSHOT", + "site.global.zk_host": "localhost:2181", + "site.global.solr_host": "${SOLR_HOST}", + "site.global.listen_port": "${SOLR.ALLOCATED_PORT}", + "site.global.xmx_val": "1g", + "site.global.xms_val": "1g", + "site.global.gc_tune": "-XX:NewRatio=3 -XX:SurvivorRatio=4 -XX:TargetSurvivorRatio=90 -XX:MaxTenuringThreshold=8 -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:ConcGCThreads=4 -XX:ParallelGCThreads=4 -XX:+CMSScavengeBeforeRemark -XX:PretenureSizeThreshold=64m -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=50 -XX:CMSMaxAbortablePrecleanTime=6000 -XX:+CMSParallelRemarkEnabled -XX:+ParallelRefProcEnabled -verbose:gc -XX:+PrintHeapAtGC -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintTenuringDistribution -XX:+PrintGCApplicationStoppedTime", + "site.global.zk_timeout": "15000", + "site.global.server_module": "--module=http", + "site.global.stop_key": "solrrocks", + "site.global.solr_opts": "" + }, + "components": { + "slider-appmaster": { + "jvm.heapsize": "512M" + }, + "SOLR": { + } + } +} http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/5960226d/app-packages/solr/metainfo.xml ---------------------------------------------------------------------- diff --git a/app-packages/solr/metainfo.xml b/app-packages/solr/metainfo.xml new file mode 100644 index 0000000..e723f94 --- /dev/null +++ b/app-packages/solr/metainfo.xml @@ -0,0 +1,68 @@ +<?xml version="1.0"?> +<!-- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> + +<metainfo> + <schemaVersion>2.0</schemaVersion> + <application> + <name>Solr</name> + <comment>Solr is an app that can run on YARN.</comment> + <version>1.0.0</version> + <exportedConfigs>None</exportedConfigs> + + <exportGroups> + <exportGroup> + <name>Servers</name> + <exports> + <export> + <name>host_port</name> + <value>http://${SOLR_HOST}:${site.global.listen_port}/</value> + </export> + <export> + <name>zkhost</name> + <value>${site.global.zk_host}</value> + </export> + </exports> + </exportGroup> + </exportGroups> + + <components> + <component> + <name>SOLR</name> + <category>SLAVE</category> + <compExports>Servers-host_port,Servers-zkhost</compExports> + <commandScript> + <script>scripts/solr_node.py</script> + <scriptType>PYTHON</scriptType> + </commandScript> + </component> + </components> + + <osSpecifics> + <osSpecific> + <osType>any</osType> + <packages> + <package> + <type>tarball</type> + <name>files/solr.tgz</name> + </package> + </packages> + </osSpecific> + </osSpecifics> + + </application> +</metainfo> http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/5960226d/app-packages/solr/package/files/solr.tgz.REPLACE ---------------------------------------------------------------------- diff --git a/app-packages/solr/package/files/solr.tgz.REPLACE b/app-packages/solr/package/files/solr.tgz.REPLACE new file mode 100644 index 0000000..5b881e8 --- /dev/null +++ b/app-packages/solr/package/files/solr.tgz.REPLACE @@ -0,0 +1,26 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + + + +Place the Solr distribution archive (.tgz) into this directory and then +update the site.global.app_root setting in appConfig-default.json, e.g. +if you put solr-5.2.0-SNAPSHOT.tgz into the files directory, then you +would set app_root to: + + +"site.global.app_root": "${AGENT_WORK_ROOT}/app/install/solr-5.2.0-SNAPSHOT" + + http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/5960226d/app-packages/solr/package/scripts/params.py ---------------------------------------------------------------------- diff --git a/app-packages/solr/package/scripts/params.py b/app-packages/solr/package/scripts/params.py new file mode 100644 index 0000000..d0a0854 --- /dev/null +++ b/app-packages/solr/package/scripts/params.py @@ -0,0 +1,42 @@ +#!/usr/bin/env python +""" +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. + +""" + +from resource_management import * + +# server configurations +config = Script.get_config() + +app_root = config['configurations']['global']['app_root'] +java64_home = config['hostLevelParams']['java_home'] +app_user = config['configurations']['global']['app_user'] +port = config['configurations']['global']['listen_port'] +zk_host = config['configurations']['global']['zk_host'] +zk_timeout = config['configurations']['global']['zk_timeout'] +gc_tune = config['configurations']['global']['gc_tune'] +xmx_val = config['configurations']['global']['xmx_val'] +xms_val = config['configurations']['global']['xms_val'] +if not xms_val: + xms_val = xmx_val +pid_file = config['configurations']['global']['pid_file'] +server_module = config['configurations']['global']['server_module'] +stop_port = int(port)-1000 +stop_key = config['configurations']['global']['stop_key'] +solr_opts = config['configurations']['global']['solr_opts'] +solr_host = config['configurations']['global']['solr_host'] \ No newline at end of file http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/5960226d/app-packages/solr/package/scripts/solr_node.py ---------------------------------------------------------------------- diff --git a/app-packages/solr/package/scripts/solr_node.py b/app-packages/solr/package/scripts/solr_node.py new file mode 100644 index 0000000..b384f1c --- /dev/null +++ b/app-packages/solr/package/scripts/solr_node.py @@ -0,0 +1,88 @@ +#!/usr/bin/env python +""" +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. + +""" + +import sys +from resource_management import * + +class Solr_Component(Script): + def install(self, env): + self.install_packages(env) + + def configure(self, env): + import params + env.set_params(params) + + def start(self, env): + import params + env.set_params(params) + self.configure(env) + start_solr_cmd = """{java64_home}/bin/java +-server +-Xss256k +-Xmx{xmx_val} +-Xms{xms_val} +{gc_tune} +{solr_opts} +-DzkClientTimeout={zk_timeout} +-DzkHost={zk_host} +-Dhost={solr_host} +-Djetty.port={port} +-DSTOP.PORT={stop_port} +-DSTOP.KEY={stop_key} +-Duser.timezone=UTC +-Dsolr.solr.home=\"{app_root}/server/solr\" +-Dsolr.install.dir=\"{app_root}\" +-Djetty.home=\"{app_root}/server\" +-Xloggc:\"{app_root}/server/logs/solr_gc.log\" +-jar start.jar {server_module}""" + + process_cmd = format(start_solr_cmd.replace("\n", " ")) + print("Starting Solr using command: "+process_cmd) + Execute(process_cmd, + logoutput=True, + wait_for_finish=False, + pid_file=params.pid_file, + poll_after = 10, + cwd=format("{app_root}/server") + ) + + + def stop(self, env): + import params + env.set_params(params) + stop_cmd = format("bin/solr stop -p {port} -k {stop_key}") + Execute(stop_cmd, + logoutput=True, + wait_for_finish=True, + cwd=format("{app_root}") + ) + + def status(self, env): + import params + env.set_params(params) + status_cmd = "bin/solr status" + Execute(status_cmd, + logoutput=True, + wait_for_finish=True, + cwd=format("{app_root}") + ) + +if __name__ == "__main__": + Solr_Component().execute() http://git-wip-us.apache.org/repos/asf/incubator-slider/blob/5960226d/app-packages/solr/resources-default.json ---------------------------------------------------------------------- diff --git a/app-packages/solr/resources-default.json b/app-packages/solr/resources-default.json new file mode 100644 index 0000000..6c8febe --- /dev/null +++ b/app-packages/solr/resources-default.json @@ -0,0 +1,16 @@ +{ + "schema" : "http://example.org/specification/v2.0.0";, + "metadata" : { + }, + "global" : { + }, + "components": { + "slider-appmaster": { + }, + "SOLR": { + "yarn.role.priority": "1", + "yarn.component.instances": "2", + "yarn.memory": "1024" + } + } +}
