I try to run kafka as slider application
On Tue, Oct 28, 2014 at 7:01 PM, [email protected] <[email protected]> wrote: > I had the same problem > > This is my appConfig.json > > { > "schema": "http://example.org/specification/v2.0.0";, > "metadata": { > }, > "global": { > "application.def": > "hdfs://localhost:9000/user/siyuan/slider_kafka.zip", > "java_home": "/usr/lib/jvm/java-7-oracle/", > "package_list": "files/kafka_2.10-0.8.1.1.tgz", > > "site.global.app_user": "siyuan", > "site.global.app_root": "${AGENT_WORK_ROOT}/app/install/slider_kafka", > "site.global.pid_file": "${AGENT_WORK_ROOT}/app/run/component.pid", > > "site.global.additional_cp": "/usr/lib/hadoop/lib/*", > "site.global.xmx_val": "256m", > "site.global.xms_val": "128m" > }, > "components": { > "slider-appmaster": { > "jvm.heapsize": "256M" > } > } > } > > On Mon, Oct 27, 2014 at 8:06 AM, Jon Maron <[email protected]> wrote: > >> I think the problem is probably in your appConfig file (e.g. referencing >> an application package from the the local filesystem as opposed to HDFS). >> Can you attach that? >> >> — Jon >> >> On Oct 27, 2014, at 10:54 AM, Pushkar Raste <[email protected]> >> wrote: >> >> > Hi, >> > I am trying to to deploy memcached package using slider (0.40.0 >> version). I >> > keep getting "Wrong FS: >> > hdfs://localhost:9000/user/root/.slider/cluster/c100, expected: >> file:///" >> > error >> > >> > I saw that 'Rui Zhang' had similar issue, but she got around by >> > setting SLIDER_CLASSPATH_EXTRA. >> > Unfortunately that solution did not work for me. Here is my setup >> > 1. slider-client.xml >> > <?xml version="1.0"?> <?xml-stylesheet type="text/xsl" >> > href="configuration.xsl"?> <!-- Licensed to the Apache Software >> Foundation >> > (ASF) under one or more contributor license agreements. See the NOTICE >> file >> > distributed with this work for additional information regarding >> copyright >> > ownership. The ASF licenses this file to You under the Apache License: >> > Version 2.0 (the "License"); you may not use this file except in >> compliance >> > with the License. You may obtain a copy of the License at >> > http://www.apache.org/licenses/LICENSE-2.0 Unless required by >> applicable >> > law or agreed to in writing: software distributed under the License is >> > distributed on an "AS IS" BASIS: WITHOUT WARRANTIES OR CONDITIONS OF ANY >> > KIND: either express or implied. See the License for the specific >> language >> > governing permissions and limitations under the License. --> <!-- >> > Properties set here are picked up in the client. --> <configuration> >> > <property> <name>slider.client.resource.origin</name> >> > <value>conf/slider-client.xml</value> <description>This is just for >> > diagnostics</description> </property> <property> >> > <name>yarn.log-aggregation-enable</name> <value>true</value> </property> >> > <property> <name>slider.yarn.queue</name> <value>default</value> >> > <description>YARN queue for the Application Master</description> >> > </property> <!-- <property> <name>yarn.resourcemanager.address</name> >> > <value>master:8032</value> </property> <property> >> <name>fs.defaultFS</name> >> > <value>hdfs://master:9090</value> </property> <property> >> > <name>yarn.resourcemanager.principal</name> >> > <value>yarn/master@MINICLUSTER</value> >> > </property> <property> <name>slider.security.enabled</name> >> > <value>true</value> </property> <property> >> > <name>dfs.namenode.kerberos.principal</name> >> > <value>hdfs/master@MINICLUSTER</value> >> > </property> --> <property> <name>yarn.application.classpath</name> >> > >> <value>/usr/local/hadoop/etc/hadoop/*:/usr/local/hadoop/share/hadoop/common/*:/usr/local/hadoop/share/hadoop/common/lib/*:/usr/local/hadoop/share/hadoop/hdfs/*:/usr/local/hadoop/share/hadoop/hdfs/lib/*:/usr/local/hadoop/share/hadoop/yarn/*:/usr/local/hadoop/share/hadoop/yarn/lib/*:/usr/local/hadoop/share/hadoop/mapreduce/*:/usr/local/hadoop/share/hadoop/mapreduce/lib/*</value> >> > </property> <property> <name>slider.zookeeper.quorum</name> >> > <value>localhost:2181</value> </property> <property> >> > <name>yarn.resourcemanager.address</name> <value>localhost:8032</value> >> > </property> <property> >> <name>yarn.resourcemanager.scheduler.address</name> >> > <value>localhost:8030</value> </property> <property> >> > <name>fs.defaultFS</name> <value>hdfs://localhost:9000</value> >> </property> >> > </configuration> >> > >> > 2. Environment variables in my ~/.bashrc >> > export JAVA_HOME=/usr/lib/jvm/java-7-openjdk-amd64 export >> > HADOOP_INSTALL=/usr/local/hadoop export PATH=$PATH:$HADOOP_INSTALL/bin >> > export PATH=$PATH:$HADOOP_INSTALL/sbin export >> > HADOOP_MAPRED_HOME=$HADOOP_INSTALL export >> > HADOOP_COMMON_HOME=$HADOOP_INSTALL export >> HADOOP_HDFS_HOME=$HADOOP_INSTALL >> > export YARN_HOME=$HADOOP_INSTALL export >> > HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_INSTALL/lib/native export >> > HADOOP_OPTS="-Djava.library.path=$HADOOP_INSTALL/lib" export >> > HADOOP_CONF_DIR=$HADOOP_INSTALL/etc/hadoop export >> > >> SLIDER_CLASSPATH_EXTRA=$HADOOP_CONF_DIR/*:/usr/local/hadoop/etc/hadoop/*:/usr/local/hadoop/share/hadoop/common/*:/usr/local/hadoop/share/hadoop/common/lib/*:/usr/local/hadoop/share/hadoop/hdfs/*:/usr/local/hadoop/share/hadoop/hdfs/lib/*:/usr/local/hadoop/share/hadoop/yarn/*:/usr/local/hadoop/share/hadoop/yarn/lib/*:/usr/local/hadoop/share/hadoop/mapreduce/*:/usr/local/hadoop/share/hadoop/mapreduce/lib/* >> > export YARN_CONF_DIR=$HADOOP_INSTALL/etc/hadoop >> > >> > >> > >> > 3. Output when I run 'python slider.py version' slider_home = >> > "/usr/local/slider/slider-0.40" slider_jvm_opts = >> > "-Djava.net.preferIPv4Stack=true -Djava.awt.headless=true -Xmx256m >> > -Djava.confdir=/usr/local/slider/slider-0.40/conf" slider_classpath = >> > >> "/usr/local/slider/slider-0.40/lib/*:/usr/local/slider/slider-0.40/conf:/usr/local/hadoop/etc/hadoop/*:/usr/local/hadoop/etc/hadoop/*:/usr/local/hadoop/share/hadoop/common/*:/usr/local/hadoop/share/hadoop/common/lib/*:/usr/local/hadoop/share/hadoop/hdfs/*:/usr/local/hadoop/share/hadoop/hdfs/lib/*:/usr/local/hadoop/share/hadoop/yarn/*:/usr/local/hadoop/share/hadoop/yarn/lib/*:/usr/local/hadoop/share/hadoop/mapreduce/*:/usr/local/hadoop/share/hadoop/mapreduce/lib/*" >> > ready to exec : ['java', '-Djava.net.preferIPv4Stack=true', >> > '-Djava.awt.headless=true', '-Xmx256m', >> > '-Djava.confdir=/usr/local/slider/slider-0.40/conf', '-classpath', >> > >> '/usr/local/slider/slider-0.40/lib/*:/usr/local/slider/slider-0.40/conf:/usr/local/hadoop/etc/hadoop/*:/usr/local/hadoop/etc/hadoop/*:/usr/local/hadoop/share/hadoop/common/*:/usr/local/hadoop/share/hadoop/common/lib/*:/usr/local/hadoop/share/hadoop/hdfs/*:/usr/local/hadoop/share/hadoop/hdfs/lib/*:/usr/local/hadoop/share/hadoop/yarn/*:/usr/local/hadoop/share/hadoop/yarn/lib/*:/usr/local/hadoop/share/hadoop/mapreduce/*:/usr/local/hadoop/share/hadoop/mapreduce/lib/*', >> > 'org.apache.slider.Slider', 'version'] SLF4J: Class path contains >> multiple >> > SLF4J bindings. 2014-10-27 14:52:17,995 [main] INFO client.RMProxy - >> > Connecting to ResourceManager at localhost/127.0.0.1:8032 SLF4J: Found >> > binding in >> > >> [jar:file:/usr/local/slider/slider-0.40/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class] >> > 2014-10-27 14:52:18,121 [main] INFO client.SliderClient - Slider >> Core-0.40 >> > Built against commit# ddf7338759 on Java 1.7.0_67 by praste SLF4J: Found >> > binding in >> > >> [jar:file:/usr/local/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class] >> > 2014-10-27 14:52:18,125 [main] INFO client.SliderClient - Compiled >> against >> > Hadoop 2.4.0 SLF4J: See >> http://www.slf4j.org/codes.html#multiple_bindings >> > for an explanation. SLF4J: Actual binding is of type >> > [org.slf4j.impl.Log4jLoggerFactory] 2014-10-27 14:52:18,129 [main] INFO >> > client.SliderClient - Hadoop runtime version branch-2.4.0 with source >> > checksum 375b2832a6641759c6eaf6e3e998147 and build date >> 2014-03-31T08:31Z >> > 2014-10-27 14:52:18,133 [main] INFO util.ExitUtil - Exiting with status >> 0 >> >> >> -- >> CONFIDENTIALITY NOTICE >> NOTICE: This message is intended for the use of the individual or entity >> to >> which it is addressed and may contain information that is confidential, >> privileged and exempt from disclosure under applicable law. If the reader >> of this message is not the intended recipient, you are hereby notified >> that >> any printing, copying, dissemination, distribution, disclosure or >> forwarding of this communication is strictly prohibited. If you have >> received this communication in error, please contact the sender >> immediately >> and delete it from your system. Thank You. >> > >
