Opps, it seems that the hadoop-hdfs-2.3.0.jar lost some dependencies. It's a hadoop-specific exception.
* Could you add the jar file that contains the missing class? (not quite sure it's in hadoop-common or hadoop-client) * Another worth trying is to add the dependency from pom file, let maven take care of it. Assume you are using hello-samza similar structure. There are two steps: ** add dependency in pom.xml of samza-job-package ** add <include>org.apache.hadoop:hadoop-hdfs</include> in samza-job-package/src/main/assembly/src.xml Then mvn clean package. Thank you. Cheers, Fang, Yan [email protected] +1 (206) 849-4108 On Mon, Aug 11, 2014 at 5:12 PM, Telles Nobrega <[email protected]> wrote: > Same thing happens. > > On 11 Aug 2014, at 21:05, Yan Fang <[email protected]> wrote: > > > Cool, we are almost there. Could you remove > > > > <property> > > <name>fs.hdfs.impl</name> > > <value>org.apache.hadoop.hdfs. > > DistributedFileSystem</value> > > <description>The FileSystem for hdfs: uris.</description> > > </property> > > > > To see how it works? > > > > > > Fang, Yan > > [email protected] > > +1 (206) 849-4108 > > > > > > On Mon, Aug 11, 2014 at 5:03 PM, Telles Nobrega <[email protected] > > > > wrote: > > > >> You may forget this last email, I was really stupid and put the files > in a > >> different folder. Now it could find the file but it’s not there yet… > >> another error came up > >> > >> Exception in thread "main" java.util.ServiceConfigurationError: > >> org.apache.hadoop.fs.FileSystem: Provider > >> org.apache.hadoop.hdfs.DistributedFileSystem could not be instantiated > >> at java.util.ServiceLoader.fail(ServiceLoader.java:224) > >> at java.util.ServiceLoader.access$100(ServiceLoader.java:181) > >> at > >> java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:377) > >> at java.util.ServiceLoader$1.next(ServiceLoader.java:445) > >> at > >> org.apache.hadoop.fs.FileSystem.loadFileSystems(FileSystem.java:2400) > >> at > >> org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2411) > >> at > >> org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2428) > >> at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:88) > >> at > >> org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2467) > >> at > org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2449) > >> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:367) > >> at org.apache.hadoop.fs.Path.getFileSystem(Path.java:287) > >> at > >> > org.apache.samza.job.yarn.ClientHelper.submitApplication(ClientHelper.scala:111) > >> at org.apache.samza.job.yarn.YarnJob.submit(YarnJob.scala:55) > >> at org.apache.samza.job.yarn.YarnJob.submit(YarnJob.scala:48) > >> at org.apache.samza.job.JobRunner.run(JobRunner.scala:62) > >> at org.apache.samza.job.JobRunner$.main(JobRunner.scala:37) > >> at org.apache.samza.job.JobRunner.main(JobRunner.scala) > >> Caused by: java.lang.NoClassDefFoundError: > >> org/apache/hadoop/conf/Configuration$DeprecationDelta > >> at > >> > org.apache.hadoop.hdfs.HdfsConfiguration.addDeprecatedKeys(HdfsConfiguration.java:66) > >> at > >> > org.apache.hadoop.hdfs.HdfsConfiguration.<clinit>(HdfsConfiguration.java:31) > >> at > >> > org.apache.hadoop.hdfs.DistributedFileSystem.<clinit>(DistributedFileSystem.java:106) > >> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native > >> Method) > >> at > >> > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) > >> at > >> > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > >> at > java.lang.reflect.Constructor.newInstance(Constructor.java:526) > >> at java.lang.Class.newInstance(Class.java:374) > >> at > >> java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:373) > >> ... 15 more > >> Caused by: java.lang.ClassNotFoundException: > >> org.apache.hadoop.conf.Configuration$DeprecationDelta > >> at java.net.URLClassLoader$1.run(URLClassLoader.java:366) > >> at java.net.URLClassLoader$1.run(URLClassLoader.java:355) > >> at java.security.AccessController.doPrivileged(Native Method) > >> at java.net.URLClassLoader.findClass(URLClassLoader.java:354) > >> at java.lang.ClassLoader.loadClass(ClassLoader.java:425) > >> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) > >> at java.lang.ClassLoader.loadClass(ClassLoader.java:358) > >> ... 24 more > >> > >> On 11 Aug 2014, at 20:45, Telles Nobrega <[email protected]> > wrote: > >> > >>> Hi, I copied hadoop-hdfs-2.3.0 to my-job/lib and it changed the error > >> which is good but the error is back to > >>> > >>> Exception in thread "main" java.lang.RuntimeException: > >> java.lang.ClassNotFoundException: Class > >> org.apache.hadoop.hdfs.DistributedFileSystem not found > >>> at > >> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1720) > >>> at > >> org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2415) > >>> at > >> org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2428) > >>> at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:88) > >>> at > >> org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2467) > >>> at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2449) > >>> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:367) > >>> at org.apache.hadoop.fs.Path.getFileSystem(Path.java:287) > >>> at > >> > org.apache.samza.job.yarn.ClientHelper.submitApplication(ClientHelper.scala:111) > >>> at org.apache.samza.job.yarn.YarnJob.submit(YarnJob.scala:55) > >>> at org.apache.samza.job.yarn.YarnJob.submit(YarnJob.scala:48) > >>> at org.apache.samza.job.JobRunner.run(JobRunner.scala:62) > >>> at org.apache.samza.job.JobRunner$.main(JobRunner.scala:37) > >>> at org.apache.samza.job.JobRunner.main(JobRunner.scala) > >>> Caused by: java.lang.ClassNotFoundException: Class > >> org.apache.hadoop.hdfs.DistributedFileSystem not found > >>> at > >> > org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1626) > >>> at > >> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1718) > >>> ... 13 more > >>> > >>> Do I need to have this lib in all nodes at the job folder or just to > >> submit? > >>> > >>> On 11 Aug 2014, at 20:11, Yan Fang <[email protected]> wrote: > >>> > >>>> Hi Telles, > >>>> > >>>> I replayed your problem and think I figured out why CLASSPATH does not > >>>> work. Because in our script bin/run-class.sh, we have the line > >>>> "CLASSPATH=$HADOOP_CONF_DIR", which actually ingores your setting. > >>>> > >>>> So a simple solution is to copy the hadoop-hdfs.jar to your samza lib > >>>> directory. Then run bin/run-job ----config-factory=... > >> --config-path=... . > >>>> Let me know how it goes. Thank you. > >>>> > >>>> Cheers, > >>>> > >>>> Fang, Yan > >>>> [email protected] > >>>> +1 (206) 849-4108 > >>>> > >>>> > >>>> On Mon, Aug 11, 2014 at 4:07 PM, Telles Nobrega < > >> [email protected]> > >>>> wrote: > >>>> > >>>>> Sure, thanks. > >>>>> > >>>>> > >>>>> On Mon, Aug 11, 2014 at 6:22 PM, Yan Fang <[email protected]> > >> wrote: > >>>>> > >>>>>> Hi Telles, > >>>>>> > >>>>>> I am not sure whether exporting the CLASSPATH works. (sometimes it > >> does > >>>>> not > >>>>>> work for me...) My suggestion is to include the hdfs jar explicitly > in > >>>>> the > >>>>>> package that you upload to hdfs. Also , remember to put the jar into > >> your > >>>>>> local samza (which is deploy/samza/lib if you go with the > hello-samza > >>>>>> tutorial) Let me know if that works. > >>>>>> > >>>>>> Cheers, > >>>>>> > >>>>>> Fang, Yan > >>>>>> [email protected] > >>>>>> +1 (206) 849-4108 > >>>>>> > >>>>>> > >>>>>> On Mon, Aug 11, 2014 at 2:04 PM, Chris Riccomini < > >>>>>> [email protected]> wrote: > >>>>>> > >>>>>>> Hey Telles, > >>>>>>> > >>>>>>> Hmm. I'm out of ideas. If Zhijie is around, he'd probably be of > use, > >>>>> but > >>>>>> I > >>>>>>> haven't heard from him in a while. > >>>>>>> > >>>>>>> I'm afraid your best bet is probably to email the YARN dev mailing > >>>>> list, > >>>>>>> since this is a YARN config issue. > >>>>>>> > >>>>>>> Cheers, > >>>>>>> Chris > >>>>>>> > >>>>>>> On 8/11/14 1:58 PM, "Telles Nobrega" <[email protected]> > >> wrote: > >>>>>>> > >>>>>>>> I exported export > >>>>>>> > >>>>>> > >>>>>> > >> > CLASSPATH=$CLASSPATH:hadoop-2.3.0/share/hadoop/hdfs/hadoop-hdfs-2.3.0.jar > >>>>>>>> and still happened the same problem. > >>>>>>>> > >>>>>>>> > >>>>>>>> On Mon, Aug 11, 2014 at 5:35 PM, Chris Riccomini < > >>>>>>>> [email protected]> wrote: > >>>>>>>> > >>>>>>>>> Hey Telles, > >>>>>>>>> > >>>>>>>>> It sounds like either the HDFS jar is missing from the classpath, > >> or > >>>>>> the > >>>>>>>>> hdfs file system needs to be configured: > >>>>>>>>> > >>>>>>>>> <property> > >>>>>>>>> <name>fs.hdfs.impl</name> > >>>>>>>>> <value>org.apache.hadoop.hdfs.DistributedFileSystem</value> > >>>>>>>>> <description>The FileSystem for hdfs: uris.</description> > >>>>>>>>> </property> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> (from > >>>>>>>>> > >>>>>>>>> > >>>>>>> > >>>>>> > >>>>> > >> > https://groups.google.com/a/cloudera.org/forum/#!topic/scm-users/lyho8ptA > >>>>>>>>> zE > >>>>>>>>> 0) > >>>>>>>>> > >>>>>>>>> I believe this will need to be configured for your NM. > >>>>>>>>> > >>>>>>>>> Cheers, > >>>>>>>>> Chris > >>>>>>>>> > >>>>>>>>> On 8/11/14 1:31 PM, "Telles Nobrega" <[email protected]> > >>>>> wrote: > >>>>>>>>> > >>>>>>>>>> Yes, it is like this: > >>>>>>>>>> > >>>>>>>>>> <configuration> > >>>>>>>>>> <property> > >>>>>>>>>> <name>dfs.datanode.data.dir</name> > >>>>>>>>>> <value>file:///home/ubuntu/hadoop-2.3.0/hdfs/datanode</value> > >>>>>>>>>> <description>Comma separated list of paths on the local > >>>>>> filesystem > >>>>>>>>> of > >>>>>>>>>> a > >>>>>>>>>> DataNode where it should store its blocks.</description> > >>>>>>>>>> </property> > >>>>>>>>>> > >>>>>>>>>> <property> > >>>>>>>>>> <name>dfs.namenode.name.dir</name> > >>>>>>>>>> <value>file:///home/ubuntu/hadoop-2.3.0/hdfs/namenode</value> > >>>>>>>>>> <description>Path on the local filesystem where the NameNode > >>>>>> stores > >>>>>>>>>> the > >>>>>>>>>> namespace and transaction logs persistently.</description> > >>>>>>>>>> </property> > >>>>>>>>>> </configuration> > >>>>>>>>>> ~ > >>>>>>>>>> > >>>>>>>>>> I saw some report that this may be a classpath problem. Does > this > >>>>>>>>> sounds > >>>>>>>>>> right to you? > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> On Mon, Aug 11, 2014 at 5:25 PM, Yan Fang <[email protected] > > > >>>>>>> wrote: > >>>>>>>>>> > >>>>>>>>>>> Hi Telles, > >>>>>>>>>>> > >>>>>>>>>>> It looks correct. Did you put the hdfs-site.xml into your > >>>>>>>>>>> HADOOP_CONF_DIR > >>>>>>>>>>> ？(such as ~/.samza/conf) > >>>>>>>>>>> > >>>>>>>>>>> Fang, Yan > >>>>>>>>>>> [email protected] > >>>>>>>>>>> +1 (206) 849-4108 > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> On Mon, Aug 11, 2014 at 1:02 PM, Telles Nobrega > >>>>>>>>>>> <[email protected]> > >>>>>>>>>>> wrote: > >>>>>>>>>>> > >>>>>>>>>>>> Hi Yan Fang, > >>>>>>>>>>>> > >>>>>>>>>>>> I was able to deploy the file to hdfs, I can see them in all > my > >>>>>>>>> nodes > >>>>>>>>>>> but > >>>>>>>>>>>> when I tried running I got this error: > >>>>>>>>>>>> > >>>>>>>>>>>> Exception in thread "main" java.io.IOException: No FileSystem > >>>>> for > >>>>>>>>>>> scheme: > >>>>>>>>>>>> hdfs > >>>>>>>>>>>> at > >>>>>>>>>>> > >>>>>>> > >>>>>> > >>>>>>> > >> org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2421) > >>>>>>>>>>>> at > >>>>>>>>>>> > >>>>>>> > >>>>>>> > >> org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2428) > >>>>>>>>>>>> at > >>>>> org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:88) > >>>>>>>>>>>> at > >>>>>>>>>>> > >>>>>>> > >>>>>>> > >> org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2467) > >>>>>>>>>>>> at > >>>>>> org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2449) > >>>>>>>>>>>> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:367) > >>>>>>>>>>>> at org.apache.hadoop.fs.Path.getFileSystem(Path.java:287) > >>>>>>>>>>>> at > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>> > >>>>>>> > >>>>>> > >>>>>>>>> > >> org.apache.samza.job.yarn.ClientHelper.submitApplication(ClientHelper.s > >>>>>>>>>>> ca > >>>>>>>>>>> la:111) > >>>>>>>>>>>> at org.apache.samza.job.yarn.YarnJob.submit(YarnJob.scala:55) > >>>>>>>>>>>> at org.apache.samza.job.yarn.YarnJob.submit(YarnJob.scala:48) > >>>>>>>>>>>> at org.apache.samza.job.JobRunner.run(JobRunner.scala:62) > >>>>>>>>>>>> at org.apache.samza.job.JobRunner$.main(JobRunner.scala:37) > >>>>>>>>>>>> at org.apache.samza.job.JobRunner.main(JobRunner.scala) > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> This is my yarn.package.path config: > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>> > >>>>>>> > >>>>>> > >>>>>>>>> > >> yarn.package.path=hdfs://telles-master-samza:50070/samza-job-package-0 > >>>>>>>>>>> .7 > >>>>>>>>>>> .0-dist.tar.gz > >>>>>>>>>>>> > >>>>>>>>>>>> Thanks in advance > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> On Mon, Aug 11, 2014 at 3:00 PM, Yan Fang < > >>>>> [email protected]> > >>>>>>>>>>> wrote: > >>>>>>>>>>>> > >>>>>>>>>>>>> Hi Telles, > >>>>>>>>>>>>> > >>>>>>>>>>>>> In terms of "*I tried pushing the tar file to HDFS but I got > >>>>> an > >>>>>>>>>>> error > >>>>>>>>>>>> from > >>>>>>>>>>>>> hadoop saying that it couldn’t find core-site.xml file*.", I > >>>>>>>>> guess > >>>>>>>>>>> you > >>>>>>>>>>>> set > >>>>>>>>>>>>> the HADOOP_CONF_DIR variable and made it point to > >>>>>> ~/.samza/conf. > >>>>>>>>> You > >>>>>>>>>>> can > >>>>>>>>>>>> do > >>>>>>>>>>>>> 1) make the HADOOP_CONF_DIR point to the directory where your > >>>>>>>>> conf > >>>>>>>>>>> files > >>>>>>>>>>>>> are, such as /etc/hadoop/conf. Or 2) copy the config files to > >>>>>>>>>>>>> ~/.samza/conf. Thank you, > >>>>>>>>>>>>> > >>>>>>>>>>>>> Cheer, > >>>>>>>>>>>>> > >>>>>>>>>>>>> Fang, Yan > >>>>>>>>>>>>> [email protected] > >>>>>>>>>>>>> +1 (206) 849-4108 > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> On Mon, Aug 11, 2014 at 7:40 AM, Chris Riccomini < > >>>>>>>>>>>>> [email protected]> wrote: > >>>>>>>>>>>>> > >>>>>>>>>>>>>> Hey Telles, > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> To get YARN working with the HTTP file system, you need to > >>>>>>>>> follow > >>>>>>>>>>> the > >>>>>>>>>>>>>> instructions on: > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>> > >>>>>> > >>>>> > >> > http://samza.incubator.apache.org/learn/tutorials/0.7.0/run-in-multi-node > >>>>>>>>>>> -y > >>>>>>>>>>>>>> arn.html > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> In the "Set Up Http Filesystem for YARN" section. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> You shouldn't need to compile anything (no Gradle, which is > >>>>>>>>> what > >>>>>>>>>>> your > >>>>>>>>>>>>>> stack trace is showing). This setup should be done for all > >>>>> of > >>>>>>>>> the > >>>>>>>>>>> NMs, > >>>>>>>>>>>>>> since they will be the ones downloading your job's package > >>>>>>>>> (from > >>>>>>>>>>>>>> yarn.package.path). > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Cheers, > >>>>>>>>>>>>>> Chris > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> On 8/9/14 9:44 PM, "Telles Nobrega" < > >>>>> [email protected] > >>>>>>> > >>>>>>>>>>> wrote: > >>>>>>>>>>>>>> > >>>>>>>>>>>>>>> Hi again, I tried installing the scala libs but the Http > >>>>>>>>> problem > >>>>>>>>>>> still > >>>>>>>>>>>>>>> occurs. I realised that I need to compile incubator samza > >>>>> in > >>>>>>>>> the > >>>>>>>>>>>>> machines > >>>>>>>>>>>>>>> that I¹m going to run the jobs, but the compilation fails > >>>>>> with > >>>>>>>>>>> this > >>>>>>>>>>>> huge > >>>>>>>>>>>>>>> message: > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> # > >>>>>>>>>>>>>>> # There is insufficient memory for the Java Runtime > >>>>>>>>> Environment > >>>>>>>>>>> to > >>>>>>>>>>>>>>> continue. > >>>>>>>>>>>>>>> # Native memory allocation (malloc) failed to allocate > >>>>>>>>> 3946053632 > >>>>>>>>>>>> bytes > >>>>>>>>>>>>>>> for committing reserved memory. > >>>>>>>>>>>>>>> # An error report file with more information is saved as: > >>>>>>>>>>>>>>> # > >>>>>> /home/ubuntu/incubator-samza/samza-kafka/hs_err_pid2506.log > >>>>>>>>>>>>>>> Could not write standard input into: Gradle Worker 13. > >>>>>>>>>>>>>>> java.io.IOException: Broken pipe > >>>>>>>>>>>>>>> at java.io.FileOutputStream.writeBytes(Native > >>>>> Method) > >>>>>>>>>>>>>>> at > >>>>>>>>>>> java.io.FileOutputStream.write(FileOutputStream.java:345) > >>>>>>>>>>>>>>> at > >>>>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>> > >>>>>>> > >> java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82) > >>>>>>>>>>>>>>> at > >>>>>>>>>>>>>> > >>>>>>>>> java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140) > >>>>>>>>>>>>>>> at > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>> > >>>>>>> > >>>>>> > >>>>>>>>>> > >> org.gradle.process.internal.streams.ExecOutputHandleRunner.run(ExecOut > >>>>>>>>>>>> pu > >>>>>>>>>>>> tH > >>>>>>>>>>>>>>> andleRunner.java:53) > >>>>>>>>>>>>>>> at > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>> > >>>>>>> > >>>>>> > >>>>>>>>>> > >> org.gradle.internal.concurrent.DefaultExecutorFactory$StoppableExecuto > >>>>>>>>>>>> rI > >>>>>>>>>>>> mp > >>>>>>>>>>>>>>> l$1.run(DefaultExecutorFactory.java:66) > >>>>>>>>>>>>>>> at > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>> > >>>>>>> > >>>>>> > >>>>>>>>>> > >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.j > >>>>>>>>>>>> av > >>>>>>>>>>>> a: > >>>>>>>>>>>>>>> 1145) > >>>>>>>>>>>>>>> at > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>> > >>>>>>> > >>>>>> > >>>>>>>>>> > >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor. > >>>>>>>>>>>> ja > >>>>>>>>>>>> va > >>>>>>>>>>>>>>> :615) > >>>>>>>>>>>>>>> at java.lang.Thread.run(Thread.java:744) > >>>>>>>>>>>>>>> Process 'Gradle Worker 13' finished with non-zero exit > >>>>>> value 1 > >>>>>>>>>>>>>>> org.gradle.process.internal.ExecException: Process 'Gradle > >>>>>>>>> Worker > >>>>>>>>>>> 13' > >>>>>>>>>>>>>>> finished with non-zero exit value 1 > >>>>>>>>>>>>>>> at > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>> > >>>>>>> > >>>>>> > >>>>>>>>>> > >> org.gradle.process.internal.DefaultExecHandle$ExecResultImpl.assertNor > >>>>>>>>>>>> ma > >>>>>>>>>>>> lE > >>>>>>>>>>>>>>> xitValue(DefaultExecHandle.java:362) > >>>>>>>>>>>>>>> at > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>> > >>>>>>> > >>>>>> > >>>>>>>>>> > >> org.gradle.process.internal.DefaultWorkerProcess.onProcessStop(Default > >>>>>>>>>>>> Wo > >>>>>>>>>>>> rk > >>>>>>>>>>>>>>> erProcess.java:89) > >>>>>>>>>>>>>>> at > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>> > >>>>>>> > >>>>>> > >>>>>>>>>> > >> org.gradle.process.internal.DefaultWorkerProcess.access$000(DefaultWor > >>>>>>>>>>>> ke > >>>>>>>>>>>> rP > >>>>>>>>>>>>>>> rocess.java:33) > >>>>>>>>>>>>>>> at > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>> > >>>>>>> > >>>>>> > >>>>>>>>>> > >> org.gradle.process.internal.DefaultWorkerProcess$1.executionFinished(D > >>>>>>>>>>>> ef > >>>>>>>>>>>> au > >>>>>>>>>>>>>>> ltWorkerProcess.java:55) > >>>>>>>>>>>>>>> at > >>>>>> sun.reflect.NativeMethodAccessorImpl.invoke0(Native > >>>>>>>>>>> Method) > >>>>>>>>>>>>>>> at > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>> > >>>>>>> > >>>>>> > >>>>>>>>>> > >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.j > >>>>>>>>>>>> av > >>>>>>>>>>>> a: > >>>>>>>>>>>>>>> 57) > >>>>>>>>>>>>>>> at > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>> > >>>>>>> > >>>>>> > >>>>>>>>>> > >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccess > >>>>>>>>>>>> or > >>>>>>>>>>>> Im > >>>>>>>>>>>>>>> pl.java:43) > >>>>>>>>>>>>>>> at java.lang.reflect.Method.invoke(Method.java:606) > >>>>>>>>>>>>>>> at > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>> > >>>>>>> > >>>>>> > >>>>>>>>>> > >> org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDi > >>>>>>>>>>>> sp > >>>>>>>>>>>> at > >>>>>>>>>>>>>>> ch.java:35) > >>>>>>>>>>>>>>> at > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>> > >>>>>>> > >>>>>> > >>>>>>>>>> > >> org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDi > >>>>>>>>>>>> sp > >>>>>>>>>>>> at > >>>>>>>>>>>>>>> ch.java:24) > >>>>>>>>>>>>>>> at > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>> > >>>>>>> > >>>>>> > >>>>>>>>>> > >> org.gradle.listener.BroadcastDispatch.dispatch(BroadcastDispatch.java: > >>>>>>>>>>>> 81 > >>>>>>>>>>>> ) > >>>>>>>>>>>>>>> at > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>> > >>>>>>> > >>>>>> > >>>>>>>>>> > >> org.gradle.listener.BroadcastDispatch.dispatch(BroadcastDispatch.java: > >>>>>>>>>>>> 30 > >>>>>>>>>>>> ) > >>>>>>>>>>>>>>> at > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>> > >>>>>>> > >>>>>> > >>>>>>>>>> > >> org.gradle.messaging.dispatch.ProxyDispatchAdapter$DispatchingInvocati > >>>>>>>>>>>> on > >>>>>>>>>>>> Ha > >>>>>>>>>>>>>>> ndler.invoke(ProxyDispatchAdapter.java:93) > >>>>>>>>>>>>>>> at com.sun.proxy.$Proxy46.executionFinished(Unknown > >>>>>>>>>>> Source) > >>>>>>>>>>>>>>> at > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>> > >>>>>>> > >>>>>> > >>>>>>>>>> > >> org.gradle.process.internal.DefaultExecHandle.setEndStateInfo(DefaultE > >>>>>>>>>>>> xe > >>>>>>>>>>>> cH > >>>>>>>>>>>>>>> andle.java:212) > >>>>>>>>>>>>>>> at > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>> > >>>>>>> > >>>>>> > >>>>>>>>>> > >> org.gradle.process.internal.DefaultExecHandle.finished(DefaultExecHand > >>>>>>>>>>>> le > >>>>>>>>>>>> .j > >>>>>>>>>>>>>>> ava:309) > >>>>>>>>>>>>>>> at > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>> > >>>>>>> > >>>>>> > >>>>>>>>>> > >> org.gradle.process.internal.ExecHandleRunner.completed(ExecHandleRunne > >>>>>>>>>>>> r. > >>>>>>>>>>>> ja > >>>>>>>>>>>>>>> va:108) > >>>>>>>>>>>>>>> at > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>> > >>>>>>> > >>>>>> > >>>>>>>>>> > >> org.gradle.process.internal.ExecHandleRunner.run(ExecHandleRunner.java > >>>>>>>>>>>> :8 > >>>>>>>>>>>> 8) > >>>>>>>>>>>>>>> at > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>> > >>>>>>> > >>>>>> > >>>>>>>>>> > >> org.gradle.internal.concurrent.DefaultExecutorFactory$StoppableExecuto > >>>>>>>>>>>> rI > >>>>>>>>>>>> mp > >>>>>>>>>>>>>>> l$1.run(DefaultExecutorFactory.java:66) > >>>>>>>>>>>>>>> at > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>> > >>>>>>> > >>>>>> > >>>>>>>>>> > >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.j > >>>>>>>>>>>> av > >>>>>>>>>>>> a: > >>>>>>>>>>>>>>> 1145) > >>>>>>>>>>>>>>> at > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>> > >>>>>>> > >>>>>> > >>>>>>>>>> > >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor. > >>>>>>>>>>>> ja > >>>>>>>>>>>> va > >>>>>>>>>>>>>>> :615) > >>>>>>>>>>>>>>> at java.lang.Thread.run(Thread.java:744) > >>>>>>>>>>>>>>> OpenJDK 64-Bit Server VM warning: INFO: > >>>>>>>>>>>>>>> os::commit_memory(0x000000070a6c0000, 3946053632, 0) > >>>>> failed; > >>>>>>>>>>>>>>> error='Cannot allocate memory' (errno=12) > >>>>>>>>>>>>>>> # > >>>>>>>>>>>>>>> # There is insufficient memory for the Java Runtime > >>>>>>>>> Environment > >>>>>>>>>>> to > >>>>>>>>>>>>>>> continue. > >>>>>>>>>>>>>>> # Native memory allocation (malloc) failed to allocate > >>>>>>>>> 3946053632 > >>>>>>>>>>>> bytes > >>>>>>>>>>>>>>> for committing reserved memory. > >>>>>>>>>>>>>>> # An error report file with more information is saved as: > >>>>>>>>>>>>>>> # > >>>>>> /home/ubuntu/incubator-samza/samza-kafka/hs_err_pid2518.log > >>>>>>>>>>>>>>> Could not write standard input into: Gradle Worker 14. > >>>>>>>>>>>>>>> java.io.IOException: Broken pipe > >>>>>>>>>>>>>>> at java.io.FileOutputStream.writeBytes(Native > >>>>> Method) > >>>>>>>>>>>>>>> at > >>>>>>>>>>> java.io.FileOutputStream.write(FileOutputStream.java:345) > >>>>>>>>>>>>>>> at > >>>>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>> > >>>>>>> > >> java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82) > >>>>>>>>>>>>>>> at > >>>>>>>>>>>>>> > >>>>>>>>> java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140) > >>>>>>>>>>>>>>> at > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>> > >>>>>>> > >>>>>> > >>>>>>>>>> > >> org.gradle.process.internal.streams.ExecOutputHandleRunner.run(ExecOut > >>>>>>>>>>>> pu > >>>>>>>>>>>> tH > >>>>>>>>>>>>>>> andleRunner.java:53) > >>>>>>>>>>>>>>> at > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>> > >>>>>>> > >>>>>> > >>>>>>>>>> > >> org.gradle.internal.concurrent.DefaultExecutorFactory$StoppableExecuto > >>>>>>>>>>>> rI > >>>>>>>>>>>> mp > >>>>>>>>>>>>>>> l$1.run(DefaultExecutorFactory.java:66) > >>>>>>>>>>>>>>> at > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>> > >>>>>>> > >>>>>> > >>>>>>>>>> > >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.j > >>>>>>>>>>>> av > >>>>>>>>>>>> a: > >>>>>>>>>>>>>>> 1145) > >>>>>>>>>>>>>>> at > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>> > >>>>>>> > >>>>>> > >>>>>>>>>> > >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor. > >>>>>>>>>>>> ja > >>>>>>>>>>>> va > >>>>>>>>>>>>>>> :615) > >>>>>>>>>>>>>>> at java.lang.Thread.run(Thread.java:744) > >>>>>>>>>>>>>>> Process 'Gradle Worker 14' finished with non-zero exit > >>>>>> value 1 > >>>>>>>>>>>>>>> org.gradle.process.internal.ExecException: Process 'Gradle > >>>>>>>>> Worker > >>>>>>>>>>> 14' > >>>>>>>>>>>>>>> finished with non-zero exit value 1 > >>>>>>>>>>>>>>> at > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>> > >>>>>>> > >>>>>> > >>>>>>>>>> > >> org.gradle.process.internal.DefaultExecHandle$ExecResultImpl.assertNor > >>>>>>>>>>>> ma > >>>>>>>>>>>> lE > >>>>>>>>>>>>>>> xitValue(DefaultExecHandle.java:362) > >>>>>>>>>>>>>>> at > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>> > >>>>>>> > >>>>>> > >>>>>>>>>> > >> org.gradle.process.internal.DefaultWorkerProcess.onProcessStop(Default > >>>>>>>>>>>> Wo > >>>>>>>>>>>> rk > >>>>>>>>>>>>>>> erProcess.java:89) > >>>>>>>>>>>>>>> at > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>> > >>>>>>> > >>>>>> > >>>>>>>>>> > >> org.gradle.process.internal.DefaultWorkerProcess.access$000(DefaultWor > >>>>>>>>>>>> ke > >>>>>>>>>>>> rP > >>>>>>>>>>>>>>> rocess.java:33) > >>>>>>>>>>>>>>> at > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>> > >>>>>>> > >>>>>> > >>>>>>>>>> > >> org.gradle.process.internal.DefaultWorkerProcess$1.executionFinished(D > >>>>>>>>>>>> ef > >>>>>>>>>>>> au > >>>>>>>>>>>>>>> ltWorkerProcess.java:55) > >>>>>>>>>>>>>>> at > >>>>>> sun.reflect.NativeMethodAccessorImpl.invoke0(Native > >>>>>>>>>>> Method) > >>>>>>>>>>>>>>> at > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>> > >>>>>>> > >>>>>> > >>>>>>>>>> > >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.j > >>>>>>>>>>>> av > >>>>>>>>>>>> a: > >>>>>>>>>>>>>>> 57) > >>>>>>>>>>>>>>> at > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>> > >>>>>>> > >>>>>> > >>>>>>>>>> > >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccess > >>>>>>>>>>>> or > >>>>>>>>>>>> Im > >>>>>>>>>>>>>>> pl.java:43) > >>>>>>>>>>>>>>> at java.lang.reflect.Method.invoke(Method.java:606) > >>>>>>>>>>>>>>> at > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>> > >>>>>>> > >>>>>> > >>>>>>>>>> > >> org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDi > >>>>>>>>>>>> sp > >>>>>>>>>>>> at > >>>>>>>>>>>>>>> ch.java:35) > >>>>>>>>>>>>>>> at > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>> > >>>>>>> > >>>>>> > >>>>>>>>>> > >> org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDi > >>>>>>>>>>>> sp > >>>>>>>>>>>> at > >>>>>>>>>>>>>>> ch.java:24) > >>>>>>>>>>>>>>> at > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>> > >>>>>>> > >>>>>> > >>>>>>>>>> > >> org.gradle.listener.BroadcastDispatch.dispatch(BroadcastDispatch.java: > >>>>>>>>>>>> 81 > >>>>>>>>>>>> ) > >>>>>>>>>>>>>>> at > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>> > >>>>>>> > >>>>>> > >>>>>>>>>> > >> org.gradle.listener.BroadcastDispatch.dispatch(BroadcastDispatch.java: > >>>>>>>>>>>> 30 > >>>>>>>>>>>> ) > >>>>>>>>>>>>>>> at > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>> > >>>>>>> > >>>>>> > >>>>>>>>>> > >> org.gradle.messaging.dispatch.ProxyDispatchAdapter$DispatchingInvocati > >>>>>>>>>>>> on > >>>>>>>>>>>> Ha > >>>>>>>>>>>>>>> ndler.invoke(ProxyDispatchAdapter.java:93) > >>>>>>>>>>>>>>> at com.sun.proxy.$Proxy46.executionFinished(Unknown > >>>>>>>>>>> Source) > >>>>>>>>>>>>>>> at > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>> > >>>>>>> > >>>>>> > >>>>>>>>>> > >> org.gradle.process.internal.DefaultExecHandle.setEndStateInfo(DefaultE > >>>>>>>>>>>> xe > >>>>>>>>>>>> cH > >>>>>>>>>>>>>>> andle.java:212) > >>>>>>>>>>>>>>> at > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>> > >>>>>>> > >>>>>> > >>>>>>>>>> > >> org.gradle.process.internal.DefaultExecHandle.finished(DefaultExecHand > >>>>>>>>>>>> le > >>>>>>>>>>>> .j > >>>>>>>>>>>>>>> ava:309) > >>>>>>>>>>>>>>> at > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>> > >>>>>>> > >>>>>> > >>>>>>>>>> > >> org.gradle.process.internal.ExecHandleRunner.completed(ExecHandleRunne > >>>>>>>>>>>> r. > >>>>>>>>>>>> ja > >>>>>>>>>>>>>>> va:108) > >>>>>>>>>>>>>>> at > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>> > >>>>>>> > >>>>>> > >>>>>>>>>> > >> org.gradle.process.internal.ExecHandleRunner.run(ExecHandleRunner.java > >>>>>>>>>>>> :8 > >>>>>>>>>>>> 8) > >>>>>>>>>>>>>>> at > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>> > >>>>>>> > >>>>>> > >>>>>>>>>> > >> org.gradle.internal.concurrent.DefaultExecutorFactory$StoppableExecuto > >>>>>>>>>>>> rI > >>>>>>>>>>>> mp > >>>>>>>>>>>>>>> l$1.run(DefaultExecutorFactory.java:66) > >>>>>>>>>>>>>>> at > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>> > >>>>>>> > >>>>>> > >>>>>>>>>> > >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.j > >>>>>>>>>>>> av > >>>>>>>>>>>> a: > >>>>>>>>>>>>>>> 1145) > >>>>>>>>>>>>>>> at > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>> > >>>>>>> > >>>>>> > >>>>>>>>>> > >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor. > >>>>>>>>>>>> ja > >>>>>>>>>>>> va > >>>>>>>>>>>>>>> :615) > >>>>>>>>>>>>>>> at java.lang.Thread.r > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> Do I need more memory for my machines? Each already has > >>>>>> 4GB. I > >>>>>>>>>>> really > >>>>>>>>>>>>>>> need to have this running. I¹m not sure which way is best > >>>>>>>>> http or > >>>>>>>>>>> hdfs > >>>>>>>>>>>>>>> which one you suggest and how can i solve my problem for > >>>>>> each > >>>>>>>>>>> case. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> Thanks in advance and sorry for bothering this much. > >>>>>>>>>>>>>>> On 10 Aug 2014, at 00:20, Telles Nobrega > >>>>>>>>>>> <[email protected]> > >>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> Hi Chris, now I have the tar file in my RM machine, and > >>>>>> the > >>>>>>>>>>> yarn > >>>>>>>>>>>> path > >>>>>>>>>>>>>>>> points to it. I changed the core-site.xml to use > >>>>>>>>> HttpFileSystem > >>>>>>>>>>>> instead > >>>>>>>>>>>>>>>> of HDFS now it is failing with > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> Application application_1407640485281_0001 failed 2 > >>>>> times > >>>>>>>>> due > >>>>>>>>>>> to > >>>>>>>>>>> AM > >>>>>>>>>>>>>>>> Container for appattempt_1407640485281_0001_000002 exited > >>>>>>>>> with > >>>>>>>>>>>>>>>> exitCode:-1000 due to: java.lang.ClassNotFoundException: > >>>>>>>>> Class > >>>>>>>>>>>>>>>> org.apache.samza.util.hadoop.HttpFileSystem not found > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> I think I can solve this just installing scala files > >>>>> from > >>>>>>>>> the > >>>>>>>>>>> samza > >>>>>>>>>>>>>>>> tutorial, can you confirm that? > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> On 09 Aug 2014, at 08:34, Telles Nobrega > >>>>>>>>>>> <[email protected] > >>>>>>>>>>>> > >>>>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> Hi Chris, > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> I think the problem is that I forgot to update the > >>>>>>>>>>>> yarn.job.package. > >>>>>>>>>>>>>>>>> I will try again to see if it works now. > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> I have one more question, how can I stop (command line) > >>>>>> the > >>>>>>>>>>> jobs > >>>>>>>>>>>>>>>>> running in my topology, for the experiment that I will > >>>>>> run, > >>>>>>>>> I > >>>>>>>>>>> need > >>>>>>>>>>>> to > >>>>>>>>>>>>>>>>> run the same job in 4 minutes intervals. So I need to > >>>>> kill > >>>>>>>>> it, > >>>>>>>>>>> clean > >>>>>>>>>>>>>>>>> the kafka topics and rerun. > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> Thanks in advance. > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> On 08 Aug 2014, at 12:41, Chris Riccomini > >>>>>>>>>>>>>>>>> <[email protected]> wrote: > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> Hey Telles, > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> Do I need to have the job folder on each machine in > >>>>> my > >>>>>>>>>>> cluster? > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> No, you should not need to do this. There are two ways > >>>>>> to > >>>>>>>>>>> deploy > >>>>>>>>>>>>> your > >>>>>>>>>>>>>>>>>> tarball to the YARN grid. One is to put it in HDFS, > >>>>> and > >>>>>>>>> the > >>>>>>>>>>> other > >>>>>>>>>>>> is > >>>>>>>>>>>>>>>>>> to > >>>>>>>>>>>>>>>>>> put it on an HTTP server. The link to running a Samza > >>>>>> job > >>>>>>>>> in > >>>>>>>>>>> a > >>>>>>>>>>>>>>>>>> multi-node > >>>>>>>>>>>>>>>>>> YARN cluster describes how to do both (either HTTP > >>>>>> server > >>>>>>>>> or > >>>>>>>>>>>> HDFS). > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> In both cases, once the tarball is put in on the > >>>>>> HTTP/HDFS > >>>>>>>>>>>>> server(s), > >>>>>>>>>>>>>>>>>> you > >>>>>>>>>>>>>>>>>> must update yarn.package.path to point to it. From > >>>>>> there, > >>>>>>>>> the > >>>>>>>>>>> YARN > >>>>>>>>>>>>> NM > >>>>>>>>>>>>>>>>>> should download it for you automatically when you > >>>>> start > >>>>>>>>> your > >>>>>>>>>>> job. > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> * Can you send along a paste of your job config? > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> Cheers, > >>>>>>>>>>>>>>>>>> Chris > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> On 8/8/14 8:04 AM, "Claudio Martins" > >>>>>>>>>>> <[email protected]> > >>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> Hi Telles, it looks to me that you forgot to update > >>>>> the > >>>>>>>>>>>>>>>>>>> "yarn.package.path" > >>>>>>>>>>>>>>>>>>> attribute in your config file for the task. > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> - Claudio Martins > >>>>>>>>>>>>>>>>>>> Head of Engineering > >>>>>>>>>>>>>>>>>>> MobileAware USA Inc. / www.mobileaware.com > >>>>>>>>>>>>>>>>>>> office: +1 617 986 5060 / mobile: +1 617 480 5288 > >>>>>>>>>>>>>>>>>>> linkedin: www.linkedin.com/in/martinsclaudio > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> On Fri, Aug 8, 2014 at 10:55 AM, Telles Nobrega > >>>>>>>>>>>>>>>>>>> <[email protected]> > >>>>>>>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> Hi, > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> this is my first time trying to run a job on a > >>>>>> multinode > >>>>>>>>>>>>>>>>>>>> environment. I > >>>>>>>>>>>>>>>>>>>> have the cluster set up, I can see in the GUI that > >>>>> all > >>>>>>>>>>> nodes > >>>>>>>>>>> are > >>>>>>>>>>>>>>>>>>>> working. > >>>>>>>>>>>>>>>>>>>> Do I need to have the job folder on each machine in > >>>>> my > >>>>>>>>>>> cluster? > >>>>>>>>>>>>>>>>>>>> - The first time I tried running with the job on the > >>>>>>>>>>> namenode > >>>>>>>>>>>>>>>>>>>> machine > >>>>>>>>>>>>>>>>>>>> and > >>>>>>>>>>>>>>>>>>>> it failed saying: > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> Application application_1407509228798_0001 failed 2 > >>>>>>>>> times > >>>>>>>>>>> due > >>>>>>>>>>> to > >>>>>>>>>>>>> AM > >>>>>>>>>>>>>>>>>>>> Container for appattempt_1407509228798_0001_000002 > >>>>>>>>> exited > >>>>>>>>>>> with > >>>>>>>>>>>>>>>>>>>> exitCode: > >>>>>>>>>>>>>>>>>>>> -1000 due to: File > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>> > >>>>>>> > >>>>>> > >>>>>>>>>>>>>>> > >> file:/home/ubuntu/alarm-samza/samza-job-package/target/samza-job- > >>>>>>>>>>>>>>>>> pa > >>>>>>>>>>>>>>>>> ck > >>>>>>>>>>>>>>>>>>>> age- > >>>>>>>>>>>>>>>>>>>> 0.7.0-dist.tar.gz > >>>>>>>>>>>>>>>>>>>> does not exist > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> So I copied the folder to each machine in my cluster > >>>>>> and > >>>>>>>>>>> got > >>>>>>>>>>>> this > >>>>>>>>>>>>>>>>>>>> error: > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> Application application_1407509228798_0002 failed 2 > >>>>>>>>> times > >>>>>>>>>>> due > >>>>>>>>>>> to > >>>>>>>>>>>>> AM > >>>>>>>>>>>>>>>>>>>> Container for appattempt_1407509228798_0002_000002 > >>>>>>>>> exited > >>>>>>>>>>> with > >>>>>>>>>>>>>>>>>>>> exitCode: > >>>>>>>>>>>>>>>>>>>> -1000 due to: Resource > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>> > >>>>>>> > >>>>>> > >>>>>>>>>>>>>>> > >> file:/home/ubuntu/alarm-samza/samza-job-package/target/samza-job- > >>>>>>>>>>>>>>>>> pa > >>>>>>>>>>>>>>>>> ck > >>>>>>>>>>>>>>>>>>>> age- > >>>>>>>>>>>>>>>>>>>> 0.7.0-dist.tar.gz > >>>>>>>>>>>>>>>>>>>> changed on src filesystem (expected 1407509168000, > >>>>> was > >>>>>>>>>>>>> 1407509434000 > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> What am I missing? > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> p.s.: I followed this > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> < > >>>>>>>>>>>>>> > >>>>>>>>>>> > >>>>>> > https://github.com/yahoo/samoa/wiki/Executing-SAMOA-with-Apache-Samz > >>>>>>>>>>>>>>>>>>>> a> > >>>>>>>>>>>>>>>>>>>> tutorial > >>>>>>>>>>>>>>>>>>>> and this > >>>>>>>>>>>>>>>>>>>> < > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>> > >> http://samza.incubator.apache.org/learn/tutorials/0.7.0/run-in-multi- > >>>>>>>>>>>>>>>>>>>> node > >>>>>>>>>>>>>>>>>>>> -yarn.html > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> to > >>>>>>>>>>>>>>>>>>>> set up the cluster. > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> Help is much appreciated. > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> Thanks in advance. > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> -- > >>>>>>>>>>>>>>>>>>>> ------------------------------------------ > >>>>>>>>>>>>>>>>>>>> Telles Mota Vidal Nobrega > >>>>>>>>>>>>>>>>>>>> M.sc. Candidate at UFCG > >>>>>>>>>>>>>>>>>>>> B.sc. in Computer Science at UFCG > >>>>>>>>>>>>>>>>>>>> Software Engineer at OpenStack Project - HP/LSD-UFCG > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> -- > >>>>>>>>>>>> ------------------------------------------ > >>>>>>>>>>>> Telles Mota Vidal Nobrega > >>>>>>>>>>>> M.sc. Candidate at UFCG > >>>>>>>>>>>> B.sc. in Computer Science at UFCG > >>>>>>>>>>>> Software Engineer at OpenStack Project - HP/LSD-UFCG > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> -- > >>>>>>>>>> ------------------------------------------ > >>>>>>>>>> Telles Mota Vidal Nobrega > >>>>>>>>>> M.sc. Candidate at UFCG > >>>>>>>>>> B.sc. in Computer Science at UFCG > >>>>>>>>>> Software Engineer at OpenStack Project - HP/LSD-UFCG > >>>>>>>>> > >>>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> -- > >>>>>>>> ------------------------------------------ > >>>>>>>> Telles Mota Vidal Nobrega > >>>>>>>> M.sc. Candidate at UFCG > >>>>>>>> B.sc. in Computer Science at UFCG > >>>>>>>> Software Engineer at OpenStack Project - HP/LSD-UFCG > >>>>>>> > >>>>>>> > >>>>>> > >>>>> > >>>>> > >>>>> > >>>>> -- > >>>>> ------------------------------------------ > >>>>> Telles Mota Vidal Nobrega > >>>>> M.sc. Candidate at UFCG > >>>>> B.sc. in Computer Science at UFCG > >>>>> Software Engineer at OpenStack Project - HP/LSD-UFCG > >>>>> > >>> > >> > >> > >
