Hi tom : which log will have info about why a process was Killed? Sent from my iPad
On Oct 28, 2011, at 11:41 PM, Tom Melendez <[email protected]> wrote: > Hi Jay, > > Are you able to look at the logs or the web interface? Can you find > out why it's getting killed? > > Also, can you verify that these ports are open and a process is > connected to them (maybe with netstat)? > > http://www.cloudera.com/blog/2009/08/hadoop-default-ports-quick-reference/ > > Thanks, > > Tom > > On Fri, Oct 28, 2011 at 7:57 PM, Jay Vyas <[email protected]> wrote: >> Thanks tom : Thats interesting.... >> >> First, I tried, and it complained that the input directory didnt exist, so I >> ran >> $> hadoop fs -mkdir /user/cloudera/input >> >> Then, I tried to do this : >> >> $> hadoop jar /usr/lib/hadoop-0.20/hadoop-examples.jar grep input output2 >> 'dfs[a-z.]+' >> >> And it seemed to start working ...... But then it abruptly printed "killed" >> somehow at the end of the job [scroll down] ? >> >> Maybe this is related to why i cant connect ..... ?! >> >> 1) the hadoop jar 11/10/14 21:34:43 WARN util.NativeCodeLoader: Unable to >> load native-hadoop library for your platform... using builtin-java classes >> where applicable >> 11/10/14 21:34:43 WARN snappy.LoadSnappy: Snappy native library not loaded >> 11/10/14 21:34:43 INFO mapred.FileInputFormat: Total input paths to process >> : 0 >> 11/10/14 21:34:44 INFO mapred.JobClient: Running job: job_201110142010_0009 >> 11/10/14 21:34:45 INFO mapred.JobClient: map 0% reduce 0% >> 11/10/14 21:34:55 INFO mapred.JobClient: map 0% reduce 100% >> 11/10/14 21:34:57 INFO mapred.JobClient: Job complete: job_201110142010_0009 >> 11/10/14 21:34:57 INFO mapred.JobClient: Counters: 14 >> 11/10/14 21:34:57 INFO mapred.JobClient: Job Counters >> 11/10/14 21:34:57 INFO mapred.JobClient: Launched reduce tasks=1 >> 11/10/14 21:34:57 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=5627 >> 11/10/14 21:34:57 INFO mapred.JobClient: Total time spent by all reduces >> waiting after reserving slots (ms)=0 >> 11/10/14 21:34:57 INFO mapred.JobClient: Total time spent by all maps >> waiting after reserving slots (ms)=0 >> 11/10/14 21:34:57 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=5050 >> 11/10/14 21:34:57 INFO mapred.JobClient: FileSystemCounters >> 11/10/14 21:34:57 INFO mapred.JobClient: FILE_BYTES_WRITTEN=53452 >> 11/10/14 21:34:57 INFO mapred.JobClient: HDFS_BYTES_WRITTEN=86 >> 11/10/14 21:34:57 INFO mapred.JobClient: Map-Reduce Framework >> 11/10/14 21:34:57 INFO mapred.JobClient: Reduce input groups=0 >> 11/10/14 21:34:57 INFO mapred.JobClient: Combine output records=0 >> 11/10/14 21:34:57 INFO mapred.JobClient: Reduce shuffle bytes=0 >> 11/10/14 21:34:57 INFO mapred.JobClient: Reduce output records=0 >> 11/10/14 21:34:57 INFO mapred.JobClient: Spilled Records=0 >> 11/10/14 21:34:57 INFO mapred.JobClient: Combine input records=0 >> 11/10/14 21:34:57 INFO mapred.JobClient: Reduce input records=0 >> 11/10/14 21:34:57 WARN mapred.JobClient: Use GenericOptionsParser for >> parsing the arguments. Applications should implement Tool for the same. >> 11/10/14 21:34:58 INFO mapred.FileInputFormat: Total input paths to process >> : 1 >> 11/10/14 21:34:58 INFO mapred.JobClient: Running job: job_201110142010_0010 >> 11/10/14 21:34:59 INFO mapred.JobClient: map 0% reduce 0% >> Killed >> >> >> On Fri, Oct 28, 2011 at 8:24 PM, Tom Melendez <[email protected]> wrote: >> >>> Hi Jay, >>> >>> Some questions for you: >>> >>> - Does the hadoop client itself work from that same machine? >>> - Are you actually able to run the hadoop example jar (in other words, >>> your setup is valid otherwise)? >>> - Is port 8020 actually available? (you can telnet or nc to it?) >>> - What does jps show on the namenode? >>> >>> Thanks, >>> >>> Tom >>> >>> On Fri, Oct 28, 2011 at 4:04 PM, Jay Vyas <[email protected]> wrote: >>>> Hi guys : Made more progress debugging my hadoop connection, but still >>>> haven't got it working...... It looks like my VM (cloudera hadoop) won't >>>> let me in. I find that there is no issue connecting to the name node - >>> that >>>> is , using hftp and 50070...... >>>> >>>> via standard HFTP as in here : >>>> >>>> //This method works fine - connecting directly to hadoop's namenode and >>>> querying the filesystem >>>> public static void main1(String[] args) throws Exception >>>> { >>>> String uri = "hftp://155.37.101.76:50070/";; >>>> >>>> System.out.println( "uri: " + uri ); >>>> Configuration conf = new Configuration(); >>>> >>>> FileSystem fs = FileSystem.get( URI.create( uri ), conf ); >>>> fs.printStatistics(); >>>> } >>>> >>>> >>>> But unfortunately, I can't get into hdfs ..... Any thoughts on this ? I >>> am >>>> modifying the uri to access port 8020 >>>> which is what is in my core-site.xml . >>>> >>>> // This fails, resulting (trys to connect over and over again, >>> eventually >>>> gives up printing "already tried to connect 20 times"....) >>>> public static void main(String[] args) >>>> { >>>> try { >>>> String uri = "hdfs://155.37.101.76:8020/"; >>>> >>>> System.out.println( "uri: " + uri ); >>>> Configuration conf = new Configuration(); >>>> >>>> FileSystem fs = FileSystem.get( URI.create( uri ), conf ); >>>> fs.printStatistics(); >>>> } catch (Exception e) { >>>> // TODO Auto-generated catch block >>>> e.printStackTrace(); >>>> } >>>> } >>>> >>>> The error message is : >>>> >>>> 11/10/28 19:03:38 INFO ipc.Client: Retrying connect to server: / >>>> 155.37.101.76:8020. Already tried 0 time(s). >>>> 11/10/28 19:03:39 INFO ipc.Client: Retrying connect to server: / >>>> 155.37.101.76:8020. Already tried 1 time(s). >>>> 11/10/28 19:03:40 INFO ipc.Client: Retrying connect to server: / >>>> 155.37.101.76:8020. Already tried 2 time(s). >>>> 11/10/28 19:03:41 INFO ipc.Client: Retrying connect to server: / >>>> 155.37.101.76:8020. Already tried 3 time(s). >>>> >>>> Any thoughts on this would be *really* be appreciated ... Thanks guys. >>>> >>> >> >> >> >> -- >> Jay Vyas >> MMSB/UCHC >>
