Yup.... Brutal :-| but you never regret fixing a bug ... Unlike ------- Sent from my iPad
On Oct 28, 2011, at 11:43 PM, Alex Gauthier <[email protected]> wrote: > Brutal Friday night. Coding < pussy. > > :) > > On Fri, Oct 28, 2011 at 8:43 PM, Alex Gauthier > <[email protected]>wrote: > >> >> >> On Fri, Oct 28, 2011 at 8:41 PM, Tom Melendez <[email protected]> wrote: >> >>> Hi Jay, >>> >>> Are you able to look at the logs or the web interface? Can you find >>> out why it's getting killed? >>> >>> Also, can you verify that these ports are open and a process is >>> connected to them (maybe with netstat)? >>> >>> http://www.cloudera.com/blog/2009/08/hadoop-default-ports-quick-reference/ >>> >>> Thanks, >>> >>> Tom >>> >>> On Fri, Oct 28, 2011 at 7:57 PM, Jay Vyas <[email protected]> wrote: >>>> Thanks tom : Thats interesting.... >>>> >>>> First, I tried, and it complained that the input directory didnt exist, >>> so I >>>> ran >>>> $> hadoop fs -mkdir /user/cloudera/input >>>> >>>> Then, I tried to do this : >>>> >>>> $> hadoop jar /usr/lib/hadoop-0.20/hadoop-examples.jar grep input >>> output2 >>>> 'dfs[a-z.]+' >>>> >>>> And it seemed to start working ...... But then it abruptly printed >>> "killed" >>>> somehow at the end of the job [scroll down] ? >>>> >>>> Maybe this is related to why i cant connect ..... ?! >>>> >>>> 1) the hadoop jar 11/10/14 21:34:43 WARN util.NativeCodeLoader: Unable >>> to >>>> load native-hadoop library for your platform... using builtin-java >>> classes >>>> where applicable >>>> 11/10/14 21:34:43 WARN snappy.LoadSnappy: Snappy native library not >>> loaded >>>> 11/10/14 21:34:43 INFO mapred.FileInputFormat: Total input paths to >>> process >>>> : 0 >>>> 11/10/14 21:34:44 INFO mapred.JobClient: Running job: >>> job_201110142010_0009 >>>> 11/10/14 21:34:45 INFO mapred.JobClient: map 0% reduce 0% >>>> 11/10/14 21:34:55 INFO mapred.JobClient: map 0% reduce 100% >>>> 11/10/14 21:34:57 INFO mapred.JobClient: Job complete: >>> job_201110142010_0009 >>>> 11/10/14 21:34:57 INFO mapred.JobClient: Counters: 14 >>>> 11/10/14 21:34:57 INFO mapred.JobClient: Job Counters >>>> 11/10/14 21:34:57 INFO mapred.JobClient: Launched reduce tasks=1 >>>> 11/10/14 21:34:57 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=5627 >>>> 11/10/14 21:34:57 INFO mapred.JobClient: Total time spent by all >>> reduces >>>> waiting after reserving slots (ms)=0 >>>> 11/10/14 21:34:57 INFO mapred.JobClient: Total time spent by all >>> maps >>>> waiting after reserving slots (ms)=0 >>>> 11/10/14 21:34:57 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=5050 >>>> 11/10/14 21:34:57 INFO mapred.JobClient: FileSystemCounters >>>> 11/10/14 21:34:57 INFO mapred.JobClient: FILE_BYTES_WRITTEN=53452 >>>> 11/10/14 21:34:57 INFO mapred.JobClient: HDFS_BYTES_WRITTEN=86 >>>> 11/10/14 21:34:57 INFO mapred.JobClient: Map-Reduce Framework >>>> 11/10/14 21:34:57 INFO mapred.JobClient: Reduce input groups=0 >>>> 11/10/14 21:34:57 INFO mapred.JobClient: Combine output records=0 >>>> 11/10/14 21:34:57 INFO mapred.JobClient: Reduce shuffle bytes=0 >>>> 11/10/14 21:34:57 INFO mapred.JobClient: Reduce output records=0 >>>> 11/10/14 21:34:57 INFO mapred.JobClient: Spilled Records=0 >>>> 11/10/14 21:34:57 INFO mapred.JobClient: Combine input records=0 >>>> 11/10/14 21:34:57 INFO mapred.JobClient: Reduce input records=0 >>>> 11/10/14 21:34:57 WARN mapred.JobClient: Use GenericOptionsParser for >>>> parsing the arguments. Applications should implement Tool for the same. >>>> 11/10/14 21:34:58 INFO mapred.FileInputFormat: Total input paths to >>> process >>>> : 1 >>>> 11/10/14 21:34:58 INFO mapred.JobClient: Running job: >>> job_201110142010_0010 >>>> 11/10/14 21:34:59 INFO mapred.JobClient: map 0% reduce 0% >>>> Killed >>>> >>>> >>>> On Fri, Oct 28, 2011 at 8:24 PM, Tom Melendez <[email protected]> wrote: >>>> >>>>> Hi Jay, >>>>> >>>>> Some questions for you: >>>>> >>>>> - Does the hadoop client itself work from that same machine? >>>>> - Are you actually able to run the hadoop example jar (in other words, >>>>> your setup is valid otherwise)? >>>>> - Is port 8020 actually available? (you can telnet or nc to it?) >>>>> - What does jps show on the namenode? >>>>> >>>>> Thanks, >>>>> >>>>> Tom >>>>> >>>>> On Fri, Oct 28, 2011 at 4:04 PM, Jay Vyas <[email protected]> >>> wrote: >>>>>> Hi guys : Made more progress debugging my hadoop connection, but >>> still >>>>>> haven't got it working...... It looks like my VM (cloudera hadoop) >>> won't >>>>>> let me in. I find that there is no issue connecting to the name node >>> - >>>>> that >>>>>> is , using hftp and 50070...... >>>>>> >>>>>> via standard HFTP as in here : >>>>>> >>>>>> //This method works fine - connecting directly to hadoop's namenode >>> and >>>>>> querying the filesystem >>>>>> public static void main1(String[] args) throws Exception >>>>>> { >>>>>> String uri = "hftp://155.37.101.76:50070/";; >>>>>> >>>>>> System.out.println( "uri: " + uri ); >>>>>> Configuration conf = new Configuration(); >>>>>> >>>>>> FileSystem fs = FileSystem.get( URI.create( uri ), conf ); >>>>>> fs.printStatistics(); >>>>>> } >>>>>> >>>>>> >>>>>> But unfortunately, I can't get into hdfs ..... Any thoughts on this ? >>> I >>>>> am >>>>>> modifying the uri to access port 8020 >>>>>> which is what is in my core-site.xml . >>>>>> >>>>>> // This fails, resulting (trys to connect over and over again, >>>>> eventually >>>>>> gives up printing "already tried to connect 20 times"....) >>>>>> public static void main(String[] args) >>>>>> { >>>>>> try { >>>>>> String uri = "hdfs://155.37.101.76:8020/"; >>>>>> >>>>>> System.out.println( "uri: " + uri ); >>>>>> Configuration conf = new Configuration(); >>>>>> >>>>>> FileSystem fs = FileSystem.get( URI.create( uri ), conf ); >>>>>> fs.printStatistics(); >>>>>> } catch (Exception e) { >>>>>> // TODO Auto-generated catch block >>>>>> e.printStackTrace(); >>>>>> } >>>>>> } >>>>>> >>>>>> The error message is : >>>>>> >>>>>> 11/10/28 19:03:38 INFO ipc.Client: Retrying connect to server: / >>>>>> 155.37.101.76:8020. Already tried 0 time(s). >>>>>> 11/10/28 19:03:39 INFO ipc.Client: Retrying connect to server: / >>>>>> 155.37.101.76:8020. Already tried 1 time(s). >>>>>> 11/10/28 19:03:40 INFO ipc.Client: Retrying connect to server: / >>>>>> 155.37.101.76:8020. Already tried 2 time(s). >>>>>> 11/10/28 19:03:41 INFO ipc.Client: Retrying connect to server: / >>>>>> 155.37.101.76:8020. Already tried 3 time(s). >>>>>> >>>>>> Any thoughts on this would be *really* be appreciated ... Thanks >>> guys. >>>>>> >>>>> >>>> >>>> >>>> >>>> -- >>>> Jay Vyas >>>> MMSB/UCHC >>>> >>> >> >>
