Ok thanks, I'll post there. I realized that the issue has to do with the extra jars that I added to the hadoop installation.
My job wasnt getting submitted becauae my tasktrackers don't seem to start if I have avro-1.7.0.jar & avro-tools-1.7.0.jar in my hadoop/lib directory. But I need these jars in order to write the avro file. Alan Sent from my iPhone On Jul 4, 2012, at 19:21, "Arun C Murthy" <a...@hortonworks.com<mailto:a...@hortonworks.com>> wrote: It's hard for folks here to help you on CDH - please ask their own user lists. Arun On Jul 4, 2012, at 8:49 AM, Alan Miller wrote: Hi, I’m trying to move from CDH3U3 to CDH4. My existing MR program works fine on CDH3U3 but I cant get it to run on CDH4. Basically my Driver class 1. queries a PG DB and writes some HashMaps to files in the Distributed Cache, 2. then writes some Avro files (avro 1.7.0) to HDFS, 3. and then triggers a MRv1 job to process the Avro files. The DC & Avro files get written so HDFS is working, but my job is not getting started. I get an error: Exception in thread "main" java.lang.NoSuchMethodError: org.apache.hadoop.ipc.RPC.getProxy() … at org.apache.hadoop.mapred.JobClient.createRPCProxy(JobClient.java:478) Seems the job doesn’t even get accepted . At MyDriver:397 (see below) I call job.submit, but that’s it. ….. 12/07/04 08:16:54 INFO MyDriver: Processing data [my-data] 12/07/04 08:16:54 INFO MyDriver: Write Avro: [Tue Jul 03 00:00:00 PDT 2012 > etime <= Tue Jul 03 23:59:59 PDT 2012] 12/07/04 08:19:47 INFO MyDriver: Initialized file /data/in/my-data_2012-07-03.avro 12/07/04 08:19:49 INFO MyDriver: Read 158285 lines, Wrote 158131 records to 1 file(s) 12/07/04 08:19:49 INFO MyDriver: Wed Jul 04 08:19:49 PDT 2012 Finished avro data /data/in/my-data_2012-07-03.avro 12/07/04 08:19:49 INFO MyDriver: Added /data/cache/fd/83206b5c-8a1c-46f3-bfb2-d8c3e949a530#q_map to distributed cache. 12/07/04 08:19:49 INFO MyDriver: Added /data/cache/fd/b2ebfeb9-bdb0-489e-8186-8e18f4416224#u_map to distributed cache. 12/07/04 08:19:49 INFO MyDriver: Added /data/cache/fd/437cfd91-aa07-4c3a-b4c9-cd4ae076f7ad#r_map to distributed cache. 12/07/04 08:19:49 INFO MyDriver: Added /data/cache/fd/9554fe48-2171-423c-ba54-6249ffc882d4#m_map to distributed cache. 12/07/04 08:19:49 INFO MyDriver: Added /data/in/y-data_2012-07-03.avro to input files list. Exception in thread "main" java.lang.NoSuchMethodError: org.apache.hadoop.ipc.RPC.getProxy(Ljava/lang/Class;JLjava/net/InetSocketAddress;Lorg/apache/hadoop/security/UserGroupInformation;Lorg/apache/hadoop/conf/Configuration;Ljavax/net/SocketFactory;)Lorg/apache/hadoop/ipc/VersionedProtocol; at org.apache.hadoop.mapred.JobClient.createRPCProxy(JobClient.java:478) at org.apache.hadoop.mapred.JobClient.init(JobClient.java:472) at org.apache.hadoop.mapred.JobClient.<init>(JobClient.java:455) at org.apache.hadoop.mapreduce.Job$1.run(Job.java:478) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232) at org.apache.hadoop.mapreduce.Job.connect(Job.java:476) at org.apache.hadoop.mapreduce.Job.submit(Job.java:464) at com.mycompany.MyDriver.run(MyDriver.java:397) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84) at com.mycompany.MyDriver.runHadoopJob(MyDriver.java:308) at com.mycompany.MyDriver.main(MyDriver.java:1532) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.RunJar.main(RunJar.java:208) Alan -- Arun C. Murthy Hortonworks Inc. http://hortonworks.com/
