Hi Anil,
Iam using chd4 with yarn.
On Sun, Jul 29, 2012 at 3:17 PM, Anil Gupta <anilgupt...@gmail.com> wrote:
> Are you using cdh4? In you cluster are you using yarn or mr1?
> Check the classpath of Hadoop by Hadoop classpath command.
>
> Best Regards,
> Anil
>
> On Jul 29, 2012, at 12:12 PM, abhiTowson cal <abhishek.dod...@gmail.com>
> wrote:
>
>> HI Anil,
>>
>> I have already tried this,but issue could not be resolved.
>>
>> Regards
>> Abhishek
>>
>> On Sun, Jul 29, 2012 at 3:05 PM, anil gupta <anilgupt...@gmail.com> wrote:
>>> Hi Abhishek,
>>>
>>> Once you make sure that whatever Harsh said in the previous email is
>>> present in the cluster and then also the job runs in Local Mode. Then try
>>> running the job with hadoop --config option.
>>> Refer to this discussion for more detail:
>>> https://groups.google.com/a/cloudera.org/forum/#!topic/cdh-user/4tMGfvJFzrg
>>>
>>> HTH,
>>> Anil
>>>
>>> On Sun, Jul 29, 2012 at 11:43 AM, Harsh J <ha...@cloudera.com> wrote:
>>>
>>>> For a job to get submitted to a cluster, you will need proper client
>>>> configurations. Have you configured your mapred-site.xml and
>>>> yarn-site.xml properly inside /etc/hadoop/conf/mapred-site.xml and
>>>> /etc/hadoop/conf/yarn-site.xml at the client node?
>>>>
>>>> On Mon, Jul 30, 2012 at 12:00 AM, abhiTowson cal
>>>> <abhishek.dod...@gmail.com> wrote:
>>>>> Hi All,
>>>>>
>>>>> I am getting problem that job is running in localrunner rather than
>>>>> the cluster enviormnent.
>>>>> And when am running the job i would not be able to see the job id in
>>>>> the resource manager UI
>>>>>
>>>>> Can you please go through the issues and let me know ASAP.
>>>>>
>>>>> sudo -u hdfs hadoop jar
>>>>> /usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar teragen
>>>>> 1000000 /benchmark/teragen/input
>>>>> 12/07/29 13:35:59 WARN conf.Configuration: session.id is deprecated.
>>>>> Instead, use dfs.metrics.session-id
>>>>> 12/07/29 13:35:59 INFO jvm.JvmMetrics: Initializing JVM Metrics with
>>>>> processName=JobTracker, sessionId=
>>>>> 12/07/29 13:35:59 INFO util.NativeCodeLoader: Loaded the native-hadoop
>>>> library
>>>>> 12/07/29 13:35:59 WARN mapred.JobClient: Use GenericOptionsParser for
>>>>> parsing the arguments. Applications should implement Tool for the
>>>>> same.
>>>>> Generating 1000000 using 1 maps with step of 1000000
>>>>> 12/07/29 13:35:59 INFO mapred.JobClient: Running job: job_local_0001
>>>>> 12/07/29 13:35:59 INFO mapred.LocalJobRunner: OutputCommitter set in
>>>> config null
>>>>> 12/07/29 13:35:59 INFO mapred.LocalJobRunner: OutputCommitter is
>>>>> org.apache.hadoop.mapred.FileOutputCommitter
>>>>> 12/07/29 13:35:59 WARN mapreduce.Counters: Group
>>>>> org.apache.hadoop.mapred.Task$Counter is deprecated. Use
>>>>> org.apache.hadoop.mapreduce.TaskCounter instead
>>>>> 12/07/29 13:35:59 INFO util.ProcessTree: setsid exited with exit code 0
>>>>> 12/07/29 13:35:59 INFO mapred.Task: Using ResourceCalculatorPlugin :
>>>>> org.apache.hadoop.util.LinuxResourceCalculatorPlugin@47c297a3
>>>>> 12/07/29 13:36:00 WARN mapreduce.Counters: Counter name
>>>>> MAP_INPUT_BYTES is deprecated. Use FileInputFormatCounters as group
>>>>> name and BYTES_READ as counter name instead
>>>>> 12/07/29 13:36:00 INFO mapred.MapTask: numReduceTasks: 0
>>>>> 12/07/29 13:36:00 INFO mapred.JobClient: map 0% reduce 0%
>>>>> 12/07/29 13:36:01 INFO mapred.Task: Task:attempt_local_0001_m_000000_0
>>>>> is done. And is in the process of commiting
>>>>> 12/07/29 13:36:01 INFO mapred.LocalJobRunner:
>>>>> 12/07/29 13:36:01 INFO mapred.Task: Task attempt_local_0001_m_000000_0
>>>>> is allowed to commit now
>>>>> 12/07/29 13:36:01 INFO mapred.FileOutputCommitter: Saved output of
>>>>> task 'attempt_local_0001_m_000000_0' to
>>>>> hdfs://hadoop-master-1/benchmark/teragen/input
>>>>> 12/07/29 13:36:01 INFO mapred.LocalJobRunner:
>>>>> 12/07/29 13:36:01 INFO mapred.Task: Task 'attempt_local_0001_m_000000_0'
>>>> done.
>>>>> 12/07/29 13:36:02 INFO mapred.JobClient: map 100% reduce 0%
>>>>> 12/07/29 13:36:02 INFO mapred.JobClient: Job complete: job_local_0001
>>>>> 12/07/29 13:36:02 INFO mapred.JobClient: Counters: 19
>>>>> 12/07/29 13:36:02 INFO mapred.JobClient: File System Counters
>>>>> 12/07/29 13:36:02 INFO mapred.JobClient: FILE: Number of bytes
>>>> read=142686
>>>>> 12/07/29 13:36:02 INFO mapred.JobClient: FILE: Number of bytes
>>>>> written=220956
>>>>> 12/07/29 13:36:02 INFO mapred.JobClient: FILE: Number of read
>>>> operations=0
>>>>> 12/07/29 13:36:02 INFO mapred.JobClient: FILE: Number of large
>>>>> read operations=0
>>>>> 12/07/29 13:36:02 INFO mapred.JobClient: FILE: Number of write
>>>> operations=0
>>>>> 12/07/29 13:36:02 INFO mapred.JobClient: HDFS: Number of bytes read=0
>>>>> 12/07/29 13:36:02 INFO mapred.JobClient: HDFS: Number of bytes
>>>>> written=100000000
>>>>> 12/07/29 13:36:02 INFO mapred.JobClient: HDFS: Number of read
>>>> operations=1
>>>>> 12/07/29 13:36:02 INFO mapred.JobClient: HDFS: Number of large
>>>>> read operations=0
>>>>> 12/07/29 13:36:02 INFO mapred.JobClient: HDFS: Number of write
>>>> operations=2
>>>>> 12/07/29 13:36:02 INFO mapred.JobClient: Map-Reduce Framework
>>>>> 12/07/29 13:36:02 INFO mapred.JobClient: Map input records=1000000
>>>>> 12/07/29 13:36:02 INFO mapred.JobClient: Map output records=1000000
>>>>> 12/07/29 13:36:02 INFO mapred.JobClient: Input split bytes=82
>>>>> 12/07/29 13:36:02 INFO mapred.JobClient: Spilled Records=0
>>>>> 12/07/29 13:36:02 INFO mapred.JobClient: CPU time spent (ms)=0
>>>>> 12/07/29 13:36:02 INFO mapred.JobClient: Physical memory (bytes)
>>>> snapshot=0
>>>>> 12/07/29 13:36:02 INFO mapred.JobClient: Virtual memory (bytes)
>>>> snapshot=0
>>>>> 12/07/29 13:36:02 INFO mapred.JobClient: Total committed heap
>>>>> usage (bytes)=124715008
>>>>> 12/07/29 13:36:02 INFO mapred.JobClient:
>>>>> org.apache.hadoop.mapreduce.lib.input.FileInputFormatCounter
>>>>>
>>>>> Regards
>>>>> Abhishek
>>>>
>>>>
>>>>
>>>> --
>>>> Harsh J
>>>>
>>>
>>>
>>>
>>> --
>>> Thanks & Regards,
>>> Anil Gupta
