I have found the error. It was a bug in my code.
On 09/24/2015 07:47 PM, xeonmailinglist wrote:
No, I am looking to the right place. Here is the output of a maptask.
I also thought that the print would come up, but it isn't showing.
That is why I am asking.
[1] Output from from one map task.
```
Log Type: stderr
Log Upload Time: 24-Sep-2015 12:45:19
Log Length: 317
Java HotSpot(TM) Client VM warning: You have loaded library
/home/xubuntu/Programs/hadoop-2.6.0/lib/native/libhadoop.so which might have
disabled stack guard. The VM will try to fix the stack guard now.
It's highly recommended that you fix the library with 'execstack -c <libfile>',
or link it with '-z noexecstack'.
Log Type: stdout
Log Upload Time: 24-Sep-2015 12:45:19
Log Length: 0
Log Type: syslog
Log Upload Time: 24-Sep-2015 12:45:19
Log Length: 2604
2015-09-24 12:45:04,569 WARN [main] org.apache.hadoop.util.NativeCodeLoader:
Unable to load native-hadoop library for your platform... using builtin-java
classes where applicable
2015-09-24 12:45:05,139 INFO [main]
org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from
hadoop-metrics2.properties
2015-09-24 12:45:05,412 INFO [main]
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period at
10 second(s).
2015-09-24 12:45:05,413 INFO [main]
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MapTask metrics system
started
2015-09-24 12:45:05,462 INFO [main] org.apache.hadoop.mapred.YarnChild:
Executing with tokens:
2015-09-24 12:45:05,463 INFO [main] org.apache.hadoop.mapred.YarnChild: Kind:
mapreduce.job, Service: job_1443113036547_0001, Ident:
(org.apache.hadoop.mapreduce.security.token.JobTokenIdentifier@1b5a082)
2015-09-24 12:45:05,847 INFO [main] org.apache.hadoop.mapred.YarnChild:
Sleeping for 0ms before retrying again. Got null now.
2015-09-24 12:45:06,915 INFO [main] org.apache.hadoop.mapred.YarnChild:
mapreduce.cluster.local.dir for child:
/tmp/hadoop-temp/nm-local-dir/usercache/xubuntu/appcache/application_1443113036547_0001
2015-09-24 12:45:07,604 INFO [main]
org.apache.hadoop.conf.Configuration.deprecation: session.id is deprecated.
Instead, use dfs.metrics.session-id
2015-09-24 12:45:09,402 INFO [main] org.apache.hadoop.mapred.Task: Using
ResourceCalculatorProcessTree : [ ]
2015-09-24 12:45:10,187 INFO [main] org.apache.hadoop.mapred.MapTask:
Processing split: hdfs://hadoop-coc-1:9000/input1/b.txt:0+21
2015-09-24 12:45:10,812 INFO [main] org.apache.hadoop.mapred.Task:
Task:attempt_1443113036547_0001_m_000000_0 is done. And is in the process of
committing
2015-09-24 12:45:10,969 INFO [main] org.apache.hadoop.mapred.Task: Task
attempt_1443113036547_0001_m_000000_0 is allowed to commit now
2015-09-24 12:45:10,993 INFO [main]
org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: Saved output of
task 'attempt_1443113036547_0001_m_000000_0' to
hdfs://192.168.10.110:9000/output1-1442847968/_temporary/1/task_1443113036547_0001_m_000000
2015-09-24 12:45:11,135 INFO [main] org.apache.hadoop.mapred.Task: Task
'attempt_1443113036547_0001_m_000000_0' done.
2015-09-24 12:45:11,135 INFO [main]
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping MapTask metrics
system...
2015-09-24 12:45:11,136 INFO [main]
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MapTask metrics system
stopped.
2015-09-24 12:45:11,136 INFO [main]
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MapTask metrics system
shutdown complete.
```
On 09/24/2015 06:42 PM, Peyman Mohajerian wrote:
You log statement should show up in the container log for that Map
class within the data node. I'm guessing you aren't looking in the
right place.
On Thu, Sep 24, 2015 at 9:54 AM, xeonmailinglist
<xeonmailingl...@gmail.com <mailto:xeonmailingl...@gmail.com>> wrote:
Does anyone know this question about logging in MapReduce. I
can't find an example or an explanation on how to print something
inside user map and reduce functions.
On 09/24/2015 04:19 PM, xeonmailinglist wrote:
Hi,
1.
I have this example of MapReduce [1], and I want to print
info in the stdout and in a log file. It seems that the logs
isn’t print anything. How can I make my class print these words?
2.
I also have set in the |yarn-site.xml| to retain log.
Although the logs are retained in the |/app-logs| dir, the
|userlogs| dir is deleted at the end of the job execution.
How can I make MapReduce to not delete files in the
|userlogs| dir?
I am using Yarn.
Thanks,
[1] Wordcount exampla with just the map part.
|public class MyWordCount { public static class MyMap extends
Mapper { Log log = LogFactory.getLog(MyWordCount.class); private
final static IntWritable one = new IntWritable(1); private Text
word = new Text(); public void map(LongWritable key, Text value,
OutputCollector<Text, IntWritable> output, Reporter reporter)
throws IOException { StringTokenizer itr = new
StringTokenizer(value.toString()); System.out.println("HERRE");
log.info <http://log.info>("HERRRRRE"); while
(itr.hasMoreTokens()) { word.set(itr.nextToken());
output.collect(word, one); } } public void run(Context context)
throws IOException, InterruptedException { setup(context); try {
while (context.nextKeyValue()) { System.out.println("Key: " +
context.getCurrentKey() + " Value: " +
context.getCurrentValue()); map(context.getCurrentKey(),
context.getCurrentValue(), context); } } finally {
cleanup(context); } } public void cleanup(Mapper.Context
context) {} } |
[2] yarn-site.xml
|<!-- job history --> <property>
<name>yarn.log-aggregation-enable</name> <value>true</value>
</property> <property>
<name>yarn.nodemanager.log.retain-seconds</name>
<value>900000</value> </property> <property>
<name>yarn.nodemanager.remote-app-log-dir</name>
<value>/app-logs</value> </property> |
