[
https://issues.apache.org/jira/browse/HADOOP-1001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tahir Hashmi updated HADOOP-1001:
---------------------------------
Attachment: TestMapOutputType.java
The problem with this test is that the exception thrown by MapTask doesn't get
propagated to the client. The best effort here is to just make sure that the
test cases fail for whatever reason when wrong key or value types are specified
and pass with correct types. Hopefully the earlier runs would've failed due to
type mismatch exceptions rather than something else. Attaching the revised
version of the test now. Here's the kind of output it produces in the logs:
$ cat build/test/TEST-org.apache.hadoop.mapred.TestMapOutputType.txt
Testsuite: org.apache.hadoop.mapred.TestMapOutputType
Tests run: 3, Failures: 0, Errors: 0, Time elapsed: 5.585 sec
------------- Standard Output ---------------
2007-04-03 00:18:02,162 WARN util.NativeCodeLoader
(NativeCodeLoader.java:<clinit>(51)) - Unable to load native-hadoop library for
your platform... using builtin-java classes where applicable
2007-04-03 00:18:02,508 INFO mapred.InputFormatBase
(InputFormatBase.java:validateInput(141)) - Total input paths to process : 1
2007-04-03 00:18:03,238 WARN mapred.LocalJobRunner
(LocalJobRunner.java:run(166)) - job_bzpgha
java.io.IOException: Type mismatch in key from map: expected
org.apache.hadoop.io.IntWritable, recieved org.apache.hadoop.io.Text
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:312)
at
org.apache.hadoop.mapred.TestMapOutputType$TextGen.map(TestMapOutputType.java:50)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:48)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:175)
at
org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:126)
2007-04-03 00:18:04,121 INFO mapred.InputFormatBase
(InputFormatBase.java:validateInput(141)) - Total input paths to process : 1
2007-04-03 00:18:04,249 WARN mapred.LocalJobRunner
(LocalJobRunner.java:run(166)) - job_4f7xh3
java.io.IOException: Type mismatch in value from map: expected
org.apache.hadoop.io.IntWritable, recieved org.apache.hadoop.io.Text
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:317)
at
org.apache.hadoop.mapred.TestMapOutputType$TextGen.map(TestMapOutputType.java:50)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:48)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:175)
at
org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:126)
2007-04-03 00:18:05,324 INFO mapred.InputFormatBase
(InputFormatBase.java:validateInput(141)) - Total input paths to process : 1
2007-04-03 00:18:05,604 INFO mapred.LocalJobRunner
(LocalJobRunner.java:progress(188)) -
file:/home/tahir/Desktop/Hadoop/build/test/test.mapred.spill/in/part0:0+150
2007-04-03 00:18:05,752 INFO mapred.LocalJobRunner
(LocalJobRunner.java:progress(188)) - reduce > reduce
------------- ---------------- ---------------
Testcase: testKeyMismatch took 3.144 sec
Testcase: testValueMismatch took 1.128 sec
Testcase: testNoMismatch took 1.302 sec
> the output of the map is not type checked against the specified types
> ---------------------------------------------------------------------
>
> Key: HADOOP-1001
> URL: https://issues.apache.org/jira/browse/HADOOP-1001
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.11.0
> Reporter: Owen O'Malley
> Assigned To: Tahir Hashmi
> Fix For: 0.13.0
>
> Attachments: 1001.patch, 1001.patch, 1001.patch,
> TestMapOutputType.java, TestMapOutputType.java
>
>
> The output of the map is not checked against the types specified in the
> JobConf leading to hard to diagnose bugs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
