[jira] [Commented] (HADOOP-9295) AbstractMapWritable throws exception when calling readFields() multiple times when the maps contain different class types
[ https://issues.apache.org/jira/browse/HADOOP-9295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13611535#comment-13611535 ] Karthik Kambatla commented on HADOOP-9295: -- Thanks [~davidparks21], and sorry for the delay. I understand the bug now. Let me update the test and work on a patch. AbstractMapWritable throws exception when calling readFields() multiple times when the maps contain different class types - Key: HADOOP-9295 URL: https://issues.apache.org/jira/browse/HADOOP-9295 Project: Hadoop Common Issue Type: Bug Components: io Affects Versions: 1.0.3 Reporter: David Parks Assignee: Karthik Kambatla Priority: Critical Attachments: MapWritableBugTest.java, test-hadoop-9295.patch Verified the trunk looks the same as 1.0.3 for this issue. When mappers output MapWritables with different class types, then they are read in on the Reducer via an iterator (multiple calls to readFields without instantiating a new object) you'll get this: java.lang.IllegalArgumentException: Id 1 exists but maps to org.me.ClassTypeOne and not org.me.ClassTypeTwo at org.apache.hadoop.io.AbstractMapWritable.addToMap(AbstractMapWritable.java:73) at org.apache.hadoop.io.AbstractMapWritable.readFields(AbstractMapWritable.java:201) It happens because AbstractMapWritable accumulates class type entries in its ClassType to ID (and vice versa) hashmaps. Those accumulating classtype-to-id hashmaps need to be cleared to support multiple calls to readFields(). I've attached a JUnit test that both demonstrates the problem and contains an embedded, fixed version of MapWritable and ArrayMapWritable (note the //TODO comments in the code where it was fixed in 2 places). If there's a better way to submit this recommended bug fix, someone please feel free to let me know. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9295) AbstractMapWritable throws exception when calling readFields() multiple times when the maps contain different class types
[
https://issues.apache.org/jira/browse/HADOOP-9295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13611545#comment-13611545
]
Karthik Kambatla commented on HADOOP-9295:
--
Actually, I don't think the test MapWritableBugTest#reproduceMapWritableBug()
tests adding MapWritable objects either. Also, if X and Y are the two different
classes named MapWritable, isn't X.readFields(Y.write(out)) expected to fail?
Shouldn't it be as below (there are two objects map1 and map2 is not because we
are calling readFields twice, but primarily because they are two different
MapWritables):
{code}
org.apache.hadoop.io.MapWritable map1 = new org.apache.hadoop.io.MapWritable();
map1.readFields(in);
Assert.assertTrue( map1.get(new Text(testKey1)) instanceof CustomWritableOne
);
MapWritable map1 = new MapWritable();
map2.readFields(in);
Assert.assertTrue( map2.get(new Text(testKey2)) instanceof CustomWritableTwo
);
{code}
AbstractMapWritable throws exception when calling readFields() multiple times
when the maps contain different class types
-
Key: HADOOP-9295
URL: https://issues.apache.org/jira/browse/HADOOP-9295
Project: Hadoop Common
Issue Type: Bug
Components: io
Affects Versions: 1.0.3
Reporter: David Parks
Assignee: Karthik Kambatla
Priority: Critical
Attachments: MapWritableBugTest.java, test-hadoop-9295.patch
Verified the trunk looks the same as 1.0.3 for this issue.
When mappers output MapWritables with different class types, then they are
read in on the Reducer via an iterator (multiple calls to readFields without
instantiating a new object) you'll get this:
java.lang.IllegalArgumentException: Id 1 exists but maps to
org.me.ClassTypeOne and not org.me.ClassTypeTwo
at
org.apache.hadoop.io.AbstractMapWritable.addToMap(AbstractMapWritable.java:73)
at
org.apache.hadoop.io.AbstractMapWritable.readFields(AbstractMapWritable.java:201)
It happens because AbstractMapWritable accumulates class type entries in its
ClassType to ID (and vice versa) hashmaps.
Those accumulating classtype-to-id hashmaps need to be cleared to support
multiple calls to readFields().
I've attached a JUnit test that both demonstrates the problem and contains an
embedded, fixed version of MapWritable and ArrayMapWritable (note the //TODO
comments in the code where it was fixed in 2 places).
If there's a better way to submit this recommended bug fix, someone please
feel free to let me know.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9295) AbstractMapWritable throws exception when calling readFields() multiple times when the maps contain different class types
[ https://issues.apache.org/jira/browse/HADOOP-9295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13611557#comment-13611557 ] David Parks commented on HADOOP-9295: - Haha, this gets confusing, you had me thinking I totally muffed up that testcase for a few minutes. The test case, and the reason I created 2 *different* MapWritable objects, serves to emulate what is happening in a hadoop Map/Reduce phase (where I actually ran into the bug). Take this example MapReduce job with 2 mappers: MapperX adds a new MapWritable with: (key - testKey1, value - CustomWritableOne) MapperY adds a new MapWritable with: (key - testKey2, value - CustomWritableTwo) Now the mappers would have done: MapperX: context.write( new Text(commonkey), mapWritableWithTestKey1 ); MapperY: context.write( new Text(commonkey), mapWritableWithTestKey2 ); In the reducer we now have an Iterator of MapWritable objects that contain 2 distinct object types that we wanted to join on commonkey. The use of testKey1 vs. testKey2 as the MapWritable key allows our reducer to identify which data type is contained in this MapWritable. This seems like a quick way to join two differently typed Writable objects for processing in the reducer. The problem happens when we read mapWritableWithTestKey1 in, then use the same object to read mapWritableWithTestKey2 in, which is exactly what the reducer is doing. AbstractMapWritable throws exception when calling readFields() multiple times when the maps contain different class types - Key: HADOOP-9295 URL: https://issues.apache.org/jira/browse/HADOOP-9295 Project: Hadoop Common Issue Type: Bug Components: io Affects Versions: 1.0.3 Reporter: David Parks Assignee: Karthik Kambatla Priority: Critical Attachments: MapWritableBugTest.java, test-hadoop-9295.patch Verified the trunk looks the same as 1.0.3 for this issue. When mappers output MapWritables with different class types, then they are read in on the Reducer via an iterator (multiple calls to readFields without instantiating a new object) you'll get this: java.lang.IllegalArgumentException: Id 1 exists but maps to org.me.ClassTypeOne and not org.me.ClassTypeTwo at org.apache.hadoop.io.AbstractMapWritable.addToMap(AbstractMapWritable.java:73) at org.apache.hadoop.io.AbstractMapWritable.readFields(AbstractMapWritable.java:201) It happens because AbstractMapWritable accumulates class type entries in its ClassType to ID (and vice versa) hashmaps. Those accumulating classtype-to-id hashmaps need to be cleared to support multiple calls to readFields(). I've attached a JUnit test that both demonstrates the problem and contains an embedded, fixed version of MapWritable and ArrayMapWritable (note the //TODO comments in the code where it was fixed in 2 places). If there's a better way to submit this recommended bug fix, someone please feel free to let me know. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9295) AbstractMapWritable throws exception when calling readFields() multiple times when the maps contain different class types
[ https://issues.apache.org/jira/browse/HADOOP-9295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13611558#comment-13611558 ] David Parks commented on HADOOP-9295: - How did you highlight your code like that anyway? AbstractMapWritable throws exception when calling readFields() multiple times when the maps contain different class types - Key: HADOOP-9295 URL: https://issues.apache.org/jira/browse/HADOOP-9295 Project: Hadoop Common Issue Type: Bug Components: io Affects Versions: 1.0.3 Reporter: David Parks Assignee: Karthik Kambatla Priority: Critical Attachments: MapWritableBugTest.java, test-hadoop-9295.patch Verified the trunk looks the same as 1.0.3 for this issue. When mappers output MapWritables with different class types, then they are read in on the Reducer via an iterator (multiple calls to readFields without instantiating a new object) you'll get this: java.lang.IllegalArgumentException: Id 1 exists but maps to org.me.ClassTypeOne and not org.me.ClassTypeTwo at org.apache.hadoop.io.AbstractMapWritable.addToMap(AbstractMapWritable.java:73) at org.apache.hadoop.io.AbstractMapWritable.readFields(AbstractMapWritable.java:201) It happens because AbstractMapWritable accumulates class type entries in its ClassType to ID (and vice versa) hashmaps. Those accumulating classtype-to-id hashmaps need to be cleared to support multiple calls to readFields(). I've attached a JUnit test that both demonstrates the problem and contains an embedded, fixed version of MapWritable and ArrayMapWritable (note the //TODO comments in the code where it was fixed in 2 places). If there's a better way to submit this recommended bug fix, someone please feel free to let me know. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9295) AbstractMapWritable throws exception when calling readFields() multiple times when the maps contain different class types
[
https://issues.apache.org/jira/browse/HADOOP-9295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13611587#comment-13611587
]
Karthik Kambatla commented on HADOOP-9295:
--
Enclosing code in { code } without spaces highlights it. :)
IIUC, the issue then is with composite MapWritables?
AbstractMapWritable throws exception when calling readFields() multiple times
when the maps contain different class types
-
Key: HADOOP-9295
URL: https://issues.apache.org/jira/browse/HADOOP-9295
Project: Hadoop Common
Issue Type: Bug
Components: io
Affects Versions: 1.0.3
Reporter: David Parks
Assignee: Karthik Kambatla
Priority: Critical
Attachments: MapWritableBugTest.java, test-hadoop-9295.patch
Verified the trunk looks the same as 1.0.3 for this issue.
When mappers output MapWritables with different class types, then they are
read in on the Reducer via an iterator (multiple calls to readFields without
instantiating a new object) you'll get this:
java.lang.IllegalArgumentException: Id 1 exists but maps to
org.me.ClassTypeOne and not org.me.ClassTypeTwo
at
org.apache.hadoop.io.AbstractMapWritable.addToMap(AbstractMapWritable.java:73)
at
org.apache.hadoop.io.AbstractMapWritable.readFields(AbstractMapWritable.java:201)
It happens because AbstractMapWritable accumulates class type entries in its
ClassType to ID (and vice versa) hashmaps.
Those accumulating classtype-to-id hashmaps need to be cleared to support
multiple calls to readFields().
I've attached a JUnit test that both demonstrates the problem and contains an
embedded, fixed version of MapWritable and ArrayMapWritable (note the //TODO
comments in the code where it was fixed in 2 places).
If there's a better way to submit this recommended bug fix, someone please
feel free to let me know.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9295) AbstractMapWritable throws exception when calling readFields() multiple times when the maps contain different class types
[ https://issues.apache.org/jira/browse/HADOOP-9295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13611596#comment-13611596 ] David Parks commented on HADOOP-9295: - Correct, MapWritable's that contain at least 2 other ? extends Writable objects that aren't already in that pre-defined list we have earlier in the conversation. AbstractMapWritable throws exception when calling readFields() multiple times when the maps contain different class types - Key: HADOOP-9295 URL: https://issues.apache.org/jira/browse/HADOOP-9295 Project: Hadoop Common Issue Type: Bug Components: io Affects Versions: 1.0.3 Reporter: David Parks Assignee: Karthik Kambatla Priority: Critical Attachments: MapWritableBugTest.java, test-hadoop-9295.patch Verified the trunk looks the same as 1.0.3 for this issue. When mappers output MapWritables with different class types, then they are read in on the Reducer via an iterator (multiple calls to readFields without instantiating a new object) you'll get this: java.lang.IllegalArgumentException: Id 1 exists but maps to org.me.ClassTypeOne and not org.me.ClassTypeTwo at org.apache.hadoop.io.AbstractMapWritable.addToMap(AbstractMapWritable.java:73) at org.apache.hadoop.io.AbstractMapWritable.readFields(AbstractMapWritable.java:201) It happens because AbstractMapWritable accumulates class type entries in its ClassType to ID (and vice versa) hashmaps. Those accumulating classtype-to-id hashmaps need to be cleared to support multiple calls to readFields(). I've attached a JUnit test that both demonstrates the problem and contains an embedded, fixed version of MapWritable and ArrayMapWritable (note the //TODO comments in the code where it was fixed in 2 places). If there's a better way to submit this recommended bug fix, someone please feel free to let me know. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9295) AbstractMapWritable throws exception when calling readFields() multiple times when the maps contain different class types
[
https://issues.apache.org/jira/browse/HADOOP-9295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13611600#comment-13611600
]
David Parks commented on HADOOP-9295:
-
Note that the fix is in there as well, it's as simple as a clear() method, and
removing the pre-cached list of common classes (which interferes with our
ability to clear() properly).
Look for the comments to call out the 2 changes (2 lines added, 33 lines
removed):
{code}
/*
* TODO BUG FIX
*/
{code}
AbstractMapWritable throws exception when calling readFields() multiple times
when the maps contain different class types
-
Key: HADOOP-9295
URL: https://issues.apache.org/jira/browse/HADOOP-9295
Project: Hadoop Common
Issue Type: Bug
Components: io
Affects Versions: 1.0.3
Reporter: David Parks
Assignee: Karthik Kambatla
Priority: Critical
Attachments: MapWritableBugTest.java, test-hadoop-9295.patch
Verified the trunk looks the same as 1.0.3 for this issue.
When mappers output MapWritables with different class types, then they are
read in on the Reducer via an iterator (multiple calls to readFields without
instantiating a new object) you'll get this:
java.lang.IllegalArgumentException: Id 1 exists but maps to
org.me.ClassTypeOne and not org.me.ClassTypeTwo
at
org.apache.hadoop.io.AbstractMapWritable.addToMap(AbstractMapWritable.java:73)
at
org.apache.hadoop.io.AbstractMapWritable.readFields(AbstractMapWritable.java:201)
It happens because AbstractMapWritable accumulates class type entries in its
ClassType to ID (and vice versa) hashmaps.
Those accumulating classtype-to-id hashmaps need to be cleared to support
multiple calls to readFields().
I've attached a JUnit test that both demonstrates the problem and contains an
embedded, fixed version of MapWritable and ArrayMapWritable (note the //TODO
comments in the code where it was fixed in 2 places).
If there's a better way to submit this recommended bug fix, someone please
feel free to let me know.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9295) AbstractMapWritable throws exception when calling readFields() multiple times when the maps contain different class types
[
https://issues.apache.org/jira/browse/HADOOP-9295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13611604#comment-13611604
]
David Parks commented on HADOOP-9295:
-
specifically, add a call to clear:
{code}
/*
* TODO BUG FIX
*/
classToIdMap.clear();
idToClassMap.clear();
{code}
and remove the pre-cached class types
{code}
/*
* TODO BUG FIX
*/
// addToMap(ArrayWritable.class,
// Byte.valueOf(Integer.valueOf(-127).byteValue()));
// addToMap(BooleanWritable.class,
// Byte.valueOf(Integer.valueOf(-126).byteValue()));
// addToMap(BytesWritable.class,
// Byte.valueOf(Integer.valueOf(-125).byteValue()));
// addToMap(FloatWritable.class,
// Byte.valueOf(Integer.valueOf(-124).byteValue()));
// addToMap(IntWritable.class,
// Byte.valueOf(Integer.valueOf(-123).byteValue()));
// addToMap(LongWritable.class,
// Byte.valueOf(Integer.valueOf(-122).byteValue()));
// addToMap(MapWritable.class,
// Byte.valueOf(Integer.valueOf(-121).byteValue()));
// addToMap(MD5Hash.class,
// Byte.valueOf(Integer.valueOf(-120).byteValue()));
// addToMap(NullWritable.class,
// Byte.valueOf(Integer.valueOf(-119).byteValue()));
// addToMap(ObjectWritable.class,
// Byte.valueOf(Integer.valueOf(-118).byteValue()));
// addToMap(SortedMapWritable.class,
// Byte.valueOf(Integer.valueOf(-117).byteValue()));
// addToMap(Text.class,
// Byte.valueOf(Integer.valueOf(-116).byteValue()));
// addToMap(TwoDArrayWritable.class,
// Byte.valueOf(Integer.valueOf(-115).byteValue()));
//
// // UTF8 is deprecated so we don't support it
//
// addToMap(VIntWritable.class,
// Byte.valueOf(Integer.valueOf(-114).byteValue()));
// addToMap(VLongWritable.class,
// Byte.valueOf(Integer.valueOf(-113).byteValue()));
{code}
AbstractMapWritable throws exception when calling readFields() multiple times
when the maps contain different class types
-
Key: HADOOP-9295
URL: https://issues.apache.org/jira/browse/HADOOP-9295
Project: Hadoop Common
Issue Type: Bug
Components: io
Affects Versions: 1.0.3
Reporter: David Parks
Assignee: Karthik Kambatla
Priority: Critical
Attachments: MapWritableBugTest.java, test-hadoop-9295.patch
Verified the trunk looks the same as 1.0.3 for this issue.
When mappers output MapWritables with different class types, then they are
read in on the Reducer via an iterator (multiple calls to readFields without
instantiating a new object) you'll get this:
java.lang.IllegalArgumentException: Id 1 exists but maps to
org.me.ClassTypeOne and not org.me.ClassTypeTwo
at
org.apache.hadoop.io.AbstractMapWritable.addToMap(AbstractMapWritable.java:73)
at
org.apache.hadoop.io.AbstractMapWritable.readFields(AbstractMapWritable.java:201)
It happens because AbstractMapWritable accumulates class type entries in its
ClassType to ID (and vice versa) hashmaps.
Those accumulating classtype-to-id hashmaps need to be cleared to support
multiple calls to readFields().
I've attached a JUnit test that both demonstrates the problem and contains an
embedded, fixed version of MapWritable and ArrayMapWritable (note the //TODO
comments in the code where it was fixed in 2 places).
If there's a better way to submit this recommended bug fix, someone please
feel free to let me know.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9295) AbstractMapWritable throws exception when calling readFields() multiple times when the maps contain different class types
[ https://issues.apache.org/jira/browse/HADOOP-9295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13606271#comment-13606271 ] David Parks commented on HADOOP-9295: - Your test missed the problem. You added two Text objects, you need to add 2 custom MapWrtiable objects to the map to trigger the problem. The reason explained: Notice this code in AbstractMapWritable: addToMap(ArrayWritable.class, Byte.valueOf(Integer.valueOf(-127).byteValue())); addToMap(BooleanWritable.class, Byte.valueOf(Integer.valueOf(-126).byteValue())); addToMap(BytesWritable.class, Byte.valueOf(Integer.valueOf(-125).byteValue())); addToMap(FloatWritable.class, Byte.valueOf(Integer.valueOf(-124).byteValue())); addToMap(IntWritable.class, Byte.valueOf(Integer.valueOf(-123).byteValue())); addToMap(LongWritable.class, Byte.valueOf(Integer.valueOf(-122).byteValue())); addToMap(MapWritable.class, Byte.valueOf(Integer.valueOf(-121).byteValue())); addToMap(MD5Hash.class, Byte.valueOf(Integer.valueOf(-120).byteValue())); addToMap(NullWritable.class, Byte.valueOf(Integer.valueOf(-119).byteValue())); addToMap(ObjectWritable.class, Byte.valueOf(Integer.valueOf(-118).byteValue())); addToMap(SortedMapWritable.class, Byte.valueOf(Integer.valueOf(-117).byteValue())); addToMap(Text.class, Byte.valueOf(Integer.valueOf(-116).byteValue())); addToMap(TwoDArrayWritable.class, Byte.valueOf(Integer.valueOf(-115).byteValue())); // UTF8 is deprecated so we don't support it addToMap(VIntWritable.class, Byte.valueOf(Integer.valueOf(-114).byteValue())); addToMap(VLongWritable.class, Byte.valueOf(Integer.valueOf(-113).byteValue())); It's adding the typical Writables to the class map by default, so any of these classes always maps correctly, this is probably why the problem was never noticed before now. It's only when you add a Writable object that isn't already in this list that it has to add it to the map, and thus encounters the bug. AbstractMapWritable throws exception when calling readFields() multiple times when the maps contain different class types - Key: HADOOP-9295 URL: https://issues.apache.org/jira/browse/HADOOP-9295 Project: Hadoop Common Issue Type: Bug Components: io Affects Versions: 1.0.3 Reporter: David Parks Assignee: Karthik Kambatla Priority: Critical Attachments: MapWritableBugTest.java, test-hadoop-9295.patch Verified the trunk looks the same as 1.0.3 for this issue. When mappers output MapWritables with different class types, then they are read in on the Reducer via an iterator (multiple calls to readFields without instantiating a new object) you'll get this: java.lang.IllegalArgumentException: Id 1 exists but maps to org.me.ClassTypeOne and not org.me.ClassTypeTwo at org.apache.hadoop.io.AbstractMapWritable.addToMap(AbstractMapWritable.java:73) at org.apache.hadoop.io.AbstractMapWritable.readFields(AbstractMapWritable.java:201) It happens because AbstractMapWritable accumulates class type entries in its ClassType to ID (and vice versa) hashmaps. Those accumulating classtype-to-id hashmaps need to be cleared to support multiple calls to readFields(). I've attached a JUnit test that both demonstrates the problem and contains an embedded, fixed version of MapWritable and ArrayMapWritable (note the //TODO comments in the code where it was fixed in 2 places). If there's a better way to submit this recommended bug fix, someone please feel free to let me know. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9295) AbstractMapWritable throws exception when calling readFields() multiple times when the maps contain different class types
[
https://issues.apache.org/jira/browse/HADOOP-9295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13606272#comment-13606272
]
David Parks commented on HADOOP-9295:
-
Further note that the two Writable objects you add *must* be of different
types, example:
class MyFirstCustomWritable extends Writable {}
class MySecondCustomWritable extends Writable {}
AbstractMapWritable throws exception when calling readFields() multiple times
when the maps contain different class types
-
Key: HADOOP-9295
URL: https://issues.apache.org/jira/browse/HADOOP-9295
Project: Hadoop Common
Issue Type: Bug
Components: io
Affects Versions: 1.0.3
Reporter: David Parks
Assignee: Karthik Kambatla
Priority: Critical
Attachments: MapWritableBugTest.java, test-hadoop-9295.patch
Verified the trunk looks the same as 1.0.3 for this issue.
When mappers output MapWritables with different class types, then they are
read in on the Reducer via an iterator (multiple calls to readFields without
instantiating a new object) you'll get this:
java.lang.IllegalArgumentException: Id 1 exists but maps to
org.me.ClassTypeOne and not org.me.ClassTypeTwo
at
org.apache.hadoop.io.AbstractMapWritable.addToMap(AbstractMapWritable.java:73)
at
org.apache.hadoop.io.AbstractMapWritable.readFields(AbstractMapWritable.java:201)
It happens because AbstractMapWritable accumulates class type entries in its
ClassType to ID (and vice versa) hashmaps.
Those accumulating classtype-to-id hashmaps need to be cleared to support
multiple calls to readFields().
I've attached a JUnit test that both demonstrates the problem and contains an
embedded, fixed version of MapWritable and ArrayMapWritable (note the //TODO
comments in the code where it was fixed in 2 places).
If there's a better way to submit this recommended bug fix, someone please
feel free to let me know.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9295) AbstractMapWritable throws exception when calling readFields() multiple times when the maps contain different class types
[ https://issues.apache.org/jira/browse/HADOOP-9295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13605402#comment-13605402 ] Karthik Kambatla commented on HADOOP-9295: -- I thought about this further and wonder if there is really a problem here. I uploaded a test (test-hadoop-9295.patch) to capture valid use of MapWritable/SortedMapWritable and the test passes. I might be missing something. [~davidparks21], can you please point out what intended behavior is not being captured in the test. AbstractMapWritable throws exception when calling readFields() multiple times when the maps contain different class types - Key: HADOOP-9295 URL: https://issues.apache.org/jira/browse/HADOOP-9295 Project: Hadoop Common Issue Type: Bug Components: io Affects Versions: 1.0.3 Reporter: David Parks Assignee: Karthik Kambatla Priority: Critical Attachments: MapWritableBugTest.java, test-hadoop-9295.patch Verified the trunk looks the same as 1.0.3 for this issue. When mappers output MapWritables with different class types, then they are read in on the Reducer via an iterator (multiple calls to readFields without instantiating a new object) you'll get this: java.lang.IllegalArgumentException: Id 1 exists but maps to org.me.ClassTypeOne and not org.me.ClassTypeTwo at org.apache.hadoop.io.AbstractMapWritable.addToMap(AbstractMapWritable.java:73) at org.apache.hadoop.io.AbstractMapWritable.readFields(AbstractMapWritable.java:201) It happens because AbstractMapWritable accumulates class type entries in its ClassType to ID (and vice versa) hashmaps. Those accumulating classtype-to-id hashmaps need to be cleared to support multiple calls to readFields(). I've attached a JUnit test that both demonstrates the problem and contains an embedded, fixed version of MapWritable and ArrayMapWritable (note the //TODO comments in the code where it was fixed in 2 places). If there's a better way to submit this recommended bug fix, someone please feel free to let me know. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9295) AbstractMapWritable throws exception when calling readFields() multiple times when the maps contain different class types
[ https://issues.apache.org/jira/browse/HADOOP-9295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13582124#comment-13582124 ] David Parks commented on HADOOP-9295: - I'd be happy to submit this as a patch, however I don't know what that entails, can you point me to some documentation of the format you want it in, or offer some help in doing that? Thanks, Dave AbstractMapWritable throws exception when calling readFields() multiple times when the maps contain different class types - Key: HADOOP-9295 URL: https://issues.apache.org/jira/browse/HADOOP-9295 Project: Hadoop Common Issue Type: Bug Components: io Affects Versions: 1.0.3 Reporter: David Parks Assignee: Karthik Kambatla Priority: Critical Attachments: MapWritableBugTest.java Verified the trunk looks the same as 1.0.3 for this issue. When mappers output MapWritables with different class types, then they are read in on the Reducer via an iterator (multiple calls to readFields without instantiating a new object) you'll get this: java.lang.IllegalArgumentException: Id 1 exists but maps to org.me.ClassTypeOne and not org.me.ClassTypeTwo at org.apache.hadoop.io.AbstractMapWritable.addToMap(AbstractMapWritable.java:73) at org.apache.hadoop.io.AbstractMapWritable.readFields(AbstractMapWritable.java:201) It happens because AbstractMapWritable accumulates class type entries in its ClassType to ID (and vice versa) hashmaps. Those accumulating classtype-to-id hashmaps need to be cleared to support multiple calls to readFields(). I've attached a JUnit test that both demonstrates the problem and contains an embedded, fixed version of MapWritable and ArrayMapWritable (note the //TODO comments in the code where it was fixed in 2 places). If there's a better way to submit this recommended bug fix, someone please feel free to let me know. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9295) AbstractMapWritable throws exception when calling readFields() multiple times when the maps contain different class types
[ https://issues.apache.org/jira/browse/HADOOP-9295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13582307#comment-13582307 ] Karthik Kambatla commented on HADOOP-9295: -- http://wiki.apache.org/hadoop/HowToContribute is a good place to start. The instructions there are for svn - you can definitely use git - http://wiki.apache.org/hadoop/GitAndHadoop Note that Hadoop uses spaces for tabs - and both tab and indentation are 2 spaces. AbstractMapWritable throws exception when calling readFields() multiple times when the maps contain different class types - Key: HADOOP-9295 URL: https://issues.apache.org/jira/browse/HADOOP-9295 Project: Hadoop Common Issue Type: Bug Components: io Affects Versions: 1.0.3 Reporter: David Parks Assignee: Karthik Kambatla Priority: Critical Attachments: MapWritableBugTest.java Verified the trunk looks the same as 1.0.3 for this issue. When mappers output MapWritables with different class types, then they are read in on the Reducer via an iterator (multiple calls to readFields without instantiating a new object) you'll get this: java.lang.IllegalArgumentException: Id 1 exists but maps to org.me.ClassTypeOne and not org.me.ClassTypeTwo at org.apache.hadoop.io.AbstractMapWritable.addToMap(AbstractMapWritable.java:73) at org.apache.hadoop.io.AbstractMapWritable.readFields(AbstractMapWritable.java:201) It happens because AbstractMapWritable accumulates class type entries in its ClassType to ID (and vice versa) hashmaps. Those accumulating classtype-to-id hashmaps need to be cleared to support multiple calls to readFields(). I've attached a JUnit test that both demonstrates the problem and contains an embedded, fixed version of MapWritable and ArrayMapWritable (note the //TODO comments in the code where it was fixed in 2 places). If there's a better way to submit this recommended bug fix, someone please feel free to let me know. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9295) AbstractMapWritable throws exception when calling readFields() multiple times when the maps contain different class types
[ https://issues.apache.org/jira/browse/HADOOP-9295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13575913#comment-13575913 ] Karthik Kambatla commented on HADOOP-9295: -- Thanks for filing this, David. Looks like you already a fix. Will you be interested in posting a patch for the same? AbstractMapWritable throws exception when calling readFields() multiple times when the maps contain different class types - Key: HADOOP-9295 URL: https://issues.apache.org/jira/browse/HADOOP-9295 Project: Hadoop Common Issue Type: Bug Components: io Affects Versions: 1.0.3 Reporter: David Parks Priority: Critical Attachments: MapWritableBugTest.java Verified the trunk looks the same as 1.0.3 for this issue. When mappers output MapWritables with different class types, then they are read in on the Reducer via an iterator (multiple calls to readFields without instantiating a new object) you'll get this: java.lang.IllegalArgumentException: Id 1 exists but maps to org.me.ClassTypeOne and not org.me.ClassTypeTwo at org.apache.hadoop.io.AbstractMapWritable.addToMap(AbstractMapWritable.java:73) at org.apache.hadoop.io.AbstractMapWritable.readFields(AbstractMapWritable.java:201) It happens because AbstractMapWritable accumulates class type entries in its ClassType to ID (and vice versa) hashmaps. Those accumulating classtype-to-id hashmaps need to be cleared to support multiple calls to readFields(). I've attached a JUnit test that both demonstrates the problem and contains an embedded, fixed version of MapWritable and ArrayMapWritable (note the //TODO comments in the code where it was fixed in 2 places). If there's a better way to submit this recommended bug fix, someone please feel free to let me know. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
