[jira] [Updated] (HADOOP-8684) Deadlock between WritableComparator and WritableComparable
[ https://issues.apache.org/jira/browse/HADOOP-8684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-8684: - Fix Version/s: (was: 3.0.0) Deadlock between WritableComparator and WritableComparable -- Key: HADOOP-8684 URL: https://issues.apache.org/jira/browse/HADOOP-8684 Project: Hadoop Common Issue Type: Bug Components: io Affects Versions: 1.0.3, 3.0.0 Reporter: Hiroshi Ikeda Assignee: Jing Zhao Priority: Minor Fix For: 0.23.4, 2.0.3-alpha Attachments: Hadoop-8684.patch, Hadoop-8684.patch, Hadoop-8684.patch, Hadoop-8684.patch, Hadoop-8684.patch, WritableComparatorDeadLockTestApp.java Classes implementing WriableComparable in Hadoop call the method WritableComparator.define() in their static initializers. This means, the classes call the method define() while thier class loading, under locking their class objects. And, the method WritableComparator.define() locks the WritableComaprator class object. On the other hand, WritableComparator.get() also locks the WritableComparator class object, and the method may create instances of the targeted comparable class, involving loading the targeted comparable class if any. This means, the method might try to lock the targeted comparable class object under locking the WritableComparator class object. There are reversed orders of locking objects, and you might fall in deadlock. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-8684) Deadlock between WritableComparator and WritableComparable
[ https://issues.apache.org/jira/browse/HADOOP-8684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans updated HADOOP-8684: Fix Version/s: 2.0.3-alpha 0.23.4 I pulled this into branch-2, and branch-0.23 Deadlock between WritableComparator and WritableComparable -- Key: HADOOP-8684 URL: https://issues.apache.org/jira/browse/HADOOP-8684 Project: Hadoop Common Issue Type: Bug Components: io Affects Versions: 1.0.3, 3.0.0 Reporter: Hiroshi Ikeda Assignee: Jing Zhao Priority: Minor Fix For: 3.0.0, 0.23.4, 2.0.3-alpha Attachments: Hadoop-8684.patch, Hadoop-8684.patch, Hadoop-8684.patch, Hadoop-8684.patch, Hadoop-8684.patch, WritableComparatorDeadLockTestApp.java Classes implementing WriableComparable in Hadoop call the method WritableComparator.define() in their static initializers. This means, the classes call the method define() while thier class loading, under locking their class objects. And, the method WritableComparator.define() locks the WritableComaprator class object. On the other hand, WritableComparator.get() also locks the WritableComparator class object, and the method may create instances of the targeted comparable class, involving loading the targeted comparable class if any. This means, the method might try to lock the targeted comparable class object under locking the WritableComparator class object. There are reversed orders of locking objects, and you might fall in deadlock. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8684) Deadlock between WritableComparator and WritableComparable
[ https://issues.apache.org/jira/browse/HADOOP-8684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas updated HADOOP-8684: Resolution: Fixed Fix Version/s: 3.0.0 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) I committed the change to Trunk. This change has been tested using the test written by Hiroshi manually. Thank you Hiroshi and Jing. Deadlock between WritableComparator and WritableComparable -- Key: HADOOP-8684 URL: https://issues.apache.org/jira/browse/HADOOP-8684 Project: Hadoop Common Issue Type: Bug Components: io Affects Versions: 1.0.3, 3.0.0 Reporter: Hiroshi Ikeda Assignee: Jing Zhao Priority: Minor Fix For: 3.0.0 Attachments: Hadoop-8684.patch, Hadoop-8684.patch, Hadoop-8684.patch, Hadoop-8684.patch, Hadoop-8684.patch, WritableComparatorDeadLockTestApp.java Classes implementing WriableComparable in Hadoop call the method WritableComparator.define() in their static initializers. This means, the classes call the method define() while thier class loading, under locking their class objects. And, the method WritableComparator.define() locks the WritableComaprator class object. On the other hand, WritableComparator.get() also locks the WritableComparator class object, and the method may create instances of the targeted comparable class, involving loading the targeted comparable class if any. This means, the method might try to lock the targeted comparable class object under locking the WritableComparator class object. There are reversed orders of locking objects, and you might fall in deadlock. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8684) Deadlock between WritableComparator and WritableComparable
[ https://issues.apache.org/jira/browse/HADOOP-8684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HADOOP-8684: -- Attachment: Hadoop-8684.patch Deadlock between WritableComparator and WritableComparable -- Key: HADOOP-8684 URL: https://issues.apache.org/jira/browse/HADOOP-8684 Project: Hadoop Common Issue Type: Bug Components: io Affects Versions: 1.0.3, 3.0.0 Reporter: Hiroshi Ikeda Assignee: Jing Zhao Priority: Minor Attachments: Hadoop-8684.patch, Hadoop-8684.patch, Hadoop-8684.patch, Hadoop-8684.patch, Hadoop-8684.patch, WritableComparatorDeadLockTestApp.java Classes implementing WriableComparable in Hadoop call the method WritableComparator.define() in their static initializers. This means, the classes call the method define() while thier class loading, under locking their class objects. And, the method WritableComparator.define() locks the WritableComaprator class object. On the other hand, WritableComparator.get() also locks the WritableComparator class object, and the method may create instances of the targeted comparable class, involving loading the targeted comparable class if any. This means, the method might try to lock the targeted comparable class object under locking the WritableComparator class object. There are reversed orders of locking objects, and you might fall in deadlock. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8684) Deadlock between WritableComparator and WritableComparable
[ https://issues.apache.org/jira/browse/HADOOP-8684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HADOOP-8684: -- Affects Version/s: 3.0.0 Status: Patch Available (was: Open) Deadlock between WritableComparator and WritableComparable -- Key: HADOOP-8684 URL: https://issues.apache.org/jira/browse/HADOOP-8684 Project: Hadoop Common Issue Type: Bug Components: io Affects Versions: 1.0.3, 3.0.0 Reporter: Hiroshi Ikeda Assignee: Jing Zhao Priority: Minor Attachments: Hadoop-8684.patch, WritableComparatorDeadLockTestApp.java Classes implementing WriableComparable in Hadoop call the method WritableComparator.define() in their static initializers. This means, the classes call the method define() while thier class loading, under locking their class objects. And, the method WritableComparator.define() locks the WritableComaprator class object. On the other hand, WritableComparator.get() also locks the WritableComparator class object, and the method may create instances of the targeted comparable class, involving loading the targeted comparable class if any. This means, the method might try to lock the targeted comparable class object under locking the WritableComparator class object. There are reversed orders of locking objects, and you might fall in deadlock. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8684) Deadlock between WritableComparator and WritableComparable
[ https://issues.apache.org/jira/browse/HADOOP-8684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HADOOP-8684: -- Attachment: Hadoop-8684.patch seems the comparators should be declared as volatile. Deadlock between WritableComparator and WritableComparable -- Key: HADOOP-8684 URL: https://issues.apache.org/jira/browse/HADOOP-8684 Project: Hadoop Common Issue Type: Bug Components: io Affects Versions: 1.0.3, 3.0.0 Reporter: Hiroshi Ikeda Assignee: Jing Zhao Priority: Minor Attachments: Hadoop-8684.patch, Hadoop-8684.patch, WritableComparatorDeadLockTestApp.java Classes implementing WriableComparable in Hadoop call the method WritableComparator.define() in their static initializers. This means, the classes call the method define() while thier class loading, under locking their class objects. And, the method WritableComparator.define() locks the WritableComaprator class object. On the other hand, WritableComparator.get() also locks the WritableComparator class object, and the method may create instances of the targeted comparable class, involving loading the targeted comparable class if any. This means, the method might try to lock the targeted comparable class object under locking the WritableComparator class object. There are reversed orders of locking objects, and you might fall in deadlock. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8684) Deadlock between WritableComparator and WritableComparable
[ https://issues.apache.org/jira/browse/HADOOP-8684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HADOOP-8684: -- Attachment: Hadoop-8684.patch We do not need the volatile so switch back to the original version. Now in the patch, the locking sequence corresponding to WritableComparator#define() is 1) lock L1 of the targeted comparable class object, and 2) lock L2 in WritableComparator, and 3) lock L3 in WritableComparator#comparators (which is a ConcurrentHashMap), and 4) release lock L3, and 5) release lock L2, and 6) release lock in L1. The lock sequence of get: lock L3 -- unlock L3 -- lock L1 of targeted comparable class object -- unlock L1 -- lock L2 -- unlock L2. So we should be able to avoid the deadlock now. Deadlock between WritableComparator and WritableComparable -- Key: HADOOP-8684 URL: https://issues.apache.org/jira/browse/HADOOP-8684 Project: Hadoop Common Issue Type: Bug Components: io Affects Versions: 1.0.3, 3.0.0 Reporter: Hiroshi Ikeda Assignee: Jing Zhao Priority: Minor Attachments: Hadoop-8684.patch, Hadoop-8684.patch, Hadoop-8684.patch, WritableComparatorDeadLockTestApp.java Classes implementing WriableComparable in Hadoop call the method WritableComparator.define() in their static initializers. This means, the classes call the method define() while thier class loading, under locking their class objects. And, the method WritableComparator.define() locks the WritableComaprator class object. On the other hand, WritableComparator.get() also locks the WritableComparator class object, and the method may create instances of the targeted comparable class, involving loading the targeted comparable class if any. This means, the method might try to lock the targeted comparable class object under locking the WritableComparator class object. There are reversed orders of locking objects, and you might fall in deadlock. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8684) Deadlock between WritableComparator and WritableComparable
[ https://issues.apache.org/jira/browse/HADOOP-8684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HADOOP-8684: -- Attachment: Hadoop-8684.patch Suresh and Hiroshi: thanks for the comments! I changed from reentrantlock to synchronized (I use the synchronized here to keep the same semantic with the original version). And the test program runs well with the patch. Deadlock between WritableComparator and WritableComparable -- Key: HADOOP-8684 URL: https://issues.apache.org/jira/browse/HADOOP-8684 Project: Hadoop Common Issue Type: Bug Components: io Affects Versions: 1.0.3, 3.0.0 Reporter: Hiroshi Ikeda Assignee: Jing Zhao Priority: Minor Attachments: Hadoop-8684.patch, Hadoop-8684.patch, Hadoop-8684.patch, Hadoop-8684.patch, WritableComparatorDeadLockTestApp.java Classes implementing WriableComparable in Hadoop call the method WritableComparator.define() in their static initializers. This means, the classes call the method define() while thier class loading, under locking their class objects. And, the method WritableComparator.define() locks the WritableComaprator class object. On the other hand, WritableComparator.get() also locks the WritableComparator class object, and the method may create instances of the targeted comparable class, involving loading the targeted comparable class if any. This means, the method might try to lock the targeted comparable class object under locking the WritableComparator class object. There are reversed orders of locking objects, and you might fall in deadlock. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8684) Deadlock between WritableComparator and WritableComparable
[ https://issues.apache.org/jira/browse/HADOOP-8684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hiroshi Ikeda updated HADOOP-8684: -- Description: Classes implementing WriableComparable in Hadoop call the method WritableComparator.define() in their static initializers. This means, the classes call the method define() while thier class loading, under locking their class objects. And, the method WritableComparator.define() locks the WritableComaprator class object. On the other hand, WritableComparator.get() also locks the WritableComparator class object, and the method may create instances of the targeted comparable class, involving loading the targeted comparable class if any. This means, the method might try to lock the targeted comparable class object under locking the WritableComparator class object. There are reversed orders of locking objects, and you might fall in deadlock. was: Classes implementing WriableComparable in Hadoop call the method WritableComparator.define() in their static initializers. This means, the classes call the method define() while thier class loading, under locking their class objects. And, the method WritableComparator.define() locks the WritableComaprator class object. On the other hand, WritableComparator.get() also locks the WritableComparator class object, and the method may create instances of the targeted comparable class, involving loading the targeted comparable class if any. This means, the method might try to lock the targeted comparable class object under locking the WritableComparator class object. There are reversed orders of locking objects, and you might fall in deadlock. Deadlock between WritableComparator and WritableComparable -- Key: HADOOP-8684 URL: https://issues.apache.org/jira/browse/HADOOP-8684 Project: Hadoop Common Issue Type: Bug Components: io Affects Versions: 1.0.3 Reporter: Hiroshi Ikeda Priority: Minor Classes implementing WriableComparable in Hadoop call the method WritableComparator.define() in their static initializers. This means, the classes call the method define() while thier class loading, under locking their class objects. And, the method WritableComparator.define() locks the WritableComaprator class object. On the other hand, WritableComparator.get() also locks the WritableComparator class object, and the method may create instances of the targeted comparable class, involving loading the targeted comparable class if any. This means, the method might try to lock the targeted comparable class object under locking the WritableComparator class object. There are reversed orders of locking objects, and you might fall in deadlock. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8684) Deadlock between WritableComparator and WritableComparable
[ https://issues.apache.org/jira/browse/HADOOP-8684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hiroshi Ikeda updated HADOOP-8684: -- Attachment: WritableComparatorDeadLockTestApp.java Sample application for the deadlock. Deadlock between WritableComparator and WritableComparable -- Key: HADOOP-8684 URL: https://issues.apache.org/jira/browse/HADOOP-8684 Project: Hadoop Common Issue Type: Bug Components: io Affects Versions: 1.0.3 Reporter: Hiroshi Ikeda Priority: Minor Attachments: WritableComparatorDeadLockTestApp.java Classes implementing WriableComparable in Hadoop call the method WritableComparator.define() in their static initializers. This means, the classes call the method define() while thier class loading, under locking their class objects. And, the method WritableComparator.define() locks the WritableComaprator class object. On the other hand, WritableComparator.get() also locks the WritableComparator class object, and the method may create instances of the targeted comparable class, involving loading the targeted comparable class if any. This means, the method might try to lock the targeted comparable class object under locking the WritableComparator class object. There are reversed orders of locking objects, and you might fall in deadlock. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
