[jira] [Updated] (HADOOP-9439) JniBasedUnixGroupsMapping: fix some crash bugs
[ https://issues.apache.org/jira/browse/HADOOP-9439?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J updated HADOOP-9439: Fix Version/s: 2.1.0-beta 3.0.0 > JniBasedUnixGroupsMapping: fix some crash bugs > -- > > Key: HADOOP-9439 > URL: https://issues.apache.org/jira/browse/HADOOP-9439 > Project: Hadoop Common > Issue Type: Bug > Components: native >Affects Versions: 2.0.4-alpha >Reporter: Colin Patrick McCabe >Assignee: Colin Patrick McCabe >Priority: Minor > Fix For: 3.0.0, 2.1.0-beta > > Attachments: HADOOP-9439.001.patch, HADOOP-9439.003.patch, > HADOOP-9439.005.patch, HADOOP-9439.006.patch, HADOOP-9439.007.patch, > HADOOP-9439.008.patch, HDFS-4640.002.patch > > > JniBasedUnixGroupsMapping has some issues. > * sometimes on error paths variables are freed prior to being initialized > * re-allocate buffers less frequently (can reuse the same buffer for multiple > calls to getgrnam) > * allow non-reentrant functions to be used, to work around client bugs > * don't throw IOException from JNI functions if the JNI functions do not > declare this checked exception. > * don't bail out if only one group name among all the ones associated with a > user can't be looked up. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9439) JniBasedUnixGroupsMapping: fix some crash bugs
[ https://issues.apache.org/jira/browse/HADOOP-9439?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Patrick McCabe updated HADOOP-9439: - Resolution: Fixed Status: Resolved (was: Patch Available) committed to trunk, branch-2, branch-2.1 > JniBasedUnixGroupsMapping: fix some crash bugs > -- > > Key: HADOOP-9439 > URL: https://issues.apache.org/jira/browse/HADOOP-9439 > Project: Hadoop Common > Issue Type: Bug > Components: native >Affects Versions: 2.0.4-alpha >Reporter: Colin Patrick McCabe >Assignee: Colin Patrick McCabe >Priority: Minor > Attachments: HADOOP-9439.001.patch, HADOOP-9439.003.patch, > HADOOP-9439.005.patch, HADOOP-9439.006.patch, HADOOP-9439.007.patch, > HADOOP-9439.008.patch, HDFS-4640.002.patch > > > JniBasedUnixGroupsMapping has some issues. > * sometimes on error paths variables are freed prior to being initialized > * re-allocate buffers less frequently (can reuse the same buffer for multiple > calls to getgrnam) > * allow non-reentrant functions to be used, to work around client bugs > * don't throw IOException from JNI functions if the JNI functions do not > declare this checked exception. > * don't bail out if only one group name among all the ones associated with a > user can't be looked up. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9439) JniBasedUnixGroupsMapping: fix some crash bugs
[ https://issues.apache.org/jira/browse/HADOOP-9439?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Nauroth updated HADOOP-9439: -- Attachment: HADOOP-9439.008.patch The patch caused unsatisfied link errors on Windows. The problem was most easily visible as a test failure in {{TestJNIGroupsMapping}}. It would be valuable to port this to the Windows side. The Windows implementation was largely based on the prior code, so it's subject to the same problems, such as the problems listed in the description here and the memory leak I reported in HADOOP-9312. Unfortunately, I'm not available to do a full port and test it right now. (Any other volunteers?) Meanwhile, I'm uploading version 8 of the patch, which is the minimal work required to prevent breaking Windows. The only thing I changed in addition to Colin's patch is JniBasedUnixGroupsMappingWin.c. I handled the signature change on {{getGroupsForUser}}. I stubbed {{anchorNative}} to do nothing and left a comment explaining that we need the full port of this patch later. > JniBasedUnixGroupsMapping: fix some crash bugs > -- > > Key: HADOOP-9439 > URL: https://issues.apache.org/jira/browse/HADOOP-9439 > Project: Hadoop Common > Issue Type: Bug > Components: native >Affects Versions: 2.0.4-alpha >Reporter: Colin Patrick McCabe >Assignee: Colin Patrick McCabe >Priority: Minor > Attachments: HADOOP-9439.001.patch, HADOOP-9439.003.patch, > HADOOP-9439.005.patch, HADOOP-9439.006.patch, HADOOP-9439.007.patch, > HADOOP-9439.008.patch, HDFS-4640.002.patch > > > JniBasedUnixGroupsMapping has some issues. > * sometimes on error paths variables are freed prior to being initialized > * re-allocate buffers less frequently (can reuse the same buffer for multiple > calls to getgrnam) > * allow non-reentrant functions to be used, to work around client bugs > * don't throw IOException from JNI functions if the JNI functions do not > declare this checked exception. > * don't bail out if only one group name among all the ones associated with a > user can't be looked up. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9439) JniBasedUnixGroupsMapping: fix some crash bugs
[ https://issues.apache.org/jira/browse/HADOOP-9439?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Patrick McCabe updated HADOOP-9439: - Attachment: HADOOP-9439.007.patch We've been seeing a lot of people hit segfaults that we believe result from non-threadsafe implementations of the getpwuid, etc interfaces. Let's change the default to locking. > JniBasedUnixGroupsMapping: fix some crash bugs > -- > > Key: HADOOP-9439 > URL: https://issues.apache.org/jira/browse/HADOOP-9439 > Project: Hadoop Common > Issue Type: Bug > Components: native >Affects Versions: 2.0.4-alpha >Reporter: Colin Patrick McCabe >Assignee: Colin Patrick McCabe >Priority: Minor > Attachments: HADOOP-9439.001.patch, HADOOP-9439.003.patch, > HADOOP-9439.005.patch, HADOOP-9439.006.patch, HADOOP-9439.007.patch, > HDFS-4640.002.patch > > > JniBasedUnixGroupsMapping has some issues. > * sometimes on error paths variables are freed prior to being initialized > * re-allocate buffers less frequently (can reuse the same buffer for multiple > calls to getgrnam) > * allow non-reentrant functions to be used, to work around client bugs > * don't throw IOException from JNI functions if the JNI functions do not > declare this checked exception. > * don't bail out if only one group name among all the ones associated with a > user can't be looked up. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9439) JniBasedUnixGroupsMapping: fix some crash bugs
[ https://issues.apache.org/jira/browse/HADOOP-9439?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Patrick McCabe updated HADOOP-9439: - Attachment: HADOOP-9439.006.patch * declare pw_lock_object as extern * put declaration of pw_lock_locked on a separate line > JniBasedUnixGroupsMapping: fix some crash bugs > -- > > Key: HADOOP-9439 > URL: https://issues.apache.org/jira/browse/HADOOP-9439 > Project: Hadoop Common > Issue Type: Bug > Components: native >Affects Versions: 2.0.4-alpha >Reporter: Colin Patrick McCabe >Assignee: Colin Patrick McCabe >Priority: Minor > Attachments: HADOOP-9439.001.patch, HADOOP-9439.003.patch, > HADOOP-9439.005.patch, HADOOP-9439.006.patch, HDFS-4640.002.patch > > > JniBasedUnixGroupsMapping has some issues. > * sometimes on error paths variables are freed prior to being initialized > * re-allocate buffers less frequently (can reuse the same buffer for multiple > calls to getgrnam) > * allow non-reentrant functions to be used, to work around client bugs > * don't throw IOException from JNI functions if the JNI functions do not > declare this checked exception. > * don't bail out if only one group name among all the ones associated with a > user can't be looked up. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9439) JniBasedUnixGroupsMapping: fix some crash bugs
[ https://issues.apache.org/jira/browse/HADOOP-9439?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Patrick McCabe updated HADOOP-9439: - Attachment: HADOOP-9439.005.patch > JniBasedUnixGroupsMapping: fix some crash bugs > -- > > Key: HADOOP-9439 > URL: https://issues.apache.org/jira/browse/HADOOP-9439 > Project: Hadoop Common > Issue Type: Bug > Components: native >Affects Versions: 2.0.4-alpha >Reporter: Colin Patrick McCabe >Assignee: Colin Patrick McCabe >Priority: Minor > Attachments: HADOOP-9439.001.patch, HADOOP-9439.003.patch, > HADOOP-9439.005.patch, HDFS-4640.002.patch > > > JniBasedUnixGroupsMapping has some issues. > * sometimes on error paths variables are freed prior to being initialized > * re-allocate buffers less frequently (can reuse the same buffer for multiple > calls to getgrnam) > * allow non-reentrant functions to be used, to work around client bugs > * don't throw IOException from JNI functions if the JNI functions do not > declare this checked exception. > * don't bail out if only one group name among all the ones associated with a > user can't be looked up. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9439) JniBasedUnixGroupsMapping: fix some crash bugs
[ https://issues.apache.org/jira/browse/HADOOP-9439?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Patrick McCabe updated HADOOP-9439: - Attachment: HADOOP-9439.003.patch bq. I don't get why empty is a parameter here... aside from it being a premature optimization (is allocating an empty array that bad?), it seems like the JNI code could just grab this field itself and return a reference, rather than taking it as a parameter. This optimization was in the existing code, so I kept it in. But actually, looking at it again, I think you're right. We might as well take it out. bq. Style nit: extra space here before getGroupsForUser ok. bq. Can we use strerror here on the return, assuming it's probably a standard errnum? good idea. +ret = hadoop_group_info_fetch(ginfo, uinfo->gids[i]); +if (!ret) { bq. Here if the group info lookup fails for a particular gid, you end up swallowing the error without any warnings, etc. Maybe instead we should just return the numeric gid as a group? This is what the 'groups' shell command does: The 'groups' shell command returns nonzero if it can't look up a group (at least according to the internet; I didn't want to mess up my groups settings to test). That would cause ShellBasedUnixGroupsMapping to get "got exception trying to get groups for user." So as far as I know, there is no scenario currently where we treat the numeric gid as a group name. We could be bug-compatible with ShellBasedUnixGroupsMapping, and refuse to look up *any* groups if one fails. However, that doesn't seem like a good idea. Instead, I added some code that fires off a log4j message in that scenario, and continues with the other groups. bq. We may have to actually share this lock and configuration with the lock used in NativeIO.c. yeah, let's do that. bq. Additionally, when I did that patch, I found it simpler to continue to use the reentrant functions wrapped with a lock – less repeated code, etc. ok. > JniBasedUnixGroupsMapping: fix some crash bugs > -- > > Key: HADOOP-9439 > URL: https://issues.apache.org/jira/browse/HADOOP-9439 > Project: Hadoop Common > Issue Type: Bug > Components: native >Affects Versions: 2.0.4-alpha >Reporter: Colin Patrick McCabe >Assignee: Colin Patrick McCabe >Priority: Minor > Attachments: HADOOP-9439.001.patch, HADOOP-9439.003.patch, > HDFS-4640.002.patch > > > JniBasedUnixGroupsMapping has some issues. > * sometimes on error paths variables are freed prior to being initialized > * re-allocate buffers less frequently (can reuse the same buffer for multiple > calls to getgrnam) > * allow non-reentrant functions to be used, to work around client bugs > * don't throw IOException from JNI functions if the JNI functions do not > declare this checked exception. > * don't bail out if only one group name among all the ones associated with a > user can't be looked up. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9439) JniBasedUnixGroupsMapping: fix some crash bugs
[ https://issues.apache.org/jira/browse/HADOOP-9439?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Patrick McCabe updated HADOOP-9439: - Attachment: HADOOP-9439.001.patch add the ability to configure JniBasedUnixGroupsMapping to use the non-reentrant getpwent and getgrent functions at runtime. They are buggy on some implementations. pass in an empty array in a threadsafe manner, duplicating the optimization we had before where we return a pre-allocated empty array when there are no groups to be found. > JniBasedUnixGroupsMapping: fix some crash bugs > -- > > Key: HADOOP-9439 > URL: https://issues.apache.org/jira/browse/HADOOP-9439 > Project: Hadoop Common > Issue Type: Bug > Components: native >Affects Versions: 2.0.4-alpha >Reporter: Colin Patrick McCabe >Assignee: Colin Patrick McCabe >Priority: Minor > Attachments: HADOOP-9439.001.patch, HDFS-4640.002.patch > > > JniBasedUnixGroupsMapping has some issues. > * sometimes on error paths variables are freed prior to being initialized > * re-allocate buffers less frequently (can reuse the same buffer for multiple > calls to getgrnam) > * allow non-reentrant functions to be used, to work around client bugs > * don't throw IOException from JNI functions if the JNI functions do not > declare this checked exception. > * don't bail out if only one group name among all the ones associated with a > user can't be looked up. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira