[jira] [Updated] (HADOOP-12001) Limiting LDAP search conflicts with posixGroup addition
[ https://issues.apache.org/jira/browse/HADOOP-12001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick White updated HADOOP-12001: --- Attachment: HADOOP-12001.002.patch same patch with updated unit tests Limiting LDAP search conflicts with posixGroup addition --- Key: HADOOP-12001 URL: https://issues.apache.org/jira/browse/HADOOP-12001 Project: Hadoop Common Issue Type: Bug Components: security Affects Versions: 2.7.0, 2.8.0 Reporter: Patrick White Assignee: Patrick White Priority: Blocker Attachments: HADOOP-12001.002.patch, HADOOP-12001.patch In HADOOP-9477, posixGroup support was added In HADOOP-10626, a limit on the returned attributes was added to speed up queries. Limiting the attributes can break the SEARCH_CONTROLS object in the context of the isPosix block, since it only asks LDAP for the groupNameAttr -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-12001) Limiting LDAP search conflicts with posixGroup addition
[ https://issues.apache.org/jira/browse/HADOOP-12001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14559724#comment-14559724 ] Patrick White commented on HADOOP-12001: Those two patches on their own don't break anything, but the combination will break posixGroups implementation as it stands. My patch corrects that. I'm new to patches in Hadoop, what additional work is required on the patch? For testing, I've run it on a cluster and verified I see the right thing when configured in posixGroups mode with the 'groups' command. I'm not 100% sure how to modify the unit tests to fit this case, but haven't looked very deeply yet. Limiting LDAP search conflicts with posixGroup addition --- Key: HADOOP-12001 URL: https://issues.apache.org/jira/browse/HADOOP-12001 Project: Hadoop Common Issue Type: Bug Components: security Affects Versions: 2.7.0, 2.8.0 Reporter: Patrick White Assignee: Patrick White Priority: Blocker Attachments: HADOOP-12001.patch In HADOOP-9477, posixGroup support was added In HADOOP-10626, a limit on the returned attributes was added to speed up queries. Limiting the attributes can break the SEARCH_CONTROLS object in the context of the isPosix block, since it only asks LDAP for the groupNameAttr -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12001) Limiting LDAP search conflicts with posixGroup addition
[ https://issues.apache.org/jira/browse/HADOOP-12001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick White updated HADOOP-12001: --- Status: Patch Available (was: Open) Limiting LDAP search conflicts with posixGroup addition --- Key: HADOOP-12001 URL: https://issues.apache.org/jira/browse/HADOOP-12001 Project: Hadoop Common Issue Type: Bug Components: security Affects Versions: 2.8.0 Reporter: Patrick White Attachments: HADOOP-12001.patch In HADOOP-9477, posixGroup support was added In HADOOP-10626, a limit on the returned attributes was added to speed up queries. Limiting the attributes can break the SEARCH_CONTROLS object in the context of the isPosix block, since it only asks LDAP for the groupNameAttr -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12001) Limiting LDAP search conflicts with posixGroup addition
[ https://issues.apache.org/jira/browse/HADOOP-12001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick White updated HADOOP-12001: --- Attachment: HADOOP-12001.patch Initial attempt at a fix for this. Shouldn't break anything that currently uses this, but enables people with weird(?) schemas to use it. Limiting LDAP search conflicts with posixGroup addition --- Key: HADOOP-12001 URL: https://issues.apache.org/jira/browse/HADOOP-12001 Project: Hadoop Common Issue Type: Bug Components: security Affects Versions: 2.8.0 Reporter: Patrick White Attachments: HADOOP-12001.patch In HADOOP-9477, posixGroup support was added In HADOOP-10626, a limit on the returned attributes was added to speed up queries. Limiting the attributes can break the SEARCH_CONTROLS object in the context of the isPosix block, since it only asks LDAP for the groupNameAttr -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HADOOP-12001) Limiting LDAP search conflicts with posixGroup addition
Patrick White created HADOOP-12001: -- Summary: Limiting LDAP search conflicts with posixGroup addition Key: HADOOP-12001 URL: https://issues.apache.org/jira/browse/HADOOP-12001 Project: Hadoop Common Issue Type: Bug Components: security Affects Versions: 2.8.0 Reporter: Patrick White In HADOOP-9477, posixGroup support was added In HADOOP-10626, a limit on the returned attributes was added to speed up queries. Limiting the attributes can break the SEARCH_CONTROLS object in the context of the isPosix block, since it only asks LDAP for the groupNameAttr -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-10554) Performance: Scan metrics for 2.4 are down compared to 0.23.9
[ https://issues.apache.org/jira/browse/HADOOP-10554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13997846#comment-13997846 ] patrick white commented on HADOOP-10554: Test results update to Description: Issue does reproduce on a small (10 node) cluster, with consistent performance difference across release lines. Numbers from the 10 node cluster actually show a larger delta than the 350 node cluster's results (10% regression), between 2.4.0 and 0.23.9, both in runtimes and throughput. Performance: Scan metrics for 2.4 are down compared to 0.23.9 - Key: HADOOP-10554 URL: https://issues.apache.org/jira/browse/HADOOP-10554 Project: Hadoop Common Issue Type: Bug Affects Versions: 2.4.0 Reporter: patrick white Performance comparison benchmarks for Scan test's runtime and throughput metrics are slightly out of 5% tolerance in 2.x compared against 0.23. The trend is consistent across later releases in both lines, latest release numbers are; Runtime: 2.4.0.0 -73.6 seconds (avg 5 passes) 0.23.9.12 -69.4 seconds (avg 5 passes) Diff: -5.7% Throughput: 2.4.0.0 - 28.67 GB/s (avg 5 passes) 0.23.9.12 - 30.41 GB/s (avg 5 passes) Diff: -6.1% Scan test is specifically measuring the average map's input read performance. The diff is consistent when run on a larger (350 node) perf environment, we are in process of seeing if this reproduces in a smaller cluster, using appropriately scaled inputs. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HADOOP-10553) Performance: AM scaleability is 10% slower in 2.4 compared to 0.23.9
patrick white created HADOOP-10553: -- Summary: Performance: AM scaleability is 10% slower in 2.4 compared to 0.23.9 Key: HADOOP-10553 URL: https://issues.apache.org/jira/browse/HADOOP-10553 Project: Hadoop Common Issue Type: Bug Affects Versions: 2.4.0 Reporter: patrick white Performance comparison benchmarks from 2.x against 0.23 shows AM scalability benchmark's runtime is approximately 10% slower in 2.4.0. The trend is consistent across later releases in both lines, latest release numbers are: 2.4.0.0 runtime 255.6 seconds (avg 5 passes) 0.23.9.12 runtime 230.4 seconds (avg 5 passes) Diff: -9.9% AM Scalability test is essentially a sleep job that measures time to launch and complete a large number of mappers. The diff is consistent and has been reproduced in both a larger (350 node, 100,000 mappers) perf environment, as well as a small (10 node, 2,900 mappers) demo cluster. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HADOOP-10554) Performance: Scan metrics for 2.4 are notably down compared to 0.23.9
patrick white created HADOOP-10554: -- Summary: Performance: Scan metrics for 2.4 are notably down compared to 0.23.9 Key: HADOOP-10554 URL: https://issues.apache.org/jira/browse/HADOOP-10554 Project: Hadoop Common Issue Type: Bug Affects Versions: 2.4.0 Reporter: patrick white Performance comparison benchmarks for Scan test's runtime and throughput metrics are slightly out of 5% tolerance in 2.x compared against 0.23. The trend is consistent across later releases in both lines, latest release numbers are; Runtime: 2.4.0.0 -73.6 seconds (avg 5 passes) 0.23.9.12 -69.4 seconds (avg 5 passes) Diff: -5.7% Throughput: 2.4.0.0 - 28.67 GB/s (avg 5 passes) 0.23.9.12 - 30.41 GB/s (avg 5 passes) Diff: -6.1% Scan test is specifically measuring the average map's input read performance. The diff is consistent when run on a larger (350 node) perf environment, we are in process of seeing if this reproduces in a smaller cluster, using appropriately scaled inputs. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-10554) Performance: Scan metrics for 2.4 are down compared to 0.23.9
[ https://issues.apache.org/jira/browse/HADOOP-10554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] patrick white updated HADOOP-10554: --- Summary: Performance: Scan metrics for 2.4 are down compared to 0.23.9 (was: Performance: Scan metrics for 2.4 are notably down compared to 0.23.9) Performance: Scan metrics for 2.4 are down compared to 0.23.9 - Key: HADOOP-10554 URL: https://issues.apache.org/jira/browse/HADOOP-10554 Project: Hadoop Common Issue Type: Bug Affects Versions: 2.4.0 Reporter: patrick white Performance comparison benchmarks for Scan test's runtime and throughput metrics are slightly out of 5% tolerance in 2.x compared against 0.23. The trend is consistent across later releases in both lines, latest release numbers are; Runtime: 2.4.0.0 -73.6 seconds (avg 5 passes) 0.23.9.12 -69.4 seconds (avg 5 passes) Diff: -5.7% Throughput: 2.4.0.0 - 28.67 GB/s (avg 5 passes) 0.23.9.12 - 30.41 GB/s (avg 5 passes) Diff: -6.1% Scan test is specifically measuring the average map's input read performance. The diff is consistent when run on a larger (350 node) perf environment, we are in process of seeing if this reproduces in a smaller cluster, using appropriately scaled inputs. -- This message was sent by Atlassian JIRA (v6.2#6252)
