[jira] [Updated] (HADOOP-8152) Expand public APIs for security library classes
[ https://issues.apache.org/jira/browse/HADOOP-8152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron T. Myers updated HADOOP-8152: --- Attachment: HADOOP-8152.patch Thanks a lot for the reviews, Eli and Alejandro. Here's an updated patch which incorporates your feedback. bq. hadoop-auth packages: Mind if we address this in a separate JIRA? Seems like a distinct issue to me. Expand public APIs for security library classes --- Key: HADOOP-8152 URL: https://issues.apache.org/jira/browse/HADOOP-8152 Project: Hadoop Common Issue Type: Improvement Components: security Affects Versions: 2.0.0 Reporter: Aaron T. Myers Assignee: Aaron T. Myers Attachments: HADOOP-8152.patch, HADOOP-8152.patch, HADOOP-8152.patch Currently projects like Hive and HBase use UserGroupInformation and SecurityUtil methods. Both of these classes are marked LimitedPrivate(HDFS,MR) but should probably be marked more generally public. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8280) Move VersionUtil/TestVersionUtil and GenericTestUtils from HDFS into Common.
[ https://issues.apache.org/jira/browse/HADOOP-8280?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron T. Myers updated HADOOP-8280: --- Summary: Move VersionUtil/TestVersionUtil and GenericTestUtils from HDFS into Common. (was: Move VersionUtil/TestVersionUtil from HDFS into Common.) Move VersionUtil/TestVersionUtil and GenericTestUtils from HDFS into Common. - Key: HADOOP-8280 URL: https://issues.apache.org/jira/browse/HADOOP-8280 Project: Hadoop Common Issue Type: Improvement Components: test, util Affects Versions: 0.23.1 Reporter: Ahmed Radwan Assignee: Ahmed Radwan Attachments: HADOOP-8280.patch We need to use VersionUtil in MAPREDUCE-4150. Moving VersionUtil/TestVersionUtil from HDFS into common will help this, especially since test-patch doesn't seem to deal well with cross-sub-project patches. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8280) Move VersionUtil/TestVersionUtil and GenericTestUtils from HDFS into Common.
[ https://issues.apache.org/jira/browse/HADOOP-8280?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron T. Myers updated HADOOP-8280: --- Target Version/s: 2.0.0 Move VersionUtil/TestVersionUtil and GenericTestUtils from HDFS into Common. - Key: HADOOP-8280 URL: https://issues.apache.org/jira/browse/HADOOP-8280 Project: Hadoop Common Issue Type: Improvement Components: test, util Affects Versions: 0.23.1 Reporter: Ahmed Radwan Assignee: Ahmed Radwan Attachments: HADOOP-8280.patch We need to use VersionUtil in MAPREDUCE-4150. Moving VersionUtil/TestVersionUtil from HDFS into common will help this, especially since test-patch doesn't seem to deal well with cross-sub-project patches. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8275) harden serialization logic against malformed or malicious input
[ https://issues.apache.org/jira/browse/HADOOP-8275?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron T. Myers updated HADOOP-8275: --- Status: Patch Available (was: Open) Hey Colin, could you please add some simple tests for this code? Also, be sure to mark it patch-available so that test-patch will run on it. (I'm doing that for you now.) harden serialization logic against malformed or malicious input --- Key: HADOOP-8275 URL: https://issues.apache.org/jira/browse/HADOOP-8275 Project: Hadoop Common Issue Type: Bug Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Priority: Minor Attachments: HADOOP-8275.001.patch harden serialization logic against malformed or malicious input -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8152) Expand public APIs for security library classes
[ https://issues.apache.org/jira/browse/HADOOP-8152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron T. Myers updated HADOOP-8152: --- Attachment: HADOOP-8152.patch Thanks a lot for the input, Todd. Here's an updated patch which adds the methods you mentioned HBase makes use of. bq. Many parts of UGI are pretty broken when multiple keytabs are in play. They should probably be fixed before marking them public. Note that this patch is marking the interfaces evolving, not stable, so if we have a good reason to change them between releases (e.g. fixing multi-keytab support) then we certainly can. Expand public APIs for security library classes --- Key: HADOOP-8152 URL: https://issues.apache.org/jira/browse/HADOOP-8152 Project: Hadoop Common Issue Type: Improvement Components: security Affects Versions: 2.0.0 Reporter: Aaron T. Myers Assignee: Aaron T. Myers Attachments: HADOOP-8152.patch, HADOOP-8152.patch Currently projects like Hive and HBase use UserGroupInformation and SecurityUtil methods. Both of these classes are marked LimitedPrivate(HDFS,MR) but should probably be marked more generally public. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8261) Har file system doesn't deal with FS URIs with a host but no port
[
https://issues.apache.org/jira/browse/HADOOP-8261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aaron T. Myers updated HADOOP-8261:
---
Attachment: HADOOP-8261.patch
Thanks a lot for the review, Todd. Here's an updated patch which addresses your
feedback. I'm going to commit this momentarily.
Har file system doesn't deal with FS URIs with a host but no port
-
Key: HADOOP-8261
URL: https://issues.apache.org/jira/browse/HADOOP-8261
Project: Hadoop Common
Issue Type: Bug
Components: fs
Affects Versions: 2.0.0
Reporter: Aaron T. Myers
Assignee: Aaron T. Myers
Attachments: HADOOP-8261-with-test-in-HDFS.patch, HADOOP-8261.patch,
HADOOP-8261.patch
If you try to run an MR job with a Hadoop Archive as the input, but the URI
you give it has no port specified (e.g. hdfs://simon) the job will fail
with an error like the following:
{noformat}
java.io.IOException: Incomplete HDFS URI, no host:
hdfs://simon:-1/user/atm/input.har/input
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8261) Har file system doesn't deal with FS URIs with a host but no port
[
https://issues.apache.org/jira/browse/HADOOP-8261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aaron T. Myers updated HADOOP-8261:
---
Attachment: HADOOP-8261.patch
Here's a patch which addresses the issue.
I've written a test for this fix, but the test was easiest written in the HDFS
project. I'm going to upload another patch in a moment which is the complete
change, but uploading just the fix in Common here so that test-patch can run.
Har file system doesn't deal with FS URIs with a host but no port
-
Key: HADOOP-8261
URL: https://issues.apache.org/jira/browse/HADOOP-8261
Project: Hadoop Common
Issue Type: Bug
Components: fs
Affects Versions: 2.0.0
Reporter: Aaron T. Myers
Assignee: Aaron T. Myers
Attachments: HADOOP-8261.patch
If you try to run an MR job with a Hadoop Archive as the input, but the URI
you give it has no port specified (e.g. hdfs://simon) the job will fail
with an error like the following:
{noformat}
java.io.IOException: Incomplete HDFS URI, no host:
hdfs://simon:-1/user/atm/input.har/input
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8261) Har file system doesn't deal with FS URIs with a host but no port
[
https://issues.apache.org/jira/browse/HADOOP-8261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aaron T. Myers updated HADOOP-8261:
---
Attachment: HADOOP-8261-with-test-in-HDFS.patch
Here's the complete patch, including cross-project test in HDFS.
Har file system doesn't deal with FS URIs with a host but no port
-
Key: HADOOP-8261
URL: https://issues.apache.org/jira/browse/HADOOP-8261
Project: Hadoop Common
Issue Type: Bug
Components: fs
Affects Versions: 2.0.0
Reporter: Aaron T. Myers
Assignee: Aaron T. Myers
Attachments: HADOOP-8261-with-test-in-HDFS.patch, HADOOP-8261.patch
If you try to run an MR job with a Hadoop Archive as the input, but the URI
you give it has no port specified (e.g. hdfs://simon) the job will fail
with an error like the following:
{noformat}
java.io.IOException: Incomplete HDFS URI, no host:
hdfs://simon:-1/user/atm/input.har/input
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8261) Har file system doesn't deal with FS URIs with a host but no port
[
https://issues.apache.org/jira/browse/HADOOP-8261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aaron T. Myers updated HADOOP-8261:
---
Status: Patch Available (was: Open)
Har file system doesn't deal with FS URIs with a host but no port
-
Key: HADOOP-8261
URL: https://issues.apache.org/jira/browse/HADOOP-8261
Project: Hadoop Common
Issue Type: Bug
Components: fs
Affects Versions: 2.0.0
Reporter: Aaron T. Myers
Assignee: Aaron T. Myers
Attachments: HADOOP-8261-with-test-in-HDFS.patch, HADOOP-8261.patch
If you try to run an MR job with a Hadoop Archive as the input, but the URI
you give it has no port specified (e.g. hdfs://simon) the job will fail
with an error like the following:
{noformat}
java.io.IOException: Incomplete HDFS URI, no host:
hdfs://simon:-1/user/atm/input.har/input
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8152) Expand public APIs for security library classes
[ https://issues.apache.org/jira/browse/HADOOP-8152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron T. Myers updated HADOOP-8152: --- Target Version/s: 2.0.0 (was: 0.23.3) Affects Version/s: (was: 0.23.3) 2.0.0 Status: Patch Available (was: Open) Expand public APIs for security library classes --- Key: HADOOP-8152 URL: https://issues.apache.org/jira/browse/HADOOP-8152 Project: Hadoop Common Issue Type: Improvement Components: security Affects Versions: 2.0.0 Reporter: Aaron T. Myers Assignee: Aaron T. Myers Attachments: HADOOP-8152.patch Currently projects like Hive and HBase use UserGroupInformation and SecurityUtil methods. Both of these classes are marked LimitedPrivate(HDFS,MR) but should probably be marked more generally public. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8152) Expand public APIs for security library classes
[ https://issues.apache.org/jira/browse/HADOOP-8152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron T. Myers updated HADOOP-8152: --- Attachment: HADOOP-8152.patch Here's a patch which addresses the issue. Let me summarize the changes: * Both UserGroupInformation and SecurityUtil are currently marked InterfaceAudience.LimitedPrivate(HDFS, MapReduce) and InterfaceStability.Evolving. This is unchanged. * This patch adds InterfaceAudience.Public and InterfaceStability.Evolving annotations to getCurrentUser, getLoginUser, createRemoteUser, createProxyUser, and both variants of doAs in UserGroupInformation. * This patch adds InterfaceAudience.Public and InterfaceStability.Evolving annotations to both variants of the method login in SecurityUtil. * This patch removes two cases of individual methods being marked InterfaceAudience.LimitedPrivate(HDFS, MapReduce) in UserGroupInformation. Since the class is already annotated the same way, these seemed redundant. My understanding of the nature of the InterfaceAudience and InterfaceStability annotations is that the most-specific annotation is what applies. Thus, just increasing the InterfaceAudience visibility of these methods should be sufficient for the purposes of dependent projects. The methods that I chose here are the ones that I'm aware of dependent projects using. If others are aware of more, I'd be happy to add them to this patch. Expand public APIs for security library classes --- Key: HADOOP-8152 URL: https://issues.apache.org/jira/browse/HADOOP-8152 Project: Hadoop Common Issue Type: Improvement Components: security Affects Versions: 2.0.0 Reporter: Aaron T. Myers Assignee: Aaron T. Myers Attachments: HADOOP-8152.patch Currently projects like Hive and HBase use UserGroupInformation and SecurityUtil methods. Both of these classes are marked LimitedPrivate(HDFS,MR) but should probably be marked more generally public. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8244) Improve comments on ByteBufferReadable.read
[ https://issues.apache.org/jira/browse/HADOOP-8244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron T. Myers updated HADOOP-8244: --- Resolution: Fixed Fix Version/s: 0.24.0 Target Version/s: 0.24.0 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) I've just committed this to trunk. Thanks a lot for the contribution, Hank. Improve comments on ByteBufferReadable.read --- Key: HADOOP-8244 URL: https://issues.apache.org/jira/browse/HADOOP-8244 Project: Hadoop Common Issue Type: Improvement Reporter: Henry Robinson Assignee: Henry Robinson Fix For: 0.24.0 Attachments: HADOOP-8244.0.patch, HADOOP-8244.1.patch There are a couple of ways in which the comment on ByteBufferReadable.read can be improved. Since this is a public-facing API with potentially many implementations, it's worth taking the time to get it right. * We should describe what can become of the byte buffer state in the case of an exception (is the limit changed? where's the position?). For the DFSInputStream implementation, position and limit are unchanged if there is an error, but I don't think that's the right think to mandate for all implementations. * We should mention explicitly that 0-byte reads are legitimate - this is particularly important in light of HDFS-3110 which detects support for direct reads by issuing a 0-byte read from libhdfs. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8184) ProtoBuf RPC engine does not need it own reply packet - it can use the IPC layer reply packet.
[ https://issues.apache.org/jira/browse/HADOOP-8184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron T. Myers updated HADOOP-8184: --- Affects Version/s: 2.0.0 Release Note: This change will affect the output of errors for some Hadoop CLI commands. Specifically, the name of the exception class will no longer appear, and instead only the text of the exception message will appear. Hadoop Flags: Incompatible change,Reviewed (was: Reviewed) ProtoBuf RPC engine does not need it own reply packet - it can use the IPC layer reply packet. -- Key: HADOOP-8184 URL: https://issues.apache.org/jira/browse/HADOOP-8184 Project: Hadoop Common Issue Type: Improvement Components: ipc Affects Versions: 2.0.0 Reporter: Sanjay Radia Assignee: Sanjay Radia Fix For: 2.0.0 Attachments: rpcFixPBHeader2.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-7416) Allow test-patch to work with cross sub-project changes
[
https://issues.apache.org/jira/browse/HADOOP-7416?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aaron T. Myers updated HADOOP-7416:
---
Attachment: HADOOP-7416.patch
Here's a patch which partially addresses the issue. This will allow for
cross-sub-project patches to apply cleanly, and will comment with a -0
patch... in the JIRA comment if this is detected. It *won't* actually run all
of the tests from all affected modules. Since that's a substantially more
involved change, and since this change should help considerably, I'd like to
punt that to a separate JIRA.
I tested this patch by creating a few uni-project and multi-project patches,
and running test-patch manually. With this change, test-patch successfully
applies and detects cross-sub-project patches.
Allow test-patch to work with cross sub-project changes
---
Key: HADOOP-7416
URL: https://issues.apache.org/jira/browse/HADOOP-7416
Project: Hadoop Common
Issue Type: Improvement
Reporter: Aaron T. Myers
Assignee: Aaron T. Myers
Attachments: HADOOP-7416.patch
Now that the sub-projects are sub-directories in the same repo, we can make
{{test-patch.sh}} not barf on cross-project patches. We would need to make
{{smart-apply-patch.sh}} not fail hard when it detects this, and probably
make {{test-patch.sh}} run all the validation checks against all affected
sub-projects.
This was inspired by HDFS-1900, which changes a config in Common that's
referenced in a bunch of places in HDFS.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8189) LdapGroupsMapping shouldn't throw away IOException
[ https://issues.apache.org/jira/browse/HADOOP-8189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron T. Myers updated HADOOP-8189: --- Resolution: Fixed Fix Version/s: 0.23.3 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) I've just committed this to trunk and branch-0.23. Thanks a lot, Natty. LdapGroupsMapping shouldn't throw away IOException -- Key: HADOOP-8189 URL: https://issues.apache.org/jira/browse/HADOOP-8189 Project: Hadoop Common Issue Type: Bug Components: security Affects Versions: 0.23.3 Reporter: Jonathan Natkins Assignee: Jonathan Natkins Fix For: 0.23.3 Attachments: HADOOP-8189.patch, HADOOP-8189.patch When extracting a password from a file during the config setup, the LdapGroupsMapping throws a RuntimeException, but doesn't bubble up the IOException that caused it -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8191) SshFenceByTcpPort uses netcat incorrectly
[
https://issues.apache.org/jira/browse/HADOOP-8191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aaron T. Myers updated HADOOP-8191:
---
Target Version/s: 0.23.3
Affects Version/s: (was: 0.24.0)
0.23.3
Moved to Common since the patch has no HDFS changes. I've also kicked Jenkins
to run test-patch on the patch.
SshFenceByTcpPort uses netcat incorrectly
-
Key: HADOOP-8191
URL: https://issues.apache.org/jira/browse/HADOOP-8191
Project: Hadoop Common
Issue Type: Bug
Components: ha
Affects Versions: 0.23.3
Reporter: Philip Zeyliger
Assignee: Todd Lipcon
Attachments: hdfs-3081.txt
SshFencyByTcpPort currently assumes that the NN is listening on localhost.
Typical setups have the namenode listening just on the hostname of the
namenode, which would lead nc -z to not catch it.
Here's an example in which the NN is running, listening on 8020, but doesn't
respond to localhost 8020.
{noformat}
[root@xxx ~]# lsof -P -p 5286 | grep -i listen
java5286 root 110u IPv41772357 TCP xxx:8020
(LISTEN)
java5286 root 121u IPv41772397 TCP xxx:50070
(LISTEN)
[root@xxx ~]# nc -z localhost 8020
[root@xxx ~]# nc -z xxx 8020
Connection to xxx 8020 port [tcp/intu-ec-svcdisc] succeeded!
{noformat}
Here's the likely offending code:
{code}
LOG.info(
Indeterminate response from trying to kill service. +
Verifying whether it is running using nc...);
rc = execCommand(session, nc -z localhost 8020);
{code}
Naively, we could rely on netcat to the correct hostname (since the NN ought
to be listening on the hostname it's configured as), or just to use fuser.
Fuser catches ports independently of what IPs they're bound to:
{noformat}
[root@xxx ~]# fuser 1234/tcp
1234/tcp: 6766 6768
[root@xxx ~]# jobs
[1]- Running nc -l localhost 1234
[2]+ Running nc -l rhel56-18.ent.cloudera.com 1234
[root@xxx ~]# sudo lsof -P | grep -i LISTEN | grep -i 1234
nc 6766 root3u IPv42563626
TCP localhost:1234 (LISTEN)
nc 6768 root3u IPv42563671
TCP xxx:1234 (LISTEN)
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8121) Active Directory Group Mapping Service
[ https://issues.apache.org/jira/browse/HADOOP-8121?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron T. Myers updated HADOOP-8121: --- Target Version/s: 0.23.3 Affects Version/s: 0.23.2 Active Directory Group Mapping Service -- Key: HADOOP-8121 URL: https://issues.apache.org/jira/browse/HADOOP-8121 Project: Hadoop Common Issue Type: New Feature Components: security Affects Versions: 0.23.2 Reporter: Jonathan Natkins Assignee: Jonathan Natkins Attachments: HADOOP-8121-common.patch, HADOOP-8121-common.patch, HADOOP-8121-hdfs.patch, HADOOP-8121.patch, HADOOP-8121.patch, HADOOP-8121.patch, HADOOP-8121.patch, HADOOP-8121.patch, HADOOP-8121.patch, HADOOP-8121.patch, HADOOP-8121.patch, HADOOP-8121.patch, HADOOP-8121.patch, HADOOP-8121.patch Planning on building a group mapping service that will go and talk directly to an Active Directory setup to get group memberships -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8121) Active Directory Group Mapping Service
[ https://issues.apache.org/jira/browse/HADOOP-8121?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron T. Myers updated HADOOP-8121: --- Resolution: Fixed Fix Version/s: 0.23.3 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) I've just committed this to trunk and branch-0.23. Thanks a lot for the contribution, Natty! Active Directory Group Mapping Service -- Key: HADOOP-8121 URL: https://issues.apache.org/jira/browse/HADOOP-8121 Project: Hadoop Common Issue Type: New Feature Components: security Affects Versions: 0.23.2 Reporter: Jonathan Natkins Assignee: Jonathan Natkins Fix For: 0.23.3 Attachments: HADOOP-8121-common.patch, HADOOP-8121-common.patch, HADOOP-8121-hdfs.patch, HADOOP-8121.patch, HADOOP-8121.patch, HADOOP-8121.patch, HADOOP-8121.patch, HADOOP-8121.patch, HADOOP-8121.patch, HADOOP-8121.patch, HADOOP-8121.patch, HADOOP-8121.patch, HADOOP-8121.patch, HADOOP-8121.patch Planning on building a group mapping service that will go and talk directly to an Active Directory setup to get group memberships -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8189) LdapGroupsMapping shouldn't throw away IOException
[ https://issues.apache.org/jira/browse/HADOOP-8189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron T. Myers updated HADOOP-8189: --- Target Version/s: 0.23.3 Affects Version/s: 0.23.3 LdapGroupsMapping shouldn't throw away IOException -- Key: HADOOP-8189 URL: https://issues.apache.org/jira/browse/HADOOP-8189 Project: Hadoop Common Issue Type: Bug Components: security Affects Versions: 0.23.3 Reporter: Jonathan Natkins Assignee: Jonathan Natkins Attachments: HADOOP-8189.patch When extracting a password from a file during the config setup, the LdapGroupsMapping throws a RuntimeException, but doesn't bubble up the IOException that caused it -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-7788) HA: Simple HealthMonitor class to watch an HAService
[ https://issues.apache.org/jira/browse/HADOOP-7788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron T. Myers updated HADOOP-7788: --- Issue Type: New Feature (was: Sub-task) Parent: (was: HADOOP-7454) HA: Simple HealthMonitor class to watch an HAService Key: HADOOP-7788 URL: https://issues.apache.org/jira/browse/HADOOP-7788 Project: Hadoop Common Issue Type: New Feature Components: ha Reporter: Todd Lipcon Assignee: Todd Lipcon Attachments: hdfs-2524.txt This is a utility class which will be part of the FailoverController. The class starts a daemon thread which periodically monitors an HAService, calling its monitorHealth function. It then generates callbacks into another class when the health status changes (eg the RPC fails or the service returns a HealthCheckFailedException) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-7788) HA: Simple HealthMonitor class to watch an HAService
[ https://issues.apache.org/jira/browse/HADOOP-7788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron T. Myers updated HADOOP-7788: --- Target Version/s: 0.24.0 (was: HA Branch (HDFS-1623)) Affects Version/s: 0.24.0 Converting to top-level issue with commit of HADOOP-7454. HA: Simple HealthMonitor class to watch an HAService Key: HADOOP-7788 URL: https://issues.apache.org/jira/browse/HADOOP-7788 Project: Hadoop Common Issue Type: New Feature Components: ha Affects Versions: 0.24.0 Reporter: Todd Lipcon Assignee: Todd Lipcon Attachments: hdfs-2524.txt This is a utility class which will be part of the FailoverController. The class starts a daemon thread which periodically monitors an HAService, calling its monitorHealth function. It then generates callbacks into another class when the health status changes (eg the RPC fails or the service returns a HealthCheckFailedException) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-7992) Add ZK client for leader election
[ https://issues.apache.org/jira/browse/HADOOP-7992?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron T. Myers updated HADOOP-7992: --- Issue Type: New Feature (was: Sub-task) Parent: (was: HADOOP-7454) Add ZK client for leader election - Key: HADOOP-7992 URL: https://issues.apache.org/jira/browse/HADOOP-7992 Project: Hadoop Common Issue Type: New Feature Components: ha Reporter: Suresh Srinivas Assignee: Bikas Saha Fix For: HA Branch (HDFS-1623) Attachments: HDFS-2681.HDFS-1623.patch, HDFS-2681.HDFS-1623.patch, HDFS-2681.HDFS-1623.patch, HDFS-2681.HDFS-1623.patch, HDFS-2681.HDFS-1623.patch, HDFS-2681.HDFS-1623.patch, HDFS-2681.HDFS-1623.patch, HDFS-2681.HDFS-1623.patch, HDFS-2681.HDFS-1623.patch, HDFS-2681.HDFS-1623.patch, HDFS-2681.txt, HDFS-2681.txt, Zookeeper based Leader Election and Monitoring Library.pdf ZKClient needs to support the following capabilities: # Ability to create a znode for co-ordinating leader election. # Ability to monitor and receive call backs when active znode status changes. # Ability to get information about the active node. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-7992) Add ZK client for leader election
[ https://issues.apache.org/jira/browse/HADOOP-7992?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron T. Myers updated HADOOP-7992: --- Issue Type: Sub-task (was: New Feature) Parent: HADOOP-7454 Add ZK client for leader election - Key: HADOOP-7992 URL: https://issues.apache.org/jira/browse/HADOOP-7992 Project: Hadoop Common Issue Type: Sub-task Components: ha Reporter: Suresh Srinivas Assignee: Bikas Saha Fix For: HA Branch (HDFS-1623) Attachments: HDFS-2681.HDFS-1623.patch, HDFS-2681.HDFS-1623.patch, HDFS-2681.HDFS-1623.patch, HDFS-2681.HDFS-1623.patch, HDFS-2681.HDFS-1623.patch, HDFS-2681.HDFS-1623.patch, HDFS-2681.HDFS-1623.patch, HDFS-2681.HDFS-1623.patch, HDFS-2681.HDFS-1623.patch, HDFS-2681.HDFS-1623.patch, HDFS-2681.txt, HDFS-2681.txt, Zookeeper based Leader Election and Monitoring Library.pdf ZKClient needs to support the following capabilities: # Ability to create a znode for co-ordinating leader election. # Ability to monitor and receive call backs when active znode status changes. # Ability to get information about the active node. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8077) HA: fencing method should be able to be configured on a per-NN or per-NS basis
[ https://issues.apache.org/jira/browse/HADOOP-8077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron T. Myers updated HADOOP-8077: --- Target Version/s: 0.24.0 (was: HA Branch (HDFS-1623)) Affects Version/s: (was: HA Branch (HDFS-1623)) 0.24.0 Converting to top-level issue with commit of HADOOP-7454. HA: fencing method should be able to be configured on a per-NN or per-NS basis -- Key: HADOOP-8077 URL: https://issues.apache.org/jira/browse/HADOOP-8077 Project: Hadoop Common Issue Type: Sub-task Components: ha Affects Versions: 0.24.0 Reporter: Todd Lipcon Currently, the fencing method configuration is global. Given that different nameservices may use different underlying storage mechanisms or different types of PDUs, it would be preferable to allow the fencing method configuration to be scoped by namenode or nameservice. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8077) HA: fencing method should be able to be configured on a per-NN or per-NS basis
[ https://issues.apache.org/jira/browse/HADOOP-8077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron T. Myers updated HADOOP-8077: --- Issue Type: Improvement (was: Sub-task) Parent: (was: HADOOP-7454) HA: fencing method should be able to be configured on a per-NN or per-NS basis -- Key: HADOOP-8077 URL: https://issues.apache.org/jira/browse/HADOOP-8077 Project: Hadoop Common Issue Type: Improvement Components: ha Affects Versions: 0.24.0 Reporter: Todd Lipcon Currently, the fencing method configuration is global. Given that different nameservices may use different underlying storage mechanisms or different types of PDUs, it would be preferable to allow the fencing method configuration to be scoped by namenode or nameservice. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8007) HA: use substitution token for fencing argument
[ https://issues.apache.org/jira/browse/HADOOP-8007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron T. Myers updated HADOOP-8007: --- Target Version/s: 0.24.0 (was: HA Branch (HDFS-1623)) Affects Version/s: (was: HA Branch (HDFS-1623)) 0.24.0 Converting to top-level issue with commit of HADOOP-7454. HA: use substitution token for fencing argument --- Key: HADOOP-8007 URL: https://issues.apache.org/jira/browse/HADOOP-8007 Project: Hadoop Common Issue Type: Sub-task Components: ha Affects Versions: 0.24.0 Reporter: Eli Collins Assignee: Eli Collins Per HADOOP-7983 currently the fencer always passes the target host:port to fence as the first argument to the fence script, it would be better to use a substitution token. That is to say, the user would configure myfence.sh $TARGETHOST foo bar and Hadoop would substitute the target. This would allow use of pre-existing scripts that might have a different ordering of arguments without a wrapper. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8007) HA: use substitution token for fencing argument
[ https://issues.apache.org/jira/browse/HADOOP-8007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron T. Myers updated HADOOP-8007: --- Issue Type: Improvement (was: Sub-task) Reporter: Aaron T. Myers (was: Eli Collins) Parent: (was: HADOOP-7454) HA: use substitution token for fencing argument --- Key: HADOOP-8007 URL: https://issues.apache.org/jira/browse/HADOOP-8007 Project: Hadoop Common Issue Type: Improvement Components: ha Affects Versions: 0.24.0 Reporter: Aaron T. Myers Assignee: Eli Collins Per HADOOP-7983 currently the fencer always passes the target host:port to fence as the first argument to the fence script, it would be better to use a substitution token. That is to say, the user would configure myfence.sh $TARGETHOST foo bar and Hadoop would substitute the target. This would allow use of pre-existing scripts that might have a different ordering of arguments without a wrapper. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8135) Add ByteBufferReadable interface to FSDataInputStream
[
https://issues.apache.org/jira/browse/HADOOP-8135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aaron T. Myers updated HADOOP-8135:
---
Resolution: Fixed
Fix Version/s: 0.24.0
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
I've just committed this. Thanks a lot for the contribution, Hank.
Add ByteBufferReadable interface to FSDataInputStream
-
Key: HADOOP-8135
URL: https://issues.apache.org/jira/browse/HADOOP-8135
Project: Hadoop Common
Issue Type: New Feature
Components: fs
Reporter: Henry Robinson
Assignee: Henry Robinson
Fix For: 0.24.0
Attachments: HADOOP-8135.2.patch, HADOOP-8135.patch
To prepare for HDFS-2834, it's useful to add an interface to
FSDataInputStream (and others inside hdfs) that adds a read(ByteBuffer...)
method as follows:
{code}
/**
* Reads up to buf.remaining() bytes into buf. Callers should use
* buf.limit(..) to control the size of the desired read.
*
* After the call, buf.position() should be unchanged, and therefore any
data
* can be immediately read from buf.
*
* @param buf
* @return - the number of bytes available to read from buf
* @throws IOException
*/
public int read(ByteBuffer buf) throws IOException;
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-7972) HA: HAServiceProtocol exceptions need to be unwrapped before it can be used
[ https://issues.apache.org/jira/browse/HADOOP-7972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron T. Myers updated HADOOP-7972: --- Target Version/s: HA Branch (HDFS-1623) Fix Version/s: HA Branch (HDFS-1623) Summary: HA: HAServiceProtocol exceptions need to be unwrapped before it can be used (was: HAServiceProtocol exceptions need to be unwrapped before it can be used) HA: HAServiceProtocol exceptions need to be unwrapped before it can be used --- Key: HADOOP-7972 URL: https://issues.apache.org/jira/browse/HADOOP-7972 Project: Hadoop Common Issue Type: Bug Components: ha Affects Versions: HA Branch (HDFS-1623) Reporter: Hari Mankude Assignee: Hari Mankude Fix For: HA Branch (HDFS-1623) Attachments: HADOOP-7972.txt, HADOOP-7972.txt Remote exceptions for all the interfaces in HAServiceProtocol class has to be unwrapped before it can be used. All the HAServiceProtocol interfaces throw exceptions of type RemoteException by default. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-7972) HA: HAServiceProtocol exceptions need to be unwrapped before it can be used
[ https://issues.apache.org/jira/browse/HADOOP-7972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron T. Myers updated HADOOP-7972: --- Issue Type: Sub-task (was: Bug) Parent: HADOOP-7454 HA: HAServiceProtocol exceptions need to be unwrapped before it can be used --- Key: HADOOP-7972 URL: https://issues.apache.org/jira/browse/HADOOP-7972 Project: Hadoop Common Issue Type: Sub-task Components: ha Affects Versions: HA Branch (HDFS-1623) Reporter: Hari Mankude Assignee: Hari Mankude Fix For: HA Branch (HDFS-1623) Attachments: HADOOP-7972.txt, HADOOP-7972.txt Remote exceptions for all the interfaces in HAServiceProtocol class has to be unwrapped before it can be used. All the HAServiceProtocol interfaces throw exceptions of type RemoteException by default. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8116) HA: RetriableCommand is using RetryPolicy incorrectly after HADOOP-7896
[ https://issues.apache.org/jira/browse/HADOOP-8116?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron T. Myers updated HADOOP-8116: --- Attachment: HADOOP-8116-HDFS-1623.patch Thanks a lot for the review, Eli. Here's an updated patch which adds the comment you asked for. I'm going to commit this in a moment, and create the JIRA you asked for. HA: RetriableCommand is using RetryPolicy incorrectly after HADOOP-7896 --- Key: HADOOP-8116 URL: https://issues.apache.org/jira/browse/HADOOP-8116 Project: Hadoop Common Issue Type: Sub-task Components: util Affects Versions: HA Branch (HDFS-1623) Reporter: Aaron T. Myers Assignee: Aaron T. Myers Attachments: HADOOP-8116-HDFS-1623.patch, HADOOP-8116-HDFS-1623.patch HADOOP-7896 (on the HA branch) refactored RetryAction from an enum to a class, and also moved the act of sleeping to delay retries from the RetryPolicy implementations into RetryInvocationHandler. RetriableCommand, in the rewritten distcp tool, uses RetryPolicy and associated classes from o.a.h.io.retry. When MAPREDUCE-2765 was merged into the HA branch, RetriableCommand wasn't adjusted accordingly to make use of the new structure of the o.a.h.io.retry classes. It's probably generally not kosher for RetriableCommand to be using the RetryPolicy classes at all, since they're not really intended to be used except by RetryInvocationHandler. But, regardless, this JIRA aims to make distcp's use of the o.a.h.io.retry classes functional again. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8116) HA: RetriableCommand is using RetryPolicy incorrectly after HADOOP-7896
[ https://issues.apache.org/jira/browse/HADOOP-8116?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron T. Myers updated HADOOP-8116: --- Attachment: HADOOP-8116-HDFS-1623.patch Here's a patch which addresses the issue. I also noticed while fixing this issue that TestCopyMapper wouldn't have worked because HADOOP-8068 results in an additional exception wrapping, which the test wasn't taking into account. HA: RetriableCommand is using RetryPolicy incorrectly after HADOOP-7896 --- Key: HADOOP-8116 URL: https://issues.apache.org/jira/browse/HADOOP-8116 Project: Hadoop Common Issue Type: Sub-task Components: util Affects Versions: HA Branch (HDFS-1623) Reporter: Aaron T. Myers Assignee: Aaron T. Myers Attachments: HADOOP-8116-HDFS-1623.patch HADOOP-7896 (on the HA branch) refactored RetryAction from an enum to a class, and also moved the act of sleeping to delay retries from the RetryPolicy implementations into RetryInvocationHandler. RetriableCommand, in the rewritten distcp tool, uses RetryPolicy and associated classes from o.a.h.io.retry. When MAPREDUCE-2765 was merged into the HA branch, RetriableCommand wasn't adjusted accordingly to make use of the new structure of the o.a.h.io.retry classes. It's probably generally not kosher for RetriableCommand to be using the RetryPolicy classes at all, since they're not really intended to be used except by RetryInvocationHandler. But, regardless, this JIRA aims to make distcp's use of the o.a.h.io.retry classes functional again. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8078) Add capability to turn on security in unit tests.
[ https://issues.apache.org/jira/browse/HADOOP-8078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron T. Myers updated HADOOP-8078: --- Component/s: test Target Version/s: 0.24.0 Affects Version/s: 0.24.0 Fix Version/s: 0.24.0 Add capability to turn on security in unit tests. - Key: HADOOP-8078 URL: https://issues.apache.org/jira/browse/HADOOP-8078 Project: Hadoop Common Issue Type: Improvement Components: test Affects Versions: 0.24.0 Reporter: Jitendra Nath Pandey Assignee: Jaimin D Jetly Fix For: 0.24.0 Attachments: HADOOP-8078.patch, HADOOP-8078.patch, HADOOP-8078.patch, HADOOP-8078.patch, dn1.keytab, nn1.keytab, user1.keytab We should be able to start a kdc server for unit tests, so that security could be turned on. This will greatly improve the coverage of unit tests. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8084) Protobuf RPC engine can be optimized to not do copying for the RPC request/response
[ https://issues.apache.org/jira/browse/HADOOP-8084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron T. Myers updated HADOOP-8084: --- Component/s: ipc Affects Version/s: 0.24.0 Fix Version/s: 0.24.0 Protobuf RPC engine can be optimized to not do copying for the RPC request/response --- Key: HADOOP-8084 URL: https://issues.apache.org/jira/browse/HADOOP-8084 Project: Hadoop Common Issue Type: Improvement Components: ipc Affects Versions: 0.24.0 Reporter: Devaraj Das Assignee: Devaraj Das Fix For: 0.24.0 Attachments: protobuf-no-copy.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8079) Proposal for enhancements to Hadoop for Windows Server and Windows Azure development and runtime environments
[ https://issues.apache.org/jira/browse/HADOOP-8079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron T. Myers updated HADOOP-8079: --- Target Version/s: 1.1.0 (was: 1.0.0) Fix Version/s: (was: 1.0.0) Hi Alexander, the fix version field should only be set once the change has been committed and the JIRA resolved. Thus, I've removed that field. Since 1.0.0 has already been released, it's not an appropriate target version. I've changed the target version to 1.1.0. One question I have - you say in the description that you've developed the patch set against Hadoop 1.0, but that you'd like to refine the patch set until it can be committed to Hadoop trunk. Is the intention to commit this both to branch-1 and trunk? Or just trunk? Proposal for enhancements to Hadoop for Windows Server and Windows Azure development and runtime environments - Key: HADOOP-8079 URL: https://issues.apache.org/jira/browse/HADOOP-8079 Project: Hadoop Common Issue Type: Improvement Components: native Affects Versions: 1.0.0 Reporter: Alexander Stojanovic Labels: hadoop Original Estimate: 2,016h Remaining Estimate: 2,016h This JIRA is intended to capture discussion around proposed work to enhance Apache Hadoop to run well on Windows. Apache Hadoop has worked on Microsoft Windows since its inception, but Windows support has never been a priority. Currently Windows works as a development and testing platform for Hadoop, but Hadoop is not natively integrated, full-featured or performance and scalability tuned for Windows Server or Windows Azure. We would like to change this and engage in a dialog with the broader community on the architectural design points for making Windows (enterprise and cloud) an excellent runtime and deployment environment for Hadoop. The Isotope team at Microsoft (names below) has developed an Apache Hadoop 1.0 patch set that addresses these performance, integration and feature gaps, allowing Apache Hadoop to be used with Azure and Windows Server without recourse to virtualization technologies such as Cygwin. We have significant interest in the deployment of Hadoop across many multi-tenant, PaaS and IaaS environments - which bring their own unique requirements. Microsoft has recently completed a CCLA with Apache and would like to contribute these enhancements back to the Apache Hadoop community. In the interest of improving Apache Hadoop so that it runs more smoothly on all platforms, including Windows, we propose first contributing this work to the Apache community by attaching it to this JIRA. From there we would like to work with the community to refine the patch set until it is ready to be merged into the Apache trunk. Your feedback solicited, Alexander Stojanovic Min Wei David Lao Lengning Liu David Zhang Asad Khan -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8079) Proposal for enhancements to Hadoop for Windows Server and Windows Azure development and runtime environments
[ https://issues.apache.org/jira/browse/HADOOP-8079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron T. Myers updated HADOOP-8079: --- Target Version/s: 0.24.0, 1.1.0 (was: 1.1.0) Got it. Thanks for the explanation. I've also added a target version of 0.24.0, which corresponds to trunk. Proposal for enhancements to Hadoop for Windows Server and Windows Azure development and runtime environments - Key: HADOOP-8079 URL: https://issues.apache.org/jira/browse/HADOOP-8079 Project: Hadoop Common Issue Type: Improvement Components: native Affects Versions: 1.0.0 Reporter: Alexander Stojanovic Labels: hadoop Original Estimate: 2,016h Remaining Estimate: 2,016h This JIRA is intended to capture discussion around proposed work to enhance Apache Hadoop to run well on Windows. Apache Hadoop has worked on Microsoft Windows since its inception, but Windows support has never been a priority. Currently Windows works as a development and testing platform for Hadoop, but Hadoop is not natively integrated, full-featured or performance and scalability tuned for Windows Server or Windows Azure. We would like to change this and engage in a dialog with the broader community on the architectural design points for making Windows (enterprise and cloud) an excellent runtime and deployment environment for Hadoop. The Isotope team at Microsoft (names below) has developed an Apache Hadoop 1.0 patch set that addresses these performance, integration and feature gaps, allowing Apache Hadoop to be used with Azure and Windows Server without recourse to virtualization technologies such as Cygwin. We have significant interest in the deployment of Hadoop across many multi-tenant, PaaS and IaaS environments - which bring their own unique requirements. Microsoft has recently completed a CCLA with Apache and would like to contribute these enhancements back to the Apache Hadoop community. In the interest of improving Apache Hadoop so that it runs more smoothly on all platforms, including Windows, we propose first contributing this work to the Apache community by attaching it to this JIRA. From there we would like to work with the community to refine the patch set until it is ready to be merged into the Apache trunk. Your feedback solicited, Alexander Stojanovic Min Wei David Lao Lengning Liu David Zhang Asad Khan -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8074) Small bug in hadoop error message for unknown commands
[
https://issues.apache.org/jira/browse/HADOOP-8074?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aaron T. Myers updated HADOOP-8074:
---
Fix Version/s: (was: 0.23.2)
Small bug in hadoop error message for unknown commands
--
Key: HADOOP-8074
URL: https://issues.apache.org/jira/browse/HADOOP-8074
Project: Hadoop Common
Issue Type: Bug
Components: scripts
Affects Versions: 0.24.0
Reporter: Eli Collins
Assignee: Colin Patrick McCabe
Priority: Trivial
The hadoop fs command should be more user friendly if the user forgets the
dash before the command. Also, this should say cat rather than at.
{noformat}
hadoop-0.24.0-SNAPSHOT $ ./bin/hadoop fs cat
at: Unknown command
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8068) HA: void methods can swallow exceptions when going through failover path
[
https://issues.apache.org/jira/browse/HADOOP-8068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aaron T. Myers updated HADOOP-8068:
---
Issue Type: Sub-task (was: Bug)
Parent: HADOOP-7454
HA: void methods can swallow exceptions when going through failover path
Key: HADOOP-8068
URL: https://issues.apache.org/jira/browse/HADOOP-8068
Project: Hadoop Common
Issue Type: Sub-task
Components: ha, ipc
Affects Versions: HA Branch (HDFS-1623)
Reporter: Todd Lipcon
Assignee: Todd Lipcon
Priority: Blocker
Attachments: hadoop-8068.txt
While running through scale testing, we saw an issue where clients were
getting LeaseExpiredExceptions. We eventually tracked it down to the fact
that some {{create}} calls had timed out (having been sent to a NN just
before it crashed) but the resulting exception was swallowed by the retry
policy code paths.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8038) Add 'ipc.client.connect.max.retries.on.timeouts' entry in core-default.xml file.
[ https://issues.apache.org/jira/browse/HADOOP-8038?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron T. Myers updated HADOOP-8038: --- Target Version/s: HA Branch (HDFS-1623) +1, the patch looks good to me. I'll commit this momentarily. Thanks Uma. Add 'ipc.client.connect.max.retries.on.timeouts' entry in core-default.xml file. Key: HADOOP-8038 URL: https://issues.apache.org/jira/browse/HADOOP-8038 Project: Hadoop Common Issue Type: Sub-task Components: conf, ha Affects Versions: HA Branch (HDFS-1623) Reporter: Uma Maheswara Rao G Assignee: Uma Maheswara Rao G Priority: Minor Attachments: HADOOP-8038.patch HADOOP-7932 added the new property 'ipc.client.connect.max.retries.on.timeouts', which needs to be added in core-default.xml with documentation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8035) Hadoop Maven build is inefficient and runs phases redundantly
[ https://issues.apache.org/jira/browse/HADOOP-8035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron T. Myers updated HADOOP-8035: --- Target Version/s: 0.24.0 Affects Version/s: (was: 0.23.0) 0.23.1 Hadoop Maven build is inefficient and runs phases redundantly - Key: HADOOP-8035 URL: https://issues.apache.org/jira/browse/HADOOP-8035 Project: Hadoop Common Issue Type: Bug Components: build Affects Versions: 0.23.1 Reporter: Andrew Bayer Assignee: Andrew Bayer Priority: Minor Attachments: HADOOP-8035.mk2.patch.txt, HADOOP-8035.patch.txt The Hadoop 0.23 Maven build has some major inefficiencies, most notably in site and javadoc. As a result of redundant plugin executions and certain plugin executions being inherited when they really shouldn't be, builds take a lot longer than they should. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8035) Hadoop Maven build is inefficient and runs phases redundantly
[ https://issues.apache.org/jira/browse/HADOOP-8035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron T. Myers updated HADOOP-8035: --- Status: Patch Available (was: Open) Marking as patch available for Andrew. Hadoop Maven build is inefficient and runs phases redundantly - Key: HADOOP-8035 URL: https://issues.apache.org/jira/browse/HADOOP-8035 Project: Hadoop Common Issue Type: Bug Components: build Affects Versions: 0.23.0 Reporter: Andrew Bayer Assignee: Andrew Bayer Priority: Minor Attachments: HADOOP-8035.patch.txt The Hadoop 0.23 Maven build has some major inefficiencies, most notably in site and javadoc. As a result of redundant plugin executions and certain plugin executions being inherited when they really shouldn't be, builds take a lot longer than they should. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8027) Visiting /jmx on the daemon web interfaces may print unnecessary error in logs
[
https://issues.apache.org/jira/browse/HADOOP-8027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aaron T. Myers updated HADOOP-8027:
---
Target Version/s: 0.24.0, 0.23.1 (was: 0.23.1, 0.24.0)
Summary: Visiting /jmx on the daemon web interfaces may print
unnecessary error in logs (was: Out of the box, visiting /jmx on the NN gives
a whole lot of errors in logs.)
Visiting /jmx on the daemon web interfaces may print unnecessary error in logs
--
Key: HADOOP-8027
URL: https://issues.apache.org/jira/browse/HADOOP-8027
Project: Hadoop Common
Issue Type: Improvement
Components: metrics
Affects Versions: 0.23.0, 0.24.0
Reporter: Harsh J
Assignee: Aaron T. Myers
Priority: Minor
Attachments: HDFS-2584.patch
Logs that follow a {{/jmx}} servlet visit:
{code}
11/11/22 12:09:52 ERROR jmx.JMXJsonServlet: getting attribute UsageThreshold
of java.lang:type=MemoryPool,name=Par Eden Space threw an exception
javax.management.RuntimeMBeanException:
java.lang.UnsupportedOperationException: Usage threshold is not supported
at
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.rethrow(DefaultMBeanServerInterceptor.java:856)
at
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.rethrowMaybeMBeanException(DefaultMBeanServerInterceptor.java:869)
at
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:670)
at
com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(JmxMBeanServer.java:638)
at
org.apache.hadoop.jmx.JMXJsonServlet.writeAttribute(JMXJsonServlet.java:314)
at
org.apache.hadoop.jmx.JMXJsonServlet.listBeans(JMXJsonServlet.java:292)
at org.apache.hadoop.jmx.JMXJsonServlet.doGet(JMXJsonServlet.java:192)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at
org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
at
org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter.doFilter(StaticUserWebFilter.java:109)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at
org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:940)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at
org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
at
org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at
org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
at
org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
at
org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
at
org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:326)
at
org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
at
org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
at
org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
at
org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
Caused by: java.lang.UnsupportedOperationException: Usage threshold is not
supported
at
sun.management.MemoryPoolImpl.getUsageThreshold(MemoryPoolImpl.java:87)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
com.sun.jmx.mbeanserver.ConvertingMethod.invokeWithOpenReturn(ConvertingMethod.java:167)
at
com.sun.jmx.mbeanserver.MXBeanIntrospector.invokeM2(MXBeanIntrospector.java:96)
at
com.sun.jmx.mbeanserver.MXBeanIntrospector.invokeM2(MXBeanIntrospector.java:33)
at
com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:208)
at
com.sun.jmx.mbeanserver.PerInterface.getAttribute(PerInterface.java:65)
at
[jira] [Updated] (HADOOP-7774) HA: Administrative CLI to control HA daemons
[ https://issues.apache.org/jira/browse/HADOOP-7774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron T. Myers updated HADOOP-7774: --- Component/s: ha HA: Administrative CLI to control HA daemons Key: HADOOP-7774 URL: https://issues.apache.org/jira/browse/HADOOP-7774 Project: Hadoop Common Issue Type: Sub-task Components: ha, util Affects Versions: HA Branch (HDFS-1623) Reporter: Aaron T. Myers Assignee: Todd Lipcon Fix For: HA Branch (HDFS-1623) Attachments: hadoop-7774.txt, hdfs-2487.txt We'll need to have some way of controlling the HA nodes while they're live, probably by adding some more commands to dfsadmin. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-7380) Add client failover functionality to o.a.h.io.(ipc|retry)
[
https://issues.apache.org/jira/browse/HADOOP-7380?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aaron T. Myers updated HADOOP-7380:
---
Component/s: ha
Add client failover functionality to o.a.h.io.(ipc|retry)
-
Key: HADOOP-7380
URL: https://issues.apache.org/jira/browse/HADOOP-7380
Project: Hadoop Common
Issue Type: Sub-task
Components: ha, ipc
Affects Versions: 0.23.0
Reporter: Aaron T. Myers
Assignee: Aaron T. Myers
Fix For: 0.23.0
Attachments: hadoop-7380-hdfs-example.patch, hadoop-7380.0.patch,
hadoop-7380.1.patch, hadoop-7380.2.patch, hdfs-7380.3.patch
Implementing client failover will likely require changes to {{o.a.h.io.ipc}}
and/or {{o.a.h.io.retry}}. This JIRA is to track those changes.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-7925) Add interface and update CLI to query current state to HAServiceProtocol
[ https://issues.apache.org/jira/browse/HADOOP-7925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron T. Myers updated HADOOP-7925: --- Component/s: ha Add interface and update CLI to query current state to HAServiceProtocol Key: HADOOP-7925 URL: https://issues.apache.org/jira/browse/HADOOP-7925 Project: Hadoop Common Issue Type: Sub-task Components: ha Affects Versions: HA Branch (HDFS-1623) Reporter: Eli Collins Assignee: Eli Collins Fix For: HA Branch (HDFS-1623) Attachments: hadoop-7925.txt, hadoop-7925.txt, hadoop-7925.txt Common side of HDFS-2679. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-7455) Introduce HA Service Protocol Interface
[ https://issues.apache.org/jira/browse/HADOOP-7455?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron T. Myers updated HADOOP-7455: --- Component/s: ha Introduce HA Service Protocol Interface --- Key: HADOOP-7455 URL: https://issues.apache.org/jira/browse/HADOOP-7455 Project: Hadoop Common Issue Type: Sub-task Components: ha, util Reporter: Suresh Srinivas Assignee: Suresh Srinivas Fix For: HA Branch (HDFS-1623) Attachments: HADOOP-7455.1.patch, HADOOP-7455.2.patch, HADOOP-7455.3.patch, HADOOP-7455.4.patch, HADOOP-7455.5.patch, HDFS-7454.patch This jira introduces a protocol interface to be implemented by services that provide HA functionality. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-7968) Errant println left in RPC.getHighestSupportedProtocol
[ https://issues.apache.org/jira/browse/HADOOP-7968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron T. Myers updated HADOOP-7968: --- Labels: newbie (was: ) Errant println left in RPC.getHighestSupportedProtocol -- Key: HADOOP-7968 URL: https://issues.apache.org/jira/browse/HADOOP-7968 Project: Hadoop Common Issue Type: Bug Components: ipc Affects Versions: 0.24.0 Reporter: Todd Lipcon Priority: Minor Labels: newbie hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/RPC.java: System.out.println(Size of protoMap for + rpcKind + = + getProtocolImplMap(rpcKind).size()); -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-7931) o.a.h.ipc.WritableRpcEngine should have a way to force initialization
[
https://issues.apache.org/jira/browse/HADOOP-7931?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aaron T. Myers updated HADOOP-7931:
---
Attachment: HADOOP-7931.patch
Patch which addresses the issue.
o.a.h.ipc.WritableRpcEngine should have a way to force initialization
-
Key: HADOOP-7931
URL: https://issues.apache.org/jira/browse/HADOOP-7931
Project: Hadoop Common
Issue Type: Bug
Components: ipc
Affects Versions: 0.24.0
Reporter: Aaron T. Myers
Assignee: Aaron T. Myers
Attachments: HADOOP-7931.patch
{{WritableRpcEngine}} currently relies on a static initializer to register
itself with {{o.a.h.ipc.Server}} as a valid {{RpcKind}}. There should be a
way to ensure this initialization has already occurred.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-7931) o.a.h.ipc.WritableRpcEngine should have a way to force initialization
[
https://issues.apache.org/jira/browse/HADOOP-7931?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aaron T. Myers updated HADOOP-7931:
---
Status: Patch Available (was: Open)
o.a.h.ipc.WritableRpcEngine should have a way to force initialization
-
Key: HADOOP-7931
URL: https://issues.apache.org/jira/browse/HADOOP-7931
Project: Hadoop Common
Issue Type: Bug
Components: ipc
Affects Versions: 0.24.0
Reporter: Aaron T. Myers
Assignee: Aaron T. Myers
Attachments: HADOOP-7931.patch
{{WritableRpcEngine}} currently relies on a static initializer to register
itself with {{o.a.h.ipc.Server}} as a valid {{RpcKind}}. There should be a
way to ensure this initialization has already occurred.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-7931) o.a.h.ipc.WritableRpcEngine should have a way to force initialization
[
https://issues.apache.org/jira/browse/HADOOP-7931?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aaron T. Myers updated HADOOP-7931:
---
Resolution: Fixed
Fix Version/s: 0.24.0
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Thanks a lot for your quick review, Suresh. I've just committed this to trunk.
o.a.h.ipc.WritableRpcEngine should have a way to force initialization
-
Key: HADOOP-7931
URL: https://issues.apache.org/jira/browse/HADOOP-7931
Project: Hadoop Common
Issue Type: Bug
Components: ipc
Affects Versions: 0.24.0
Reporter: Aaron T. Myers
Assignee: Aaron T. Myers
Fix For: 0.24.0
Attachments: HADOOP-7931.patch
{{WritableRpcEngine}} currently relies on a static initializer to register
itself with {{o.a.h.ipc.Server}} as a valid {{RpcKind}}. There should be a
way to ensure this initialization has already occurred.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-7928) HA: Client failover policy is incorrectly trying to fail over all IOExceptions
[
https://issues.apache.org/jira/browse/HADOOP-7928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aaron T. Myers updated HADOOP-7928:
---
Attachment: HADOOP-7928-HDFS-1623.patch
Here's a patch which addresses the issue.
HA: Client failover policy is incorrectly trying to fail over all IOExceptions
--
Key: HADOOP-7928
URL: https://issues.apache.org/jira/browse/HADOOP-7928
Project: Hadoop Common
Issue Type: Sub-task
Components: ha, ipc
Affects Versions: HA Branch (HDFS-1623)
Reporter: Todd Lipcon
Assignee: Aaron T. Myers
Priority: Blocker
Attachments: HADOOP-7928-HDFS-1623.patch
In the {{FailoverOnNetworkExceptionRetry}} implementation, we're returning
FAILOVER_AND_RETRY for any IOException, so long as the call is idempotent.
So, for example, {{getFileInfo}} on a file you don't have access to throws
AccessControlException which incorrectly triggers a failover. We should not
failover on RemoteExceptions unless they are StandbyException.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-7896) HA: if both NNs are in Standby mode, client needs to try failing back and forth several times with sleeps
[ https://issues.apache.org/jira/browse/HADOOP-7896?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron T. Myers updated HADOOP-7896: --- Attachment: HADOOP-7896-HDFS-1623.patch Thanks a lot for the review, Eli. Here's an updated patch which addresses your comments. HA: if both NNs are in Standby mode, client needs to try failing back and forth several times with sleeps - Key: HADOOP-7896 URL: https://issues.apache.org/jira/browse/HADOOP-7896 Project: Hadoop Common Issue Type: Sub-task Components: ha, ipc Affects Versions: HA Branch (HDFS-1623) Reporter: Todd Lipcon Assignee: Aaron T. Myers Priority: Critical Attachments: HADOOP-7896-HDFS-1623.patch, HADOOP-7896-HDFS-1623.patch For a manual failover, there may be an intermediate state for a non-trivial amount of time where both NNs are in standby mode. Currently, the failover proxy will immediately failover on receiving this exception from the first NN, and when it hits the same exception on the second NN, it immediately fails. It should probably fail back and forth nearly indefinitely if both NNs are in Standby mode. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-7896) HA: if both NNs are in Standby mode, client needs to try failing back and forth several times with sleeps
[ https://issues.apache.org/jira/browse/HADOOP-7896?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron T. Myers updated HADOOP-7896: --- Attachment: HADOOP-7896-HDFS-1623.patch Thanks again for the review, Eli. I agree that that synchronization is unnecessary. Here's an updated patch which removes that sync. I'm going to commit this momentarily unless there are further objections. HA: if both NNs are in Standby mode, client needs to try failing back and forth several times with sleeps - Key: HADOOP-7896 URL: https://issues.apache.org/jira/browse/HADOOP-7896 Project: Hadoop Common Issue Type: Sub-task Components: ha, ipc Affects Versions: HA Branch (HDFS-1623) Reporter: Todd Lipcon Assignee: Aaron T. Myers Priority: Critical Attachments: HADOOP-7896-HDFS-1623.patch, HADOOP-7896-HDFS-1623.patch, HADOOP-7896-HDFS-1623.patch For a manual failover, there may be an intermediate state for a non-trivial amount of time where both NNs are in standby mode. Currently, the failover proxy will immediately failover on receiving this exception from the first NN, and when it hits the same exception on the second NN, it immediately fails. It should probably fail back and forth nearly indefinitely if both NNs are in Standby mode. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-7896) HA: if both NNs are in Standby mode, client needs to try failing back and forth several times with sleeps
[ https://issues.apache.org/jira/browse/HADOOP-7896?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron T. Myers updated HADOOP-7896: --- Attachment: HADOOP-7896-HDFS-1623.patch Here's a patch which addresses the issue. HA: if both NNs are in Standby mode, client needs to try failing back and forth several times with sleeps - Key: HADOOP-7896 URL: https://issues.apache.org/jira/browse/HADOOP-7896 Project: Hadoop Common Issue Type: Sub-task Components: ipc Affects Versions: HA Branch (HDFS-1623) Reporter: Todd Lipcon Assignee: Aaron T. Myers Priority: Critical Attachments: HADOOP-7896-HDFS-1623.patch For a manual failover, there may be an intermediate state for a non-trivial amount of time where both NNs are in standby mode. Currently, the failover proxy will immediately failover on receiving this exception from the first NN, and when it hits the same exception on the second NN, it immediately fails. It should probably fail back and forth nearly indefinitely if both NNs are in Standby mode. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-7888) TestFailoverProxy fails intermittently on trunk
[ https://issues.apache.org/jira/browse/HADOOP-7888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron T. Myers updated HADOOP-7888: --- Resolution: Fixed Fix Version/s: 0.24.0 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) I've just committed this to trunk. Thanks a lot, Jason! TestFailoverProxy fails intermittently on trunk --- Key: HADOOP-7888 URL: https://issues.apache.org/jira/browse/HADOOP-7888 Project: Hadoop Common Issue Type: Bug Components: test Affects Versions: 0.24.0 Reporter: Jason Lowe Assignee: Jason Lowe Fix For: 0.24.0 Attachments: hadoop-7888.patch TestFailoverProxy can fail intermittently with the failures occurring in testConcurrentMethodFailures(). The test has a race condition where the two threads may be sequentially invoking the unreliable interface rather than concurrently. Currently the proxy provider's getProxy() method contains the thread synchronization to enforce a concurrent invocation, but examining the source to RetryInvocationHandler.invoke() shows that the call to getProxy() during failover is too late to enforce a truly concurrent invocation. For this particular test, one thread could race ahead and block on the CountDownLatch in getProxy() before the other thread even enters RetryInvocationHandler.invoke(). If that happens the second thread will cache the newly updated value for proxyProviderFailoverCount, since the failover has mostly been processed by the original thread. Therefore the second thread ends up assuming no other thread is present, performs a failover, and the test fails because two failovers occurred instead of one. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-7774) HA: Administrative CLI to control HA daemons
[ https://issues.apache.org/jira/browse/HADOOP-7774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron T. Myers updated HADOOP-7774: --- Issue Type: Sub-task (was: New Feature) Parent: HADOOP-7454 HA: Administrative CLI to control HA daemons Key: HADOOP-7774 URL: https://issues.apache.org/jira/browse/HADOOP-7774 Project: Hadoop Common Issue Type: Sub-task Components: util Affects Versions: HA Branch (HDFS-1623) Reporter: Aaron T. Myers Assignee: Todd Lipcon Attachments: hdfs-2487.txt We'll need to have some way of controlling the HA nodes while they're live, probably by adding some more commands to dfsadmin. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-7695) RPC.stopProxy can throw unintended exception while logging error
[
https://issues.apache.org/jira/browse/HADOOP-7695?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aaron T. Myers updated HADOOP-7695:
---
Resolution: Fixed
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Thanks a lot for the review, Todd. I've just committed this.
Thanks also to Uma for noticing this issue.
RPC.stopProxy can throw unintended exception while logging error
Key: HADOOP-7695
URL: https://issues.apache.org/jira/browse/HADOOP-7695
Project: Hadoop Common
Issue Type: Bug
Components: ipc
Affects Versions: 0.24.0
Reporter: Aaron T. Myers
Assignee: Aaron T. Myers
Fix For: 0.24.0
Attachments: HADOOP-7695.patch
{{RPC.stopProxy}} includes the following lines in case of error:
{code}
LOG.error(Could not get invocation handler + invocationHandler +
for proxy + proxy + , or invocation handler is not
closeable.);
{code}
Trouble is, the {{proxy}} object is usually backed by {{WritableRpcEngine}},
which will fail in the event {{toString}} is called on one of its proxy
objects. See HADOOP-7694 for more details on that issue. Until that's
addressed, we might as well change the log message in {{RPC.stopProxy}} to
not call {{toString()}} on {{proxy}}.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-7717) Move handling of concurrent client fail-overs to RetryInvocationHandler
[
https://issues.apache.org/jira/browse/HADOOP-7717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aaron T. Myers updated HADOOP-7717:
---
Resolution: Fixed
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Thanks a lot for the review, Todd. I've just committed this.
Move handling of concurrent client fail-overs to RetryInvocationHandler
---
Key: HADOOP-7717
URL: https://issues.apache.org/jira/browse/HADOOP-7717
Project: Hadoop Common
Issue Type: Improvement
Components: ipc
Affects Versions: 0.24.0
Reporter: Aaron T. Myers
Assignee: Aaron T. Myers
Fix For: 0.24.0
Attachments: HADOOP-7717.patch, HADOOP-7717.patch
Currently every implementation of a {{FailoverProxyProvider}} will need to
perform its own synchronization to ensure that multiple concurrent failed
calls to a single client proxy object don't result in multiple client fail
over events. It would be better to put this logic in
{{RetryInvocationHandler.invoke}}.
This is based on feedback provided by Todd in [this
comment|https://issues.apache.org/jira/browse/HDFS-1973?focusedCommentId=13119567page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13119567].
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-7717) Move handling of concurrent client fail-overs to RetryInvocationHandler
[
https://issues.apache.org/jira/browse/HADOOP-7717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aaron T. Myers updated HADOOP-7717:
---
Status: Patch Available (was: Open)
Move handling of concurrent client fail-overs to RetryInvocationHandler
---
Key: HADOOP-7717
URL: https://issues.apache.org/jira/browse/HADOOP-7717
Project: Hadoop Common
Issue Type: Improvement
Components: ipc
Affects Versions: 0.24.0
Reporter: Aaron T. Myers
Assignee: Aaron T. Myers
Fix For: 0.24.0
Attachments: HADOOP-7717.patch
Currently every implementation of a {{FailoverProxyProvider}} will need to
perform its own synchronization to ensure that multiple concurrent failed
calls to a single client proxy object don't result in multiple client fail
over events. It would be better to put this logic in
{{RetryInvocationHandler.invoke}}.
This is based on feedback provided by Todd in [this
comment|https://issues.apache.org/jira/browse/HDFS-1973?focusedCommentId=13119567page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13119567].
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-7717) Move handling of concurrent client fail-overs to RetryInvocationHandler
[
https://issues.apache.org/jira/browse/HADOOP-7717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aaron T. Myers updated HADOOP-7717:
---
Attachment: HADOOP-7717.patch
Thanks a lot for the review, Todd. I agree with your analysis. Here's an
updated patch which addresses your concern.
I also took this opportunity to change some of the fail-over count variable
names, which hopefully will make things more clear.
Move handling of concurrent client fail-overs to RetryInvocationHandler
---
Key: HADOOP-7717
URL: https://issues.apache.org/jira/browse/HADOOP-7717
Project: Hadoop Common
Issue Type: Improvement
Components: ipc
Affects Versions: 0.24.0
Reporter: Aaron T. Myers
Assignee: Aaron T. Myers
Fix For: 0.24.0
Attachments: HADOOP-7717.patch, HADOOP-7717.patch
Currently every implementation of a {{FailoverProxyProvider}} will need to
perform its own synchronization to ensure that multiple concurrent failed
calls to a single client proxy object don't result in multiple client fail
over events. It would be better to put this logic in
{{RetryInvocationHandler.invoke}}.
This is based on feedback provided by Todd in [this
comment|https://issues.apache.org/jira/browse/HDFS-1973?focusedCommentId=13119567page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13119567].
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-7695) RPC.stopProxy can throw unintended exception while logging error
[
https://issues.apache.org/jira/browse/HADOOP-7695?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aaron T. Myers updated HADOOP-7695:
---
Attachment: HADOOP-7695.patch
Here's a patch which addresses the issue.
Uma, there's not much reason to check that {{invocationHandler}} is null, since
that object is likely not in fact a proxy object, so it should be safe to call
{{toString()}} on it. I also prefer to explicitly include {{null}} in the
string, instead of {{proxy}} which happens to be {{null}}, to make things more
clear.
Does this look OK to you?
RPC.stopProxy can throw unintended exception while logging error
Key: HADOOP-7695
URL: https://issues.apache.org/jira/browse/HADOOP-7695
Project: Hadoop Common
Issue Type: Bug
Components: ipc
Affects Versions: 0.24.0
Reporter: Aaron T. Myers
Assignee: Aaron T. Myers
Fix For: 0.24.0
Attachments: HADOOP-7695.patch
{{RPC.stopProxy}} includes the following lines in case of error:
{code}
LOG.error(Could not get invocation handler + invocationHandler +
for proxy + proxy + , or invocation handler is not
closeable.);
{code}
Trouble is, the {{proxy}} object is usually backed by {{WritableRpcEngine}},
which will fail in the event {{toString}} is called on one of its proxy
objects. See HADOOP-7694 for more details on that issue. Until that's
addressed, we might as well change the log message in {{RPC.stopProxy}} to
not call {{toString()}} on {{proxy}}.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
