[jira] [Updated] (HADOOP-9523) Provide a generic IBM java vendor flag in PlatformName.java to support non-Sun JREs
[ https://issues.apache.org/jira/browse/HADOOP-9523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tian Hong Wang updated HADOOP-9523: --- Attachment: HADOOP-9523-fix.patch Provide a generic IBM java vendor flag in PlatformName.java to support non-Sun JREs --- Key: HADOOP-9523 URL: https://issues.apache.org/jira/browse/HADOOP-9523 Project: Hadoop Common Issue Type: Improvement Affects Versions: 2.0.4-alpha Reporter: Tian Hong Wang Assignee: Tian Hong Wang Labels: patch Fix For: 2.0.5-beta Attachments: HADOOP-9523-fix.patch, HADOOP-9523.patch, HADOOP-9523-v1.patch, HADOOP-9523-v2.patch There are several different points between Sun jdk IBM jdk, so there is a need to provide a generic IBM java vendor flag. So enhance PlatformName.java to add Java vendor information. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9523) Provide a generic IBM java vendor flag in PlatformName.java to support non-Sun JREs
[
https://issues.apache.org/jira/browse/HADOOP-9523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13658090#comment-13658090
]
Suresh Srinivas commented on HADOOP-9523:
-
First, this is not an incompatible change. However, we can revert this patch
and do a patch so that the downstream project does not need to change. If you
do so, I will revert this patch and commit a new one. Please add
InterfaceAudience.LimitedPrivate({HBase}) to the class in the new version of
the patch.
Provide a generic IBM java vendor flag in PlatformName.java to support
non-Sun JREs
---
Key: HADOOP-9523
URL: https://issues.apache.org/jira/browse/HADOOP-9523
Project: Hadoop Common
Issue Type: Improvement
Affects Versions: 2.0.4-alpha
Reporter: Tian Hong Wang
Assignee: Tian Hong Wang
Labels: patch
Fix For: 2.0.5-beta
Attachments: HADOOP-9523-fix.patch, HADOOP-9523.patch,
HADOOP-9523-v1.patch, HADOOP-9523-v2.patch
There are several different points between Sun jdk IBM jdk, so there is a
need to provide a generic IBM java vendor flag. So enhance PlatformName.java
to add Java vendor information.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9523) Provide a generic IBM java vendor flag in PlatformName.java to support non-Sun JREs
[ https://issues.apache.org/jira/browse/HADOOP-9523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13658094#comment-13658094 ] Tian Hong Wang commented on HADOOP-9523: Suresh, I am not sure if this patch should be undo or the code in hbase should be changed because of interfaceAudience.Private in hadoop core. But anyway, I add a fix patch to support hbase startup script. Provide a generic IBM java vendor flag in PlatformName.java to support non-Sun JREs --- Key: HADOOP-9523 URL: https://issues.apache.org/jira/browse/HADOOP-9523 Project: Hadoop Common Issue Type: Improvement Affects Versions: 2.0.4-alpha Reporter: Tian Hong Wang Assignee: Tian Hong Wang Labels: patch Fix For: 2.0.5-beta Attachments: HADOOP-9523-fix.patch, HADOOP-9523.patch, HADOOP-9523-v1.patch, HADOOP-9523-v2.patch There are several different points between Sun jdk IBM jdk, so there is a need to provide a generic IBM java vendor flag. So enhance PlatformName.java to add Java vendor information. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9523) Provide a generic IBM java vendor flag in PlatformName.java to support non-Sun JREs
[ https://issues.apache.org/jira/browse/HADOOP-9523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tian Hong Wang updated HADOOP-9523: --- Attachment: (was: HADOOP-9523-fix.patch) Provide a generic IBM java vendor flag in PlatformName.java to support non-Sun JREs --- Key: HADOOP-9523 URL: https://issues.apache.org/jira/browse/HADOOP-9523 Project: Hadoop Common Issue Type: Improvement Affects Versions: 2.0.4-alpha Reporter: Tian Hong Wang Assignee: Tian Hong Wang Labels: patch Fix For: 2.0.5-beta Attachments: HADOOP-9523-fix.patch, HADOOP-9523.patch, HADOOP-9523-v1.patch, HADOOP-9523-v2.patch There are several different points between Sun jdk IBM jdk, so there is a need to provide a generic IBM java vendor flag. So enhance PlatformName.java to add Java vendor information. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9523) Provide a generic IBM java vendor flag in PlatformName.java to support non-Sun JREs
[ https://issues.apache.org/jira/browse/HADOOP-9523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tian Hong Wang updated HADOOP-9523: --- Attachment: HADOOP-9523-fix.patch Provide a generic IBM java vendor flag in PlatformName.java to support non-Sun JREs --- Key: HADOOP-9523 URL: https://issues.apache.org/jira/browse/HADOOP-9523 Project: Hadoop Common Issue Type: Improvement Affects Versions: 2.0.4-alpha Reporter: Tian Hong Wang Assignee: Tian Hong Wang Labels: patch Fix For: 2.0.5-beta Attachments: HADOOP-9523-fix.patch, HADOOP-9523.patch, HADOOP-9523-v1.patch, HADOOP-9523-v2.patch There are several different points between Sun jdk IBM jdk, so there is a need to provide a generic IBM java vendor flag. So enhance PlatformName.java to add Java vendor information. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9523) Provide a generic IBM java vendor flag in PlatformName.java to support non-Sun JREs
[ https://issues.apache.org/jira/browse/HADOOP-9523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13658105#comment-13658105 ] Tian Hong Wang commented on HADOOP-9523: Suresh, I add a new patch HADOOP-9523-fix.patch, you needn't revert the old patch, just commit HADOOP-9523-fix.patch. Provide a generic IBM java vendor flag in PlatformName.java to support non-Sun JREs --- Key: HADOOP-9523 URL: https://issues.apache.org/jira/browse/HADOOP-9523 Project: Hadoop Common Issue Type: Improvement Affects Versions: 2.0.4-alpha Reporter: Tian Hong Wang Assignee: Tian Hong Wang Labels: patch Fix For: 2.0.5-beta Attachments: HADOOP-9523-fix.patch, HADOOP-9523.patch, HADOOP-9523-v1.patch, HADOOP-9523-v2.patch There are several different points between Sun jdk IBM jdk, so there is a need to provide a generic IBM java vendor flag. So enhance PlatformName.java to add Java vendor information. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (HADOOP-9523) Provide a generic IBM java vendor flag in PlatformName.java to support non-Sun JREs
[ https://issues.apache.org/jira/browse/HADOOP-9523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tian Hong Wang reopened HADOOP-9523: Provide a generic IBM java vendor flag in PlatformName.java to support non-Sun JREs --- Key: HADOOP-9523 URL: https://issues.apache.org/jira/browse/HADOOP-9523 Project: Hadoop Common Issue Type: Improvement Affects Versions: 2.0.4-alpha Reporter: Tian Hong Wang Assignee: Tian Hong Wang Labels: patch Fix For: 2.0.5-beta Attachments: HADOOP-9523-fix.patch, HADOOP-9523.patch, HADOOP-9523-v1.patch, HADOOP-9523-v2.patch There are several different points between Sun jdk IBM jdk, so there is a need to provide a generic IBM java vendor flag. So enhance PlatformName.java to add Java vendor information. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9523) Provide a generic IBM java vendor flag in PlatformName.java to support non-Sun JREs
[ https://issues.apache.org/jira/browse/HADOOP-9523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tian Hong Wang updated HADOOP-9523: --- Status: Patch Available (was: Reopened) Provide a generic IBM java vendor flag in PlatformName.java to support non-Sun JREs --- Key: HADOOP-9523 URL: https://issues.apache.org/jira/browse/HADOOP-9523 Project: Hadoop Common Issue Type: Improvement Affects Versions: 2.0.4-alpha Reporter: Tian Hong Wang Assignee: Tian Hong Wang Labels: patch Fix For: 2.0.5-beta Attachments: HADOOP-9523-fix.patch, HADOOP-9523.patch, HADOOP-9523-v1.patch, HADOOP-9523-v2.patch There are several different points between Sun jdk IBM jdk, so there is a need to provide a generic IBM java vendor flag. So enhance PlatformName.java to add Java vendor information. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9523) Provide a generic IBM java vendor flag in PlatformName.java to support non-Sun JREs
[ https://issues.apache.org/jira/browse/HADOOP-9523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13658110#comment-13658110 ] Suresh Srinivas commented on HADOOP-9523: - You can post this fix to HADOOP-9563. Please assign that jira to yourself. Thanks for quickly addressing the issue. Provide a generic IBM java vendor flag in PlatformName.java to support non-Sun JREs --- Key: HADOOP-9523 URL: https://issues.apache.org/jira/browse/HADOOP-9523 Project: Hadoop Common Issue Type: Improvement Affects Versions: 2.0.4-alpha Reporter: Tian Hong Wang Assignee: Tian Hong Wang Labels: patch Fix For: 2.0.5-beta Attachments: HADOOP-9523-fix.patch, HADOOP-9523.patch, HADOOP-9523-v1.patch, HADOOP-9523-v2.patch There are several different points between Sun jdk IBM jdk, so there is a need to provide a generic IBM java vendor flag. So enhance PlatformName.java to add Java vendor information. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HADOOP-9563) Fix incompatibility introduced by HADOOP-9523
[ https://issues.apache.org/jira/browse/HADOOP-9563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tian Hong Wang reassigned HADOOP-9563: -- Assignee: Tian Hong Wang Fix incompatibility introduced by HADOOP-9523 - Key: HADOOP-9563 URL: https://issues.apache.org/jira/browse/HADOOP-9563 Project: Hadoop Common Issue Type: Bug Components: util Affects Versions: 3.0.0, 2.0.5-beta Reporter: Kihwal Lee Assignee: Tian Hong Wang HADOOP-9523 changed the output format of the platform name util, so hbase won't start unless platform name is specifically set. We could add a command line option to output in the new extended format and keep the old format when no option is given. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9564) DFSClient$DFSOutputStream.closeInternal locks up waiting for namenode.complete
[
https://issues.apache.org/jira/browse/HADOOP-9564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13658114#comment-13658114
]
Suresh Srinivas commented on HADOOP-9564:
-
Can you please see if you can duplicate this issue on an Apache release. It
most likely will happen on Apache release as well. If not, is it a good idea to
move this to CDH related jiras?
DFSClient$DFSOutputStream.closeInternal locks up waiting for namenode.complete
--
Key: HADOOP-9564
URL: https://issues.apache.org/jira/browse/HADOOP-9564
Project: Hadoop Common
Issue Type: Bug
Components: fs
Reporter: Jin Feng
Priority: Minor
Hi,
Our component uses FileSystem.copyFromLocalFile to copy a local file to HDFS
cluster. It's working fine in production environment. Its integration tests
used to run fine on our dev's local Mac laptop until recently (exact point of
time unknown) our tests started to freeze up very frequently with this stack:
{code}
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for 0x000152f41378 (a
java.util.concurrent.FutureTask$Sync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:994)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1303)
at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:248)
at java.util.concurrent.FutureTask.get(FutureTask.java:111)
at org.apache.hadoop.ipc.Client$Connection.sendParam(Client.java:790)
- locked 0x00014f568720 (a java.lang.Object)
at org.apache.hadoop.ipc.Client.call(Client.java:1080)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:226)
at $Proxy37.complete(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
at $Proxy37.complete(Unknown Source)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.closeInternal(DFSClient.java:3566)
- locked 0x000152f3f658 (a
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.close(DFSClient.java:3481)
at
org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:61)
at
org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:86)
at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:59)
at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:89)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:224)
at
org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:1295)
{code}
our version is 0.20.2.cdh3u2-t1.
In the test suite, we use org.apache.hadoop.hdfs.MiniDFSCluster. I've
searched around couldn't find anything resembles this symptom, any helps are
really appreciated!
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9563) Fix incompatibility introduced by HADOOP-9523
[ https://issues.apache.org/jira/browse/HADOOP-9563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13658115#comment-13658115 ] Tian Hong Wang commented on HADOOP-9563: Add patch to support hbase startup script. Fix incompatibility introduced by HADOOP-9523 - Key: HADOOP-9563 URL: https://issues.apache.org/jira/browse/HADOOP-9563 Project: Hadoop Common Issue Type: Bug Components: util Affects Versions: 3.0.0, 2.0.5-beta Reporter: Kihwal Lee Assignee: Tian Hong Wang Attachments: HADOOP-9563.patch HADOOP-9523 changed the output format of the platform name util, so hbase won't start unless platform name is specifically set. We could add a command line option to output in the new extended format and keep the old format when no option is given. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9563) Fix incompatibility introduced by HADOOP-9523
[ https://issues.apache.org/jira/browse/HADOOP-9563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tian Hong Wang updated HADOOP-9563: --- Attachment: HADOOP-9563.patch Fix incompatibility introduced by HADOOP-9523 - Key: HADOOP-9563 URL: https://issues.apache.org/jira/browse/HADOOP-9563 Project: Hadoop Common Issue Type: Bug Components: util Affects Versions: 3.0.0, 2.0.5-beta Reporter: Kihwal Lee Assignee: Tian Hong Wang Attachments: HADOOP-9563.patch HADOOP-9523 changed the output format of the platform name util, so hbase won't start unless platform name is specifically set. We could add a command line option to output in the new extended format and keep the old format when no option is given. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9563) Fix incompatibility introduced by HADOOP-9523
[ https://issues.apache.org/jira/browse/HADOOP-9563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tian Hong Wang updated HADOOP-9563: --- Status: Patch Available (was: Open) Fix incompatibility introduced by HADOOP-9523 - Key: HADOOP-9563 URL: https://issues.apache.org/jira/browse/HADOOP-9563 Project: Hadoop Common Issue Type: Bug Components: util Affects Versions: 3.0.0, 2.0.5-beta Reporter: Kihwal Lee Assignee: Tian Hong Wang Attachments: HADOOP-9563.patch HADOOP-9523 changed the output format of the platform name util, so hbase won't start unless platform name is specifically set. We could add a command line option to output in the new extended format and keep the old format when no option is given. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9523) Provide a generic IBM java vendor flag in PlatformName.java to support non-Sun JREs
[ https://issues.apache.org/jira/browse/HADOOP-9523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13658117#comment-13658117 ] Tian Hong Wang commented on HADOOP-9523: Suresh, fix patch moves to HADOOP-9563.patch Provide a generic IBM java vendor flag in PlatformName.java to support non-Sun JREs --- Key: HADOOP-9523 URL: https://issues.apache.org/jira/browse/HADOOP-9523 Project: Hadoop Common Issue Type: Improvement Affects Versions: 2.0.4-alpha Reporter: Tian Hong Wang Assignee: Tian Hong Wang Labels: patch Fix For: 2.0.5-beta Attachments: HADOOP-9523-fix.patch, HADOOP-9523.patch, HADOOP-9523-v1.patch, HADOOP-9523-v2.patch There are several different points between Sun jdk IBM jdk, so there is a need to provide a generic IBM java vendor flag. So enhance PlatformName.java to add Java vendor information. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9523) Provide a generic IBM java vendor flag in PlatformName.java to support non-Sun JREs
[ https://issues.apache.org/jira/browse/HADOOP-9523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tian Hong Wang updated HADOOP-9523: --- Status: Open (was: Patch Available) Provide a generic IBM java vendor flag in PlatformName.java to support non-Sun JREs --- Key: HADOOP-9523 URL: https://issues.apache.org/jira/browse/HADOOP-9523 Project: Hadoop Common Issue Type: Improvement Affects Versions: 2.0.4-alpha Reporter: Tian Hong Wang Assignee: Tian Hong Wang Labels: patch Fix For: 2.0.5-beta Attachments: HADOOP-9523-fix.patch, HADOOP-9523.patch, HADOOP-9523-v1.patch, HADOOP-9523-v2.patch There are several different points between Sun jdk IBM jdk, so there is a need to provide a generic IBM java vendor flag. So enhance PlatformName.java to add Java vendor information. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9564) DFSClient$DFSOutputStream.closeInternal locks up waiting for namenode.complete
[
https://issues.apache.org/jira/browse/HADOOP-9564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13658121#comment-13658121
]
Lohit Vijayarenu commented on HADOOP-9564:
--
Will try to see if this is something specific to environment and update this
JIRA.
DFSClient$DFSOutputStream.closeInternal locks up waiting for namenode.complete
--
Key: HADOOP-9564
URL: https://issues.apache.org/jira/browse/HADOOP-9564
Project: Hadoop Common
Issue Type: Bug
Components: fs
Reporter: Jin Feng
Priority: Minor
Hi,
Our component uses FileSystem.copyFromLocalFile to copy a local file to HDFS
cluster. It's working fine in production environment. Its integration tests
used to run fine on our dev's local Mac laptop until recently (exact point of
time unknown) our tests started to freeze up very frequently with this stack:
{code}
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for 0x000152f41378 (a
java.util.concurrent.FutureTask$Sync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:994)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1303)
at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:248)
at java.util.concurrent.FutureTask.get(FutureTask.java:111)
at org.apache.hadoop.ipc.Client$Connection.sendParam(Client.java:790)
- locked 0x00014f568720 (a java.lang.Object)
at org.apache.hadoop.ipc.Client.call(Client.java:1080)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:226)
at $Proxy37.complete(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
at $Proxy37.complete(Unknown Source)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.closeInternal(DFSClient.java:3566)
- locked 0x000152f3f658 (a
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.close(DFSClient.java:3481)
at
org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:61)
at
org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:86)
at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:59)
at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:89)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:224)
at
org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:1295)
{code}
our version is 0.20.2.cdh3u2-t1.
In the test suite, we use org.apache.hadoop.hdfs.MiniDFSCluster. I've
searched around couldn't find anything resembles this symptom, any helps are
really appreciated!
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9523) Provide a generic IBM java vendor flag in PlatformName.java to support non-Sun JREs
[
https://issues.apache.org/jira/browse/HADOOP-9523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13658128#comment-13658128
]
Hadoop QA commented on HADOOP-9523:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12583286/HADOOP-9523-fix.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/2542//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/2542//console
This message is automatically generated.
Provide a generic IBM java vendor flag in PlatformName.java to support
non-Sun JREs
---
Key: HADOOP-9523
URL: https://issues.apache.org/jira/browse/HADOOP-9523
Project: Hadoop Common
Issue Type: Improvement
Affects Versions: 2.0.4-alpha
Reporter: Tian Hong Wang
Assignee: Tian Hong Wang
Labels: patch
Fix For: 2.0.5-beta
Attachments: HADOOP-9523-fix.patch, HADOOP-9523.patch,
HADOOP-9523-v1.patch, HADOOP-9523-v2.patch
There are several different points between Sun jdk IBM jdk, so there is a
need to provide a generic IBM java vendor flag. So enhance PlatformName.java
to add Java vendor information.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9563) Fix incompatibility introduced by HADOOP-9523
[
https://issues.apache.org/jira/browse/HADOOP-9563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13658130#comment-13658130
]
Hadoop QA commented on HADOOP-9563:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12583287/HADOOP-9563.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/2543//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/2543//console
This message is automatically generated.
Fix incompatibility introduced by HADOOP-9523
-
Key: HADOOP-9563
URL: https://issues.apache.org/jira/browse/HADOOP-9563
Project: Hadoop Common
Issue Type: Bug
Components: util
Affects Versions: 3.0.0, 2.0.5-beta
Reporter: Kihwal Lee
Assignee: Tian Hong Wang
Attachments: HADOOP-9563.patch
HADOOP-9523 changed the output format of the platform name util, so hbase
won't start unless platform name is specifically set.
We could add a command line option to output in the new extended format and
keep the old format when no option is given.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9563) Fix incompatibility introduced by HADOOP-9523
[ https://issues.apache.org/jira/browse/HADOOP-9563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13658138#comment-13658138 ] Tian Hong Wang commented on HADOOP-9563: Suresh, can you include this patch? Fix incompatibility introduced by HADOOP-9523 - Key: HADOOP-9563 URL: https://issues.apache.org/jira/browse/HADOOP-9563 Project: Hadoop Common Issue Type: Bug Components: util Affects Versions: 3.0.0, 2.0.5-beta Reporter: Kihwal Lee Assignee: Tian Hong Wang Attachments: HADOOP-9563.patch HADOOP-9523 changed the output format of the platform name util, so hbase won't start unless platform name is specifically set. We could add a command line option to output in the new extended format and keep the old format when no option is given. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9563) Fix incompatibility introduced by HADOOP-9523
[ https://issues.apache.org/jira/browse/HADOOP-9563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13658146#comment-13658146 ] Suresh Srinivas commented on HADOOP-9563: - +1 for the patch. This essentially reverts the output of PlatformName main to what was prior to HADOOP-9523. Fix incompatibility introduced by HADOOP-9523 - Key: HADOOP-9563 URL: https://issues.apache.org/jira/browse/HADOOP-9563 Project: Hadoop Common Issue Type: Bug Components: util Affects Versions: 3.0.0, 2.0.5-beta Reporter: Kihwal Lee Assignee: Tian Hong Wang Attachments: HADOOP-9563.patch HADOOP-9523 changed the output format of the platform name util, so hbase won't start unless platform name is specifically set. We could add a command line option to output in the new extended format and keep the old format when no option is given. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9563) Fix incompatibility introduced by HADOOP-9523
[ https://issues.apache.org/jira/browse/HADOOP-9563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas updated HADOOP-9563: Resolution: Fixed Fix Version/s: 2.0.5-beta Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) I committed the patch to trunk and branch-2. Thank you Tian for quickly fixing this issue. Fix incompatibility introduced by HADOOP-9523 - Key: HADOOP-9563 URL: https://issues.apache.org/jira/browse/HADOOP-9563 Project: Hadoop Common Issue Type: Bug Components: util Affects Versions: 3.0.0, 2.0.5-beta Reporter: Kihwal Lee Assignee: Tian Hong Wang Fix For: 2.0.5-beta Attachments: HADOOP-9563.patch HADOOP-9523 changed the output format of the platform name util, so hbase won't start unless platform name is specifically set. We could add a command line option to output in the new extended format and keep the old format when no option is given. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9563) Fix incompatibility introduced by HADOOP-9523
[ https://issues.apache.org/jira/browse/HADOOP-9563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13658151#comment-13658151 ] Tian Hong Wang commented on HADOOP-9563: Thanks Suresh for your comment. Fix incompatibility introduced by HADOOP-9523 - Key: HADOOP-9563 URL: https://issues.apache.org/jira/browse/HADOOP-9563 Project: Hadoop Common Issue Type: Bug Components: util Affects Versions: 3.0.0, 2.0.5-beta Reporter: Kihwal Lee Assignee: Tian Hong Wang Fix For: 2.0.5-beta Attachments: HADOOP-9563.patch HADOOP-9523 changed the output format of the platform name util, so hbase won't start unless platform name is specifically set. We could add a command line option to output in the new extended format and keep the old format when no option is given. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9523) Provide a generic IBM java vendor flag in PlatformName.java to support non-Sun JREs
[ https://issues.apache.org/jira/browse/HADOOP-9523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas updated HADOOP-9523: Hadoop Flags: Reviewed (was: Incompatible change,Reviewed) Removed incompatible flag with HADOOP-9563 undoing the change that causes issues for the downstream project. Provide a generic IBM java vendor flag in PlatformName.java to support non-Sun JREs --- Key: HADOOP-9523 URL: https://issues.apache.org/jira/browse/HADOOP-9523 Project: Hadoop Common Issue Type: Improvement Affects Versions: 2.0.4-alpha Reporter: Tian Hong Wang Assignee: Tian Hong Wang Labels: patch Fix For: 2.0.5-beta Attachments: HADOOP-9523-fix.patch, HADOOP-9523.patch, HADOOP-9523-v1.patch, HADOOP-9523-v2.patch There are several different points between Sun jdk IBM jdk, so there is a need to provide a generic IBM java vendor flag. So enhance PlatformName.java to add Java vendor information. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9563) Fix incompatibility introduced by HADOOP-9523
[ https://issues.apache.org/jira/browse/HADOOP-9563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13658157#comment-13658157 ] Hudson commented on HADOOP-9563: Integrated in Hadoop-trunk-Commit #3755 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/3755/]) HADOOP-9563. Fix incompatibility introduced by HADOOP-9523. Contributed by Tian Hong Wang. (Revision 1482709) Result = SUCCESS suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1482709 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/PlatformName.java Fix incompatibility introduced by HADOOP-9523 - Key: HADOOP-9563 URL: https://issues.apache.org/jira/browse/HADOOP-9563 Project: Hadoop Common Issue Type: Bug Components: util Affects Versions: 3.0.0, 2.0.5-beta Reporter: Kihwal Lee Assignee: Tian Hong Wang Fix For: 2.0.5-beta Attachments: HADOOP-9563.patch HADOOP-9523 changed the output format of the platform name util, so hbase won't start unless platform name is specifically set. We could add a command line option to output in the new extended format and keep the old format when no option is given. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9523) Provide a generic IBM java vendor flag in PlatformName.java to support non-Sun JREs
[ https://issues.apache.org/jira/browse/HADOOP-9523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13658158#comment-13658158 ] Hudson commented on HADOOP-9523: Integrated in Hadoop-trunk-Commit #3755 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/3755/]) HADOOP-9563. Fix incompatibility introduced by HADOOP-9523. Contributed by Tian Hong Wang. (Revision 1482709) Result = SUCCESS suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1482709 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/PlatformName.java Provide a generic IBM java vendor flag in PlatformName.java to support non-Sun JREs --- Key: HADOOP-9523 URL: https://issues.apache.org/jira/browse/HADOOP-9523 Project: Hadoop Common Issue Type: Improvement Affects Versions: 2.0.4-alpha Reporter: Tian Hong Wang Assignee: Tian Hong Wang Labels: patch Fix For: 2.0.5-beta Attachments: HADOOP-9523-fix.patch, HADOOP-9523.patch, HADOOP-9523-v1.patch, HADOOP-9523-v2.patch There are several different points between Sun jdk IBM jdk, so there is a need to provide a generic IBM java vendor flag. So enhance PlatformName.java to add Java vendor information. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HADOOP-9523) Provide a generic IBM java vendor flag in PlatformName.java to support non-Sun JREs
[ https://issues.apache.org/jira/browse/HADOOP-9523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas resolved HADOOP-9523. - Resolution: Fixed Marking the issue as resolved. Provide a generic IBM java vendor flag in PlatformName.java to support non-Sun JREs --- Key: HADOOP-9523 URL: https://issues.apache.org/jira/browse/HADOOP-9523 Project: Hadoop Common Issue Type: Improvement Affects Versions: 2.0.4-alpha Reporter: Tian Hong Wang Assignee: Tian Hong Wang Labels: patch Fix For: 2.0.5-beta Attachments: HADOOP-9523-fix.patch, HADOOP-9523.patch, HADOOP-9523-v1.patch, HADOOP-9523-v2.patch There are several different points between Sun jdk IBM jdk, so there is a need to provide a generic IBM java vendor flag. So enhance PlatformName.java to add Java vendor information. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9562) Create REST interface for HDFS health data
[
https://issues.apache.org/jira/browse/HADOOP-9562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13658168#comment-13658168
]
Suresh Srinivas commented on HADOOP-9562:
-
{quote}
The HDFS health screen (dfshealth.jsp) displays basic Version, Security and
Health information concerning the NameNode, currently this information is
accessible from classes in the org.apache.hadoop,hdfs.server.namenode package
and cannot be accessed outside the NameNode.
{quote}
This is not correct. While doing standardized REST APIs may not be a bad idea,
all of this information is exposed through {{FSNamesystemMBean}} and
{{NameNodeMXBean}} and the corresponding JMX http bridge (see HADOOP-7144 and
related jiras). This is what currently Ambari project uses.
Create REST interface for HDFS health data
--
Key: HADOOP-9562
URL: https://issues.apache.org/jira/browse/HADOOP-9562
Project: Hadoop Common
Issue Type: New Feature
Components: fs
Affects Versions: 2.0.4-alpha
Reporter: Trevor Lorimer
Priority: Minor
The HDFS health screen (dfshealth.jsp) displays basic Version, Security and
Health information concerning the NameNode, currently this information is
accessible from classes in the org.apache.hadoop,hdfs.server.namenode package
and cannot be accessed outside the NameNode. This becomes prevalent if the
data is required to be displayed using a new user interface.
The proposal is to create a REST interface to expose all the information
displayed on dfshealth.jsp using GET methods. Wrapper classes will be created
to serve the data to the REST root resource within the hadoop-hdfs project.
This will enable the HDFS health screen information to be accessed remotely.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (HADOOP-9562) Create REST interface for HDFS health data
[
https://issues.apache.org/jira/browse/HADOOP-9562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13658168#comment-13658168
]
Suresh Srinivas edited comment on HADOOP-9562 at 5/15/13 8:29 AM:
--
{quote}
The HDFS health screen (dfshealth.jsp) displays basic Version, Security and
Health information concerning the NameNode, currently this information is
accessible from classes in the org.apache.hadoop,hdfs.server.namenode package
and cannot be accessed outside the NameNode.
{quote}
This is not correct. While doing standardized REST APIs may not be a bad idea,
all of this information is exposed in namnode webUI is exposed through
{{FSNamesystemMBean}} and {{NameNodeMXBean}} and the corresponding JMX http
bridge (see HADOOP-7144 and related jiras). This is what currently Ambari
project uses.
was (Author: sureshms):
{quote}
The HDFS health screen (dfshealth.jsp) displays basic Version, Security and
Health information concerning the NameNode, currently this information is
accessible from classes in the org.apache.hadoop,hdfs.server.namenode package
and cannot be accessed outside the NameNode.
{quote}
This is not correct. While doing standardized REST APIs may not be a bad idea,
all of this information is exposed through {{FSNamesystemMBean}} and
{{NameNodeMXBean}} and the corresponding JMX http bridge (see HADOOP-7144 and
related jiras). This is what currently Ambari project uses.
Create REST interface for HDFS health data
--
Key: HADOOP-9562
URL: https://issues.apache.org/jira/browse/HADOOP-9562
Project: Hadoop Common
Issue Type: New Feature
Components: fs
Affects Versions: 2.0.4-alpha
Reporter: Trevor Lorimer
Priority: Minor
The HDFS health screen (dfshealth.jsp) displays basic Version, Security and
Health information concerning the NameNode, currently this information is
accessible from classes in the org.apache.hadoop,hdfs.server.namenode package
and cannot be accessed outside the NameNode. This becomes prevalent if the
data is required to be displayed using a new user interface.
The proposal is to create a REST interface to expose all the information
displayed on dfshealth.jsp using GET methods. Wrapper classes will be created
to serve the data to the REST root resource within the hadoop-hdfs project.
This will enable the HDFS health screen information to be accessed remotely.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9435) Minimal code change to support IBM jvm build dependency
[ https://issues.apache.org/jira/browse/HADOOP-9435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13658193#comment-13658193 ] Tian Hong Wang commented on HADOOP-9435: Thanks Colin for your comment. Minimal code change to support IBM jvm build dependency --- Key: HADOOP-9435 URL: https://issues.apache.org/jira/browse/HADOOP-9435 Project: Hadoop Common Issue Type: Bug Components: build Reporter: Tian Hong Wang Assignee: Tian Hong Wang Labels: patch Attachments: HADOOP-9435.patch, HADOOP-9435-v1.patch When native build hadoop-common-project with IBM java using command like: mvn package -Pnative it will exist the following errors. [exec] -- Configuring incomplete, errors occurred! [exec] JAVA_HOME=, JAVA_JVM_LIBRARY=/home/louis/ibm-java-i386-60/jre/lib/i386/classic/libjvm.so [exec] JAVA_INCLUDE_PATH=/home/louis/ibm-java-i386-60/include, JAVA_INCLUDE_PATH2=JAVA_INCLUDE_PATH2-NOTFOUND [exec] CMake Error at JNIFlags.cmake:113 (MESSAGE): [exec] Failed to find a viable JVM installation under JAVA_HOME. [exec] Call Stack (most recent call first): [exec] CMakeLists.txt:24 (include) The reason is that IBM java uses $JAVA_HOME/include/jniport.h instead of $JAVA_HOME/include/jni_md.h in non-IBM java. [exec] /usr/lib/jvm/java-1.6.0-ibm-1.6.0.12.0.x86_64/jre/lib/amd64/default/libjvm.so: undefined reference to `dlsym' [exec] /usr/lib/jvm/java-1.6.0-ibm-1.6.0.12.0.x86_64/jre/lib/amd64/default/libjvm.so: undefined reference to `dlerror' [exec] /usr/lib/jvm/java-1.6.0-ibm-1.6.0.12.0.x86_64/jre/lib/amd64/default/libjvm.so: undefined reference to `dladdr' [exec] /usr/lib/jvm/java-1.6.0-ibm-1.6.0.12.0.x86_64/jre/lib/amd64/default/libjvm.so: undefined reference to `dlopen' [exec] /usr/lib/jvm/java-1.6.0-ibm-1.6.0.12.0.x86_64/jre/lib/amd64/default/libjvm.so: undefined reference to `dlclose' [exec] collect2: ld returned 1 exit status [exec] make[2]: *** [test_libhdfs_ops] Error 1 [exec] make[1]: *** [CMakeFiles/test_libhdfs_ops.dir/all] Error 2 [exec] make: *** [all] Error The reason is libjvm.so need libdl when linking. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9421) Generalize SASL Support with Protocol Buffer
[ https://issues.apache.org/jira/browse/HADOOP-9421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13658221#comment-13658221 ] Kai Zheng commented on HADOOP-9421: --- Luke good point. It’s great to resolve this issue if “we have a mechanism to evolve SASL mechanisms/protocols negotiation without breaking general RPC backward compatibility” with the RPC cleanup. With this support new authentication mechanisms can continue to be implemented in current way as Daryn might be doing. And based on the mentioned mechanism hopefully an opaque token can also be employed or at least supported without breaking RPC backward compatibility as this JIRA desires. Such opaque token is needed by token authentication and single sign on in (HADOOP-9392) where new authentication mechanism can be implemented and plugin-ed via the token without involving to change this RPC work again. So my question would be: 1. Do we have any limit for the optional token field and what properties it must be of? Variable of bytes would be great; 2. Do we have a flag field or something like that to mark the token type in client side, and in server side it can interpret the token accordingly to the type? 3. Hopefully we can add token authentication handler with relevant callbacks in elegant way. Thanks for your consideration. Generalize SASL Support with Protocol Buffer Key: HADOOP-9421 URL: https://issues.apache.org/jira/browse/HADOOP-9421 Project: Hadoop Common Issue Type: Sub-task Affects Versions: 2.0.3-alpha Reporter: Sanjay Radia Assignee: Junping Du Attachments: HADOOP-9421.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9392) Token based authentication and Single Sign On
[ https://issues.apache.org/jira/browse/HADOOP-9392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13658225#comment-13658225 ] Kai Zheng commented on HADOOP-9392: --- Larry - You use an extra Service Access Token to “eliminate unnecessary re-authentication, limit the amount of damage that a compromised token can enable and ensure freshness of authorization related attributes”, however TokenAuth addresses these concerns without requiring two types of tokens. The Identity Token is cached at the client and will be reused for subsequent requests to avoid re-authentication. Tokens in the cache are expired at a configurable interval, typically a few hours instead of days. When the client requests access to a service, the token provided by the client is authenticated to the TAS configured for the service and at that time the TAS validates the token, verifies the associated identity is valid and also if the principal is still allowed to access the service. Furthermore identity attributes should be of the same freshness with the identity itself. As with other attributes that can be used for authorization, they should be collected and evaluated on a per request basis, and they often involve resources other than coarse-grained service level access considerations. Such attributes are often retrieved during authorization enforcement from Attribute Authority(AA), for example in XACML context it should be in PDP. We cover fine-grained authorization in the Unified Authorization Framework (HADOOP-9466) based on TokenAuth. If there’s a relationship between a Service Access Token and an OAuth Access Token, I’m not sure it’s in the right way. In OAuth, the Access Token is issued by Authorization Server in an Authorization Grant from the Resource Owner. Do you intend for the HSSO service to take on the role of an OAuth Resource Owner? If yes then how will you handle service users? Please consider if we can avoid introducing additional tokens like the Service Access Token. Probably we can address your design concerns in other ways. Also, do we need two similar tokens for authentication and SSO in Hadoop (the Identity Token in TokenAuth, and the Cluster Access Token in HSSO)? I believe Hadoop only needs one. Cleanup of redundant constructs emitted from the two partially overlapping proposals here I would think is the first step toward aligning these efforts. We should consider a deployment model where we build HSSO layered on TAS. Sounds like a good plan for us to meet at the Hadoop Summit. I will try but not sure if I can make it. Others from the team who are also working on this will be there and participate in the discussion. If there is a bridge, I will try to join the bridge. Thanks for setting this up. Token based authentication and Single Sign On - Key: HADOOP-9392 URL: https://issues.apache.org/jira/browse/HADOOP-9392 Project: Hadoop Common Issue Type: New Feature Components: security Reporter: Kai Zheng Assignee: Kai Zheng Fix For: 3.0.0 Attachments: token-based-authn-plus-sso.pdf This is an umbrella entry for one of project Rhino’s topic, for details of project Rhino, please refer to https://github.com/intel-hadoop/project-rhino/. The major goal for this entry as described in project Rhino was “Core, HDFS, ZooKeeper, and HBase currently support Kerberos authentication at the RPC layer, via SASL. However this does not provide valuable attributes such as group membership, classification level, organizational identity, or support for user defined attributes. Hadoop components must interrogate external resources for discovering these attributes and at scale this is problematic. There is also no consistent delegation model. HDFS has a simple delegation capability, and only Oozie can take limited advantage of it. We will implement a common token based authentication framework to decouple internal user and service authentication from external mechanisms used to support it (like Kerberos)” We’d like to start our work from Hadoop-Common and try to provide common facilities by extending existing authentication framework which support: 1.Pluggable token provider interface 2.Pluggable token verification protocol and interface 3.Security mechanism to distribute secrets in cluster nodes 4.Delegation model of user authentication -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9307) BufferedFSInputStream.read returns wrong results after certain seeks
[ https://issues.apache.org/jira/browse/HADOOP-9307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13658229#comment-13658229 ] Hudson commented on HADOOP-9307: Integrated in Hadoop-Yarn-trunk #210 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/210/]) HADOOP-9307. BufferedFSInputStream.read returns wrong results after certain seeks. Contributed by Todd Lipcon. (Revision 1482377) Result = SUCCESS todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1482377 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/BufferedFSInputStream.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestLocalFileSystem.java BufferedFSInputStream.read returns wrong results after certain seeks Key: HADOOP-9307 URL: https://issues.apache.org/jira/browse/HADOOP-9307 Project: Hadoop Common Issue Type: Bug Components: fs Affects Versions: 1.1.1, 2.0.2-alpha Reporter: Todd Lipcon Assignee: Todd Lipcon Fix For: 3.0.0, 2.0.5-beta Attachments: hadoop-9307-branch-1.txt, hadoop-9307.txt After certain sequences of seek/read, BufferedFSInputStream can silently return data from the wrong part of the file. Further description in first comment below. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9220) Unnecessary transition to standby in ActiveStandbyElector
[ https://issues.apache.org/jira/browse/HADOOP-9220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13658230#comment-13658230 ] Hudson commented on HADOOP-9220: Integrated in Hadoop-Yarn-trunk #210 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/210/]) HADOOP-9220. Unnecessary transition to standby in ActiveStandbyElector. Contributed by Tom White and Todd Lipcon. (Revision 1482401) Result = SUCCESS todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1482401 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ha/ActiveStandbyElector.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ha/DummyHAService.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ha/TestZKFailoverController.java Unnecessary transition to standby in ActiveStandbyElector - Key: HADOOP-9220 URL: https://issues.apache.org/jira/browse/HADOOP-9220 Project: Hadoop Common Issue Type: Bug Components: ha Reporter: Tom White Assignee: Tom White Priority: Critical Fix For: 3.0.0, 2.0.5-beta Attachments: HADOOP-9220.patch, HADOOP-9220.patch, hadoop-9220.txt When performing a manual failover from one HA node to a second, under some circumstances the second node will transition from standby - active - standby - active. This is with automatic failover enabled, so there is a ZK cluster doing leader election. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9563) Fix incompatibility introduced by HADOOP-9523
[ https://issues.apache.org/jira/browse/HADOOP-9563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13658231#comment-13658231 ] Hudson commented on HADOOP-9563: Integrated in Hadoop-Yarn-trunk #210 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/210/]) HADOOP-9563. Fix incompatibility introduced by HADOOP-9523. Contributed by Tian Hong Wang. (Revision 1482709) Result = SUCCESS suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1482709 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/PlatformName.java Fix incompatibility introduced by HADOOP-9523 - Key: HADOOP-9563 URL: https://issues.apache.org/jira/browse/HADOOP-9563 Project: Hadoop Common Issue Type: Bug Components: util Affects Versions: 3.0.0, 2.0.5-beta Reporter: Kihwal Lee Assignee: Tian Hong Wang Fix For: 2.0.5-beta Attachments: HADOOP-9563.patch HADOOP-9523 changed the output format of the platform name util, so hbase won't start unless platform name is specifically set. We could add a command line option to output in the new extended format and keep the old format when no option is given. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9560) metrics2#JvmMetrics should have max memory size of JVM
[ https://issues.apache.org/jira/browse/HADOOP-9560?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13658232#comment-13658232 ] Hudson commented on HADOOP-9560: Integrated in Hadoop-Yarn-trunk #210 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/210/]) HADOOP-9560. metrics2#JvmMetrics should have max memory size of JVM. Contributed by Tsuyoshi Ozawa. (Revision 1482372) Result = SUCCESS suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1482372 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/source/JvmMetrics.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/source/JvmMetricsInfo.java metrics2#JvmMetrics should have max memory size of JVM -- Key: HADOOP-9560 URL: https://issues.apache.org/jira/browse/HADOOP-9560 Project: Hadoop Common Issue Type: Improvement Components: metrics Affects Versions: 3.0.0, 2.0.5-beta Reporter: Tsuyoshi OZAWA Assignee: Tsuyoshi OZAWA Priority: Minor Fix For: 2.0.5-beta Attachments: HADOOP-9560.1.patch metrics#JvmMetrics have the max memory size of JVM specified by -Xmx option. metrics2#JvmMetrics doesn't have it but it should also have max memory size of JVM, because it's useful for users. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9523) Provide a generic IBM java vendor flag in PlatformName.java to support non-Sun JREs
[ https://issues.apache.org/jira/browse/HADOOP-9523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13658235#comment-13658235 ] Hudson commented on HADOOP-9523: Integrated in Hadoop-Yarn-trunk #210 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/210/]) HADOOP-9563. Fix incompatibility introduced by HADOOP-9523. Contributed by Tian Hong Wang. (Revision 1482709) Result = SUCCESS suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1482709 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/PlatformName.java Provide a generic IBM java vendor flag in PlatformName.java to support non-Sun JREs --- Key: HADOOP-9523 URL: https://issues.apache.org/jira/browse/HADOOP-9523 Project: Hadoop Common Issue Type: Improvement Affects Versions: 2.0.4-alpha Reporter: Tian Hong Wang Assignee: Tian Hong Wang Labels: patch Fix For: 2.0.5-beta Attachments: HADOOP-9523-fix.patch, HADOOP-9523.patch, HADOOP-9523-v1.patch, HADOOP-9523-v2.patch There are several different points between Sun jdk IBM jdk, so there is a need to provide a generic IBM java vendor flag. So enhance PlatformName.java to add Java vendor information. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9440) Unit Test: hadoop-common2.0.3 TestIPC fails on protobuf2.5.0
[ https://issues.apache.org/jira/browse/HADOOP-9440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13658239#comment-13658239 ] Harsh J commented on HADOOP-9440: - Why do we get a protobuf exception in the first place? We shouldn't be modifying the test to capture an IOException as we don't expect that to happen. Please correct my understanding if am wrong. Unit Test: hadoop-common2.0.3 TestIPC fails on protobuf2.5.0 Key: HADOOP-9440 URL: https://issues.apache.org/jira/browse/HADOOP-9440 Project: Hadoop Common Issue Type: Bug Components: test Affects Versions: 2.0.5-beta Reporter: Tian Hong Wang Labels: patch Attachments: HADOOP-9440.patch TestIPC runs normally if use protobuf2.4.1 or below version. But if using protobuf2.5.0, TestIPC.testIpcTimeout TestIPC.testIpcConnectTimeout will fail. java.io.IOException: Failed on local exception: com.google.protobuf.InvalidProtocolBufferException: 500 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/127.0.0.1:50850 remote=louis-ThinkPad-T410/127.0.0.1:50353]; Host Details : local host is: louis-ThinkPad-T410/127.0.0.1; destination host is: louis-ThinkPad-T410:50353; at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:761) at org.apache.hadoop.ipc.Client.call(Client.java:1239) at org.apache.hadoop.ipc.Client.call(Client.java:1163) at org.apache.hadoop.ipc.TestIPC.testIpcTimeout(TestIPC.java:492) testIpcConnectTimeout(org.apache.hadoop.ipc.TestIPC) Time elapsed: 2009 sec ERROR! java.io.IOException: Failed on local exception: com.google.protobuf.InvalidProtocolBufferException: 2000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/127.0.0.1:51304 remote=louis-ThinkPad-T410/127.0.0.1:39525]; Host Details : local host is: louis-ThinkPad-T410/127.0.0.1; destination host is: louis-ThinkPad-T410:39525; at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:761) at org.apache.hadoop.ipc.Client.call(Client.java:1239) at org.apache.hadoop.ipc.Client.call(Client.java:1163) at org.apache.hadoop.ipc.TestIPC.testIpcConnectTimeout(TestIPC.java:515) TestIPC.testIpcTimeout TestIPC.testIpcConnectTimeout fails because it catches the com.google.protobuf.InvalidProtocolBufferException not SocketTimeoutException. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9046) provide unit-test coverage of class org.apache.hadoop.fs.DelegationTokenRenewer.RenewActionT
[ https://issues.apache.org/jira/browse/HADOOP-9046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ivan A. Veselovsky updated HADOOP-9046: --- Resolution: Cannot Reproduce Status: Resolved (was: Patch Available) Closing this request because: 1) After https://issues.apache.org/jira/browse/HADOOP-9549 coverage of the DelegationTokenRenewer is high enough in branch-2 and trunk. 2) In branch-0.23 class DelegationTokenRenewer resides in another package, so its coverage does not directly affect coverage of package o.a.h.fs . (The fix was initially done in order to elevate coverage of o.a.h.fs package). provide unit-test coverage of class org.apache.hadoop.fs.DelegationTokenRenewer.RenewActionT -- Key: HADOOP-9046 URL: https://issues.apache.org/jira/browse/HADOOP-9046 Project: Hadoop Common Issue Type: Test Affects Versions: 0.23.6 Reporter: Ivan A. Veselovsky Assignee: Ivan A. Veselovsky Priority: Minor Attachments: HADOOP-9046-branch-0.23--c.patch, HADOOP-9046-branch-0.23--d.patch, HADOOP-9046-branch-0.23-over-9049.patch, HADOOP-9046-branch-0.23--over-HDFS-4567.patch, HADOOP-9046-branch-0.23.patch, HADOOP-9046-branch-2--over-HDFS-4567.patch, HADOOP-9046--c.patch, HADOOP-9046--d.patch, HADOOP-9046--e.patch, HADOOP-9046-over-9049.patch, HADOOP-9046.patch, HADOOP-9046-trunk--over-HDFS-4567.patch The class org.apache.hadoop.fs.DelegationTokenRenewer.RenewActionT has zero coverage in entire cumulative test run. Provide test(s) to cover this class. Note: the request submitted to HDFS project because the class likely to be tested by tests in that project. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9392) Token based authentication and Single Sign On
[ https://issues.apache.org/jira/browse/HADOOP-9392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13658295#comment-13658295 ] Larry McCay commented on HADOOP-9392: - Hi Kai - my previous response hadn't really intended to compare the designs as much as correct something that - based on your interpretation - must have been understated in my client interaction overview on HADOOP-9533. Any decent design would strive to address these same goals - I didn't mean to imply that your's doesn't. Ultimately, we just want to make sure that we are covering all relevant goals and usecases in our converged approach. I mentioned the OAuth similarity as an analogue with an existing token based protocol for addressing restricted capabilities of a given token. The same type of analogue can be drawn between HADOOP-9533 and Kerberos. The cluster access token is similar to the TGT while the service access token is much like a service ticket. Service user scenarios are covered in an upcoming overview document for In-cluster trust establishment and service:service authentication that will be published on HADOOP-9533. I do agree that keeping the number of tokens that are required for a client/user-agent to keep track of is a good idea from the complexity and developer experience perspectives. We will however have to strike a balance between simplicity and a design that meets all of our goals. Enumerating the properties of each design at this point will provide little value. Instead, I propose that we articulate all of the relevant goals and usecases of the required design. Including attack vectors and scenarios that we must address with the ultimate design. Deployment scenarios of the Hadoop cluster will be important to consider as well. We will also need to take into account the capabilities of all the clients in the ecosystem for supporting it. With some of this groundwork done, we can get together at the Summit with an agenda item to reconcile those goals and usecases into a converged design. Discussions around the deployment scenarios will help drive how any layering will be done between Knox at the perimeter and HSSO and TAS inside the cluster. I do have some questions about the authorization approach as well. We can keep that as a separate discussion - probably better had on the HADOOP-9466 Jira? I will look into having a bridge available for the get together. Token based authentication and Single Sign On - Key: HADOOP-9392 URL: https://issues.apache.org/jira/browse/HADOOP-9392 Project: Hadoop Common Issue Type: New Feature Components: security Reporter: Kai Zheng Assignee: Kai Zheng Fix For: 3.0.0 Attachments: token-based-authn-plus-sso.pdf This is an umbrella entry for one of project Rhino’s topic, for details of project Rhino, please refer to https://github.com/intel-hadoop/project-rhino/. The major goal for this entry as described in project Rhino was “Core, HDFS, ZooKeeper, and HBase currently support Kerberos authentication at the RPC layer, via SASL. However this does not provide valuable attributes such as group membership, classification level, organizational identity, or support for user defined attributes. Hadoop components must interrogate external resources for discovering these attributes and at scale this is problematic. There is also no consistent delegation model. HDFS has a simple delegation capability, and only Oozie can take limited advantage of it. We will implement a common token based authentication framework to decouple internal user and service authentication from external mechanisms used to support it (like Kerberos)” We’d like to start our work from Hadoop-Common and try to provide common facilities by extending existing authentication framework which support: 1.Pluggable token provider interface 2.Pluggable token verification protocol and interface 3.Security mechanism to distribute secrets in cluster nodes 4.Delegation model of user authentication -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9307) BufferedFSInputStream.read returns wrong results after certain seeks
[ https://issues.apache.org/jira/browse/HADOOP-9307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13658338#comment-13658338 ] Hudson commented on HADOOP-9307: Integrated in Hadoop-Hdfs-trunk #1399 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1399/]) HADOOP-9307. BufferedFSInputStream.read returns wrong results after certain seeks. Contributed by Todd Lipcon. (Revision 1482377) Result = FAILURE todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1482377 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/BufferedFSInputStream.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestLocalFileSystem.java BufferedFSInputStream.read returns wrong results after certain seeks Key: HADOOP-9307 URL: https://issues.apache.org/jira/browse/HADOOP-9307 Project: Hadoop Common Issue Type: Bug Components: fs Affects Versions: 1.1.1, 2.0.2-alpha Reporter: Todd Lipcon Assignee: Todd Lipcon Fix For: 3.0.0, 2.0.5-beta Attachments: hadoop-9307-branch-1.txt, hadoop-9307.txt After certain sequences of seek/read, BufferedFSInputStream can silently return data from the wrong part of the file. Further description in first comment below. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9220) Unnecessary transition to standby in ActiveStandbyElector
[ https://issues.apache.org/jira/browse/HADOOP-9220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13658339#comment-13658339 ] Hudson commented on HADOOP-9220: Integrated in Hadoop-Hdfs-trunk #1399 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1399/]) HADOOP-9220. Unnecessary transition to standby in ActiveStandbyElector. Contributed by Tom White and Todd Lipcon. (Revision 1482401) Result = FAILURE todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1482401 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ha/ActiveStandbyElector.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ha/DummyHAService.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ha/TestZKFailoverController.java Unnecessary transition to standby in ActiveStandbyElector - Key: HADOOP-9220 URL: https://issues.apache.org/jira/browse/HADOOP-9220 Project: Hadoop Common Issue Type: Bug Components: ha Reporter: Tom White Assignee: Tom White Priority: Critical Fix For: 3.0.0, 2.0.5-beta Attachments: HADOOP-9220.patch, HADOOP-9220.patch, hadoop-9220.txt When performing a manual failover from one HA node to a second, under some circumstances the second node will transition from standby - active - standby - active. This is with automatic failover enabled, so there is a ZK cluster doing leader election. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9563) Fix incompatibility introduced by HADOOP-9523
[ https://issues.apache.org/jira/browse/HADOOP-9563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13658340#comment-13658340 ] Hudson commented on HADOOP-9563: Integrated in Hadoop-Hdfs-trunk #1399 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1399/]) HADOOP-9563. Fix incompatibility introduced by HADOOP-9523. Contributed by Tian Hong Wang. (Revision 1482709) Result = FAILURE suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1482709 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/PlatformName.java Fix incompatibility introduced by HADOOP-9523 - Key: HADOOP-9563 URL: https://issues.apache.org/jira/browse/HADOOP-9563 Project: Hadoop Common Issue Type: Bug Components: util Affects Versions: 3.0.0, 2.0.5-beta Reporter: Kihwal Lee Assignee: Tian Hong Wang Fix For: 2.0.5-beta Attachments: HADOOP-9563.patch HADOOP-9523 changed the output format of the platform name util, so hbase won't start unless platform name is specifically set. We could add a command line option to output in the new extended format and keep the old format when no option is given. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9560) metrics2#JvmMetrics should have max memory size of JVM
[ https://issues.apache.org/jira/browse/HADOOP-9560?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13658341#comment-13658341 ] Hudson commented on HADOOP-9560: Integrated in Hadoop-Hdfs-trunk #1399 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1399/]) HADOOP-9560. metrics2#JvmMetrics should have max memory size of JVM. Contributed by Tsuyoshi Ozawa. (Revision 1482372) Result = FAILURE suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1482372 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/source/JvmMetrics.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/source/JvmMetricsInfo.java metrics2#JvmMetrics should have max memory size of JVM -- Key: HADOOP-9560 URL: https://issues.apache.org/jira/browse/HADOOP-9560 Project: Hadoop Common Issue Type: Improvement Components: metrics Affects Versions: 3.0.0, 2.0.5-beta Reporter: Tsuyoshi OZAWA Assignee: Tsuyoshi OZAWA Priority: Minor Fix For: 2.0.5-beta Attachments: HADOOP-9560.1.patch metrics#JvmMetrics have the max memory size of JVM specified by -Xmx option. metrics2#JvmMetrics doesn't have it but it should also have max memory size of JVM, because it's useful for users. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9523) Provide a generic IBM java vendor flag in PlatformName.java to support non-Sun JREs
[ https://issues.apache.org/jira/browse/HADOOP-9523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13658344#comment-13658344 ] Hudson commented on HADOOP-9523: Integrated in Hadoop-Hdfs-trunk #1399 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1399/]) HADOOP-9563. Fix incompatibility introduced by HADOOP-9523. Contributed by Tian Hong Wang. (Revision 1482709) Result = FAILURE suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1482709 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/PlatformName.java Provide a generic IBM java vendor flag in PlatformName.java to support non-Sun JREs --- Key: HADOOP-9523 URL: https://issues.apache.org/jira/browse/HADOOP-9523 Project: Hadoop Common Issue Type: Improvement Affects Versions: 2.0.4-alpha Reporter: Tian Hong Wang Assignee: Tian Hong Wang Labels: patch Fix For: 2.0.5-beta Attachments: HADOOP-9523-fix.patch, HADOOP-9523.patch, HADOOP-9523-v1.patch, HADOOP-9523-v2.patch There are several different points between Sun jdk IBM jdk, so there is a need to provide a generic IBM java vendor flag. So enhance PlatformName.java to add Java vendor information. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9307) BufferedFSInputStream.read returns wrong results after certain seeks
[ https://issues.apache.org/jira/browse/HADOOP-9307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13658374#comment-13658374 ] Hudson commented on HADOOP-9307: Integrated in Hadoop-Mapreduce-trunk #1426 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1426/]) HADOOP-9307. BufferedFSInputStream.read returns wrong results after certain seeks. Contributed by Todd Lipcon. (Revision 1482377) Result = SUCCESS todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1482377 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/BufferedFSInputStream.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestLocalFileSystem.java BufferedFSInputStream.read returns wrong results after certain seeks Key: HADOOP-9307 URL: https://issues.apache.org/jira/browse/HADOOP-9307 Project: Hadoop Common Issue Type: Bug Components: fs Affects Versions: 1.1.1, 2.0.2-alpha Reporter: Todd Lipcon Assignee: Todd Lipcon Fix For: 3.0.0, 2.0.5-beta Attachments: hadoop-9307-branch-1.txt, hadoop-9307.txt After certain sequences of seek/read, BufferedFSInputStream can silently return data from the wrong part of the file. Further description in first comment below. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9220) Unnecessary transition to standby in ActiveStandbyElector
[ https://issues.apache.org/jira/browse/HADOOP-9220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13658375#comment-13658375 ] Hudson commented on HADOOP-9220: Integrated in Hadoop-Mapreduce-trunk #1426 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1426/]) HADOOP-9220. Unnecessary transition to standby in ActiveStandbyElector. Contributed by Tom White and Todd Lipcon. (Revision 1482401) Result = SUCCESS todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1482401 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ha/ActiveStandbyElector.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ha/DummyHAService.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ha/TestZKFailoverController.java Unnecessary transition to standby in ActiveStandbyElector - Key: HADOOP-9220 URL: https://issues.apache.org/jira/browse/HADOOP-9220 Project: Hadoop Common Issue Type: Bug Components: ha Reporter: Tom White Assignee: Tom White Priority: Critical Fix For: 3.0.0, 2.0.5-beta Attachments: HADOOP-9220.patch, HADOOP-9220.patch, hadoop-9220.txt When performing a manual failover from one HA node to a second, under some circumstances the second node will transition from standby - active - standby - active. This is with automatic failover enabled, so there is a ZK cluster doing leader election. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9563) Fix incompatibility introduced by HADOOP-9523
[ https://issues.apache.org/jira/browse/HADOOP-9563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13658376#comment-13658376 ] Hudson commented on HADOOP-9563: Integrated in Hadoop-Mapreduce-trunk #1426 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1426/]) HADOOP-9563. Fix incompatibility introduced by HADOOP-9523. Contributed by Tian Hong Wang. (Revision 1482709) Result = SUCCESS suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1482709 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/PlatformName.java Fix incompatibility introduced by HADOOP-9523 - Key: HADOOP-9563 URL: https://issues.apache.org/jira/browse/HADOOP-9563 Project: Hadoop Common Issue Type: Bug Components: util Affects Versions: 3.0.0, 2.0.5-beta Reporter: Kihwal Lee Assignee: Tian Hong Wang Fix For: 2.0.5-beta Attachments: HADOOP-9563.patch HADOOP-9523 changed the output format of the platform name util, so hbase won't start unless platform name is specifically set. We could add a command line option to output in the new extended format and keep the old format when no option is given. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9560) metrics2#JvmMetrics should have max memory size of JVM
[ https://issues.apache.org/jira/browse/HADOOP-9560?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13658377#comment-13658377 ] Hudson commented on HADOOP-9560: Integrated in Hadoop-Mapreduce-trunk #1426 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1426/]) HADOOP-9560. metrics2#JvmMetrics should have max memory size of JVM. Contributed by Tsuyoshi Ozawa. (Revision 1482372) Result = SUCCESS suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1482372 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/source/JvmMetrics.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/source/JvmMetricsInfo.java metrics2#JvmMetrics should have max memory size of JVM -- Key: HADOOP-9560 URL: https://issues.apache.org/jira/browse/HADOOP-9560 Project: Hadoop Common Issue Type: Improvement Components: metrics Affects Versions: 3.0.0, 2.0.5-beta Reporter: Tsuyoshi OZAWA Assignee: Tsuyoshi OZAWA Priority: Minor Fix For: 2.0.5-beta Attachments: HADOOP-9560.1.patch metrics#JvmMetrics have the max memory size of JVM specified by -Xmx option. metrics2#JvmMetrics doesn't have it but it should also have max memory size of JVM, because it's useful for users. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9523) Provide a generic IBM java vendor flag in PlatformName.java to support non-Sun JREs
[ https://issues.apache.org/jira/browse/HADOOP-9523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13658380#comment-13658380 ] Hudson commented on HADOOP-9523: Integrated in Hadoop-Mapreduce-trunk #1426 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1426/]) HADOOP-9563. Fix incompatibility introduced by HADOOP-9523. Contributed by Tian Hong Wang. (Revision 1482709) Result = SUCCESS suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1482709 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/PlatformName.java Provide a generic IBM java vendor flag in PlatformName.java to support non-Sun JREs --- Key: HADOOP-9523 URL: https://issues.apache.org/jira/browse/HADOOP-9523 Project: Hadoop Common Issue Type: Improvement Affects Versions: 2.0.4-alpha Reporter: Tian Hong Wang Assignee: Tian Hong Wang Labels: patch Fix For: 2.0.5-beta Attachments: HADOOP-9523-fix.patch, HADOOP-9523.patch, HADOOP-9523-v1.patch, HADOOP-9523-v2.patch There are several different points between Sun jdk IBM jdk, so there is a need to provide a generic IBM java vendor flag. So enhance PlatformName.java to add Java vendor information. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9523) Provide a generic IBM java vendor flag in PlatformName.java to support non-Sun JREs
[ https://issues.apache.org/jira/browse/HADOOP-9523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13658390#comment-13658390 ] Tian Hong Wang commented on HADOOP-9523: Suresh, talked with IBM hadoop dev team internally, we will push the patchset later after internal testing with other hadoop components successfully at first. So could you please revert the patch of HADOOP-9523 HADOOP-9563? Thanks. Provide a generic IBM java vendor flag in PlatformName.java to support non-Sun JREs --- Key: HADOOP-9523 URL: https://issues.apache.org/jira/browse/HADOOP-9523 Project: Hadoop Common Issue Type: Improvement Affects Versions: 2.0.4-alpha Reporter: Tian Hong Wang Assignee: Tian Hong Wang Labels: patch Fix For: 2.0.5-beta Attachments: HADOOP-9523-fix.patch, HADOOP-9523.patch, HADOOP-9523-v1.patch, HADOOP-9523-v2.patch There are several different points between Sun jdk IBM jdk, so there is a need to provide a generic IBM java vendor flag. So enhance PlatformName.java to add Java vendor information. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9563) Fix incompatibility introduced by HADOOP-9523
[ https://issues.apache.org/jira/browse/HADOOP-9563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13658391#comment-13658391 ] Tian Hong Wang commented on HADOOP-9563: Suresh, talked with IBM hadoop dev team internally, we will push the patchset later after internal testing with other hadoop components successfully at first. So could you please revert the patch of HADOOP-9523 HADOOP-9563? Thanks. Fix incompatibility introduced by HADOOP-9523 - Key: HADOOP-9563 URL: https://issues.apache.org/jira/browse/HADOOP-9563 Project: Hadoop Common Issue Type: Bug Components: util Affects Versions: 3.0.0, 2.0.5-beta Reporter: Kihwal Lee Assignee: Tian Hong Wang Fix For: 2.0.5-beta Attachments: HADOOP-9563.patch HADOOP-9523 changed the output format of the platform name util, so hbase won't start unless platform name is specifically set. We could add a command line option to output in the new extended format and keep the old format when no option is given. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9435) Minimal code change to support IBM jvm build dependency
[ https://issues.apache.org/jira/browse/HADOOP-9435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13658393#comment-13658393 ] Suresh Srinivas commented on HADOOP-9435: - +1 from me as well. I will commit this shortly. Minimal code change to support IBM jvm build dependency --- Key: HADOOP-9435 URL: https://issues.apache.org/jira/browse/HADOOP-9435 Project: Hadoop Common Issue Type: Bug Components: build Reporter: Tian Hong Wang Assignee: Tian Hong Wang Labels: patch Attachments: HADOOP-9435.patch, HADOOP-9435-v1.patch When native build hadoop-common-project with IBM java using command like: mvn package -Pnative it will exist the following errors. [exec] -- Configuring incomplete, errors occurred! [exec] JAVA_HOME=, JAVA_JVM_LIBRARY=/home/louis/ibm-java-i386-60/jre/lib/i386/classic/libjvm.so [exec] JAVA_INCLUDE_PATH=/home/louis/ibm-java-i386-60/include, JAVA_INCLUDE_PATH2=JAVA_INCLUDE_PATH2-NOTFOUND [exec] CMake Error at JNIFlags.cmake:113 (MESSAGE): [exec] Failed to find a viable JVM installation under JAVA_HOME. [exec] Call Stack (most recent call first): [exec] CMakeLists.txt:24 (include) The reason is that IBM java uses $JAVA_HOME/include/jniport.h instead of $JAVA_HOME/include/jni_md.h in non-IBM java. [exec] /usr/lib/jvm/java-1.6.0-ibm-1.6.0.12.0.x86_64/jre/lib/amd64/default/libjvm.so: undefined reference to `dlsym' [exec] /usr/lib/jvm/java-1.6.0-ibm-1.6.0.12.0.x86_64/jre/lib/amd64/default/libjvm.so: undefined reference to `dlerror' [exec] /usr/lib/jvm/java-1.6.0-ibm-1.6.0.12.0.x86_64/jre/lib/amd64/default/libjvm.so: undefined reference to `dladdr' [exec] /usr/lib/jvm/java-1.6.0-ibm-1.6.0.12.0.x86_64/jre/lib/amd64/default/libjvm.so: undefined reference to `dlopen' [exec] /usr/lib/jvm/java-1.6.0-ibm-1.6.0.12.0.x86_64/jre/lib/amd64/default/libjvm.so: undefined reference to `dlclose' [exec] collect2: ld returned 1 exit status [exec] make[2]: *** [test_libhdfs_ops] Error 1 [exec] make[1]: *** [CMakeFiles/test_libhdfs_ops.dir/all] Error 2 [exec] make: *** [all] Error The reason is libjvm.so need libdl when linking. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9563) Fix incompatibility introduced by HADOOP-9523
[ https://issues.apache.org/jira/browse/HADOOP-9563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13658405#comment-13658405 ] Suresh Srinivas commented on HADOOP-9563: - bq. Suresh, talked with IBM hadoop dev team internally, Both HADOOP-9523 and HADOOP-9563 look like reasonable changes to make. I am not sure I understand this and why the patch should be reverted in Apache? bq. we will push the patchset later after internal testing with other hadoop components successfully at first Please do create new jiras if you uncover more issues. Fix incompatibility introduced by HADOOP-9523 - Key: HADOOP-9563 URL: https://issues.apache.org/jira/browse/HADOOP-9563 Project: Hadoop Common Issue Type: Bug Components: util Affects Versions: 3.0.0, 2.0.5-beta Reporter: Kihwal Lee Assignee: Tian Hong Wang Fix For: 2.0.5-beta Attachments: HADOOP-9563.patch HADOOP-9523 changed the output format of the platform name util, so hbase won't start unless platform name is specifically set. We could add a command line option to output in the new extended format and keep the old format when no option is given. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9563) Fix incompatibility introduced by HADOOP-9523
[ https://issues.apache.org/jira/browse/HADOOP-9563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13658418#comment-13658418 ] Tian Hong Wang commented on HADOOP-9563: Suresh, IBM internal team decide to have a whole testing of this patch. I am asked to undo this patch. So Suresh, please revert patch of HADOOP-9523 and HADOOP-9563 for me now. We will resubmit it later after whole testing. Thanks. Fix incompatibility introduced by HADOOP-9523 - Key: HADOOP-9563 URL: https://issues.apache.org/jira/browse/HADOOP-9563 Project: Hadoop Common Issue Type: Bug Components: util Affects Versions: 3.0.0, 2.0.5-beta Reporter: Kihwal Lee Assignee: Tian Hong Wang Fix For: 2.0.5-beta Attachments: HADOOP-9563.patch HADOOP-9523 changed the output format of the platform name util, so hbase won't start unless platform name is specifically set. We could add a command line option to output in the new extended format and keep the old format when no option is given. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9563) Fix incompatibility introduced by HADOOP-9523
[ https://issues.apache.org/jira/browse/HADOOP-9563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13658528#comment-13658528 ] Suresh Srinivas commented on HADOOP-9563: - This is a strange request. As I said earlier, these are trivial changes to add a constant to be reused. I do not understand how this could affect any testing. IBM internal team decide to have a whole testing of this patch. I am asked to undo this patch. is not a good enough reason for me to revert this patch. Please explain the technical reasons why this causes problems and hence should be reverted. bq. We will resubmit it later after whole testing. Please make updated patches available in a separate jira when your testing completes. Feel free to change the code introduced in these two jiras in the new jiras, if they are issues. Fix incompatibility introduced by HADOOP-9523 - Key: HADOOP-9563 URL: https://issues.apache.org/jira/browse/HADOOP-9563 Project: Hadoop Common Issue Type: Bug Components: util Affects Versions: 3.0.0, 2.0.5-beta Reporter: Kihwal Lee Assignee: Tian Hong Wang Fix For: 2.0.5-beta Attachments: HADOOP-9563.patch HADOOP-9523 changed the output format of the platform name util, so hbase won't start unless platform name is specifically set. We could add a command line option to output in the new extended format and keep the old format when no option is given. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9439) JniBasedUnixGroupsMapping: fix some crash bugs
[
https://issues.apache.org/jira/browse/HADOOP-9439?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Colin Patrick McCabe updated HADOOP-9439:
-
Attachment: HADOOP-9439.003.patch
bq. I don't get why empty is a parameter here... aside from it being a
premature optimization (is allocating an empty array that bad?), it seems like
the JNI code could just grab this field itself and return a reference,
rather than taking it as a parameter.
This optimization was in the existing code, so I kept it in. But actually,
looking at it again, I think you're right. We might as well take it out.
bq. Style nit: extra space here before getGroupsForUser
ok.
bq. Can we use strerror here on the return, assuming it's probably a standard
errnum?
good idea.
+ret = hadoop_group_info_fetch(ginfo, uinfo-gids[i]);
+if (!ret) {
bq. Here if the group info lookup fails for a particular gid, you end up
swallowing the error without any warnings, etc. Maybe instead we should just
return the numeric gid as a group? This is what the 'groups' shell command
does:
The 'groups' shell command returns nonzero if it can't look up a group (at
least according to the internet; I didn't want to mess up my groups settings to
test). That would cause ShellBasedUnixGroupsMapping to get got exception
trying to get groups for user. So as far as I know, there is no scenario
currently where we treat the numeric gid as a group name.
We could be bug-compatible with ShellBasedUnixGroupsMapping, and refuse to look
up *any* groups if one fails. However, that doesn't seem like a good idea.
Instead, I added some code that fires off a log4j message in that scenario, and
continues with the other groups.
bq. We may have to actually share this lock and configuration with the lock
used in NativeIO.c.
yeah, let's do that.
bq. Additionally, when I did that patch, I found it simpler to continue to use
the reentrant functions wrapped with a lock – less repeated code, etc.
ok.
JniBasedUnixGroupsMapping: fix some crash bugs
--
Key: HADOOP-9439
URL: https://issues.apache.org/jira/browse/HADOOP-9439
Project: Hadoop Common
Issue Type: Bug
Components: native
Affects Versions: 2.0.4-alpha
Reporter: Colin Patrick McCabe
Assignee: Colin Patrick McCabe
Priority: Minor
Attachments: HADOOP-9439.001.patch, HADOOP-9439.003.patch,
HDFS-4640.002.patch
JniBasedUnixGroupsMapping has some issues.
* sometimes on error paths variables are freed prior to being initialized
* re-allocate buffers less frequently (can reuse the same buffer for multiple
calls to getgrnam)
* allow non-reentrant functions to be used, to work around client bugs
* don't throw IOException from JNI functions if the JNI functions do not
declare this checked exception.
* don't bail out if only one group name among all the ones associated with a
user can't be looked up.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9439) JniBasedUnixGroupsMapping: fix some crash bugs
[
https://issues.apache.org/jira/browse/HADOOP-9439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13658686#comment-13658686
]
Hadoop QA commented on HADOOP-9439:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12583352/HADOOP-9439.003.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/2544//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/2544//console
This message is automatically generated.
JniBasedUnixGroupsMapping: fix some crash bugs
--
Key: HADOOP-9439
URL: https://issues.apache.org/jira/browse/HADOOP-9439
Project: Hadoop Common
Issue Type: Bug
Components: native
Affects Versions: 2.0.4-alpha
Reporter: Colin Patrick McCabe
Assignee: Colin Patrick McCabe
Priority: Minor
Attachments: HADOOP-9439.001.patch, HADOOP-9439.003.patch,
HDFS-4640.002.patch
JniBasedUnixGroupsMapping has some issues.
* sometimes on error paths variables are freed prior to being initialized
* re-allocate buffers less frequently (can reuse the same buffer for multiple
calls to getgrnam)
* allow non-reentrant functions to be used, to work around client bugs
* don't throw IOException from JNI functions if the JNI functions do not
declare this checked exception.
* don't bail out if only one group name among all the ones associated with a
user can't be looked up.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9562) Create REST interface for HDFS health data
[
https://issues.apache.org/jira/browse/HADOOP-9562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13658733#comment-13658733
]
Steve Loughran commented on HADOOP-9562:
of course, if it is just GET ops, then making dfshealth a valid XHTML page with
id's for every span/div containing data lets you grab this stuff by
GET-parse-xpath operations, such as {{//html/body/div[id=safemode][0]}}.
You wouldn't need any extra pages, just some tests to verify that the served up
content would parse and the relevant IDs resolve
Create REST interface for HDFS health data
--
Key: HADOOP-9562
URL: https://issues.apache.org/jira/browse/HADOOP-9562
Project: Hadoop Common
Issue Type: New Feature
Components: fs
Affects Versions: 2.0.4-alpha
Reporter: Trevor Lorimer
Priority: Minor
The HDFS health screen (dfshealth.jsp) displays basic Version, Security and
Health information concerning the NameNode, currently this information is
accessible from classes in the org.apache.hadoop,hdfs.server.namenode package
and cannot be accessed outside the NameNode. This becomes prevalent if the
data is required to be displayed using a new user interface.
The proposal is to create a REST interface to expose all the information
displayed on dfshealth.jsp using GET methods. Wrapper classes will be created
to serve the data to the REST root resource within the hadoop-hdfs project.
This will enable the HDFS health screen information to be accessed remotely.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9421) Generalize SASL Support with Protocol Buffer
[ https://issues.apache.org/jira/browse/HADOOP-9421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13658842#comment-13658842 ] Sanjay Radia commented on HADOOP-9421: -- bq. Looks like we need to limit the scope of this jira, which is a subtask of the RPC cleanup Luke, part of the problem is that the title of the Jira suggests that the scope is wide. Please change the title to Convert SASL to use ProtoBuf and add lengths for non-blocking processing. I agree that we should limit the scope. I would have been happy to just add the length to the reply. Generalize SASL Support with Protocol Buffer Key: HADOOP-9421 URL: https://issues.apache.org/jira/browse/HADOOP-9421 Project: Hadoop Common Issue Type: Sub-task Affects Versions: 2.0.3-alpha Reporter: Sanjay Radia Assignee: Junping Du Attachments: HADOOP-9421.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9517) Define Hadoop Compatibility
[ https://issues.apache.org/jira/browse/HADOOP-9517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13658848#comment-13658848 ] Tsz Wo (Nicholas), SZE commented on HADOOP-9517: Hi Karthik, Generally, Hadoop welcomes new comers. We like to help them to understand the code and contribute to the project. New contributors are better starting with some easy issues or issues with less impact. I think this JIRA Define Hadoop Compatibility is a very important issue and it is more suitable for some committers who has a deep understanding of Hadoop to work on it. Would you agree? Please forgive me if I have overlooked anything. Define Hadoop Compatibility --- Key: HADOOP-9517 URL: https://issues.apache.org/jira/browse/HADOOP-9517 Project: Hadoop Common Issue Type: Bug Components: documentation Reporter: Arun C Murthy Assignee: Karthik Kambatla Attachments: hadoop-9517.patch, hadoop-9517.patch, hadoop-9517.patch, hadoop-9517.patch As we get ready to call hadoop-2 stable we need to better define 'Hadoop Compatibility'. http://wiki.apache.org/hadoop/Compatibility is a start, let's document requirements clearly and completely. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9421) Convert SASL to use ProtoBuf and add lengths for non-blocking processing
[ https://issues.apache.org/jira/browse/HADOOP-9421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du updated HADOOP-9421: --- Summary: Convert SASL to use ProtoBuf and add lengths for non-blocking processing (was: Generalize SASL Support with Protocol Buffer) Convert SASL to use ProtoBuf and add lengths for non-blocking processing Key: HADOOP-9421 URL: https://issues.apache.org/jira/browse/HADOOP-9421 Project: Hadoop Common Issue Type: Sub-task Affects Versions: 2.0.3-alpha Reporter: Sanjay Radia Assignee: Junping Du Attachments: HADOOP-9421.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9421) Convert SASL to use ProtoBuf and add lengths for non-blocking processing
[ https://issues.apache.org/jira/browse/HADOOP-9421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13658944#comment-13658944 ] Junping Du commented on HADOOP-9421: Update the title now. Will deliver a updated patch with limited scope today or tomorrow. Convert SASL to use ProtoBuf and add lengths for non-blocking processing Key: HADOOP-9421 URL: https://issues.apache.org/jira/browse/HADOOP-9421 Project: Hadoop Common Issue Type: Sub-task Affects Versions: 2.0.3-alpha Reporter: Sanjay Radia Assignee: Junping Du Attachments: HADOOP-9421.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9517) Define Hadoop Compatibility
[ https://issues.apache.org/jira/browse/HADOOP-9517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13658951#comment-13658951 ] Eli Collins commented on HADOOP-9517: - Hey Nicholas, per [the thread on common-dev@|http://s.apache.org/VE1] this is something Karthik and I worked on together. Karthik, is an active contributor to MR and YARN (why you haven't bumped into him yet), and, in my opinion, has sufficient experience to contribute on important issues. Newcomers are capable of working on important and impactful things (just consider the people you've been working on with on HDFS snapshots). We would love your constructive feedback on the draft posted if you have any. Define Hadoop Compatibility --- Key: HADOOP-9517 URL: https://issues.apache.org/jira/browse/HADOOP-9517 Project: Hadoop Common Issue Type: Bug Components: documentation Reporter: Arun C Murthy Assignee: Karthik Kambatla Attachments: hadoop-9517.patch, hadoop-9517.patch, hadoop-9517.patch, hadoop-9517.patch As we get ready to call hadoop-2 stable we need to better define 'Hadoop Compatibility'. http://wiki.apache.org/hadoop/Compatibility is a start, let's document requirements clearly and completely. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (HADOOP-9517) Define Hadoop Compatibility
[ https://issues.apache.org/jira/browse/HADOOP-9517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13658951#comment-13658951 ] Eli Collins edited comment on HADOOP-9517 at 5/15/13 10:48 PM: --- Hey Nicholas, per [the thread on common-dev@|http://s.apache.org/VE1] this is something Karthik and I worked on together. Karthik, is an active contributor to MR and YARN (why you haven't bumped into him yet), and, in my opinion, has sufficient experience to contribute on important issues. Newcomers are capable of working on important and impactful things (just consider the people you've been working with on HDFS snapshots). We would love your constructive feedback on the draft posted if you have any. was (Author: eli): Hey Nicholas, per [the thread on common-dev@|http://s.apache.org/VE1] this is something Karthik and I worked on together. Karthik, is an active contributor to MR and YARN (why you haven't bumped into him yet), and, in my opinion, has sufficient experience to contribute on important issues. Newcomers are capable of working on important and impactful things (just consider the people you've been working on with on HDFS snapshots). We would love your constructive feedback on the draft posted if you have any. Define Hadoop Compatibility --- Key: HADOOP-9517 URL: https://issues.apache.org/jira/browse/HADOOP-9517 Project: Hadoop Common Issue Type: Bug Components: documentation Reporter: Arun C Murthy Assignee: Karthik Kambatla Attachments: hadoop-9517.patch, hadoop-9517.patch, hadoop-9517.patch, hadoop-9517.patch As we get ready to call hadoop-2 stable we need to better define 'Hadoop Compatibility'. http://wiki.apache.org/hadoop/Compatibility is a start, let's document requirements clearly and completely. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-9565) Add a Blobstore interface to add to blobstore FileSystems
Steve Loughran created HADOOP-9565:
--
Summary: Add a Blobstore interface to add to blobstore FileSystems
Key: HADOOP-9565
URL: https://issues.apache.org/jira/browse/HADOOP-9565
Project: Hadoop Common
Issue Type: Sub-task
Components: fs
Affects Versions: 2.0.4-alpha
Reporter: Steve Loughran
Priority: Minor
We can make the fact that some {{FileSystem}} implementations are really
blobstores, with different atomicity and consistency guarantees, by adding a
{{Blobstore}} interface to add to them.
This could also be a place to add a {{Copy(Path,Path)}} method, assuming that
all blobstores implement at server-side copy operation as a substitute for
rename.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9421) Convert SASL to use ProtoBuf and add lengths for non-blocking processing
[ https://issues.apache.org/jira/browse/HADOOP-9421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13658991#comment-13658991 ] Sanjay Radia commented on HADOOP-9421: -- Junping, can you add code to ensure that an old sasl request will get a response along the lines of not right version protocol. For example, the rpc layer responds with the old protocol format error message. This helps when folks connect old client to new servers. Convert SASL to use ProtoBuf and add lengths for non-blocking processing Key: HADOOP-9421 URL: https://issues.apache.org/jira/browse/HADOOP-9421 Project: Hadoop Common Issue Type: Sub-task Affects Versions: 2.0.3-alpha Reporter: Sanjay Radia Assignee: Junping Du Attachments: HADOOP-9421.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9421) Convert SASL to use ProtoBuf and add lengths for non-blocking processing
[ https://issues.apache.org/jira/browse/HADOOP-9421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13659041#comment-13659041 ] Luke Lu commented on HADOOP-9421: - [~drankye]: converting the wireformat to protobuf for both SASL request and response will address all your concerns as protobuf allows adding new fields without breaking backward compatibility. [~sanjay.radia]: since the old sasl request doesn't exist for RPC v9 yet, I think we can get away with not handling SASL version mismatch until the next sasl version when. :) Convert SASL to use ProtoBuf and add lengths for non-blocking processing Key: HADOOP-9421 URL: https://issues.apache.org/jira/browse/HADOOP-9421 Project: Hadoop Common Issue Type: Sub-task Affects Versions: 2.0.3-alpha Reporter: Sanjay Radia Assignee: Junping Du Attachments: HADOOP-9421.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9517) Define Hadoop Compatibility
[ https://issues.apache.org/jira/browse/HADOOP-9517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13659064#comment-13659064 ] Tsz Wo (Nicholas), SZE commented on HADOOP-9517: ... Karthik, is an active contributor to MR and YARN (why you haven't bumped into him yet), ... I do realize that Karthik have worked on MR and YARN but not much in HDFS. That's why I think that he may not be the best person to define HDFS compatibility, which is very important to HDFS. Would you agree? Define Hadoop Compatibility --- Key: HADOOP-9517 URL: https://issues.apache.org/jira/browse/HADOOP-9517 Project: Hadoop Common Issue Type: Bug Components: documentation Reporter: Arun C Murthy Assignee: Karthik Kambatla Attachments: hadoop-9517.patch, hadoop-9517.patch, hadoop-9517.patch, hadoop-9517.patch As we get ready to call hadoop-2 stable we need to better define 'Hadoop Compatibility'. http://wiki.apache.org/hadoop/Compatibility is a start, let's document requirements clearly and completely. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9517) Define Hadoop Compatibility
[ https://issues.apache.org/jira/browse/HADOOP-9517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13659076#comment-13659076 ] Karthik Kambatla commented on HADOOP-9517: -- Hey [~szetszwo], thanks for chipping in. Your valuable feedback and insight will surely help ironing out the kinks in the proposal. I am eager to hear more feedback on the patch, would gladly incorporate additions/changes. Meanwhile, [~eli] and I are working on a strawman policy for items we currently don't have policies for (Currently, there is NO policy) and will post it as soon as we get to a reasonable shape. Define Hadoop Compatibility --- Key: HADOOP-9517 URL: https://issues.apache.org/jira/browse/HADOOP-9517 Project: Hadoop Common Issue Type: Bug Components: documentation Reporter: Arun C Murthy Assignee: Karthik Kambatla Attachments: hadoop-9517.patch, hadoop-9517.patch, hadoop-9517.patch, hadoop-9517.patch As we get ready to call hadoop-2 stable we need to better define 'Hadoop Compatibility'. http://wiki.apache.org/hadoop/Compatibility is a start, let's document requirements clearly and completely. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9517) Define Hadoop Compatibility
[ https://issues.apache.org/jira/browse/HADOOP-9517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13659084#comment-13659084 ] Eli Collins commented on HADOOP-9517: - Nicholas, see my point about Karthik and I working on this *together*. Working on something together means that multiple perspectives Karthik (mostly focuses on YARN/MR) and myself (mostly focused on Common/HDFS) were incorporated. The point of this jira is not to pick one person to define compatibility, most of the stuff covered in this document are rules and guidelines that have existed in the project for years, this is just writing them up. Do have an objection to anything particular in the document? Define Hadoop Compatibility --- Key: HADOOP-9517 URL: https://issues.apache.org/jira/browse/HADOOP-9517 Project: Hadoop Common Issue Type: Bug Components: documentation Reporter: Arun C Murthy Assignee: Karthik Kambatla Attachments: hadoop-9517.patch, hadoop-9517.patch, hadoop-9517.patch, hadoop-9517.patch As we get ready to call hadoop-2 stable we need to better define 'Hadoop Compatibility'. http://wiki.apache.org/hadoop/Compatibility is a start, let's document requirements clearly and completely. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9140) Cleanup rpc PB protos
[ https://issues.apache.org/jira/browse/HADOOP-9140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo (Nicholas), SZE updated HADOOP-9140: --- Component/s: ipc Fix Version/s: 2.0.5-beta Merged to branch-2. Cleanup rpc PB protos - Key: HADOOP-9140 URL: https://issues.apache.org/jira/browse/HADOOP-9140 Project: Hadoop Common Issue Type: Sub-task Components: ipc Reporter: Sanjay Radia Assignee: Sanjay Radia Fix For: 2.0.5-beta Attachments: Hadoop-9140-2.patch, Hadoop-9140-3.patch, Hadoop-9140-4.patch, Hadoop-9140.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9218) Document the Rpc-wrappers used internally
[ https://issues.apache.org/jira/browse/HADOOP-9218?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sanjay Radia updated HADOOP-9218: - Component/s: ipc Fix Version/s: 2.0.5-beta Merged into branch2 Document the Rpc-wrappers used internally - Key: HADOOP-9218 URL: https://issues.apache.org/jira/browse/HADOOP-9218 Project: Hadoop Common Issue Type: Sub-task Components: ipc Reporter: Sanjay Radia Assignee: Sanjay Radia Fix For: 2.0.5-beta Attachments: hadoop-9218.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9163) The rpc msg in ProtobufRpcEngine.proto should be moved out to avoid an extra copy
[ https://issues.apache.org/jira/browse/HADOOP-9163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sanjay Radia updated HADOOP-9163: - Component/s: ipc Fix Version/s: 2.0.5-beta Hadoop Flags: Incompatible change Merged into branch2 The rpc msg in ProtobufRpcEngine.proto should be moved out to avoid an extra copy -- Key: HADOOP-9163 URL: https://issues.apache.org/jira/browse/HADOOP-9163 Project: Hadoop Common Issue Type: Sub-task Components: ipc Reporter: Sanjay Radia Assignee: Sanjay Radia Fix For: 2.0.5-beta Attachments: Hadoop-9163-2.patch, Hadoop-9163-3.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9151) Include RPC error info in RpcResponseHeader instead of sending it separately
[ https://issues.apache.org/jira/browse/HADOOP-9151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sanjay Radia updated HADOOP-9151: - Component/s: ipc Description: (was: ) Fix Version/s: 2.0.5-beta merged into branch2 Include RPC error info in RpcResponseHeader instead of sending it separately Key: HADOOP-9151 URL: https://issues.apache.org/jira/browse/HADOOP-9151 Project: Hadoop Common Issue Type: Sub-task Components: ipc Reporter: Sanjay Radia Assignee: Sanjay Radia Fix For: 2.0.5-beta Attachments: HADOOP-9151-2.patch, HADOOP-9151-3.patch, HADOOP-9151-4.patch, HADOOP-9151-5.patch, HADOOP-9151.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9380) Add totalLength to rpc response
[ https://issues.apache.org/jira/browse/HADOOP-9380?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sanjay Radia updated HADOOP-9380: - Component/s: ipc Fix Version/s: 2.0.5-beta merged into branch2 Add totalLength to rpc response --- Key: HADOOP-9380 URL: https://issues.apache.org/jira/browse/HADOOP-9380 Project: Hadoop Common Issue Type: Sub-task Components: ipc Reporter: Sanjay Radia Assignee: Sanjay Radia Fix For: 2.0.5-beta Attachments: HADOOP-9380-2.patch, HADOOP-9380-4.patch, HADOOP-9380-5.patch, HADOOP-9380-6.patch, HADOOP-9380.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9425) Add error codes to rpc-response
[ https://issues.apache.org/jira/browse/HADOOP-9425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sanjay Radia updated HADOOP-9425: - Component/s: ipc Fix Version/s: 2.0.5-beta Merged into branch2 Add error codes to rpc-response --- Key: HADOOP-9425 URL: https://issues.apache.org/jira/browse/HADOOP-9425 Project: Hadoop Common Issue Type: Sub-task Components: ipc Reporter: Sanjay Radia Assignee: Sanjay Radia Fix For: 2.0.5-beta Attachments: HADOOP-9425-1.patch, HADOOP-9425-2.patch, HADOOP-9425-3.patch, HADOOP-9425-4.patch, HADOOP-9425-5.patch, HADOOP-9425-6.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HADOOP-9564) DFSClient$DFSOutputStream.closeInternal locks up waiting for namenode.complete
[
https://issues.apache.org/jira/browse/HADOOP-9564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth reassigned HADOOP-9564:
-
Assignee: Chris Nauroth
DFSClient$DFSOutputStream.closeInternal locks up waiting for namenode.complete
--
Key: HADOOP-9564
URL: https://issues.apache.org/jira/browse/HADOOP-9564
Project: Hadoop Common
Issue Type: Bug
Components: fs
Reporter: Jin Feng
Assignee: Chris Nauroth
Priority: Minor
Hi,
Our component uses FileSystem.copyFromLocalFile to copy a local file to HDFS
cluster. It's working fine in production environment. Its integration tests
used to run fine on our dev's local Mac laptop until recently (exact point of
time unknown) our tests started to freeze up very frequently with this stack:
{code}
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for 0x000152f41378 (a
java.util.concurrent.FutureTask$Sync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:994)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1303)
at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:248)
at java.util.concurrent.FutureTask.get(FutureTask.java:111)
at org.apache.hadoop.ipc.Client$Connection.sendParam(Client.java:790)
- locked 0x00014f568720 (a java.lang.Object)
at org.apache.hadoop.ipc.Client.call(Client.java:1080)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:226)
at $Proxy37.complete(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
at $Proxy37.complete(Unknown Source)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.closeInternal(DFSClient.java:3566)
- locked 0x000152f3f658 (a
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.close(DFSClient.java:3481)
at
org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:61)
at
org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:86)
at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:59)
at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:89)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:224)
at
org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:1295)
{code}
our version is 0.20.2.cdh3u2-t1.
In the test suite, we use org.apache.hadoop.hdfs.MiniDFSCluster. I've
searched around couldn't find anything resembles this symptom, any helps are
really appreciated!
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9564) DFSClient$DFSOutputStream.closeInternal locks up waiting for namenode.complete
[
https://issues.apache.org/jira/browse/HADOOP-9564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth updated HADOOP-9564:
--
Assignee: (was: Chris Nauroth)
DFSClient$DFSOutputStream.closeInternal locks up waiting for namenode.complete
--
Key: HADOOP-9564
URL: https://issues.apache.org/jira/browse/HADOOP-9564
Project: Hadoop Common
Issue Type: Bug
Components: fs
Reporter: Jin Feng
Priority: Minor
Hi,
Our component uses FileSystem.copyFromLocalFile to copy a local file to HDFS
cluster. It's working fine in production environment. Its integration tests
used to run fine on our dev's local Mac laptop until recently (exact point of
time unknown) our tests started to freeze up very frequently with this stack:
{code}
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for 0x000152f41378 (a
java.util.concurrent.FutureTask$Sync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:994)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1303)
at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:248)
at java.util.concurrent.FutureTask.get(FutureTask.java:111)
at org.apache.hadoop.ipc.Client$Connection.sendParam(Client.java:790)
- locked 0x00014f568720 (a java.lang.Object)
at org.apache.hadoop.ipc.Client.call(Client.java:1080)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:226)
at $Proxy37.complete(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
at $Proxy37.complete(Unknown Source)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.closeInternal(DFSClient.java:3566)
- locked 0x000152f3f658 (a
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.close(DFSClient.java:3481)
at
org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:61)
at
org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:86)
at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:59)
at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:89)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:224)
at
org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:1295)
{code}
our version is 0.20.2.cdh3u2-t1.
In the test suite, we use org.apache.hadoop.hdfs.MiniDFSCluster. I've
searched around couldn't find anything resembles this symptom, any helps are
really appreciated!
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9392) Token based authentication and Single Sign On
[ https://issues.apache.org/jira/browse/HADOOP-9392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13659243#comment-13659243 ] Kai Zheng commented on HADOOP-9392: --- Thomas - Identity Token represents the identity and will be used to access all Hadoop services. However, any service can have its own TAS deployed to authenticate the identity token. Of course in most cases they can share the same TAS instance. The token sure has an expiration time and user can also initiate token expire command to expire it earlier. The token can be cached and the cache is to be expired at a configurable internal. Token based authentication and Single Sign On - Key: HADOOP-9392 URL: https://issues.apache.org/jira/browse/HADOOP-9392 Project: Hadoop Common Issue Type: New Feature Components: security Reporter: Kai Zheng Assignee: Kai Zheng Fix For: 3.0.0 Attachments: token-based-authn-plus-sso.pdf This is an umbrella entry for one of project Rhino’s topic, for details of project Rhino, please refer to https://github.com/intel-hadoop/project-rhino/. The major goal for this entry as described in project Rhino was “Core, HDFS, ZooKeeper, and HBase currently support Kerberos authentication at the RPC layer, via SASL. However this does not provide valuable attributes such as group membership, classification level, organizational identity, or support for user defined attributes. Hadoop components must interrogate external resources for discovering these attributes and at scale this is problematic. There is also no consistent delegation model. HDFS has a simple delegation capability, and only Oozie can take limited advantage of it. We will implement a common token based authentication framework to decouple internal user and service authentication from external mechanisms used to support it (like Kerberos)” We’d like to start our work from Hadoop-Common and try to provide common facilities by extending existing authentication framework which support: 1.Pluggable token provider interface 2.Pluggable token verification protocol and interface 3.Security mechanism to distribute secrets in cluster nodes 4.Delegation model of user authentication -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
