date:20130514

[
https://issues.apache.org/jira/browse/HADOOP-9459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Todd Lipcon updated HADOOP-9459:

Resolution: Fixed
Fix Version/s: 2.0.5-beta
3.0.0
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)

Committed to trunk and branch-2. Thanks for tracking down this tricky bug, Vinay

ActiveStandbyElector can join election even before Service HEALTHY, and
results in null data at ActiveBreadCrumb

Key: HADOOP-9459
URL: https://issues.apache.org/jira/browse/HADOOP-9459
Project: Hadoop Common
Issue Type: Bug
Components: ha
Affects Versions: 2.0.2-alpha
Reporter: Vinay
Assignee: Vinay
Priority: Critical
Fix For: 3.0.0, 2.0.5-beta

Attachments: HDFS-4463.patch, hdfs-4463.txt

ActiveStandbyElector can store null at ActiveBreadCrumb in the below race
condition. At further all failovers will fail resulting NPE.
1. ZKFC restarted.
2. due to machine busy, first zk connection is expired even before the health
monitoring returned the status.
3. On re-establishment transitionToActive will be called, at this time
appData will be null,
4. So now ActiveBreadCrumb will have null.
5. After this any failovers will fail throwing
{noformat}java.lang.NullPointerException
at
org.apache.hadoop.util.StringUtils.byteToHexString(StringUtils.java:171)
at
org.apache.hadoop.ha.ActiveStandbyElector.fenceOldActive(ActiveStandbyElector.java:892)
at
org.apache.hadoop.ha.ActiveStandbyElector.becomeActive(ActiveStandbyElector.java:797)
at
org.apache.hadoop.ha.ActiveStandbyElector.processResult(ActiveStandbyElector.java:475)
at
org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:545)
at
org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:497){noformat}
Should not join the election before service is HEALTHY

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-9307) BufferedFSInputStream.read returns wrong results after certain seeks


[ 
https://issues.apache.org/jira/browse/HADOOP-9307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13656861#comment-13656861
 ] 

Todd Lipcon commented on HADOOP-9307:
-

Hey Steve. I agree that improving the general cross-filesystem testing is a 
worthy goal. But, this is a simple bug in an existing implementation, and the 
patch adds a specific unit test. Given that this breaks HBase running on the 
local filesystem, I don't think it makes sense to block fixing it on a much 
bigger project like standardizing tests.

 BufferedFSInputStream.read returns wrong results after certain seeks
 

 Key: HADOOP-9307
 URL: https://issues.apache.org/jira/browse/HADOOP-9307
 Project: Hadoop Common
  Issue Type: Bug
  Components: fs
Affects Versions: 1.1.1, 2.0.2-alpha
Reporter: Todd Lipcon
Assignee: Todd Lipcon
 Attachments: hadoop-9307.txt


 After certain sequences of seek/read, BufferedFSInputStream can silently 
 return data from the wrong part of the file. Further description in first 
 comment below.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-9307) BufferedFSInputStream.read returns wrong results after certain seeks

2013-05-14 Thread Harsh J (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-9307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13656872#comment-13656872
 ] 

Harsh J commented on HADOOP-9307:
-

+1 - The change and the added regression test looks good. I tested it without 
the fix as well. Nice find Todd!

 BufferedFSInputStream.read returns wrong results after certain seeks
 

 Key: HADOOP-9307
 URL: https://issues.apache.org/jira/browse/HADOOP-9307
 Project: Hadoop Common
  Issue Type: Bug
  Components: fs
Affects Versions: 1.1.1, 2.0.2-alpha
Reporter: Todd Lipcon
Assignee: Todd Lipcon
 Attachments: hadoop-9307.txt


 After certain sequences of seek/read, BufferedFSInputStream can silently 
 return data from the wrong part of the file. Further description in first 
 comment below.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HADOOP-9435) Minimal code change to support IBM jvm build dependency


 [ 
https://issues.apache.org/jira/browse/HADOOP-9435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tian Hong Wang updated HADOOP-9435:
---

Summary: Minimal code change to support IBM jvm build dependency  (was: 
Native build hadoop-common-project fails on $JAVA_HOME/include/jni_md.h using 
ibm java)

 Minimal code change to support IBM jvm build dependency
 ---

 Key: HADOOP-9435
 URL: https://issues.apache.org/jira/browse/HADOOP-9435
 Project: Hadoop Common
  Issue Type: Bug
  Components: build
Affects Versions: 2.0.3-alpha
Reporter: Tian Hong Wang
  Labels: patch
 Fix For: 2.0.3-alpha

 Attachments: HADOOP-9435.patch


 When native build hadoop-common-project with IBM java using command like: 
 mvn package -Pnative
 it will exist the following errors.
  [exec] -- Configuring incomplete, errors occurred!
  [exec] JAVA_HOME=, 
 JAVA_JVM_LIBRARY=/home/louis/ibm-java-i386-60/jre/lib/i386/classic/libjvm.so
  [exec] JAVA_INCLUDE_PATH=/home/louis/ibm-java-i386-60/include, 
 JAVA_INCLUDE_PATH2=JAVA_INCLUDE_PATH2-NOTFOUND
  [exec] CMake Error at JNIFlags.cmake:113 (MESSAGE):
  [exec]   Failed to find a viable JVM installation under JAVA_HOME.
  [exec] Call Stack (most recent call first):
  [exec]   CMakeLists.txt:24 (include)
 The reason is that IBM java uses $JAVA_HOME/include/jniport.h instead of 
 $JAVA_HOME/include/jni_md.h in Oracle java.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HADOOP-9435) Minimal code change to support IBM jvm build dependency


 [ 
https://issues.apache.org/jira/browse/HADOOP-9435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tian Hong Wang updated HADOOP-9435:
---

Description: 
When native build hadoop-common-project with IBM java using command like: 
mvn package -Pnative
it will exist the following errors.
 [exec] -- Configuring incomplete, errors occurred!
 [exec] JAVA_HOME=, 
JAVA_JVM_LIBRARY=/home/louis/ibm-java-i386-60/jre/lib/i386/classic/libjvm.so
 [exec] JAVA_INCLUDE_PATH=/home/louis/ibm-java-i386-60/include, 
JAVA_INCLUDE_PATH2=JAVA_INCLUDE_PATH2-NOTFOUND
 [exec] CMake Error at JNIFlags.cmake:113 (MESSAGE):
 [exec]   Failed to find a viable JVM installation under JAVA_HOME.
 [exec] Call Stack (most recent call first):
 [exec]   CMakeLists.txt:24 (include)

The reason is that IBM java uses $JAVA_HOME/include/jniport.h instead of 
$JAVA_HOME/include/jni_md.h in non-IBM java.

[exec] 
/usr/lib/jvm/java-1.6.0-ibm-1.6.0.12.0.x86_64/jre/lib/amd64/default/libjvm.so: 
undefined reference to `dlsym'
 [exec] 
/usr/lib/jvm/java-1.6.0-ibm-1.6.0.12.0.x86_64/jre/lib/amd64/default/libjvm.so: 
undefined reference to `dlerror'
 [exec] 
/usr/lib/jvm/java-1.6.0-ibm-1.6.0.12.0.x86_64/jre/lib/amd64/default/libjvm.so: 
undefined reference to `dladdr'
 [exec] 
/usr/lib/jvm/java-1.6.0-ibm-1.6.0.12.0.x86_64/jre/lib/amd64/default/libjvm.so: 
undefined reference to `dlopen'
 [exec] 
/usr/lib/jvm/java-1.6.0-ibm-1.6.0.12.0.x86_64/jre/lib/amd64/default/libjvm.so: 
undefined reference to `dlclose'
 [exec] collect2: ld returned 1 exit status
 [exec] make[2]: *** [test_libhdfs_ops] Error 1
 [exec] make[1]: *** [CMakeFiles/test_libhdfs_ops.dir/all] Error 2
 [exec] make: *** [all] Error 

The reason is libjvm.so need libdl when linking.

  was:
When native build hadoop-common-project with IBM java using command like: 
mvn package -Pnative
it will exist the following errors.
 [exec] -- Configuring incomplete, errors occurred!
 [exec] JAVA_HOME=, 
JAVA_JVM_LIBRARY=/home/louis/ibm-java-i386-60/jre/lib/i386/classic/libjvm.so
 [exec] JAVA_INCLUDE_PATH=/home/louis/ibm-java-i386-60/include, 
JAVA_INCLUDE_PATH2=JAVA_INCLUDE_PATH2-NOTFOUND
 [exec] CMake Error at JNIFlags.cmake:113 (MESSAGE):
 [exec]   Failed to find a viable JVM installation under JAVA_HOME.
 [exec] Call Stack (most recent call first):
 [exec]   CMakeLists.txt:24 (include)

The reason is that IBM java uses $JAVA_HOME/include/jniport.h instead of 
$JAVA_HOME/include/jni_md.h in Oracle java.



 Minimal code change to support IBM jvm build dependency
 ---

 Key: HADOOP-9435
 URL: https://issues.apache.org/jira/browse/HADOOP-9435
 Project: Hadoop Common
  Issue Type: Bug
  Components: build
Affects Versions: 2.0.3-alpha
Reporter: Tian Hong Wang
  Labels: patch
 Fix For: 2.0.3-alpha

 Attachments: HADOOP-9435.patch


 When native build hadoop-common-project with IBM java using command like: 
 mvn package -Pnative
 it will exist the following errors.
  [exec] -- Configuring incomplete, errors occurred!
  [exec] JAVA_HOME=, 
 JAVA_JVM_LIBRARY=/home/louis/ibm-java-i386-60/jre/lib/i386/classic/libjvm.so
  [exec] JAVA_INCLUDE_PATH=/home/louis/ibm-java-i386-60/include, 
 JAVA_INCLUDE_PATH2=JAVA_INCLUDE_PATH2-NOTFOUND
  [exec] CMake Error at JNIFlags.cmake:113 (MESSAGE):
  [exec]   Failed to find a viable JVM installation under JAVA_HOME.
  [exec] Call Stack (most recent call first):
  [exec]   CMakeLists.txt:24 (include)
 The reason is that IBM java uses $JAVA_HOME/include/jniport.h instead of 
 $JAVA_HOME/include/jni_md.h in non-IBM java.
 [exec] 
 /usr/lib/jvm/java-1.6.0-ibm-1.6.0.12.0.x86_64/jre/lib/amd64/default/libjvm.so:
  undefined reference to `dlsym'
  [exec] 
 /usr/lib/jvm/java-1.6.0-ibm-1.6.0.12.0.x86_64/jre/lib/amd64/default/libjvm.so:
  undefined reference to `dlerror'
  [exec] 
 /usr/lib/jvm/java-1.6.0-ibm-1.6.0.12.0.x86_64/jre/lib/amd64/default/libjvm.so:
  undefined reference to `dladdr'
  [exec] 
 /usr/lib/jvm/java-1.6.0-ibm-1.6.0.12.0.x86_64/jre/lib/amd64/default/libjvm.so:
  undefined reference to `dlopen'
  [exec] 
 /usr/lib/jvm/java-1.6.0-ibm-1.6.0.12.0.x86_64/jre/lib/amd64/default/libjvm.so:
  undefined reference to `dlclose'
  [exec] collect2: ld returned 1 exit status
  [exec] make[2]: *** [test_libhdfs_ops] Error 1
  [exec] make[1]: *** [CMakeFiles/test_libhdfs_ops.dir/all] Error 2
  [exec] make: *** [all] Error 
 The reason is libjvm.so need libdl when linking.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HADOOP-9435) Minimal code change to support IBM jvm build dependency


 [ 
https://issues.apache.org/jira/browse/HADOOP-9435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tian Hong Wang updated HADOOP-9435:
---

Status: Open  (was: Patch Available)

 Minimal code change to support IBM jvm build dependency
 ---

 Key: HADOOP-9435
 URL: https://issues.apache.org/jira/browse/HADOOP-9435
 Project: Hadoop Common
  Issue Type: Bug
  Components: build
Affects Versions: 2.0.3-alpha
Reporter: Tian Hong Wang
  Labels: patch
 Fix For: 2.0.3-alpha

 Attachments: HADOOP-9435.patch


 When native build hadoop-common-project with IBM java using command like: 
 mvn package -Pnative
 it will exist the following errors.
  [exec] -- Configuring incomplete, errors occurred!
  [exec] JAVA_HOME=, 
 JAVA_JVM_LIBRARY=/home/louis/ibm-java-i386-60/jre/lib/i386/classic/libjvm.so
  [exec] JAVA_INCLUDE_PATH=/home/louis/ibm-java-i386-60/include, 
 JAVA_INCLUDE_PATH2=JAVA_INCLUDE_PATH2-NOTFOUND
  [exec] CMake Error at JNIFlags.cmake:113 (MESSAGE):
  [exec]   Failed to find a viable JVM installation under JAVA_HOME.
  [exec] Call Stack (most recent call first):
  [exec]   CMakeLists.txt:24 (include)
 The reason is that IBM java uses $JAVA_HOME/include/jniport.h instead of 
 $JAVA_HOME/include/jni_md.h in non-IBM java.
 [exec] 
 /usr/lib/jvm/java-1.6.0-ibm-1.6.0.12.0.x86_64/jre/lib/amd64/default/libjvm.so:
  undefined reference to `dlsym'
  [exec] 
 /usr/lib/jvm/java-1.6.0-ibm-1.6.0.12.0.x86_64/jre/lib/amd64/default/libjvm.so:
  undefined reference to `dlerror'
  [exec] 
 /usr/lib/jvm/java-1.6.0-ibm-1.6.0.12.0.x86_64/jre/lib/amd64/default/libjvm.so:
  undefined reference to `dladdr'
  [exec] 
 /usr/lib/jvm/java-1.6.0-ibm-1.6.0.12.0.x86_64/jre/lib/amd64/default/libjvm.so:
  undefined reference to `dlopen'
  [exec] 
 /usr/lib/jvm/java-1.6.0-ibm-1.6.0.12.0.x86_64/jre/lib/amd64/default/libjvm.so:
  undefined reference to `dlclose'
  [exec] collect2: ld returned 1 exit status
  [exec] make[2]: *** [test_libhdfs_ops] Error 1
  [exec] make[1]: *** [CMakeFiles/test_libhdfs_ops.dir/all] Error 2
  [exec] make: *** [all] Error 
 The reason is libjvm.so need libdl when linking.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HADOOP-9435) Minimal code change to support IBM jvm build dependency


 [ 
https://issues.apache.org/jira/browse/HADOOP-9435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tian Hong Wang updated HADOOP-9435:
---

Affects Version/s: (was: 2.0.3-alpha)

 Minimal code change to support IBM jvm build dependency
 ---

 Key: HADOOP-9435
 URL: https://issues.apache.org/jira/browse/HADOOP-9435
 Project: Hadoop Common
  Issue Type: Bug
  Components: build
Reporter: Tian Hong Wang
  Labels: patch
 Fix For: 2.0.3-alpha

 Attachments: HADOOP-9435.patch


 When native build hadoop-common-project with IBM java using command like: 
 mvn package -Pnative
 it will exist the following errors.
  [exec] -- Configuring incomplete, errors occurred!
  [exec] JAVA_HOME=, 
 JAVA_JVM_LIBRARY=/home/louis/ibm-java-i386-60/jre/lib/i386/classic/libjvm.so
  [exec] JAVA_INCLUDE_PATH=/home/louis/ibm-java-i386-60/include, 
 JAVA_INCLUDE_PATH2=JAVA_INCLUDE_PATH2-NOTFOUND
  [exec] CMake Error at JNIFlags.cmake:113 (MESSAGE):
  [exec]   Failed to find a viable JVM installation under JAVA_HOME.
  [exec] Call Stack (most recent call first):
  [exec]   CMakeLists.txt:24 (include)
 The reason is that IBM java uses $JAVA_HOME/include/jniport.h instead of 
 $JAVA_HOME/include/jni_md.h in non-IBM java.
 [exec] 
 /usr/lib/jvm/java-1.6.0-ibm-1.6.0.12.0.x86_64/jre/lib/amd64/default/libjvm.so:
  undefined reference to `dlsym'
  [exec] 
 /usr/lib/jvm/java-1.6.0-ibm-1.6.0.12.0.x86_64/jre/lib/amd64/default/libjvm.so:
  undefined reference to `dlerror'
  [exec] 
 /usr/lib/jvm/java-1.6.0-ibm-1.6.0.12.0.x86_64/jre/lib/amd64/default/libjvm.so:
  undefined reference to `dladdr'
  [exec] 
 /usr/lib/jvm/java-1.6.0-ibm-1.6.0.12.0.x86_64/jre/lib/amd64/default/libjvm.so:
  undefined reference to `dlopen'
  [exec] 
 /usr/lib/jvm/java-1.6.0-ibm-1.6.0.12.0.x86_64/jre/lib/amd64/default/libjvm.so:
  undefined reference to `dlclose'
  [exec] collect2: ld returned 1 exit status
  [exec] make[2]: *** [test_libhdfs_ops] Error 1
  [exec] make[1]: *** [CMakeFiles/test_libhdfs_ops.dir/all] Error 2
  [exec] make: *** [all] Error 
 The reason is libjvm.so need libdl when linking.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HADOOP-9435) Minimal code change to support IBM jvm build dependency


 [ 
https://issues.apache.org/jira/browse/HADOOP-9435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tian Hong Wang updated HADOOP-9435:
---

Fix Version/s: (was: 2.0.3-alpha)

 Minimal code change to support IBM jvm build dependency
 ---

 Key: HADOOP-9435
 URL: https://issues.apache.org/jira/browse/HADOOP-9435
 Project: Hadoop Common
  Issue Type: Bug
  Components: build
Reporter: Tian Hong Wang
  Labels: patch
 Attachments: HADOOP-9435.patch


 When native build hadoop-common-project with IBM java using command like: 
 mvn package -Pnative
 it will exist the following errors.
  [exec] -- Configuring incomplete, errors occurred!
  [exec] JAVA_HOME=, 
 JAVA_JVM_LIBRARY=/home/louis/ibm-java-i386-60/jre/lib/i386/classic/libjvm.so
  [exec] JAVA_INCLUDE_PATH=/home/louis/ibm-java-i386-60/include, 
 JAVA_INCLUDE_PATH2=JAVA_INCLUDE_PATH2-NOTFOUND
  [exec] CMake Error at JNIFlags.cmake:113 (MESSAGE):
  [exec]   Failed to find a viable JVM installation under JAVA_HOME.
  [exec] Call Stack (most recent call first):
  [exec]   CMakeLists.txt:24 (include)
 The reason is that IBM java uses $JAVA_HOME/include/jniport.h instead of 
 $JAVA_HOME/include/jni_md.h in non-IBM java.
 [exec] 
 /usr/lib/jvm/java-1.6.0-ibm-1.6.0.12.0.x86_64/jre/lib/amd64/default/libjvm.so:
  undefined reference to `dlsym'
  [exec] 
 /usr/lib/jvm/java-1.6.0-ibm-1.6.0.12.0.x86_64/jre/lib/amd64/default/libjvm.so:
  undefined reference to `dlerror'
  [exec] 
 /usr/lib/jvm/java-1.6.0-ibm-1.6.0.12.0.x86_64/jre/lib/amd64/default/libjvm.so:
  undefined reference to `dladdr'
  [exec] 
 /usr/lib/jvm/java-1.6.0-ibm-1.6.0.12.0.x86_64/jre/lib/amd64/default/libjvm.so:
  undefined reference to `dlopen'
  [exec] 
 /usr/lib/jvm/java-1.6.0-ibm-1.6.0.12.0.x86_64/jre/lib/amd64/default/libjvm.so:
  undefined reference to `dlclose'
  [exec] collect2: ld returned 1 exit status
  [exec] make[2]: *** [test_libhdfs_ops] Error 1
  [exec] make[1]: *** [CMakeFiles/test_libhdfs_ops.dir/all] Error 2
  [exec] make: *** [all] Error 
 The reason is libjvm.so need libdl when linking.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Assigned] (HADOOP-9435) Minimal code change to support IBM jvm build dependency


 [ 
https://issues.apache.org/jira/browse/HADOOP-9435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tian Hong Wang reassigned HADOOP-9435:
--

Assignee: Tian Hong Wang

 Minimal code change to support IBM jvm build dependency
 ---

 Key: HADOOP-9435
 URL: https://issues.apache.org/jira/browse/HADOOP-9435
 Project: Hadoop Common
  Issue Type: Bug
  Components: build
Reporter: Tian Hong Wang
Assignee: Tian Hong Wang
  Labels: patch
 Attachments: HADOOP-9435.patch


 When native build hadoop-common-project with IBM java using command like: 
 mvn package -Pnative
 it will exist the following errors.
  [exec] -- Configuring incomplete, errors occurred!
  [exec] JAVA_HOME=, 
 JAVA_JVM_LIBRARY=/home/louis/ibm-java-i386-60/jre/lib/i386/classic/libjvm.so
  [exec] JAVA_INCLUDE_PATH=/home/louis/ibm-java-i386-60/include, 
 JAVA_INCLUDE_PATH2=JAVA_INCLUDE_PATH2-NOTFOUND
  [exec] CMake Error at JNIFlags.cmake:113 (MESSAGE):
  [exec]   Failed to find a viable JVM installation under JAVA_HOME.
  [exec] Call Stack (most recent call first):
  [exec]   CMakeLists.txt:24 (include)
 The reason is that IBM java uses $JAVA_HOME/include/jniport.h instead of 
 $JAVA_HOME/include/jni_md.h in non-IBM java.
 [exec] 
 /usr/lib/jvm/java-1.6.0-ibm-1.6.0.12.0.x86_64/jre/lib/amd64/default/libjvm.so:
  undefined reference to `dlsym'
  [exec] 
 /usr/lib/jvm/java-1.6.0-ibm-1.6.0.12.0.x86_64/jre/lib/amd64/default/libjvm.so:
  undefined reference to `dlerror'
  [exec] 
 /usr/lib/jvm/java-1.6.0-ibm-1.6.0.12.0.x86_64/jre/lib/amd64/default/libjvm.so:
  undefined reference to `dladdr'
  [exec] 
 /usr/lib/jvm/java-1.6.0-ibm-1.6.0.12.0.x86_64/jre/lib/amd64/default/libjvm.so:
  undefined reference to `dlopen'
  [exec] 
 /usr/lib/jvm/java-1.6.0-ibm-1.6.0.12.0.x86_64/jre/lib/amd64/default/libjvm.so:
  undefined reference to `dlclose'
  [exec] collect2: ld returned 1 exit status
  [exec] make[2]: *** [test_libhdfs_ops] Error 1
  [exec] make[1]: *** [CMakeFiles/test_libhdfs_ops.dir/all] Error 2
  [exec] make: *** [all] Error 
 The reason is libjvm.so need libdl when linking.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HADOOP-9435) Minimal code change to support IBM jvm build dependency


 [ 
https://issues.apache.org/jira/browse/HADOOP-9435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tian Hong Wang updated HADOOP-9435:
---

Attachment: HADOOP-9435-v1.patch

Resubmit patch to resolve two build dependency issues using IBM jvm.

 Minimal code change to support IBM jvm build dependency
 ---

 Key: HADOOP-9435
 URL: https://issues.apache.org/jira/browse/HADOOP-9435
 Project: Hadoop Common
  Issue Type: Bug
  Components: build
Reporter: Tian Hong Wang
Assignee: Tian Hong Wang
  Labels: patch
 Attachments: HADOOP-9435.patch, HADOOP-9435-v1.patch


 When native build hadoop-common-project with IBM java using command like: 
 mvn package -Pnative
 it will exist the following errors.
  [exec] -- Configuring incomplete, errors occurred!
  [exec] JAVA_HOME=, 
 JAVA_JVM_LIBRARY=/home/louis/ibm-java-i386-60/jre/lib/i386/classic/libjvm.so
  [exec] JAVA_INCLUDE_PATH=/home/louis/ibm-java-i386-60/include, 
 JAVA_INCLUDE_PATH2=JAVA_INCLUDE_PATH2-NOTFOUND
  [exec] CMake Error at JNIFlags.cmake:113 (MESSAGE):
  [exec]   Failed to find a viable JVM installation under JAVA_HOME.
  [exec] Call Stack (most recent call first):
  [exec]   CMakeLists.txt:24 (include)
 The reason is that IBM java uses $JAVA_HOME/include/jniport.h instead of 
 $JAVA_HOME/include/jni_md.h in non-IBM java.
 [exec] 
 /usr/lib/jvm/java-1.6.0-ibm-1.6.0.12.0.x86_64/jre/lib/amd64/default/libjvm.so:
  undefined reference to `dlsym'
  [exec] 
 /usr/lib/jvm/java-1.6.0-ibm-1.6.0.12.0.x86_64/jre/lib/amd64/default/libjvm.so:
  undefined reference to `dlerror'
  [exec] 
 /usr/lib/jvm/java-1.6.0-ibm-1.6.0.12.0.x86_64/jre/lib/amd64/default/libjvm.so:
  undefined reference to `dladdr'
  [exec] 
 /usr/lib/jvm/java-1.6.0-ibm-1.6.0.12.0.x86_64/jre/lib/amd64/default/libjvm.so:
  undefined reference to `dlopen'
  [exec] 
 /usr/lib/jvm/java-1.6.0-ibm-1.6.0.12.0.x86_64/jre/lib/amd64/default/libjvm.so:
  undefined reference to `dlclose'
  [exec] collect2: ld returned 1 exit status
  [exec] make[2]: *** [test_libhdfs_ops] Error 1
  [exec] make[1]: *** [CMakeFiles/test_libhdfs_ops.dir/all] Error 2
  [exec] make: *** [all] Error 
 The reason is libjvm.so need libdl when linking.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HADOOP-9435) Minimal code change to support IBM jvm build dependency


 [ 
https://issues.apache.org/jira/browse/HADOOP-9435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tian Hong Wang updated HADOOP-9435:
---

Status: Patch Available  (was: Open)

 Minimal code change to support IBM jvm build dependency
 ---

 Key: HADOOP-9435
 URL: https://issues.apache.org/jira/browse/HADOOP-9435
 Project: Hadoop Common
  Issue Type: Bug
  Components: build
Reporter: Tian Hong Wang
Assignee: Tian Hong Wang
  Labels: patch
 Attachments: HADOOP-9435.patch, HADOOP-9435-v1.patch


 When native build hadoop-common-project with IBM java using command like: 
 mvn package -Pnative
 it will exist the following errors.
  [exec] -- Configuring incomplete, errors occurred!
  [exec] JAVA_HOME=, 
 JAVA_JVM_LIBRARY=/home/louis/ibm-java-i386-60/jre/lib/i386/classic/libjvm.so
  [exec] JAVA_INCLUDE_PATH=/home/louis/ibm-java-i386-60/include, 
 JAVA_INCLUDE_PATH2=JAVA_INCLUDE_PATH2-NOTFOUND
  [exec] CMake Error at JNIFlags.cmake:113 (MESSAGE):
  [exec]   Failed to find a viable JVM installation under JAVA_HOME.
  [exec] Call Stack (most recent call first):
  [exec]   CMakeLists.txt:24 (include)
 The reason is that IBM java uses $JAVA_HOME/include/jniport.h instead of 
 $JAVA_HOME/include/jni_md.h in non-IBM java.
 [exec] 
 /usr/lib/jvm/java-1.6.0-ibm-1.6.0.12.0.x86_64/jre/lib/amd64/default/libjvm.so:
  undefined reference to `dlsym'
  [exec] 
 /usr/lib/jvm/java-1.6.0-ibm-1.6.0.12.0.x86_64/jre/lib/amd64/default/libjvm.so:
  undefined reference to `dlerror'
  [exec] 
 /usr/lib/jvm/java-1.6.0-ibm-1.6.0.12.0.x86_64/jre/lib/amd64/default/libjvm.so:
  undefined reference to `dladdr'
  [exec] 
 /usr/lib/jvm/java-1.6.0-ibm-1.6.0.12.0.x86_64/jre/lib/amd64/default/libjvm.so:
  undefined reference to `dlopen'
  [exec] 
 /usr/lib/jvm/java-1.6.0-ibm-1.6.0.12.0.x86_64/jre/lib/amd64/default/libjvm.so:
  undefined reference to `dlclose'
  [exec] collect2: ld returned 1 exit status
  [exec] make[2]: *** [test_libhdfs_ops] Error 1
  [exec] make[1]: *** [CMakeFiles/test_libhdfs_ops.dir/all] Error 2
  [exec] make: *** [all] Error 
 The reason is libjvm.so need libdl when linking.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HADOOP-9046) provide unit-test coverage of class org.apache.hadoop.fs.DelegationTokenRenewer.RenewActionT

2013-05-14 Thread Ivan A. Veselovsky (JIRA)


 [ 
https://issues.apache.org/jira/browse/HADOOP-9046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ivan A. Veselovsky updated HADOOP-9046:
---

Affects Version/s: (was: 2.0.3-alpha)
   (was: 3.0.0)

 provide unit-test coverage of class 
 org.apache.hadoop.fs.DelegationTokenRenewer.RenewActionT
 --

 Key: HADOOP-9046
 URL: https://issues.apache.org/jira/browse/HADOOP-9046
 Project: Hadoop Common
  Issue Type: Test
Affects Versions: 0.23.6
Reporter: Ivan A. Veselovsky
Assignee: Ivan A. Veselovsky
Priority: Minor
 Attachments: HADOOP-9046-branch-0.23--c.patch, 
 HADOOP-9046-branch-0.23--d.patch, HADOOP-9046-branch-0.23-over-9049.patch, 
 HADOOP-9046-branch-0.23--over-HDFS-4567.patch, HADOOP-9046-branch-0.23.patch, 
 HADOOP-9046-branch-2--over-HDFS-4567.patch, HADOOP-9046--c.patch, 
 HADOOP-9046--d.patch, HADOOP-9046--e.patch, HADOOP-9046-over-9049.patch, 
 HADOOP-9046.patch, HADOOP-9046-trunk--over-HDFS-4567.patch


 The class org.apache.hadoop.fs.DelegationTokenRenewer.RenewActionT has zero 
 coverage in entire cumulative test run. Provide test(s) to cover this class.
 Note: the request submitted to HDFS project because the class likely to be 
 tested by tests in that project.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-9046) provide unit-test coverage of class org.apache.hadoop.fs.DelegationTokenRenewer.RenewActionT

2013-05-14 Thread Ivan A. Veselovsky (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-9046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13656918#comment-13656918
 ] 

Ivan A. Veselovsky commented on HADOOP-9046:


Since fix https://issues.apache.org/jira/browse/HADOOP-9549 the patch is no 
longer applicable to branches trunk and branch-2 (needs merge).
It looks like there is no big reason to merge over 9549 because after the fix 
9549 the coverage of the target classes is enough.
So, the suggested patch is now applicable to branch-0.23 only.
Modifying the Affected versions field accordingly.

 provide unit-test coverage of class 
 org.apache.hadoop.fs.DelegationTokenRenewer.RenewActionT
 --

 Key: HADOOP-9046
 URL: https://issues.apache.org/jira/browse/HADOOP-9046
 Project: Hadoop Common
  Issue Type: Test
Affects Versions: 0.23.6
Reporter: Ivan A. Veselovsky
Assignee: Ivan A. Veselovsky
Priority: Minor
 Attachments: HADOOP-9046-branch-0.23--c.patch, 
 HADOOP-9046-branch-0.23--d.patch, HADOOP-9046-branch-0.23-over-9049.patch, 
 HADOOP-9046-branch-0.23--over-HDFS-4567.patch, HADOOP-9046-branch-0.23.patch, 
 HADOOP-9046-branch-2--over-HDFS-4567.patch, HADOOP-9046--c.patch, 
 HADOOP-9046--d.patch, HADOOP-9046--e.patch, HADOOP-9046-over-9049.patch, 
 HADOOP-9046.patch, HADOOP-9046-trunk--over-HDFS-4567.patch


 The class org.apache.hadoop.fs.DelegationTokenRenewer.RenewActionT has zero 
 coverage in entire cumulative test run. Provide test(s) to cover this class.
 Note: the request submitted to HDFS project because the class likely to be 
 tested by tests in that project.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-9220) Unnecessary transition to standby in ActiveStandbyElector

2013-05-14 Thread Tom White (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-9220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13656937#comment-13656937
 ] 

Tom White commented on HADOOP-9220:
---

+1 for Todd's patch. I tested it manually and the original problem no longer 
occurs with the patch.

Nit: did you mean to introduce {{new Exception}} in the debug line?

 Unnecessary transition to standby in ActiveStandbyElector
 -

 Key: HADOOP-9220
 URL: https://issues.apache.org/jira/browse/HADOOP-9220
 Project: Hadoop Common
  Issue Type: Bug
  Components: ha
Reporter: Tom White
Assignee: Tom White
Priority: Critical
 Attachments: HADOOP-9220.patch, HADOOP-9220.patch, hadoop-9220.txt


 When performing a manual failover from one HA node to a second, under some 
 circumstances the second node will transition from standby - active - 
 standby - active. This is with automatic failover enabled, so there is a ZK 
 cluster doing leader election.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-9459) ActiveStandbyElector can join election even before Service HEALTHY, and results in null data at ActiveBreadCrumb

[
https://issues.apache.org/jira/browse/HADOOP-9459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13656952#comment-13656952
]

Hudson commented on HADOOP-9459:

Integrated in Hadoop-Yarn-trunk #209 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk/209/])
HADOOP-9459. ActiveStandbyElector can join election even before Service
HEALTHY, and results in null data at ActiveBreadCrumb. Contributed by Vinay and
Todd Lipcon. (Revision 1482227)

Result = SUCCESS
todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1482227
Files :
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
*
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ha/ActiveStandbyElector.java
*
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ha/TestActiveStandbyElector.java

ActiveStandbyElector can join election even before Service HEALTHY, and
results in null data at ActiveBreadCrumb

Attachments: HDFS-4463.patch, hdfs-4463.txt

[jira] [Commented] (HADOOP-9435) Minimal code change to support IBM jvm build dependency

2013-05-14 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-9435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13656965#comment-13656965
 ] 

Hadoop QA commented on HADOOP-9435:
---

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12583101/HADOOP-9435-v1.patch
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:red}-1 tests included{color}.  The patch doesn't appear to include 
any new or modified tests.
Please justify why no new tests are needed for this 
patch.
Also please list what manual steps were performed to 
verify this patch.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  The javadoc tool did not generate any 
warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 core tests{color}.  The patch passed unit tests in 
hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs.

{color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: 
https://builds.apache.org/job/PreCommit-HADOOP-Build/2540//testReport/
Console output: 
https://builds.apache.org/job/PreCommit-HADOOP-Build/2540//console

This message is automatically generated.

 Minimal code change to support IBM jvm build dependency
 ---

 Key: HADOOP-9435
 URL: https://issues.apache.org/jira/browse/HADOOP-9435
 Project: Hadoop Common
  Issue Type: Bug
  Components: build
Reporter: Tian Hong Wang
Assignee: Tian Hong Wang
  Labels: patch
 Attachments: HADOOP-9435.patch, HADOOP-9435-v1.patch


 When native build hadoop-common-project with IBM java using command like: 
 mvn package -Pnative
 it will exist the following errors.
  [exec] -- Configuring incomplete, errors occurred!
  [exec] JAVA_HOME=, 
 JAVA_JVM_LIBRARY=/home/louis/ibm-java-i386-60/jre/lib/i386/classic/libjvm.so
  [exec] JAVA_INCLUDE_PATH=/home/louis/ibm-java-i386-60/include, 
 JAVA_INCLUDE_PATH2=JAVA_INCLUDE_PATH2-NOTFOUND
  [exec] CMake Error at JNIFlags.cmake:113 (MESSAGE):
  [exec]   Failed to find a viable JVM installation under JAVA_HOME.
  [exec] Call Stack (most recent call first):
  [exec]   CMakeLists.txt:24 (include)
 The reason is that IBM java uses $JAVA_HOME/include/jniport.h instead of 
 $JAVA_HOME/include/jni_md.h in non-IBM java.
 [exec] 
 /usr/lib/jvm/java-1.6.0-ibm-1.6.0.12.0.x86_64/jre/lib/amd64/default/libjvm.so:
  undefined reference to `dlsym'
  [exec] 
 /usr/lib/jvm/java-1.6.0-ibm-1.6.0.12.0.x86_64/jre/lib/amd64/default/libjvm.so:
  undefined reference to `dlerror'
  [exec] 
 /usr/lib/jvm/java-1.6.0-ibm-1.6.0.12.0.x86_64/jre/lib/amd64/default/libjvm.so:
  undefined reference to `dladdr'
  [exec] 
 /usr/lib/jvm/java-1.6.0-ibm-1.6.0.12.0.x86_64/jre/lib/amd64/default/libjvm.so:
  undefined reference to `dlopen'
  [exec] 
 /usr/lib/jvm/java-1.6.0-ibm-1.6.0.12.0.x86_64/jre/lib/amd64/default/libjvm.so:
  undefined reference to `dlclose'
  [exec] collect2: ld returned 1 exit status
  [exec] make[2]: *** [test_libhdfs_ops] Error 1
  [exec] make[1]: *** [CMakeFiles/test_libhdfs_ops.dir/all] Error 2
  [exec] make: *** [all] Error 
 The reason is libjvm.so need libdl when linking.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-9459) ActiveStandbyElector can join election even before Service HEALTHY, and results in null data at ActiveBreadCrumb

[
https://issues.apache.org/jira/browse/HADOOP-9459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13657035#comment-13657035
]

Hudson commented on HADOOP-9459:

Integrated in Hadoop-Hdfs-trunk #1398 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/1398/])
HADOOP-9459. ActiveStandbyElector can join election even before Service
HEALTHY, and results in null data at ActiveBreadCrumb. Contributed by Vinay and
Todd Lipcon. (Revision 1482227)

Result = FAILURE
todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1482227
Files :
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
*
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ha/ActiveStandbyElector.java
*
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ha/TestActiveStandbyElector.java

ActiveStandbyElector can join election even before Service HEALTHY, and
results in null data at ActiveBreadCrumb

Attachments: HDFS-4463.patch, hdfs-4463.txt

[jira] [Commented] (HADOOP-9459) ActiveStandbyElector can join election even before Service HEALTHY, and results in null data at ActiveBreadCrumb

[
https://issues.apache.org/jira/browse/HADOOP-9459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13657071#comment-13657071
]

Hudson commented on HADOOP-9459:

Integrated in Hadoop-Mapreduce-trunk #1425 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1425/])
HADOOP-9459. ActiveStandbyElector can join election even before Service
HEALTHY, and results in null data at ActiveBreadCrumb. Contributed by Vinay and
Todd Lipcon. (Revision 1482227)

ActiveStandbyElector can join election even before Service HEALTHY, and
results in null data at ActiveBreadCrumb

Attachments: HDFS-4463.patch, hdfs-4463.txt

[jira] [Commented] (HADOOP-9046) provide unit-test coverage of class org.apache.hadoop.fs.DelegationTokenRenewer.RenewActionT

2013-05-14 Thread Robert Joseph Evans (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-9046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13657095#comment-13657095
 ] 

Robert Joseph Evans commented on HADOOP-9046:
-

Ivan, I am sorry I dropped the ball on this, and did not repond sooner.  Sadly 
though the only time we want to put something into just 0.23 without going into 
trunk/branch-2 first is if there is a bug that only exists in branch-0.23.  
This is a very rare occurrence.  If you think the coverage is good enough on 
branch-2 and trunk then would it be acceptable if we resolve this JIRA instead.

 provide unit-test coverage of class 
 org.apache.hadoop.fs.DelegationTokenRenewer.RenewActionT
 --

 Key: HADOOP-9046
 URL: https://issues.apache.org/jira/browse/HADOOP-9046
 Project: Hadoop Common
  Issue Type: Test
Affects Versions: 0.23.6
Reporter: Ivan A. Veselovsky
Assignee: Ivan A. Veselovsky
Priority: Minor
 Attachments: HADOOP-9046-branch-0.23--c.patch, 
 HADOOP-9046-branch-0.23--d.patch, HADOOP-9046-branch-0.23-over-9049.patch, 
 HADOOP-9046-branch-0.23--over-HDFS-4567.patch, HADOOP-9046-branch-0.23.patch, 
 HADOOP-9046-branch-2--over-HDFS-4567.patch, HADOOP-9046--c.patch, 
 HADOOP-9046--d.patch, HADOOP-9046--e.patch, HADOOP-9046-over-9049.patch, 
 HADOOP-9046.patch, HADOOP-9046-trunk--over-HDFS-4567.patch


 The class org.apache.hadoop.fs.DelegationTokenRenewer.RenewActionT has zero 
 coverage in entire cumulative test run. Provide test(s) to cover this class.
 Note: the request submitted to HDFS project because the class likely to be 
 tested by tests in that project.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HADOOP-9560) metrics2#JvmMetrics should have max memory size of JVM


 [ 
https://issues.apache.org/jira/browse/HADOOP-9560?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Suresh Srinivas updated HADOOP-9560:


   Resolution: Fixed
Fix Version/s: 2.0.5-beta
 Hadoop Flags: Reviewed
   Status: Resolved  (was: Patch Available)

I have committed this change to trunk and branch-2.

Thank you Tsuyoshi!

 metrics2#JvmMetrics should have max memory size of JVM
 --

 Key: HADOOP-9560
 URL: https://issues.apache.org/jira/browse/HADOOP-9560
 Project: Hadoop Common
  Issue Type: Improvement
  Components: metrics
Affects Versions: 3.0.0, 2.0.5-beta
Reporter: Tsuyoshi OZAWA
Assignee: Tsuyoshi OZAWA
Priority: Minor
 Fix For: 2.0.5-beta

 Attachments: HADOOP-9560.1.patch


 metrics#JvmMetrics have the max memory size of JVM specified by -Xmx option.
 metrics2#JvmMetrics doesn't have it but it should also have max memory size 
 of JVM, because it's useful for users.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HADOOP-9307) BufferedFSInputStream.read returns wrong results after certain seeks


 [ 
https://issues.apache.org/jira/browse/HADOOP-9307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Todd Lipcon updated HADOOP-9307:


Attachment: hadoop-9307-branch-1.txt

I committed to branch-2 and trunk. Here's a backport to branch-1 as well

 BufferedFSInputStream.read returns wrong results after certain seeks
 

 Key: HADOOP-9307
 URL: https://issues.apache.org/jira/browse/HADOOP-9307
 Project: Hadoop Common
  Issue Type: Bug
  Components: fs
Affects Versions: 1.1.1, 2.0.2-alpha
Reporter: Todd Lipcon
Assignee: Todd Lipcon
 Attachments: hadoop-9307-branch-1.txt, hadoop-9307.txt


 After certain sequences of seek/read, BufferedFSInputStream can silently 
 return data from the wrong part of the file. Further description in first 
 comment below.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HADOOP-9307) BufferedFSInputStream.read returns wrong results after certain seeks


 [ 
https://issues.apache.org/jira/browse/HADOOP-9307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Todd Lipcon updated HADOOP-9307:


Target Version/s: 2.0.5-beta, 1.3.0  (was: 2.0.5-beta)
   Fix Version/s: 2.0.5-beta
  3.0.0

 BufferedFSInputStream.read returns wrong results after certain seeks
 

 Key: HADOOP-9307
 URL: https://issues.apache.org/jira/browse/HADOOP-9307
 Project: Hadoop Common
  Issue Type: Bug
  Components: fs
Affects Versions: 1.1.1, 2.0.2-alpha
Reporter: Todd Lipcon
Assignee: Todd Lipcon
 Fix For: 3.0.0, 2.0.5-beta

 Attachments: hadoop-9307-branch-1.txt, hadoop-9307.txt


 After certain sequences of seek/read, BufferedFSInputStream can silently 
 return data from the wrong part of the file. Further description in first 
 comment below.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-9560) metrics2#JvmMetrics should have max memory size of JVM

2013-05-14 Thread Tsuyoshi OZAWA (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-9560?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13657133#comment-13657133
 ] 

Tsuyoshi OZAWA commented on HADOOP-9560:


Thank you too, Suresh!

 metrics2#JvmMetrics should have max memory size of JVM
 --

 Key: HADOOP-9560
 URL: https://issues.apache.org/jira/browse/HADOOP-9560
 Project: Hadoop Common
  Issue Type: Improvement
  Components: metrics
Affects Versions: 3.0.0, 2.0.5-beta
Reporter: Tsuyoshi OZAWA
Assignee: Tsuyoshi OZAWA
Priority: Minor
 Fix For: 2.0.5-beta

 Attachments: HADOOP-9560.1.patch


 metrics#JvmMetrics have the max memory size of JVM specified by -Xmx option.
 metrics2#JvmMetrics doesn't have it but it should also have max memory size 
 of JVM, because it's useful for users.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-9220) Unnecessary transition to standby in ActiveStandbyElector


[ 
https://issues.apache.org/jira/browse/HADOOP-9220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13657135#comment-13657135
 ] 

Todd Lipcon commented on HADOOP-9220:
-

Woops, nope, I'll take out that 'new Exception' on commit. I was just using it 
to help debug. Good catch.

 Unnecessary transition to standby in ActiveStandbyElector
 -

 Key: HADOOP-9220
 URL: https://issues.apache.org/jira/browse/HADOOP-9220
 Project: Hadoop Common
  Issue Type: Bug
  Components: ha
Reporter: Tom White
Assignee: Tom White
Priority: Critical
 Attachments: HADOOP-9220.patch, HADOOP-9220.patch, hadoop-9220.txt


 When performing a manual failover from one HA node to a second, under some 
 circumstances the second node will transition from standby - active - 
 standby - active. This is with automatic failover enabled, so there is a ZK 
 cluster doing leader election.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HADOOP-9220) Unnecessary transition to standby in ActiveStandbyElector


 [ 
https://issues.apache.org/jira/browse/HADOOP-9220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Todd Lipcon updated HADOOP-9220:


   Resolution: Fixed
Fix Version/s: 2.0.5-beta
   3.0.0
 Hadoop Flags: Reviewed
   Status: Resolved  (was: Patch Available)

Committed to branch-2 and trunk. thanks Tom for tracking this down

 Unnecessary transition to standby in ActiveStandbyElector
 -

 Key: HADOOP-9220
 URL: https://issues.apache.org/jira/browse/HADOOP-9220
 Project: Hadoop Common
  Issue Type: Bug
  Components: ha
Reporter: Tom White
Assignee: Tom White
Priority: Critical
 Fix For: 3.0.0, 2.0.5-beta

 Attachments: HADOOP-9220.patch, HADOOP-9220.patch, hadoop-9220.txt


 When performing a manual failover from one HA node to a second, under some 
 circumstances the second node will transition from standby - active - 
 standby - active. This is with automatic failover enabled, so there is a ZK 
 cluster doing leader election.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-9220) Unnecessary transition to standby in ActiveStandbyElector


[ 
https://issues.apache.org/jira/browse/HADOOP-9220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13657144#comment-13657144
 ] 

Hudson commented on HADOOP-9220:


Integrated in Hadoop-trunk-Commit #3750 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/3750/])
HADOOP-9220. Unnecessary transition to standby in ActiveStandbyElector. 
Contributed by Tom White and Todd Lipcon. (Revision 1482401)

 Result = SUCCESS
todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1482401
Files : 
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ha/ActiveStandbyElector.java
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ha/DummyHAService.java
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ha/TestZKFailoverController.java


 Unnecessary transition to standby in ActiveStandbyElector
 -

 Key: HADOOP-9220
 URL: https://issues.apache.org/jira/browse/HADOOP-9220
 Project: Hadoop Common
  Issue Type: Bug
  Components: ha
Reporter: Tom White
Assignee: Tom White
Priority: Critical
 Fix For: 3.0.0, 2.0.5-beta

 Attachments: HADOOP-9220.patch, HADOOP-9220.patch, hadoop-9220.txt


 When performing a manual failover from one HA node to a second, under some 
 circumstances the second node will transition from standby - active - 
 standby - active. This is with automatic failover enabled, so there is a ZK 
 cluster doing leader election.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-9079) LocalDirAllocator throws ArithmeticException


[ 
https://issues.apache.org/jira/browse/HADOOP-9079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13657160#comment-13657160
 ] 

Todd Lipcon commented on HADOOP-9079:
-

Could you make a unit test by making a directory, calling setReadable(false) 
and setExecutable(false), and then trying to set up a LocalDirAllocator inside 
it?

 LocalDirAllocator throws ArithmeticException
 

 Key: HADOOP-9079
 URL: https://issues.apache.org/jira/browse/HADOOP-9079
 Project: Hadoop Common
  Issue Type: Bug
Affects Versions: 2.0.3-alpha
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang
Priority: Minor
 Attachments: hadoop-9079-v2.txt, trunk-9079.patch


 2012-11-19 22:07:41,709 WARN  [IPC Server handler 0 on 38671] 
 nodemanager.NMAuditLogger(150): USER=UnknownUserIP= 
 OPERATION=Stop Container RequestTARGET=ContainerManagerImpl 
 RESULT=FAILURE  DESCRIPTION=Trying to stop unknown container!   
 APPID=application_1353391620476_0001
 CONTAINERID=container_1353391620476_0001_01_10
 java.lang.ArithmeticException: / by zero
   at 
 org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:368)
   at 
 org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:150)
   at 
 org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:131)
   at 
 org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:115)
   at 
 org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:263)
   at 
 org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:849)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-6344) rm and rmr fail to correctly move the user's files to the trash prior to deleting when they are over quota.

2013-05-14 Thread Albert XU (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-6344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13657166#comment-13657166
 ] 

Albert XU commented on HADOOP-6344:
---

It could be reproduced in the Hadoop-1.0.4

# bin/hadoop dfs -rmr input
13/05/15 00:14:15 WARN util.NativeCodeLoader: Unable to load native-hadoop 
library for your platform... using builtin-java classes where applicable
13/05/15 00:14:15 WARN fs.Trash: Can't create trash directory: 
file:/C:/Documents and Settings/XU 
Zheng/.Trash/Current/C:/cygwin/usr/hadoop-1.0.4
Problem with Trash.Failed to set permissions of path: C:\Documents and 
Settings\XU Zheng\.Trash\Current\C:\cygwin\usr\hadoop-1.0.4 to 0700. Consider 
using -skipTrash option
rmr: Failed to move to trash: file:/C:/cygwin/usr/hadoop-1.0.4/input



 rm and rmr fail to correctly move the user's files to the trash prior to 
 deleting when they are over quota.  
 -

 Key: HADOOP-6344
 URL: https://issues.apache.org/jira/browse/HADOOP-6344
 Project: Hadoop Common
  Issue Type: Bug
  Components: fs
Affects Versions: 0.20.0, 0.20.1, 0.21.0, 0.22.0
Reporter: gary murry
Assignee: Jakob Homan
 Fix For: 0.21.0, 0.22.0

 Attachments: HDFS-740-for-Y20.patch, HDFS-740.patch


 With trash turned on, if a user is over his quota and does a rm (or rmr), the 
 file is deleted without a copy being placed in the trash.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HADOOP-9079) LocalDirAllocator throws ArithmeticException

2013-05-14 Thread Jimmy Xiang (JIRA)


 [ 
https://issues.apache.org/jira/browse/HADOOP-9079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jimmy Xiang updated HADOOP-9079:


Assignee: (was: Jimmy Xiang)

 LocalDirAllocator throws ArithmeticException
 

 Key: HADOOP-9079
 URL: https://issues.apache.org/jira/browse/HADOOP-9079
 Project: Hadoop Common
  Issue Type: Bug
Affects Versions: 2.0.3-alpha
Reporter: Jimmy Xiang
Priority: Minor
 Attachments: hadoop-9079-v2.txt, trunk-9079.patch


 2012-11-19 22:07:41,709 WARN  [IPC Server handler 0 on 38671] 
 nodemanager.NMAuditLogger(150): USER=UnknownUserIP= 
 OPERATION=Stop Container RequestTARGET=ContainerManagerImpl 
 RESULT=FAILURE  DESCRIPTION=Trying to stop unknown container!   
 APPID=application_1353391620476_0001
 CONTAINERID=container_1353391620476_0001_01_10
 java.lang.ArithmeticException: / by zero
   at 
 org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:368)
   at 
 org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:150)
   at 
 org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:131)
   at 
 org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:115)
   at 
 org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:263)
   at 
 org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:849)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HADOOP-9079) LocalDirAllocator throws ArithmeticException

2013-05-14 Thread Jimmy Xiang (JIRA)


 [ 
https://issues.apache.org/jira/browse/HADOOP-9079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jimmy Xiang updated HADOOP-9079:


Status: Open  (was: Patch Available)

 LocalDirAllocator throws ArithmeticException
 

 Key: HADOOP-9079
 URL: https://issues.apache.org/jira/browse/HADOOP-9079
 Project: Hadoop Common
  Issue Type: Bug
Affects Versions: 2.0.3-alpha
Reporter: Jimmy Xiang
Priority: Minor
 Attachments: hadoop-9079-v2.txt, trunk-9079.patch


 2012-11-19 22:07:41,709 WARN  [IPC Server handler 0 on 38671] 
 nodemanager.NMAuditLogger(150): USER=UnknownUserIP= 
 OPERATION=Stop Container RequestTARGET=ContainerManagerImpl 
 RESULT=FAILURE  DESCRIPTION=Trying to stop unknown container!   
 APPID=application_1353391620476_0001
 CONTAINERID=container_1353391620476_0001_01_10
 java.lang.ArithmeticException: / by zero
   at 
 org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:368)
   at 
 org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:150)
   at 
 org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:131)
   at 
 org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:115)
   at 
 org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:263)
   at 
 org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:849)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-9079) LocalDirAllocator throws ArithmeticException

2013-05-14 Thread Jimmy Xiang (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-9079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13657189#comment-13657189
 ] 

Jimmy Xiang commented on HADOOP-9079:
-

I can work on it later when I get some time.  Please feel free to take it over 
if anybody wants to help.

 LocalDirAllocator throws ArithmeticException
 

 Key: HADOOP-9079
 URL: https://issues.apache.org/jira/browse/HADOOP-9079
 Project: Hadoop Common
  Issue Type: Bug
Affects Versions: 2.0.3-alpha
Reporter: Jimmy Xiang
Priority: Minor
 Attachments: hadoop-9079-v2.txt, trunk-9079.patch


 2012-11-19 22:07:41,709 WARN  [IPC Server handler 0 on 38671] 
 nodemanager.NMAuditLogger(150): USER=UnknownUserIP= 
 OPERATION=Stop Container RequestTARGET=ContainerManagerImpl 
 RESULT=FAILURE  DESCRIPTION=Trying to stop unknown container!   
 APPID=application_1353391620476_0001
 CONTAINERID=container_1353391620476_0001_01_10
 java.lang.ArithmeticException: / by zero
   at 
 org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:368)
   at 
 org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:150)
   at 
 org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:131)
   at 
 org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:115)
   at 
 org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:263)
   at 
 org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:849)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-9435) Minimal code change to support IBM jvm build dependency

2013-05-14 Thread Colin Patrick McCabe (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-9435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13657235#comment-13657235
 ] 

Colin Patrick McCabe commented on HADOOP-9435:
--

I built this against Sun JDK 6 and it worked fine.

It looks good to me, although I didn't test against an IBM JVM.

 Minimal code change to support IBM jvm build dependency
 ---

 Key: HADOOP-9435
 URL: https://issues.apache.org/jira/browse/HADOOP-9435
 Project: Hadoop Common
  Issue Type: Bug
  Components: build
Reporter: Tian Hong Wang
Assignee: Tian Hong Wang
  Labels: patch
 Attachments: HADOOP-9435.patch, HADOOP-9435-v1.patch


 When native build hadoop-common-project with IBM java using command like: 
 mvn package -Pnative
 it will exist the following errors.
  [exec] -- Configuring incomplete, errors occurred!
  [exec] JAVA_HOME=, 
 JAVA_JVM_LIBRARY=/home/louis/ibm-java-i386-60/jre/lib/i386/classic/libjvm.so
  [exec] JAVA_INCLUDE_PATH=/home/louis/ibm-java-i386-60/include, 
 JAVA_INCLUDE_PATH2=JAVA_INCLUDE_PATH2-NOTFOUND
  [exec] CMake Error at JNIFlags.cmake:113 (MESSAGE):
  [exec]   Failed to find a viable JVM installation under JAVA_HOME.
  [exec] Call Stack (most recent call first):
  [exec]   CMakeLists.txt:24 (include)
 The reason is that IBM java uses $JAVA_HOME/include/jniport.h instead of 
 $JAVA_HOME/include/jni_md.h in non-IBM java.
 [exec] 
 /usr/lib/jvm/java-1.6.0-ibm-1.6.0.12.0.x86_64/jre/lib/amd64/default/libjvm.so:
  undefined reference to `dlsym'
  [exec] 
 /usr/lib/jvm/java-1.6.0-ibm-1.6.0.12.0.x86_64/jre/lib/amd64/default/libjvm.so:
  undefined reference to `dlerror'
  [exec] 
 /usr/lib/jvm/java-1.6.0-ibm-1.6.0.12.0.x86_64/jre/lib/amd64/default/libjvm.so:
  undefined reference to `dladdr'
  [exec] 
 /usr/lib/jvm/java-1.6.0-ibm-1.6.0.12.0.x86_64/jre/lib/amd64/default/libjvm.so:
  undefined reference to `dlopen'
  [exec] 
 /usr/lib/jvm/java-1.6.0-ibm-1.6.0.12.0.x86_64/jre/lib/amd64/default/libjvm.so:
  undefined reference to `dlclose'
  [exec] collect2: ld returned 1 exit status
  [exec] make[2]: *** [test_libhdfs_ops] Error 1
  [exec] make[1]: *** [CMakeFiles/test_libhdfs_ops.dir/all] Error 2
  [exec] make: *** [all] Error 
 The reason is libjvm.so need libdl when linking.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-9079) LocalDirAllocator throws ArithmeticException

2013-05-14 Thread Ted Yu (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-9079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13657241#comment-13657241
 ] 

Ted Yu commented on HADOOP-9079:


I made modification in TestLocalDirAllocator#testRWBufferDirBecomesRO:
{code}
Index: 
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestLocalDirAllocator.java
===
--- 
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestLocalDirAllocator.java
   (revision 1482467)
+++ 
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestLocalDirAllocator.java
   (working copy)
@@ -215,7 +215,9 @@
   validateTempDirCreation(buildBufferDir(ROOT, nextDirIdx));

   // change buffer directory 2 to be read only
-  new File(new Path(dir4).toUri().getPath()).setReadOnly();
+  File f0 = new File(new Path(dir4).toUri().getPath());
+  f0.setReadable(false);
+  f0.setExecutable(false);
   validateTempDirCreation(dir3);
   validateTempDirCreation(dir3);
 } finally {
{code}
When I ran the test, I got:
{code}
initializationError(org.apache.hadoop.fs.TestLocalDirAllocator)  Time elapsed: 
5 sec   FAILURE!
java.lang.AssertionError:
  at org.junit.Assert.fail(Assert.java:91)
  at org.junit.Assert.assertTrue(Assert.java:43)
  at org.junit.Assert.assertTrue(Assert.java:54)
  at 
org.apache.hadoop.fs.TestLocalDirAllocator.rmBufferDirs(TestLocalDirAllocator.java:104)
  at 
org.apache.hadoop.fs.TestLocalDirAllocator.clinit(TestLocalDirAllocator.java:71)
{code}

 LocalDirAllocator throws ArithmeticException
 

 Key: HADOOP-9079
 URL: https://issues.apache.org/jira/browse/HADOOP-9079
 Project: Hadoop Common
  Issue Type: Bug
Affects Versions: 2.0.3-alpha
Reporter: Jimmy Xiang
Priority: Minor
 Attachments: hadoop-9079-v2.txt, trunk-9079.patch


 2012-11-19 22:07:41,709 WARN  [IPC Server handler 0 on 38671] 
 nodemanager.NMAuditLogger(150): USER=UnknownUserIP= 
 OPERATION=Stop Container RequestTARGET=ContainerManagerImpl 
 RESULT=FAILURE  DESCRIPTION=Trying to stop unknown container!   
 APPID=application_1353391620476_0001
 CONTAINERID=container_1353391620476_0001_01_10
 java.lang.ArithmeticException: / by zero
   at 
 org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:368)
   at 
 org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:150)
   at 
 org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:131)
   at 
 org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:115)
   at 
 org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:263)
   at 
 org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:849)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-9533) Centralized Hadoop SSO/Token Server

2013-05-14 Thread Brian Swan (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-9533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13657251#comment-13657251
 ] 

Brian Swan commented on HADOOP-9533:


Thanks for the heavy lifting on this, Larry. This issue is headed in the right 
direction. As you point out, it addresses the use cases of cloud, on-premises, 
hybrid enterprise deployments, and 3rd party integration scenarios - all of 
which are broadly important and are also our (Microsoft) high priority 
scenarios.

To Daryn's comment about getting together at Hadoop Summit - I think that's a 
great idea. I plan to be there and look forward to discussions on how to 
rationalize related work and collaborate on the delivery of it.

 Centralized Hadoop SSO/Token Server
 ---

 Key: HADOOP-9533
 URL: https://issues.apache.org/jira/browse/HADOOP-9533
 Project: Hadoop Common
  Issue Type: New Feature
  Components: security
Reporter: Larry McCay
 Attachments: HSSO-Interaction-Overview-rev-1.docx, 
 HSSO-Interaction-Overview-rev-1.pdf


 This is an umbrella Jira filing to oversee a set of proposals for introducing 
 a new master service for Hadoop Single Sign On (HSSO).
 There is an increasing need for pluggable authentication providers that 
 authenticate both users and services as well as validate tokens in order to 
 federate identities authenticated by trusted IDPs. These IDPs may be deployed 
 within the enterprise or third-party IDPs that are external to the enterprise.
 These needs speak to a specific pain point: which is a narrow integration 
 path into the enterprise identity infrastructure. Kerberos is a fine solution 
 for those that already have it in place or are willing to adopt its use but 
 there remains a class of user that finds this unacceptable and needs to 
 integrate with a wider variety of identity management solutions.
 Another specific pain point is that of rolling and distributing keys. A 
 related and integral part of the HSSO server is library called the Credential 
 Management Framework (CMF), which will be a common library for easing the 
 management of secrets, keys and credentials.
 Initially, the existing delegation, block access and job tokens will continue 
 to be utilized. There may be some changes required to leverage a PKI based 
 signature facility rather than shared secrets. This is a means to simplify 
 the solution for the pain point of distributing shared secrets.
 This project will primarily centralize the responsibility of authentication 
 and federation into a single service that is trusted across the Hadoop 
 cluster and optionally across multiple clusters. This greatly simplifies a 
 number of things in the Hadoop ecosystem:
 1.a single token format that is used across all of Hadoop regardless of 
 authentication method
 2.a single service to have pluggable providers instead of all services
 3.a single token authority that would be trusted across the cluster/s and 
 through PKI encryption be able to easily issue cryptographically verifiable 
 tokens
 4.automatic rolling of the token authority’s keys and publishing of the 
 public key for easy access by those parties that need to verify incoming 
 tokens
 5.use of PKI for signatures eliminates the need for securely sharing and 
 distributing shared secrets
 In addition to serving as the internal Hadoop SSO service this service will 
 be leveraged by the Knox Gateway from the cluster perimeter in order to 
 acquire the Hadoop cluster tokens. The same token mechanism that is used for 
 internal services will be used to represent user identities. Providing for 
 interesting scenarios such as SSO across Hadoop clusters within an enterprise 
 and/or into the cloud.
 The HSSO service will be comprised of three major components and capabilities:
 1.Federating IDP – authenticates users/services and issues the common 
 Hadoop token
 2.Federating SP – validates the token of trusted external IDPs and issues 
 the common Hadoop token
 3.Token Authority – management of the common Hadoop tokens – including: 
 a.Issuance 
 b.Renewal
 c.Revocation
 As this is a meta Jira for tracking this overall effort, the details of the 
 individual efforts will be submitted along with the child Jira filings.
 Hadoop-Common would seem to be the most appropriate home for such a service 
 and its related common facilities. We will also leverage and extend existing 
 common mechanisms as appropriate.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-9523) Provide a generic IBM java vendor flag in PlatformName.java to support non-Sun JREs


[ 
https://issues.apache.org/jira/browse/HADOOP-9523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13657258#comment-13657258
 ] 

Kihwal Lee commented on HADOOP-9523:


This is an incompatible change. For example, hbase startup script can have an 
issue.

{noformat}
JAVA_PLATFORM=`CLASSPATH=${CLASSPATH} ${JAVA} 
org.apache.hadoop.util.PlatformName | sed -e s/ /_/g`
{noformat}

Instead of changing the output format, we could have added something like -a 
for extended output.

 Provide a generic IBM java vendor flag in PlatformName.java to support 
 non-Sun JREs
 ---

 Key: HADOOP-9523
 URL: https://issues.apache.org/jira/browse/HADOOP-9523
 Project: Hadoop Common
  Issue Type: Improvement
Affects Versions: 2.0.4-alpha
Reporter: Tian Hong Wang
Assignee: Tian Hong Wang
  Labels: patch
 Fix For: 2.0.5-beta

 Attachments: HADOOP-9523.patch, HADOOP-9523-v1.patch, 
 HADOOP-9523-v2.patch


 There are several different points between Sun jdk  IBM jdk, so there is a 
 need to provide a generic IBM java vendor flag. So enhance PlatformName.java 
 to add Java vendor information.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HADOOP-9563) Fix incompatibility introduced by HADOOP-9523

Kihwal Lee created HADOOP-9563:
--

 Summary: Fix incompatibility introduced by HADOOP-9523
 Key: HADOOP-9563
 URL: https://issues.apache.org/jira/browse/HADOOP-9563
 Project: Hadoop Common
  Issue Type: Bug
  Components: util
Reporter: Kihwal Lee


HADOOP-9523 changed the output format of the platform name util, so hbase won't 
start unless platform name is specifically set.

We could add a command line option to output in the new extended format and 
keep the old format when no option is given.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HADOOP-9563) Fix incompatibility introduced by HADOOP-9523


 [ 
https://issues.apache.org/jira/browse/HADOOP-9563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kihwal Lee updated HADOOP-9563:
---

Affects Version/s: 2.0.5-beta
   3.0.0

 Fix incompatibility introduced by HADOOP-9523
 -

 Key: HADOOP-9563
 URL: https://issues.apache.org/jira/browse/HADOOP-9563
 Project: Hadoop Common
  Issue Type: Bug
  Components: util
Affects Versions: 3.0.0, 2.0.5-beta
Reporter: Kihwal Lee

 HADOOP-9523 changed the output format of the platform name util, so hbase 
 won't start unless platform name is specifically set.
 We could add a command line option to output in the new extended format and 
 keep the old format when no option is given.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-9562) Create REST interface for HDFS health data

2013-05-14 Thread Steve Loughran (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-9562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13657276#comment-13657276
 ] 

Steve Loughran commented on HADOOP-9562:


This would be good.
# must include DFS safe mode status
# assuming federation is supported, what are the implications?
# what about HFDS HA? Allow all NNs to do an enum of all NNs  status?
# are there any security implications? If so, access should be restricted. 
# I'd like to have error codes in the pages, so I can do a {{curl -m 60 
http://namenode:50020/service/dfslive.jsp}} and expect an error code if dfs is 
not up and out out of safemode. The use case would be : let bash scripts (inc 
Linux HA scripts) verify FS is live just by looking at the return code of curl 
or {{GET}}.


 Create REST interface for HDFS health data
 --

 Key: HADOOP-9562
 URL: https://issues.apache.org/jira/browse/HADOOP-9562
 Project: Hadoop Common
  Issue Type: New Feature
  Components: fs
Affects Versions: 2.0.4-alpha
Reporter: Trevor Lorimer
Priority: Minor

 The HDFS health screen (dfshealth.jsp) displays basic Version, Security and 
 Health information concerning the NameNode, currently this information is 
 accessible from classes in the org.apache.hadoop,hdfs.server.namenode package 
 and cannot be accessed outside the NameNode. This becomes prevalent if the 
 data is required to be displayed using a new user interface.
 The proposal is to create a REST interface to expose all the information 
 displayed on dfshealth.jsp using GET methods. Wrapper classes will be created 
 to serve the data to the REST root resource within the hadoop-hdfs project.
 This will enable the HDFS health screen information to be accessed remotely.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-9523) Provide a generic IBM java vendor flag in PlatformName.java to support non-Sun JREs


[ 
https://issues.apache.org/jira/browse/HADOOP-9523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13657282#comment-13657282
 ] 

Suresh Srinivas commented on HADOOP-9523:
-

[~kihwal] The class does not have any InterfaceAudience annotation. That means 
this it is private and there should not be any expectation of compatibility. If 
classes get used in the downstream projects, it would be good to annotate 
classes.

 Provide a generic IBM java vendor flag in PlatformName.java to support 
 non-Sun JREs
 ---

 Key: HADOOP-9523
 URL: https://issues.apache.org/jira/browse/HADOOP-9523
 Project: Hadoop Common
  Issue Type: Improvement
Affects Versions: 2.0.4-alpha
Reporter: Tian Hong Wang
Assignee: Tian Hong Wang
  Labels: patch
 Fix For: 2.0.5-beta

 Attachments: HADOOP-9523.patch, HADOOP-9523-v1.patch, 
 HADOOP-9523-v2.patch


 There are several different points between Sun jdk  IBM jdk, so there is a 
 need to provide a generic IBM java vendor flag. So enhance PlatformName.java 
 to add Java vendor information.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-9523) Provide a generic IBM java vendor flag in PlatformName.java to support non-Sun JREs


[ 
https://issues.apache.org/jira/browse/HADOOP-9523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13657309#comment-13657309
 ] 

Kihwal Lee commented on HADOOP-9523:


That's clear for cases where classes are used programmatically. In this case, 
however, it was a command line interface that caused the issue. I am not sure 
how we can effectively mark and inform users of the stability of command line 
interfaces. I think users expect command line interfaces to be more stable or 
backward compatible than API.

 Provide a generic IBM java vendor flag in PlatformName.java to support 
 non-Sun JREs
 ---

 Key: HADOOP-9523
 URL: https://issues.apache.org/jira/browse/HADOOP-9523
 Project: Hadoop Common
  Issue Type: Improvement
Affects Versions: 2.0.4-alpha
Reporter: Tian Hong Wang
Assignee: Tian Hong Wang
  Labels: patch
 Fix For: 2.0.5-beta

 Attachments: HADOOP-9523.patch, HADOOP-9523-v1.patch, 
 HADOOP-9523-v2.patch


 There are several different points between Sun jdk  IBM jdk, so there is a 
 need to provide a generic IBM java vendor flag. So enhance PlatformName.java 
 to add Java vendor information.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-9439) JniBasedUnixGroupsMapping: fix some crash bugs


[ 
https://issues.apache.org/jira/browse/HADOOP-9439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13657348#comment-13657348
 ] 

Todd Lipcon commented on HADOOP-9439:
-

{code}
+  static String empty[] = new String[0];
{code}

Should probably be static final, right? And rename to {{NO_GROUPS}} or 
{{EMPTY_STIRNG_ARRAY}} or something a little more descriptive



{code}
+  native static String[] getGroupsForUser(String username, String empty[],
+boolean reentrant);
{code}

I don't get why {{empty}} is a parameter here... aside from it being a 
premature optimization (is allocating an empty array that bad?), it seems like 
the JNI code could just grab this field itself and return a reference, rather 
than taking it as a parameter.


{code}
+  groups = getGroupsForUser(user, empty, useReentrant );
{code}
Style nit: extra space here before ')'


{code}
+  snprintf(buf, sizeof(buf), getgrouplist error %d, ret);
+  THROW(env, java/lang/RuntimeException, buf);
{code}

Can we use strerror here on the return, assuming it's probably a standard 
errnum?


{code}
+ret = hadoop_group_info_fetch(ginfo, uinfo-gids[i]);
+if (!ret) {
{code}
Here if the group info lookup fails for a particular gid, you end up swallowing 
the error without any warnings, etc. Maybe instead we should just return the 
numeric gid as a group? This is what the 'groups' shell command does:

{code}
todd@todd-w510:~$ groups todd
todd : groups: cannot find name for group ID 1050
1050 adm dialout cdrom audio video plugdev lpadmin admin sambashare mrtest 
wireshark
{code}


- I noticed you use different locks for getgrnam and getpwnam. But when I was 
working on HADOOP-7156, I found that the buggy implementations were racey in 
other parts of their code -- ie a concurrent getgrnam and a concurrent getpwnam 
might race with eachother and cause either to fail. I know that you added this 
function to deal with a crash we saw on a production cluster - did you verify 
that with the same buggy underlying implementation that the workaround prevents 
the issue?

We may have to actually share this lock and configuration with the lock used in 
NativeIO.c.

Additionally, when I did that patch, I found it simpler to continue to use the 
reentrant functions wrapped with a lock -- less repeated code, etc. 

 JniBasedUnixGroupsMapping: fix some crash bugs
 --

 Key: HADOOP-9439
 URL: https://issues.apache.org/jira/browse/HADOOP-9439
 Project: Hadoop Common
  Issue Type: Bug
  Components: native
Affects Versions: 2.0.4-alpha
Reporter: Colin Patrick McCabe
Assignee: Colin Patrick McCabe
Priority: Minor
 Attachments: HADOOP-9439.001.patch, HDFS-4640.002.patch


 JniBasedUnixGroupsMapping has some issues.
 * sometimes on error paths variables are freed prior to being initialized
 * re-allocate buffers less frequently (can reuse the same buffer for multiple 
 calls to getgrnam)
 * allow non-reentrant functions to be used, to work around client bugs
 * don't throw IOException from JNI functions if the JNI functions do not 
 declare this checked exception.
 * don't bail out if only one group name among all the ones associated with a 
 user can't be looked up.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-9563) Fix incompatibility introduced by HADOOP-9523


[ 
https://issues.apache.org/jira/browse/HADOOP-9563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13657397#comment-13657397
 ] 

Kihwal Lee commented on HADOOP-9563:


I am not sure about the effectiveness of annotation since it was command line 
interface invoked from the start-up script.

 Fix incompatibility introduced by HADOOP-9523
 -

 Key: HADOOP-9563
 URL: https://issues.apache.org/jira/browse/HADOOP-9563
 Project: Hadoop Common
  Issue Type: Bug
  Components: util
Affects Versions: 3.0.0, 2.0.5-beta
Reporter: Kihwal Lee

 HADOOP-9523 changed the output format of the platform name util, so hbase 
 won't start unless platform name is specifically set.
 We could add a command line option to output in the new extended format and 
 keep the old format when no option is given.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HADOOP-9523) Provide a generic IBM java vendor flag in PlatformName.java to support non-Sun JREs


 [ 
https://issues.apache.org/jira/browse/HADOOP-9523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Todd Lipcon updated HADOOP-9523:


Hadoop Flags: Incompatible change,Reviewed  (was: Reviewed)

Adding Incompatible Change for now, so folks are aware of it.

 Provide a generic IBM java vendor flag in PlatformName.java to support 
 non-Sun JREs
 ---

 Key: HADOOP-9523
 URL: https://issues.apache.org/jira/browse/HADOOP-9523
 Project: Hadoop Common
  Issue Type: Improvement
Affects Versions: 2.0.4-alpha
Reporter: Tian Hong Wang
Assignee: Tian Hong Wang
  Labels: patch
 Fix For: 2.0.5-beta

 Attachments: HADOOP-9523.patch, HADOOP-9523-v1.patch, 
 HADOOP-9523-v2.patch


 There are several different points between Sun jdk  IBM jdk, so there is a 
 need to provide a generic IBM java vendor flag. So enhance PlatformName.java 
 to add Java vendor information.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-9346) Upgrading to protoc 2.5.0 fails the build

2013-05-14 Thread Ravi Prakash (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-9346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13657411#comment-13657411
 ] 

Ravi Prakash commented on HADOOP-9346:
--

Hi Harsh. Thanks. This is a thorn in my side as well.

The code builds and I tried wordcount on Fedora 19 + protobuf-2.5.

However on Fedora 19 + protobuf 2.4 even though the build succeeds with the 
patch, the DN doesn't come up:

2013-05-14 14:26:30,916 [DataNode: 
[file:/tmp/hadoop-raviprak/hdfs/data1,file:/tmp/hadoop-raviprak/hdfs/data2]  
heartbeating to NAMENODE-HOSTNAME/NAMENODE-IP:9000] WARN 
org.apache.hadoop.hdfs.server.datanode.DataNode: Unexpected exception in block 
pool Block pool registering (storage id unknown) service to 
NAMENODE-HOSTNAME/NAMENODE-IP:9000
java.lang.UnsupportedOperationException: This is supposed to be overridden by 
subclasses.
at 
com.google.protobuf.GeneratedMessage.getUnknownFields(GeneratedMessage.java:180)
at 
org.apache.hadoop.hdfs.protocol.proto.HdfsProtos$VersionRequestProto.getSerializedSize(HdfsProtos.java:20296)
at 
com.google.protobuf.AbstractMessageLite.toByteString(AbstractMessageLite.java:49)
at 
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.constructRpcRequest(ProtobufRpcEngine.java:149)
at 
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:193)
at com.sun.proxy.$Proxy9.versionRequest(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at 
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:164)
at 
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:83)
at com.sun.proxy.$Proxy9.versionRequest(Unknown Source)
at 
org.apache.hadoop.hdfs.protocolPB.DatanodeProtocolClientSideTranslatorPB.versionRequest(DatanodeProtocolClientSideTranslatorPB.java:245)
at 
org.apache.hadoop.hdfs.server.datanode.BPServiceActor.retrieveNamespaceInfo(BPServiceActor.java:163)
at 
org.apache.hadoop.hdfs.server.datanode.BPServiceActor.connectToNNAndHandshake(BPServiceActor.java:217)
at 
org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:664)
at java.lang.Thread.run(Thread.java:722)

Were you able to get the daemon up? Is this something wrong with my 
environment? Without the patch the daemon came up and I was able to run a 
wordcount


 Upgrading to protoc 2.5.0 fails the build
 -

 Key: HADOOP-9346
 URL: https://issues.apache.org/jira/browse/HADOOP-9346
 Project: Hadoop Common
  Issue Type: Task
  Components: build
Affects Versions: 3.0.0
Reporter: Harsh J
Assignee: Harsh J
Priority: Minor
  Labels: protobuf
 Attachments: HADOOP-9346.patch


 Reported over the impala lists, one of the errors received is:
 {code}
 src/hadoop-common-project/hadoop-common/target/generated-sources/java/org/apache/hadoop/ha/proto/ZKFCProtocolProtos.java:[104,37]
  can not find symbol.
 symbol： class Parser
 location： package com.google.protobuf
 {code}
 Worth looking into as we'll eventually someday bump our protobuf deps.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-9562) Create REST interface for HDFS health data

2013-05-14 Thread Chris Nauroth (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-9562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13657413#comment-13657413
 ] 

Chris Nauroth commented on HADOOP-9562:
---

{quote}
I'd like to have error codes in the pages, so I can do a curl -m 60 
http://namenode:50020/service/dfslive.jsp and expect an error code if dfs is 
not up and out out of safemode. The use case would be : let bash scripts (inc 
Linux HA scripts) verify FS is live just by looking at the return code of curl 
or GET.
{quote}

One difficulty is that we don't start the HTTP server until relatively late in 
initialization.  If you have a ton of edits, then you'll suffer through a long 
checkpoint before status is visible, either in the web page or a REST 
interface.  I have patches queued up in HDFS-4249 and its sub-tasks to start 
the HTTP server sooner and display more details about the early phases of 
namenode startup.


 Create REST interface for HDFS health data
 --

 Key: HADOOP-9562
 URL: https://issues.apache.org/jira/browse/HADOOP-9562
 Project: Hadoop Common
  Issue Type: New Feature
  Components: fs
Affects Versions: 2.0.4-alpha
Reporter: Trevor Lorimer
Priority: Minor

 The HDFS health screen (dfshealth.jsp) displays basic Version, Security and 
 Health information concerning the NameNode, currently this information is 
 accessible from classes in the org.apache.hadoop,hdfs.server.namenode package 
 and cannot be accessed outside the NameNode. This becomes prevalent if the 
 data is required to be displayed using a new user interface.
 The proposal is to create a REST interface to expose all the information 
 displayed on dfshealth.jsp using GET methods. Wrapper classes will be created 
 to serve the data to the REST root resource within the hadoop-hdfs project.
 This will enable the HDFS health screen information to be accessed remotely.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-9523) Provide a generic IBM java vendor flag in PlatformName.java to support non-Sun JREs


[ 
https://issues.apache.org/jira/browse/HADOOP-9523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13657424#comment-13657424
 ] 

Suresh Srinivas commented on HADOOP-9523:
-

bq.  In this case, however, it was a command line interface...
It is the output of the main right? Irrespective of programmatically or command 
line, we need to make sure class is backward compatible. That means method 
return values or output cannot change.

bq. Adding Incompatible Change for now, so folks are aware of it.
I actually think this is not necessary unless it has appropriate 
InterfaceAudience. The whole idea of such classification was to say, if some 
thing changes in such classes and if you use it, be ready to change.

 Provide a generic IBM java vendor flag in PlatformName.java to support 
 non-Sun JREs
 ---

 Key: HADOOP-9523
 URL: https://issues.apache.org/jira/browse/HADOOP-9523
 Project: Hadoop Common
  Issue Type: Improvement
Affects Versions: 2.0.4-alpha
Reporter: Tian Hong Wang
Assignee: Tian Hong Wang
  Labels: patch
 Fix For: 2.0.5-beta

 Attachments: HADOOP-9523.patch, HADOOP-9523-v1.patch, 
 HADOOP-9523-v2.patch


 There are several different points between Sun jdk  IBM jdk, so there is a 
 need to provide a generic IBM java vendor flag. So enhance PlatformName.java 
 to add Java vendor information.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HADOOP-9564) DFSClient$DFSOutputStream.closeInternal locks up waiting for namenode.complete

2013-05-14 Thread Jin Feng (JIRA)

Jin Feng created HADOOP-9564:


 Summary: DFSClient$DFSOutputStream.closeInternal locks up waiting 
for namenode.complete
 Key: HADOOP-9564
 URL: https://issues.apache.org/jira/browse/HADOOP-9564
 Project: Hadoop Common
  Issue Type: Bug
  Components: fs
Reporter: Jin Feng
Priority: Minor


Hi,

Our component uses FileSystem.copyFromLocalFile to copy a local file to HDFS 
cluster. It's working fine in production environment. Its integration tests 
used to run fine on our dev's local Mac laptop until recently (exact point of 
time unknown) our tests started to freeze up very frequently with this stack:

{code}
com.twitter.ads.billing.spendaggregator.jobs.maintenance.billinghourlyarchive.BillingHourlyAggArchiveJobIT-runner
 prio=5 tid=0x7f9ce52cb800 nid=0x4a503 waiting on condition 
[0x000165815000]
   java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  0x000152f41378 (a 
java.util.concurrent.FutureTask$Sync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:994)
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1303)
at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:248)
at java.util.concurrent.FutureTask.get(FutureTask.java:111)
at org.apache.hadoop.ipc.Client$Connection.sendParam(Client.java:790)
- locked 0x00014f568720 (a java.lang.Object)
at org.apache.hadoop.ipc.Client.call(Client.java:1080)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:226)
at $Proxy37.complete(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at 
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
at 
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
at $Proxy37.complete(Unknown Source)
at 
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.closeInternal(DFSClient.java:3566)
- locked 0x000152f3f658 (a 
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream)
at 
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.close(DFSClient.java:3481)
at 
org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:61)
at 
org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:86)
at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:59)
at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:89)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:224)
at 
org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:1295)


{code}

our version is 0.20.2.cdh3u2-t1.

In the test suite, we use org.apache.hadoop.hdfs.MiniDFSCluster. I've searched 
around couldn't find anything resembles this symptom, any helps are really 
appreciated!


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-8562) Enhancements to support Hadoop on Windows Server and Windows Azure environments


[ 
https://issues.apache.org/jira/browse/HADOOP-8562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13657695#comment-13657695
 ] 

Suresh Srinivas commented on HADOOP-8562:
-

The changes from this jira and related jiras from HDFS and YARN has been in 
trunk for some time now. If there are no objections, I plan on merging these 
changes to branch-2 early next week.

 Enhancements to support Hadoop on Windows Server and Windows Azure 
 environments
 ---

 Key: HADOOP-8562
 URL: https://issues.apache.org/jira/browse/HADOOP-8562
 Project: Hadoop Common
  Issue Type: New Feature
Affects Versions: 3.0.0
Reporter: Bikas Saha
Assignee: Bikas Saha
 Fix For: 3.0.0

 Attachments: branch-trunk-win.min-notest.patch, 
 branch-trunk-win-min.patch, branch-trunk-win.min.patch, 
 branch-trunk-win.patch, branch-trunk-win.patch, branch-trunk-win.patch, 
 branch-trunk-win.patch, branch-trunk-win.patch, branch-trunk-win.patch, 
 branch-trunk-win.patch, branch-trunk-win.patch, branch-trunk-win.patch, 
 branch-trunk-win.patch, branch-trunk-win.patch, test-untar.tar, test-untar.tgz


 This JIRA tracks the work that needs to be done on trunk to enable Hadoop to 
 run on Windows Server and Azure environments. This incorporates porting 
 relevant work from the similar effort on branch 1 tracked via HADOOP-8079.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HADOOP-9564) DFSClient$DFSOutputStream.closeInternal locks up waiting for namenode.complete

2013-05-14 Thread Jin Feng (JIRA)


 [ 
https://issues.apache.org/jira/browse/HADOOP-9564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jin Feng updated HADOOP-9564:
-

Description: 
Hi,

Our component uses FileSystem.copyFromLocalFile to copy a local file to HDFS 
cluster. It's working fine in production environment. Its integration tests 
used to run fine on our dev's local Mac laptop until recently (exact point of 
time unknown) our tests started to freeze up very frequently with this stack:

{code}
   java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  0x000152f41378 (a 
java.util.concurrent.FutureTask$Sync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:994)
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1303)
at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:248)
at java.util.concurrent.FutureTask.get(FutureTask.java:111)
at org.apache.hadoop.ipc.Client$Connection.sendParam(Client.java:790)
- locked 0x00014f568720 (a java.lang.Object)
at org.apache.hadoop.ipc.Client.call(Client.java:1080)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:226)
at $Proxy37.complete(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at 
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
at 
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
at $Proxy37.complete(Unknown Source)
at 
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.closeInternal(DFSClient.java:3566)
- locked 0x000152f3f658 (a 
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream)
at 
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.close(DFSClient.java:3481)
at 
org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:61)
at 
org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:86)
at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:59)
at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:89)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:224)
at 
org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:1295)


{code}

our version is 0.20.2.cdh3u2-t1.

In the test suite, we use org.apache.hadoop.hdfs.MiniDFSCluster. I've searched 
around couldn't find anything resembles this symptom, any helps are really 
appreciated!


  was:
Hi,

Our component uses FileSystem.copyFromLocalFile to copy a local file to HDFS 
cluster. It's working fine in production environment. Its integration tests 
used to run fine on our dev's local Mac laptop until recently (exact point of 
time unknown) our tests started to freeze up very frequently with this stack:

{code}
com.twitter.ads.billing.spendaggregator.jobs.maintenance.billinghourlyarchive.BillingHourlyAggArchiveJobIT-runner
 prio=5 tid=0x7f9ce52cb800 nid=0x4a503 waiting on condition 
[0x000165815000]
   java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  0x000152f41378 (a 
java.util.concurrent.FutureTask$Sync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:994)
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1303)
at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:248)
at java.util.concurrent.FutureTask.get(FutureTask.java:111)
at org.apache.hadoop.ipc.Client$Connection.sendParam(Client.java:790)
- locked 0x00014f568720 (a java.lang.Object)
at org.apache.hadoop.ipc.Client.call(Client.java:1080)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:226)
at $Proxy37.complete(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at

[jira] [Commented] (HADOOP-9564) DFSClient$DFSOutputStream.closeInternal locks up waiting for namenode.complete

2013-05-14 Thread Jin Feng (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-9564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13657707#comment-13657707
 ] 

Jin Feng commented on HADOOP-9564:
--

This is captured from our test output, seems like the DataBlockScanner is 
really slow coming up with verification results even though our data/file 
generated in the tests are minimal.

{noformat}
13/05/04 06:50:55 INFO hdfs.StateChange: BLOCK* NameSystem.allocateBlock: 
our_test_file_name.lzo-f773a37f-3dac-4337-a6cb-004fb94c1d31. 
blk_-8485988660073681466_1002
13/05/04 06:50:55 INFO datanode.DataNode: Receiving block 
blk_-8485988660073681466_1002 src: /127.0.0.1:42563 dest: /127.0.0.1:35830
13/05/04 06:50:55 INFO DataNode.clienttrace: src: /127.0.0.1:42563, dest: 
/127.0.0.1:35830, bytes: 303, op: HDFS_WRITE, cliID: DFSClient_-854844208, 
offset: 0, srvID: DS-1070312150-10.35.8.106-35830-1367650255272, blockid: 
blk_-8485988660073681466_1002, duration: 778000
13/05/04 06:50:55 INFO datanode.DataNode: PacketResponder 0 for block 
blk_-8485988660073681466_1002 terminating
13/05/04 06:50:55 INFO hdfs.StateChange: BLOCK* NameSystem.addStoredBlock: 
blockMap updated: 127.0.0.1:35830 is added to blk_-8485988660073681466_1002 
size 303
13/05/04 06:52:48 INFO datanode.DataBlockScanner: Verification succeeded for 
blk_-8485988660073681466_1002
13/05/04 07:00:40 INFO datanode.DataBlockScanner: Verification succeeded for 
blk_586310994067086116_1001
{noformat}


Could this be related to this bug: HADOOP-4584?

 DFSClient$DFSOutputStream.closeInternal locks up waiting for namenode.complete
 --

 Key: HADOOP-9564
 URL: https://issues.apache.org/jira/browse/HADOOP-9564
 Project: Hadoop Common
  Issue Type: Bug
  Components: fs
Reporter: Jin Feng
Priority: Minor

 Hi,
 Our component uses FileSystem.copyFromLocalFile to copy a local file to HDFS 
 cluster. It's working fine in production environment. Its integration tests 
 used to run fine on our dev's local Mac laptop until recently (exact point of 
 time unknown) our tests started to freeze up very frequently with this stack:
 {code}
java.lang.Thread.State: WAITING (parking)
   at sun.misc.Unsafe.park(Native Method)
   - parking to wait for  0x000152f41378 (a 
 java.util.concurrent.FutureTask$Sync)
   at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
   at 
 java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
   at 
 java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:994)
   at 
 java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1303)
   at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:248)
   at java.util.concurrent.FutureTask.get(FutureTask.java:111)
   at org.apache.hadoop.ipc.Client$Connection.sendParam(Client.java:790)
   - locked 0x00014f568720 (a java.lang.Object)
   at org.apache.hadoop.ipc.Client.call(Client.java:1080)
   at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:226)
   at $Proxy37.complete(Unknown Source)
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   at 
 sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
   at 
 sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   at java.lang.reflect.Method.invoke(Method.java:601)
   at 
 org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
   at 
 org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
   at $Proxy37.complete(Unknown Source)
   at 
 org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.closeInternal(DFSClient.java:3566)
   - locked 0x000152f3f658 (a 
 org.apache.hadoop.hdfs.DFSClient$DFSOutputStream)
   at 
 org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.close(DFSClient.java:3481)
   at 
 org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:61)
   at 
 org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:86)
   at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:59)
   at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:89)
   at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:224)
   at 
 org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:1295)
 
 
 {code}
 our version is 0.20.2.cdh3u2-t1.
 In the test suite, we use org.apache.hadoop.hdfs.MiniDFSCluster. I've 
 searched around couldn't find anything resembles this symptom, any helps are 
 really appreciated!

--
This message is

[jira] [Commented] (HADOOP-9523) Provide a generic IBM java vendor flag in PlatformName.java to support non-Sun JREs


[ 
https://issues.apache.org/jira/browse/HADOOP-9523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13657815#comment-13657815
 ] 

Tian Hong Wang commented on HADOOP-9523:


Suresh, could you please undo this patch for it has caused some incompatible 
change?

 Provide a generic IBM java vendor flag in PlatformName.java to support 
 non-Sun JREs
 ---

 Key: HADOOP-9523
 URL: https://issues.apache.org/jira/browse/HADOOP-9523
 Project: Hadoop Common
  Issue Type: Improvement
Affects Versions: 2.0.4-alpha
Reporter: Tian Hong Wang
Assignee: Tian Hong Wang
  Labels: patch
 Fix For: 2.0.5-beta

 Attachments: HADOOP-9523.patch, HADOOP-9523-v1.patch, 
 HADOOP-9523-v2.patch


 There are several different points between Sun jdk  IBM jdk, so there is a 
 need to provide a generic IBM java vendor flag. So enhance PlatformName.java 
 to add Java vendor information.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-9564) DFSClient$DFSOutputStream.closeInternal locks up waiting for namenode.complete


[ 
https://issues.apache.org/jira/browse/HADOOP-9564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13657832#comment-13657832
 ] 

Todd Lipcon commented on HADOOP-9564:
-

What's the thread doing on the NN side? Keep in mind you're on a very old 
version (and a twitter-local build that no one else has), so you may not have a 
lot of luck getting people to help you diagnose farther.

 DFSClient$DFSOutputStream.closeInternal locks up waiting for namenode.complete
 --

 Key: HADOOP-9564
 URL: https://issues.apache.org/jira/browse/HADOOP-9564
 Project: Hadoop Common
  Issue Type: Bug
  Components: fs
Reporter: Jin Feng
Priority: Minor

 Hi,
 Our component uses FileSystem.copyFromLocalFile to copy a local file to HDFS 
 cluster. It's working fine in production environment. Its integration tests 
 used to run fine on our dev's local Mac laptop until recently (exact point of 
 time unknown) our tests started to freeze up very frequently with this stack:
 {code}
java.lang.Thread.State: WAITING (parking)
   at sun.misc.Unsafe.park(Native Method)
   - parking to wait for  0x000152f41378 (a 
 java.util.concurrent.FutureTask$Sync)
   at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
   at 
 java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
   at 
 java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:994)
   at 
 java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1303)
   at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:248)
   at java.util.concurrent.FutureTask.get(FutureTask.java:111)
   at org.apache.hadoop.ipc.Client$Connection.sendParam(Client.java:790)
   - locked 0x00014f568720 (a java.lang.Object)
   at org.apache.hadoop.ipc.Client.call(Client.java:1080)
   at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:226)
   at $Proxy37.complete(Unknown Source)
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   at 
 sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
   at 
 sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   at java.lang.reflect.Method.invoke(Method.java:601)
   at 
 org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
   at 
 org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
   at $Proxy37.complete(Unknown Source)
   at 
 org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.closeInternal(DFSClient.java:3566)
   - locked 0x000152f3f658 (a 
 org.apache.hadoop.hdfs.DFSClient$DFSOutputStream)
   at 
 org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.close(DFSClient.java:3481)
   at 
 org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:61)
   at 
 org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:86)
   at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:59)
   at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:89)
   at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:224)
   at 
 org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:1295)
 
 
 {code}
 our version is 0.20.2.cdh3u2-t1.
 In the test suite, we use org.apache.hadoop.hdfs.MiniDFSCluster. I've 
 searched around couldn't find anything resembles this symptom, any helps are 
 really appreciated!

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-9346) Upgrading to protoc 2.5.0 fails the build

2013-05-14 Thread Harsh J (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-9346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13657834#comment-13657834
 ] 

Harsh J commented on HADOOP-9346:
-

Thanks Ravi!

No I did not actually build a package and run the daemons. The error does look 
protobuf related. I'll investigate!

 Upgrading to protoc 2.5.0 fails the build
 -

 Key: HADOOP-9346
 URL: https://issues.apache.org/jira/browse/HADOOP-9346
 Project: Hadoop Common
  Issue Type: Task
  Components: build
Affects Versions: 3.0.0
Reporter: Harsh J
Assignee: Harsh J
Priority: Minor
  Labels: protobuf
 Attachments: HADOOP-9346.patch


 Reported over the impala lists, one of the errors received is:
 {code}
 src/hadoop-common-project/hadoop-common/target/generated-sources/java/org/apache/hadoop/ha/proto/ZKFCProtocolProtos.java:[104,37]
  can not find symbol.
 symbol： class Parser
 location： package com.google.protobuf
 {code}
 Worth looking into as we'll eventually someday bump our protobuf deps.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HADOOP-9346) Upgrading to protoc 2.5.0 fails the build

2013-05-14 Thread Harsh J (JIRA)


 [ 
https://issues.apache.org/jira/browse/HADOOP-9346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Harsh J updated HADOOP-9346:


Status: Open  (was: Patch Available)

Cancelling patch while we address Ravi's find (and any other such subtle issues 
that tests are seemingly not capturing)

 Upgrading to protoc 2.5.0 fails the build
 -

 Key: HADOOP-9346
 URL: https://issues.apache.org/jira/browse/HADOOP-9346
 Project: Hadoop Common
  Issue Type: Task
  Components: build
Affects Versions: 3.0.0
Reporter: Harsh J
Assignee: Harsh J
Priority: Minor
  Labels: protobuf
 Attachments: HADOOP-9346.patch


 Reported over the impala lists, one of the errors received is:
 {code}
 src/hadoop-common-project/hadoop-common/target/generated-sources/java/org/apache/hadoop/ha/proto/ZKFCProtocolProtos.java:[104,37]
  can not find symbol.
 symbol： class Parser
 location： package com.google.protobuf
 {code}
 Worth looking into as we'll eventually someday bump our protobuf deps.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-9392) Token based authentication and Single Sign On

2013-05-14 Thread Thomas NGUY (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-9392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13658067#comment-13658067
 ] 

Thomas NGUY commented on HADOOP-9392:
-

From my point of view, we have two different designs to SSO in Hadoop but not 
necessary incompatible.
Concerning the « Token » design, in HADOOP-9533, the Service Access Token 
targets a specific ressource (defined by the service URL) and have a low 
lifetime while in HADOOP-9392, the Identity Token can be used for any services? 
(belonging to at trusted tokenrealm) and doesnt have an expiration time.
Both of them carry extended attributed for fined-grained access control 
decisions or for the service itself. 
I'm just curious to learn more about how the Unified Authorization Framework 
(Hadoop-9466) would used to common token to make decisions.  

 Token based authentication and Single Sign On
 -

 Key: HADOOP-9392
 URL: https://issues.apache.org/jira/browse/HADOOP-9392
 Project: Hadoop Common
  Issue Type: New Feature
  Components: security
Reporter: Kai Zheng
Assignee: Kai Zheng
 Fix For: 3.0.0

 Attachments: token-based-authn-plus-sso.pdf


 This is an umbrella entry for one of project Rhino’s topic, for details of 
 project Rhino, please refer to 
 https://github.com/intel-hadoop/project-rhino/. The major goal for this entry 
 as described in project Rhino was 
  
 “Core, HDFS, ZooKeeper, and HBase currently support Kerberos authentication 
 at the RPC layer, via SASL. However this does not provide valuable attributes 
 such as group membership, classification level, organizational identity, or 
 support for user defined attributes. Hadoop components must interrogate 
 external resources for discovering these attributes and at scale this is 
 problematic. There is also no consistent delegation model. HDFS has a simple 
 delegation capability, and only Oozie can take limited advantage of it. We 
 will implement a common token based authentication framework to decouple 
 internal user and service authentication from external mechanisms used to 
 support it (like Kerberos)”
  
 We’d like to start our work from Hadoop-Common and try to provide common 
 facilities by extending existing authentication framework which support:
 1.Pluggable token provider interface 
 2.Pluggable token verification protocol and interface
 3.Security mechanism to distribute secrets in cluster nodes
 4.Delegation model of user authentication

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-9346) Upgrading to protoc 2.5.0 fails the build