date:20121009


 [ 
https://issues.apache.org/jira/browse/HADOOP-8901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Colin Patrick McCabe updated HADOOP-8901:
-

Attachment: HADOOP-8901.003.patch

Multi-platform-ish version.

 GZip and Snappy support may not work without unversioned libraries
 --

 Key: HADOOP-8901
 URL: https://issues.apache.org/jira/browse/HADOOP-8901
 Project: Hadoop Common
  Issue Type: Bug
  Components: native
Affects Versions: 2.0.3-alpha
Reporter: Colin Patrick McCabe
Assignee: Colin Patrick McCabe
Priority: Minor
 Attachments: HADOOP-8901.001.patch, HADOOP-8901.002.patch, 
 HADOOP-8901.003.patch


 Currently, we use {{dlopen}} to open {{libz.so}} and {{libsnappy.so}}, to get 
 Gzip and Snappy support, respectively.
 However, this is not correct; we should be dlopening {{libsnappy.so.1}} 
 instead.  The versionless form of the shared library is not commonly 
 installed except by development packages.  Also, we may run into subtle 
 compatibility problems if a new version of libsnappy comes out.
 Thanks to Brandon Vargo for reporting this bug.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-8901) GZip and Snappy support may not work without unversioned libraries


[ 
https://issues.apache.org/jira/browse/HADOOP-8901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13472191#comment-13472191
 ] 

Hadoop QA commented on HADOOP-8901:
---

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12548366/HADOOP-8901.003.patch
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:red}-1 tests included{color}.  The patch doesn't appear to include 
any new or modified tests.
Please justify why no new tests are needed for this 
patch.
Also please list what manual steps were performed to 
verify this patch.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  The javadoc tool did not generate any 
warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 core tests{color}.  The patch passed unit tests in 
hadoop-common-project/hadoop-common.

{color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: 
https://builds.apache.org/job/PreCommit-HADOOP-Build/1587//testReport/
Console output: 
https://builds.apache.org/job/PreCommit-HADOOP-Build/1587//console

This message is automatically generated.

 GZip and Snappy support may not work without unversioned libraries
 --

 Key: HADOOP-8901
 URL: https://issues.apache.org/jira/browse/HADOOP-8901
 Project: Hadoop Common
  Issue Type: Bug
  Components: native
Affects Versions: 2.0.3-alpha
Reporter: Colin Patrick McCabe
Assignee: Colin Patrick McCabe
Priority: Minor
 Attachments: HADOOP-8901.001.patch, HADOOP-8901.002.patch, 
 HADOOP-8901.003.patch


 Currently, we use {{dlopen}} to open {{libz.so}} and {{libsnappy.so}}, to get 
 Gzip and Snappy support, respectively.
 However, this is not correct; we should be dlopening {{libsnappy.so.1}} 
 instead.  The versionless form of the shared library is not commonly 
 installed except by development packages.  Also, we may run into subtle 
 compatibility problems if a new version of libsnappy comes out.
 Thanks to Brandon Vargo for reporting this bug.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HADOOP-8904) Hadoop does not close output file / does not call Mapper.cleanup if exception in map

2012-10-09 Thread Daniel Dai (JIRA)

Daniel Dai created HADOOP-8904:
--

 Summary:  Hadoop does not close output file / does not call 
Mapper.cleanup if exception in map
 Key: HADOOP-8904
 URL: https://issues.apache.org/jira/browse/HADOOP-8904
 Project: Hadoop Common
  Issue Type: Bug
Affects Versions: 1-win
Reporter: Daniel Dai


Find this in Pig unit test TestStore under Windows. There are dangling files 
because map does not close the file when exception happens in map(). In 
Windows, Hadoop will not remove a file if it is not closed. This happens in 
reduce() as well.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HADOOP-8904) Hadoop does not close output file / does not call Mapper.cleanup if exception in map

2012-10-09 Thread Daniel Dai (JIRA)


 [ 
https://issues.apache.org/jira/browse/HADOOP-8904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Dai updated HADOOP-8904:
---

Attachment: HADOOP-8904-1.patch

  Hadoop does not close output file / does not call Mapper.cleanup if 
 exception in map
 -

 Key: HADOOP-8904
 URL: https://issues.apache.org/jira/browse/HADOOP-8904
 Project: Hadoop Common
  Issue Type: Bug
Affects Versions: 1-win
Reporter: Daniel Dai
 Attachments: HADOOP-8904-1.patch


 Find this in Pig unit test TestStore under Windows. There are dangling files 
 because map does not close the file when exception happens in map(). In 
 Windows, Hadoop will not remove a file if it is not closed. This happens in 
 reduce() as well.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HADOOP-8904) Hadoop does not close output file / does not call Mapper.cleanup if exception in map

2012-10-09 Thread Daniel Dai (JIRA)


 [ 
https://issues.apache.org/jira/browse/HADOOP-8904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Dai updated HADOOP-8904:
---

Status: Patch Available  (was: Open)

  Hadoop does not close output file / does not call Mapper.cleanup if 
 exception in map
 -

 Key: HADOOP-8904
 URL: https://issues.apache.org/jira/browse/HADOOP-8904
 Project: Hadoop Common
  Issue Type: Bug
Affects Versions: 1-win
Reporter: Daniel Dai
 Attachments: HADOOP-8904-1.patch


 Find this in Pig unit test TestStore under Windows. There are dangling files 
 because map does not close the file when exception happens in map(). In 
 Windows, Hadoop will not remove a file if it is not closed. This happens in 
 reduce() as well.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-8904) Hadoop does not close output file / does not call Mapper.cleanup if exception in map


[ 
https://issues.apache.org/jira/browse/HADOOP-8904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13472207#comment-13472207
 ] 

Hadoop QA commented on HADOOP-8904:
---

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12548375/HADOOP-8904-1.patch
  against trunk revision .

{color:red}-1 patch{color}.  The patch command could not apply the patch.

Console output: 
https://builds.apache.org/job/PreCommit-HADOOP-Build/1588//console

This message is automatically generated.

  Hadoop does not close output file / does not call Mapper.cleanup if 
 exception in map
 -

 Key: HADOOP-8904
 URL: https://issues.apache.org/jira/browse/HADOOP-8904
 Project: Hadoop Common
  Issue Type: Bug
Affects Versions: 1-win
Reporter: Daniel Dai
 Attachments: HADOOP-8904-1.patch


 Find this in Pig unit test TestStore under Windows. There are dangling files 
 because map does not close the file when exception happens in map(). In 
 Windows, Hadoop will not remove a file if it is not closed. This happens in 
 reduce() as well.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HADOOP-8891) Remove DelegationTokenRenewer and its logic from WebHdfsFileSystem and HftpFileSystem


 [ 
https://issues.apache.org/jira/browse/HADOOP-8891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Karthik Kambatla updated HADOOP-8891:
-

Attachment: HADOOP-8891.patch

 Remove DelegationTokenRenewer and its logic from WebHdfsFileSystem and 
 HftpFileSystem
 -

 Key: HADOOP-8891
 URL: https://issues.apache.org/jira/browse/HADOOP-8891
 Project: Hadoop Common
  Issue Type: Improvement
Affects Versions: 2.0.1-alpha
Reporter: Karthik Kambatla
Assignee: Karthik Kambatla
 Attachments: HADOOP-8891.patch, HADOOP-8891.patch


 Moved the HDFS part of HADOOP-8852 to HDFS-4009 along with other sub-tasks. 
 Created this to track the removal of DelegationTokenRenewer alone.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HADOOP-8891) Remove DelegationTokenRenewer and its logic from WebHdfsFileSystem and HftpFileSystem


 [ 
https://issues.apache.org/jira/browse/HADOOP-8891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Karthik Kambatla updated HADOOP-8891:
-

Status: Open  (was: Patch Available)

 Remove DelegationTokenRenewer and its logic from WebHdfsFileSystem and 
 HftpFileSystem
 -

 Key: HADOOP-8891
 URL: https://issues.apache.org/jira/browse/HADOOP-8891
 Project: Hadoop Common
  Issue Type: Improvement
Affects Versions: 2.0.1-alpha
Reporter: Karthik Kambatla
Assignee: Karthik Kambatla
 Attachments: HADOOP-8891.patch, HADOOP-8891.patch


 Moved the HDFS part of HADOOP-8852 to HDFS-4009 along with other sub-tasks. 
 Created this to track the removal of DelegationTokenRenewer alone.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-8878) uppercase namenode hostname causes hadoop dfs calls with webhdfs filesystem and fsck to fail when security is on

2012-10-09 Thread Steve Loughran (JIRA)

[
https://issues.apache.org/jira/browse/HADOOP-8878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13472245#comment-13472245
]

Steve Loughran commented on HADOOP-8878:

-1: needs to use {{String.toLowerCase(Locale.EN_US)}} to work reliably round
the world. You don't want to field unreplicable support cases from Turkey.

uppercase namenode hostname causes hadoop dfs calls with webhdfs filesystem
and fsck to fail when security is on

Key: HADOOP-8878
URL: https://issues.apache.org/jira/browse/HADOOP-8878
Project: Hadoop Common
Issue Type: Bug
Affects Versions: 1.0.3, 1.1.0, 1.2.0, 3.0.0
Reporter: Arpit Gupta
Assignee: Arpit Gupta
Attachments: HADOOP-8878.branch-1.patch, HADOOP-8878.branch-1.patch,
HADOOP-8878.patch, HADOOP-8878.patch

This was noticed on a secure cluster where the namenode had an upper case
hostname and the following command was issued
hadoop dfs -ls webhdfs://NN:PORT/PATH
the above command failed because delegation token retrieval failed.
Upon looking at the kerberos logs it was determined that we tried to get the
ticket for kerberos principal with upper case hostnames and that host did not
exit in kerberos. We should convert the hostnames to lower case. Take a look
at HADOOP-7988 where the same fix was applied on a different class.
I have noticed this issue exists on branch-1. Will investigate trunk and
branch-2 and update accordingly.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-8891) Remove DelegationTokenRenewer and its logic from WebHdfsFileSystem and HftpFileSystem


[ 
https://issues.apache.org/jira/browse/HADOOP-8891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13472246#comment-13472246
 ] 

Karthik Kambatla commented on HADOOP-8891:
--

I am a little confused with {{HttpFSFileSystem}}. {{#initialize()}} doesn't 
check for an existing delegation token, nor does it sign up for token renewal. 
However, it still implements {{getDelegationToken()}} and 
{{setDelegationToken()}}, I am not sure where these methods are actually 
invoked from.

Will investigate this further. Meanwhile, I have uploaded a patch that removes 
{{DelegationTokenRenewer}} and updates {{WebHdfsFileSystem}} and 
{{HftpFileSystem}} accordingly. It would really help if someone could take a 
look at the available patch and any thoughts on how to proceed with 
{{HttpFSFileSystem}}. 

Thanks
Karthik

 Remove DelegationTokenRenewer and its logic from WebHdfsFileSystem and 
 HftpFileSystem
 -

 Key: HADOOP-8891
 URL: https://issues.apache.org/jira/browse/HADOOP-8891
 Project: Hadoop Common
  Issue Type: Improvement
Affects Versions: 2.0.1-alpha
Reporter: Karthik Kambatla
Assignee: Karthik Kambatla
 Attachments: HADOOP-8891.patch, HADOOP-8891.patch


 Moved the HDFS part of HADOOP-8852 to HDFS-4009 along with other sub-tasks. 
 Created this to track the removal of DelegationTokenRenewer alone.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-8887) Use a Maven plugin to build the native code using CMake

2012-10-09 Thread Tom White (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-8887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13472272#comment-13472272
 ] 

Tom White commented on HADOOP-8887:
---

This plugin looks general purpose and it's in the org.apache.maven package, so 
perhaps it should go in the Maven Plugins project?

 Use a Maven plugin to build the native code using CMake
 ---

 Key: HADOOP-8887
 URL: https://issues.apache.org/jira/browse/HADOOP-8887
 Project: Hadoop Common
  Issue Type: Improvement
  Components: build
Affects Versions: 2.0.3-alpha
Reporter: Colin Patrick McCabe
Assignee: Colin Patrick McCabe
Priority: Minor
 Attachments: HADOOP-8887.001.patch, HADOOP-8887.002.patch, 
 HADOOP-8887.003.patch, HADOOP-8887.004.patch


 Currently, we build the native code using ant-build invocations.  Although 
 this works, it has some limitations:
 * compiler warning messages are hidden, which can cause people to check in 
 code with warnings unintentionally
 * there is no framework for running native unit tests; instead, we use ad-hoc 
 constructs involving shell scripts
 * the antrun code is very platform specific
 * there is no way to run a specific native unit test
 * it's more or less impossible for scripts like test-patch.sh to separate a 
 native test failing from the build itself failing (no files are created) or 
 to enumerate which native tests failed.
 Using a native Maven plugin would overcome these limitations.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-8886) Remove KFS support


[ 
https://issues.apache.org/jira/browse/HADOOP-8886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13472340#comment-13472340
 ] 

Hudson commented on HADOOP-8886:


Integrated in Hadoop-Hdfs-trunk #1190 (See 
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/1190/])
HADOOP-8886. Remove KFS support. Contributed by Eli Collins (Revision 
1395820)

 Result = SUCCESS
eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1395820
Files : 
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-common/dev-support/findbugsExcludeFile.xml
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/pom.xml
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/kfs
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/resources/META-INF/services/org.apache.hadoop.fs.FileSystem
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/resources/core-default.xml
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/kfs


 Remove KFS support
 --

 Key: HADOOP-8886
 URL: https://issues.apache.org/jira/browse/HADOOP-8886
 Project: Hadoop Common
  Issue Type: Improvement
  Components: fs
Reporter: Eli Collins
Assignee: Eli Collins
 Fix For: 3.0.0

 Attachments: hadoop-8886.txt


 KFS is no longer maintained (is replaced by QFS, which HADOOP-8885 is 
 adding), let's remove it.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-8894) GenericTestUtils.waitFor should dump thread stacks on timeout


[ 
https://issues.apache.org/jira/browse/HADOOP-8894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13472337#comment-13472337
 ] 

Hudson commented on HADOOP-8894:


Integrated in Hadoop-Hdfs-trunk #1190 (See 
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/1190/])
HADOOP-8894. GenericTestUtils.waitFor should dump thread stacks on timeout. 
Contributed by Todd Lipcon. (Revision 1395825)

 Result = SUCCESS
todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1395825
Files : 
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/test/GenericTestUtils.java
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/test/TimedOutTestsListener.java


 GenericTestUtils.waitFor should dump thread stacks on timeout
 -

 Key: HADOOP-8894
 URL: https://issues.apache.org/jira/browse/HADOOP-8894
 Project: Hadoop Common
  Issue Type: Improvement
Reporter: Todd Lipcon
Assignee: Todd Lipcon
 Fix For: 3.0.0, 2.0.3-alpha

 Attachments: hadoop-8894.txt, hadoop-8894.txt


 Many tests use this utility to wait for a condition to become true. In the 
 event that it times out, we should dump all the thread stack traces, in case 
 the timeout was due to a deadlock. This should make it easier to debug 
 scenarios like HDFS-4001.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-8884) DEBUG should be WARN for DEBUG util.NativeCodeLoader: Failed to load native-hadoop with error: java.lang.UnsatisfiedLinkError


[ 
https://issues.apache.org/jira/browse/HADOOP-8884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13472339#comment-13472339
 ] 

Hudson commented on HADOOP-8884:


Integrated in Hadoop-Hdfs-trunk #1190 (See 
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/1190/])
Revert initial commit of HADOOP-8884 per feedback from Todd. (Revision 
1395783)
HADOOP-8884. DEBUG should be WARN for DEBUG util.NativeCodeLoader: Failed to 
load native-hadoop with error: java.lang.UnsatisfiedLinkError. Contributed by 
Anthony Rojas. (Revision 1395714)

 Result = SUCCESS
atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1395783
Files : 
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/NativeCodeLoader.java

atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1395714
Files : 
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/NativeCodeLoader.java


 DEBUG should be WARN for DEBUG util.NativeCodeLoader: Failed to load 
 native-hadoop with error: java.lang.UnsatisfiedLinkError
 -

 Key: HADOOP-8884
 URL: https://issues.apache.org/jira/browse/HADOOP-8884
 Project: Hadoop Common
  Issue Type: Bug
  Components: util
Affects Versions: 2.0.1-alpha
Reporter: Anthony Rojas
Assignee: Anthony Rojas
 Fix For: 2.0.3-alpha

 Attachments: HADOOP-8884.patch, HADOOP-8884.patch, 
 HADOOP-8884-v2.patch


 Recommending to change the following debug message and promote it to a 
 warning instead:
 12/07/02 18:41:44 DEBUG util.NativeCodeLoader: Failed to load native-hadoop 
 with error: java.lang.UnsatisfiedLinkError: 
 /usr/lib/hadoop/lib/native/libhadoop.so.1.0.0: /lib64/libc.so.6: version 
 `GLIBC_2.6' not found (required by 
 /usr/lib/hadoop/lib/native/libhadoop.so.1.0.0)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-8804) Improve Web UIs when the wildcard address is used


[ 
https://issues.apache.org/jira/browse/HADOOP-8804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13472342#comment-13472342
 ] 

Hudson commented on HADOOP-8804:


Integrated in Hadoop-Hdfs-trunk #1190 (See 
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/1190/])
HADOOP-8804. Improve Web UIs when the wildcard address is used. Contributed 
by Senthil Kumar (Revision 1395703)

 Result = SUCCESS
eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1395703
Files : 
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/StringUtils.java
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestStringUtils.java


 Improve Web UIs when the wildcard address is used
 -

 Key: HADOOP-8804
 URL: https://issues.apache.org/jira/browse/HADOOP-8804
 Project: Hadoop Common
  Issue Type: Improvement
Affects Versions: 1.0.0, 2.0.0-alpha
Reporter: Eli Collins
Assignee: Senthil V Kumar
Priority: Minor
  Labels: newbie
 Fix For: 2.0.3-alpha

 Attachments: DisplayOptions.jpg, HADOOP-8804-1.0.patch, 
 HADOOP-8804-1.1.patch, HADOOP-8804-1.1.patch, HADOOP-8804-1.1.patch, 
 HADOOP-8804-trunk.patch, HADOOP-8804-trunk.patch, HADOOP-8804-trunk.patch, 
 HADOOP-8804-trunk.patch, HADOOP-8804-trunk.patch


 When IPC addresses are bound to the wildcard (ie the default config) the NN, 
 JT (and probably RM etc) Web UIs are a little goofy. Eg 0 Hadoop Map/Reduce 
 Administration and NameNode '0.0.0.0:18021' (active). Let's improve them.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-8894) GenericTestUtils.waitFor should dump thread stacks on timeout


[ 
https://issues.apache.org/jira/browse/HADOOP-8894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13472394#comment-13472394
 ] 

Hudson commented on HADOOP-8894:


Integrated in Hadoop-Mapreduce-trunk #1221 (See 
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1221/])
HADOOP-8894. GenericTestUtils.waitFor should dump thread stacks on timeout. 
Contributed by Todd Lipcon. (Revision 1395825)

 Result = SUCCESS
todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1395825
Files : 
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/test/GenericTestUtils.java
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/test/TimedOutTestsListener.java


 GenericTestUtils.waitFor should dump thread stacks on timeout
 -

 Key: HADOOP-8894
 URL: https://issues.apache.org/jira/browse/HADOOP-8894
 Project: Hadoop Common
  Issue Type: Improvement
Reporter: Todd Lipcon
Assignee: Todd Lipcon
 Fix For: 3.0.0, 2.0.3-alpha

 Attachments: hadoop-8894.txt, hadoop-8894.txt


 Many tests use this utility to wait for a condition to become true. In the 
 event that it times out, we should dump all the thread stack traces, in case 
 the timeout was due to a deadlock. This should make it easier to debug 
 scenarios like HDFS-4001.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-8886) Remove KFS support


[ 
https://issues.apache.org/jira/browse/HADOOP-8886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13472397#comment-13472397
 ] 

Hudson commented on HADOOP-8886:


Integrated in Hadoop-Mapreduce-trunk #1221 (See 
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1221/])
HADOOP-8886. Remove KFS support. Contributed by Eli Collins (Revision 
1395820)

 Result = SUCCESS
eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1395820
Files : 
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-common/dev-support/findbugsExcludeFile.xml
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/pom.xml
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/kfs
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/resources/META-INF/services/org.apache.hadoop.fs.FileSystem
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/resources/core-default.xml
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/kfs


 Remove KFS support
 --

 Key: HADOOP-8886
 URL: https://issues.apache.org/jira/browse/HADOOP-8886
 Project: Hadoop Common
  Issue Type: Improvement
  Components: fs
Reporter: Eli Collins
Assignee: Eli Collins
 Fix For: 3.0.0

 Attachments: hadoop-8886.txt


 KFS is no longer maintained (is replaced by QFS, which HADOOP-8885 is 
 adding), let's remove it.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-8884) DEBUG should be WARN for DEBUG util.NativeCodeLoader: Failed to load native-hadoop with error: java.lang.UnsatisfiedLinkError


[ 
https://issues.apache.org/jira/browse/HADOOP-8884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13472396#comment-13472396
 ] 

Hudson commented on HADOOP-8884:


Integrated in Hadoop-Mapreduce-trunk #1221 (See 
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1221/])
Revert initial commit of HADOOP-8884 per feedback from Todd. (Revision 
1395783)
HADOOP-8884. DEBUG should be WARN for DEBUG util.NativeCodeLoader: Failed to 
load native-hadoop with error: java.lang.UnsatisfiedLinkError. Contributed by 
Anthony Rojas. (Revision 1395714)

 Result = SUCCESS
atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1395783
Files : 
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/NativeCodeLoader.java

atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1395714
Files : 
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/NativeCodeLoader.java


 DEBUG should be WARN for DEBUG util.NativeCodeLoader: Failed to load 
 native-hadoop with error: java.lang.UnsatisfiedLinkError
 -

 Key: HADOOP-8884
 URL: https://issues.apache.org/jira/browse/HADOOP-8884
 Project: Hadoop Common
  Issue Type: Bug
  Components: util
Affects Versions: 2.0.1-alpha
Reporter: Anthony Rojas
Assignee: Anthony Rojas
 Fix For: 2.0.3-alpha

 Attachments: HADOOP-8884.patch, HADOOP-8884.patch, 
 HADOOP-8884-v2.patch


 Recommending to change the following debug message and promote it to a 
 warning instead:
 12/07/02 18:41:44 DEBUG util.NativeCodeLoader: Failed to load native-hadoop 
 with error: java.lang.UnsatisfiedLinkError: 
 /usr/lib/hadoop/lib/native/libhadoop.so.1.0.0: /lib64/libc.so.6: version 
 `GLIBC_2.6' not found (required by 
 /usr/lib/hadoop/lib/native/libhadoop.so.1.0.0)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-8804) Improve Web UIs when the wildcard address is used


[ 
https://issues.apache.org/jira/browse/HADOOP-8804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13472399#comment-13472399
 ] 

Hudson commented on HADOOP-8804:


Integrated in Hadoop-Mapreduce-trunk #1221 (See 
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1221/])
HADOOP-8804. Improve Web UIs when the wildcard address is used. Contributed 
by Senthil Kumar (Revision 1395703)

 Result = SUCCESS
eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1395703
Files : 
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/StringUtils.java
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestStringUtils.java


 Improve Web UIs when the wildcard address is used
 -

 Key: HADOOP-8804
 URL: https://issues.apache.org/jira/browse/HADOOP-8804
 Project: Hadoop Common
  Issue Type: Improvement
Affects Versions: 1.0.0, 2.0.0-alpha
Reporter: Eli Collins
Assignee: Senthil V Kumar
Priority: Minor
  Labels: newbie
 Fix For: 2.0.3-alpha

 Attachments: DisplayOptions.jpg, HADOOP-8804-1.0.patch, 
 HADOOP-8804-1.1.patch, HADOOP-8804-1.1.patch, HADOOP-8804-1.1.patch, 
 HADOOP-8804-trunk.patch, HADOOP-8804-trunk.patch, HADOOP-8804-trunk.patch, 
 HADOOP-8804-trunk.patch, HADOOP-8804-trunk.patch


 When IPC addresses are bound to the wildcard (ie the default config) the NN, 
 JT (and probably RM etc) Web UIs are a little goofy. Eg 0 Hadoop Map/Reduce 
 Administration and NameNode '0.0.0.0:18021' (active). Let's improve them.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-8895) TokenRenewer should be an interface, it is currently a fully abstract class

2012-10-09 Thread Tom White (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-8895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13472404#comment-13472404
 ] 

Tom White commented on HADOOP-8895:
---

Abstract classes are easier to evolve than interfaces, since you can add a 
method with a default implementation without breaking implementors. 
TokenRenewer is only for use internally so it doesn't matter so much, but I 
would still leave it as it is since it's always possible to create a new 
implementation class that extends the abstract class.

 TokenRenewer should be an interface, it is currently a fully abstract class
 ---

 Key: HADOOP-8895
 URL: https://issues.apache.org/jira/browse/HADOOP-8895
 Project: Hadoop Common
  Issue Type: Bug
Affects Versions: 2.0.1-alpha
Reporter: Karthik Kambatla
Assignee: Karthik Kambatla
Priority: Minor
 Attachments: HADOOP-8895.patch


 TokenRenewer is a fully abstract class. Making it an interface will allow 
 classes extending other classes to implement the interface.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-8895) TokenRenewer should be an interface, it is currently a fully abstract class

[
https://issues.apache.org/jira/browse/HADOOP-8895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13472412#comment-13472412
]

Daryn Sharp commented on HADOOP-8895:
-

bq. For instance, if a WebHdfsFileSystem could implement TokenRenewer, we might
not need DelegationTokenRenewer.Renewable.

Correct to the second half. The {{DelegationTokenRenewer.Renewable}} interface
only existed for the code being removed, so it too should be removed.

I actually intended for {{TokenRenewer}} to be implemented as an interface for
classes implementing tokens. This idea was rejected in favor of using lower
level renewers that bypassed the token issuing class. I believe part of the
idea as a class is to force simple lightweight renewer objects since the
service loader will instantiate all of them.

Also, the filesystems currently cannot directly implement {{TokenRenewer}}
because the service loader instantiation won't be usable. It will either need
to instantiate another copy of itself or its lower level client with the
token's address, or will need to initialize itself - but filesystems are not
designed to be initialized multiple times.

This is why I think it's not needed.

TokenRenewer should be an interface, it is currently a fully abstract class
---

Key: HADOOP-8895
URL: https://issues.apache.org/jira/browse/HADOOP-8895
Project: Hadoop Common
Issue Type: Bug
Affects Versions: 2.0.1-alpha
Reporter: Karthik Kambatla
Assignee: Karthik Kambatla
Priority: Minor
Attachments: HADOOP-8895.patch

TokenRenewer is a fully abstract class. Making it an interface will allow
classes extending other classes to implement the interface.

[jira] [Updated] (HADOOP-8878) uppercase namenode hostname causes hadoop dfs calls with webhdfs filesystem and fsck to fail when security is on

[
https://issues.apache.org/jira/browse/HADOOP-8878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Arpit Gupta updated HADOOP-8878:

Status: Open (was: Patch Available)

canceling the patch to adress Steve's comments.

uppercase namenode hostname causes hadoop dfs calls with webhdfs filesystem
and fsck to fail when security is on

[jira] [Commented] (HADOOP-8899) Classpath exceeds maximum OS limit

[
https://issues.apache.org/jira/browse/HADOOP-8899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13472503#comment-13472503
]

Alejandro Abdelnur commented on HADOOP-8899:

You don't want to unjar and jar, as you'll lose in the process any JDK services
definitions in the JARs. I'd suggest the following alternate approach: all JARs
in the DistributedCache classpath should be symlink-ed to a
distributedcache-lib/ dir in the current directory and then
'distributedcached-lib/*' should be added to the classpath.

Classpath exceeds maximum OS limit
--

Key: HADOOP-8899
URL: https://issues.apache.org/jira/browse/HADOOP-8899
Project: Hadoop Common
Issue Type: Bug
Affects Versions: 1-win
Reporter: Ahmed El Baz

We hit errors in TT due to long classpath value. One example is Oozie trying
to start a hive job, and it fails with the following error:
java.io.IOException: Command exceeds the OS command length limit: 8192,
command: set CLASSPATH=….
The classpath includes Hadoop Core + Hive Jars which are in the distributed
cache. This is causing the classpath to be too long giving the error above. A
viable long term fix is to generate a temporary JAR file in the task cache
directory which includes all elements in the long classpath, and then provide
this as the -classpath argument for the JVM to be spawned. This is gated
for Windows only.

[jira] [Commented] (HADOOP-8899) Classpath exceeds maximum OS limit

[
https://issues.apache.org/jira/browse/HADOOP-8899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13472507#comment-13472507
]

Alejandro Abdelnur commented on HADOOP-8899:

The potential downside of my suggested approach is that you lose the capability
to order JARs in the classpath as '*' will be resolved in system specific way
(I've seen this being done differently in different unix versions); which in my
opinion should not be an issue as you should not have duplicate JARs in your
classpath to start. The other downside is that you won't, from the logs, know
exactly what is in the classpath, you'll only see a 'distributedcache-lib/*'
entry in the classpath.

Classpath exceeds maximum OS limit
--

Key: HADOOP-8899
URL: https://issues.apache.org/jira/browse/HADOOP-8899
Project: Hadoop Common
Issue Type: Bug
Affects Versions: 1-win
Reporter: Ahmed El Baz

[jira] [Updated] (HADOOP-8903) support HADOOP_USER_CLASSPATH_FIRST in windows hadoop cmd

2012-10-09 Thread Suresh Srinivas (JIRA)


 [ 
https://issues.apache.org/jira/browse/HADOOP-8903?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Suresh Srinivas updated HADOOP-8903:


Assignee: Thejas M Nair

 support HADOOP_USER_CLASSPATH_FIRST in windows hadoop cmd
 -

 Key: HADOOP-8903
 URL: https://issues.apache.org/jira/browse/HADOOP-8903
 Project: Hadoop Common
  Issue Type: Bug
  Components: scripts
Affects Versions: 1-win
Reporter: Thejas M Nair
Assignee: Thejas M Nair
 Attachments: HADOOP-8903.1.patch


 HADOOP_USER_CLASSPATH_FIRST environment variable has been documented in 
 hadoop.cmd comments, but the code to make it work is missing .

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Resolved] (HADOOP-8903) support HADOOP_USER_CLASSPATH_FIRST in windows hadoop cmd

2012-10-09 Thread Suresh Srinivas (JIRA)


 [ 
https://issues.apache.org/jira/browse/HADOOP-8903?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Suresh Srinivas resolved HADOOP-8903.
-

  Resolution: Fixed
Hadoop Flags: Reviewed

+1 for the patch. I committed it.

Thank you Thejas for the patch and Ivan for the review.

 support HADOOP_USER_CLASSPATH_FIRST in windows hadoop cmd
 -

 Key: HADOOP-8903
 URL: https://issues.apache.org/jira/browse/HADOOP-8903
 Project: Hadoop Common
  Issue Type: Bug
  Components: scripts
Affects Versions: 1-win
Reporter: Thejas M Nair
Assignee: Thejas M Nair
 Attachments: HADOOP-8903.1.patch


 HADOOP_USER_CLASSPATH_FIRST environment variable has been documented in 
 hadoop.cmd comments, but the code to make it work is missing .

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-8899) Classpath exceeds maximum OS limit

2012-10-09 Thread Joep Rottinghuis (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-8899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13472529#comment-13472529
 ] 

Joep Rottinghuis commented on HADOOP-8899:
--

Admittedly I have not kept up with this lately, but aren't symlinks  (MKLINK) 
limited to NTFS?
Does that interopt well  with Cygwin?
Not sure if that is truly a restriction anymore these days...

 Classpath exceeds maximum OS limit
 --

 Key: HADOOP-8899
 URL: https://issues.apache.org/jira/browse/HADOOP-8899
 Project: Hadoop Common
  Issue Type: Bug
Affects Versions: 1-win
Reporter: Ahmed El Baz

 We hit errors in TT due to long classpath value. One example is Oozie trying 
 to start a hive job, and it fails with the following error:
  java.io.IOException: Command exceeds the OS command length limit: 8192, 
 command: set CLASSPATH=….
 The classpath includes Hadoop Core + Hive Jars which are in the distributed 
 cache. This is causing the classpath to be too long giving the error above. A 
 viable long term fix is to generate a temporary JAR file in the task cache 
 directory which includes all elements in the long classpath, and then provide 
 this as the -classpath argument for the JVM to be spawned. This is gated 
 for Windows only.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-8904) Hadoop does not close output file / does not call Mapper.cleanup if exception in map

2012-10-09 Thread Todd Lipcon (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-8904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13472549#comment-13472549
 ] 

Todd Lipcon commented on HADOOP-8904:
-

This seems like a bug fix which is relevant for branch-1 as well as newer 
branches, not just branch-1-win, right? Is trunk/branch-2 affected?

  Hadoop does not close output file / does not call Mapper.cleanup if 
 exception in map
 -

 Key: HADOOP-8904
 URL: https://issues.apache.org/jira/browse/HADOOP-8904
 Project: Hadoop Common
  Issue Type: Bug
Affects Versions: 1-win
Reporter: Daniel Dai
 Attachments: HADOOP-8904-1.patch


 Find this in Pig unit test TestStore under Windows. There are dangling files 
 because map does not close the file when exception happens in map(). In 
 Windows, Hadoop will not remove a file if it is not closed. This happens in 
 reduce() as well.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HADOOP-8885) Need to add fs shim to use QFS

2012-10-09 Thread thilee (JIRA)


 [ 
https://issues.apache.org/jira/browse/HADOOP-8885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

thilee updated HADOOP-8885:
---

Fix Version/s: 1.0.2
   1.0.3
   0.23.3
   2.0.0-alpha
   2.0.1-alpha
   3.0.0
Affects Version/s: (was: 2.0.1-alpha)
   (was: 2.0.0-alpha)
   (was: 1.0.3)
   (was: 1.0.2)
   (was: 0.23.3)
   Status: Patch Available  (was: Open)

 Need to add fs shim to use QFS
 --

 Key: HADOOP-8885
 URL: https://issues.apache.org/jira/browse/HADOOP-8885
 Project: Hadoop Common
  Issue Type: New Feature
  Components: fs
Reporter: thilee
Assignee: thilee
 Fix For: 3.0.0, 2.0.1-alpha, 2.0.0-alpha, 0.23.3, 1.0.3, 1.0.2

 Attachments: HADOOP-8885-branch-1.patch, HADOOP-8885-trunk.patch

   Original Estimate: 168h
  Remaining Estimate: 168h

 Quantcast has released QFS 1.0 (http://quantcast.github.com/qfs), a C++ 
 distributed filesystem based on Kosmos File System(KFS). QFS comes with 
 various feature, performance, and stability improvements over KFS.
 A hadoop 'fs' shim needs be added to support QFS through 'qfs://' URIs.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HADOOP-8905) Add metrics for HTTP Server

2012-10-09 Thread Todd Lipcon (JIRA)

Todd Lipcon created HADOOP-8905:
---

 Summary: Add metrics for HTTP Server
 Key: HADOOP-8905
 URL: https://issues.apache.org/jira/browse/HADOOP-8905
 Project: Hadoop Common
  Issue Type: Improvement
  Components: metrics
Affects Versions: 3.0.0
Reporter: Todd Lipcon


Currently we don't expose any metrics about the HTTP server. It would be useful 
to be able to monitor the following:
- Number of threads currently actively serving servlet requests
- Total number of requests served
- Perhaps break down time/count by endpoint (eg /jmx, /conf, various JSPs)

This becomes more important as http-based protocols like webhdfs become more 
common

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-8885) Need to add fs shim to use QFS

[
https://issues.apache.org/jira/browse/HADOOP-8885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13472563#comment-13472563
]

Hadoop QA commented on HADOOP-8885:
---

{color:red}-1 overall{color}. Here are the results of testing the latest
attachment

http://issues.apache.org/jira/secure/attachment/12548323/HADOOP-8885-trunk.patch
against trunk revision .

{color:green}+1 @author{color}. The patch does not contain any @author
tags.

{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.

{color:red}-1 javac{color:red}. The patch appears to cause the build to
fail.

Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/1589//console

This message is automatically generated.

Need to add fs shim to use QFS
--

Key: HADOOP-8885
URL: https://issues.apache.org/jira/browse/HADOOP-8885
Project: Hadoop Common
Issue Type: New Feature
Components: fs
Reporter: thilee
Assignee: thilee
Fix For: 1.0.2, 1.0.3, 0.23.3, 2.0.0-alpha, 3.0.0, 2.0.1-alpha

Attachments: HADOOP-8885-branch-1.patch, HADOOP-8885-trunk.patch

Original Estimate: 168h
Remaining Estimate: 168h

Quantcast has released QFS 1.0 (http://quantcast.github.com/qfs), a C++
distributed filesystem based on Kosmos File System(KFS). QFS comes with
various feature, performance, and stability improvements over KFS.
A hadoop 'fs' shim needs be added to support QFS through 'qfs://' URIs.

[jira] [Commented] (HADOOP-8905) Add metrics for HTTP Server

2012-10-09 Thread Aaron T. Myers (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-8905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13472566#comment-13472566
 ] 

Aaron T. Myers commented on HADOOP-8905:


Might also be nice to use the percentile metrics support to record high 
percentile response times.

 Add metrics for HTTP Server
 ---

 Key: HADOOP-8905
 URL: https://issues.apache.org/jira/browse/HADOOP-8905
 Project: Hadoop Common
  Issue Type: Improvement
  Components: metrics
Affects Versions: 3.0.0
Reporter: Todd Lipcon
  Labels: newbie

 Currently we don't expose any metrics about the HTTP server. It would be 
 useful to be able to monitor the following:
 - Number of threads currently actively serving servlet requests
 - Total number of requests served
 - Perhaps break down time/count by endpoint (eg /jmx, /conf, various JSPs)
 This becomes more important as http-based protocols like webhdfs become more 
 common

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-8891) Remove DelegationTokenRenewer and its logic from WebHdfsFileSystem and HftpFileSystem


[ 
https://issues.apache.org/jira/browse/HADOOP-8891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13472569#comment-13472569
 ] 

Daryn Sharp commented on HADOOP-8891:
-

Are some of these changes (like removal of the renewer) duplicated in your 
other jiras?  If so, that will cause merge conflicts.

I still need to look at {{HttpFSFileSystem}} to see about how it does or should 
find a token, but: {{getDelegationToken}} is used by job submission, so it must 
remain.  {{setDelegationToken}} is only used by the renewer you are removing, 
so it should be removed.

 Remove DelegationTokenRenewer and its logic from WebHdfsFileSystem and 
 HftpFileSystem
 -

 Key: HADOOP-8891
 URL: https://issues.apache.org/jira/browse/HADOOP-8891
 Project: Hadoop Common
  Issue Type: Improvement
Affects Versions: 2.0.1-alpha
Reporter: Karthik Kambatla
Assignee: Karthik Kambatla
 Attachments: HADOOP-8891.patch, HADOOP-8891.patch


 Moved the HDFS part of HADOOP-8852 to HDFS-4009 along with other sub-tasks. 
 Created this to track the removal of DelegationTokenRenewer alone.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-8899) Classpath exceeds maximum OS limit


[ 
https://issues.apache.org/jira/browse/HADOOP-8899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13472571#comment-13472571
 ] 

Alejandro Abdelnur commented on HADOOP-8899:


Argh, forgot about that. Maybe in win distributed cache symlinks should be 
simulated with copies. Then the proposed approach would work

Alejandro




 Classpath exceeds maximum OS limit
 --

 Key: HADOOP-8899
 URL: https://issues.apache.org/jira/browse/HADOOP-8899
 Project: Hadoop Common
  Issue Type: Bug
Affects Versions: 1-win
Reporter: Ahmed El Baz

 We hit errors in TT due to long classpath value. One example is Oozie trying 
 to start a hive job, and it fails with the following error:
  java.io.IOException: Command exceeds the OS command length limit: 8192, 
 command: set CLASSPATH=….
 The classpath includes Hadoop Core + Hive Jars which are in the distributed 
 cache. This is causing the classpath to be too long giving the error above. A 
 viable long term fix is to generate a temporary JAR file in the task cache 
 directory which includes all elements in the long classpath, and then provide 
 this as the -classpath argument for the JVM to be spawned. This is gated 
 for Windows only.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-8608) Add Configuration API for parsing time durations

2012-10-09 Thread Robert Joseph Evans (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-8608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13472579#comment-13472579
 ] 

Robert Joseph Evans commented on HADOOP-8608:
-

I like the new API.  I do have a few minor comments, none of which I see as 
blocking this from going in.

The first is to not use StringBuilder for the warning.  It seems cleaner and 
more compact to have 

{code}
LOG.warn(No unit for  + name +  ( + vStr +) assuming  
  + unit);
{code} over {code}
StringBuilder sb = new StringBuilder(No unit for )
  .append(name).append( ().append(vStr).append() assuming )
  .append(unit.toString());
LOG.warn(sb.toString());
{code}

I would also like to see more lenient parsing, but that is something that can 
come later as the need shows up.  I can see people wanting to write 1.5 days 
over 36h.  But like I said before this is not a big deal because 36h is a 
whole lot easier to understand and less error prone compared to 12960.

 Add Configuration API for parsing time durations
 

 Key: HADOOP-8608
 URL: https://issues.apache.org/jira/browse/HADOOP-8608
 Project: Hadoop Common
  Issue Type: Improvement
  Components: conf
Affects Versions: 3.0.0
Reporter: Todd Lipcon
 Attachments: 8608-0.patch, 8608-1.patch


 Hadoop has a lot of configurations which specify durations or intervals of 
 time. Unfortunately these different configurations have little consistency in 
 units - eg some are in milliseconds, some in seconds, and some in minutes. 
 This makes it difficult for users to configure, since they have to always 
 refer back to docs to remember the unit for each property.
 The proposed solution is to add an API like {{Configuration.getTimeDuration}} 
 which allows the user to specify the units with a postfix. For example, 
 10ms, 10s, 10m, 10h, or even 10d. For backwards-compatibility, if 
 the user does not specify a unit, the API can specify the default unit, and 
 warn the user that they should specify an explicit unit instead.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HADOOP-8900) BuiltInGzipDecompressor : java.io.IOException: stored gzip size doesn't match decompressed size (Slavik Krassovsky)


 [ 
https://issues.apache.org/jira/browse/HADOOP-8900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Slavik Krassovsky updated HADOOP-8900:
--

Affects Version/s: 1-win

 BuiltInGzipDecompressor : java.io.IOException: stored gzip size doesn't match 
 decompressed size (Slavik Krassovsky)
 ---

 Key: HADOOP-8900
 URL: https://issues.apache.org/jira/browse/HADOOP-8900
 Project: Hadoop Common
  Issue Type: Bug
Affects Versions: 1-win, 2.0.1-alpha
 Environment: Encountered failure when processing large GZIP file
Reporter: Slavik Krassovsky
Assignee: Andy Isaacson
 Attachments: BuiltInGzipDecompressor2.patch, hadoop8900.txt


 Encountered failure when processing large GZIP file
 • Gz: Failed in 1hrs, 13mins, 57sec with the error:
  ¸java.io.IOException: IO error in map input file 
 hdfs://localhost:9000/Halo4/json_m/gz/NewFileCat.txt.gz
  at 
 org.apache.hadoop.mapred.MapTask$TrackedRecordReader.moveToNext(MapTask.java:242)
  at 
 org.apache.hadoop.mapred.MapTask$TrackedRecordReader.next(MapTask.java:216)
  at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:48)
  at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:435)
  at org.apache.hadoop.mapred.MapTask.run(MapTask.java:371)
  at org.apache.hadoop.mapred.Child$4.run(Child.java:266)
  at java.security.AccessController.doPrivileged(Native Method)
  at javax.security.auth.Subject.doAs(Subject.java:415)
  at 
 org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
  at org.apache.hadoop.mapred.Child.main(Child.java:260)
  Caused by: java.io.IOException: stored gzip size doesn't match decompressed 
 size
  at 
 org.apache.hadoop.io.compress.zlib.BuiltInGzipDecompressor.executeTrailerState(BuiltInGzipDecompressor.java:389)
  at 
 org.apache.hadoop.io.compress.zlib.BuiltInGzipDecompressor.decompress(BuiltInGzipDecompressor.java:224)
  at 
 org.apache.hadoop.io.compress.DecompressorStream.decompress(DecompressorStream.java:82)
  at 
 org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:76)
  at java.io.InputStream.read(InputStream.java:102)
  at org.apache.hadoop.util.LineReader.readLine(LineReader.java:134)
  at org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:136)
  at org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:40)
  at 
 org.apache.hadoop.hive.ql.io.HiveRecordReader.doNext(HiveRecordReader.java:66)
  at 
 org.apache.hadoop.hive.ql.io.HiveRecordReader.doNext(HiveRecordReader.java:32)
  at 
 org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.next(HiveContextAwareRecordReader.java:67)
  at 
 org.apache.hadoop.mapred.MapTask$TrackedRecordReader.moveToNext(MapTask.java:236)
  ... 9 more

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-8900) BuiltInGzipDecompressor : java.io.IOException: stored gzip size doesn't match decompressed size (Slavik Krassovsky)


[ 
https://issues.apache.org/jira/browse/HADOOP-8900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13472582#comment-13472582
 ] 

Slavik Krassovsky commented on HADOOP-8900:
---

The 
http://issues.apache.org/jira/secure/attachment/12548348/BuiltInGzipDecompressor2.patch
 is for 1-win branch.

 BuiltInGzipDecompressor : java.io.IOException: stored gzip size doesn't match 
 decompressed size (Slavik Krassovsky)
 ---

 Key: HADOOP-8900
 URL: https://issues.apache.org/jira/browse/HADOOP-8900
 Project: Hadoop Common
  Issue Type: Bug
Affects Versions: 1-win, 2.0.1-alpha
 Environment: Encountered failure when processing large GZIP file
Reporter: Slavik Krassovsky
Assignee: Andy Isaacson
 Attachments: BuiltInGzipDecompressor2.patch, hadoop8900.txt


 Encountered failure when processing large GZIP file
 • Gz: Failed in 1hrs, 13mins, 57sec with the error:
  ¸java.io.IOException: IO error in map input file 
 hdfs://localhost:9000/Halo4/json_m/gz/NewFileCat.txt.gz
  at 
 org.apache.hadoop.mapred.MapTask$TrackedRecordReader.moveToNext(MapTask.java:242)
  at 
 org.apache.hadoop.mapred.MapTask$TrackedRecordReader.next(MapTask.java:216)
  at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:48)
  at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:435)
  at org.apache.hadoop.mapred.MapTask.run(MapTask.java:371)
  at org.apache.hadoop.mapred.Child$4.run(Child.java:266)
  at java.security.AccessController.doPrivileged(Native Method)
  at javax.security.auth.Subject.doAs(Subject.java:415)
  at 
 org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
  at org.apache.hadoop.mapred.Child.main(Child.java:260)
  Caused by: java.io.IOException: stored gzip size doesn't match decompressed 
 size
  at 
 org.apache.hadoop.io.compress.zlib.BuiltInGzipDecompressor.executeTrailerState(BuiltInGzipDecompressor.java:389)
  at 
 org.apache.hadoop.io.compress.zlib.BuiltInGzipDecompressor.decompress(BuiltInGzipDecompressor.java:224)
  at 
 org.apache.hadoop.io.compress.DecompressorStream.decompress(DecompressorStream.java:82)
  at 
 org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:76)
  at java.io.InputStream.read(InputStream.java:102)
  at org.apache.hadoop.util.LineReader.readLine(LineReader.java:134)
  at org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:136)
  at org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:40)
  at 
 org.apache.hadoop.hive.ql.io.HiveRecordReader.doNext(HiveRecordReader.java:66)
  at 
 org.apache.hadoop.hive.ql.io.HiveRecordReader.doNext(HiveRecordReader.java:32)
  at 
 org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.next(HiveContextAwareRecordReader.java:67)
  at 
 org.apache.hadoop.mapred.MapTask$TrackedRecordReader.moveToNext(MapTask.java:236)
  ... 9 more

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HADOOP-8878) uppercase namenode hostname causes hadoop dfs calls with webhdfs filesystem and fsck to fail when security is on

[
https://issues.apache.org/jira/browse/HADOOP-8878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Arpit Gupta updated HADOOP-8878:

Attachment: HADOOP-8878.branch-1.patch

patch for branch-1 with local in the toLowerCase.

uppercase namenode hostname causes hadoop dfs calls with webhdfs filesystem
and fsck to fail when security is on

[jira] [Updated] (HADOOP-8878) uppercase namenode hostname causes hadoop dfs calls with webhdfs filesystem and fsck to fail when security is on


 [ 
https://issues.apache.org/jira/browse/HADOOP-8878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Arpit Gupta updated HADOOP-8878:


Attachment: HADOOP-8878.patch

patch for trunk which is using LOCALE.US in toLowerCase

 uppercase namenode hostname causes hadoop dfs calls with webhdfs filesystem 
 and fsck to fail when security is on
 

 Key: HADOOP-8878
 URL: https://issues.apache.org/jira/browse/HADOOP-8878
 Project: Hadoop Common
  Issue Type: Bug
Affects Versions: 1.0.3, 1.1.0, 1.2.0, 3.0.0
Reporter: Arpit Gupta
Assignee: Arpit Gupta
 Attachments: HADOOP-8878.branch-1.patch, HADOOP-8878.branch-1.patch, 
 HADOOP-8878.branch-1.patch, HADOOP-8878.patch, HADOOP-8878.patch, 
 HADOOP-8878.patch


 This was noticed on a secure cluster where the namenode had an upper case 
 hostname and the following command was issued
 hadoop dfs -ls webhdfs://NN:PORT/PATH
 the above command failed because delegation token retrieval failed.
 Upon looking at the kerberos logs it was determined that we tried to get the 
 ticket for kerberos principal with upper case hostnames and that host did not 
 exit in kerberos. We should convert the hostnames to lower case. Take a look 
 at HADOOP-7988 where the same fix was applied on a different class.
 I have noticed this issue exists on branch-1. Will investigate trunk and 
 branch-2 and update accordingly.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HADOOP-8878) uppercase namenode hostname causes hadoop dfs calls with webhdfs filesystem and fsck to fail when security is on


 [ 
https://issues.apache.org/jira/browse/HADOOP-8878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Arpit Gupta updated HADOOP-8878:


Status: Patch Available  (was: Open)

 uppercase namenode hostname causes hadoop dfs calls with webhdfs filesystem 
 and fsck to fail when security is on
 

 Key: HADOOP-8878
 URL: https://issues.apache.org/jira/browse/HADOOP-8878
 Project: Hadoop Common
  Issue Type: Bug
Affects Versions: 1.0.3, 1.1.0, 1.2.0, 3.0.0
Reporter: Arpit Gupta
Assignee: Arpit Gupta
 Attachments: HADOOP-8878.branch-1.patch, HADOOP-8878.branch-1.patch, 
 HADOOP-8878.branch-1.patch, HADOOP-8878.patch, HADOOP-8878.patch, 
 HADOOP-8878.patch


 This was noticed on a secure cluster where the namenode had an upper case 
 hostname and the following command was issued
 hadoop dfs -ls webhdfs://NN:PORT/PATH
 the above command failed because delegation token retrieval failed.
 Upon looking at the kerberos logs it was determined that we tried to get the 
 ticket for kerberos principal with upper case hostnames and that host did not 
 exit in kerberos. We should convert the hostnames to lower case. Take a look 
 at HADOOP-7988 where the same fix was applied on a different class.
 I have noticed this issue exists on branch-1. Will investigate trunk and 
 branch-2 and update accordingly.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-8882) uppercase namenode host name causes fsck to fail when useKsslAuth is on


[ 
https://issues.apache.org/jira/browse/HADOOP-8882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13472597#comment-13472597
 ] 

Arpit Gupta commented on HADOOP-8882:
-

once HADOOP-8878 gets committed i will change this patch to use the method in 
KerberosUtil so we are going through the same code path.

 uppercase namenode host name causes fsck to fail when useKsslAuth is on
 ---

 Key: HADOOP-8882
 URL: https://issues.apache.org/jira/browse/HADOOP-8882
 Project: Hadoop Common
  Issue Type: Bug
Affects Versions: 1.0.3, 1.1.0, 1.2.0
Reporter: Arpit Gupta
Assignee: Arpit Gupta
 Attachments: HADOOP-8882.branch-1.patch


 {code}
  public static void fetchServiceTicket(URL remoteHost) throws IOException {
 if(!UserGroupInformation.isSecurityEnabled())
   return;
 
 String serviceName = host/ + remoteHost.getHost();
 {code}
 the hostname should be converted to lower case. Saw this in branch 1, will 
 look at trunk and update the bug accordingly.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-8878) uppercase namenode hostname causes hadoop dfs calls with webhdfs filesystem and fsck to fail when security is on


[ 
https://issues.apache.org/jira/browse/HADOOP-8878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13472610#comment-13472610
 ] 

Hadoop QA commented on HADOOP-8878:
---

{color:green}+1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12548435/HADOOP-8878.patch
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 1 new 
or modified test files.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  The javadoc tool did not generate any 
warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 core tests{color}.  The patch passed unit tests in 
hadoop-common-project/hadoop-auth.

{color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: 
https://builds.apache.org/job/PreCommit-HADOOP-Build/1590//testReport/
Console output: 
https://builds.apache.org/job/PreCommit-HADOOP-Build/1590//console

This message is automatically generated.

 uppercase namenode hostname causes hadoop dfs calls with webhdfs filesystem 
 and fsck to fail when security is on
 

 Key: HADOOP-8878
 URL: https://issues.apache.org/jira/browse/HADOOP-8878
 Project: Hadoop Common
  Issue Type: Bug
Affects Versions: 1.0.3, 1.1.0, 1.2.0, 3.0.0
Reporter: Arpit Gupta
Assignee: Arpit Gupta
 Attachments: HADOOP-8878.branch-1.patch, HADOOP-8878.branch-1.patch, 
 HADOOP-8878.branch-1.patch, HADOOP-8878.patch, HADOOP-8878.patch, 
 HADOOP-8878.patch


 This was noticed on a secure cluster where the namenode had an upper case 
 hostname and the following command was issued
 hadoop dfs -ls webhdfs://NN:PORT/PATH
 the above command failed because delegation token retrieval failed.
 Upon looking at the kerberos logs it was determined that we tried to get the 
 ticket for kerberos principal with upper case hostnames and that host did not 
 exit in kerberos. We should convert the hostnames to lower case. Take a look 
 at HADOOP-7988 where the same fix was applied on a different class.
 I have noticed this issue exists on branch-1. Will investigate trunk and 
 branch-2 and update accordingly.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-8878) uppercase namenode hostname causes hadoop dfs calls with webhdfs filesystem and fsck to fail when security is on


[ 
https://issues.apache.org/jira/browse/HADOOP-8878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13472613#comment-13472613
 ] 

Arpit Gupta commented on HADOOP-8878:
-

test patch output for branch-1

{code}
[exec] 
 [exec] -1 overall.  
 [exec] 
 [exec] +1 @author.  The patch does not contain any @author tags.
 [exec] 
 [exec] +1 tests included.  The patch appears to include 2 new or 
modified tests.
 [exec] 
 [exec] +1 javadoc.  The javadoc tool did not generate any warning 
messages.
 [exec] 
 [exec] +1 javac.  The applied patch does not increase the total number 
of javac compiler warnings.
 [exec] 
 [exec] -1 findbugs.  The patch appears to introduce 9 new Findbugs 
(version 1.3.9) warnings.
 [exec] 
 [exec] 
 [exec] 
 [exec] 
 [exec] 
==
 [exec] 
==
 [exec] Finished build.
 [exec] 
==
 [exec] 
==
 [exec] 
 [exec] 
{code}

findbugs warnings are not related to this patch.

 uppercase namenode hostname causes hadoop dfs calls with webhdfs filesystem 
 and fsck to fail when security is on
 

 Key: HADOOP-8878
 URL: https://issues.apache.org/jira/browse/HADOOP-8878
 Project: Hadoop Common
  Issue Type: Bug
Affects Versions: 1.0.3, 1.1.0, 1.2.0, 3.0.0
Reporter: Arpit Gupta
Assignee: Arpit Gupta
 Attachments: HADOOP-8878.branch-1.patch, HADOOP-8878.branch-1.patch, 
 HADOOP-8878.branch-1.patch, HADOOP-8878.patch, HADOOP-8878.patch, 
 HADOOP-8878.patch


 This was noticed on a secure cluster where the namenode had an upper case 
 hostname and the following command was issued
 hadoop dfs -ls webhdfs://NN:PORT/PATH
 the above command failed because delegation token retrieval failed.
 Upon looking at the kerberos logs it was determined that we tried to get the 
 ticket for kerberos principal with upper case hostnames and that host did not 
 exit in kerberos. We should convert the hostnames to lower case. Take a look 
 at HADOOP-7988 where the same fix was applied on a different class.
 I have noticed this issue exists on branch-1. Will investigate trunk and 
 branch-2 and update accordingly.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-8895) TokenRenewer should be an interface, it is currently a fully abstract class


[ 
https://issues.apache.org/jira/browse/HADOOP-8895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13472615#comment-13472615
 ] 

Karthik Kambatla commented on HADOOP-8895:
--

Fair enough. Thanks Tom and Daryn. I ll close this JIRA as won't fix. We can 
may be re-open should a need arise. Thanks again.

 TokenRenewer should be an interface, it is currently a fully abstract class
 ---

 Key: HADOOP-8895
 URL: https://issues.apache.org/jira/browse/HADOOP-8895
 Project: Hadoop Common
  Issue Type: Bug
Affects Versions: 2.0.1-alpha
Reporter: Karthik Kambatla
Assignee: Karthik Kambatla
Priority: Minor
 Attachments: HADOOP-8895.patch


 TokenRenewer is a fully abstract class. Making it an interface will allow 
 classes extending other classes to implement the interface.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HADOOP-8895) TokenRenewer should be an interface, it is currently a fully abstract class


 [ 
https://issues.apache.org/jira/browse/HADOOP-8895?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Karthik Kambatla updated HADOOP-8895:
-

Resolution: Won't Fix
Status: Resolved  (was: Patch Available)

 TokenRenewer should be an interface, it is currently a fully abstract class
 ---

 Key: HADOOP-8895
 URL: https://issues.apache.org/jira/browse/HADOOP-8895
 Project: Hadoop Common
  Issue Type: Bug
Affects Versions: 2.0.1-alpha
Reporter: Karthik Kambatla
Assignee: Karthik Kambatla
Priority: Minor
 Attachments: HADOOP-8895.patch


 TokenRenewer is a fully abstract class. Making it an interface will allow 
 classes extending other classes to implement the interface.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-8900) BuiltInGzipDecompressor : java.io.IOException: stored gzip size doesn't match decompressed size (Slavik Krassovsky)


[ 
https://issues.apache.org/jira/browse/HADOOP-8900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13472616#comment-13472616
 ] 

Colin Patrick McCabe commented on HADOOP-8900:
--

Looks good to me.

 BuiltInGzipDecompressor : java.io.IOException: stored gzip size doesn't match 
 decompressed size (Slavik Krassovsky)
 ---

 Key: HADOOP-8900
 URL: https://issues.apache.org/jira/browse/HADOOP-8900
 Project: Hadoop Common
  Issue Type: Bug
Affects Versions: 1-win, 2.0.1-alpha
 Environment: Encountered failure when processing large GZIP file
Reporter: Slavik Krassovsky
Assignee: Andy Isaacson
 Attachments: BuiltInGzipDecompressor2.patch, hadoop8900.txt


 Encountered failure when processing large GZIP file
 • Gz: Failed in 1hrs, 13mins, 57sec with the error:
  ¸java.io.IOException: IO error in map input file 
 hdfs://localhost:9000/Halo4/json_m/gz/NewFileCat.txt.gz
  at 
 org.apache.hadoop.mapred.MapTask$TrackedRecordReader.moveToNext(MapTask.java:242)
  at 
 org.apache.hadoop.mapred.MapTask$TrackedRecordReader.next(MapTask.java:216)
  at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:48)
  at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:435)
  at org.apache.hadoop.mapred.MapTask.run(MapTask.java:371)
  at org.apache.hadoop.mapred.Child$4.run(Child.java:266)
  at java.security.AccessController.doPrivileged(Native Method)
  at javax.security.auth.Subject.doAs(Subject.java:415)
  at 
 org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
  at org.apache.hadoop.mapred.Child.main(Child.java:260)
  Caused by: java.io.IOException: stored gzip size doesn't match decompressed 
 size
  at 
 org.apache.hadoop.io.compress.zlib.BuiltInGzipDecompressor.executeTrailerState(BuiltInGzipDecompressor.java:389)
  at 
 org.apache.hadoop.io.compress.zlib.BuiltInGzipDecompressor.decompress(BuiltInGzipDecompressor.java:224)
  at 
 org.apache.hadoop.io.compress.DecompressorStream.decompress(DecompressorStream.java:82)
  at 
 org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:76)
  at java.io.InputStream.read(InputStream.java:102)
  at org.apache.hadoop.util.LineReader.readLine(LineReader.java:134)
  at org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:136)
  at org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:40)
  at 
 org.apache.hadoop.hive.ql.io.HiveRecordReader.doNext(HiveRecordReader.java:66)
  at 
 org.apache.hadoop.hive.ql.io.HiveRecordReader.doNext(HiveRecordReader.java:32)
  at 
 org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.next(HiveContextAwareRecordReader.java:67)
  at 
 org.apache.hadoop.mapred.MapTask$TrackedRecordReader.moveToNext(MapTask.java:236)
  ... 9 more

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-8891) Remove DelegationTokenRenewer and its logic from WebHdfsFileSystem and HftpFileSystem

[
https://issues.apache.org/jira/browse/HADOOP-8891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13472631#comment-13472631
]

Karthik Kambatla commented on HADOOP-8891:
--

bq. Are some of these changes (like removal of the renewer) duplicated in your
other jiras? If so, that will cause merge conflicts.

Yes. I created the other JIRAs thinking it would make it easier, but now
realize that there could be conflicts. I was thinking of finishing this one,
and see if anything else needs to be done for the other JIRAs. Otherwise, can
close them out as duplicates.

bq. setDelegationToken is only used by the renewer you are removing, so it
should be removed.

{{TestHttpFSWithKerberos}} calls {{setDelegationToken()}} on all these
filesystems. I am trying to figure out if there is any way around it, short of
removing the tests.

Remove DelegationTokenRenewer and its logic from WebHdfsFileSystem and
HftpFileSystem
-

Key: HADOOP-8891
URL: https://issues.apache.org/jira/browse/HADOOP-8891
Project: Hadoop Common
Issue Type: Improvement
Affects Versions: 2.0.1-alpha
Reporter: Karthik Kambatla
Assignee: Karthik Kambatla
Attachments: HADOOP-8891.patch, HADOOP-8891.patch

Moved the HDFS part of HADOOP-8852 to HDFS-4009 along with other sub-tasks.
Created this to track the removal of DelegationTokenRenewer alone.

[jira] [Created] (HADOOP-8906) paths with multiple globs are unreliable

Daryn Sharp created HADOOP-8906:
---

 Summary: paths with multiple globs are unreliable
 Key: HADOOP-8906
 URL: https://issues.apache.org/jira/browse/HADOOP-8906
 Project: Hadoop Common
  Issue Type: Bug
  Components: fs
Affects Versions: 2.0.0-alpha, 0.23.0, 3.0.0
Reporter: Daryn Sharp
Priority: Critical


Let's say we have have a structure of $date/$user/stuff/file.  Multiple globs 
are unreliable unless every directory in the structure exists.

These work:
date*/user
date*/user/stuff
date*/user/stuff/file

These fail:
date*/user/*
date*/user/*/*
date*/user/stu*
date*/user/stu*/*
date*/user/stu*/file
date*/user/stuff/*
date*/user/stuff/f*

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-8906) paths with multiple globs are unreliable


[ 
https://issues.apache.org/jira/browse/HADOOP-8906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13472696#comment-13472696
 ] 

Daryn Sharp commented on HADOOP-8906:
-

It appears that after a glob, all the intervening path components before the 
next glob _must_ exist.

 paths with multiple globs are unreliable
 

 Key: HADOOP-8906
 URL: https://issues.apache.org/jira/browse/HADOOP-8906
 Project: Hadoop Common
  Issue Type: Bug
  Components: fs
Affects Versions: 0.23.0, 2.0.0-alpha, 3.0.0
Reporter: Daryn Sharp
Priority: Critical

 Let's say we have have a structure of $date/$user/stuff/file.  Multiple 
 globs are unreliable unless every directory in the structure exists.
 These work:
 date*/user
 date*/user/stuff
 date*/user/stuff/file
 These fail:
 date*/user/*
 date*/user/*/*
 date*/user/stu*
 date*/user/stu*/*
 date*/user/stu*/file
 date*/user/stuff/*
 date*/user/stuff/f*

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-8906) paths with multiple globs are unreliable


[ 
https://issues.apache.org/jira/browse/HADOOP-8906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13472709#comment-13472709
 ] 

Daryn Sharp commented on HADOOP-8906:
-

There's more multi-glob bugginess. Given {{/file}}, and {{/dir/file}}:
# {{/\*/file}} will fail with a permission denied because it tries to treat 
{{/file}} as a directory since it matches the first *.
# Worse yet, {{/\*/file\*}} will return BOTH {{/file}} and {{/dir/file}}!

 paths with multiple globs are unreliable
 

 Key: HADOOP-8906
 URL: https://issues.apache.org/jira/browse/HADOOP-8906
 Project: Hadoop Common
  Issue Type: Bug
  Components: fs
Affects Versions: 0.23.0, 2.0.0-alpha, 3.0.0
Reporter: Daryn Sharp
Priority: Critical

 Let's say we have have a structure of $date/$user/stuff/file.  Multiple 
 globs are unreliable unless every directory in the structure exists.
 These work:
 date*/user
 date*/user/stuff
 date*/user/stuff/file
 These fail:
 date*/user/*
 date*/user/*/*
 date*/user/stu*
 date*/user/stu*/*
 date*/user/stu*/file
 date*/user/stuff/*
 date*/user/stuff/f*

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HADOOP-8902) Enable Gridmix v1 v2 benchmarks on Windows platform

2012-10-09 Thread Mike Liddell (JIRA)


 [ 
https://issues.apache.org/jira/browse/HADOOP-8902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mike Liddell updated HADOOP-8902:
-

Status: Open  (was: Patch Available)

 Enable Gridmix v1  v2 benchmarks on Windows platform
 -

 Key: HADOOP-8902
 URL: https://issues.apache.org/jira/browse/HADOOP-8902
 Project: Hadoop Common
  Issue Type: Bug
  Components: benchmarks
Affects Versions: 1-win
Reporter: Mike Liddell
 Attachments: HADOOP-8902.patch


 Gridmix v1 and v2 benchmarks do not run on Windows as they require bash 
 shell.  These scripts have been ported to Windows cmd-scripts.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HADOOP-8907) Provide means to look for zlib1.dll next to haodop.dll on Windows

2012-10-09 Thread Ivan Mitic (JIRA)

Ivan Mitic created HADOOP-8907:
--

 Summary: Provide means to look for zlib1.dll next to haodop.dll on 
Windows
 Key: HADOOP-8907
 URL: https://issues.apache.org/jira/browse/HADOOP-8907
 Project: Hadoop Common
  Issue Type: Bug
Affects Versions: 1-win
Reporter: Ivan Mitic
Assignee: Ivan Mitic


This change is dependent on HADOOP-8564.

Instead of asking end users to install zlib1.dll to system32 on Windows, it 
would be more convenient to just copy the dll next to hadoop.dll. 

More context and patch coming after HADOOP-8564.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HADOOP-8899) Classpath exceeds maximum OS limit

2012-10-09 Thread Ahmed El Baz (JIRA)

[
https://issues.apache.org/jira/browse/HADOOP-8899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Ahmed El Baz updated HADOOP-8899:
-

Attachment: HADOOP-8899.branch-1-win.classpath.patch

Please note the following:

- The generated Jar does not encapsulate any other jars, it simply references
all entries in the classpath in its Manifest, so this is just another way to
propagate the long list of Jars to the tasks to launch.

- Classpath elements are referenced in the generated Jar's Manifest in the same
order.

- This change is only gated on Windows where the Java classpath argument now
has the generated Jar (which references the other ones), instead of the long
list which is giving the problem. For non-Windows OS, the behavior is just the
same as before.

- We do not explicitly unjar the file, we feed it to Jave to lookup the
classpath elements from the manifest.

- The patch also includes testing performance changes (in TestJvmManager): in
Windows, launchTask is only affected by ~3%, in Linux no impact since the logic
is the same.

I have attached a patch with the changes (including test changes)

Classpath exceeds maximum OS limit
--

Key: HADOOP-8899
URL: https://issues.apache.org/jira/browse/HADOOP-8899
Project: Hadoop Common
Issue Type: Bug
Affects Versions: 1-win
Reporter: Ahmed El Baz
Attachments: HADOOP-8899.branch-1-win.classpath.patch

[jira] [Updated] (HADOOP-8899) Classpath exceeds maximum OS limit

2012-10-09 Thread Ahmed El Baz (JIRA)


 [ 
https://issues.apache.org/jira/browse/HADOOP-8899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ahmed El Baz updated HADOOP-8899:
-

Status: Patch Available  (was: Open)

 Classpath exceeds maximum OS limit
 --

 Key: HADOOP-8899
 URL: https://issues.apache.org/jira/browse/HADOOP-8899
 Project: Hadoop Common
  Issue Type: Bug
Affects Versions: 1-win
Reporter: Ahmed El Baz
 Attachments: HADOOP-8899.branch-1-win.classpath.patch


 We hit errors in TT due to long classpath value. One example is Oozie trying 
 to start a hive job, and it fails with the following error:
  java.io.IOException: Command exceeds the OS command length limit: 8192, 
 command: set CLASSPATH=….
 The classpath includes Hadoop Core + Hive Jars which are in the distributed 
 cache. This is causing the classpath to be too long giving the error above. A 
 viable long term fix is to generate a temporary JAR file in the task cache 
 directory which includes all elements in the long classpath, and then provide 
 this as the -classpath argument for the JVM to be spawned. This is gated 
 for Windows only.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-8899) Classpath exceeds maximum OS limit

[
https://issues.apache.org/jira/browse/HADOOP-8899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13472718#comment-13472718
]

Hadoop QA commented on HADOOP-8899:
---

{color:red}-1 overall{color}. Here are the results of testing the latest
attachment

http://issues.apache.org/jira/secure/attachment/12548456/HADOOP-8899.branch-1-win.classpath.patch
against trunk revision .

{color:red}-1 patch{color}. The patch command could not apply the patch.

Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/1591//console

This message is automatically generated.

Classpath exceeds maximum OS limit
--

[jira] [Updated] (HADOOP-8608) Add Configuration API for parsing time durations

2012-10-09 Thread Chris Douglas (JIRA)

[
https://issues.apache.org/jira/browse/HADOOP-8608?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Chris Douglas updated HADOOP-8608:
--

Attachment: 8608-2.patch

Updated patch.

More complex parsing- units, fractions, composites like 1d3h2s, etc.- seems
like overkill for the use cases we have.

Add Configuration API for parsing time durations

Key: HADOOP-8608
URL: https://issues.apache.org/jira/browse/HADOOP-8608
Project: Hadoop Common
Issue Type: Improvement
Components: conf
Affects Versions: 3.0.0
Reporter: Todd Lipcon
Attachments: 8608-0.patch, 8608-1.patch, 8608-2.patch

Hadoop has a lot of configurations which specify durations or intervals of
time. Unfortunately these different configurations have little consistency in
units - eg some are in milliseconds, some in seconds, and some in minutes.
This makes it difficult for users to configure, since they have to always
refer back to docs to remember the unit for each property.
The proposed solution is to add an API like {{Configuration.getTimeDuration}}
which allows the user to specify the units with a postfix. For example,
10ms, 10s, 10m, 10h, or even 10d. For backwards-compatibility, if
the user does not specify a unit, the API can specify the default unit, and
warn the user that they should specify an explicit unit instead.

[jira] [Commented] (HADOOP-8608) Add Configuration API for parsing time durations

2012-10-09 Thread Chris Douglas (JIRA)

[
https://issues.apache.org/jira/browse/HADOOP-8608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13472734#comment-13472734
]

Chris Douglas commented on HADOOP-8608:
---

But again: it's a matter of taste, so if someone wants to be more thorough,
they should feel free take point. It may change the return type and we don't
need multiple time-oriented APIs to configuration vars.

Add Configuration API for parsing time durations

Key: HADOOP-8608
URL: https://issues.apache.org/jira/browse/HADOOP-8608
Project: Hadoop Common
Issue Type: Improvement
Components: conf
Affects Versions: 3.0.0
Reporter: Todd Lipcon
Priority: Minor
Attachments: 8608-0.patch, 8608-1.patch, 8608-2.patch

[jira] [Updated] (HADOOP-8907) Provide means to look for zlib1.dll next to hadoop.dll on Windows

2012-10-09 Thread Aaron T. Myers (JIRA)


 [ 
https://issues.apache.org/jira/browse/HADOOP-8907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aaron T. Myers updated HADOOP-8907:
---

Summary: Provide means to look for zlib1.dll next to hadoop.dll on Windows  
(was: Provide means to look for zlib1.dll next to haodop.dll on Windows)

 Provide means to look for zlib1.dll next to hadoop.dll on Windows
 -

 Key: HADOOP-8907
 URL: https://issues.apache.org/jira/browse/HADOOP-8907
 Project: Hadoop Common
  Issue Type: Bug
Affects Versions: 1-win
Reporter: Ivan Mitic
Assignee: Ivan Mitic

 This change is dependent on HADOOP-8564.
 Instead of asking end users to install zlib1.dll to system32 on Windows, it 
 would be more convenient to just copy the dll next to hadoop.dll. 
 More context and patch coming after HADOOP-8564.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Assigned] (HADOOP-8906) paths with multiple globs are unreliable


 [ 
https://issues.apache.org/jira/browse/HADOOP-8906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daryn Sharp reassigned HADOOP-8906:
---

Assignee: Daryn Sharp

 paths with multiple globs are unreliable
 

 Key: HADOOP-8906
 URL: https://issues.apache.org/jira/browse/HADOOP-8906
 Project: Hadoop Common
  Issue Type: Bug
  Components: fs
Affects Versions: 0.23.0, 2.0.0-alpha, 3.0.0
Reporter: Daryn Sharp
Assignee: Daryn Sharp
Priority: Critical

 Let's say we have have a structure of $date/$user/stuff/file.  Multiple 
 globs are unreliable unless every directory in the structure exists.
 These work:
 date*/user
 date*/user/stuff
 date*/user/stuff/file
 These fail:
 date*/user/*
 date*/user/*/*
 date*/user/stu*
 date*/user/stu*/*
 date*/user/stu*/file
 date*/user/stuff/*
 date*/user/stuff/f*

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-8906) paths with multiple globs are unreliable


[ 
https://issues.apache.org/jira/browse/HADOOP-8906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13472752#comment-13472752
 ] 

Daryn Sharp commented on HADOOP-8906:
-

This appears to be the unfortunate fallout of the change to make {{listStatus}} 
throw an exception when the path doesn't exist.

 paths with multiple globs are unreliable
 

 Key: HADOOP-8906
 URL: https://issues.apache.org/jira/browse/HADOOP-8906
 Project: Hadoop Common
  Issue Type: Bug
  Components: fs
Affects Versions: 0.23.0, 2.0.0-alpha, 3.0.0
Reporter: Daryn Sharp
Priority: Critical

 Let's say we have have a structure of $date/$user/stuff/file.  Multiple 
 globs are unreliable unless every directory in the structure exists.
 These work:
 date*/user
 date*/user/stuff
 date*/user/stuff/file
 These fail:
 date*/user/*
 date*/user/*/*
 date*/user/stu*
 date*/user/stu*/*
 date*/user/stu*/file
 date*/user/stuff/*
 date*/user/stuff/f*

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-8887) Use a Maven plugin to build the native code using CMake


[ 
https://issues.apache.org/jira/browse/HADOOP-8887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13472758#comment-13472758
 ] 

Colin Patrick McCabe commented on HADOOP-8887:
--

It would be nice to spin it off as a separate project, but that seems like it 
would just make it more difficult to iterate on the design in the short term, 
which is what we will need to get it truly polished.  If it's a separate 
project, we'll have backwards compatibility constraints, and also need two 
changes (one to add a feature to it, another to update Hadoop to use the new 
version and features) for every change.

Does it make sense to put it in Hadoop for now, and then spin it off into 
another project later?

 Use a Maven plugin to build the native code using CMake
 ---

 Key: HADOOP-8887
 URL: https://issues.apache.org/jira/browse/HADOOP-8887
 Project: Hadoop Common
  Issue Type: Improvement
  Components: build
Affects Versions: 2.0.3-alpha
Reporter: Colin Patrick McCabe
Assignee: Colin Patrick McCabe
Priority: Minor
 Attachments: HADOOP-8887.001.patch, HADOOP-8887.002.patch, 
 HADOOP-8887.003.patch, HADOOP-8887.004.patch


 Currently, we build the native code using ant-build invocations.  Although 
 this works, it has some limitations:
 * compiler warning messages are hidden, which can cause people to check in 
 code with warnings unintentionally
 * there is no framework for running native unit tests; instead, we use ad-hoc 
 constructs involving shell scripts
 * the antrun code is very platform specific
 * there is no way to run a specific native unit test
 * it's more or less impossible for scripts like test-patch.sh to separate a 
 native test failing from the build itself failing (no files are created) or 
 to enumerate which native tests failed.
 Using a native Maven plugin would overcome these limitations.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HADOOP-8076) hadoop (1.x) ant build fetches ivy JAR every time


 [ 
https://issues.apache.org/jira/browse/HADOOP-8076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alejandro Abdelnur updated HADOOP-8076:
---

Status: Open  (was: Patch Available)

 hadoop (1.x) ant build fetches ivy JAR every time
 -

 Key: HADOOP-8076
 URL: https://issues.apache.org/jira/browse/HADOOP-8076
 Project: Hadoop Common
  Issue Type: Improvement
  Components: build
Affects Versions: 1.0.0
Reporter: Alejandro Abdelnur
Assignee: Alejandro Abdelnur
 Attachments: HADOOP-8076.patch


 the get .. task does a timestamp check, as ivy JAR is final release it 
 should use the skip if already downloaded check.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HADOOP-8292) TableMapping does not refresh when topology is updated


 [ 
https://issues.apache.org/jira/browse/HADOOP-8292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alejandro Abdelnur updated HADOOP-8292:
---

Status: Open  (was: Patch Available)

canceling patch for now as it may need more discussion.

 TableMapping does not refresh when topology is updated
 --

 Key: HADOOP-8292
 URL: https://issues.apache.org/jira/browse/HADOOP-8292
 Project: Hadoop Common
  Issue Type: Bug
Affects Versions: 2.0.0-alpha
Reporter: Philip Zeyliger
Assignee: Alejandro Abdelnur
 Attachments: HADOOP-8292.patch, HADOOP-8292.patch


 HADOOP-7030 introduced TableMapping, an implementation of DNSToSwitchMapping 
 which uses a file to map from IPs/hosts to their racks.  It's intended to 
 replace ScriptBasedMapping for cases where the latter was just a complicated 
 way of looking up the rack in a file.
 Though there was discussion of it on the JIRA, the TableMapping 
 implementation is not 'refreshable'.  i.e., if you want to add a host to your 
 cluster, and that host wasn't in the topology file to begin with, it will 
 never be added.
 TableMapping should refresh, either based on a command that can be executed, 
 or, perhaps, if the file on disk changes.
 I'll also point out that TableMapping extends CachedDNSToSwitchMapping, but, 
 since it does no refreshing, I don't see what the caching gets you: I think 
 the cache ends up being a second copy of the underlying map, always.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HADOOP-8883) Anonymous fallback in KerberosAuthenticator is broken

2012-10-09 Thread Robert Kanter (JIRA)


 [ 
https://issues.apache.org/jira/browse/HADOOP-8883?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Kanter updated HADOOP-8883:
--

Priority: Major  (was: Minor)

 Anonymous fallback in KerberosAuthenticator is broken
 -

 Key: HADOOP-8883
 URL: https://issues.apache.org/jira/browse/HADOOP-8883
 Project: Hadoop Common
  Issue Type: Bug
Affects Versions: 2.0.3-alpha
Reporter: Robert Kanter
Assignee: Robert Kanter
  Labels: security
 Fix For: 2.0.3-alpha

 Attachments: HADOOP-8883.patch


 HADOOP-8855 changed KerberosAuthenticator to handle when the JDK did the 
 SPNEGO already; but this change broke using the fallback authenticator 
 (PseudoAuthenticator) with an anonymous user (see OOZIE-1010).  

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-8887) Use a Maven plugin to build the native code using CMake

[
https://issues.apache.org/jira/browse/HADOOP-8887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13472786#comment-13472786
]

Alejandro Abdelnur commented on HADOOP-8887:

* plugin root package should be org.apache.hadoop if in Hadoop.

* source directory should be settable via an 'source' property which defaults
to ${basedir}/src/main/native

* build directory should be settable via an 'outputDirectory' property which
defaults to ${project.build.directory}/native if not set.

* what is the diff between the output and target params in the CompileMojo? Do
we need both? see prev comment on naming.

* CleanMojo, why do we need this one? 'mvn clean' already takes care of it.

* what is the diff between cmake-generate and cmake-compile? Do we need 2
different Mojos? Do we gain something from it?

Use a Maven plugin to build the native code using CMake
---

Key: HADOOP-8887
URL: https://issues.apache.org/jira/browse/HADOOP-8887
Project: Hadoop Common
Issue Type: Improvement
Components: build
Affects Versions: 2.0.3-alpha
Reporter: Colin Patrick McCabe
Assignee: Colin Patrick McCabe
Priority: Minor
Attachments: HADOOP-8887.001.patch, HADOOP-8887.002.patch,
HADOOP-8887.003.patch, HADOOP-8887.004.patch

Currently, we build the native code using ant-build invocations. Although
this works, it has some limitations:
* compiler warning messages are hidden, which can cause people to check in
code with warnings unintentionally
* there is no framework for running native unit tests; instead, we use ad-hoc
constructs involving shell scripts
* the antrun code is very platform specific
* there is no way to run a specific native unit test
* it's more or less impossible for scripts like test-patch.sh to separate a
native test failing from the build itself failing (no files are created) or
to enumerate which native tests failed.
Using a native Maven plugin would overcome these limitations.

[jira] [Assigned] (HADOOP-6496) HttpServer sends wrong content-type for CSS files (and others)

2012-10-09 Thread Ivan Mitic (JIRA)

[
https://issues.apache.org/jira/browse/HADOOP-6496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Ivan Mitic reassigned HADOOP-6496:
--

Assignee: Ivan Mitic (was: Todd Lipcon)

Assigning to myself to prepare the backport patch. The initial backport patch I
attached introduced unit test regression, will post a new patch soon.

HttpServer sends wrong content-type for CSS files (and others)
--

Key: HADOOP-6496
URL: https://issues.apache.org/jira/browse/HADOOP-6496
Project: Hadoop Common
Issue Type: Bug
Affects Versions: 0.21.0, 0.22.0
Reporter: Lars Francke
Assignee: Ivan Mitic
Priority: Minor
Fix For: 0.22.0

Attachments: HADOOP-6496.branch-1.1.backport.patch, hadoop-6496.txt,
hadoop-6496.txt

CSS files are send as text/html causing problems if the HTML page is rendered
in standards mode. The HDFS interface for example still works because it is
rendered in quirks mode, the HBase interface doesn't work because it is
rendered in standards mode. See HBASE-2110 for more details.
I've had a quick look at HttpServer but I'm too unfamiliar with it to see the
problem. I think this started happening with HADOOP-6441 which would lead me
to believe that the filter is called for every request and not only *.jsp and
*.html. I'd consider this a bug but I don't know enough about this to provide
a fix.

[jira] [Updated] (HADOOP-8906) paths with multiple globs are unreliable


 [ 
https://issues.apache.org/jira/browse/HADOOP-8906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daryn Sharp updated HADOOP-8906:


Attachment: HADOOP-8906.patch

Here's what I think will work for multi-level globs.  Haven't checked the other 
issues in the comments yet.

 paths with multiple globs are unreliable
 

 Key: HADOOP-8906
 URL: https://issues.apache.org/jira/browse/HADOOP-8906
 Project: Hadoop Common
  Issue Type: Bug
  Components: fs
Affects Versions: 0.23.0, 2.0.0-alpha, 3.0.0
Reporter: Daryn Sharp
Assignee: Daryn Sharp
Priority: Critical
 Attachments: HADOOP-8906.patch


 Let's say we have have a structure of $date/$user/stuff/file.  Multiple 
 globs are unreliable unless every directory in the structure exists.
 These work:
 date*/user
 date*/user/stuff
 date*/user/stuff/file
 These fail:
 date*/user/*
 date*/user/*/*
 date*/user/stu*
 date*/user/stu*/*
 date*/user/stu*/file
 date*/user/stuff/*
 date*/user/stuff/f*

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-8906) paths with multiple globs are unreliable


[ 
https://issues.apache.org/jira/browse/HADOOP-8906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13472818#comment-13472818
 ] 

Daryn Sharp commented on HADOOP-8906:
-

It doesn't fix all cases though...  There's another place.

 paths with multiple globs are unreliable
 

 Key: HADOOP-8906
 URL: https://issues.apache.org/jira/browse/HADOOP-8906
 Project: Hadoop Common
  Issue Type: Bug
  Components: fs
Affects Versions: 0.23.0, 2.0.0-alpha, 3.0.0
Reporter: Daryn Sharp
Assignee: Daryn Sharp
Priority: Critical
 Attachments: HADOOP-8906.patch


 Let's say we have have a structure of $date/$user/stuff/file.  Multiple 
 globs are unreliable unless every directory in the structure exists.
 These work:
 date*/user
 date*/user/stuff
 date*/user/stuff/file
 These fail:
 date*/user/*
 date*/user/*/*
 date*/user/stu*
 date*/user/stu*/*
 date*/user/stu*/file
 date*/user/stuff/*
 date*/user/stuff/f*

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HADOOP-8908) 'winutils.exe' code refactory

Chuan Liu created HADOOP-8908:
-

 Summary: 'winutils.exe' code refactory 
 Key: HADOOP-8908
 URL: https://issues.apache.org/jira/browse/HADOOP-8908
 Project: Hadoop Common
  Issue Type: Improvement
Affects Versions: 1-win
Reporter: Chuan Liu
Priority: Minor
 Fix For: 1-win


We want to split the existing 'winutil.exe' code into a library project and a 
executable project. The library project will be able to be linked and used in 
future projects. It is also good software engineering practice to introduce the 
modularity into the project.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HADOOP-8908) 'winutils.exe' code refactory


 [ 
https://issues.apache.org/jira/browse/HADOOP-8908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chuan Liu updated HADOOP-8908:
--

Description: We want to split the existing 'winutil.exe' code into a 
library project and an executable project. The library project will generate a 
static library, which will be able to be linked and used in future projects. It 
is also good software engineering practice to introduce the modularity into the 
project.  (was: We want to split the existing 'winutil.exe' code into a library 
project and a executable project. The library project will be able to be linked 
and used in future projects. It is also good software engineering practice to 
introduce the modularity into the project.)

 'winutils.exe' code refactory 
 --

 Key: HADOOP-8908
 URL: https://issues.apache.org/jira/browse/HADOOP-8908
 Project: Hadoop Common
  Issue Type: Improvement
Affects Versions: 1-win
Reporter: Chuan Liu
Priority: Minor
 Fix For: 1-win


 We want to split the existing 'winutil.exe' code into a library project and 
 an executable project. The library project will generate a static library, 
 which will be able to be linked and used in future projects. It is also good 
 software engineering practice to introduce the modularity into the project.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-8887) Use a Maven plugin to build the native code using CMake

[
https://issues.apache.org/jira/browse/HADOOP-8887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13472842#comment-13472842
]

Colin Patrick McCabe commented on HADOOP-8887:
--

bq. plugin root package should be org.apache.hadoop if in Hadoop.

OK.

bq. source directory should be settable via an 'source' property which defaults
to ${basedir}/src/main/native

It is, via GenerateMojo#source

bq. build directory should be settable via an 'outputDirectory' property which
defaults to ${project.build.directory}/native if not set.

It is, via GenerateMojo#output and CompileMojo#output.

bq. what is the diff between the output and target params in the CompileMojo?
Do we need both? see prev comment on naming.

Build target would be something like Debug, Production, etc. output is a
directory. I will add a comment explaining this.

bq. CleanMojo, why do we need this one? 'mvn clean' already takes care of it.

'mvn clean' will delete the 'target' directory, but we don't enforce the
concept that the CMake-ng output directory is inside that directory. We could
enforce this, and then make get rid of the clean target? However, we also
might need this for when we're supporting Windows, maybe?

bq. what is the diff between cmake-generate and cmake-compile? Do we need 2
different Mojos? Do we gain something from it?

cmake-generate runs the cmake application to create the Makefiles.
cmake-compile actually runs Make on these generated files. It seems natural to
separate these two steps. However, I don't have a specific reason why it has
to be implemented this way -- we could combine both steps into one. I was
trying to go with the spirit of Maven, which separates code generation and
compilation.

Use a Maven plugin to build the native code using CMake
---

[jira] [Updated] (HADOOP-8900) BuiltInGzipDecompressor : java.io.IOException: stored gzip size doesn't match decompressed size (Slavik Krassovsky)

2012-10-09 Thread Andy Isaacson (JIRA)


 [ 
https://issues.apache.org/jira/browse/HADOOP-8900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andy Isaacson updated HADOOP-8900:
--

Attachment: hadoop8900-2.txt

New patch for trunk:
* fix all examples of {{long0x}} in the tree (adds TestVLong).
* verified that 4GB+1 is the relevant edge case, 2GB+1 does not trigger the 
failure.

I'm still a bit unhappy at the long runtime, but 100 seconds is not *that* long 
by the standards of this test suite, so maybe it's worthwhile.

 BuiltInGzipDecompressor : java.io.IOException: stored gzip size doesn't match 
 decompressed size (Slavik Krassovsky)
 ---

 Key: HADOOP-8900
 URL: https://issues.apache.org/jira/browse/HADOOP-8900
 Project: Hadoop Common
  Issue Type: Bug
Affects Versions: 1-win, 2.0.1-alpha
 Environment: Encountered failure when processing large GZIP file
Reporter: Slavik Krassovsky
Assignee: Andy Isaacson
 Attachments: BuiltInGzipDecompressor2.patch, hadoop8900-2.txt, 
 hadoop8900.txt


 Encountered failure when processing large GZIP file
 • Gz: Failed in 1hrs, 13mins, 57sec with the error:
  ¸java.io.IOException: IO error in map input file 
 hdfs://localhost:9000/Halo4/json_m/gz/NewFileCat.txt.gz
  at 
 org.apache.hadoop.mapred.MapTask$TrackedRecordReader.moveToNext(MapTask.java:242)
  at 
 org.apache.hadoop.mapred.MapTask$TrackedRecordReader.next(MapTask.java:216)
  at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:48)
  at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:435)
  at org.apache.hadoop.mapred.MapTask.run(MapTask.java:371)
  at org.apache.hadoop.mapred.Child$4.run(Child.java:266)
  at java.security.AccessController.doPrivileged(Native Method)
  at javax.security.auth.Subject.doAs(Subject.java:415)
  at 
 org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
  at org.apache.hadoop.mapred.Child.main(Child.java:260)
  Caused by: java.io.IOException: stored gzip size doesn't match decompressed 
 size
  at 
 org.apache.hadoop.io.compress.zlib.BuiltInGzipDecompressor.executeTrailerState(BuiltInGzipDecompressor.java:389)
  at 
 org.apache.hadoop.io.compress.zlib.BuiltInGzipDecompressor.decompress(BuiltInGzipDecompressor.java:224)
  at 
 org.apache.hadoop.io.compress.DecompressorStream.decompress(DecompressorStream.java:82)
  at 
 org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:76)
  at java.io.InputStream.read(InputStream.java:102)
  at org.apache.hadoop.util.LineReader.readLine(LineReader.java:134)
  at org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:136)
  at org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:40)
  at 
 org.apache.hadoop.hive.ql.io.HiveRecordReader.doNext(HiveRecordReader.java:66)
  at 
 org.apache.hadoop.hive.ql.io.HiveRecordReader.doNext(HiveRecordReader.java:32)
  at 
 org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.next(HiveContextAwareRecordReader.java:67)
  at 
 org.apache.hadoop.mapred.MapTask$TrackedRecordReader.moveToNext(MapTask.java:236)
  ... 9 more

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-8900) BuiltInGzipDecompressor : java.io.IOException: stored gzip size doesn't match decompressed size (Slavik Krassovsky)


[ 
https://issues.apache.org/jira/browse/HADOOP-8900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13472851#comment-13472851
 ] 

Colin Patrick McCabe commented on HADOOP-8900:
--

It seems like the long runtime could be avoided by implementing an InputStream 
subclass that returns synthetic data.  GZip is a block-based compression 
algorithm-- could one simply keep returning the same block over and over, 
followed by the 4-byte footer?

It's kind of annoying to have to use 4GB of temporary space for the unit tests 
as well.  I only have 9GB free in total on my drive now-- a test like this 
could easily push me over the limit.

 BuiltInGzipDecompressor : java.io.IOException: stored gzip size doesn't match 
 decompressed size (Slavik Krassovsky)
 ---

 Key: HADOOP-8900
 URL: https://issues.apache.org/jira/browse/HADOOP-8900
 Project: Hadoop Common
  Issue Type: Bug
Affects Versions: 1-win, 2.0.1-alpha
 Environment: Encountered failure when processing large GZIP file
Reporter: Slavik Krassovsky
Assignee: Andy Isaacson
 Attachments: BuiltInGzipDecompressor2.patch, hadoop8900-2.txt, 
 hadoop8900.txt


 Encountered failure when processing large GZIP file
 • Gz: Failed in 1hrs, 13mins, 57sec with the error:
  ¸java.io.IOException: IO error in map input file 
 hdfs://localhost:9000/Halo4/json_m/gz/NewFileCat.txt.gz
  at 
 org.apache.hadoop.mapred.MapTask$TrackedRecordReader.moveToNext(MapTask.java:242)
  at 
 org.apache.hadoop.mapred.MapTask$TrackedRecordReader.next(MapTask.java:216)
  at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:48)
  at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:435)
  at org.apache.hadoop.mapred.MapTask.run(MapTask.java:371)
  at org.apache.hadoop.mapred.Child$4.run(Child.java:266)
  at java.security.AccessController.doPrivileged(Native Method)
  at javax.security.auth.Subject.doAs(Subject.java:415)
  at 
 org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
  at org.apache.hadoop.mapred.Child.main(Child.java:260)
  Caused by: java.io.IOException: stored gzip size doesn't match decompressed 
 size
  at 
 org.apache.hadoop.io.compress.zlib.BuiltInGzipDecompressor.executeTrailerState(BuiltInGzipDecompressor.java:389)
  at 
 org.apache.hadoop.io.compress.zlib.BuiltInGzipDecompressor.decompress(BuiltInGzipDecompressor.java:224)
  at 
 org.apache.hadoop.io.compress.DecompressorStream.decompress(DecompressorStream.java:82)
  at 
 org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:76)
  at java.io.InputStream.read(InputStream.java:102)
  at org.apache.hadoop.util.LineReader.readLine(LineReader.java:134)
  at org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:136)
  at org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:40)
  at 
 org.apache.hadoop.hive.ql.io.HiveRecordReader.doNext(HiveRecordReader.java:66)
  at 
 org.apache.hadoop.hive.ql.io.HiveRecordReader.doNext(HiveRecordReader.java:32)
  at 
 org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.next(HiveContextAwareRecordReader.java:67)
  at 
 org.apache.hadoop.mapred.MapTask$TrackedRecordReader.moveToNext(MapTask.java:236)
  ... 9 more

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-8900) BuiltInGzipDecompressor : java.io.IOException: stored gzip size doesn't match decompressed size (Slavik Krassovsky)


[ 
https://issues.apache.org/jira/browse/HADOOP-8900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13472855#comment-13472855
 ] 

Colin Patrick McCabe commented on HADOOP-8900:
--

Another, perhaps easier way to avoid the disk space cost would be to have a 
gzip output stream and feed it into the gzip input stream (never touch disk).

 BuiltInGzipDecompressor : java.io.IOException: stored gzip size doesn't match 
 decompressed size (Slavik Krassovsky)
 ---

 Key: HADOOP-8900
 URL: https://issues.apache.org/jira/browse/HADOOP-8900
 Project: Hadoop Common
  Issue Type: Bug
Affects Versions: 1-win, 2.0.1-alpha
 Environment: Encountered failure when processing large GZIP file
Reporter: Slavik Krassovsky
Assignee: Andy Isaacson
 Attachments: BuiltInGzipDecompressor2.patch, hadoop8900-2.txt, 
 hadoop8900.txt


 Encountered failure when processing large GZIP file
 • Gz: Failed in 1hrs, 13mins, 57sec with the error:
  ¸java.io.IOException: IO error in map input file 
 hdfs://localhost:9000/Halo4/json_m/gz/NewFileCat.txt.gz
  at 
 org.apache.hadoop.mapred.MapTask$TrackedRecordReader.moveToNext(MapTask.java:242)
  at 
 org.apache.hadoop.mapred.MapTask$TrackedRecordReader.next(MapTask.java:216)
  at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:48)
  at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:435)
  at org.apache.hadoop.mapred.MapTask.run(MapTask.java:371)
  at org.apache.hadoop.mapred.Child$4.run(Child.java:266)
  at java.security.AccessController.doPrivileged(Native Method)
  at javax.security.auth.Subject.doAs(Subject.java:415)
  at 
 org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
  at org.apache.hadoop.mapred.Child.main(Child.java:260)
  Caused by: java.io.IOException: stored gzip size doesn't match decompressed 
 size
  at 
 org.apache.hadoop.io.compress.zlib.BuiltInGzipDecompressor.executeTrailerState(BuiltInGzipDecompressor.java:389)
  at 
 org.apache.hadoop.io.compress.zlib.BuiltInGzipDecompressor.decompress(BuiltInGzipDecompressor.java:224)
  at 
 org.apache.hadoop.io.compress.DecompressorStream.decompress(DecompressorStream.java:82)
  at 
 org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:76)
  at java.io.InputStream.read(InputStream.java:102)
  at org.apache.hadoop.util.LineReader.readLine(LineReader.java:134)
  at org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:136)
  at org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:40)
  at 
 org.apache.hadoop.hive.ql.io.HiveRecordReader.doNext(HiveRecordReader.java:66)
  at 
 org.apache.hadoop.hive.ql.io.HiveRecordReader.doNext(HiveRecordReader.java:32)
  at 
 org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.next(HiveContextAwareRecordReader.java:67)
  at 
 org.apache.hadoop.mapred.MapTask$TrackedRecordReader.moveToNext(MapTask.java:236)
  ... 9 more

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-8887) Use a Maven plugin to build the native code using CMake


[ 
https://issues.apache.org/jira/browse/HADOOP-8887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13472857#comment-13472857
 ] 

Alejandro Abdelnur commented on HADOOP-8887:



* Unless I'm missing something, I don't see patch anything like the following 
to define defaults:

{code}
 * @parameter  expression=${source} 
default-value=${basedir}/src/main/native
 * @parameter  expression=${output} 
default-value=${project.build.directory}/native
{code}

* CleanMojo for windows, I don't see how, but if that is the case we should 
introduce it then if needed, not now just in case.

* generate/make, I'd merge them in a single Mojo as there are not usecases for 
using them apart, plus generate does not generate sources but make files.

* where test code lives? does compiled test stuff need to be segregated in 
target/test-native?

As an additional reason for removing clean  merging generate/make is that the 
amount of plugins config in the POMs are reduced from 3 to 1.

I have not run the patch yet, but I assume that ALL generated/created files are 
done under output and the source tree is not polluted with files created at 
build time.



 Use a Maven plugin to build the native code using CMake
 ---

 Key: HADOOP-8887
 URL: https://issues.apache.org/jira/browse/HADOOP-8887
 Project: Hadoop Common
  Issue Type: Improvement
  Components: build
Affects Versions: 2.0.3-alpha
Reporter: Colin Patrick McCabe
Assignee: Colin Patrick McCabe
Priority: Minor
 Attachments: HADOOP-8887.001.patch, HADOOP-8887.002.patch, 
 HADOOP-8887.003.patch, HADOOP-8887.004.patch


 Currently, we build the native code using ant-build invocations.  Although 
 this works, it has some limitations:
 * compiler warning messages are hidden, which can cause people to check in 
 code with warnings unintentionally
 * there is no framework for running native unit tests; instead, we use ad-hoc 
 constructs involving shell scripts
 * the antrun code is very platform specific
 * there is no way to run a specific native unit test
 * it's more or less impossible for scripts like test-patch.sh to separate a 
 native test failing from the build itself failing (no files are created) or 
 to enumerate which native tests failed.
 Using a native Maven plugin would overcome these limitations.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-8900) BuiltInGzipDecompressor : java.io.IOException: stored gzip size doesn't match decompressed size (Slavik Krassovsky)


[ 
https://issues.apache.org/jira/browse/HADOOP-8900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13472859#comment-13472859
 ] 

Slavik Krassovsky commented on HADOOP-8900:
---

Andy, let's go with your patch. 
+1
Could you please port it to branch-1 that that we could integrate it to 
branch-1-win 

 BuiltInGzipDecompressor : java.io.IOException: stored gzip size doesn't match 
 decompressed size (Slavik Krassovsky)
 ---

 Key: HADOOP-8900
 URL: https://issues.apache.org/jira/browse/HADOOP-8900
 Project: Hadoop Common
  Issue Type: Bug
Affects Versions: 1-win, 2.0.1-alpha
 Environment: Encountered failure when processing large GZIP file
Reporter: Slavik Krassovsky
Assignee: Andy Isaacson
 Attachments: BuiltInGzipDecompressor2.patch, hadoop8900-2.txt, 
 hadoop8900.txt


 Encountered failure when processing large GZIP file
 • Gz: Failed in 1hrs, 13mins, 57sec with the error:
  ¸java.io.IOException: IO error in map input file 
 hdfs://localhost:9000/Halo4/json_m/gz/NewFileCat.txt.gz
  at 
 org.apache.hadoop.mapred.MapTask$TrackedRecordReader.moveToNext(MapTask.java:242)
  at 
 org.apache.hadoop.mapred.MapTask$TrackedRecordReader.next(MapTask.java:216)
  at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:48)
  at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:435)
  at org.apache.hadoop.mapred.MapTask.run(MapTask.java:371)
  at org.apache.hadoop.mapred.Child$4.run(Child.java:266)
  at java.security.AccessController.doPrivileged(Native Method)
  at javax.security.auth.Subject.doAs(Subject.java:415)
  at 
 org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
  at org.apache.hadoop.mapred.Child.main(Child.java:260)
  Caused by: java.io.IOException: stored gzip size doesn't match decompressed 
 size
  at 
 org.apache.hadoop.io.compress.zlib.BuiltInGzipDecompressor.executeTrailerState(BuiltInGzipDecompressor.java:389)
  at 
 org.apache.hadoop.io.compress.zlib.BuiltInGzipDecompressor.decompress(BuiltInGzipDecompressor.java:224)
  at 
 org.apache.hadoop.io.compress.DecompressorStream.decompress(DecompressorStream.java:82)
  at 
 org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:76)
  at java.io.InputStream.read(InputStream.java:102)
  at org.apache.hadoop.util.LineReader.readLine(LineReader.java:134)
  at org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:136)
  at org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:40)
  at 
 org.apache.hadoop.hive.ql.io.HiveRecordReader.doNext(HiveRecordReader.java:66)
  at 
 org.apache.hadoop.hive.ql.io.HiveRecordReader.doNext(HiveRecordReader.java:32)
  at 
 org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.next(HiveContextAwareRecordReader.java:67)
  at 
 org.apache.hadoop.mapred.MapTask$TrackedRecordReader.moveToNext(MapTask.java:236)
  ... 9 more

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Assigned] (HADOOP-8908) 'winutils.exe' code refactory


 [ 
https://issues.apache.org/jira/browse/HADOOP-8908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chuan Liu reassigned HADOOP-8908:
-

Assignee: Chuan Liu

 'winutils.exe' code refactory 
 --

 Key: HADOOP-8908
 URL: https://issues.apache.org/jira/browse/HADOOP-8908
 Project: Hadoop Common
  Issue Type: Improvement
Affects Versions: 1-win
Reporter: Chuan Liu
Assignee: Chuan Liu
Priority: Minor
 Fix For: 1-win

 Attachments: HADOOP-8908-branch-1-win.patch


 We want to split the existing 'winutil.exe' code into a library project and 
 an executable project. The library project will generate a static library, 
 which will be able to be linked and used in future projects. It is also good 
 software engineering practice to introduce the modularity into the project.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HADOOP-8908) 'winutils.exe' code refactory


 [ 
https://issues.apache.org/jira/browse/HADOOP-8908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chuan Liu updated HADOOP-8908:
--

Attachment: HADOOP-8908-branch-1-win.patch

 'winutils.exe' code refactory 
 --

 Key: HADOOP-8908
 URL: https://issues.apache.org/jira/browse/HADOOP-8908
 Project: Hadoop Common
  Issue Type: Improvement
Affects Versions: 1-win
Reporter: Chuan Liu
Priority: Minor
 Fix For: 1-win

 Attachments: HADOOP-8908-branch-1-win.patch


 We want to split the existing 'winutil.exe' code into a library project and 
 an executable project. The library project will generate a static library, 
 which will be able to be linked and used in future projects. It is also good 
 software engineering practice to introduce the modularity into the project.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-8900) BuiltInGzipDecompressor : java.io.IOException: stored gzip size doesn't match decompressed size (Slavik Krassovsky)


[ 
https://issues.apache.org/jira/browse/HADOOP-8900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13472860#comment-13472860
 ] 

Slavik Krassovsky commented on HADOOP-8900:
---

Andy, let's go with your patch. 
+1
Could you please port it to branch-1 that that we could integrate it to 
branch-1-win

 BuiltInGzipDecompressor : java.io.IOException: stored gzip size doesn't match 
 decompressed size (Slavik Krassovsky)
 ---

 Key: HADOOP-8900
 URL: https://issues.apache.org/jira/browse/HADOOP-8900
 Project: Hadoop Common
  Issue Type: Bug
Affects Versions: 1-win, 2.0.1-alpha
 Environment: Encountered failure when processing large GZIP file
Reporter: Slavik Krassovsky
Assignee: Andy Isaacson
 Attachments: BuiltInGzipDecompressor2.patch, hadoop8900-2.txt, 
 hadoop8900.txt


 Encountered failure when processing large GZIP file
 • Gz: Failed in 1hrs, 13mins, 57sec with the error:
  ¸java.io.IOException: IO error in map input file 
 hdfs://localhost:9000/Halo4/json_m/gz/NewFileCat.txt.gz
  at 
 org.apache.hadoop.mapred.MapTask$TrackedRecordReader.moveToNext(MapTask.java:242)
  at 
 org.apache.hadoop.mapred.MapTask$TrackedRecordReader.next(MapTask.java:216)
  at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:48)
  at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:435)
  at org.apache.hadoop.mapred.MapTask.run(MapTask.java:371)
  at org.apache.hadoop.mapred.Child$4.run(Child.java:266)
  at java.security.AccessController.doPrivileged(Native Method)
  at javax.security.auth.Subject.doAs(Subject.java:415)
  at 
 org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
  at org.apache.hadoop.mapred.Child.main(Child.java:260)
  Caused by: java.io.IOException: stored gzip size doesn't match decompressed 
 size
  at 
 org.apache.hadoop.io.compress.zlib.BuiltInGzipDecompressor.executeTrailerState(BuiltInGzipDecompressor.java:389)
  at 
 org.apache.hadoop.io.compress.zlib.BuiltInGzipDecompressor.decompress(BuiltInGzipDecompressor.java:224)
  at 
 org.apache.hadoop.io.compress.DecompressorStream.decompress(DecompressorStream.java:82)
  at 
 org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:76)
  at java.io.InputStream.read(InputStream.java:102)
  at org.apache.hadoop.util.LineReader.readLine(LineReader.java:134)
  at org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:136)
  at org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:40)
  at 
 org.apache.hadoop.hive.ql.io.HiveRecordReader.doNext(HiveRecordReader.java:66)
  at 
 org.apache.hadoop.hive.ql.io.HiveRecordReader.doNext(HiveRecordReader.java:32)
  at 
 org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.next(HiveContextAwareRecordReader.java:67)
  at 
 org.apache.hadoop.mapred.MapTask$TrackedRecordReader.moveToNext(MapTask.java:236)
  ... 9 more

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-8887) Use a Maven plugin to build the native code using CMake

[
https://issues.apache.org/jira/browse/HADOOP-8887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13472864#comment-13472864
]

Colin Patrick McCabe commented on HADOOP-8887:
--

bq. [defaults]

I'll add defaults for those parameters.

bq. CleanMojo for windows, I don't see how, but if that is the case we should
introduce it then if needed, not now just in case.

My big concern is just that people will put their output directories somewhere
other than in a directory that gets auto-cleaned. I think if we skip the clean
step, we have to somehow enforce the constraint that the output directory gets
cleared by mvn clean; otherwise we're just creating hidden dangers for the
future.

bq. generate/make, I'd merge them in a single Mojo as there are not usecases
for using them apart, plus generate does not generate sources but make files.

OK. I'm fine with merging the cmake and make steps. It probably will make the
pom.xml files shorter, which we can always use.

bq. I have not run the patch yet, but I assume that ALL generated/created files
are done under output and the source tree is not polluted with files created at
build time.

Yep.

Use a Maven plugin to build the native code using CMake
---

[jira] [Commented] (HADOOP-8887) Use a Maven plugin to build the native code using CMake


[ 
https://issues.apache.org/jira/browse/HADOOP-8887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13472866#comment-13472866
 ] 

Alejandro Abdelnur commented on HADOOP-8887:


got it, ok on clean. on the defaults, make sure you have tests for them.

You've missed one question from prev comment: where test code lives? does 
compiled test stuff need to be segregated in target/test-native?


 Use a Maven plugin to build the native code using CMake
 ---

 Key: HADOOP-8887
 URL: https://issues.apache.org/jira/browse/HADOOP-8887
 Project: Hadoop Common
  Issue Type: Improvement
  Components: build
Affects Versions: 2.0.3-alpha
Reporter: Colin Patrick McCabe
Assignee: Colin Patrick McCabe
Priority: Minor
 Attachments: HADOOP-8887.001.patch, HADOOP-8887.002.patch, 
 HADOOP-8887.003.patch, HADOOP-8887.004.patch


 Currently, we build the native code using ant-build invocations.  Although 
 this works, it has some limitations:
 * compiler warning messages are hidden, which can cause people to check in 
 code with warnings unintentionally
 * there is no framework for running native unit tests; instead, we use ad-hoc 
 constructs involving shell scripts
 * the antrun code is very platform specific
 * there is no way to run a specific native unit test
 * it's more or less impossible for scripts like test-patch.sh to separate a 
 native test failing from the build itself failing (no files are created) or 
 to enumerate which native tests failed.
 Using a native Maven plugin would overcome these limitations.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-8900) BuiltInGzipDecompressor : java.io.IOException: stored gzip size doesn't match decompressed size (Slavik Krassovsky)


[ 
https://issues.apache.org/jira/browse/HADOOP-8900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13472867#comment-13472867
 ] 

Hadoop QA commented on HADOOP-8900:
---

{color:green}+1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12548490/hadoop8900-2.txt
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 2 new 
or modified test files.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  The javadoc tool did not generate any 
warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 core tests{color}.  The patch passed unit tests in 
hadoop-common-project/hadoop-common.

{color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: 
https://builds.apache.org/job/PreCommit-HADOOP-Build/1592//testReport/
Console output: 
https://builds.apache.org/job/PreCommit-HADOOP-Build/1592//console

This message is automatically generated.

 BuiltInGzipDecompressor : java.io.IOException: stored gzip size doesn't match 
 decompressed size (Slavik Krassovsky)
 ---

 Key: HADOOP-8900
 URL: https://issues.apache.org/jira/browse/HADOOP-8900
 Project: Hadoop Common
  Issue Type: Bug
Affects Versions: 1-win, 2.0.1-alpha
 Environment: Encountered failure when processing large GZIP file
Reporter: Slavik Krassovsky
Assignee: Andy Isaacson
 Attachments: BuiltInGzipDecompressor2.patch, hadoop8900-2.txt, 
 hadoop8900.txt


 Encountered failure when processing large GZIP file
 • Gz: Failed in 1hrs, 13mins, 57sec with the error:
  ¸java.io.IOException: IO error in map input file 
 hdfs://localhost:9000/Halo4/json_m/gz/NewFileCat.txt.gz
  at 
 org.apache.hadoop.mapred.MapTask$TrackedRecordReader.moveToNext(MapTask.java:242)
  at 
 org.apache.hadoop.mapred.MapTask$TrackedRecordReader.next(MapTask.java:216)
  at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:48)
  at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:435)
  at org.apache.hadoop.mapred.MapTask.run(MapTask.java:371)
  at org.apache.hadoop.mapred.Child$4.run(Child.java:266)
  at java.security.AccessController.doPrivileged(Native Method)
  at javax.security.auth.Subject.doAs(Subject.java:415)
  at 
 org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
  at org.apache.hadoop.mapred.Child.main(Child.java:260)
  Caused by: java.io.IOException: stored gzip size doesn't match decompressed 
 size
  at 
 org.apache.hadoop.io.compress.zlib.BuiltInGzipDecompressor.executeTrailerState(BuiltInGzipDecompressor.java:389)
  at 
 org.apache.hadoop.io.compress.zlib.BuiltInGzipDecompressor.decompress(BuiltInGzipDecompressor.java:224)
  at 
 org.apache.hadoop.io.compress.DecompressorStream.decompress(DecompressorStream.java:82)
  at 
 org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:76)
  at java.io.InputStream.read(InputStream.java:102)
  at org.apache.hadoop.util.LineReader.readLine(LineReader.java:134)
  at org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:136)
  at org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:40)
  at 
 org.apache.hadoop.hive.ql.io.HiveRecordReader.doNext(HiveRecordReader.java:66)
  at 
 org.apache.hadoop.hive.ql.io.HiveRecordReader.doNext(HiveRecordReader.java:32)
  at 
 org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.next(HiveContextAwareRecordReader.java:67)
  at 
 org.apache.hadoop.mapred.MapTask$TrackedRecordReader.moveToNext(MapTask.java:236)
  ... 9 more

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-8887) Use a Maven plugin to build the native code using CMake


[ 
https://issues.apache.org/jira/browse/HADOOP-8887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13472875#comment-13472875
 ] 

Colin Patrick McCabe commented on HADOOP-8887:
--

bq. does compiled test stuff need to be segregated in target/test-native?

No.

 Use a Maven plugin to build the native code using CMake
 ---

 Key: HADOOP-8887
 URL: https://issues.apache.org/jira/browse/HADOOP-8887
 Project: Hadoop Common
  Issue Type: Improvement
  Components: build
Affects Versions: 2.0.3-alpha
Reporter: Colin Patrick McCabe
Assignee: Colin Patrick McCabe
Priority: Minor
 Attachments: HADOOP-8887.001.patch, HADOOP-8887.002.patch, 
 HADOOP-8887.003.patch, HADOOP-8887.004.patch


 Currently, we build the native code using ant-build invocations.  Although 
 this works, it has some limitations:
 * compiler warning messages are hidden, which can cause people to check in 
 code with warnings unintentionally
 * there is no framework for running native unit tests; instead, we use ad-hoc 
 constructs involving shell scripts
 * the antrun code is very platform specific
 * there is no way to run a specific native unit test
 * it's more or less impossible for scripts like test-patch.sh to separate a 
 native test failing from the build itself failing (no files are created) or 
 to enumerate which native tests failed.
 Using a native Maven plugin would overcome these limitations.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Resolved] (HADOOP-8858) backport branch-trunk-win: HADOOP-8234 Enable user group mappings on Windows


 [ 
https://issues.apache.org/jira/browse/HADOOP-8858?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chris Nauroth resolved HADOOP-8858.
---

Resolution: Invalid

Discarding this Jira, because this functionality will go to trunk in a 
different form.

 backport branch-trunk-win: HADOOP-8234 Enable user group mappings on Windows
 

 Key: HADOOP-8858
 URL: https://issues.apache.org/jira/browse/HADOOP-8858
 Project: Hadoop Common
  Issue Type: Task
  Components: security
Affects Versions: trunk-win
Reporter: Chris Nauroth
Assignee: Chris Nauroth
 Attachments: HADOOP-8858-branch-trunk-win.patch


 Backport the code for HADOOP-8234 to enable user group mappings on Windows.  
 This code had been committed to branch-1-win.  This issue tracks backporting 
 to branch-trunk-win, in preparation for merging all the way to trunk.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HADOOP-8891) Remove DelegationTokenRenewer and its logic from FileSystems using it


 [ 
https://issues.apache.org/jira/browse/HADOOP-8891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Karthik Kambatla updated HADOOP-8891:
-

Summary: Remove DelegationTokenRenewer and its logic from FileSystems using 
it  (was: Remove DelegationTokenRenewer and its logic from WebHdfsFileSystem 
and HftpFileSystem)

 Remove DelegationTokenRenewer and its logic from FileSystems using it
 -

 Key: HADOOP-8891
 URL: https://issues.apache.org/jira/browse/HADOOP-8891
 Project: Hadoop Common
  Issue Type: Improvement
Affects Versions: 2.0.1-alpha
Reporter: Karthik Kambatla
Assignee: Karthik Kambatla
 Attachments: HADOOP-8891.patch, HADOOP-8891.patch


 Moved the HDFS part of HADOOP-8852 to HDFS-4009 along with other sub-tasks. 
 Created this to track the removal of DelegationTokenRenewer alone.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HADOOP-8891) Remove DelegationTokenRenewer and its logic from FileSystems using it


 [ 
https://issues.apache.org/jira/browse/HADOOP-8891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Karthik Kambatla updated HADOOP-8891:
-

Attachment: HADOOP-8891.patch

Updated patch addressing Daryn's comments.

TestHttpFSWithKerberos fails both before and after the patch.

Unlike WebHdfs and Hftp, HttpFSFileSystem doesn't check UGI for tokens at init. 
But, that is out of the scope of this JIRA - can create another for it.

 Remove DelegationTokenRenewer and its logic from FileSystems using it
 -

 Key: HADOOP-8891
 URL: https://issues.apache.org/jira/browse/HADOOP-8891
 Project: Hadoop Common
  Issue Type: Improvement
Affects Versions: 2.0.1-alpha
Reporter: Karthik Kambatla
Assignee: Karthik Kambatla
 Attachments: HADOOP-8891.patch, HADOOP-8891.patch, HADOOP-8891.patch


 Moved the HDFS part of HADOOP-8852 to HDFS-4009 along with other sub-tasks. 
 Created this to track the removal of DelegationTokenRenewer alone.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HADOOP-8891) Remove DelegationTokenRenewer and its logic from FileSystems using it


 [ 
https://issues.apache.org/jira/browse/HADOOP-8891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Karthik Kambatla updated HADOOP-8891:
-

Status: Patch Available  (was: Open)

 Remove DelegationTokenRenewer and its logic from FileSystems using it
 -

 Key: HADOOP-8891
 URL: https://issues.apache.org/jira/browse/HADOOP-8891
 Project: Hadoop Common
  Issue Type: Improvement
Affects Versions: 2.0.1-alpha
Reporter: Karthik Kambatla
Assignee: Karthik Kambatla
 Attachments: HADOOP-8891.patch, HADOOP-8891.patch, HADOOP-8891.patch


 Moved the HDFS part of HADOOP-8852 to HDFS-4009 along with other sub-tasks. 
 Created this to track the removal of DelegationTokenRenewer alone.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HADOOP-8909) Maven protoc calls must not depend on external sh script

Chris Nauroth created HADOOP-8909:
-

 Summary: Maven protoc calls must not depend on external sh script
 Key: HADOOP-8909
 URL: https://issues.apache.org/jira/browse/HADOOP-8909
 Project: Hadoop Common
  Issue Type: Improvement
  Components: build
Affects Versions: 3.0.0
Reporter: Chris Nauroth
Assignee: Chris Nauroth


Currently, several pom.xml files rely on external shell scripting to call 
protoc.  The sh binary may not be available on all developers' machines (e.g. 
Windows without Cygwin).  This issue tracks removal of that dependency.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HADOOP-8909) Maven protoc calls must not depend on external sh script

[
https://issues.apache.org/jira/browse/HADOOP-8909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Chris Nauroth updated HADOOP-8909:
--

Attachment: HADOOP-8909.patch

The attached patch removes the sh scripting and instead uses exec-maven-plugin,
which is the same way that the yarn modules call protoc. I tested the change
successfully on Mac, Windows with Cygwin, and Windows without Cygwin.

There is a small loss of functionality in that the shell script performed
wildcard expansion, so that developers could have added new .proto files
without needing to make corresponding updates to pom.xml. Adding new .proto
files is a rare event, so this shouldn't be too cumbersome. There is a
Protocol Buffers Maven plugin under development that could solve this more
elegantly, but it hasn't been released yet.

Maven protoc calls must not depend on external sh script

Key: HADOOP-8909
URL: https://issues.apache.org/jira/browse/HADOOP-8909
Project: Hadoop Common
Issue Type: Improvement
Components: build
Affects Versions: 3.0.0
Reporter: Chris Nauroth
Assignee: Chris Nauroth
Attachments: HADOOP-8909.patch

Currently, several pom.xml files rely on external shell scripting to call
protoc. The sh binary may not be available on all developers' machines (e.g.
Windows without Cygwin). This issue tracks removal of that dependency.

[jira] [Work started] (HADOOP-8909) Maven protoc calls must not depend on external sh script


 [ 
https://issues.apache.org/jira/browse/HADOOP-8909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on HADOOP-8909 started by Chris Nauroth.

 Maven protoc calls must not depend on external sh script
 

 Key: HADOOP-8909
 URL: https://issues.apache.org/jira/browse/HADOOP-8909
 Project: Hadoop Common
  Issue Type: Improvement
  Components: build
Affects Versions: 3.0.0
Reporter: Chris Nauroth
Assignee: Chris Nauroth
 Attachments: HADOOP-8909.patch


 Currently, several pom.xml files rely on external shell scripting to call 
 protoc.  The sh binary may not be available on all developers' machines (e.g. 
 Windows without Cygwin).  This issue tracks removal of that dependency.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-8908) 'winutils.exe' code refactory

2012-10-09 Thread Ivan Mitic (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-8908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13472910#comment-13472910
 ] 

Ivan Mitic commented on HADOOP-8908:


Thanks Chuan, I already reviewed this and it looks good to me.

 'winutils.exe' code refactory 
 --

 Key: HADOOP-8908
 URL: https://issues.apache.org/jira/browse/HADOOP-8908
 Project: Hadoop Common
  Issue Type: Improvement
Affects Versions: 1-win
Reporter: Chuan Liu
Assignee: Chuan Liu
Priority: Minor
 Fix For: 1-win

 Attachments: HADOOP-8908-branch-1-win.patch


 We want to split the existing 'winutil.exe' code into a library project and 
 an executable project. The library project will generate a static library, 
 which will be able to be linked and used in future projects. It is also good 
 software engineering practice to introduce the modularity into the project.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-8909) Maven protoc calls must not depend on external sh script

[
https://issues.apache.org/jira/browse/HADOOP-8909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13472914#comment-13472914
]

Hadoop QA commented on HADOOP-8909:
---

{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12548502/HADOOP-8909.patch
against trunk revision .

{color:green}+1 @author{color}. The patch does not contain any @author
tags.

{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.

{color:red}-1 javac{color:red}. The patch appears to cause the build to
fail.

Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/1594//console

This message is automatically generated.

Maven protoc calls must not depend on external sh script

[jira] [Commented] (HADOOP-8891) Remove DelegationTokenRenewer and its logic from FileSystems using it


[ 
https://issues.apache.org/jira/browse/HADOOP-8891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13472915#comment-13472915
 ] 

Hadoop QA commented on HADOOP-8891:
---

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12548498/HADOOP-8891.patch
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 2 new 
or modified test files.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  The javadoc tool did not generate any 
warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:red}-1 findbugs{color}.  The patch appears to introduce 1 new 
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 core tests{color}.  The patch passed unit tests in 
hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs 
hadoop-hdfs-project/hadoop-hdfs-httpfs.

{color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: 
https://builds.apache.org/job/PreCommit-HADOOP-Build/1593//testReport/
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HADOOP-Build/1593//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
Console output: 
https://builds.apache.org/job/PreCommit-HADOOP-Build/1593//console

This message is automatically generated.

 Remove DelegationTokenRenewer and its logic from FileSystems using it
 -

 Key: HADOOP-8891
 URL: https://issues.apache.org/jira/browse/HADOOP-8891
 Project: Hadoop Common
  Issue Type: Improvement
Affects Versions: 2.0.1-alpha
Reporter: Karthik Kambatla
Assignee: Karthik Kambatla
 Attachments: HADOOP-8891.patch, HADOOP-8891.patch, HADOOP-8891.patch


 Moved the HDFS part of HADOOP-8852 to HDFS-4009 along with other sub-tasks. 
 Created this to track the removal of DelegationTokenRenewer alone.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HADOOP-6311) Add support for unix domain sockets to JNI libs


 [ 
https://issues.apache.org/jira/browse/HADOOP-6311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Colin Patrick McCabe updated HADOOP-6311:
-

Attachment: HADOOP-6311.021.patch

Incorporate some style changes suggested by Andy Isaacson.

 Add support for unix domain sockets to JNI libs
 ---

 Key: HADOOP-6311
 URL: https://issues.apache.org/jira/browse/HADOOP-6311
 Project: Hadoop Common
  Issue Type: New Feature
  Components: native
Affects Versions: 0.20.0
Reporter: Todd Lipcon
Assignee: Colin Patrick McCabe
 Attachments: 6311-trunk-inprogress.txt, HADOOP-6311.014.patch, 
 HADOOP-6311.016.patch, HADOOP-6311.018.patch, HADOOP-6311.020b.patch, 
 HADOOP-6311.020.patch, HADOOP-6311.021.patch, HADOOP-6311-0.patch, 
 HADOOP-6311-1.patch, hadoop-6311.txt


 For HDFS-347 we need to use unix domain sockets. This JIRA is to include a 
 library in common which adds a o.a.h.net.unix package based on the code from 
 Android (apache 2 license)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-8909) Maven protoc calls must not depend on external sh script


[ 
https://issues.apache.org/jira/browse/HADOOP-8909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13472919#comment-13472919
 ] 

Chris Nauroth commented on HADOOP-8909:
---

The changes are in pom.xml files only, so no new tests.

 Maven protoc calls must not depend on external sh script
 

 Key: HADOOP-8909
 URL: https://issues.apache.org/jira/browse/HADOOP-8909
 Project: Hadoop Common
  Issue Type: Improvement
  Components: build
Affects Versions: 3.0.0
Reporter: Chris Nauroth
Assignee: Chris Nauroth
 Attachments: HADOOP-8909.patch


 Currently, several pom.xml files rely on external shell scripting to call 
 protoc.  The sh binary may not be available on all developers' machines (e.g. 
 Windows without Cygwin).  This issue tracks removal of that dependency.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-6311) Add support for unix domain sockets to JNI libs


[ 
https://issues.apache.org/jira/browse/HADOOP-6311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13472927#comment-13472927
 ] 

Hadoop QA commented on HADOOP-6311:
---

{color:green}+1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12548504/HADOOP-6311.021.patch
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 2 new 
or modified test files.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  The javadoc tool did not generate any 
warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 core tests{color}.  The patch passed unit tests in 
hadoop-common-project/hadoop-common.

{color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: 
https://builds.apache.org/job/PreCommit-HADOOP-Build/1595//testReport/
Console output: 
https://builds.apache.org/job/PreCommit-HADOOP-Build/1595//console

This message is automatically generated.

 Add support for unix domain sockets to JNI libs
 ---

 Key: HADOOP-6311
 URL: https://issues.apache.org/jira/browse/HADOOP-6311
 Project: Hadoop Common
  Issue Type: New Feature
  Components: native
Affects Versions: 0.20.0
Reporter: Todd Lipcon
Assignee: Colin Patrick McCabe
 Attachments: 6311-trunk-inprogress.txt, HADOOP-6311.014.patch, 
 HADOOP-6311.016.patch, HADOOP-6311.018.patch, HADOOP-6311.020b.patch, 
 HADOOP-6311.020.patch, HADOOP-6311.021.patch, HADOOP-6311-0.patch, 
 HADOOP-6311-1.patch, hadoop-6311.txt


 For HDFS-347 we need to use unix domain sockets. This JIRA is to include a 
 library in common which adds a o.a.h.net.unix package based on the code from 
 Android (apache 2 license)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-8908) 'winutils.exe' code refactory

2012-10-09 Thread Bikas Saha (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-8908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13472938#comment-13472938
 ] 

Bikas Saha commented on HADOOP-8908:


Looks like a simple renaming of files refactor to create libwinutils and 
changing references to it. +1

 'winutils.exe' code refactory 
 --

 Key: HADOOP-8908
 URL: https://issues.apache.org/jira/browse/HADOOP-8908
 Project: Hadoop Common
  Issue Type: Improvement
Affects Versions: 1-win
Reporter: Chuan Liu
Assignee: Chuan Liu
Priority: Minor
 Fix For: 1-win

 Attachments: HADOOP-8908-branch-1-win.patch


 We want to split the existing 'winutil.exe' code into a library project and 
 an executable project. The library project will generate a static library, 
 which will be able to be linked and used in future projects. It is also good 
 software engineering practice to introduce the modularity into the project.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-8869) Links at the bottom of the jobdetails page do not render correctly in IE9

2012-10-09 Thread Bikas Saha (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-8869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13472947#comment-13472947
 ] 

Bikas Saha commented on HADOOP-8869:


Looks like a simple fix. +1

 Links at the bottom of the jobdetails page do not render correctly in IE9
 -

 Key: HADOOP-8869
 URL: https://issues.apache.org/jira/browse/HADOOP-8869
 Project: Hadoop Common
  Issue Type: Bug
Affects Versions: 1-win
Reporter: Ivan Mitic
Assignee: Ivan Mitic
 Attachments: Fixed_IE_Chrome_FF.png, 
 HADOOP-8869.branch-1-win.ie_links.patch, IE9.png, OtherBrowsers.png


 See attached screen shoots IE9.png vs OtherBrowsers.png

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-8868) FileUtil#chmod should normalize the path before calling into shell APIs

2012-10-09 Thread Bikas Saha (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-8868?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13472952#comment-13472952
 ] 

Bikas Saha commented on HADOOP-8868:


So we are using JAVA API to resolve the path to a normalized form? Ideally the 
FileUtil method could take File arguments instead of strings but we'd like to 
avoid changing the public API.
In what cases can we get a mix of slashes on the string path?

 FileUtil#chmod should normalize the path before calling into shell APIs
 ---

 Key: HADOOP-8868
 URL: https://issues.apache.org/jira/browse/HADOOP-8868
 Project: Hadoop Common
  Issue Type: Bug
Affects Versions: 1-win
Reporter: Ivan Mitic
Assignee: Ivan Mitic
 Attachments: HADOOP-8868.branch-1-win.chmod.patch


 We have seen cases where paths passed in from FileUtil#chmod to Shell APIs 
 can contain both forward and backward slashes on Windows.
 This causes problems, since some Windows APIs do not work well with mixed 
 slashes.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HADOOP-8564) Port and extend Hadoop native libraries for Windows to address datanode concurrent reading and writing issue