date:20121017


 [ 
https://issues.apache.org/jira/browse/HADOOP-8937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Suresh Srinivas updated HADOOP-8937:


Labels:   (was: newbie)

removing newbie label.

 ClientProtocol should support a way to get DataNodeInfo for a particular data 
 node.
 ---

 Key: HADOOP-8937
 URL: https://issues.apache.org/jira/browse/HADOOP-8937
 Project: Hadoop Common
  Issue Type: Improvement
  Components: fs
Affects Versions: 1.0.3
Reporter: Sameer Vaishampayan
Priority: Minor
 Fix For: 2.0.3-alpha


 HBase project needs a way to find if a DataNode is running local or not on a 
 given host. The current way is too expensive in which getDatanodeReport needs 
 to be called which returns information for all data nodes in the cluster.
 https://issues.apache.org/jira/browse/HBASE-6398

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HADOOP-8937) ClientProtocol should support a way to get DataNodeInfo for a particular data node.


 [ 
https://issues.apache.org/jira/browse/HADOOP-8937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Suresh Srinivas updated HADOOP-8937:


Tags:   (was: noob)

 ClientProtocol should support a way to get DataNodeInfo for a particular data 
 node.
 ---

 Key: HADOOP-8937
 URL: https://issues.apache.org/jira/browse/HADOOP-8937
 Project: Hadoop Common
  Issue Type: Improvement
  Components: fs
Affects Versions: 1.0.3
Reporter: Sameer Vaishampayan
Priority: Minor
 Fix For: 2.0.3-alpha


 HBase project needs a way to find if a DataNode is running local or not on a 
 given host. The current way is too expensive in which getDatanodeReport needs 
 to be called which returns information for all data nodes in the cluster.
 https://issues.apache.org/jira/browse/HBASE-6398

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Resolved] (HADOOP-8936) Local users should be able to query a domain user's groups on Windows


 [ 
https://issues.apache.org/jira/browse/HADOOP-8936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Suresh Srinivas resolved HADOOP-8936.
-

  Resolution: Fixed
Hadoop Flags: Reviewed

+1 for the patch. I committed it to branch-1-win. Thank you Chuan.

 Local users should be able to query a domain user's groups on Windows
 -

 Key: HADOOP-8936
 URL: https://issues.apache.org/jira/browse/HADOOP-8936
 Project: Hadoop Common
  Issue Type: Bug
Affects Versions: 1-win
Reporter: Chuan Liu
Assignee: Chuan Liu
Priority: Minor
 Fix For: 1-win

 Attachments: HADOOP-8936-branch-1-win.patch


 When Hadoop run by a local user, and a domain user submit a job, Hadoop will 
 need to get the local groups for the domain user. This fails in 'winutils' 
 now because we tried to query domain controller for domain users and local 
 users does not have the permission to do so. We should fix the problem so 
 that local users should be able to query a domain user's local groups.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-8564) Port and extend Hadoop native libraries for Windows to address datanode concurrent reading and writing issue

[
https://issues.apache.org/jira/browse/HADOOP-8564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13477648#comment-13477648
]

Suresh Srinivas commented on HADOOP-8564:
-

Todd, please post if you have any comments. Otherwise I am going to commit this
tomorrow.

Port and extend Hadoop native libraries for Windows to address datanode
concurrent reading and writing issue

Key: HADOOP-8564
URL: https://issues.apache.org/jira/browse/HADOOP-8564
Project: Hadoop Common
Issue Type: Bug
Components: io
Affects Versions: 1-win
Reporter: Chuan Liu
Assignee: Chuan Liu
Attachments: HADOOP-8564-branch-1-win.patch

HDFS files are made up of blocks. First, let’s look at writing. When the data
is written to datanode, an active or temporary file is created to receive
packets. After the last packet for the block is received, we will finalize
the block. One step during finalization is to rename the block file to a new
directory. The relevant code can be found via the call sequence:
FSDataSet.finalizeBlockInternal - FSDir.addBlock.
{code}
if ( ! metaData.renameTo( newmeta ) ||
! src.renameTo( dest ) ) {
throw new IOException( could not move files for + b +
from tmp to +
dest.getAbsolutePath() );
}
{code}
Let’s then switch to reading. On HDFS, it is expected the client can also
read these unfinished blocks. So when the read calls from client reach
datanode, the datanode will open an input stream on the unfinished block file.
The problem comes in when the file is opened for reading while the datanode
receives last packet from client and try to rename the finished block file.
This operation will succeed on Linux, but not on Windows . The behavior can
be modified on Windows to open the file with FILE_SHARE_DELETE flag on, i.e.
sharing the delete (including renaming) permission with other processes while
opening the file. There is also a Java bug ([id
6357433|http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6357433]) reported
a while back on this. However, since this behavior exists for Java on Windows
since JDK 1.0, the Java developers do not want to break the backward
compatibility on this behavior. Instead, a new file system API is proposed in
JDK 7.
As outlined in the [Java forum|http://www.java.net/node/645421] by the Java
developer (kbr), there are three ways to fix the problem:
# Use different mechanism in the application in dealing with files.
# Create a new implementation of InputStream abstract class using Windows
native code.
# Patch JDK with a private patch that alters FileInputStream behavior.
For the third option, it cannot fix the problem for users using Oracle JDK.
We discussed some options for the first approach. For example one option is
to use two phase renaming, i.e. first hardlink; then remove the old hardlink
when read is finished. This option was thought to be rather pervasive.
Another option discussed is to change the HDFS behavior on Windows by not
allowing client reading unfinished blocks. However this behavior change is
thought to be problematic and may affect other application build on top of
HDFS.
For all the reasons discussed above, we will use the second approach to
address the problem.
If there are better options to fix the problem, we would also like to hear
about them.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HADOOP-8869) Links at the bottom of the jobdetails page do not render correctly in IE9


 [ 
https://issues.apache.org/jira/browse/HADOOP-8869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Suresh Srinivas updated HADOOP-8869:


  Resolution: Fixed
Hadoop Flags: Reviewed
  Status: Resolved  (was: Patch Available)

+1 for the patch.

I committed it to branch-1-win. Thank Ivan.

 Links at the bottom of the jobdetails page do not render correctly in IE9
 -

 Key: HADOOP-8869
 URL: https://issues.apache.org/jira/browse/HADOOP-8869
 Project: Hadoop Common
  Issue Type: Bug
Affects Versions: 1-win
Reporter: Ivan Mitic
Assignee: Ivan Mitic
 Attachments: Fixed_IE_Chrome_FF.png, 
 HADOOP-8869.branch-1-win.ie_links.patch, IE9.png, OtherBrowsers.png


 See attached screen shoots IE9.png vs OtherBrowsers.png

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-6496) HttpServer sends wrong content-type for CSS files (and others)


[ 
https://issues.apache.org/jira/browse/HADOOP-6496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13477658#comment-13477658
 ] 

Suresh Srinivas commented on HADOOP-6496:
-

Ivan, the branch-1.1 patch is missing the addition of file test.css, changes to 
build.xml, HttpServerFunctionalTest.java. Is that intentional?

 HttpServer sends wrong content-type for CSS files (and others)
 --

 Key: HADOOP-6496
 URL: https://issues.apache.org/jira/browse/HADOOP-6496
 Project: Hadoop Common
  Issue Type: Bug
Affects Versions: 0.21.0, 0.22.0
Reporter: Lars Francke
Assignee: Ivan Mitic
Priority: Minor
 Fix For: 0.22.0

 Attachments: HADOOP-6496.branch-1.1.backport.2.patch, 
 HADOOP-6496.branch-1.1.backport.patch, hadoop-6496.txt, hadoop-6496.txt


 CSS files are send as text/html causing problems if the HTML page is rendered 
 in standards mode. The HDFS interface for example still works because it is 
 rendered in quirks mode, the HBase interface doesn't work because it is 
 rendered in standards mode. See HBASE-2110 for more details.
 I've had a quick look at HttpServer but I'm too unfamiliar with it to see the 
 problem. I think this started happening with HADOOP-6441 which would lead me 
 to believe that the filter is called for every request and not only *.jsp and 
 *.html. I'd consider this a bug but I don't know enough about this to provide 
 a fix.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-8925) Remove packaging


[ 
https://issues.apache.org/jira/browse/HADOOP-8925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13477666#comment-13477666
 ] 

Hadoop QA commented on HADOOP-8925:
---

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12549441/hadoop-8925.txt
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:red}-1 tests included{color}.  The patch doesn't appear to include 
any new or modified tests.
Please justify why no new tests are needed for this 
patch.
Also please list what manual steps were performed to 
verify this patch.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  The javadoc tool did not generate any 
warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:red}-1 core tests{color}.  The patch failed these unit tests in 
hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs:

  
org.apache.hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks

{color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: 
https://builds.apache.org/job/PreCommit-HADOOP-Build/1639//testReport/
Console output: 
https://builds.apache.org/job/PreCommit-HADOOP-Build/1639//console

This message is automatically generated.

 Remove packaging
 

 Key: HADOOP-8925
 URL: https://issues.apache.org/jira/browse/HADOOP-8925
 Project: Hadoop Common
  Issue Type: Improvement
  Components: build
Affects Versions: 2.0.0-alpha
Reporter: Eli Collins
Assignee: Eli Collins
 Attachments: hadoop-8925.txt


 Per discussion on HADOOP-8809, now that Bigtop is TLP and supports Hadoop v2 
 let's remove the Hadoop packaging from trunk and branch-2. We should remove 
 it anyway since it no longer part of the build post mavenization, was not 
 updated post MR1 (there's no MR2/YARN packaging) and is not maintained.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-6496) HttpServer sends wrong content-type for CSS files (and others)

[
https://issues.apache.org/jira/browse/HADOOP-6496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13477669#comment-13477669
]

Ivan Mitic commented on HADOOP-6496:

Thanks for reviewing Suresh.

bq. the branch-1.1 patch is missing the addition of file test.css, changes to
build.xml, HttpServerFunctionalTest.java. Is that intentional?
HttpServerFunctionalTest does not exist in branch-1, that is why I don't have
any changes around it.
Other changes (build.xml/test.css) are only related to adding a new
{{test.css}} test file. However, that introduced unittest regressions in
branch-1, since Jetty ended up resolving resources against the local test path
{{build/test/webapps}}. Specifically, {{test.classpath}} would have
{{build/test/webapps}} before {{build/webapps}} causing many tests dependent on
files under {{build/webapps}} to fail, since HttpServer binds to an incorrect
path.

As an alternative, I used an existing {{/static/hadoop.css}} test file. This
way we have both the fix and a test covering the new functionality. Let me know
if this makes sense or if you have other proposals.

HttpServer sends wrong content-type for CSS files (and others)
--

Key: HADOOP-6496
URL: https://issues.apache.org/jira/browse/HADOOP-6496
Project: Hadoop Common
Issue Type: Bug
Affects Versions: 0.21.0, 0.22.0
Reporter: Lars Francke
Assignee: Ivan Mitic
Priority: Minor
Fix For: 0.22.0

Attachments: HADOOP-6496.branch-1.1.backport.2.patch,
HADOOP-6496.branch-1.1.backport.patch, hadoop-6496.txt, hadoop-6496.txt

CSS files are send as text/html causing problems if the HTML page is rendered
in standards mode. The HDFS interface for example still works because it is
rendered in quirks mode, the HBase interface doesn't work because it is
rendered in standards mode. See HBASE-2110 for more details.
I've had a quick look at HttpServer but I'm too unfamiliar with it to see the
problem. I think this started happening with HADOOP-6441 which would lead me
to believe that the filter is called for every request and not only *.jsp and
*.html. I'd consider this a bug but I don't know enough about this to provide
a fix.

[jira] [Commented] (HADOOP-8935) Make 'winutils ls' show the SID if the owner does not exist on the system


[ 
https://issues.apache.org/jira/browse/HADOOP-8935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13477677#comment-13477677
 ] 

Ivan Mitic commented on HADOOP-8935:


Thanks Chuan for posting the patch, fix looks good to me, +1

 Make 'winutils ls' show the SID if the owner does not exist on the system
 -

 Key: HADOOP-8935
 URL: https://issues.apache.org/jira/browse/HADOOP-8935
 Project: Hadoop Common
  Issue Type: Bug
Affects Versions: 1-win
Reporter: Chuan Liu
Assignee: Chuan Liu
Priority: Minor
 Fix For: 1-win

 Attachments: HADOOP-8935-branch-1-win.patch


 Right now, 'winutils ls' will fail if the file belongs to a user SID that 
 does not exist on the system. E.g. the user is deleted. Previously, this is 
 only a hypothesis scenario. However, we have seen some failures in the Azure 
 deployment where the OS is re-imaged, and renders the old SID invalid. 
 [~jgordon] proposed to display the SID itself in the invalid SID case similar 
 to the situation on Linux. This JIRA is created to track this proposal.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-6496) HttpServer sends wrong content-type for CSS files (and others)

[
https://issues.apache.org/jira/browse/HADOOP-6496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13477680#comment-13477680
]

Suresh Srinivas commented on HADOOP-6496:
-

Thanks for the explanation. +1 for the branch-1 patch.

HttpServer sends wrong content-type for CSS files (and others)
--

Attachments: HADOOP-6496.branch-1.1.backport.2.patch,
HADOOP-6496.branch-1.1.backport.patch, hadoop-6496.txt, hadoop-6496.txt

[jira] [Resolved] (HADOOP-6496) HttpServer sends wrong content-type for CSS files (and others)


 [ 
https://issues.apache.org/jira/browse/HADOOP-6496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Suresh Srinivas resolved HADOOP-6496.
-

   Resolution: Fixed
Fix Version/s: 1.2.0

 HttpServer sends wrong content-type for CSS files (and others)
 --

 Key: HADOOP-6496
 URL: https://issues.apache.org/jira/browse/HADOOP-6496
 Project: Hadoop Common
  Issue Type: Bug
Affects Versions: 0.21.0, 0.22.0
Reporter: Lars Francke
Assignee: Ivan Mitic
Priority: Minor
 Fix For: 1.2.0, 0.22.0

 Attachments: HADOOP-6496.branch-1.1.backport.2.patch, 
 HADOOP-6496.branch-1.1.backport.patch, hadoop-6496.txt, hadoop-6496.txt


 CSS files are send as text/html causing problems if the HTML page is rendered 
 in standards mode. The HDFS interface for example still works because it is 
 rendered in quirks mode, the HBase interface doesn't work because it is 
 rendered in standards mode. See HBASE-2110 for more details.
 I've had a quick look at HttpServer but I'm too unfamiliar with it to see the 
 problem. I think this started happening with HADOOP-6441 which would lead me 
 to believe that the filter is called for every request and not only *.jsp and 
 *.html. I'd consider this a bug but I don't know enough about this to provide 
 a fix.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HADOOP-6496) HttpServer sends wrong content-type for CSS files (and others)


 [ 
https://issues.apache.org/jira/browse/HADOOP-6496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Suresh Srinivas updated HADOOP-6496:


Fix Version/s: 1-win

 HttpServer sends wrong content-type for CSS files (and others)
 --

 Key: HADOOP-6496
 URL: https://issues.apache.org/jira/browse/HADOOP-6496
 Project: Hadoop Common
  Issue Type: Bug
Affects Versions: 0.21.0, 0.22.0
Reporter: Lars Francke
Assignee: Ivan Mitic
Priority: Minor
 Fix For: 1.2.0, 0.22.0, 1-win

 Attachments: HADOOP-6496.branch-1.1.backport.2.patch, 
 HADOOP-6496.branch-1.1.backport.patch, hadoop-6496.txt, hadoop-6496.txt


 CSS files are send as text/html causing problems if the HTML page is rendered 
 in standards mode. The HDFS interface for example still works because it is 
 rendered in quirks mode, the HBase interface doesn't work because it is 
 rendered in standards mode. See HBASE-2110 for more details.
 I've had a quick look at HttpServer but I'm too unfamiliar with it to see the 
 problem. I think this started happening with HADOOP-6441 which would lead me 
 to believe that the filter is called for every request and not only *.jsp and 
 *.html. I'd consider this a bug but I don't know enough about this to provide 
 a fix.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HADOOP-7827) jsp pages missing DOCTYPE


 [ 
https://issues.apache.org/jira/browse/HADOOP-7827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ivan Mitic updated HADOOP-7827:
---

Attachment: HADOOP-7827.branch-1.1.backport.patch

Attaching the branch-1.1 compatible patch.

 jsp pages missing DOCTYPE
 -

 Key: HADOOP-7827
 URL: https://issues.apache.org/jira/browse/HADOOP-7827
 Project: Hadoop Common
  Issue Type: Bug
Affects Versions: 0.20.203.0
Reporter: Dave Vronay
Assignee: Dave Vronay
Priority: Trivial
 Attachments: HADOOP-6496.branch-1.1.backport.patch, 
 HADOOP-7827-branch-0.20-security.patch, 
 HADOOP-7827.branch-1.1.backport.patch, HADOOP-7827.patch, HADOOP-7827.patch

   Original Estimate: 168h
  Remaining Estimate: 168h

 The various jsp pages in the UI are all missing a DOCTYPE declaration.  This 
 causes the pages to render incorrectly on some browsers, such as IE9.  Every 
 UI page should have a valid tag, such as !DOCTYPE HTML, as their first 
 line.  There are 31 files that need to be changed, all in the 
 core\src\webapps tree.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-7827) jsp pages missing DOCTYPE


[ 
https://issues.apache.org/jira/browse/HADOOP-7827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13477698#comment-13477698
 ] 

Ivan Mitic commented on HADOOP-7827:


Since HADOOP-6496 is backported to branch-1.1, it should be fine to also commit 
{{HADOOP-7827.branch-1.1.backport.patch}} to branch-1.1.

 jsp pages missing DOCTYPE
 -

 Key: HADOOP-7827
 URL: https://issues.apache.org/jira/browse/HADOOP-7827
 Project: Hadoop Common
  Issue Type: Bug
Affects Versions: 0.20.203.0
Reporter: Dave Vronay
Assignee: Dave Vronay
Priority: Trivial
 Attachments: HADOOP-6496.branch-1.1.backport.patch, 
 HADOOP-7827-branch-0.20-security.patch, 
 HADOOP-7827.branch-1.1.backport.patch, HADOOP-7827.patch, HADOOP-7827.patch

   Original Estimate: 168h
  Remaining Estimate: 168h

 The various jsp pages in the UI are all missing a DOCTYPE declaration.  This 
 causes the pages to render incorrectly on some browsers, such as IE9.  Every 
 UI page should have a valid tag, such as !DOCTYPE HTML, as their first 
 line.  There are 31 files that need to be changed, all in the 
 core\src\webapps tree.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-8931) Add Java version to startup message


[ 
https://issues.apache.org/jira/browse/HADOOP-8931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13477764#comment-13477764
 ] 

Hudson commented on HADOOP-8931:


Integrated in Hadoop-Yarn-trunk #6 (See 
[https://builds.apache.org/job/Hadoop-Yarn-trunk/6/])
HADOOP-8931. Add Java version to startup message. Contributed by Eli 
Collins (Revision 1398998)

 Result = FAILURE
eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1398998
Files : 
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/StringUtils.java


 Add Java version to startup message 
 

 Key: HADOOP-8931
 URL: https://issues.apache.org/jira/browse/HADOOP-8931
 Project: Hadoop Common
  Issue Type: Improvement
Affects Versions: 1.0.0, 2.0.0-alpha
Reporter: Eli Collins
Assignee: Eli Collins
Priority: Trivial
 Fix For: 1.2.0, 2.0.3-alpha

 Attachments: hadoop-8931-b1.txt, hadoop-8931.txt


 I often look at logs and have to track down the java version they were run 
 with, it would be useful if we logged this as part of the startup message.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-8926) hadoop.util.PureJavaCrc32 cache hit-ratio is low for static data


[ 
https://issues.apache.org/jira/browse/HADOOP-8926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13477766#comment-13477766
 ] 

Hudson commented on HADOOP-8926:


Integrated in Hadoop-Yarn-trunk #6 (See 
[https://builds.apache.org/job/Hadoop-Yarn-trunk/6/])
HADOOP-8926. hadoop.util.PureJavaCrc32 cache hit-ratio is low for static 
data (Gopal V via bobby) (Revision 1399005)

 Result = FAILURE
bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1399005
Files : 
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/PureJavaCrc32.java
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/PureJavaCrc32C.java
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestPureJavaCrc32.java


 hadoop.util.PureJavaCrc32 cache hit-ratio is low for static data
 

 Key: HADOOP-8926
 URL: https://issues.apache.org/jira/browse/HADOOP-8926
 Project: Hadoop Common
  Issue Type: Improvement
  Components: util
Affects Versions: 2.0.3-alpha
 Environment: Ubuntu 10.10 i386 
Reporter: Gopal V
Assignee: Gopal V
Priority: Trivial
  Labels: optimization
 Fix For: 3.0.0, 2.0.3-alpha

 Attachments: crc32-faster+readable.patch, crc32-faster+test.patch, 
 pure-crc32-cache-hit.patch


 While running microbenchmarks for HDFS write codepath, a significant part of 
 the CPU fraction was consumed by the DataChecksum.update(). 
 The attached patch converts the static arrays in CRC32 into a single linear 
 array for a performance boost in the inner loop.
 milli-seconds for 1Gig (16400 loop over a 64kb chunk) 
 || platform || original || cache-aware || improvement ||
 | x86 | 3894 | 2304 | 40.83 |
 | x86_64 | 2131 | 1826 | 14 | 
 The performance improvement on x86 is rather larger than the 64bit case, due 
 to the extra register/stack pressure caused by the static arrays.
 A closer analysis of the PureJavaCrc32 JIT code shows the following assembly 
 fragment
 {code}
   0x40f1e345: mov$0x184,%ecx
   0x40f1e34a: mov0x4415b560(%ecx),%ecx  ;*getstatic T8_5
 ; - PureJavaCrc32::update@95 (line 61)
 ;   {oop('PureJavaCrc32')}
   0x40f1e350: mov%ecx,0x2c(%esp)
 {code}
 Basically, the static variables T8_0 through to T8_7 are being spilled to the 
 stack because of register pressure. The x86_64 case has a lower likelihood of 
 such pessimistic JIT code due to the increased number of registers.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-8922) Provide alternate JSONP output for JMXJsonServlet to allow javascript in browser dashboard


[ 
https://issues.apache.org/jira/browse/HADOOP-8922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13477768#comment-13477768
 ] 

Hudson commented on HADOOP-8922:


Integrated in Hadoop-Yarn-trunk #6 (See 
[https://builds.apache.org/job/Hadoop-Yarn-trunk/6/])
HADOOP-8922. Provide alternate JSONP output for JMXJsonServlet to allow 
javascript in browser dashboard (Damien Hardy via bobby) (Revision 1398904)

 Result = FAILURE
bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1398904
Files : 
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/jmx/JMXJsonServlet.java
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/jmx/TestJMXJsonServlet.java


 Provide alternate JSONP output for JMXJsonServlet to allow javascript in 
 browser dashboard
 --

 Key: HADOOP-8922
 URL: https://issues.apache.org/jira/browse/HADOOP-8922
 Project: Hadoop Common
  Issue Type: Improvement
  Components: metrics
Affects Versions: 2.0.0-alpha
Reporter: Damien Hardy
Priority: Trivial
  Labels: newbie, patch
 Fix For: 3.0.0, 2.0.3-alpha

 Attachments: HADOOP-8922-3.patch, HADOOP-8922-4-branch-2.patch, 
 test.html


 JMXJsonServlet may provide a JSONP alternative to JSON to allow javascript in 
 browser GUI to make requests.
 For security purpose about XSS, browser limit request on other 
 domain[¹|#ref1] so that metrics from cluster nodes cannot be used in a full 
 js interface.
 An example of this kind of dashboard is the bigdesk[²|#ref2] plugin for 
 ElasticSearch.
 In order to achieve that the servlet should detect a GET parameter 
 (callback=) and modify the response by surrounding the Json value with 
 ( and ); [³|#ref3]
 value  is variable and should be provide by client as callback 
 parameter value.
 {anchor:ref1}[1] 
 https://developer.mozilla.org/en-US/docs/Same_origin_policy_for_JavaScript
 {anchor:ref2}[2] https://github.com/lukas-vlcek/bigdesk
 {anchor:ref3}[3] http://en.wikipedia.org/wiki/JSONP

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-8923) WEBUI shows an intermediatory page when the cookie expires.

[
https://issues.apache.org/jira/browse/HADOOP-8923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13477769#comment-13477769
]

Hudson commented on HADOOP-8923:

Integrated in Hadoop-Yarn-trunk #6 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk/6/])
HADOOP-8923. JNI-based user-group mapping modules can be too chatty on
lookup failures. Contributed by Kihwal Lee. (Revision 1398883)

Result = FAILURE
suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1398883
Files :
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
*
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/JniBasedUnixGroupsMapping.java
*
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/JniBasedUnixGroupsNetgroupMapping.java

WEBUI shows an intermediatory page when the cookie expires.
---

Key: HADOOP-8923
URL: https://issues.apache.org/jira/browse/HADOOP-8923
Project: Hadoop Common
Issue Type: Improvement
Components: security
Affects Versions: 1.1.0
Reporter: Benoy Antony
Assignee: Benoy Antony
Priority: Minor
Attachments: HADOOP-8923.patch

The WEBUI does Authentication (SPNEGO/Custom) and then drops a cookie.
Once the cookie expires, the webui displays a page saying that
authentication token expired. The user has to refresh the page to get
authenticated again. This page can be avoided and the user can authenticated
without showing such a page to the user.
Also the when the cookie expires, a warning is logged. But there is no need
to log this as this is not of any significance.

[jira] [Commented] (HADOOP-8883) Anonymous fallback in KerberosAuthenticator is broken


[ 
https://issues.apache.org/jira/browse/HADOOP-8883?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=1341#comment-1341
 ] 

Hudson commented on HADOOP-8883:


Integrated in Hadoop-Yarn-trunk #6 (See 
[https://builds.apache.org/job/Hadoop-Yarn-trunk/6/])
HADOOP-8883. Anonymous fallback in KerberosAuthenticator is broken. 
(rkanter via tucu) (Revision 1398895)

 Result = FAILURE
tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1398895
Files : 
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/client/KerberosAuthenticator.java
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/client/AuthenticatorTestCase.java
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/client/TestKerberosAuthenticator.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt


 Anonymous fallback in KerberosAuthenticator is broken
 -

 Key: HADOOP-8883
 URL: https://issues.apache.org/jira/browse/HADOOP-8883
 Project: Hadoop Common
  Issue Type: Bug
Affects Versions: 2.0.3-alpha
Reporter: Robert Kanter
Assignee: Robert Kanter
  Labels: security
 Fix For: 2.0.3-alpha

 Attachments: HADOOP-8883.patch


 HADOOP-8855 changed KerberosAuthenticator to handle when the JDK did the 
 SPNEGO already; but this change broke using the fallback authenticator 
 (PseudoAuthenticator) with an anonymous user (see OOZIE-1010).  

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HADOOP-8938) add option to do better diags of startup configuration

Steve Loughran created HADOOP-8938:
--

 Summary: add option to do better diags of startup configuration
 Key: HADOOP-8938
 URL: https://issues.apache.org/jira/browse/HADOOP-8938
 Project: Hadoop Common
  Issue Type: New Feature
  Components: bin
Affects Versions: 2.0.2-alpha, 1.1.0
Reporter: Steve Loughran
Priority: Minor


HADOOP-8931 shows a symptom of a larger problem: we need better diagnostics of 
what all the environment variables and settings going through the hadoop 
scripts to find out why something isn't working. 

Ideally some command line parameter to the scripts (or even: a new environment 
variable) could trigger more display of critical parameters.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-8938) add option to do better diags of startup configuration


[ 
https://issues.apache.org/jira/browse/HADOOP-8938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=1348#comment-1348
 ] 

Steve Loughran commented on HADOOP-8938:


Things that are often useful to know
* Java home: value and what is at the far end
* Hadoop conf directory picked up (symlink games can cause confusion)
* which *-site.xml files are being picked up (full path)
* full path to commands
* native lib directory and contents

This isn't stuff that needs to be printed every run, so it should be something 
that could be turned on if needed.

 add option to do better diags of startup configuration
 --

 Key: HADOOP-8938
 URL: https://issues.apache.org/jira/browse/HADOOP-8938
 Project: Hadoop Common
  Issue Type: New Feature
  Components: bin
Affects Versions: 1.1.0, 2.0.2-alpha
Reporter: Steve Loughran
Priority: Minor

 HADOOP-8931 shows a symptom of a larger problem: we need better diagnostics 
 of what all the environment variables and settings going through the hadoop 
 scripts to find out why something isn't working. 
 Ideally some command line parameter to the scripts (or even: a new 
 environment variable) could trigger more display of critical parameters.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HADOOP-8938) add option to do better diags of startup configuration


 [ 
https://issues.apache.org/jira/browse/HADOOP-8938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Steve Loughran updated HADOOP-8938:
---

Attachment: hdiag.py

This is not integrated with bin/hadoop; its what I use after execing all the 
preamble scripts , especially {{hadoop-config.sh}} to see what the environment 
variables currently are. It shows what I've found useful in the past

 add option to do better diags of startup configuration
 --

 Key: HADOOP-8938
 URL: https://issues.apache.org/jira/browse/HADOOP-8938
 Project: Hadoop Common
  Issue Type: New Feature
  Components: bin
Affects Versions: 1.1.0, 2.0.2-alpha
Reporter: Steve Loughran
Priority: Minor
 Attachments: hdiag.py


 HADOOP-8931 shows a symptom of a larger problem: we need better diagnostics 
 of what all the environment variables and settings going through the hadoop 
 scripts to find out why something isn't working. 
 Ideally some command line parameter to the scripts (or even: a new 
 environment variable) could trigger more display of critical parameters.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-8932) JNI-based user-group mapping modules can be too chatty on lookup failures


[ 
https://issues.apache.org/jira/browse/HADOOP-8932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13477826#comment-13477826
 ] 

Hudson commented on HADOOP-8932:


Integrated in Hadoop-Hdfs-0.23-Build #407 (See 
[https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/407/])
HADOOP-8932. Merging change 1398883 from trunk. (Revision 1398886)

 Result = UNSTABLE
suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1398886
Files : 
* 
/hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt
* 
/hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/JniBasedUnixGroupsMapping.java
* 
/hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/JniBasedUnixGroupsNetgroupMapping.java


 JNI-based user-group mapping modules can be too chatty on lookup failures
 -

 Key: HADOOP-8932
 URL: https://issues.apache.org/jira/browse/HADOOP-8932
 Project: Hadoop Common
  Issue Type: Improvement
  Components: security
Affects Versions: 3.0.0, 2.0.3-alpha, 0.23.5
Reporter: Kihwal Lee
Assignee: Kihwal Lee
 Fix For: 3.0.0, 2.0.3-alpha, 0.23.5

 Attachments: hadoop-8932.patch.txt


 On a user/group lookup failure, JniBasedUnixGroupsMapping and 
 JniBasedUnixGroupsNetgroupMapping are logging the
 full stack trace at WARN level.  Since the caller of these methods is already 
 logging errors, this is not needed.  In branch-1, just one line is logged, so 
 we don't need this change there.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-8931) Add Java version to startup message


[ 
https://issues.apache.org/jira/browse/HADOOP-8931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13477833#comment-13477833
 ] 

Hudson commented on HADOOP-8931:


Integrated in Hadoop-Hdfs-trunk #1198 (See 
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/1198/])
HADOOP-8931. Add Java version to startup message. Contributed by Eli 
Collins (Revision 1398998)

 Result = SUCCESS
eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1398998
Files : 
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/StringUtils.java


 Add Java version to startup message 
 

 Key: HADOOP-8931
 URL: https://issues.apache.org/jira/browse/HADOOP-8931
 Project: Hadoop Common
  Issue Type: Improvement
Affects Versions: 1.0.0, 2.0.0-alpha
Reporter: Eli Collins
Assignee: Eli Collins
Priority: Trivial
 Fix For: 1.2.0, 2.0.3-alpha

 Attachments: hadoop-8931-b1.txt, hadoop-8931.txt


 I often look at logs and have to track down the java version they were run 
 with, it would be useful if we logged this as part of the startup message.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-8926) hadoop.util.PureJavaCrc32 cache hit-ratio is low for static data


[ 
https://issues.apache.org/jira/browse/HADOOP-8926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13477835#comment-13477835
 ] 

Hudson commented on HADOOP-8926:


Integrated in Hadoop-Hdfs-trunk #1198 (See 
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/1198/])
HADOOP-8926. hadoop.util.PureJavaCrc32 cache hit-ratio is low for static 
data (Gopal V via bobby) (Revision 1399005)

 Result = SUCCESS
bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1399005
Files : 
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/PureJavaCrc32.java
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/PureJavaCrc32C.java
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestPureJavaCrc32.java


 hadoop.util.PureJavaCrc32 cache hit-ratio is low for static data
 

 Key: HADOOP-8926
 URL: https://issues.apache.org/jira/browse/HADOOP-8926
 Project: Hadoop Common
  Issue Type: Improvement
  Components: util
Affects Versions: 2.0.3-alpha
 Environment: Ubuntu 10.10 i386 
Reporter: Gopal V
Assignee: Gopal V
Priority: Trivial
  Labels: optimization
 Fix For: 3.0.0, 2.0.3-alpha

 Attachments: crc32-faster+readable.patch, crc32-faster+test.patch, 
 pure-crc32-cache-hit.patch


 While running microbenchmarks for HDFS write codepath, a significant part of 
 the CPU fraction was consumed by the DataChecksum.update(). 
 The attached patch converts the static arrays in CRC32 into a single linear 
 array for a performance boost in the inner loop.
 milli-seconds for 1Gig (16400 loop over a 64kb chunk) 
 || platform || original || cache-aware || improvement ||
 | x86 | 3894 | 2304 | 40.83 |
 | x86_64 | 2131 | 1826 | 14 | 
 The performance improvement on x86 is rather larger than the 64bit case, due 
 to the extra register/stack pressure caused by the static arrays.
 A closer analysis of the PureJavaCrc32 JIT code shows the following assembly 
 fragment
 {code}
   0x40f1e345: mov$0x184,%ecx
   0x40f1e34a: mov0x4415b560(%ecx),%ecx  ;*getstatic T8_5
 ; - PureJavaCrc32::update@95 (line 61)
 ;   {oop('PureJavaCrc32')}
   0x40f1e350: mov%ecx,0x2c(%esp)
 {code}
 Basically, the static variables T8_0 through to T8_7 are being spilled to the 
 stack because of register pressure. The x86_64 case has a lower likelihood of 
 such pessimistic JIT code due to the increased number of registers.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-8922) Provide alternate JSONP output for JMXJsonServlet to allow javascript in browser dashboard


[ 
https://issues.apache.org/jira/browse/HADOOP-8922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13477837#comment-13477837
 ] 

Hudson commented on HADOOP-8922:


Integrated in Hadoop-Hdfs-trunk #1198 (See 
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/1198/])
HADOOP-8922. Provide alternate JSONP output for JMXJsonServlet to allow 
javascript in browser dashboard (Damien Hardy via bobby) (Revision 1398904)

 Result = SUCCESS
bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1398904
Files : 
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/jmx/JMXJsonServlet.java
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/jmx/TestJMXJsonServlet.java


 Provide alternate JSONP output for JMXJsonServlet to allow javascript in 
 browser dashboard
 --

 Key: HADOOP-8922
 URL: https://issues.apache.org/jira/browse/HADOOP-8922
 Project: Hadoop Common
  Issue Type: Improvement
  Components: metrics
Affects Versions: 2.0.0-alpha
Reporter: Damien Hardy
Priority: Trivial
  Labels: newbie, patch
 Fix For: 3.0.0, 2.0.3-alpha

 Attachments: HADOOP-8922-3.patch, HADOOP-8922-4-branch-2.patch, 
 test.html


 JMXJsonServlet may provide a JSONP alternative to JSON to allow javascript in 
 browser GUI to make requests.
 For security purpose about XSS, browser limit request on other 
 domain[¹|#ref1] so that metrics from cluster nodes cannot be used in a full 
 js interface.
 An example of this kind of dashboard is the bigdesk[²|#ref2] plugin for 
 ElasticSearch.
 In order to achieve that the servlet should detect a GET parameter 
 (callback=) and modify the response by surrounding the Json value with 
 ( and ); [³|#ref3]
 value  is variable and should be provide by client as callback 
 parameter value.
 {anchor:ref1}[1] 
 https://developer.mozilla.org/en-US/docs/Same_origin_policy_for_JavaScript
 {anchor:ref2}[2] https://github.com/lukas-vlcek/bigdesk
 {anchor:ref3}[3] http://en.wikipedia.org/wiki/JSONP

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-8923) WEBUI shows an intermediatory page when the cookie expires.

[
https://issues.apache.org/jira/browse/HADOOP-8923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13477838#comment-13477838
]

Hudson commented on HADOOP-8923:

Integrated in Hadoop-Hdfs-trunk #1198 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/1198/])
HADOOP-8923. JNI-based user-group mapping modules can be too chatty on
lookup failures. Contributed by Kihwal Lee. (Revision 1398883)

Result = SUCCESS
suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1398883
Files :
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
*
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/JniBasedUnixGroupsMapping.java
*
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/JniBasedUnixGroupsNetgroupMapping.java

WEBUI shows an intermediatory page when the cookie expires.
---

[jira] [Commented] (HADOOP-8883) Anonymous fallback in KerberosAuthenticator is broken


[ 
https://issues.apache.org/jira/browse/HADOOP-8883?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13477840#comment-13477840
 ] 

Hudson commented on HADOOP-8883:


Integrated in Hadoop-Hdfs-trunk #1198 (See 
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/1198/])
HADOOP-8883. Anonymous fallback in KerberosAuthenticator is broken. 
(rkanter via tucu) (Revision 1398895)

 Result = SUCCESS
tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1398895
Files : 
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/client/KerberosAuthenticator.java
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/client/AuthenticatorTestCase.java
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/client/TestKerberosAuthenticator.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt


 Anonymous fallback in KerberosAuthenticator is broken
 -

 Key: HADOOP-8883
 URL: https://issues.apache.org/jira/browse/HADOOP-8883
 Project: Hadoop Common
  Issue Type: Bug
Affects Versions: 2.0.3-alpha
Reporter: Robert Kanter
Assignee: Robert Kanter
  Labels: security
 Fix For: 2.0.3-alpha

 Attachments: HADOOP-8883.patch


 HADOOP-8855 changed KerberosAuthenticator to handle when the JDK did the 
 SPNEGO already; but this change broke using the fallback authenticator 
 (PseudoAuthenticator) with an anonymous user (see OOZIE-1010).  

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-8901) GZip and Snappy support may not work without unversioned libraries


[ 
https://issues.apache.org/jira/browse/HADOOP-8901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13477855#comment-13477855
 ] 

Hudson commented on HADOOP-8901:


Integrated in Hadoop-Mapreduce-trunk #1228 (See 
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1228/])
HADOOP-8901. GZip and Snappy support may not work without unversioned 
libraries. Contributed by Colin Patrick McCabe. (Revision 1398416)

 Result = FAILURE
todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1398416
Files : 
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/CMakeLists.txt


 GZip and Snappy support may not work without unversioned libraries
 --

 Key: HADOOP-8901
 URL: https://issues.apache.org/jira/browse/HADOOP-8901
 Project: Hadoop Common
  Issue Type: Bug
  Components: native
Affects Versions: 2.0.3-alpha
Reporter: Colin Patrick McCabe
Assignee: Colin Patrick McCabe
Priority: Minor
 Fix For: 3.0.0, 2.0.3-alpha

 Attachments: HADOOP-8901.001.patch, HADOOP-8901.002.patch, 
 HADOOP-8901.003.patch


 Currently, we use {{dlopen}} to open {{libz.so}} and {{libsnappy.so}}, to get 
 Gzip and Snappy support, respectively.
 However, this is not correct; we should be dlopening {{libsnappy.so.1}} 
 instead.  The versionless form of the shared library is not commonly 
 installed except by development packages.  Also, we may run into subtle 
 compatibility problems if a new version of libsnappy comes out.
 Thanks to Brandon Vargo for reporting this bug.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-8920) Add more javadoc to metrics2 related classes


[ 
https://issues.apache.org/jira/browse/HADOOP-8920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13477857#comment-13477857
 ] 

Hudson commented on HADOOP-8920:


Integrated in Hadoop-Mapreduce-trunk #1228 (See 
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1228/])
HADOOP-8920. Add more javadoc to metrics2 related classes. Contributed by 
Suresh Srinivas. (Revision 1398640)

 Result = FAILURE
suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1398640
Files : 
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/MetricsCollector.java
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/MetricsFilter.java
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/MetricsInfo.java
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/MetricsRecord.java
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/MetricsRecordBuilder.java
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/MetricsSink.java
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/MetricsSource.java
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/MetricsSystem.java
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/annotation/Metric.java
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/impl/MetricsRecordBuilderImpl.java
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/impl/MetricsRecordImpl.java
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/impl/MetricsSystemImpl.java
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/lib/DefaultMetricsSystem.java
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/lib/MetricsSourceBuilder.java
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/lib/MutableMetricsFactory.java


 Add more javadoc to metrics2 related classes
 

 Key: HADOOP-8920
 URL: https://issues.apache.org/jira/browse/HADOOP-8920
 Project: Hadoop Common
  Issue Type: Improvement
  Components: metrics
Affects Versions: 1.0.0, 0.23.0, 2.0.0-alpha, 3.0.0
Reporter: Suresh Srinivas
Assignee: Suresh Srinivas
Priority: Minor
 Fix For: 3.0.0

 Attachments: HADOOP-8920.patch


 Metrics2 related code can is very sparsely documented. Here is patch that 
 adds javadoc that should help some of the code easier to browse and 
 understand.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-8931) Add Java version to startup message


[ 
https://issues.apache.org/jira/browse/HADOOP-8931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13477860#comment-13477860
 ] 

Hudson commented on HADOOP-8931:


Integrated in Hadoop-Mapreduce-trunk #1228 (See 
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1228/])
HADOOP-8931. Add Java version to startup message. Contributed by Eli 
Collins (Revision 1398998)

 Result = FAILURE
eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1398998
Files : 
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/StringUtils.java


 Add Java version to startup message 
 

 Key: HADOOP-8931
 URL: https://issues.apache.org/jira/browse/HADOOP-8931
 Project: Hadoop Common
  Issue Type: Improvement
Affects Versions: 1.0.0, 2.0.0-alpha
Reporter: Eli Collins
Assignee: Eli Collins
Priority: Trivial
 Fix For: 1.2.0, 2.0.3-alpha

 Attachments: hadoop-8931-b1.txt, hadoop-8931.txt


 I often look at logs and have to track down the java version they were run 
 with, it would be useful if we logged this as part of the startup message.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-8926) hadoop.util.PureJavaCrc32 cache hit-ratio is low for static data


[ 
https://issues.apache.org/jira/browse/HADOOP-8926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13477863#comment-13477863
 ] 

Hudson commented on HADOOP-8926:


Integrated in Hadoop-Mapreduce-trunk #1228 (See 
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1228/])
HADOOP-8926. hadoop.util.PureJavaCrc32 cache hit-ratio is low for static 
data (Gopal V via bobby) (Revision 1399005)

 Result = FAILURE
bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1399005
Files : 
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/PureJavaCrc32.java
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/PureJavaCrc32C.java
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestPureJavaCrc32.java


 hadoop.util.PureJavaCrc32 cache hit-ratio is low for static data
 

 Key: HADOOP-8926
 URL: https://issues.apache.org/jira/browse/HADOOP-8926
 Project: Hadoop Common
  Issue Type: Improvement
  Components: util
Affects Versions: 2.0.3-alpha
 Environment: Ubuntu 10.10 i386 
Reporter: Gopal V
Assignee: Gopal V
Priority: Trivial
  Labels: optimization
 Fix For: 3.0.0, 2.0.3-alpha

 Attachments: crc32-faster+readable.patch, crc32-faster+test.patch, 
 pure-crc32-cache-hit.patch


 While running microbenchmarks for HDFS write codepath, a significant part of 
 the CPU fraction was consumed by the DataChecksum.update(). 
 The attached patch converts the static arrays in CRC32 into a single linear 
 array for a performance boost in the inner loop.
 milli-seconds for 1Gig (16400 loop over a 64kb chunk) 
 || platform || original || cache-aware || improvement ||
 | x86 | 3894 | 2304 | 40.83 |
 | x86_64 | 2131 | 1826 | 14 | 
 The performance improvement on x86 is rather larger than the 64bit case, due 
 to the extra register/stack pressure caused by the static arrays.
 A closer analysis of the PureJavaCrc32 JIT code shows the following assembly 
 fragment
 {code}
   0x40f1e345: mov$0x184,%ecx
   0x40f1e34a: mov0x4415b560(%ecx),%ecx  ;*getstatic T8_5
 ; - PureJavaCrc32::update@95 (line 61)
 ;   {oop('PureJavaCrc32')}
   0x40f1e350: mov%ecx,0x2c(%esp)
 {code}
 Basically, the static variables T8_0 through to T8_7 are being spilled to the 
 stack because of register pressure. The x86_64 case has a lower likelihood of 
 such pessimistic JIT code due to the increased number of registers.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-8922) Provide alternate JSONP output for JMXJsonServlet to allow javascript in browser dashboard


[ 
https://issues.apache.org/jira/browse/HADOOP-8922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13477866#comment-13477866
 ] 

Hudson commented on HADOOP-8922:


Integrated in Hadoop-Mapreduce-trunk #1228 (See 
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1228/])
HADOOP-8922. Provide alternate JSONP output for JMXJsonServlet to allow 
javascript in browser dashboard (Damien Hardy via bobby) (Revision 1398904)

 Result = FAILURE
bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1398904
Files : 
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/jmx/JMXJsonServlet.java
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/jmx/TestJMXJsonServlet.java


 Provide alternate JSONP output for JMXJsonServlet to allow javascript in 
 browser dashboard
 --

 Key: HADOOP-8922
 URL: https://issues.apache.org/jira/browse/HADOOP-8922
 Project: Hadoop Common
  Issue Type: Improvement
  Components: metrics
Affects Versions: 2.0.0-alpha
Reporter: Damien Hardy
Priority: Trivial
  Labels: newbie, patch
 Fix For: 3.0.0, 2.0.3-alpha

 Attachments: HADOOP-8922-3.patch, HADOOP-8922-4-branch-2.patch, 
 test.html


 JMXJsonServlet may provide a JSONP alternative to JSON to allow javascript in 
 browser GUI to make requests.
 For security purpose about XSS, browser limit request on other 
 domain[¹|#ref1] so that metrics from cluster nodes cannot be used in a full 
 js interface.
 An example of this kind of dashboard is the bigdesk[²|#ref2] plugin for 
 ElasticSearch.
 In order to achieve that the servlet should detect a GET parameter 
 (callback=) and modify the response by surrounding the Json value with 
 ( and ); [³|#ref3]
 value  is variable and should be provide by client as callback 
 parameter value.
 {anchor:ref1}[1] 
 https://developer.mozilla.org/en-US/docs/Same_origin_policy_for_JavaScript
 {anchor:ref2}[2] https://github.com/lukas-vlcek/bigdesk
 {anchor:ref3}[3] http://en.wikipedia.org/wiki/JSONP

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-8923) WEBUI shows an intermediatory page when the cookie expires.

[
https://issues.apache.org/jira/browse/HADOOP-8923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13477867#comment-13477867
]

Hudson commented on HADOOP-8923:

Integrated in Hadoop-Mapreduce-trunk #1228 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1228/])
HADOOP-8923. JNI-based user-group mapping modules can be too chatty on
lookup failures. Contributed by Kihwal Lee. (Revision 1398883)

WEBUI shows an intermediatory page when the cookie expires.
---

[jira] [Commented] (HADOOP-8883) Anonymous fallback in KerberosAuthenticator is broken


[ 
https://issues.apache.org/jira/browse/HADOOP-8883?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13477869#comment-13477869
 ] 

Hudson commented on HADOOP-8883:


Integrated in Hadoop-Mapreduce-trunk #1228 (See 
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1228/])
HADOOP-8883. Anonymous fallback in KerberosAuthenticator is broken. 
(rkanter via tucu) (Revision 1398895)

 Result = FAILURE
tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1398895
Files : 
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/client/KerberosAuthenticator.java
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/client/AuthenticatorTestCase.java
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/client/TestKerberosAuthenticator.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt


 Anonymous fallback in KerberosAuthenticator is broken
 -

 Key: HADOOP-8883
 URL: https://issues.apache.org/jira/browse/HADOOP-8883
 Project: Hadoop Common
  Issue Type: Bug
Affects Versions: 2.0.3-alpha
Reporter: Robert Kanter
Assignee: Robert Kanter
  Labels: security
 Fix For: 2.0.3-alpha

 Attachments: HADOOP-8883.patch


 HADOOP-8855 changed KerberosAuthenticator to handle when the JDK did the 
 SPNEGO already; but this change broke using the fallback authenticator 
 (PseudoAuthenticator) with an anonymous user (see OOZIE-1010).  

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-8929) Add toString, other improvements for SampleQuantiles


[ 
https://issues.apache.org/jira/browse/HADOOP-8929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13477872#comment-13477872
 ] 

Hudson commented on HADOOP-8929:


Integrated in Hadoop-Mapreduce-trunk #1228 (See 
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1228/])
HADOOP-8929. Add toString, other improvements for SampleQuantiles. 
Contributed by Todd Lipcon. (Revision 1398658)

 Result = FAILURE
todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1398658
Files : 
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/lib/MutableQuantiles.java
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/util/Quantile.java
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/util/SampleQuantiles.java
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/metrics2/util/TestSampleQuantiles.java


 Add toString, other improvements for SampleQuantiles
 

 Key: HADOOP-8929
 URL: https://issues.apache.org/jira/browse/HADOOP-8929
 Project: Hadoop Common
  Issue Type: Improvement
  Components: metrics
Affects Versions: 3.0.0, 2.0.2-alpha
Reporter: Todd Lipcon
Assignee: Todd Lipcon
 Fix For: 3.0.0, 2.0.3-alpha

 Attachments: hadoop-8929.txt


 The new SampleQuantiles class is useful in the context of benchmarks, but 
 currently there is no way to print it out outside the context of a metrics 
 sink. It would be nice to have a convenient way to stringify it for logging, 
 etc.
 Also:
 - made it Comparable and changed the HashMap to TreeMap so that the printout 
 is in ascending percentile order. Given that this map is always very small, 
 and snapshot() is only called once a minute or so, the runtime/memory 
 differences between treemap and hashmap should be negligible.
 - changed the behavior to return null instead of throw, because all the 
 catching, etc, got pretty ugly. In implementing toString, I figured I'd clean 
 up the other behavior along the way.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-8928) Add ability to reset topologies on master nodes

2012-10-17 Thread Shinichi Yamashita (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-8928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13477915#comment-13477915
 ] 

Shinichi Yamashita commented on HADOOP-8928:


The improvement necessary to clear topology cache is as follows.

1. Define method to clear cache in CachedDNSToSwitchMapping
2. Define method to call 1. from NameNode and JobTracker
3. Add a protocol about 2.
4. Add a method about cache clear to DFSAdmin and MRAdmin
5. Add cache clear command to hadoop script

 Add ability to reset topologies on master nodes
 ---

 Key: HADOOP-8928
 URL: https://issues.apache.org/jira/browse/HADOOP-8928
 Project: Hadoop Common
  Issue Type: Improvement
  Components: net
Reporter: Shinichi Yamashita

 For a topology decision of DataNode and TaskTracker, ScriptBasedMapping 
 (probably TableMapping) confirms HashMap first.
 To decide topology of DataNode and TaskTracker again, it is necessary to 
 restart NameNode and JobTracker.
 Therefore, it is necessary to change (or clear) HashMap function without 
 restarting NameNode and JobTracker.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-7827) jsp pages missing DOCTYPE


[ 
https://issues.apache.org/jira/browse/HADOOP-7827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13477976#comment-13477976
 ] 

Suresh Srinivas commented on HADOOP-7827:
-

+1 for the branch-1.1 patch. Straightforward port to add !DOCTYPE HTML

 jsp pages missing DOCTYPE
 -

 Key: HADOOP-7827
 URL: https://issues.apache.org/jira/browse/HADOOP-7827
 Project: Hadoop Common
  Issue Type: Bug
Affects Versions: 0.20.203.0
Reporter: Dave Vronay
Assignee: Dave Vronay
Priority: Trivial
 Attachments: HADOOP-6496.branch-1.1.backport.patch, 
 HADOOP-7827-branch-0.20-security.patch, 
 HADOOP-7827.branch-1.1.backport.patch, HADOOP-7827.patch, HADOOP-7827.patch

   Original Estimate: 168h
  Remaining Estimate: 168h

 The various jsp pages in the UI are all missing a DOCTYPE declaration.  This 
 causes the pages to render incorrectly on some browsers, such as IE9.  Every 
 UI page should have a valid tag, such as !DOCTYPE HTML, as their first 
 line.  There are 31 files that need to be changed, all in the 
 core\src\webapps tree.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HADOOP-7827) jsp pages missing DOCTYPE


 [ 
https://issues.apache.org/jira/browse/HADOOP-7827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Suresh Srinivas updated HADOOP-7827:


Fix Version/s: 2.0.0-alpha
   1.2.0
   Status: Patch Available  (was: Reopened)

I committed the patch to branch-1. Thank you Ivan.

 jsp pages missing DOCTYPE
 -

 Key: HADOOP-7827
 URL: https://issues.apache.org/jira/browse/HADOOP-7827
 Project: Hadoop Common
  Issue Type: Bug
Affects Versions: 0.20.203.0
Reporter: Dave Vronay
Assignee: Dave Vronay
Priority: Trivial
 Fix For: 1.2.0, 2.0.0-alpha

 Attachments: HADOOP-6496.branch-1.1.backport.patch, 
 HADOOP-7827-branch-0.20-security.patch, 
 HADOOP-7827.branch-1.1.backport.patch, HADOOP-7827.patch, HADOOP-7827.patch

   Original Estimate: 168h
  Remaining Estimate: 168h

 The various jsp pages in the UI are all missing a DOCTYPE declaration.  This 
 causes the pages to render incorrectly on some browsers, such as IE9.  Every 
 UI page should have a valid tag, such as !DOCTYPE HTML, as their first 
 line.  There are 31 files that need to be changed, all in the 
 core\src\webapps tree.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-7827) jsp pages missing DOCTYPE

[
https://issues.apache.org/jira/browse/HADOOP-7827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13477983#comment-13477983
]

Hadoop QA commented on HADOOP-7827:
---

{color:red}-1 overall{color}. Here are the results of testing the latest
attachment

http://issues.apache.org/jira/secure/attachment/12549458/HADOOP-7827.branch-1.1.backport.patch
against trunk revision .

{color:red}-1 patch{color}. The patch command could not apply the patch.

Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/1640//console

This message is automatically generated.

jsp pages missing DOCTYPE
-

Key: HADOOP-7827
URL: https://issues.apache.org/jira/browse/HADOOP-7827
Project: Hadoop Common
Issue Type: Bug
Affects Versions: 0.20.203.0
Reporter: Dave Vronay
Assignee: Dave Vronay
Priority: Trivial
Fix For: 1.2.0, 2.0.0-alpha

Attachments: HADOOP-6496.branch-1.1.backport.patch,
HADOOP-7827-branch-0.20-security.patch,
HADOOP-7827.branch-1.1.backport.patch, HADOOP-7827.patch, HADOOP-7827.patch

Original Estimate: 168h
Remaining Estimate: 168h

The various jsp pages in the UI are all missing a DOCTYPE declaration. This
causes the pages to render incorrectly on some browsers, such as IE9. Every
UI page should have a valid tag, such as !DOCTYPE HTML, as their first
line. There are 31 files that need to be changed, all in the
core\src\webapps tree.

[jira] [Commented] (HADOOP-8779) Use tokens regardless of authentication type

2012-10-17 Thread Daryn Sharp (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-8779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13478008#comment-13478008
 ] 

Daryn Sharp commented on HADOOP-8779:
-

I believe you are misunderstanding the patch, or I have an unintentional bug.  
To save people a trip to HDFS-4056, here's the relevant info:
bq. SIMPLE is still supported as it currently is today. The difference is that 
a SIMPLE auth connection may also use a token, whereas today an insecure 
cluster will kick a client back into SIMPLE auth if it attempts to negotiate 
DIGEST-MD5 with a token. Clients are not strictly required to use a token.

 Use tokens regardless of authentication type
 

 Key: HADOOP-8779
 URL: https://issues.apache.org/jira/browse/HADOOP-8779
 Project: Hadoop Common
  Issue Type: New Feature
  Components: fs, security
Affects Versions: 3.0.0, 2.0.2-alpha
Reporter: Daryn Sharp
Assignee: Daryn Sharp

 Security is a combination of authentication and authorization (tokens).  
 Authorization may be granted independently of the authentication model.  
 Tokens should be used regardless of simple or kerberos authentication.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-8938) add option to do better diags of startup configuration

2012-10-17 Thread Daryn Sharp (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-8938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13478011#comment-13478011
 ] 

Daryn Sharp commented on HADOOP-8938:
-

Although by no means perfect and I'm not promoting it as the solution, I often 
use sh -x hadoop-cmd to aid in debugging.

 add option to do better diags of startup configuration
 --

 Key: HADOOP-8938
 URL: https://issues.apache.org/jira/browse/HADOOP-8938
 Project: Hadoop Common
  Issue Type: New Feature
  Components: bin
Affects Versions: 1.1.0, 2.0.2-alpha
Reporter: Steve Loughran
Priority: Minor
 Attachments: hdiag.py


 HADOOP-8931 shows a symptom of a larger problem: we need better diagnostics 
 of what all the environment variables and settings going through the hadoop 
 scripts to find out why something isn't working. 
 Ideally some command line parameter to the scripts (or even: a new 
 environment variable) could trigger more display of critical parameters.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HADOOP-7827) jsp pages missing DOCTYPE


 [ 
https://issues.apache.org/jira/browse/HADOOP-7827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Suresh Srinivas updated HADOOP-7827:


  Resolution: Fixed
Target Version/s: 1.2.0  (was: 1.1.0)
  Status: Resolved  (was: Patch Available)

 jsp pages missing DOCTYPE
 -

 Key: HADOOP-7827
 URL: https://issues.apache.org/jira/browse/HADOOP-7827
 Project: Hadoop Common
  Issue Type: Bug
Affects Versions: 0.20.203.0
Reporter: Dave Vronay
Assignee: Dave Vronay
Priority: Trivial
 Fix For: 1.2.0, 2.0.0-alpha

 Attachments: HADOOP-6496.branch-1.1.backport.patch, 
 HADOOP-7827-branch-0.20-security.patch, 
 HADOOP-7827.branch-1.1.backport.patch, HADOOP-7827.patch, HADOOP-7827.patch

   Original Estimate: 168h
  Remaining Estimate: 168h

 The various jsp pages in the UI are all missing a DOCTYPE declaration.  This 
 causes the pages to render incorrectly on some browsers, such as IE9.  Every 
 UI page should have a valid tag, such as !DOCTYPE HTML, as their first 
 line.  There are 31 files that need to be changed, all in the 
 core\src\webapps tree.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-8888) add the ability to suppress the deprecated warnings when using hadoop cli

2012-10-17 Thread Harsh J (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13478048#comment-13478048
 ] 

Harsh J commented on HADOOP-:
-

Hi Arpit,

I've not seen a pressing need for this yet, given that the messages are 
informative enough. So I'm ok if we're not doing this.

Maybe it would be worth it in future, for the code, to have like a single 
deprecation log channel, toggle-able at the global, package and class levels.

Same could be done for scripts I think.

 add the ability to suppress the deprecated warnings when using hadoop cli
 -

 Key: HADOOP-
 URL: https://issues.apache.org/jira/browse/HADOOP-
 Project: Hadoop Common
  Issue Type: Improvement
Affects Versions: 3.0.0
Reporter: Arpit Gupta
Assignee: Arpit Gupta

 some this similar to what HADOOP_HOME_WARN_SUPPRESS is used for in branch-1
 May be we can introduce
 HADOOP_DEPRECATED_WARN_SUPPRESS
 which if set to yes will suppress the various warnings that are thrown.
 For example commands like
 {code}
 hadoop dfs
 hadoop jar
 {code}
 etc will print out deprecated warnings.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Resolved] (HADOOP-8935) Make 'winutils ls' show the SID if the owner does not exist on the system


 [ 
https://issues.apache.org/jira/browse/HADOOP-8935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Suresh Srinivas resolved HADOOP-8935.
-

  Resolution: Fixed
Hadoop Flags: Reviewed

+1. I committed the patch to branch-1-win.

Thank you Chuan.

 Make 'winutils ls' show the SID if the owner does not exist on the system
 -

 Key: HADOOP-8935
 URL: https://issues.apache.org/jira/browse/HADOOP-8935
 Project: Hadoop Common
  Issue Type: Bug
Affects Versions: 1-win
Reporter: Chuan Liu
Assignee: Chuan Liu
Priority: Minor
 Fix For: 1-win

 Attachments: HADOOP-8935-branch-1-win.patch


 Right now, 'winutils ls' will fail if the file belongs to a user SID that 
 does not exist on the system. E.g. the user is deleted. Previously, this is 
 only a hypothesis scenario. However, we have seen some failures in the Azure 
 deployment where the OS is re-imaged, and renders the old SID invalid. 
 [~jgordon] proposed to display the SID itself in the invalid SID case similar 
 to the situation on Linux. This JIRA is created to track this proposal.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-8899) Classpath exceeds maximum OS limit

[
https://issues.apache.org/jira/browse/HADOOP-8899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13478069#comment-13478069
]

Suresh Srinivas commented on HADOOP-8899:
-

Alejandro and Joep, please look at Ahmed's comments. I am +1 for this change
given maven surefire.

I will commit this in a day or so, if there are no further comments. Comments
post that can be addressed in a separate jira.

Classpath exceeds maximum OS limit
--

Key: HADOOP-8899
URL: https://issues.apache.org/jira/browse/HADOOP-8899
Project: Hadoop Common
Issue Type: Bug
Affects Versions: 1-win
Reporter: Ahmed El Baz
Attachments: HADOOP-8899.branch-1-win.classpath.patch

We hit errors in TT due to long classpath value. One example is Oozie trying
to start a hive job, and it fails with the following error:
java.io.IOException: Command exceeds the OS command length limit: 8192,
command: set CLASSPATH=….
The classpath includes Hadoop Core + Hive Jars which are in the distributed
cache. This is causing the classpath to be too long giving the error above. A
viable long term fix is to generate a temporary JAR file in the task cache
directory which includes all elements in the long classpath, and then provide
this as the -classpath argument for the JVM to be spawned. This is gated
for Windows only.

[jira] [Commented] (HADOOP-8938) add option to do better diags of startup configuration


[ 
https://issues.apache.org/jira/browse/HADOOP-8938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13478084#comment-13478084
 ] 

Steve Loughran commented on HADOOP-8938:


@Daryn -that and bash -v ... they do require understanding of bash though.

 add option to do better diags of startup configuration
 --

 Key: HADOOP-8938
 URL: https://issues.apache.org/jira/browse/HADOOP-8938
 Project: Hadoop Common
  Issue Type: New Feature
  Components: bin
Affects Versions: 1.1.0, 2.0.2-alpha
Reporter: Steve Loughran
Priority: Minor
 Attachments: hdiag.py


 HADOOP-8931 shows a symptom of a larger problem: we need better diagnostics 
 of what all the environment variables and settings going through the hadoop 
 scripts to find out why something isn't working. 
 Ideally some command line parameter to the scripts (or even: a new 
 environment variable) could trigger more display of critical parameters.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HADOOP-8900) BuiltInGzipDecompressor throws IOException - stored gzip size doesn't match decompressed size


 [ 
https://issues.apache.org/jira/browse/HADOOP-8900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Suresh Srinivas updated HADOOP-8900:


Summary: BuiltInGzipDecompressor throws IOException - stored gzip size 
doesn't match decompressed size  (was: BuiltInGzipDecompressor : 
java.io.IOException: stored gzip size doesn't match decompressed size (Slavik 
Krassovsky))

 BuiltInGzipDecompressor throws IOException - stored gzip size doesn't match 
 decompressed size
 -

 Key: HADOOP-8900
 URL: https://issues.apache.org/jira/browse/HADOOP-8900
 Project: Hadoop Common
  Issue Type: Bug
Affects Versions: 1-win, 2.0.1-alpha
 Environment: Encountered failure when processing large GZIP file
Reporter: Slavik Krassovsky
Assignee: Andy Isaacson
 Attachments: BuiltInGzipDecompressor2.patch, hadoop8900-2.txt, 
 hadoop8900.txt


 Encountered failure when processing large GZIP file
 • Gz: Failed in 1hrs, 13mins, 57sec with the error:
  ¸java.io.IOException: IO error in map input file 
 hdfs://localhost:9000/Halo4/json_m/gz/NewFileCat.txt.gz
  at 
 org.apache.hadoop.mapred.MapTask$TrackedRecordReader.moveToNext(MapTask.java:242)
  at 
 org.apache.hadoop.mapred.MapTask$TrackedRecordReader.next(MapTask.java:216)
  at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:48)
  at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:435)
  at org.apache.hadoop.mapred.MapTask.run(MapTask.java:371)
  at org.apache.hadoop.mapred.Child$4.run(Child.java:266)
  at java.security.AccessController.doPrivileged(Native Method)
  at javax.security.auth.Subject.doAs(Subject.java:415)
  at 
 org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
  at org.apache.hadoop.mapred.Child.main(Child.java:260)
  Caused by: java.io.IOException: stored gzip size doesn't match decompressed 
 size
  at 
 org.apache.hadoop.io.compress.zlib.BuiltInGzipDecompressor.executeTrailerState(BuiltInGzipDecompressor.java:389)
  at 
 org.apache.hadoop.io.compress.zlib.BuiltInGzipDecompressor.decompress(BuiltInGzipDecompressor.java:224)
  at 
 org.apache.hadoop.io.compress.DecompressorStream.decompress(DecompressorStream.java:82)
  at 
 org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:76)
  at java.io.InputStream.read(InputStream.java:102)
  at org.apache.hadoop.util.LineReader.readLine(LineReader.java:134)
  at org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:136)
  at org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:40)
  at 
 org.apache.hadoop.hive.ql.io.HiveRecordReader.doNext(HiveRecordReader.java:66)
  at 
 org.apache.hadoop.hive.ql.io.HiveRecordReader.doNext(HiveRecordReader.java:32)
  at 
 org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.next(HiveContextAwareRecordReader.java:67)
  at 
 org.apache.hadoop.mapred.MapTask$TrackedRecordReader.moveToNext(MapTask.java:236)
  ... 9 more

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-8899) Classpath exceeds maximum OS limit

[
https://issues.apache.org/jira/browse/HADOOP-8899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13478197#comment-13478197
]

Alejandro Abdelnur commented on HADOOP-8899:

I think the approach is not correct, reiterating on my prev comment. if more
than one of the JARs being consolidated contains a META-INF service definition
for the same service (ie o.a.h.fs.FileSystem,
o.a.h.io.compress.CompressionCodec, o.a.h.security.SecurityInfo), then things
will not work as expected as you'll lose some of those files in the
consolidation.

Classpath exceeds maximum OS limit
--

[jira] [Commented] (HADOOP-8564) Port and extend Hadoop native libraries for Windows to address datanode concurrent reading and writing issue

2012-10-17 Thread Todd Lipcon (JIRA)

[
https://issues.apache.org/jira/browse/HADOOP-8564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13478207#comment-13478207
]

Todd Lipcon commented on HADOOP-8564:
-

Sorry, I missed that the new patch was uploaded. Can I have a couple days to
review it? It's a big patch. If you want to go ahead and commit to the branch,
that's OK so long as review feedback can be addressed afterwards.

Port and extend Hadoop native libraries for Windows to address datanode
concurrent reading and writing issue

[jira] [Commented] (HADOOP-8900) BuiltInGzipDecompressor throws IOException - stored gzip size doesn't match decompressed size


[ 
https://issues.apache.org/jira/browse/HADOOP-8900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13478225#comment-13478225
 ] 

Hudson commented on HADOOP-8900:


Integrated in Hadoop-trunk-Commit #2882 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/2882/])
HADOOP-8900. BuiltInGzipDecompressor throws IOException - stored gzip size 
doesn't match decompressed size. Contributed by Slavik Krassovsky. (Revision 
1399377)

 Result = SUCCESS
suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1399377
Files : 
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/zlib/BuiltInGzipDecompressor.java
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/compress/TestCodec.java
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/file/tfile/TestVLong.java


 BuiltInGzipDecompressor throws IOException - stored gzip size doesn't match 
 decompressed size
 -

 Key: HADOOP-8900
 URL: https://issues.apache.org/jira/browse/HADOOP-8900
 Project: Hadoop Common
  Issue Type: Bug
Affects Versions: 1-win, 2.0.1-alpha
 Environment: Encountered failure when processing large GZIP file
Reporter: Slavik Krassovsky
Assignee: Andy Isaacson
 Attachments: BuiltInGzipDecompressor2.patch, hadoop8900-2.txt, 
 hadoop8900.txt


 Encountered failure when processing large GZIP file
 • Gz: Failed in 1hrs, 13mins, 57sec with the error:
  ¸java.io.IOException: IO error in map input file 
 hdfs://localhost:9000/Halo4/json_m/gz/NewFileCat.txt.gz
  at 
 org.apache.hadoop.mapred.MapTask$TrackedRecordReader.moveToNext(MapTask.java:242)
  at 
 org.apache.hadoop.mapred.MapTask$TrackedRecordReader.next(MapTask.java:216)
  at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:48)
  at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:435)
  at org.apache.hadoop.mapred.MapTask.run(MapTask.java:371)
  at org.apache.hadoop.mapred.Child$4.run(Child.java:266)
  at java.security.AccessController.doPrivileged(Native Method)
  at javax.security.auth.Subject.doAs(Subject.java:415)
  at 
 org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
  at org.apache.hadoop.mapred.Child.main(Child.java:260)
  Caused by: java.io.IOException: stored gzip size doesn't match decompressed 
 size
  at 
 org.apache.hadoop.io.compress.zlib.BuiltInGzipDecompressor.executeTrailerState(BuiltInGzipDecompressor.java:389)
  at 
 org.apache.hadoop.io.compress.zlib.BuiltInGzipDecompressor.decompress(BuiltInGzipDecompressor.java:224)
  at 
 org.apache.hadoop.io.compress.DecompressorStream.decompress(DecompressorStream.java:82)
  at 
 org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:76)
  at java.io.InputStream.read(InputStream.java:102)
  at org.apache.hadoop.util.LineReader.readLine(LineReader.java:134)
  at org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:136)
  at org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:40)
  at 
 org.apache.hadoop.hive.ql.io.HiveRecordReader.doNext(HiveRecordReader.java:66)
  at 
 org.apache.hadoop.hive.ql.io.HiveRecordReader.doNext(HiveRecordReader.java:32)
  at 
 org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.next(HiveContextAwareRecordReader.java:67)
  at 
 org.apache.hadoop.mapred.MapTask$TrackedRecordReader.moveToNext(MapTask.java:236)
  ... 9 more

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-8925) Remove packaging

2012-10-17 Thread Eli Collins (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-8925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13478245#comment-13478245
 ] 

Eli Collins commented on HADOOP-8925:
-

Here's the thread btw: http://s.apache.org/mIS

 Remove packaging
 

 Key: HADOOP-8925
 URL: https://issues.apache.org/jira/browse/HADOOP-8925
 Project: Hadoop Common
  Issue Type: Improvement
  Components: build
Affects Versions: 2.0.0-alpha
Reporter: Eli Collins
Assignee: Eli Collins
 Attachments: hadoop-8925.txt


 Per discussion on HADOOP-8809, now that Bigtop is TLP and supports Hadoop v2 
 let's remove the Hadoop packaging from trunk and branch-2. We should remove 
 it anyway since it no longer part of the build post mavenization, was not 
 updated post MR1 (there's no MR2/YARN packaging) and is not maintained.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HADOOP-8900) BuiltInGzipDecompressor throws IOException - stored gzip size doesn't match decompressed size


 [ 
https://issues.apache.org/jira/browse/HADOOP-8900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Suresh Srinivas updated HADOOP-8900:


Fix Version/s: 2.0.3-alpha
   3.0.0

+1 for the patch.

I committed the patch to trunk and branch-2. I will post it to branch-1, once 
the patch for it is posted.

 BuiltInGzipDecompressor throws IOException - stored gzip size doesn't match 
 decompressed size
 -

 Key: HADOOP-8900
 URL: https://issues.apache.org/jira/browse/HADOOP-8900
 Project: Hadoop Common
  Issue Type: Bug
Affects Versions: 1-win, 2.0.1-alpha
 Environment: Encountered failure when processing large GZIP file
Reporter: Slavik Krassovsky
Assignee: Andy Isaacson
 Fix For: 3.0.0, 2.0.3-alpha

 Attachments: BuiltInGzipDecompressor2.patch, hadoop8900-2.txt, 
 hadoop8900.txt


 Encountered failure when processing large GZIP file
 • Gz: Failed in 1hrs, 13mins, 57sec with the error:
  ¸java.io.IOException: IO error in map input file 
 hdfs://localhost:9000/Halo4/json_m/gz/NewFileCat.txt.gz
  at 
 org.apache.hadoop.mapred.MapTask$TrackedRecordReader.moveToNext(MapTask.java:242)
  at 
 org.apache.hadoop.mapred.MapTask$TrackedRecordReader.next(MapTask.java:216)
  at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:48)
  at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:435)
  at org.apache.hadoop.mapred.MapTask.run(MapTask.java:371)
  at org.apache.hadoop.mapred.Child$4.run(Child.java:266)
  at java.security.AccessController.doPrivileged(Native Method)
  at javax.security.auth.Subject.doAs(Subject.java:415)
  at 
 org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
  at org.apache.hadoop.mapred.Child.main(Child.java:260)
  Caused by: java.io.IOException: stored gzip size doesn't match decompressed 
 size
  at 
 org.apache.hadoop.io.compress.zlib.BuiltInGzipDecompressor.executeTrailerState(BuiltInGzipDecompressor.java:389)
  at 
 org.apache.hadoop.io.compress.zlib.BuiltInGzipDecompressor.decompress(BuiltInGzipDecompressor.java:224)
  at 
 org.apache.hadoop.io.compress.DecompressorStream.decompress(DecompressorStream.java:82)
  at 
 org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:76)
  at java.io.InputStream.read(InputStream.java:102)
  at org.apache.hadoop.util.LineReader.readLine(LineReader.java:134)
  at org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:136)
  at org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:40)
  at 
 org.apache.hadoop.hive.ql.io.HiveRecordReader.doNext(HiveRecordReader.java:66)
  at 
 org.apache.hadoop.hive.ql.io.HiveRecordReader.doNext(HiveRecordReader.java:32)
  at 
 org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.next(HiveContextAwareRecordReader.java:67)
  at 
 org.apache.hadoop.mapred.MapTask$TrackedRecordReader.moveToNext(MapTask.java:236)
  ... 9 more

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-8887) Use a Maven plugin to build the native code using CMake


[ 
https://issues.apache.org/jira/browse/HADOOP-8887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13478267#comment-13478267
 ] 

Alejandro Abdelnur commented on HADOOP-8887:


+1 pending jenkins.

 Use a Maven plugin to build the native code using CMake
 ---

 Key: HADOOP-8887
 URL: https://issues.apache.org/jira/browse/HADOOP-8887
 Project: Hadoop Common
  Issue Type: Improvement
  Components: build
Affects Versions: 2.0.3-alpha
Reporter: Colin Patrick McCabe
Assignee: Colin Patrick McCabe
Priority: Minor
 Attachments: HADOOP-8887.001.patch, HADOOP-8887.002.patch, 
 HADOOP-8887.003.patch, HADOOP-8887.004.patch, HADOOP-8887.005.patch


 Currently, we build the native code using ant-build invocations.  Although 
 this works, it has some limitations:
 * compiler warning messages are hidden, which can cause people to check in 
 code with warnings unintentionally
 * there is no framework for running native unit tests; instead, we use ad-hoc 
 constructs involving shell scripts
 * the antrun code is very platform specific
 * there is no way to run a specific native unit test
 * it's more or less impossible for scripts like test-patch.sh to separate a 
 native test failing from the build itself failing (no files are created) or 
 to enumerate which native tests failed.
 Using a native Maven plugin would overcome these limitations.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-8925) Remove packaging


[ 
https://issues.apache.org/jira/browse/HADOOP-8925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13478300#comment-13478300
 ] 

Alejandro Abdelnur commented on HADOOP-8925:


+1, did a tarball build to verify nothing is broken.

 Remove packaging
 

 Key: HADOOP-8925
 URL: https://issues.apache.org/jira/browse/HADOOP-8925
 Project: Hadoop Common
  Issue Type: Improvement
  Components: build
Affects Versions: 2.0.0-alpha
Reporter: Eli Collins
Assignee: Eli Collins
 Attachments: hadoop-8925.txt


 Per discussion on HADOOP-8809, now that Bigtop is TLP and supports Hadoop v2 
 let's remove the Hadoop packaging from trunk and branch-2. We should remove 
 it anyway since it no longer part of the build post mavenization, was not 
 updated post MR1 (there's no MR2/YARN packaging) and is not maintained.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-8926) hadoop.util.PureJavaCrc32 cache hit-ratio is low for static data


[ 
https://issues.apache.org/jira/browse/HADOOP-8926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13478329#comment-13478329
 ] 

Tsz Wo (Nicholas), SZE commented on HADOOP-8926:


Gopal, Amazing works!  There are 10% or more performance improvement from your 
patch.

java.version = 1.6.0_35
java.runtime.name = Java(TM) SE Runtime Environment
java.runtime.version = 1.6.0_35-b10-428-11M3811
java.vm.version = 20.10-b01-428
java.vm.vendor = Apple Inc.
java.vm.name = Java HotSpot(TM) 64-Bit Server VM
java.vm.specification.version = 1.0
java.specification.version = 1.6
os.arch = x86_64
os.name = Mac OS X
os.version = 10.7.4

Performance Table (The unit is MB/sec)
|| Num Bytes ||CRC32 || PureJavaCrc32_8926 || PureJavaCrc32 ||
|  1 |11.896 | 129.832 |164.845 |
|  2 |24.097 | 192.742 |210.266 |
|  4 |46.274 | 222.059 |233.902 |
|  8 |82.332 | 488.716 |438.514 |
| 16 |   131.682 | 587.312 |602.784 |
| 32 |   187.265 | 796.510 |760.628 |
| 64 |   237.088 | 938.650 |891.017 |
|128 |   264.795 |1049.774 |913.666 |
|256 |   291.785 |1095.084 |987.380 |
|512 |   298.590 |1126.067 |   1002.899 |
|   1024 |   305.349 |1152.375 |   1040.211 |
|   2048 |   309.342 |1119.713 |   1033.258 |
|   4096 |   309.162 |1170.767 |   1047.746 |
|   8192 |   321.775 |1189.724 |   1053.065 |
|  16384 |   320.457 |1181.128 |   1060.138 |
|  32768 |   324.524 |1169.965 |   1050.610 |
|  65536 |   322.380 |1160.471 |   1053.854 |
| 131072 |   315.983 |1138.223 |   1009.193 |
| 262144 |   324.293 |1190.476 |   1020.782 |
| 524288 |   316.003 |1136.979 |   1015.389 |
|1048576 |   321.715 |1081.465 |   1033.750 |
|2097152 |   318.330 |1189.680 |   1072.054 |
|4194304 |   316.710 |1138.496 |   1024.352 |
|8388608 |   315.701 |1124.909 |   1030.505 |
|   16777216 |   325.575 |1154.724 |   1031.285 |


 hadoop.util.PureJavaCrc32 cache hit-ratio is low for static data
 

 Key: HADOOP-8926
 URL: https://issues.apache.org/jira/browse/HADOOP-8926
 Project: Hadoop Common
  Issue Type: Improvement
  Components: util
Affects Versions: 2.0.3-alpha
 Environment: Ubuntu 10.10 i386 
Reporter: Gopal V
Assignee: Gopal V
Priority: Trivial
  Labels: optimization
 Fix For: 3.0.0, 2.0.3-alpha

 Attachments: crc32-faster+readable.patch, crc32-faster+test.patch, 
 pure-crc32-cache-hit.patch


 While running microbenchmarks for HDFS write codepath, a significant part of 
 the CPU fraction was consumed by the DataChecksum.update(). 
 The attached patch converts the static arrays in CRC32 into a single linear 
 array for a performance boost in the inner loop.
 milli-seconds for 1Gig (16400 loop over a 64kb chunk) 
 || platform || original || cache-aware || improvement ||
 | x86 | 3894 | 2304 | 40.83 |
 | x86_64 | 2131 | 1826 | 14 | 
 The performance improvement on x86 is rather larger than the 64bit case, due 
 to the extra register/stack pressure caused by the static arrays.
 A closer analysis of the PureJavaCrc32 JIT code shows the following assembly 
 fragment
 {code}
   0x40f1e345: mov$0x184,%ecx
   0x40f1e34a: mov0x4415b560(%ecx),%ecx  ;*getstatic T8_5
 ; - PureJavaCrc32::update@95 (line 61)
 ;   {oop('PureJavaCrc32')}
   0x40f1e350: mov%ecx,0x2c(%esp)
 {code}
 Basically, the static variables T8_0 through to T8_7 are being spilled to the 
 stack because of register pressure. The x86_64 case has a lower likelihood of 
 such pessimistic JIT code due to the increased number of registers.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-8779) Use tokens regardless of authentication type

2012-10-17 Thread Kan Zhang (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-8779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13478331#comment-13478331
 ] 

Kan Zhang commented on HADOOP-8779:
---

On the server side, you can support both SIMPLE and DIEGST-MD5 (token) 
simultaneously. It's up to the client to choose which method to use. But the 
client side code needs to make a decision. Without an option to specify the 
desired behavior, the client code can only default to one behavior. See more 
discussion in HDFS-4056.

 Use tokens regardless of authentication type
 

 Key: HADOOP-8779
 URL: https://issues.apache.org/jira/browse/HADOOP-8779
 Project: Hadoop Common
  Issue Type: New Feature
  Components: fs, security
Affects Versions: 3.0.0, 2.0.2-alpha
Reporter: Daryn Sharp
Assignee: Daryn Sharp

 Security is a combination of authentication and authorization (tokens).  
 Authorization may be granted independently of the authentication model.  
 Tokens should be used regardless of simple or kerberos authentication.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HADOOP-8926) hadoop.util.PureJavaCrc32 cache hit-ratio is low for static data


 [ 
https://issues.apache.org/jira/browse/HADOOP-8926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tsz Wo (Nicholas), SZE updated HADOOP-8926:
---

Priority: Major  (was: Trivial)

 hadoop.util.PureJavaCrc32 cache hit-ratio is low for static data
 

 Key: HADOOP-8926
 URL: https://issues.apache.org/jira/browse/HADOOP-8926
 Project: Hadoop Common
  Issue Type: Improvement
  Components: util
Affects Versions: 2.0.3-alpha
 Environment: Ubuntu 10.10 i386 
Reporter: Gopal V
Assignee: Gopal V
  Labels: optimization
 Fix For: 3.0.0, 2.0.3-alpha

 Attachments: crc32-faster+readable.patch, crc32-faster+test.patch, 
 pure-crc32-cache-hit.patch


 While running microbenchmarks for HDFS write codepath, a significant part of 
 the CPU fraction was consumed by the DataChecksum.update(). 
 The attached patch converts the static arrays in CRC32 into a single linear 
 array for a performance boost in the inner loop.
 milli-seconds for 1Gig (16400 loop over a 64kb chunk) 
 || platform || original || cache-aware || improvement ||
 | x86 | 3894 | 2304 | 40.83 |
 | x86_64 | 2131 | 1826 | 14 | 
 The performance improvement on x86 is rather larger than the 64bit case, due 
 to the extra register/stack pressure caused by the static arrays.
 A closer analysis of the PureJavaCrc32 JIT code shows the following assembly 
 fragment
 {code}
   0x40f1e345: mov$0x184,%ecx
   0x40f1e34a: mov0x4415b560(%ecx),%ecx  ;*getstatic T8_5
 ; - PureJavaCrc32::update@95 (line 61)
 ;   {oop('PureJavaCrc32')}
   0x40f1e350: mov%ecx,0x2c(%esp)
 {code}
 Basically, the static variables T8_0 through to T8_7 are being spilled to the 
 stack because of register pressure. The x86_64 case has a lower likelihood of 
 such pessimistic JIT code due to the increased number of registers.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-8930) Cumulative code coverage calculation

2012-10-17 Thread Robert Joseph Evans (JIRA)

[
https://issues.apache.org/jira/browse/HADOOP-8930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13478338#comment-13478338
]

Robert Joseph Evans commented on HADOOP-8930:
-

The change looks fine to me, Clover seems to build and run at least as well as
it did before. I am +1 for the change.

Cumulative code coverage calculation

Key: HADOOP-8930
URL: https://issues.apache.org/jira/browse/HADOOP-8930
Project: Hadoop Common
Issue Type: Improvement
Components: test
Affects Versions: 0.23.3, 2.0.2-alpha
Reporter: Andrey Klochkov
Assignee: Andrey Klochkov
Attachments: HADOOP-8930-branch-0.23.patch, HADOOP-8930.patch

When analyzing code coverage in Hadoop Core, we noticed that some coverage
gaps are caused by the way the coverage calculation is done currently. More
specifically, right now coverage can not be calculated for the whole Core at
once, but can only be calculated separately for top level modules like
common-project, hadoop-hdfs-project etc.
At the same time, some code in particular modules is tested by tests in other
modules of Core. For example, org.apache.hadoop.fs from
hadoop-common-project/hadoop-common is not covered there but it's covered by
tests under hadoop-hdfs-project.
To enable calculation of cumulative code coverage it's needed to move
Clover profile definition up one level, from hadoop-project/pom.xml to the
top level pom.xml (hadoop-main).
Patch both for 0.23 and 2.x will be attached shortly.

[jira] [Commented] (HADOOP-8564) Port and extend Hadoop native libraries for Windows to address datanode concurrent reading and writing issue

[
https://issues.apache.org/jira/browse/HADOOP-8564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13478340#comment-13478340
]

Suresh Srinivas commented on HADOOP-8564:
-

I will commit this patch, since there are other patches that are dependent on
this.

One you post your review comments, it can be addressed in another Jira.

Port and extend Hadoop native libraries for Windows to address datanode
concurrent reading and writing issue

[jira] [Commented] (HADOOP-8887) Use a Maven plugin to build the native code using CMake


[ 
https://issues.apache.org/jira/browse/HADOOP-8887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13478341#comment-13478341
 ] 

Hadoop QA commented on HADOOP-8887:
---

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12548991/HADOOP-8887.005.patch
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 3 new 
or modified test files.

  {color:red}-1 javac{color}.  The applied patch generated 2031 javac 
compiler warnings (more than the trunk's current 2030 warnings).

{color:red}-1 javadoc{color}.  The javadoc tool appears to have generated 4 
warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 core tests{color}.  The patch passed unit tests in 
dev-support/cmake-maven-ng-plugin hadoop-common-project/hadoop-common 
hadoop-hdfs-project/hadoop-hdfs 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager.

{color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: 
https://builds.apache.org/job/PreCommit-HADOOP-Build/1641//testReport/
Javac warnings: 
https://builds.apache.org/job/PreCommit-HADOOP-Build/1641//artifact/trunk/patchprocess/diffJavacWarnings.txt
Console output: 
https://builds.apache.org/job/PreCommit-HADOOP-Build/1641//console

This message is automatically generated.

 Use a Maven plugin to build the native code using CMake
 ---

 Key: HADOOP-8887
 URL: https://issues.apache.org/jira/browse/HADOOP-8887
 Project: Hadoop Common
  Issue Type: Improvement
  Components: build
Affects Versions: 2.0.3-alpha
Reporter: Colin Patrick McCabe
Assignee: Colin Patrick McCabe
Priority: Minor
 Attachments: HADOOP-8887.001.patch, HADOOP-8887.002.patch, 
 HADOOP-8887.003.patch, HADOOP-8887.004.patch, HADOOP-8887.005.patch


 Currently, we build the native code using ant-build invocations.  Although 
 this works, it has some limitations:
 * compiler warning messages are hidden, which can cause people to check in 
 code with warnings unintentionally
 * there is no framework for running native unit tests; instead, we use ad-hoc 
 constructs involving shell scripts
 * the antrun code is very platform specific
 * there is no way to run a specific native unit test
 * it's more or less impossible for scripts like test-patch.sh to separate a 
 native test failing from the build itself failing (no files are created) or 
 to enumerate which native tests failed.
 Using a native Maven plugin would overcome these limitations.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-8930) Cumulative code coverage calculation


[ 
https://issues.apache.org/jira/browse/HADOOP-8930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13478365#comment-13478365
 ] 

Hudson commented on HADOOP-8930:


Integrated in Hadoop-trunk-Commit #2883 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/2883/])
HADOOP-8930. Cumulative code coverage calculation (Andrey Klochkov via 
bobby) (Revision 1399426)

 Result = SUCCESS
bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1399426
Files : 
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
* /hadoop/common/trunk/hadoop-project/pom.xml
* /hadoop/common/trunk/pom.xml


 Cumulative code coverage calculation
 

 Key: HADOOP-8930
 URL: https://issues.apache.org/jira/browse/HADOOP-8930
 Project: Hadoop Common
  Issue Type: Improvement
  Components: test
Affects Versions: 0.23.3, 2.0.2-alpha
Reporter: Andrey Klochkov
Assignee: Andrey Klochkov
 Attachments: HADOOP-8930-branch-0.23.patch, HADOOP-8930.patch


 When analyzing code coverage in Hadoop Core, we noticed that some coverage 
 gaps are caused by the way the coverage calculation is done currently. More 
 specifically, right now coverage can not be calculated for the whole Core at 
 once, but can only be calculated separately for top level modules like 
 common-project, hadoop-hdfs-project etc. 
 At the same time, some code in particular modules is tested by tests in other 
 modules of Core. For example, org.apache.hadoop.fs from 
 hadoop-common-project/hadoop-common is not covered there but it's covered by 
 tests under hadoop-hdfs-project.
 To enable calculation of cumulative code coverage it's needed to move 
 Clover profile definition up one level, from hadoop-project/pom.xml to the 
 top level pom.xml (hadoop-main).
 Patch both for 0.23 and 2.x will be attached shortly.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HADOOP-8930) Cumulative code coverage calculation

2012-10-17 Thread Robert Joseph Evans (JIRA)


 [ 
https://issues.apache.org/jira/browse/HADOOP-8930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Joseph Evans updated HADOOP-8930:


   Resolution: Fixed
Fix Version/s: 0.23.5
   2.0.3-alpha
   3.0.0
   Status: Resolved  (was: Patch Available)

Thanks Andrey,

I pulled this into trunk, branch-2, and branch-0.23.

 Cumulative code coverage calculation
 

 Key: HADOOP-8930
 URL: https://issues.apache.org/jira/browse/HADOOP-8930
 Project: Hadoop Common
  Issue Type: Improvement
  Components: test
Affects Versions: 0.23.3, 2.0.2-alpha
Reporter: Andrey Klochkov
Assignee: Andrey Klochkov
 Fix For: 3.0.0, 2.0.3-alpha, 0.23.5

 Attachments: HADOOP-8930-branch-0.23.patch, HADOOP-8930.patch


 When analyzing code coverage in Hadoop Core, we noticed that some coverage 
 gaps are caused by the way the coverage calculation is done currently. More 
 specifically, right now coverage can not be calculated for the whole Core at 
 once, but can only be calculated separately for top level modules like 
 common-project, hadoop-hdfs-project etc. 
 At the same time, some code in particular modules is tested by tests in other 
 modules of Core. For example, org.apache.hadoop.fs from 
 hadoop-common-project/hadoop-common is not covered there but it's covered by 
 tests under hadoop-hdfs-project.
 To enable calculation of cumulative code coverage it's needed to move 
 Clover profile definition up one level, from hadoop-project/pom.xml to the 
 top level pom.xml (hadoop-main).
 Patch both for 0.23 and 2.x will be attached shortly.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HADOOP-8902) Enable Gridmix v1 v2 benchmarks on Windows platform

2012-10-17 Thread Mike Liddell (JIRA)


 [ 
https://issues.apache.org/jira/browse/HADOOP-8902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mike Liddell updated HADOOP-8902:
-

Attachment: HADOOP-8902.branch-1-win.contribscripts.patch

 Enable Gridmix v1  v2 benchmarks on Windows platform
 -

 Key: HADOOP-8902
 URL: https://issues.apache.org/jira/browse/HADOOP-8902
 Project: Hadoop Common
  Issue Type: Bug
  Components: benchmarks
Affects Versions: 1-win
Reporter: Mike Liddell
 Attachments: HADOOP-8902.branch-1-win.contribscripts.patch, 
 HADOOP-8902.patch


 Gridmix v1 and v2 benchmarks do not run on Windows as they require bash 
 shell.  These scripts have been ported to Windows cmd-scripts.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-8902) Enable Gridmix v1 v2 benchmarks on Windows platform

2012-10-17 Thread Mike Liddell (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-8902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13478374#comment-13478374
 ] 

Mike Liddell commented on HADOOP-8902:
--

patch updated: HADOOP-8902.branch-1-win.contribscripts.patch

 Enable Gridmix v1  v2 benchmarks on Windows platform
 -

 Key: HADOOP-8902
 URL: https://issues.apache.org/jira/browse/HADOOP-8902
 Project: Hadoop Common
  Issue Type: Bug
  Components: benchmarks
Affects Versions: 1-win
Reporter: Mike Liddell
 Attachments: HADOOP-8902.branch-1-win.contribscripts.patch, 
 HADOOP-8902.patch


 Gridmix v1 and v2 benchmarks do not run on Windows as they require bash 
 shell.  These scripts have been ported to Windows cmd-scripts.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-8902) Enable Gridmix v1 v2 benchmarks on Windows platform


[ 
https://issues.apache.org/jira/browse/HADOOP-8902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13478375#comment-13478375
 ] 

Ivan Mitic commented on HADOOP-8902:


Thanks Mike, new patch looks good, +1

 Enable Gridmix v1  v2 benchmarks on Windows platform
 -

 Key: HADOOP-8902
 URL: https://issues.apache.org/jira/browse/HADOOP-8902
 Project: Hadoop Common
  Issue Type: Bug
  Components: benchmarks
Affects Versions: 1-win
Reporter: Mike Liddell
 Attachments: HADOOP-8902.branch-1-win.contribscripts.patch, 
 HADOOP-8902.patch


 Gridmix v1 and v2 benchmarks do not run on Windows as they require bash 
 shell.  These scripts have been ported to Windows cmd-scripts.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-8779) Use tokens regardless of authentication type

2012-10-17 Thread Daryn Sharp (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-8779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13478377#comment-13478377
 ] 

Daryn Sharp commented on HADOOP-8779:
-

The RPC client will use DIGEST-MD5 if and only if there's a token in the UGI.  
The rpc client used to only look for tokens if auth is kerberos, but 
HADOOP-8784 changed the client to always look for tokens regardless of auth 
method.  I believe your real question, which is legitimate, is whether the job 
client chooses to get fs tokens.  We can discuss that on this jira because it's 
unrelated to HDFS-4056.

 Use tokens regardless of authentication type
 

 Key: HADOOP-8779
 URL: https://issues.apache.org/jira/browse/HADOOP-8779
 Project: Hadoop Common
  Issue Type: New Feature
  Components: fs, security
Affects Versions: 3.0.0, 2.0.2-alpha
Reporter: Daryn Sharp
Assignee: Daryn Sharp

 Security is a combination of authentication and authorization (tokens).  
 Authorization may be granted independently of the authentication model.  
 Tokens should be used regardless of simple or kerberos authentication.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-8427) Convert Forrest docs to APT


[ 
https://issues.apache.org/jira/browse/HADOOP-8427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13478437#comment-13478437
 ] 

Tsz Wo (Nicholas), SZE commented on HADOOP-8427:


Hi Andy, could we first remove the out-of-date doc and then convert it to APT?  
It would be much easier to see the changes.

 Convert Forrest docs to APT
 ---

 Key: HADOOP-8427
 URL: https://issues.apache.org/jira/browse/HADOOP-8427
 Project: Hadoop Common
  Issue Type: Task
  Components: documentation
Affects Versions: 2.0.0-alpha
Reporter: Eli Collins
Assignee: Andy Isaacson
  Labels: newbie
 Attachments: hadoop8427.txt


 Some of the forrest docs content in src/docs/src/documentation/content/xdocs 
 has not yet been converted to APT and moved to src/site/apt. Let's convert 
 the forrest docs that haven't been converted yet to new APT content in 
 hadoop-common/src/site/apt (and link the new content into 
 hadoop-project/src/site/apt/index.apt.vm) and remove all forrest dependencies.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Closed] (HADOOP-7973) DistributedFileSystem close has severe consequences


 [ 
https://issues.apache.org/jira/browse/HADOOP-7973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Foley closed HADOOP-7973.
--


Closed upon release of 1.1.0.

 DistributedFileSystem close has severe consequences
 ---

 Key: HADOOP-7973
 URL: https://issues.apache.org/jira/browse/HADOOP-7973
 Project: Hadoop Common
  Issue Type: Bug
  Components: fs
Affects Versions: 1.0.0
Reporter: Daryn Sharp
Assignee: Daryn Sharp
Priority: Critical
 Attachments: HADOOP-7973-2.patch, HADOOP-7973-3.patch, 
 HADOOP-7973-4.patch, HADOOP-7973-5.patch, HADOOP-7973.patch


 The way {{FileSystem#close}} works is very problematic.  Since the 
 {{FileSystems}} are cached, any {{close}} by any caller will cause problems 
 for every other reference to it.  Will add more detail in the comments.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Closed] (HADOOP-8166) Remove JDK 1.5 dependency from building forrest docs


 [ 
https://issues.apache.org/jira/browse/HADOOP-8166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Foley closed HADOOP-8166.
--


 Remove JDK 1.5 dependency from building forrest docs
 

 Key: HADOOP-8166
 URL: https://issues.apache.org/jira/browse/HADOOP-8166
 Project: Hadoop Common
  Issue Type: Improvement
  Components: documentation
Affects Versions: 0.20.203.0, 0.20.204.0, 0.20.205.0, 1.0.0, 1.0.1
Reporter: Mark Butler
 Attachments: forrest.patch, hadoop-8166.txt


 Currently Hadoop requires both JDK 1.6 and JDK 1.5. JDK 1.5 is a requirement 
 of Forrest. It is easy to remove the latter requirement by turning off 
 forrest.validate.sitemap and forrest.validate.skins.stylesheets.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HADOOP-8409) Fix TestCommandLineJobSubmission and TestGenericOptionsParser to work for windows


 [ 
https://issues.apache.org/jira/browse/HADOOP-8409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Foley updated HADOOP-8409:
---

Target Version/s: 1-win  (was: 1.1.0)
   Fix Version/s: 1-win

 Fix TestCommandLineJobSubmission and TestGenericOptionsParser to work for 
 windows
 -

 Key: HADOOP-8409
 URL: https://issues.apache.org/jira/browse/HADOOP-8409
 Project: Hadoop Common
  Issue Type: Bug
  Components: fs, test, util
Affects Versions: 1.0.0
Reporter: Ivan Mitic
Assignee: Ivan Mitic
 Fix For: 1-win

 Attachments: HADOOP-8409-branch-1-win.2.patch, 
 HADOOP-8409-branch-1-win.3.patch, HADOOP-8409-branch-1-win.4.patch, 
 HADOOP-8409-branch-1-win.patch

   Original Estimate: 168h
  Remaining Estimate: 168h

 There are multiple places in prod and test code where Windows paths are not 
 handled properly. From a high level this could be summarized with:
 1. Windows paths are not necessarily valid DFS paths (while Unix paths are)
 2. Windows paths are not necessarily valid URIs (while Unix paths are)
 #1 causes a number of tests to fail because they implicitly assume that local 
 paths are valid DFS paths (by extracting the DFS test path from for example 
 test.build.data property)
 #2 causes issues when URIs are directly created on path strings passed in by 
 the user

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HADOOP-8414) Address problems related to localhost resolving to 127.0.0.1 on Windows


 [ 
https://issues.apache.org/jira/browse/HADOOP-8414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Foley updated HADOOP-8414:
---

Target Version/s: 1-win  (was: 1.1.0)
   Fix Version/s: 1-win

 Address problems related to localhost resolving to 127.0.0.1 on Windows
 ---

 Key: HADOOP-8414
 URL: https://issues.apache.org/jira/browse/HADOOP-8414
 Project: Hadoop Common
  Issue Type: Bug
  Components: fs, test
Affects Versions: 1.0.0
Reporter: Ivan Mitic
Assignee: Ivan Mitic
 Fix For: 1-win

 Attachments: HADOOP-8414-branch-1-win(2).patch, 
 HADOOP-8414-branch-1-win(3).patch, HADOOP-8414-branch-1-win.patch, 
 HADOOP-8414-branch-1-win.patch


 Localhost resolves to 127.0.0.1 on Windows and that causes the following 
 tests to fail:
  - TestHarFileSystem
  - TestCLI
  - TestSaslRPC
 This Jira tracks fixing these tests and other possible places that have 
 similar issue.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HADOOP-8887) Use a Maven plugin to build the native code using CMake

2012-10-17 Thread Colin Patrick McCabe (JIRA)


 [ 
https://issues.apache.org/jira/browse/HADOOP-8887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Colin Patrick McCabe updated HADOOP-8887:
-

Attachment: HADOOP-8887.006.patch

This new version should fix the javac warnings.

The javadoc warnings will be suppressed when the patch is actually checked in 
(I bumped {{OK_JAVADOC_WARNINGS}} to 14).  The source of the Javadoc warnings 
is that we're getting warnings about a few annotations which are necessary for 
Maven plugins (It warns about @goal and @phase, but Maven needs these).

 Use a Maven plugin to build the native code using CMake
 ---

 Key: HADOOP-8887
 URL: https://issues.apache.org/jira/browse/HADOOP-8887
 Project: Hadoop Common
  Issue Type: Improvement
  Components: build
Affects Versions: 2.0.3-alpha
Reporter: Colin Patrick McCabe
Assignee: Colin Patrick McCabe
Priority: Minor
 Attachments: HADOOP-8887.001.patch, HADOOP-8887.002.patch, 
 HADOOP-8887.003.patch, HADOOP-8887.004.patch, HADOOP-8887.005.patch, 
 HADOOP-8887.006.patch


 Currently, we build the native code using ant-build invocations.  Although 
 this works, it has some limitations:
 * compiler warning messages are hidden, which can cause people to check in 
 code with warnings unintentionally
 * there is no framework for running native unit tests; instead, we use ad-hoc 
 constructs involving shell scripts
 * the antrun code is very platform specific
 * there is no way to run a specific native unit test
 * it's more or less impossible for scripts like test-patch.sh to separate a 
 native test failing from the build itself failing (no files are created) or 
 to enumerate which native tests failed.
 Using a native Maven plugin would overcome these limitations.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-8779) Use tokens regardless of authentication type

2012-10-17 Thread Kan Zhang (JIRA)

[
https://issues.apache.org/jira/browse/HADOOP-8779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13478488#comment-13478488
]

Kan Zhang commented on HADOOP-8779:
---

Well, the job client scenario is just an example. The deeper issue is whether
the client should make its decision on what auth method to use based on
configuration, or on what credentials are currently available. I think the
former is better and easier to reason. If the required credentials are not
available, it should complain rather than automatically switch to make a
different type of connection (a task switching from token to SIMPLE would
defeat your testing purpose). Using credential precedence may work in simpler
scenarios (like we had before), but may not work well when more combinations of
initial and subsequent auth methods need to be supported.

Use tokens regardless of authentication type

Key: HADOOP-8779
URL: https://issues.apache.org/jira/browse/HADOOP-8779
Project: Hadoop Common
Issue Type: New Feature
Components: fs, security
Affects Versions: 3.0.0, 2.0.2-alpha
Reporter: Daryn Sharp
Assignee: Daryn Sharp

Security is a combination of authentication and authorization (tokens).
Authorization may be granted independently of the authentication model.
Tokens should be used regardless of simple or kerberos authentication.

[jira] [Updated] (HADOOP-8564) Port and extend Hadoop native libraries for Windows to address datanode concurrent reading and writing issue

[
https://issues.apache.org/jira/browse/HADOOP-8564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Suresh Srinivas updated HADOOP-8564:

Attachment: HADOOP-8564-branch-1-win.patch

Attaching the patch with indentation changed to spaces and CRLF changed LF.

Port and extend Hadoop native libraries for Windows to address datanode
concurrent reading and writing issue

[jira] [Resolved] (HADOOP-8564) Port and extend Hadoop native libraries for Windows to address datanode concurrent reading and writing issue

[
https://issues.apache.org/jira/browse/HADOOP-8564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Suresh Srinivas resolved HADOOP-8564.
-

Resolution: Fixed
Fix Version/s: 1-win
Hadoop Flags: Reviewed

+1 for the patch. I committed it to branch-1-win. Thank you Chuan.

Todd, please do post your comments. It will be addressed by a separate jira.

Port and extend Hadoop native libraries for Windows to address datanode
concurrent reading and writing issue

Attachments: HADOOP-8564-branch-1-win.patch,
HADOOP-8564-branch-1-win.patch

[jira] [Commented] (HADOOP-8564) Port and extend Hadoop native libraries for Windows to address datanode concurrent reading and writing issue

[
https://issues.apache.org/jira/browse/HADOOP-8564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13478538#comment-13478538
]

Suresh Srinivas commented on HADOOP-8564:
-

I reverted the commit.

Chuan the patch fails build:
{noformat}
[javac] .../src/test/org/apache/hadoop/io/nativeio/TestNativeIO.java:83:
cannot find symbol
[javac] symbol : method setOwner(java.io.File,java.lang.String,nulltype)
[javac] location: class org.apache.hadoop.fs.FileUtil
[javac] FileUtil.setOwner(testFile, username, null);
{noformat}

Is this patch dependent on any other jira?

Port and extend Hadoop native libraries for Windows to address datanode
concurrent reading and writing issue

Attachments: HADOOP-8564-branch-1-win.patch,
HADOOP-8564-branch-1-win.patch

[jira] [Commented] (HADOOP-8887) Use a Maven plugin to build the native code using CMake


[ 
https://issues.apache.org/jira/browse/HADOOP-8887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13478550#comment-13478550
 ] 

Hadoop QA commented on HADOOP-8887:
---

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12549601/HADOOP-8887.006.patch
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 3 new 
or modified test files.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:red}-1 javadoc{color}.  The javadoc tool appears to have generated 4 
warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:red}-1 core tests{color}.  The patch failed these unit tests in 
dev-support/cmake-maven-ng-plugin hadoop-common-project/hadoop-common 
hadoop-hdfs-project/hadoop-hdfs 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager:

  
org.apache.hadoop.hdfs.server.namenode.ha.TestInitializeSharedEdits
  org.apache.hadoop.hdfs.server.balancer.TestBalancer

{color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: 
https://builds.apache.org/job/PreCommit-HADOOP-Build/1642//testReport/
Console output: 
https://builds.apache.org/job/PreCommit-HADOOP-Build/1642//console

This message is automatically generated.

 Use a Maven plugin to build the native code using CMake
 ---

 Key: HADOOP-8887
 URL: https://issues.apache.org/jira/browse/HADOOP-8887
 Project: Hadoop Common
  Issue Type: Improvement
  Components: build
Affects Versions: 2.0.3-alpha
Reporter: Colin Patrick McCabe
Assignee: Colin Patrick McCabe
Priority: Minor
 Attachments: HADOOP-8887.001.patch, HADOOP-8887.002.patch, 
 HADOOP-8887.003.patch, HADOOP-8887.004.patch, HADOOP-8887.005.patch, 
 HADOOP-8887.006.patch


 Currently, we build the native code using ant-build invocations.  Although 
 this works, it has some limitations:
 * compiler warning messages are hidden, which can cause people to check in 
 code with warnings unintentionally
 * there is no framework for running native unit tests; instead, we use ad-hoc 
 constructs involving shell scripts
 * the antrun code is very platform specific
 * there is no way to run a specific native unit test
 * it's more or less impossible for scripts like test-patch.sh to separate a 
 native test failing from the build itself failing (no files are created) or 
 to enumerate which native tests failed.
 Using a native Maven plugin would overcome these limitations.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-8427) Convert Forrest docs to APT

2012-10-17 Thread Andy Isaacson (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-8427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13478568#comment-13478568
 ] 

Andy Isaacson commented on HADOOP-8427:
---

bq. Hi Andy, could we first remove the out-of-date doc and then convert it to 
APT? It would be much easier to see the changes.

I'm not sure what you're asking for, could you be more specific?

Since all the filenames and all the content is changing, this diff is going to 
be monstrous in any case.  I can post a separate here's the rm command to run 
and then a here's the new markup, but the new markup (in APT) is completely 
different from the old markup (in Forrest XML).

Pages that I'm planning to convert and fully update to 2.0:

http://hadoop.apache.org/docs/r1.0.3/file_system_shell.html
http://hadoop.apache.org/docs/r1.0.3/commands_manual.html

Pages that I'm planning to convert without editing for 2.0 content:

http://hadoop.apache.org/docs/r1.1.0/mapred_tutorial.html
http://hadoop.apache.org/docs/r1.0.3/cluster_setup.html
http://hadoop.apache.org/docs/r1.0.3/deployment_layout.html
http://hadoop.apache.org/docs/r1.0.3/native_libraries.html
http://hadoop.apache.org/docs/r1.0.3/service_level_auth.html
http://hadoop.apache.org/docs/r1.0.3/cluster_setup.html

Pages that I'm planning to delete as no longer relevant:

http://hadoop.apache.org/docs/r1.0.3/HttpAuthentication.html

I've moved mapred\_tutorial and cluster\_setup from plan to delete to plan 
to convert without editing since my last comment.  I'm not sure if 
HttpAuthentication is salvageable, would appreciate input on that.  In any 
case, it's fairly easy to resurrect a page.

In fact, I suppose the right thing is simply to leave xdocs/*.xml in place 
(unused), adding apt.vm versions as they're converted, then deleting the unused 
.xml after they are completely redundant.  I'll post a new patch to that effect.

 Convert Forrest docs to APT
 ---

 Key: HADOOP-8427
 URL: https://issues.apache.org/jira/browse/HADOOP-8427
 Project: Hadoop Common
  Issue Type: Task
  Components: documentation
Affects Versions: 2.0.0-alpha
Reporter: Eli Collins
Assignee: Andy Isaacson
  Labels: newbie
 Attachments: hadoop8427.txt


 Some of the forrest docs content in src/docs/src/documentation/content/xdocs 
 has not yet been converted to APT and moved to src/site/apt. Let's convert 
 the forrest docs that haven't been converted yet to new APT content in 
 hadoop-common/src/site/apt (and link the new content into 
 hadoop-project/src/site/apt/index.apt.vm) and remove all forrest dependencies.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HADOOP-8939) Backport HADOOP-7457: remove cn-docs from branch-1

Tsz Wo (Nicholas), SZE created HADOOP-8939:
--

 Summary: Backport HADOOP-7457: remove cn-docs from branch-1
 Key: HADOOP-8939
 URL: https://issues.apache.org/jira/browse/HADOOP-8939
 Project: Hadoop Common
  Issue Type: Improvement
  Components: documentation
Reporter: Tsz Wo (Nicholas), SZE


The cn-docs in branch-1 are also out-dated

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-8823) ant package target should not depend on cn-docs


[ 
https://issues.apache.org/jira/browse/HADOOP-8823?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13478576#comment-13478576
 ] 

Tsz Wo (Nicholas), SZE commented on HADOOP-8823:


Filed HADOOP-8939 for removing cn-docs from branch-1 since I don't have time to 
work on it.

 ant package target should not depend on cn-docs
 ---

 Key: HADOOP-8823
 URL: https://issues.apache.org/jira/browse/HADOOP-8823
 Project: Hadoop Common
  Issue Type: Improvement
  Components: build
Reporter: Tsz Wo (Nicholas), SZE
Assignee: Tsz Wo (Nicholas), SZE
 Attachments: c8823_20120918.patch


 In branch-1, the package target depends on cn-docs but the doc is already 
 outdated.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-8427) Convert Forrest docs to APT


[ 
https://issues.apache.org/jira/browse/HADOOP-8427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13478582#comment-13478582
 ] 

Tsz Wo (Nicholas), SZE commented on HADOOP-8427:


Hi Andy, I means do the convert and update in two patches/JIRAs.   Then, it 
will be easier to see what are being updated.  BTW, thanks for working on this.

 Convert Forrest docs to APT
 ---

 Key: HADOOP-8427
 URL: https://issues.apache.org/jira/browse/HADOOP-8427
 Project: Hadoop Common
  Issue Type: Task
  Components: documentation
Affects Versions: 2.0.0-alpha
Reporter: Eli Collins
Assignee: Andy Isaacson
  Labels: newbie
 Attachments: hadoop8427.txt


 Some of the forrest docs content in src/docs/src/documentation/content/xdocs 
 has not yet been converted to APT and moved to src/site/apt. Let's convert 
 the forrest docs that haven't been converted yet to new APT content in 
 hadoop-common/src/site/apt (and link the new content into 
 hadoop-project/src/site/apt/index.apt.vm) and remove all forrest dependencies.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-8874) HADOOP_HOME and -Dhadoop.home (from hadoop wrapper script) are not uniformly handled

[
https://issues.apache.org/jira/browse/HADOOP-8874?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13478598#comment-13478598
]

Ivan Mitic commented on HADOOP-8874:

Thanks John for the patch! The patch will also help troubleshoot/root cause
issues where winutils.exe cannot be found (oppose to just silently ignoring the
failure).

Two minor comments:
- Can you please recreate the patch using the {{--no-prefix}} git flag?
Otherwise, the patch does not apply automatically.
- I see a few lines above 80 chars, would be good to shrink them

Otherwise, it looks good.

HADOOP_HOME and -Dhadoop.home (from hadoop wrapper script) are not uniformly
handled

Key: HADOOP-8874
URL: https://issues.apache.org/jira/browse/HADOOP-8874
Project: Hadoop Common
Issue Type: Sub-task
Components: scripts, security
Affects Versions: 1-win
Environment: Called from external process with -D flag vs HADOOP_HOME
set.
Reporter: John Gordon
Labels: security
Fix For: 1-win

Attachments: fix_home.patch

There is a -D flag to set hadoop.home, which is specified in the hadoop
wrapper scripts. This is particularly useful if you want SxS execution of
two or more versions of hadoop (e.g. rolling upgrade). However, it isn't
honored at all. HADOOP_HOME is used in 3-4 places to find non-java hadoop
components such as schedulers, scripts, shared libraries, or with the Windows
changes -- binaries.
Ideally, these should all resolve the path in a consistent manner, and
callers shuold have a similar onus applied when trying to resolve an invalid
path to their components. This is particularly relevant to scripts or
binaries that may have security impact, as absolute path resolution is
generally safer and more stable than relative path resolution.

[jira] [Commented] (HADOOP-8427) Convert Forrest docs to APT

2012-10-17 Thread Andy Isaacson (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-8427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13478605#comment-13478605
 ] 

Andy Isaacson commented on HADOOP-8427:
---

bq. do the convert and update in two patches/JIRAs

OK, that's what I'll end up doing on this jira -- a diff creating the new 
.apt.vm files, after which the old .xml files can simply be svn rmed.

 Convert Forrest docs to APT
 ---

 Key: HADOOP-8427
 URL: https://issues.apache.org/jira/browse/HADOOP-8427
 Project: Hadoop Common
  Issue Type: Task
  Components: documentation
Affects Versions: 2.0.0-alpha
Reporter: Eli Collins
Assignee: Andy Isaacson
  Labels: newbie
 Attachments: hadoop8427-1.txt, hadoop8427.txt


 Some of the forrest docs content in src/docs/src/documentation/content/xdocs 
 has not yet been converted to APT and moved to src/site/apt. Let's convert 
 the forrest docs that haven't been converted yet to new APT content in 
 hadoop-common/src/site/apt (and link the new content into 
 hadoop-project/src/site/apt/index.apt.vm) and remove all forrest dependencies.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HADOOP-8427) Convert Forrest docs to APT

2012-10-17 Thread Andy Isaacson (JIRA)


 [ 
https://issues.apache.org/jira/browse/HADOOP-8427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andy Isaacson updated HADOOP-8427:
--

Attachment: hadoop8427-1.txt

 Convert Forrest docs to APT
 ---

 Key: HADOOP-8427
 URL: https://issues.apache.org/jira/browse/HADOOP-8427
 Project: Hadoop Common
  Issue Type: Task
  Components: documentation
Affects Versions: 2.0.0-alpha
Reporter: Eli Collins
Assignee: Andy Isaacson
  Labels: newbie
 Attachments: hadoop8427-1.txt, hadoop8427.txt


 Some of the forrest docs content in src/docs/src/documentation/content/xdocs 
 has not yet been converted to APT and moved to src/site/apt. Let's convert 
 the forrest docs that haven't been converted yet to new APT content in 
 hadoop-common/src/site/apt (and link the new content into 
 hadoop-project/src/site/apt/index.apt.vm) and remove all forrest dependencies.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-8564) Port and extend Hadoop native libraries for Windows to address datanode concurrent reading and writing issue

2012-10-17 Thread Chuan Liu (JIRA)

[
https://issues.apache.org/jira/browse/HADOOP-8564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13478620#comment-13478620
]

Chuan Liu commented on HADOOP-8564:
---

Let me try it on my machine.

Port and extend Hadoop native libraries for Windows to address datanode
concurrent reading and writing issue

Attachments: HADOOP-8564-branch-1-win.patch,
HADOOP-8564-branch-1-win.patch

[jira] [Updated] (HADOOP-8823) ant package target should not depend on cn-docs


 [ 
https://issues.apache.org/jira/browse/HADOOP-8823?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Suresh Srinivas updated HADOOP-8823:


Affects Version/s: 1.0.0

 ant package target should not depend on cn-docs
 ---

 Key: HADOOP-8823
 URL: https://issues.apache.org/jira/browse/HADOOP-8823
 Project: Hadoop Common
  Issue Type: Improvement
  Components: build
Affects Versions: 1.0.0
Reporter: Tsz Wo (Nicholas), SZE
Assignee: Tsz Wo (Nicholas), SZE
 Attachments: c8823_20120918.patch


 In branch-1, the package target depends on cn-docs but the doc is already 
 outdated.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-8823) ant package target should not depend on cn-docs


[ 
https://issues.apache.org/jira/browse/HADOOP-8823?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13478627#comment-13478627
 ] 

Suresh Srinivas commented on HADOOP-8823:
-

+1 for the patch.

 ant package target should not depend on cn-docs
 ---

 Key: HADOOP-8823
 URL: https://issues.apache.org/jira/browse/HADOOP-8823
 Project: Hadoop Common
  Issue Type: Improvement
  Components: build
Affects Versions: 1.0.0
Reporter: Tsz Wo (Nicholas), SZE
Assignee: Tsz Wo (Nicholas), SZE
 Attachments: c8823_20120918.patch


 In branch-1, the package target depends on cn-docs but the doc is already 
 outdated.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Assigned] (HADOOP-8904) Hadoop does not close output file / does not call Mapper.cleanup if exception in map


 [ 
https://issues.apache.org/jira/browse/HADOOP-8904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Suresh Srinivas reassigned HADOOP-8904:
---

Assignee: Daniel Dai

  Hadoop does not close output file / does not call Mapper.cleanup if 
 exception in map
 -

 Key: HADOOP-8904
 URL: https://issues.apache.org/jira/browse/HADOOP-8904
 Project: Hadoop Common
  Issue Type: Bug
Affects Versions: 1-win
Reporter: Daniel Dai
Assignee: Daniel Dai
 Attachments: HADOOP-8904-1.patch


 Find this in Pig unit test TestStore under Windows. There are dangling files 
 because map does not close the file when exception happens in map(). In 
 Windows, Hadoop will not remove a file if it is not closed. This happens in 
 reduce() as well.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-8904) Hadoop does not close output file / does not call Mapper.cleanup if exception in map


[ 
https://issues.apache.org/jira/browse/HADOOP-8904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13478648#comment-13478648
 ] 

Suresh Srinivas commented on HADOOP-8904:
-

Daniel, can you please also change the indentation in this patch to two spaces.

  Hadoop does not close output file / does not call Mapper.cleanup if 
 exception in map
 -

 Key: HADOOP-8904
 URL: https://issues.apache.org/jira/browse/HADOOP-8904
 Project: Hadoop Common
  Issue Type: Bug
Affects Versions: 1-win
Reporter: Daniel Dai
 Attachments: HADOOP-8904-1.patch


 Find this in Pig unit test TestStore under Windows. There are dangling files 
 because map does not close the file when exception happens in map(). In 
 Windows, Hadoop will not remove a file if it is not closed. This happens in 
 reduce() as well.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-8899) Classpath exceeds maximum OS limit

[
https://issues.apache.org/jira/browse/HADOOP-8899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13478656#comment-13478656
]

Suresh Srinivas commented on HADOOP-8899:
-

Alejandro, this is manifest only jar. See similar things done in maven surefire
-
http://maven.apache.org/plugins/maven-surefire-plugin/examples/class-loading.html.

bq. then things will not work as expected as you'll lose some of those files in
the consolidation
I do not understand given above comment. All you have done is create a jar with
manifest file that has classpath. What do mean by things will not work as
expected?

Classpath exceeds maximum OS limit
--

[jira] [Commented] (HADOOP-8564) Port and extend Hadoop native libraries for Windows to address datanode concurrent reading and writing issue

2012-10-17 Thread Chuan Liu (JIRA)

[
https://issues.apache.org/jira/browse/HADOOP-8564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13478658#comment-13478658
]

Chuan Liu commented on HADOOP-8564:
---

Suresh, sorry for the break.

This indeed depends on another JIRA HADOOP-8763.

FileUtil.setOwner() method was added in that JIRA.

Port and extend Hadoop native libraries for Windows to address datanode
concurrent reading and writing issue

Attachments: HADOOP-8564-branch-1-win.patch,
HADOOP-8564-branch-1-win.patch

[jira] [Commented] (HADOOP-8899) Classpath exceeds maximum OS limit