[jira] [Commented] (HADOOP-11518) Prefix-based per-RPC server configuration

2015-02-02 Thread Hadoop QA (JIRA)

[ 
https://issues.apache.org/jira/browse/HADOOP-11518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14302034#comment-14302034
 ] 

Hadoop QA commented on HADOOP-11518:


{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12695968/HADOOP-11518.v2.patch
  against trunk revision 3472e3b.

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 1 new 
or modified test files.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  There were no new javadoc warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 2.0.3) warnings.

{color:red}-1 release audit{color}.  The applied patch generated 1 
release audit warnings.

{color:green}+1 core tests{color}.  The patch passed unit tests in 
hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs.

Test results: 
https://builds.apache.org/job/PreCommit-HADOOP-Build/5557//testReport/
Release audit warnings: 
https://builds.apache.org/job/PreCommit-HADOOP-Build/5557//artifact/patchprocess/patchReleaseAuditProblems.txt
Console output: 
https://builds.apache.org/job/PreCommit-HADOOP-Build/5557//console

This message is automatically generated.

 Prefix-based per-RPC server configuration
 -

 Key: HADOOP-11518
 URL: https://issues.apache.org/jira/browse/HADOOP-11518
 Project: Hadoop Common
  Issue Type: Improvement
  Components: ipc
Reporter: Kihwal Lee
Assignee: Kihwal Lee
 Attachments: HADOOP-11518.patch, HADOOP-11518.v2.patch


 If a process runs multiple RPC servers, there is no way to individually set 
 their IPC parameters beyond what's supported by the RPC Builder.  If a 
 configuration is shared among many nodes, this is more problematic. 
 In this jira, a prefix-based per-server configuration is proposed.  When an 
 RPC server is built, a prefix can be specified, which will then be prepended 
 to keys when retrieving configured values.  There will be an option to 
 fall-back to using the key without the prefix when no value is found with the 
 prefix, thus making this feature compatible with the current way of 
 configuring servers.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HADOOP-10948) SwiftNativeFileSystem's directory is incompatible with Swift and Horizon

2015-02-02 Thread David Dobbins (JIRA)

[ 
https://issues.apache.org/jira/browse/HADOOP-10948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14302051#comment-14302051
 ] 

David Dobbins commented on HADOOP-10948:


Is there an avenue for maintaining backwards compatibility? It seems the only 
risk is for directories already created in swift with SwiftNativeFileSystem.  
Would it suffice to only create new directories using the trailing-slash model 
but to recognize either model for the purposes of reading existing directories?

 SwiftNativeFileSystem's directory is incompatible with Swift and Horizon
 

 Key: HADOOP-10948
 URL: https://issues.apache.org/jira/browse/HADOOP-10948
 Project: Hadoop Common
  Issue Type: Bug
  Components: fs/swift
Affects Versions: 3.0.0
Reporter: Kazuki OIKAWA
Assignee: Kazuki OIKAWA
 Attachments: HADOOP-10948-2.patch, HADOOP-10948.patch


 SwiftNativeFileSystem's directory representation is zero-byte file.
 But in Swift / Horizon, directory representation is a trailing-slash.
 This incompatibility has the following issues.
 * SwiftNativeFileSystem can't see pseudo-directory made by OpenStack Horizon
 * Swift/Horizon can't see pseudo-directory made by SwiftNativeFileSystem. But 
 Swift/Horizon see a zero-byte file instead of that pseudo-directory.
 * SwiftNativeFileSystem can't see a file if there is no intermediate 
 pseudo-directory object.
 * SwiftNativeFileSystem makes two objects when making a single directory
 (e.g. hadoop fs -mkdir swift://test.test/dir/ = dir and dir/ created)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HADOOP-11442) hadoop-azure: Create test jar

2015-02-02 Thread Hudson (JIRA)

[ 
https://issues.apache.org/jira/browse/HADOOP-11442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14301821#comment-14301821
 ] 

Hudson commented on HADOOP-11442:
-

SUCCESS: Integrated in Hadoop-trunk-Commit #6980 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/6980/])
HADOOP-11442. hadoop-azure: Create test jar. Contributed by Shashank 
Khandelwal. (cnauroth: rev 1c09ca2ba4d97d827cc6e86ef6bd2930bc77b5ae)
* hadoop-tools/hadoop-azure/pom.xml
* hadoop-common-project/hadoop-common/CHANGES.txt


 hadoop-azure: Create test jar
 -

 Key: HADOOP-11442
 URL: https://issues.apache.org/jira/browse/HADOOP-11442
 Project: Hadoop Common
  Issue Type: Improvement
  Components: tools
 Environment: windows, azure
Reporter: shashank
Assignee: Chris Nauroth
 Fix For: 2.7.0

 Attachments: HADOOP-11442.patch, HADOOP-11442v1.patch, 
 HADOOP-11442v2.patch


 pom of hadoop-azure project to needs to be modified to create a test jar as 
 well. This test jar is required to run test cases of Windowsazuretablesink 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HADOOP-11518) Prefix-based per-RPC server configuration

2015-02-02 Thread Kihwal Lee (JIRA)

[ 
https://issues.apache.org/jira/browse/HADOOP-11518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14302068#comment-14302068
 ] 

Kihwal Lee commented on HADOOP-11518:
-

The release audit is fixed in YARN-3113.

 Prefix-based per-RPC server configuration
 -

 Key: HADOOP-11518
 URL: https://issues.apache.org/jira/browse/HADOOP-11518
 Project: Hadoop Common
  Issue Type: Improvement
  Components: ipc
Reporter: Kihwal Lee
Assignee: Kihwal Lee
 Attachments: HADOOP-11518.patch, HADOOP-11518.v2.patch


 If a process runs multiple RPC servers, there is no way to individually set 
 their IPC parameters beyond what's supported by the RPC Builder.  If a 
 configuration is shared among many nodes, this is more problematic. 
 In this jira, a prefix-based per-server configuration is proposed.  When an 
 RPC server is built, a prefix can be specified, which will then be prepended 
 to keys when retrieving configured values.  There will be an option to 
 fall-back to using the key without the prefix when no value is found with the 
 prefix, thus making this feature compatible with the current way of 
 configuring servers.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HADOOP-11506) Configuration.get() is unnecessarily slow

2015-02-02 Thread Hadoop QA (JIRA)

[ 
https://issues.apache.org/jira/browse/HADOOP-11506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14302026#comment-14302026
 ] 

Hadoop QA commented on HADOOP-11506:


{color:green}+1 overall{color}.  Here are the results of testing the latest 
attachment 
  
http://issues.apache.org/jira/secure/attachment/12696002/HADOOP-11506.004.patch
  against trunk revision 5f9a0dd.

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 1 new 
or modified test files.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  There were no new javadoc warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 2.0.3) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 core tests{color}.  The patch passed unit tests in 
hadoop-common-project/hadoop-common.

Test results: 
https://builds.apache.org/job/PreCommit-HADOOP-Build/5559//testReport/
Console output: 
https://builds.apache.org/job/PreCommit-HADOOP-Build/5559//console

This message is automatically generated.

 Configuration.get() is unnecessarily slow
 -

 Key: HADOOP-11506
 URL: https://issues.apache.org/jira/browse/HADOOP-11506
 Project: Hadoop Common
  Issue Type: Bug
Reporter: Dmitriy V. Ryaboy
Assignee: Gera Shegalov
 Attachments: HADOOP-11506.001.patch, HADOOP-11506.002.patch, 
 HADOOP-11506.003.patch, HADOOP-11506.004.patch


 Profiling several large Hadoop jobs, we discovered that a surprising amount 
 of time was spent inside Configuration.get, more specifically, in regex 
 matching caused by the substituteVars call.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Assigned] (HADOOP-10867) NPE in heartbeat when the configured topology script doesn't exist

2015-02-02 Thread Mauro Murari (JIRA)

 [ 
https://issues.apache.org/jira/browse/HADOOP-10867?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mauro Murari reassigned HADOOP-10867:
-

Assignee: Mauro Murari  (was: Ivan Mitic)

 NPE in heartbeat when the configured topology script doesn't exist
 --

 Key: HADOOP-10867
 URL: https://issues.apache.org/jira/browse/HADOOP-10867
 Project: Hadoop Common
  Issue Type: Bug
Affects Versions: 1.0.3
Reporter: Vinod Kumar Vavilapalli
Assignee: Mauro Murari
  Labels: newbie





--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HADOOP-10181) GangliaContext does not work with multicast ganglia setup

2015-02-02 Thread Andrew Johnson (JIRA)

[ 
https://issues.apache.org/jira/browse/HADOOP-10181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14301995#comment-14301995
 ] 

Andrew Johnson commented on HADOOP-10181:
-

Great, thanks!

 GangliaContext does not work with multicast ganglia setup
 -

 Key: HADOOP-10181
 URL: https://issues.apache.org/jira/browse/HADOOP-10181
 Project: Hadoop Common
  Issue Type: Bug
  Components: metrics
Affects Versions: 2.6.0
Reporter: Andrew Otto
Assignee: Andrew Johnson
Priority: Minor
  Labels: ganglia, hadoop, metrics, multicast
 Fix For: 2.7.0

 Attachments: HADOOP-10181.001.patch, HADOOP-10181.002.patch, 
 HADOOP-10181.003.patch


 The GangliaContext class which is used to send Hadoop metrics to Ganglia uses 
 a DatagramSocket to send these metrics.  This works fine for Ganglia 
 multicast setups that are all on the same VLAN.  However, when working with 
 multiple VLANs, a packet sent via DatagramSocket to a multicast address will 
 end up with a TTL of 1.  Multicast TTL indicates the number of network hops 
 for which a particular multicast packet is valid.  The packets sent by 
 GangliaContext do not make it to ganglia aggregrators on the same multicast 
 group, but in different VLANs.
 To fix, we'd need a configuration property that specifies that multicast is 
 to be used, and another that allows setting of the multicast packet TTL.  
 With these set, we could then use MulticastSocket setTimeToLive() instead of 
 just plain ol' DatagramSocket.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HADOOP-11104) org.apache.hadoop.metrics2.lib.MetricsRegistry needs numerical parameter checking

2015-02-02 Thread Ray Chiang (JIRA)

[ 
https://issues.apache.org/jira/browse/HADOOP-11104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14301769#comment-14301769
 ] 

Ray Chiang commented on HADOOP-11104:
-

RE: Release audit

Sorting icons.psd not part of the patch

 org.apache.hadoop.metrics2.lib.MetricsRegistry needs numerical parameter 
 checking
 -

 Key: HADOOP-11104
 URL: https://issues.apache.org/jira/browse/HADOOP-11104
 Project: Hadoop Common
  Issue Type: Bug
Reporter: Ray Chiang
Assignee: Ray Chiang
Priority: Minor
  Labels: newbie
 Attachments: HADOOP-11104.001.patch


 Passing a negative value to the interval field of 
 MetricsRegistry#newQuantiles should throw a MetricsException with a clear 
 error message.  The current stack trace looks something like:
 java.lang.IllegalArgumentException: null
 at 
 java.util.concurrent.ScheduledThreadPoolExecutor.scheduleAtFixedRate(ScheduledThreadPoolExecutor.java:420)
 at 
 org.apache.hadoop.metrics2.lib.MutableQuantiles.init(MutableQuantiles.java:107)
 at 
 org.apache.hadoop.metrics2.lib.MetricsRegistry.newQuantiles(MetricsRegistry.java:200)
 Along similar lines, should the other methods like 
 MetricsRegistry#newCounter() also have parameter checking for negative 
 int/long values?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HADOOP-11525) FileSystem should expose some performance characteristics for caller (e.g., FsShell) to choose the right algorithm.

2015-02-02 Thread Steve Loughran (JIRA)

[ 
https://issues.apache.org/jira/browse/HADOOP-11525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14301861#comment-14301861
 ] 

Steve Loughran commented on HADOOP-11525:
-

Mark it as included; I did lift your patch to the FS client to make it the 
first real use case.

Now, one thing that could be good would to be to make some publishing of the 
semantics something that every FS does -but then, life gets complex fast. For 
the HADOOP-9361 contract tests I did make every FS publish their rules, [such 
as for 
localfs|https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/test/resources/contract/localfs.xml]
 ... but we kept them in test/resources as they were intended to be only stuff 
that tests should care about when measuring how different the other 
implementations were from HDFS, using the flags to verify that their different 
behavior was at least as expected. I'd hate to have code checks everywhere that 
looks for filesystem quirks. Either code for HDFS or target the lowest common 
denominator of an object store with write-on-close and minimal consistency 
guarantees. And for that, we need to identify those critical code paths that 
assume HDFS but end up with the latter

 FileSystem should expose some performance characteristics for caller (e.g., 
 FsShell) to choose the right algorithm.
 ---

 Key: HADOOP-11525
 URL: https://issues.apache.org/jira/browse/HADOOP-11525
 Project: Hadoop Common
  Issue Type: Improvement
  Components: tools
Affects Versions: 2.6.0
Reporter: Lei (Eddy) Xu
Assignee: Lei (Eddy) Xu
 Attachments: HADOOP-11525.000.patch


 When running {{hadoop fs -put}},  {{FsShell}} creates a {{._COPYING_.}} file 
 on the target directory, and then renames it to target file when the write is 
 done. However, for some targeted systems, such as S3, Azure and Swift, a 
 partial failure write request (i.e., {{PUT}}) has not side effect, while the 
 {{rename}} operation is expensive. 
 {{FileSystem}} should expose some characteristics so that the operation such 
 as {{CommandWithDestination#copyStreamToTarget()}} can detect and choose the 
 right way to do.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (HADOOP-11506) Configuration.get() is unnecessarily slow

2015-02-02 Thread Gera Shegalov (JIRA)

 [ 
https://issues.apache.org/jira/browse/HADOOP-11506?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gera Shegalov updated HADOOP-11506:
---
Attachment: HADOOP-11506.004.patch

Minor fix: I had an unintended HashSet size specification:
{code}
diff --git 
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
 
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
index 9380d68..c47e771 100644
--- 
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
+++ 
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
@@ -963,7 +963,7 @@ private String substituteVars(String expr) {
   final int afterRightBrace = varBounds[SUB_END_IDX] + }.length();
   final String refVar = eval.substring(dollar, afterRightBrace);
   if (evalSet == null) {
-evalSet = new HashSetString(1);
+evalSet = new HashSetString();
   }
   if (!evalSet.add(refVar)) {
 return expr; // return original expression if there is a loop
{code}

 Configuration.get() is unnecessarily slow
 -

 Key: HADOOP-11506
 URL: https://issues.apache.org/jira/browse/HADOOP-11506
 Project: Hadoop Common
  Issue Type: Bug
Reporter: Dmitriy V. Ryaboy
Assignee: Gera Shegalov
 Attachments: HADOOP-11506.001.patch, HADOOP-11506.002.patch, 
 HADOOP-11506.003.patch, HADOOP-11506.004.patch


 Profiling several large Hadoop jobs, we discovered that a surprising amount 
 of time was spent inside Configuration.get, more specifically, in regex 
 matching caused by the substituteVars call.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (HADOOP-5454) SortedMapWritable: readFields() will not clear values before deserialization

2015-02-02 Thread Mauro Murari (JIRA)

 [ 
https://issues.apache.org/jira/browse/HADOOP-5454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mauro Murari updated HADOOP-5454:
-
Assignee: (was: Mauro Murari)

 SortedMapWritable: readFields() will not clear values before deserialization
 

 Key: HADOOP-5454
 URL: https://issues.apache.org/jira/browse/HADOOP-5454
 Project: Hadoop Common
  Issue Type: Bug
Affects Versions: 0.19.1
Reporter: Stefan Podkowinski

 In case SortedMapWritable is used as value in a reducer, the user must 
 explicitly call clear() on the map between iterating values. This is because 
 SortedMapWritable will be reused once instantiated, but consecutive calls to 
 readFields() will not reset the maps internal state, as e.g. done by 
 MapWritable. Please add this.instance.clear(); on top of readFields().
 You may also want to consider HADOOP-5028 for fixing another issue with this 
 class.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (HADOOP-11529) Fix findbugs warnings in hadoop-archives

2015-02-02 Thread Masatake Iwasaki (JIRA)

 [ 
https://issues.apache.org/jira/browse/HADOOP-11529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Masatake Iwasaki updated HADOOP-11529:
--
Status: Patch Available  (was: Open)

 Fix findbugs warnings in hadoop-archives
 

 Key: HADOOP-11529
 URL: https://issues.apache.org/jira/browse/HADOOP-11529
 Project: Hadoop Common
  Issue Type: Bug
  Components: tools
Reporter: Masatake Iwasaki
Assignee: Masatake Iwasaki
Priority: Minor
 Attachments: HADOOP-11529.001.patch


 {code}
 NPPossible null pointer dereference of reader in 
 org.apache.hadoop.tools.HadoopArchives$HArchiveInputFormat.getSplits(JobConf, 
 int) on exception path
 DmFound reliance on default encoding in 
 org.apache.hadoop.tools.HadoopArchives$HArchivesReducer.close(): 
 String.getBytes()
 DmFound reliance on default encoding in 
 org.apache.hadoop.tools.HadoopArchives$HArchivesReducer.configure(JobConf): 
 String.getBytes()
 DmFound reliance on default encoding in 
 org.apache.hadoop.tools.HadoopArchives$HArchivesReducer.reduce(IntWritable, 
 Iterator, OutputCollector, Reporter): String.getBytes()
 {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Assigned] (HADOOP-3602) DataInputBuffer::getLength returns the end position, not the length

2015-02-02 Thread Mauro Murari (JIRA)

 [ 
https://issues.apache.org/jira/browse/HADOOP-3602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mauro Murari reassigned HADOOP-3602:


Assignee: Mauro Murari

 DataInputBuffer::getLength returns the end position, not the length
 ---

 Key: HADOOP-3602
 URL: https://issues.apache.org/jira/browse/HADOOP-3602
 Project: Hadoop Common
  Issue Type: Bug
Reporter: Chris Douglas
Assignee: Mauro Murari

 In DataInputBuffer, a call to reset(byte[] buffer, offset, len) resets the 
 data read by the buffer. However, when the offset is not zero, getLength 
 returns the index marking the end of valid data in the buffer and not the 
 length as specified in the last call to reset. The consequence is that 
 consumers of DataInputBuffers may inaccurately gauge the length of the input 
 given them.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Assigned] (HADOOP-10646) KeyProvider buildVersionName method should be moved to a utils class

2015-02-02 Thread Mauro Murari (JIRA)

 [ 
https://issues.apache.org/jira/browse/HADOOP-10646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mauro Murari reassigned HADOOP-10646:
-

Assignee: Mauro Murari  (was: Alejandro Abdelnur)

 KeyProvider buildVersionName method should be moved to a utils class
 

 Key: HADOOP-10646
 URL: https://issues.apache.org/jira/browse/HADOOP-10646
 Project: Hadoop Common
  Issue Type: Bug
  Components: security
Affects Versions: 3.0.0
Reporter: Alejandro Abdelnur
Assignee: Mauro Murari

 The buildVersionName() method should not be part of the KeyProvider public 
 API because keyversions could be opaque (not built based on the keyname and 
 key generation counter).
 KeyProvider implementations may choose to use buildVersionName() for reasons 
 such as described in HADOOP-10611.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HADOOP-11529) Fix findbugs warnings in hadoop-archives

2015-02-02 Thread Hadoop QA (JIRA)

[ 
https://issues.apache.org/jira/browse/HADOOP-11529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14302556#comment-14302556
 ] 

Hadoop QA commented on HADOOP-11529:


{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  
http://issues.apache.org/jira/secure/attachment/12696049/HADOOP-11529.002.patch
  against trunk revision 8acc5e9.

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:red}-1 tests included{color}.  The patch doesn't appear to include 
any new or modified tests.
Please justify why no new tests are needed for this 
patch.
Also please list what manual steps were performed to 
verify this patch.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  There were no new javadoc warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 2.0.3) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 core tests{color}.  The patch passed unit tests in 
hadoop-tools/hadoop-archives.

Test results: 
https://builds.apache.org/job/PreCommit-HADOOP-Build/5561//testReport/
Console output: 
https://builds.apache.org/job/PreCommit-HADOOP-Build/5561//console

This message is automatically generated.

 Fix findbugs warnings in hadoop-archives
 

 Key: HADOOP-11529
 URL: https://issues.apache.org/jira/browse/HADOOP-11529
 Project: Hadoop Common
  Issue Type: Bug
  Components: tools
Reporter: Masatake Iwasaki
Assignee: Masatake Iwasaki
Priority: Minor
 Attachments: HADOOP-11529.001.patch, HADOOP-11529.002.patch


 {code}
 NPPossible null pointer dereference of reader in 
 org.apache.hadoop.tools.HadoopArchives$HArchiveInputFormat.getSplits(JobConf, 
 int) on exception path
 DmFound reliance on default encoding in 
 org.apache.hadoop.tools.HadoopArchives$HArchivesReducer.close(): 
 String.getBytes()
 DmFound reliance on default encoding in 
 org.apache.hadoop.tools.HadoopArchives$HArchivesReducer.configure(JobConf): 
 String.getBytes()
 DmFound reliance on default encoding in 
 org.apache.hadoop.tools.HadoopArchives$HArchivesReducer.reduce(IntWritable, 
 Iterator, OutputCollector, Reporter): String.getBytes()
 {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HADOOP-11162) Unclosed InputStream in ApplicationClassLoader

2015-02-02 Thread Mauro Murari (JIRA)

[ 
https://issues.apache.org/jira/browse/HADOOP-11162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14302561#comment-14302561
 ] 

Mauro Murari commented on HADOOP-11162:
---

It was closed, please see this commit 
https://github.com/apache/hadoop/commit/825923f7b9a8771ff5b3f08646e96304d1acc367



 Unclosed InputStream in ApplicationClassLoader
 --

 Key: HADOOP-11162
 URL: https://issues.apache.org/jira/browse/HADOOP-11162
 Project: Hadoop Common
  Issue Type: Bug
Reporter: Ted Yu
Priority: Minor

 {code}
   static {
 InputStream is = null;
 {code}
 The above InputStream is not closed upon leaving the static block.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (HADOOP-11529) Fix findbugs warnings in hadoop-archives

2015-02-02 Thread Masatake Iwasaki (JIRA)

 [ 
https://issues.apache.org/jira/browse/HADOOP-11529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Masatake Iwasaki updated HADOOP-11529:
--
Attachment: HADOOP-11529.001.patch

attaching patch.

 Fix findbugs warnings in hadoop-archives
 

 Key: HADOOP-11529
 URL: https://issues.apache.org/jira/browse/HADOOP-11529
 Project: Hadoop Common
  Issue Type: Bug
  Components: tools
Reporter: Masatake Iwasaki
Assignee: Masatake Iwasaki
Priority: Minor
 Attachments: HADOOP-11529.001.patch


 {code}
 NPPossible null pointer dereference of reader in 
 org.apache.hadoop.tools.HadoopArchives$HArchiveInputFormat.getSplits(JobConf, 
 int) on exception path
 DmFound reliance on default encoding in 
 org.apache.hadoop.tools.HadoopArchives$HArchivesReducer.close(): 
 String.getBytes()
 DmFound reliance on default encoding in 
 org.apache.hadoop.tools.HadoopArchives$HArchivesReducer.configure(JobConf): 
 String.getBytes()
 DmFound reliance on default encoding in 
 org.apache.hadoop.tools.HadoopArchives$HArchivesReducer.reduce(IntWritable, 
 Iterator, OutputCollector, Reporter): String.getBytes()
 {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (HADOOP-11529) Fix findbugs warnings in hadoop-archives

2015-02-02 Thread Masatake Iwasaki (JIRA)

 [ 
https://issues.apache.org/jira/browse/HADOOP-11529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Masatake Iwasaki updated HADOOP-11529:
--
Attachment: HADOOP-11529.002.patch

Thanks for the comment [~wheat9]!  I updated the patch.

 Fix findbugs warnings in hadoop-archives
 

 Key: HADOOP-11529
 URL: https://issues.apache.org/jira/browse/HADOOP-11529
 Project: Hadoop Common
  Issue Type: Bug
  Components: tools
Reporter: Masatake Iwasaki
Assignee: Masatake Iwasaki
Priority: Minor
 Attachments: HADOOP-11529.001.patch, HADOOP-11529.002.patch


 {code}
 NPPossible null pointer dereference of reader in 
 org.apache.hadoop.tools.HadoopArchives$HArchiveInputFormat.getSplits(JobConf, 
 int) on exception path
 DmFound reliance on default encoding in 
 org.apache.hadoop.tools.HadoopArchives$HArchivesReducer.close(): 
 String.getBytes()
 DmFound reliance on default encoding in 
 org.apache.hadoop.tools.HadoopArchives$HArchivesReducer.configure(JobConf): 
 String.getBytes()
 DmFound reliance on default encoding in 
 org.apache.hadoop.tools.HadoopArchives$HArchivesReducer.reduce(IntWritable, 
 Iterator, OutputCollector, Reporter): String.getBytes()
 {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (HADOOP-10646) KeyProvider buildVersionName method should be moved to a utils class

2015-02-02 Thread Mauro Murari (JIRA)

 [ 
https://issues.apache.org/jira/browse/HADOOP-10646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mauro Murari updated HADOOP-10646:
--
Assignee: (was: Mauro Murari)

 KeyProvider buildVersionName method should be moved to a utils class
 

 Key: HADOOP-10646
 URL: https://issues.apache.org/jira/browse/HADOOP-10646
 Project: Hadoop Common
  Issue Type: Bug
  Components: security
Affects Versions: 3.0.0
Reporter: Alejandro Abdelnur

 The buildVersionName() method should not be part of the KeyProvider public 
 API because keyversions could be opaque (not built based on the keyname and 
 key generation counter).
 KeyProvider implementations may choose to use buildVersionName() for reasons 
 such as described in HADOOP-10611.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (HADOOP-3602) DataInputBuffer::getLength returns the end position, not the length

2015-02-02 Thread Mauro Murari (JIRA)

 [ 
https://issues.apache.org/jira/browse/HADOOP-3602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mauro Murari updated HADOOP-3602:
-
Assignee: (was: Mauro Murari)

 DataInputBuffer::getLength returns the end position, not the length
 ---

 Key: HADOOP-3602
 URL: https://issues.apache.org/jira/browse/HADOOP-3602
 Project: Hadoop Common
  Issue Type: Bug
Reporter: Chris Douglas

 In DataInputBuffer, a call to reset(byte[] buffer, offset, len) resets the 
 data read by the buffer. However, when the offset is not zero, getLength 
 returns the index marking the end of valid data in the buffer and not the 
 length as specified in the last call to reset. The consequence is that 
 consumers of DataInputBuffers may inaccurately gauge the length of the input 
 given them.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HADOOP-11529) Fix findbugs warnings in hadoop-archives

2015-02-02 Thread Haohui Mai (JIRA)

[ 
https://issues.apache.org/jira/browse/HADOOP-11529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14302325#comment-14302325
 ] 

Haohui Mai commented on HADOOP-11529:
-

LGTM. Minor nit:

{code}
-reader.close();
+if (reader != null) {
+  reader.close();
+}
{code}

You can use Java 7's try-with-resource statement here.

 Fix findbugs warnings in hadoop-archives
 

 Key: HADOOP-11529
 URL: https://issues.apache.org/jira/browse/HADOOP-11529
 Project: Hadoop Common
  Issue Type: Bug
  Components: tools
Reporter: Masatake Iwasaki
Assignee: Masatake Iwasaki
Priority: Minor
 Attachments: HADOOP-11529.001.patch


 {code}
 NPPossible null pointer dereference of reader in 
 org.apache.hadoop.tools.HadoopArchives$HArchiveInputFormat.getSplits(JobConf, 
 int) on exception path
 DmFound reliance on default encoding in 
 org.apache.hadoop.tools.HadoopArchives$HArchivesReducer.close(): 
 String.getBytes()
 DmFound reliance on default encoding in 
 org.apache.hadoop.tools.HadoopArchives$HArchivesReducer.configure(JobConf): 
 String.getBytes()
 DmFound reliance on default encoding in 
 org.apache.hadoop.tools.HadoopArchives$HArchivesReducer.reduce(IntWritable, 
 Iterator, OutputCollector, Reporter): String.getBytes()
 {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HADOOP-11529) Fix findbugs warnings in hadoop-archives

2015-02-02 Thread Haohui Mai (JIRA)

[ 
https://issues.apache.org/jira/browse/HADOOP-11529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14302493#comment-14302493
 ] 

Haohui Mai commented on HADOOP-11529:
-

+1 pending jenkins.

 Fix findbugs warnings in hadoop-archives
 

 Key: HADOOP-11529
 URL: https://issues.apache.org/jira/browse/HADOOP-11529
 Project: Hadoop Common
  Issue Type: Bug
  Components: tools
Reporter: Masatake Iwasaki
Assignee: Masatake Iwasaki
Priority: Minor
 Attachments: HADOOP-11529.001.patch, HADOOP-11529.002.patch


 {code}
 NPPossible null pointer dereference of reader in 
 org.apache.hadoop.tools.HadoopArchives$HArchiveInputFormat.getSplits(JobConf, 
 int) on exception path
 DmFound reliance on default encoding in 
 org.apache.hadoop.tools.HadoopArchives$HArchivesReducer.close(): 
 String.getBytes()
 DmFound reliance on default encoding in 
 org.apache.hadoop.tools.HadoopArchives$HArchivesReducer.configure(JobConf): 
 String.getBytes()
 DmFound reliance on default encoding in 
 org.apache.hadoop.tools.HadoopArchives$HArchivesReducer.reduce(IntWritable, 
 Iterator, OutputCollector, Reporter): String.getBytes()
 {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HADOOP-11104) org.apache.hadoop.metrics2.lib.MetricsRegistry needs numerical parameter checking

2015-02-02 Thread Ray Chiang (JIRA)

[ 
https://issues.apache.org/jira/browse/HADOOP-11104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14302593#comment-14302593
 ] 

Ray Chiang commented on HADOOP-11104:
-

It looks like the release audit problem is fixed in YARN-3113.

 org.apache.hadoop.metrics2.lib.MetricsRegistry needs numerical parameter 
 checking
 -

 Key: HADOOP-11104
 URL: https://issues.apache.org/jira/browse/HADOOP-11104
 Project: Hadoop Common
  Issue Type: Bug
Reporter: Ray Chiang
Assignee: Ray Chiang
Priority: Minor
  Labels: newbie
 Attachments: HADOOP-11104.001.patch


 Passing a negative value to the interval field of 
 MetricsRegistry#newQuantiles should throw a MetricsException with a clear 
 error message.  The current stack trace looks something like:
 java.lang.IllegalArgumentException: null
 at 
 java.util.concurrent.ScheduledThreadPoolExecutor.scheduleAtFixedRate(ScheduledThreadPoolExecutor.java:420)
 at 
 org.apache.hadoop.metrics2.lib.MutableQuantiles.init(MutableQuantiles.java:107)
 at 
 org.apache.hadoop.metrics2.lib.MetricsRegistry.newQuantiles(MetricsRegistry.java:200)
 Along similar lines, should the other methods like 
 MetricsRegistry#newCounter() also have parameter checking for negative 
 int/long values?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HADOOP-11529) Fix findbugs warnings in hadoop-archives

2015-02-02 Thread Hadoop QA (JIRA)

[ 
https://issues.apache.org/jira/browse/HADOOP-11529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14302321#comment-14302321
 ] 

Hadoop QA commented on HADOOP-11529:


{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  
http://issues.apache.org/jira/secure/attachment/12696028/HADOOP-11529.001.patch
  against trunk revision 8acc5e9.

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:red}-1 tests included{color}.  The patch doesn't appear to include 
any new or modified tests.
Please justify why no new tests are needed for this 
patch.
Also please list what manual steps were performed to 
verify this patch.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  There were no new javadoc warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 2.0.3) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 core tests{color}.  The patch passed unit tests in 
hadoop-tools/hadoop-archives.

Test results: 
https://builds.apache.org/job/PreCommit-HADOOP-Build/5560//testReport/
Console output: 
https://builds.apache.org/job/PreCommit-HADOOP-Build/5560//console

This message is automatically generated.

 Fix findbugs warnings in hadoop-archives
 

 Key: HADOOP-11529
 URL: https://issues.apache.org/jira/browse/HADOOP-11529
 Project: Hadoop Common
  Issue Type: Bug
  Components: tools
Reporter: Masatake Iwasaki
Assignee: Masatake Iwasaki
Priority: Minor
 Attachments: HADOOP-11529.001.patch


 {code}
 NPPossible null pointer dereference of reader in 
 org.apache.hadoop.tools.HadoopArchives$HArchiveInputFormat.getSplits(JobConf, 
 int) on exception path
 DmFound reliance on default encoding in 
 org.apache.hadoop.tools.HadoopArchives$HArchivesReducer.close(): 
 String.getBytes()
 DmFound reliance on default encoding in 
 org.apache.hadoop.tools.HadoopArchives$HArchivesReducer.configure(JobConf): 
 String.getBytes()
 DmFound reliance on default encoding in 
 org.apache.hadoop.tools.HadoopArchives$HArchivesReducer.reduce(IntWritable, 
 Iterator, OutputCollector, Reporter): String.getBytes()
 {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HADOOP-11520) Clean incomplete multi-part uploads in S3A tests

2015-02-02 Thread Hadoop QA (JIRA)

[ 
https://issues.apache.org/jira/browse/HADOOP-11520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14301260#comment-14301260
 ] 

Hadoop QA commented on HADOOP-11520:


{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  
http://issues.apache.org/jira/secure/attachment/12695622/HADOOP-11520.001.patch
  against trunk revision ffc75d6.

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 1 new 
or modified test files.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  There were no new javadoc warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 2.0.3) warnings.

{color:red}-1 release audit{color}.  The applied patch generated 1 
release audit warnings.

{color:green}+1 core tests{color}.  The patch passed unit tests in 
hadoop-tools/hadoop-aws.

Test results: 
https://builds.apache.org/job/PreCommit-HADOOP-Build///testReport/
Release audit warnings: 
https://builds.apache.org/job/PreCommit-HADOOP-Build///artifact/patchprocess/patchReleaseAuditProblems.txt
Console output: 
https://builds.apache.org/job/PreCommit-HADOOP-Build///console

This message is automatically generated.

 Clean incomplete multi-part uploads in S3A tests
 

 Key: HADOOP-11520
 URL: https://issues.apache.org/jira/browse/HADOOP-11520
 Project: Hadoop Common
  Issue Type: Improvement
  Components: fs/s3
Affects Versions: 2.6.0
Reporter: Thomas Demoor
Assignee: Thomas Demoor
Priority: Minor
 Fix For: 2.7.0

 Attachments: HADOOP-11520.001.patch


 As proposed in HADOOP-11488. This patch activates the purging functionality 
 of s3a at the start of each test. This cleans up any in-progress multi-part 
 uploads in the test bucket, preventing unknowing users from eternally paying 
 Amazon for the space of the already uploaded parts of previous tests that 
 failed during a multi-part upload. 
 People who have run the s3a tests should run a single test (evidently after 
 this patch is applied) against all their testbuckets (or manually abort 
 multipart).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (HADOOP-11520) Clean incomplete multi-part uploads in S3A tests

2015-02-02 Thread Steve Loughran (JIRA)

 [ 
https://issues.apache.org/jira/browse/HADOOP-11520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Steve Loughran updated HADOOP-11520:

Status: Patch Available  (was: Open)

 Clean incomplete multi-part uploads in S3A tests
 

 Key: HADOOP-11520
 URL: https://issues.apache.org/jira/browse/HADOOP-11520
 Project: Hadoop Common
  Issue Type: Improvement
  Components: fs/s3
Affects Versions: 2.6.0
Reporter: Thomas Demoor
Assignee: Thomas Demoor
Priority: Minor
 Fix For: 2.7.0

 Attachments: HADOOP-11520.001.patch


 As proposed in HADOOP-11488. This patch activates the purging functionality 
 of s3a at the start of each test. This cleans up any in-progress multi-part 
 uploads in the test bucket, preventing unknowing users from eternally paying 
 Amazon for the space of the already uploaded parts of previous tests that 
 failed during a multi-part upload. 
 People who have run the s3a tests should run a single test (evidently after 
 this patch is applied) against all their testbuckets (or manually abort 
 multipart).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HADOOP-8640) DU thread transient failures propagate to callers

2015-02-02 Thread Elad Itzhakian (JIRA)

[ 
https://issues.apache.org/jira/browse/HADOOP-8640?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14301084#comment-14301084
 ] 

Elad Itzhakian commented on HADOOP-8640:


Getting this in 2.6.0 too, during TestDFSIO
{quote}
2014-12-15 20:40:19,944 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: 
IOException in BlockReceiver constructor. Cause is
org.apache.hadoop.util.Shell$ExitCodeException: du: cannot access 
`/data3/dfs/dn/current/BP-1172228382-36.0.0.112-1418564551057/current/rbw/blk_1073754273':
 No such file or directory
at org.apache.hadoop.util.Shell.runCommand(Shell.java:511)
at org.apache.hadoop.util.Shell.run(Shell.java:424)
at org.apache.hadoop.fs.DU.run(DU.java:190)
at org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:119)
at java.lang.Thread.run(Thread.java:745)
{quote}

 DU thread transient failures propagate to callers
 -

 Key: HADOOP-8640
 URL: https://issues.apache.org/jira/browse/HADOOP-8640
 Project: Hadoop Common
  Issue Type: Bug
  Components: fs, io
Affects Versions: 2.0.0-alpha, 1.2.1
Reporter: Todd Lipcon

 When running some stress tests, I saw a failure where the DURefreshThread 
 failed due to the filesystem changing underneath it:
 {code}
 org.apache.hadoop.util.Shell$ExitCodeException: du: cannot access 
 `/data/4/dfs/dn/current/BP-1928785663-172.20.90.20-1343880685858/current/rbw/blk_4637779214690837894':
  No such file or directory
 {code}
 (the block was probably finalized while the du process was running, which 
 caused it to fail)
 The next block write, then, called {{getUsed()}}, and the exception got 
 propagated causing the write to fail. Since it was a pseudo-distributed 
 cluster, the client was unable to pick a different node to write to and 
 failed.
 The current behavior of propagating the exception to the next (and only the 
 next) caller doesn't seem well-thought-out.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Created] (HADOOP-11535) TableMapping related tests failed due to 'correct' resolving for test hostname

2015-02-02 Thread Kai Zheng (JIRA)
Kai Zheng created HADOOP-11535:
--

 Summary: TableMapping related tests failed due to 'correct' 
resolving for test hostname
 Key: HADOOP-11535
 URL: https://issues.apache.org/jira/browse/HADOOP-11535
 Project: Hadoop Common
  Issue Type: Test
Reporter: Kai Zheng
Assignee: Kai Zheng
Priority: Minor


When mvn test in my environment, it reported the following.
{noformat}
Failed tests: 
  TestTableMapping.testClearingCachedMappings:144 expected:/[rack1] but 
was:/[default-rack]
  TestTableMapping.testTableCaching:79 expected:/[rack1] but 
was:/[default-rack]
  TestTableMapping.testResolve:56 expected:/[rack1] but was:/[default-rack]
{noformat}

It's caused by the good resolving for the 'bad test' hostname 'a.b.c' as 
follows.
{noformat}
[drankye@zkdesk hadoop-common-project]$ ping a.b.c
PING a.b.c (220.250.64.228) 56(84) bytes of data.
{noformat}

I understand it may happen in just my local environment, and document this just 
in case others also meet this. We may use even worse hostname than 'a.b.c' to 
avoid such situation.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HADOOP-11462) TestSocketIOWithTimeout needs change for PowerPC platform

2015-02-02 Thread Ayappan (JIRA)

[ 
https://issues.apache.org/jira/browse/HADOOP-11462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14302802#comment-14302802
 ] 

Ayappan commented on HADOOP-11462:
--

Above build failure is not due to this patch since the failure happens in 
mapreduce related testcase whereas this patch is for TestSocketIOWithTimeout ( 
hadoop-common module ). Check out 
https://www.mail-archive.com/mapreduce-dev@hadoop.apache.org/msg12794.html

 TestSocketIOWithTimeout needs change for PowerPC platform
 -

 Key: HADOOP-11462
 URL: https://issues.apache.org/jira/browse/HADOOP-11462
 Project: Hadoop Common
  Issue Type: Bug
  Components: test
Affects Versions: 2.5.2
 Environment: PowerPC
Reporter: Ayappan
Assignee: Ayappan
 Fix For: 2.7.0

 Attachments: HADOOP-9627-v1.patch, HADOOP-9627-v2.patch


 TestSocketIOWithTimeout uses a block size of 4192 bytes to simulate a partial 
 write. This seems to be a valid in x86 architecture where the default minimum 
 blocksize is 4096. 
 This testcase fails in PowerPC where the default minimum block size is 65536 
 bytes (64KB). So for PowerPC, using a blocksize little more than 64K , say 
 6(65536 + 19) holds good for this scenario.
 I attached a patch here where i made it very general by introducing 
 NativeIO.POSIX.getCacheManipulator().getOperatingSystemPageSize() to get the 
 page size.
 I tested my patch in both ppc64 and x86 linux machines.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HADOOP-11043) Use Path Instead of String in DistCp Tests

2015-02-02 Thread Tsuyoshi OZAWA (JIRA)

[ 
https://issues.apache.org/jira/browse/HADOOP-11043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14302800#comment-14302800
 ] 

Tsuyoshi OZAWA commented on HADOOP-11043:
-

Cancelling a patch.

 Use Path Instead of String in DistCp Tests
 --

 Key: HADOOP-11043
 URL: https://issues.apache.org/jira/browse/HADOOP-11043
 Project: Hadoop Common
  Issue Type: Improvement
  Components: tools/distcp
Reporter: Gary Steelman
Assignee: Gary Steelman
Priority: Minor
 Attachments: HADOOP-11043.1.patch


 Many DistCp tests mix String and Path for files for no real reason. This 
 patch aligns use to Path.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Resolved] (HADOOP-11532) RAT checker complaining about PSD images

2015-02-02 Thread Jason Lowe (JIRA)

 [ 
https://issues.apache.org/jira/browse/HADOOP-11532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Lowe resolved HADOOP-11532.
-
Resolution: Duplicate

This is a duplicate of YARN-3113.

 RAT checker complaining about PSD images
 

 Key: HADOOP-11532
 URL: https://issues.apache.org/jira/browse/HADOOP-11532
 Project: Hadoop Common
  Issue Type: Bug
  Components: build
Affects Versions: 3.0.0
Reporter: Steve Loughran

 Jenkins is rejecting builds as {{Sorting icons.psd}} doesn't have an ASF 
 header.
 {code}
  !? 
 /home/jenkins/jenkins-slave/workspace/PreCommit-HADOOP-Build/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/webapps/static/dt-1.9.4/images/Sorting
  icons.psd
 Lines that start with ? in the release audit report indicate files that 
 do not have an Apache license header.
 {code}
 It's a layered photoshop image that either needs to be excluded from RAT or 
 cut from the source tree



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (HADOOP-11293) Factor OSType out from Shell

2015-02-02 Thread Steve Loughran (JIRA)

 [ 
https://issues.apache.org/jira/browse/HADOOP-11293?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Steve Loughran updated HADOOP-11293:

Status: Open  (was: Patch Available)

 Factor OSType out from Shell
 

 Key: HADOOP-11293
 URL: https://issues.apache.org/jira/browse/HADOOP-11293
 Project: Hadoop Common
  Issue Type: Improvement
  Components: fs, util
Reporter: Yongjun Zhang
Assignee: Yongjun Zhang
 Attachments: HADOOP-11293.001.patch, HADOOP-11293.002.patch, 
 HADOOP-11293.003.patch, HADOOP-11293.004.patch, HADOOP-11293.005.patch


 Currently the code that detects the OS type is located in Shell.java. Code 
 that need to check OS type refers to Shell, even if no other stuff of Shell 
 is needed. 
 I am proposing to refactor OSType out to  its own class, so to make the 
 OSType easier to access and the dependency cleaner.
  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HADOOP-11293) Factor OSType out from Shell

2015-02-02 Thread Steve Loughran (JIRA)

[ 
https://issues.apache.org/jira/browse/HADOOP-11293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14301374#comment-14301374
 ] 

Steve Loughran commented on HADOOP-11293:
-

lets bounce Jenkins and see what happens again...there's sometimes conflict 
with other tests on the same host

 Factor OSType out from Shell
 

 Key: HADOOP-11293
 URL: https://issues.apache.org/jira/browse/HADOOP-11293
 Project: Hadoop Common
  Issue Type: Improvement
  Components: fs, util
Reporter: Yongjun Zhang
Assignee: Yongjun Zhang
 Attachments: HADOOP-11293.001.patch, HADOOP-11293.002.patch, 
 HADOOP-11293.003.patch, HADOOP-11293.004.patch, HADOOP-11293.005.patch


 Currently the code that detects the OS type is located in Shell.java. Code 
 that need to check OS type refers to Shell, even if no other stuff of Shell 
 is needed. 
 I am proposing to refactor OSType out to  its own class, so to make the 
 OSType easier to access and the dependency cleaner.
  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (HADOOP-11293) Factor OSType out from Shell

2015-02-02 Thread Steve Loughran (JIRA)

 [ 
https://issues.apache.org/jira/browse/HADOOP-11293?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Steve Loughran updated HADOOP-11293:

Status: Patch Available  (was: Open)

 Factor OSType out from Shell
 

 Key: HADOOP-11293
 URL: https://issues.apache.org/jira/browse/HADOOP-11293
 Project: Hadoop Common
  Issue Type: Improvement
  Components: fs, util
Reporter: Yongjun Zhang
Assignee: Yongjun Zhang
 Attachments: HADOOP-11293.001.patch, HADOOP-11293.002.patch, 
 HADOOP-11293.003.patch, HADOOP-11293.004.patch, HADOOP-11293.005.patch


 Currently the code that detects the OS type is located in Shell.java. Code 
 that need to check OS type refers to Shell, even if no other stuff of Shell 
 is needed. 
 I am proposing to refactor OSType out to  its own class, so to make the 
 OSType easier to access and the dependency cleaner.
  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HADOOP-11520) Clean incomplete multi-part uploads in S3A tests

2015-02-02 Thread Thomas Demoor (JIRA)

[ 
https://issues.apache.org/jira/browse/HADOOP-11520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14301344#comment-14301344
 ] 

Thomas Demoor commented on HADOOP-11520:


unrelated error. photoshop documents lol?

 Clean incomplete multi-part uploads in S3A tests
 

 Key: HADOOP-11520
 URL: https://issues.apache.org/jira/browse/HADOOP-11520
 Project: Hadoop Common
  Issue Type: Improvement
  Components: fs/s3
Affects Versions: 2.6.0
Reporter: Thomas Demoor
Assignee: Thomas Demoor
Priority: Minor
 Fix For: 2.7.0

 Attachments: HADOOP-11520.001.patch


 As proposed in HADOOP-11488. This patch activates the purging functionality 
 of s3a at the start of each test. This cleans up any in-progress multi-part 
 uploads in the test bucket, preventing unknowing users from eternally paying 
 Amazon for the space of the already uploaded parts of previous tests that 
 failed during a multi-part upload. 
 People who have run the s3a tests should run a single test (evidently after 
 this patch is applied) against all their testbuckets (or manually abort 
 multipart).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HADOOP-11293) Factor OSType out from Shell

2015-02-02 Thread Yongjun Zhang (JIRA)

[ 
https://issues.apache.org/jira/browse/HADOOP-11293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14301412#comment-14301412
 ] 

Yongjun Zhang commented on HADOOP-11293:


Hi Steve, many thanks for your help!


 Factor OSType out from Shell
 

 Key: HADOOP-11293
 URL: https://issues.apache.org/jira/browse/HADOOP-11293
 Project: Hadoop Common
  Issue Type: Improvement
  Components: fs, util
Reporter: Yongjun Zhang
Assignee: Yongjun Zhang
 Attachments: HADOOP-11293.001.patch, HADOOP-11293.002.patch, 
 HADOOP-11293.003.patch, HADOOP-11293.004.patch, HADOOP-11293.005.patch


 Currently the code that detects the OS type is located in Shell.java. Code 
 that need to check OS type refers to Shell, even if no other stuff of Shell 
 is needed. 
 I am proposing to refactor OSType out to  its own class, so to make the 
 OSType easier to access and the dependency cleaner.
  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HADOOP-11535) TableMapping related tests failed due to 'correct' resolving for test hostname

2015-02-02 Thread Steve Loughran (JIRA)

[ 
https://issues.apache.org/jira/browse/HADOOP-11535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14301436#comment-14301436
 ] 

Steve Loughran commented on HADOOP-11535:
-

It's traditional to use {{something.example.org}} as the IETF mandates that 
these hostnames must never resolve. However if you are using Verizon fibre @ 
home, it may still resolve for you. There is an implicit expectation that a 
network doesn't work as defined.
 

 TableMapping related tests failed due to 'correct' resolving for test hostname
 --

 Key: HADOOP-11535
 URL: https://issues.apache.org/jira/browse/HADOOP-11535
 Project: Hadoop Common
  Issue Type: Test
Reporter: Kai Zheng
Assignee: Kai Zheng
Priority: Minor

 When mvn test in my environment, it reported the following.
 {noformat}
 Failed tests: 
   TestTableMapping.testClearingCachedMappings:144 expected:/[rack1] but 
 was:/[default-rack]
   TestTableMapping.testTableCaching:79 expected:/[rack1] but 
 was:/[default-rack]
   TestTableMapping.testResolve:56 expected:/[rack1] but 
 was:/[default-rack]
 {noformat}
 It's caused by the good resolving for the 'bad test' hostname 'a.b.c' as 
 follows.
 {noformat}
 [drankye@zkdesk hadoop-common-project]$ ping a.b.c
 PING a.b.c (220.250.64.228) 56(84) bytes of data.
 {noformat}
 I understand it may happen in just my local environment, and document this 
 just in case others also meet this. We may use even worse hostname than 
 'a.b.c' to avoid such situation.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (HADOOP-11104) org.apache.hadoop.metrics2.lib.MetricsRegistry needs numerical parameter checking

2015-02-02 Thread Ray Chiang (JIRA)

 [ 
https://issues.apache.org/jira/browse/HADOOP-11104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ray Chiang updated HADOOP-11104:

Attachment: HADOOP-11104.001.patch

Initial version

 org.apache.hadoop.metrics2.lib.MetricsRegistry needs numerical parameter 
 checking
 -

 Key: HADOOP-11104
 URL: https://issues.apache.org/jira/browse/HADOOP-11104
 Project: Hadoop Common
  Issue Type: Bug
Reporter: Ray Chiang
Assignee: Ray Chiang
Priority: Minor
  Labels: newbie
 Attachments: HADOOP-11104.001.patch


 Passing a negative value to the interval field of 
 MetricsRegistry#newQuantiles should throw a MetricsException with a clear 
 error message.  The current stack trace looks something like:
 java.lang.IllegalArgumentException: null
 at 
 java.util.concurrent.ScheduledThreadPoolExecutor.scheduleAtFixedRate(ScheduledThreadPoolExecutor.java:420)
 at 
 org.apache.hadoop.metrics2.lib.MutableQuantiles.init(MutableQuantiles.java:107)
 at 
 org.apache.hadoop.metrics2.lib.MetricsRegistry.newQuantiles(MetricsRegistry.java:200)
 Along similar lines, should the other methods like 
 MetricsRegistry#newCounter() also have parameter checking for negative 
 int/long values?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HADOOP-9565) Add a Blobstore interface to add to blobstore FileSystems

2015-02-02 Thread Chris Nauroth (JIRA)

[ 
https://issues.apache.org/jira/browse/HADOOP-9565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14301606#comment-14301606
 ] 

Chris Nauroth commented on HADOOP-9565:
---

This is nice work, Steve.  Thanks for putting this together.  Here are a few 
comments:

# Have you considered attaching semantics directly to {{FileSystem}} and 
{{FileContext}} instead of introducing the {{ObjectStore}} abstract base class? 
 This could give us a modest simplification of calling code, because it 
wouldn't have to downcast, either directly or implicitly by taking the extra 
hop through the static {{ObjectStore#hasStoreSemantics}}.
# Shall we change {{StoreSemantics}} to an enum and then phrase the API in 
terms of {{EnumSetStoreSemantics}} for improved type safety?  There is 
existing precedent for use of {{EnumSet}} in the {{FileSystem}} API, so I 
expect callers will find it familiar.
# I saw JavaDoc typos on {{ATOMIC_DIR_DELETE}} and 
{{DISTINCT_FILES_AND_DIRECTORIES}}.


 Add a Blobstore interface to add to blobstore FileSystems
 -

 Key: HADOOP-9565
 URL: https://issues.apache.org/jira/browse/HADOOP-9565
 Project: Hadoop Common
  Issue Type: Improvement
  Components: fs, fs/s3, fs/swift
Affects Versions: 2.6.0
Reporter: Steve Loughran
Assignee: Steve Loughran
 Attachments: HADOOP-9565-001.patch, HADOOP-9565-002.patch, 
 HADOOP-9565-003.patch


 We can make the fact that some {{FileSystem}} implementations are really 
 blobstores, with different atomicity and consistency guarantees, by adding a 
 {{Blobstore}} interface to add to them. 
 This could also be a place to add a {{Copy(Path,Path)}} method, assuming that 
 all blobstores implement at server-side copy operation as a substitute for 
 rename.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HADOOP-11506) Configuration.get() is unnecessarily slow

2015-02-02 Thread Andrew Wang (JIRA)

[ 
https://issues.apache.org/jira/browse/HADOOP-11506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14301624#comment-14301624
 ] 

Andrew Wang commented on HADOOP-11506:
--

+1 thanks Gera

 Configuration.get() is unnecessarily slow
 -

 Key: HADOOP-11506
 URL: https://issues.apache.org/jira/browse/HADOOP-11506
 Project: Hadoop Common
  Issue Type: Bug
Reporter: Dmitriy V. Ryaboy
Assignee: Gera Shegalov
 Attachments: HADOOP-11506.001.patch, HADOOP-11506.002.patch, 
 HADOOP-11506.003.patch


 Profiling several large Hadoop jobs, we discovered that a surprising amount 
 of time was spent inside Configuration.get, more specifically, in regex 
 matching caused by the substituteVars call.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (HADOOP-11104) org.apache.hadoop.metrics2.lib.MetricsRegistry needs numerical parameter checking

2015-02-02 Thread Ray Chiang (JIRA)

 [ 
https://issues.apache.org/jira/browse/HADOOP-11104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ray Chiang updated HADOOP-11104:

Status: Patch Available  (was: Open)

 org.apache.hadoop.metrics2.lib.MetricsRegistry needs numerical parameter 
 checking
 -

 Key: HADOOP-11104
 URL: https://issues.apache.org/jira/browse/HADOOP-11104
 Project: Hadoop Common
  Issue Type: Bug
Reporter: Ray Chiang
Assignee: Ray Chiang
Priority: Minor
  Labels: newbie
 Attachments: HADOOP-11104.001.patch


 Passing a negative value to the interval field of 
 MetricsRegistry#newQuantiles should throw a MetricsException with a clear 
 error message.  The current stack trace looks something like:
 java.lang.IllegalArgumentException: null
 at 
 java.util.concurrent.ScheduledThreadPoolExecutor.scheduleAtFixedRate(ScheduledThreadPoolExecutor.java:420)
 at 
 org.apache.hadoop.metrics2.lib.MutableQuantiles.init(MutableQuantiles.java:107)
 at 
 org.apache.hadoop.metrics2.lib.MetricsRegistry.newQuantiles(MetricsRegistry.java:200)
 Along similar lines, should the other methods like 
 MetricsRegistry#newCounter() also have parameter checking for negative 
 int/long values?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HADOOP-11525) FileSystem should expose some performance characteristics for caller (e.g., FsShell) to choose the right algorithm.

2015-02-02 Thread Chris Nauroth (JIRA)

[ 
https://issues.apache.org/jira/browse/HADOOP-11525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14301613#comment-14301613
 ] 

Chris Nauroth commented on HADOOP-11525:


Thanks, Steve.  I had a hunch there was an existing jira.

I'm going to back off on my suggestion of a new {{FileSystem}} method for 
transactional put.  Even if we did that, we'd still need the 
characteristics/semantics querying approach for other reasons, and I don't see 
a benefit to expanding the public API footprint in multiple different ways.

Eddy, I see that the HADOOP-9565 patch does include the same basic {{FsShell}} 
change that you want to achieve.  Are you comfortable resolving this as 
duplicate and participating over there?

 FileSystem should expose some performance characteristics for caller (e.g., 
 FsShell) to choose the right algorithm.
 ---

 Key: HADOOP-11525
 URL: https://issues.apache.org/jira/browse/HADOOP-11525
 Project: Hadoop Common
  Issue Type: Improvement
  Components: tools
Affects Versions: 2.6.0
Reporter: Lei (Eddy) Xu
Assignee: Lei (Eddy) Xu
 Attachments: HADOOP-11525.000.patch


 When running {{hadoop fs -put}},  {{FsShell}} creates a {{._COPYING_.}} file 
 on the target directory, and then renames it to target file when the write is 
 done. However, for some targeted systems, such as S3, Azure and Swift, a 
 partial failure write request (i.e., {{PUT}}) has not side effect, while the 
 {{rename}} operation is expensive. 
 {{FileSystem}} should expose some characteristics so that the operation such 
 as {{CommandWithDestination#copyStreamToTarget()}} can detect and choose the 
 right way to do.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HADOOP-11506) Configuration.get() is unnecessarily slow

2015-02-02 Thread Gera Shegalov (JIRA)

[ 
https://issues.apache.org/jira/browse/HADOOP-11506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14301611#comment-14301611
 ] 

Gera Shegalov commented on HADOOP-11506:


The release audit warning is tracked by YARN-3113. The patch is reviewable, 
[~andrew.wang].

 Configuration.get() is unnecessarily slow
 -

 Key: HADOOP-11506
 URL: https://issues.apache.org/jira/browse/HADOOP-11506
 Project: Hadoop Common
  Issue Type: Bug
Reporter: Dmitriy V. Ryaboy
Assignee: Gera Shegalov
 Attachments: HADOOP-11506.001.patch, HADOOP-11506.002.patch, 
 HADOOP-11506.003.patch


 Profiling several large Hadoop jobs, we discovered that a surprising amount 
 of time was spent inside Configuration.get, more specifically, in regex 
 matching caused by the substituteVars call.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HADOOP-10101) Update guava dependency to the latest version

2015-02-02 Thread Tsuyoshi OZAWA (JIRA)

[ 
https://issues.apache.org/jira/browse/HADOOP-10101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14301615#comment-14301615
 ] 

Tsuyoshi OZAWA commented on HADOOP-10101:
-

Thanks Steve for your review. I'll check the javac warnings.

 Update guava dependency to the latest version
 -

 Key: HADOOP-10101
 URL: https://issues.apache.org/jira/browse/HADOOP-10101
 Project: Hadoop Common
  Issue Type: Improvement
Affects Versions: 2.2.0, 2.6.0
Reporter: Rakesh R
Assignee: Vinayakumar B
 Attachments: HADOOP-10101-002.patch, HADOOP-10101-004.patch, 
 HADOOP-10101-005.patch, HADOOP-10101-006.patch, HADOOP-10101-007.patch, 
 HADOOP-10101-008.patch, HADOOP-10101.patch, HADOOP-10101.patch


 The existing guava version is 11.0.2 which is quite old. This 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (HADOOP-11518) Prefix-based per-RPC server configuration

2015-02-02 Thread Kihwal Lee (JIRA)

 [ 
https://issues.apache.org/jira/browse/HADOOP-11518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kihwal Lee updated HADOOP-11518:

Attachment: HADOOP-11518.v2.patch

The new patch modifies {{Configuration}} class directly for the prefix support, 
instead of adding {{PrefixedConfiguration}}. 

 Prefix-based per-RPC server configuration
 -

 Key: HADOOP-11518
 URL: https://issues.apache.org/jira/browse/HADOOP-11518
 Project: Hadoop Common
  Issue Type: Improvement
  Components: ipc
Reporter: Kihwal Lee
Assignee: Kihwal Lee
 Attachments: HADOOP-11518.patch, HADOOP-11518.v2.patch


 If a process runs multiple RPC servers, there is no way to individually set 
 their IPC parameters beyond what's supported by the RPC Builder.  If a 
 configuration is shared among many nodes, this is more problematic. 
 In this jira, a prefix-based per-server configuration is proposed.  When an 
 RPC server is built, a prefix can be specified, which will then be prepended 
 to keys when retrieving configured values.  There will be an option to 
 fall-back to using the key without the prefix when no value is found with the 
 prefix, thus making this feature compatible with the current way of 
 configuring servers.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HADOOP-11494) Lock acquisition on WrappedInputStream#unwrappedRpcBuffer may race with another thread

2015-02-02 Thread Hudson (JIRA)

[ 
https://issues.apache.org/jira/browse/HADOOP-11494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14301633#comment-14301633
 ] 

Hudson commented on HADOOP-11494:
-

FAILURE: Integrated in Hadoop-trunk-Commit #6977 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/6977/])
HADOOP-11494. Lock acquisition on WrappedInputStream#unwrappedRpcBuffer may 
race with another thread. Contributed by Ted Yu. (benoy: rev 
3472e3bd6c50558870b86c9ccfea5072385fa991)
* 
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/SaslRpcClient.java


 Lock acquisition on WrappedInputStream#unwrappedRpcBuffer may race with 
 another thread
 --

 Key: HADOOP-11494
 URL: https://issues.apache.org/jira/browse/HADOOP-11494
 Project: Hadoop Common
  Issue Type: Bug
Reporter: Ted Yu
Assignee: Ted Yu
Priority: Minor
 Attachments: hadoop-11494-001.patch, hadoop-11494-002.patch


 In SaslRpcClient, starting at line 576:
 {code}
 public int read(byte[] buf, int off, int len) throws IOException {
   synchronized(unwrappedRpcBuffer) {
 // fill the buffer with the next RPC message
 if (unwrappedRpcBuffer.remaining() == 0) {
   readNextRpcPacket();
 }
 {code}
 readNextRpcPacket() may assign another ByteBuffer to unwrappedRpcBuffer, 
 making the lock on previous ByteBuffer not useful.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (HADOOP-11525) FileSystem should expose some performance characteristics for caller (e.g., FsShell) to choose the right algorithm.

2015-02-02 Thread Lei (Eddy) Xu (JIRA)

 [ 
https://issues.apache.org/jira/browse/HADOOP-11525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lei (Eddy) Xu updated HADOOP-11525:
---
Resolution: Duplicate
Status: Resolved  (was: Patch Available)

[~stev...@iseran.com] Thanks for working on HADOOP-9565.  It looks like a 
better solution than this one. I will close this one instead.

[~cnauroth] Sure. I'd love to resolve this as duplicated. 


 FileSystem should expose some performance characteristics for caller (e.g., 
 FsShell) to choose the right algorithm.
 ---

 Key: HADOOP-11525
 URL: https://issues.apache.org/jira/browse/HADOOP-11525
 Project: Hadoop Common
  Issue Type: Improvement
  Components: tools
Affects Versions: 2.6.0
Reporter: Lei (Eddy) Xu
Assignee: Lei (Eddy) Xu
 Attachments: HADOOP-11525.000.patch


 When running {{hadoop fs -put}},  {{FsShell}} creates a {{._COPYING_.}} file 
 on the target directory, and then renames it to target file when the write is 
 done. However, for some targeted systems, such as S3, Azure and Swift, a 
 partial failure write request (i.e., {{PUT}}) has not side effect, while the 
 {{rename}} operation is expensive. 
 {{FileSystem}} should expose some characteristics so that the operation such 
 as {{CommandWithDestination#copyStreamToTarget()}} can detect and choose the 
 right way to do.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (HADOOP-11494) Lock acquisition on WrappedInputStream#unwrappedRpcBuffer may race with another thread

2015-02-02 Thread Benoy Antony (JIRA)

 [ 
https://issues.apache.org/jira/browse/HADOOP-11494?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Benoy Antony updated HADOOP-11494:
--
Fix Version/s: 2.7.0

 Lock acquisition on WrappedInputStream#unwrappedRpcBuffer may race with 
 another thread
 --

 Key: HADOOP-11494
 URL: https://issues.apache.org/jira/browse/HADOOP-11494
 Project: Hadoop Common
  Issue Type: Bug
Reporter: Ted Yu
Assignee: Ted Yu
Priority: Minor
 Fix For: 2.7.0

 Attachments: hadoop-11494-001.patch, hadoop-11494-002.patch


 In SaslRpcClient, starting at line 576:
 {code}
 public int read(byte[] buf, int off, int len) throws IOException {
   synchronized(unwrappedRpcBuffer) {
 // fill the buffer with the next RPC message
 if (unwrappedRpcBuffer.remaining() == 0) {
   readNextRpcPacket();
 }
 {code}
 readNextRpcPacket() may assign another ByteBuffer to unwrappedRpcBuffer, 
 making the lock on previous ByteBuffer not useful.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (HADOOP-11494) Lock acquisition on WrappedInputStream#unwrappedRpcBuffer may race with another thread

2015-02-02 Thread Benoy Antony (JIRA)

 [ 
https://issues.apache.org/jira/browse/HADOOP-11494?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Benoy Antony updated HADOOP-11494:
--
Resolution: Fixed
Status: Resolved  (was: Patch Available)

committed to trunk and branch-2. Thank you Ted.

 Lock acquisition on WrappedInputStream#unwrappedRpcBuffer may race with 
 another thread
 --

 Key: HADOOP-11494
 URL: https://issues.apache.org/jira/browse/HADOOP-11494
 Project: Hadoop Common
  Issue Type: Bug
Reporter: Ted Yu
Assignee: Ted Yu
Priority: Minor
 Fix For: 2.7.0

 Attachments: hadoop-11494-001.patch, hadoop-11494-002.patch


 In SaslRpcClient, starting at line 576:
 {code}
 public int read(byte[] buf, int off, int len) throws IOException {
   synchronized(unwrappedRpcBuffer) {
 // fill the buffer with the next RPC message
 if (unwrappedRpcBuffer.remaining() == 0) {
   readNextRpcPacket();
 }
 {code}
 readNextRpcPacket() may assign another ByteBuffer to unwrappedRpcBuffer, 
 making the lock on previous ByteBuffer not useful.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HADOOP-11442) hadoop-azure: Create test jar

2015-02-02 Thread Hadoop QA (JIRA)

[ 
https://issues.apache.org/jira/browse/HADOOP-11442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14301768#comment-14301768
 ] 

Hadoop QA commented on HADOOP-11442:


{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12695981/HADOOP-11442v2.patch
  against trunk revision f33c99b.

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:red}-1 tests included{color}.  The patch doesn't appear to include 
any new or modified tests.
Please justify why no new tests are needed for this 
patch.
Also please list what manual steps were performed to 
verify this patch.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  There were no new javadoc warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 2.0.3) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 core tests{color}.  The patch passed unit tests in 
hadoop-tools/hadoop-azure.

Test results: 
https://builds.apache.org/job/PreCommit-HADOOP-Build/5558//testReport/
Console output: 
https://builds.apache.org/job/PreCommit-HADOOP-Build/5558//console

This message is automatically generated.

 hadoop-azure: Create test jar
 -

 Key: HADOOP-11442
 URL: https://issues.apache.org/jira/browse/HADOOP-11442
 Project: Hadoop Common
  Issue Type: Improvement
  Components: tools
 Environment: windows, azure
Reporter: shashank
Assignee: Chris Nauroth
 Attachments: HADOOP-11442.patch, HADOOP-11442v1.patch, 
 HADOOP-11442v2.patch


 pom of hadoop-azure project to needs to be modified to create a test jar as 
 well. This test jar is required to run test cases of Windowsazuretablesink 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (HADOOP-11442) hadoop-azure: Create test jar

2015-02-02 Thread Chris Nauroth (JIRA)

 [ 
https://issues.apache.org/jira/browse/HADOOP-11442?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chris Nauroth updated HADOOP-11442:
---
  Resolution: Fixed
   Fix Version/s: 2.7.0
Target Version/s: 2.7.0
Hadoop Flags: Reviewed
  Status: Resolved  (was: Patch Available)

I committed this to trunk and branch-2.  Shashank, thank you for the 
contribution.

 hadoop-azure: Create test jar
 -

 Key: HADOOP-11442
 URL: https://issues.apache.org/jira/browse/HADOOP-11442
 Project: Hadoop Common
  Issue Type: Improvement
  Components: tools
 Environment: windows, azure
Reporter: shashank
Assignee: Chris Nauroth
 Fix For: 2.7.0

 Attachments: HADOOP-11442.patch, HADOOP-11442v1.patch, 
 HADOOP-11442v2.patch


 pom of hadoop-azure project to needs to be modified to create a test jar as 
 well. This test jar is required to run test cases of Windowsazuretablesink 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HADOOP-11506) Configuration.get() is unnecessarily slow

2015-02-02 Thread Gera Shegalov (JIRA)

[ 
https://issues.apache.org/jira/browse/HADOOP-11506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14301691#comment-14301691
 ] 

Gera Shegalov commented on HADOOP-11506:


Thank you, Andrew. I'll wait for a couple of days to see if there are more 
comments before committing.

 Configuration.get() is unnecessarily slow
 -

 Key: HADOOP-11506
 URL: https://issues.apache.org/jira/browse/HADOOP-11506
 Project: Hadoop Common
  Issue Type: Bug
Reporter: Dmitriy V. Ryaboy
Assignee: Gera Shegalov
 Attachments: HADOOP-11506.001.patch, HADOOP-11506.002.patch, 
 HADOOP-11506.003.patch


 Profiling several large Hadoop jobs, we discovered that a surprising amount 
 of time was spent inside Configuration.get, more specifically, in regex 
 matching caused by the substituteVars call.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (HADOOP-11442) hadoop-azure: Create test jar

2015-02-02 Thread Chris Nauroth (JIRA)

 [ 
https://issues.apache.org/jira/browse/HADOOP-11442?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chris Nauroth updated HADOOP-11442:
---
Attachment: HADOOP-11442v2.patch

[~shkhande], thank you for the update.  Here is the same patch, but converted 
to Linux line endings.  I expect Jenkins will be able to run with this.  +1 
pending Jenkins.

 hadoop-azure: Create test jar
 -

 Key: HADOOP-11442
 URL: https://issues.apache.org/jira/browse/HADOOP-11442
 Project: Hadoop Common
  Issue Type: Improvement
  Components: tools
 Environment: windows, azure
Reporter: shashank
Assignee: Chris Nauroth
 Attachments: HADOOP-11442.patch, HADOOP-11442v1.patch, 
 HADOOP-11442v2.patch


 pom of hadoop-azure project to needs to be modified to create a test jar as 
 well. This test jar is required to run test cases of Windowsazuretablesink 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HADOOP-11104) org.apache.hadoop.metrics2.lib.MetricsRegistry needs numerical parameter checking

2015-02-02 Thread Hadoop QA (JIRA)

[ 
https://issues.apache.org/jira/browse/HADOOP-11104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14301718#comment-14301718
 ] 

Hadoop QA commented on HADOOP-11104:


{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  
http://issues.apache.org/jira/secure/attachment/12695961/HADOOP-11104.001.patch
  against trunk revision ffc75d6.

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 1 new 
or modified test files.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  There were no new javadoc warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 2.0.3) warnings.

{color:red}-1 release audit{color}.  The applied patch generated 1 
release audit warnings.

{color:green}+1 core tests{color}.  The patch passed unit tests in 
hadoop-common-project/hadoop-common.

Test results: 
https://builds.apache.org/job/PreCommit-HADOOP-Build/5556//testReport/
Release audit warnings: 
https://builds.apache.org/job/PreCommit-HADOOP-Build/5556//artifact/patchprocess/patchReleaseAuditProblems.txt
Console output: 
https://builds.apache.org/job/PreCommit-HADOOP-Build/5556//console

This message is automatically generated.

 org.apache.hadoop.metrics2.lib.MetricsRegistry needs numerical parameter 
 checking
 -

 Key: HADOOP-11104
 URL: https://issues.apache.org/jira/browse/HADOOP-11104
 Project: Hadoop Common
  Issue Type: Bug
Reporter: Ray Chiang
Assignee: Ray Chiang
Priority: Minor
  Labels: newbie
 Attachments: HADOOP-11104.001.patch


 Passing a negative value to the interval field of 
 MetricsRegistry#newQuantiles should throw a MetricsException with a clear 
 error message.  The current stack trace looks something like:
 java.lang.IllegalArgumentException: null
 at 
 java.util.concurrent.ScheduledThreadPoolExecutor.scheduleAtFixedRate(ScheduledThreadPoolExecutor.java:420)
 at 
 org.apache.hadoop.metrics2.lib.MutableQuantiles.init(MutableQuantiles.java:107)
 at 
 org.apache.hadoop.metrics2.lib.MetricsRegistry.newQuantiles(MetricsRegistry.java:200)
 Along similar lines, should the other methods like 
 MetricsRegistry#newCounter() also have parameter checking for negative 
 int/long values?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HADOOP-10181) GangliaContext does not work with multicast ganglia setup

2015-02-02 Thread Hudson (JIRA)

[ 
https://issues.apache.org/jira/browse/HADOOP-10181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14301739#comment-14301739
 ] 

Hudson commented on HADOOP-10181:
-

SUCCESS: Integrated in Hadoop-trunk-Commit #6979 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/6979/])
HADOOP-10181. GangliaContext does not work with multicast ganglia setup. 
Contributed by Andrew Johnson. (cnauroth: rev 
8004a002307940176cc188657c68e85171a5b5a8)
* hadoop-common-project/hadoop-common/CHANGES.txt
* 
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics/ganglia/GangliaContext.java
* 
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/sink/ganglia/AbstractGangliaSink.java
* 
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/metrics2/sink/ganglia/TestGangliaSink.java
* 
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/metrics/ganglia/TestGangliaContext.java
* 
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics/ganglia/package.html


 GangliaContext does not work with multicast ganglia setup
 -

 Key: HADOOP-10181
 URL: https://issues.apache.org/jira/browse/HADOOP-10181
 Project: Hadoop Common
  Issue Type: Bug
  Components: metrics
Affects Versions: 2.6.0
Reporter: Andrew Otto
Assignee: Andrew Johnson
Priority: Minor
  Labels: ganglia, hadoop, metrics, multicast
 Attachments: HADOOP-10181.001.patch, HADOOP-10181.002.patch, 
 HADOOP-10181.003.patch


 The GangliaContext class which is used to send Hadoop metrics to Ganglia uses 
 a DatagramSocket to send these metrics.  This works fine for Ganglia 
 multicast setups that are all on the same VLAN.  However, when working with 
 multiple VLANs, a packet sent via DatagramSocket to a multicast address will 
 end up with a TTL of 1.  Multicast TTL indicates the number of network hops 
 for which a particular multicast packet is valid.  The packets sent by 
 GangliaContext do not make it to ganglia aggregrators on the same multicast 
 group, but in different VLANs.
 To fix, we'd need a configuration property that specifies that multicast is 
 to be used, and another that allows setting of the multicast packet TTL.  
 With these set, we could then use MulticastSocket setTimeToLive() instead of 
 just plain ol' DatagramSocket.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (HADOOP-10181) GangliaContext does not work with multicast ganglia setup

2015-02-02 Thread Chris Nauroth (JIRA)

 [ 
https://issues.apache.org/jira/browse/HADOOP-10181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chris Nauroth updated HADOOP-10181:
---
   Resolution: Fixed
Fix Version/s: 2.7.0
 Release Note: Hadoop metrics sent to Ganglia over multicast now support 
optional configuration of socket TTL.  The default TTL is 1, which preserves 
the behavior of prior Hadoop versions.  Clusters that span multiple 
subnets/VLANs will likely want to increase this.
   Status: Resolved  (was: Patch Available)

I committed this to trunk and branch-2.  Andrew, thank you for contributing 
this patch.

 GangliaContext does not work with multicast ganglia setup
 -

 Key: HADOOP-10181
 URL: https://issues.apache.org/jira/browse/HADOOP-10181
 Project: Hadoop Common
  Issue Type: Bug
  Components: metrics
Affects Versions: 2.6.0
Reporter: Andrew Otto
Assignee: Andrew Johnson
Priority: Minor
  Labels: ganglia, hadoop, metrics, multicast
 Fix For: 2.7.0

 Attachments: HADOOP-10181.001.patch, HADOOP-10181.002.patch, 
 HADOOP-10181.003.patch


 The GangliaContext class which is used to send Hadoop metrics to Ganglia uses 
 a DatagramSocket to send these metrics.  This works fine for Ganglia 
 multicast setups that are all on the same VLAN.  However, when working with 
 multiple VLANs, a packet sent via DatagramSocket to a multicast address will 
 end up with a TTL of 1.  Multicast TTL indicates the number of network hops 
 for which a particular multicast packet is valid.  The packets sent by 
 GangliaContext do not make it to ganglia aggregrators on the same multicast 
 group, but in different VLANs.
 To fix, we'd need a configuration property that specifies that multicast is 
 to be used, and another that allows setting of the multicast packet TTL.  
 With these set, we could then use MulticastSocket setTimeToLive() instead of 
 just plain ol' DatagramSocket.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HADOOP-11534) Minor improvements for raw erasure coders

2015-02-02 Thread Vinayakumar B (JIRA)

[ 
https://issues.apache.org/jira/browse/HADOOP-11534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14301032#comment-14301032
 ] 

Vinayakumar B commented on HADOOP-11534:


+1 for the latest changes. Will commit soon

 Minor improvements for raw erasure coders
 -

 Key: HADOOP-11534
 URL: https://issues.apache.org/jira/browse/HADOOP-11534
 Project: Hadoop Common
  Issue Type: Improvement
Affects Versions: HDFS-EC
Reporter: Kai Zheng
Assignee: Kai Zheng
Priority: Minor
 Fix For: HDFS-EC

 Attachments: HADOOP-11534-v1.patch, HADOOP-11534-v2.patch


 For the raw erasure coder API codes introduced by HADOOP-11514, there're some 
 minor improvements that were noticed and can be done separately.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (HADOOP-11534) Minor improvements for raw erasure coders

2015-02-02 Thread Vinayakumar B (JIRA)

 [ 
https://issues.apache.org/jira/browse/HADOOP-11534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vinayakumar B updated HADOOP-11534:
---
  Resolution: Fixed
Hadoop Flags: Reviewed
  Status: Resolved  (was: Patch Available)

Committed to HDFS-EC branch

 Minor improvements for raw erasure coders
 -

 Key: HADOOP-11534
 URL: https://issues.apache.org/jira/browse/HADOOP-11534
 Project: Hadoop Common
  Issue Type: Sub-task
Affects Versions: HDFS-EC
Reporter: Kai Zheng
Assignee: Kai Zheng
Priority: Minor
 Fix For: HDFS-EC

 Attachments: HADOOP-11534-v1.patch, HADOOP-11534-v2.patch


 For the raw erasure coder API codes introduced by HADOOP-11514, there're some 
 minor improvements that were noticed and can be done separately.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (HADOOP-11534) Minor improvements for raw erasure coders

2015-02-02 Thread Vinayakumar B (JIRA)

 [ 
https://issues.apache.org/jira/browse/HADOOP-11534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vinayakumar B updated HADOOP-11534:
---
Issue Type: Sub-task  (was: Improvement)
Parent: HADOOP-11264

 Minor improvements for raw erasure coders
 -

 Key: HADOOP-11534
 URL: https://issues.apache.org/jira/browse/HADOOP-11534
 Project: Hadoop Common
  Issue Type: Sub-task
Affects Versions: HDFS-EC
Reporter: Kai Zheng
Assignee: Kai Zheng
Priority: Minor
 Fix For: HDFS-EC

 Attachments: HADOOP-11534-v1.patch, HADOOP-11534-v2.patch


 For the raw erasure coder API codes introduced by HADOOP-11514, there're some 
 minor improvements that were noticed and can be done separately.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HADOOP-11534) Minor improvements for raw erasure coders

2015-02-02 Thread Kai Zheng (JIRA)

[ 
https://issues.apache.org/jira/browse/HADOOP-11534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14301042#comment-14301042
 ] 

Kai Zheng commented on HADOOP-11534:


Thanks [~vinayrpet] for the commit. 

 Minor improvements for raw erasure coders
 -

 Key: HADOOP-11534
 URL: https://issues.apache.org/jira/browse/HADOOP-11534
 Project: Hadoop Common
  Issue Type: Sub-task
Affects Versions: HDFS-EC
Reporter: Kai Zheng
Assignee: Kai Zheng
Priority: Minor
 Fix For: HDFS-EC

 Attachments: HADOOP-11534-v1.patch, HADOOP-11534-v2.patch


 For the raw erasure coder API codes introduced by HADOOP-11514, there're some 
 minor improvements that were noticed and can be done separately.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)