date:20150330

[
https://issues.apache.org/jira/browse/HADOOP-11552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Siddharth Seth updated HADOOP-11552:

Status: Patch Available (was: Open)

Allow handoff on the server side for RPC requests
-

Key: HADOOP-11552
URL: https://issues.apache.org/jira/browse/HADOOP-11552
Project: Hadoop Common
Issue Type: Improvement
Components: ipc
Reporter: Siddharth Seth
Assignee: Siddharth Seth
Attachments: HADOOP-11552.1.wip.txt, HADOOP-11552.2.txt

An RPC server handler thread is tied up for each incoming RPC request. This
isn't ideal, since this essentially implies that RPC operations should be
short lived, and most operations which could take time end up falling back to
a polling mechanism.
Some use cases where this is useful.
- YARN submitApplication - which currently submits, followed by a poll to
check if the application is accepted while the submit operation is written
out to storage. This can be collapsed into a single call.
- YARN allocate - requests and allocations use the same protocol. New
allocations are received via polling.
The allocate protocol could be split into a request/heartbeat along with a
'awaitResponse'. The request/heartbeat is sent only when there's a request or
on a much longer heartbeat interval. awaitResponse is always left active with
the RM - and returns the moment something is available.
MapReduce/Tez task to AM communication is another example of this pattern.
The same pattern of splitting calls can be used for other protocols as well.
This should serve to improve latency, as well as reduce network traffic since
the keep-alive heartbeat can be sent less frequently.
I believe there's some cases in HDFS as well, where the DN gets told to
perform some operations when they heartbeat into the NN.

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HADOOP-11552) Allow handoff on the server side for RPC requests

[
https://issues.apache.org/jira/browse/HADOOP-11552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Siddharth Seth updated HADOOP-11552:

Attachment: HADOOP-11552.2.txt

Updated patch with another couple of tests, and error handling.

[~sanjay.radia], [~daryn] - please review. Would appreciate it if you could pay
additional attention to ensuring this doesn't break the regular flow. I've
tried to keep the changes to a minimal for the regular flow.

Allow handoff on the server side for RPC requests
-

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-11774) listStatus in FTPFileSystem fails with connection reset

2015-03-30 Thread Haris AL (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-11774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14386294#comment-14386294
 ] 

Haris AL commented on HADOOP-11774:
---

FTPFileSystem.disconnect() internally calls FTPClient-logout() and 
FTPClient-disconnect(). Both these methods throws Exceptions.If a connection 
exception happens, we are likely not being able to connect with FTP Server. 
calling logout() will try to connect to FTP server again and access resources.

 listStatus in FTPFileSystem fails with connection reset
 ---

 Key: HADOOP-11774
 URL: https://issues.apache.org/jira/browse/HADOOP-11774
 Project: Hadoop Common
  Issue Type: Bug
  Components: fs
Affects Versions: 2.3.0
 Environment: Remote FTP located in Windows NT FTP
Reporter: Krishnamoorthy Dharmalingam

 Following exception trace raised when FTPFileSystem.listStatus() called in 
 Passive/active mode.
 Caused by: java.net.SocketException: Connection reset
   at java.net.SocketInputStream.read(SocketInputStream.java:196)
   at java.net.SocketInputStream.read(SocketInputStream.java:122)
   at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:283)
   at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:325)
   at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:177)
   at java.io.InputStreamReader.read(InputStreamReader.java:184)
   at java.io.BufferedReader.fill(BufferedReader.java:154)
   at java.io.BufferedReader.read(BufferedReader.java:175)
   at 
 org.apache.commons.net.io.CRLFLineReader.readLine(CRLFLineReader.java:58)
   at org.apache.commons.net.ftp.FTP.__getReply(FTP.java:310)
   at org.apache.commons.net.ftp.FTP.__getReply(FTP.java:290)
   at org.apache.commons.net.ftp.FTP.sendCommand(FTP.java:479)
   at org.apache.commons.net.ftp.FTP.sendCommand(FTP.java:552)
   at org.apache.commons.net.ftp.FTP.sendCommand(FTP.java:601)
   at org.apache.commons.net.ftp.FTP.quit(FTP.java:809)
   at org.apache.commons.net.ftp.FTPClient.logout(FTPClient.java:979)
   at 
 org.apache.hadoop.fs.ftp.FTPFileSystem.disconnect(FTPFileSystem.java:151)
   at 
 org.apache.hadoop.fs.ftp.FTPFileSystem.getFileStatus(FTPFileSystem.java:395)
   at org.apache.hadoop.fs.FileSystem.isFile(FileSystem.java:1424)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-11552) Allow handoff on the server side for RPC requests

[
https://issues.apache.org/jira/browse/HADOOP-11552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14386266#comment-14386266
]

Hadoop QA commented on HADOOP-11552:

{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12708089/HADOOP-11552.2.txt
against trunk revision 1ed9fb7.

{color:green}+1 @author{color}. The patch does not contain any @author
tags.

{color:green}+1 tests included{color}. The patch appears to include 4 new
or modified test files.

{color:red}-1 javac{color:red}. The patch appears to cause the build to
fail.

Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/6023//console

This message is automatically generated.

Allow handoff on the server side for RPC requests
-

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HADOOP-11772) RPC Invoker relies on static ClientCache which has synchronized(this) blocks


 [ 
https://issues.apache.org/jira/browse/HADOOP-11772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Akira AJISAKA updated HADOOP-11772:
---
Component/s: performance

 RPC Invoker relies on static ClientCache which has synchronized(this) blocks
 

 Key: HADOOP-11772
 URL: https://issues.apache.org/jira/browse/HADOOP-11772
 Project: Hadoop Common
  Issue Type: Sub-task
  Components: ipc, performance
Reporter: Gopal V
Assignee: Akira AJISAKA
 Attachments: HADOOP-11772-001.patch, sync-client-bt.png, 
 sync-client-threads.png


 {code}
   private static ClientCache CLIENTS=new ClientCache();
 ...
 this.client = CLIENTS.getClient(conf, factory);
 {code}
 Meanwhile in ClientCache
 {code}
 public synchronized Client getClient(Configuration conf,
   SocketFactory factory, Class? extends Writable valueClass) {
 ...
Client client = clients.get(factory);
 if (client == null) {
   client = new Client(valueClass, conf, factory);
   clients.put(factory, client);
 } else {
   client.incCount();
 }
 {code}
 All invokers end up calling these methods, resulting in IPC clients choking 
 up.
 !sync-client-threads.png!
 !sync-client-bt.png!



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-11753) TestS3AContractOpen#testOpenReadZeroByteFile fails due to negative range header

2015-03-30 Thread Thomas Demoor (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-11753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14386427#comment-14386427
 ] 

Thomas Demoor commented on HADOOP-11753:


Not sure we should close this. The change you propose seems harmless (the 
result is the same, the entire object is returned) and it makes the code more 
readable (one no longer needs to know the HTTP spec) so I'm OK with it (+ it 
makes your life easier). [~ste...@apache.org] what do you think?

Your other change (HADOOP-11742) is higher risk, we want to be really sure you 
don't break other backends and AWS is the standard against which we can all run 
tests, so some more justification is required.

 TestS3AContractOpen#testOpenReadZeroByteFile fails due to negative range 
 header
 ---

 Key: HADOOP-11753
 URL: https://issues.apache.org/jira/browse/HADOOP-11753
 Project: Hadoop Common
  Issue Type: Bug
  Components: fs/s3
Affects Versions: 3.0.0, 2.7.0
Reporter: Takenori Sato
Assignee: Takenori Sato
 Attachments: HADOOP-11753-branch-2.7.001.patch


 _TestS3AContractOpen#testOpenReadZeroByteFile_ fails as follows.
 {code}
 testOpenReadZeroByteFile(org.apache.hadoop.fs.contract.s3a.TestS3AContractOpen)
   Time elapsed: 3.312 sec   ERROR!
 com.amazonaws.services.s3.model.AmazonS3Exception: Status Code: 416, AWS 
 Service: Amazon S3, AWS Request ID: A58A95E0D36811E4, AWS Error Code: 
 InvalidRange, AWS Error Message: The requested range cannot be satisfied.
   at 
 com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:798)
   at 
 com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:421)
   at 
 com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:232)
   at 
 com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3528)
   at 
 com.amazonaws.services.s3.AmazonS3Client.getObject(AmazonS3Client.java:)
   at 
 org.apache.hadoop.fs.s3a.S3AInputStream.reopen(S3AInputStream.java:91)
   at 
 org.apache.hadoop.fs.s3a.S3AInputStream.openIfNeeded(S3AInputStream.java:62)
   at org.apache.hadoop.fs.s3a.S3AInputStream.read(S3AInputStream.java:127)
   at java.io.FilterInputStream.read(FilterInputStream.java:83)
   at 
 org.apache.hadoop.fs.contract.AbstractContractOpenTest.testOpenReadZeroByteFile(AbstractContractOpenTest.java:66)
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   at 
 sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
   at 
 sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   at java.lang.reflect.Method.invoke(Method.java:606)
   at 
 org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
   at 
 org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
   at 
 org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
   at 
 org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
   at 
 org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
   at 
 org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
   at 
 org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
 {code}
 This is because the header is wrong when calling _S3AInputStream#read_ after 
 _S3AInputStream#open_.
 {code}
 Range: bytes=0--1
 * from 0 to -1
 {code}
 Tested on the latest branch-2.7.
 {quote}
 $ git log
 commit d286673c602524af08935ea132c8afd181b6e2e4
 Author: Jitendra Pandey Jitendra@Jitendra-Pandeys-MacBook-Pro-4.local
 Date:   Tue Mar 24 16:17:06 2015 -0700
 {quote}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-11753) TestS3AContractOpen#testOpenReadZeroByteFile fails due to negative range header

2015-03-30 Thread Takenori Sato (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-11753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14386493#comment-14386493
 ] 

Takenori Sato commented on HADOOP-11753:


Thanks, it makes sense. I will discuss internally.

 TestS3AContractOpen#testOpenReadZeroByteFile fails due to negative range 
 header
 ---

 Key: HADOOP-11753
 URL: https://issues.apache.org/jira/browse/HADOOP-11753
 Project: Hadoop Common
  Issue Type: Bug
  Components: fs/s3
Affects Versions: 3.0.0, 2.7.0
Reporter: Takenori Sato
Assignee: Takenori Sato
 Attachments: HADOOP-11753-branch-2.7.001.patch


 _TestS3AContractOpen#testOpenReadZeroByteFile_ fails as follows.
 {code}
 testOpenReadZeroByteFile(org.apache.hadoop.fs.contract.s3a.TestS3AContractOpen)
   Time elapsed: 3.312 sec   ERROR!
 com.amazonaws.services.s3.model.AmazonS3Exception: Status Code: 416, AWS 
 Service: Amazon S3, AWS Request ID: A58A95E0D36811E4, AWS Error Code: 
 InvalidRange, AWS Error Message: The requested range cannot be satisfied.
   at 
 com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:798)
   at 
 com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:421)
   at 
 com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:232)
   at 
 com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3528)
   at 
 com.amazonaws.services.s3.AmazonS3Client.getObject(AmazonS3Client.java:)
   at 
 org.apache.hadoop.fs.s3a.S3AInputStream.reopen(S3AInputStream.java:91)
   at 
 org.apache.hadoop.fs.s3a.S3AInputStream.openIfNeeded(S3AInputStream.java:62)
   at org.apache.hadoop.fs.s3a.S3AInputStream.read(S3AInputStream.java:127)
   at java.io.FilterInputStream.read(FilterInputStream.java:83)
   at 
 org.apache.hadoop.fs.contract.AbstractContractOpenTest.testOpenReadZeroByteFile(AbstractContractOpenTest.java:66)
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   at 
 sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
   at 
 sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   at java.lang.reflect.Method.invoke(Method.java:606)
   at 
 org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
   at 
 org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
   at 
 org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
   at 
 org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
   at 
 org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
   at 
 org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
   at 
 org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
 {code}
 This is because the header is wrong when calling _S3AInputStream#read_ after 
 _S3AInputStream#open_.
 {code}
 Range: bytes=0--1
 * from 0 to -1
 {code}
 Tested on the latest branch-2.7.
 {quote}
 $ git log
 commit d286673c602524af08935ea132c8afd181b6e2e4
 Author: Jitendra Pandey Jitendra@Jitendra-Pandeys-MacBook-Pro-4.local
 Date:   Tue Mar 24 16:17:06 2015 -0700
 {quote}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-11753) TestS3AContractOpen#testOpenReadZeroByteFile fails due to negative range header


[ 
https://issues.apache.org/jira/browse/HADOOP-11753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14386456#comment-14386456
 ] 

Steve Loughran commented on HADOOP-11753:
-

[~tsato] Can I add one suggestion -that you build Hadoop (ideally branch-2  
trunk) in your local jenkins server and run the S3 module tests against your 
implementation? That way regressions in the Hadoop code will get picked up 
within hours, rather than months.

 TestS3AContractOpen#testOpenReadZeroByteFile fails due to negative range 
 header
 ---

 Key: HADOOP-11753
 URL: https://issues.apache.org/jira/browse/HADOOP-11753
 Project: Hadoop Common
  Issue Type: Bug
  Components: fs/s3
Affects Versions: 3.0.0, 2.7.0
Reporter: Takenori Sato
Assignee: Takenori Sato
 Attachments: HADOOP-11753-branch-2.7.001.patch


 _TestS3AContractOpen#testOpenReadZeroByteFile_ fails as follows.
 {code}
 testOpenReadZeroByteFile(org.apache.hadoop.fs.contract.s3a.TestS3AContractOpen)
   Time elapsed: 3.312 sec   ERROR!
 com.amazonaws.services.s3.model.AmazonS3Exception: Status Code: 416, AWS 
 Service: Amazon S3, AWS Request ID: A58A95E0D36811E4, AWS Error Code: 
 InvalidRange, AWS Error Message: The requested range cannot be satisfied.
   at 
 com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:798)
   at 
 com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:421)
   at 
 com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:232)
   at 
 com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3528)
   at 
 com.amazonaws.services.s3.AmazonS3Client.getObject(AmazonS3Client.java:)
   at 
 org.apache.hadoop.fs.s3a.S3AInputStream.reopen(S3AInputStream.java:91)
   at 
 org.apache.hadoop.fs.s3a.S3AInputStream.openIfNeeded(S3AInputStream.java:62)
   at org.apache.hadoop.fs.s3a.S3AInputStream.read(S3AInputStream.java:127)
   at java.io.FilterInputStream.read(FilterInputStream.java:83)
   at 
 org.apache.hadoop.fs.contract.AbstractContractOpenTest.testOpenReadZeroByteFile(AbstractContractOpenTest.java:66)
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   at 
 sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
   at 
 sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   at java.lang.reflect.Method.invoke(Method.java:606)
   at 
 org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
   at 
 org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
   at 
 org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
   at 
 org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
   at 
 org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
   at 
 org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
   at 
 org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
 {code}
 This is because the header is wrong when calling _S3AInputStream#read_ after 
 _S3AInputStream#open_.
 {code}
 Range: bytes=0--1
 * from 0 to -1
 {code}
 Tested on the latest branch-2.7.
 {quote}
 $ git log
 commit d286673c602524af08935ea132c8afd181b6e2e4
 Author: Jitendra Pandey Jitendra@Jitendra-Pandeys-MacBook-Pro-4.local
 Date:   Tue Mar 24 16:17:06 2015 -0700
 {quote}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HADOOP-11772) RPC Invoker relies on static ClientCache which has synchronized(this) blocks


 [ 
https://issues.apache.org/jira/browse/HADOOP-11772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Akira AJISAKA updated HADOOP-11772:
---
Target Version/s: 2.8.0
  Status: Patch Available  (was: Open)

 RPC Invoker relies on static ClientCache which has synchronized(this) blocks
 

 Key: HADOOP-11772
 URL: https://issues.apache.org/jira/browse/HADOOP-11772
 Project: Hadoop Common
  Issue Type: Sub-task
  Components: ipc
Reporter: Gopal V
Assignee: Akira AJISAKA
 Attachments: HADOOP-11772-001.patch, sync-client-bt.png, 
 sync-client-threads.png


 {code}
   private static ClientCache CLIENTS=new ClientCache();
 ...
 this.client = CLIENTS.getClient(conf, factory);
 {code}
 Meanwhile in ClientCache
 {code}
 public synchronized Client getClient(Configuration conf,
   SocketFactory factory, Class? extends Writable valueClass) {
 ...
Client client = clients.get(factory);
 if (client == null) {
   client = new Client(valueClass, conf, factory);
   clients.put(factory, client);
 } else {
   client.incCount();
 }
 {code}
 All invokers end up calling these methods, resulting in IPC clients choking 
 up.
 !sync-client-threads.png!
 !sync-client-bt.png!



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (HADOOP-11772) RPC Invoker relies on static ClientCache which has synchronized(this) blocks


 [ 
https://issues.apache.org/jira/browse/HADOOP-11772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Akira AJISAKA reassigned HADOOP-11772:
--

Assignee: Akira AJISAKA

 RPC Invoker relies on static ClientCache which has synchronized(this) blocks
 

 Key: HADOOP-11772
 URL: https://issues.apache.org/jira/browse/HADOOP-11772
 Project: Hadoop Common
  Issue Type: Sub-task
  Components: ipc
Reporter: Gopal V
Assignee: Akira AJISAKA
 Attachments: sync-client-bt.png, sync-client-threads.png


 {code}
   private static ClientCache CLIENTS=new ClientCache();
 ...
 this.client = CLIENTS.getClient(conf, factory);
 {code}
 Meanwhile in ClientCache
 {code}
 public synchronized Client getClient(Configuration conf,
   SocketFactory factory, Class? extends Writable valueClass) {
 ...
Client client = clients.get(factory);
 if (client == null) {
   client = new Client(valueClass, conf, factory);
   clients.put(factory, client);
 } else {
   client.incCount();
 }
 {code}
 All invokers end up calling these methods, resulting in IPC clients choking 
 up.
 !sync-client-threads.png!
 !sync-client-bt.png!



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HADOOP-11772) RPC Invoker relies on static ClientCache which has synchronized(this) blocks


 [ 
https://issues.apache.org/jira/browse/HADOOP-11772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Akira AJISAKA updated HADOOP-11772:
---
Attachment: HADOOP-11772-001.patch

Thanks [~gopalv] for the report. Attaching a patch to
* Use ConcurrentHashMap for caching clients
* Remove unnecessarily synchronization

 RPC Invoker relies on static ClientCache which has synchronized(this) blocks
 

 Key: HADOOP-11772
 URL: https://issues.apache.org/jira/browse/HADOOP-11772
 Project: Hadoop Common
  Issue Type: Sub-task
  Components: ipc
Reporter: Gopal V
Assignee: Akira AJISAKA
 Attachments: HADOOP-11772-001.patch, sync-client-bt.png, 
 sync-client-threads.png


 {code}
   private static ClientCache CLIENTS=new ClientCache();
 ...
 this.client = CLIENTS.getClient(conf, factory);
 {code}
 Meanwhile in ClientCache
 {code}
 public synchronized Client getClient(Configuration conf,
   SocketFactory factory, Class? extends Writable valueClass) {
 ...
Client client = clients.get(factory);
 if (client == null) {
   client = new Client(valueClass, conf, factory);
   clients.put(factory, client);
 } else {
   client.incCount();
 }
 {code}
 All invokers end up calling these methods, resulting in IPC clients choking 
 up.
 !sync-client-threads.png!
 !sync-client-bt.png!



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-11753) TestS3AContractOpen#testOpenReadZeroByteFile fails due to negative range header


[ 
https://issues.apache.org/jira/browse/HADOOP-11753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14386454#comment-14386454
 ] 

Steve Loughran commented on HADOOP-11753:
-

It is harmless, and would stop other things getting confused...I think it could 
fit into 2.8 with enough regression testing (which is why I didn't rush to get 
it into the 2.7.0 RC)

 HADOOP-11742 is different, though the test changes would seem something worth 
including —so as to act as a regression test for S3a re-implementations.

Anyone re-implementing the S3 protocols does have to consider it to be 
incompletely specified: how applications use it must be taken as the implicit 
API specification, much as the HDFS team accepts HBase's use of HDFS as the 
reference there. A failure of s3a to work with your implementation should be 
taken as a discovery of a new quirk in the API as much as a defect in the S3 
client.

 TestS3AContractOpen#testOpenReadZeroByteFile fails due to negative range 
 header
 ---

 Key: HADOOP-11753
 URL: https://issues.apache.org/jira/browse/HADOOP-11753
 Project: Hadoop Common
  Issue Type: Bug
  Components: fs/s3
Affects Versions: 3.0.0, 2.7.0
Reporter: Takenori Sato
Assignee: Takenori Sato
 Attachments: HADOOP-11753-branch-2.7.001.patch


 _TestS3AContractOpen#testOpenReadZeroByteFile_ fails as follows.
 {code}
 testOpenReadZeroByteFile(org.apache.hadoop.fs.contract.s3a.TestS3AContractOpen)
   Time elapsed: 3.312 sec   ERROR!
 com.amazonaws.services.s3.model.AmazonS3Exception: Status Code: 416, AWS 
 Service: Amazon S3, AWS Request ID: A58A95E0D36811E4, AWS Error Code: 
 InvalidRange, AWS Error Message: The requested range cannot be satisfied.
   at 
 com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:798)
   at 
 com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:421)
   at 
 com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:232)
   at 
 com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3528)
   at 
 com.amazonaws.services.s3.AmazonS3Client.getObject(AmazonS3Client.java:)
   at 
 org.apache.hadoop.fs.s3a.S3AInputStream.reopen(S3AInputStream.java:91)
   at 
 org.apache.hadoop.fs.s3a.S3AInputStream.openIfNeeded(S3AInputStream.java:62)
   at org.apache.hadoop.fs.s3a.S3AInputStream.read(S3AInputStream.java:127)
   at java.io.FilterInputStream.read(FilterInputStream.java:83)
   at 
 org.apache.hadoop.fs.contract.AbstractContractOpenTest.testOpenReadZeroByteFile(AbstractContractOpenTest.java:66)
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   at 
 sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
   at 
 sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   at java.lang.reflect.Method.invoke(Method.java:606)
   at 
 org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
   at 
 org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
   at 
 org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
   at 
 org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
   at 
 org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
   at 
 org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
   at 
 org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
 {code}
 This is because the header is wrong when calling _S3AInputStream#read_ after 
 _S3AInputStream#open_.
 {code}
 Range: bytes=0--1
 * from 0 to -1
 {code}
 Tested on the latest branch-2.7.
 {quote}
 $ git log
 commit d286673c602524af08935ea132c8afd181b6e2e4
 Author: Jitendra Pandey Jitendra@Jitendra-Pandeys-MacBook-Pro-4.local
 Date:   Tue Mar 24 16:17:06 2015 -0700
 {quote}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-11772) RPC Invoker relies on static ClientCache which has synchronized(this) blocks


[ 
https://issues.apache.org/jira/browse/HADOOP-11772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14386445#comment-14386445
 ] 

Hadoop QA commented on HADOOP-11772:


{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  
http://issues.apache.org/jira/secure/attachment/12708113/HADOOP-11772-001.patch
  against trunk revision 1ed9fb7.

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:red}-1 tests included{color}.  The patch doesn't appear to include 
any new or modified tests.
Please justify why no new tests are needed for this 
patch.
Also please list what manual steps were performed to 
verify this patch.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  There were no new javadoc warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 2.0.3) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 core tests{color}.  The patch passed unit tests in 
hadoop-common-project/hadoop-common.

Test results: 
https://builds.apache.org/job/PreCommit-HADOOP-Build/6024//testReport/
Console output: 
https://builds.apache.org/job/PreCommit-HADOOP-Build/6024//console

This message is automatically generated.

 RPC Invoker relies on static ClientCache which has synchronized(this) blocks
 

 Key: HADOOP-11772
 URL: https://issues.apache.org/jira/browse/HADOOP-11772
 Project: Hadoop Common
  Issue Type: Sub-task
  Components: ipc, performance
Reporter: Gopal V
Assignee: Akira AJISAKA
 Attachments: HADOOP-11772-001.patch, sync-client-bt.png, 
 sync-client-threads.png


 {code}
   private static ClientCache CLIENTS=new ClientCache();
 ...
 this.client = CLIENTS.getClient(conf, factory);
 {code}
 Meanwhile in ClientCache
 {code}
 public synchronized Client getClient(Configuration conf,
   SocketFactory factory, Class? extends Writable valueClass) {
 ...
Client client = clients.get(factory);
 if (client == null) {
   client = new Client(valueClass, conf, factory);
   clients.put(factory, client);
 } else {
   client.incCount();
 }
 {code}
 All invokers end up calling these methods, resulting in IPC clients choking 
 up.
 !sync-client-threads.png!
 !sync-client-bt.png!



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HADOOP-10604) CryptoFileSystem decorator using xAttrs and KeyProvider

2015-03-30 Thread Priyank Rastogi (JIRA)


 [ 
https://issues.apache.org/jira/browse/HADOOP-10604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Priyank Rastogi updated HADOOP-10604:
-
Fix Version/s: (was: fs-encryption (HADOOP-10150 and HDFS-6134))

 CryptoFileSystem decorator using xAttrs and KeyProvider
 ---

 Key: HADOOP-10604
 URL: https://issues.apache.org/jira/browse/HADOOP-10604
 Project: Hadoop Common
  Issue Type: New Feature
  Components: fs
Affects Versions: fs-encryption (HADOOP-10150 and HDFS-6134)
Reporter: Alejandro Abdelnur
Assignee: Yi Liu
 Attachments: HADOOP-10604.1.patch, HADOOP-10604.patch


 A FileSystem implementation that wraps an existing filesystem and provides 
 encryption. It will require the underlying filesystem to support xAttrs. It  
 will use the KeyProvider API to retrieve encryption keys.
 This is mostly the work in the patch HADOOP-10150 minus the crypto streams



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-10300) Allowed deferred sending of call responses

2015-03-30 Thread Yi Liu (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-10300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14386708#comment-14386708
 ] 

Yi Liu commented on HADOOP-10300:
-

{code}
  public void sendResponse() throws IOException {
  int count = responseWaitCount.decrementAndGet();
  assert count = 0 : response has already been sent;
  if (count == 0) {
if (rpcResponse == null) {
  // needed by postponed operations to indicate an exception has
  // occurred.  it's too late to re-encode the response so just
  // drop the connection.  unlikely to occur in practice but in tests
  connection.close();
} else {
  connection.sendResponse(this);
}
  }
}
{code}

In real case, {{rpcResponse}} has value before {{sendResponse}}, so it seems 
{{if (rpcResponse == null)}} will not happen.  Can we remove 
{{connection.close()}} and modify the test which makes this happen?

 Allowed deferred sending of call responses
 --

 Key: HADOOP-10300
 URL: https://issues.apache.org/jira/browse/HADOOP-10300
 Project: Hadoop Common
  Issue Type: Sub-task
  Components: ipc
Affects Versions: 2.0.0-alpha, 3.0.0
Reporter: Daryn Sharp
Assignee: Daryn Sharp
 Attachments: HADOOP-10300.patch, HADOOP-10300.patch


 RPC handlers currently do not return until the RPC call completes and 
 response is sent, or a partially sent response has been queued for the 
 responder.  It would be useful for a proxy method to notify the handler to 
 not yet the send the call's response.
 An potential use case is a namespace handler in the NN might want to return 
 before the edit log is synced so it can service more requests and allow 
 increased batching of edits per sync.  Background syncing could later trigger 
 the sending of the call response to the client.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HADOOP-11597) Factor OSType out from Shell: change in common


 [ 
https://issues.apache.org/jira/browse/HADOOP-11597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Steve Loughran updated HADOOP-11597:

Target Version/s: 2.8.0
  Status: Patch Available  (was: Open)

 Factor OSType out from Shell: change in common
 --

 Key: HADOOP-11597
 URL: https://issues.apache.org/jira/browse/HADOOP-11597
 Project: Hadoop Common
  Issue Type: Sub-task
  Components: util
Affects Versions: 2.7.0
Reporter: Yongjun Zhang
Assignee: Yongjun Zhang
 Attachments: HADOOP-11597.001.patch, HADOOP-11597.002.patch


 This jira is to cover the hadoop-common-project change for HADOOP-11293 
 (Factor OSType out from Shell).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HADOOP-11597) Factor OSType out from Shell: change in common


 [ 
https://issues.apache.org/jira/browse/HADOOP-11597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Steve Loughran updated HADOOP-11597:

Status: Open  (was: Patch Available)

 Factor OSType out from Shell: change in common
 --

 Key: HADOOP-11597
 URL: https://issues.apache.org/jira/browse/HADOOP-11597
 Project: Hadoop Common
  Issue Type: Sub-task
  Components: util
Affects Versions: 2.7.0
Reporter: Yongjun Zhang
Assignee: Yongjun Zhang
 Attachments: HADOOP-11597.001.patch, HADOOP-11597.002.patch


 This jira is to cover the hadoop-common-project change for HADOOP-11293 
 (Factor OSType out from Shell).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HADOOP-11660) Add support for hardware crc on ARM aarch64 architecture


 [ 
https://issues.apache.org/jira/browse/HADOOP-11660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Edward Nevill updated HADOOP-11660:
---
Status: Patch Available  (was: Open)

Revised patch to include requested changes

 Add support for hardware crc on ARM aarch64 architecture
 

 Key: HADOOP-11660
 URL: https://issues.apache.org/jira/browse/HADOOP-11660
 Project: Hadoop Common
  Issue Type: Improvement
  Components: native
Affects Versions: 3.0.0
 Environment: ARM aarch64 development platform
Reporter: Edward Nevill
Assignee: Edward Nevill
Priority: Minor
  Labels: performance
 Attachments: jira-11660.patch

   Original Estimate: 48h
  Remaining Estimate: 48h

 This patch adds support for hardware crc for ARM's new 64 bit architecture
 The patch is completely conditionalized on __aarch64__
 I have only added support for the non pipelined version as I benchmarked the 
 pipelined version on aarch64 and it showed no performance improvement.
 The aarch64 version supports both Castagnoli and Zlib CRCs as both of these 
 are supported on ARM aarch64 hardwre.
 To benchmark this I modified the test_bulk_crc32 test to print out the time 
 taken to CRC a 1MB dataset 1000 times.
 Before:
 CRC 1048576 bytes @ 512 bytes per checksum X 1000 iterations = 2.55
 CRC 1048576 bytes @ 512 bytes per checksum X 1000 iterations = 2.55
 After:
 CRC 1048576 bytes @ 512 bytes per checksum X 1000 iterations = 0.57
 CRC 1048576 bytes @ 512 bytes per checksum X 1000 iterations = 0.57
 So this represents a 5X performance improvement on raw CRC calculation.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-11660) Add support for hardware crc on ARM aarch64 architecture


[ 
https://issues.apache.org/jira/browse/HADOOP-11660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14386859#comment-14386859
 ] 

Edward Nevill commented on HADOOP-11660:


Hi,

I have revised the patch to include the changes requested above. I have also 
updated test_bulk_crc32.c so it prints out the times for 16384 bytes @ 512 
bytes per checksum X 100 iterations for both the Castagnoli and Zlib 
polynomials.

The following are the results I get for x86_64 before and after. I have done 5 
runs of each.

BEFORE

{code}
[ed@mylittlepony hadoop]$ 
./hadoop-common-project/hadoop-common/target/native/test_bulk_crc32
CRC 16384 bytes @ 512 bytes per checksum X 100 iterations = 1.12
CRC 16384 bytes @ 512 bytes per checksum X 100 iterations = 13.8
./hadoop-common-project/hadoop-common/target/native/test_bulk_crc32: SUCCESS.
[ed@mylittlepony hadoop]$ 
./hadoop-common-project/hadoop-common/target/native/test_bulk_crc32
CRC 16384 bytes @ 512 bytes per checksum X 100 iterations = 1.12
CRC 16384 bytes @ 512 bytes per checksum X 100 iterations = 13.84
./hadoop-common-project/hadoop-common/target/native/test_bulk_crc32: SUCCESS.
[ed@mylittlepony hadoop]$ 
./hadoop-common-project/hadoop-common/target/native/test_bulk_crc32
CRC 16384 bytes @ 512 bytes per checksum X 100 iterations = 1.1
CRC 16384 bytes @ 512 bytes per checksum X 100 iterations = 13.85
./hadoop-common-project/hadoop-common/target/native/test_bulk_crc32: SUCCESS.
[ed@mylittlepony hadoop]$ 
./hadoop-common-project/hadoop-common/target/native/test_bulk_crc32
CRC 16384 bytes @ 512 bytes per checksum X 100 iterations = 1.12
CRC 16384 bytes @ 512 bytes per checksum X 100 iterations = 13.94
./hadoop-common-project/hadoop-common/target/native/test_bulk_crc32: SUCCESS.
[ed@mylittlepony hadoop]$ 
./hadoop-common-project/hadoop-common/target/native/test_bulk_crc32
CRC 16384 bytes @ 512 bytes per checksum X 100 iterations = 1.12
CRC 16384 bytes @ 512 bytes per checksum X 100 iterations = 13.81
./hadoop-common-project/hadoop-common/target/native/test_bulk_crc32: SUCCESS.
{code}

AFTER

{code}
[ed@mylittlepony hadoop]$ 
./hadoop-common-project/hadoop-common/target/native/test_bulk_crc32
CRC 16384 bytes @ 512 bytes per checksum X 100 iterations = 1.11
CRC 16384 bytes @ 512 bytes per checksum X 100 iterations = 13.92
./hadoop-common-project/hadoop-common/target/native/test_bulk_crc32: SUCCESS.
[ed@mylittlepony hadoop]$ 
./hadoop-common-project/hadoop-common/target/native/test_bulk_crc32
CRC 16384 bytes @ 512 bytes per checksum X 100 iterations = 1.12
CRC 16384 bytes @ 512 bytes per checksum X 100 iterations = 13.99
./hadoop-common-project/hadoop-common/target/native/test_bulk_crc32: SUCCESS.
[ed@mylittlepony hadoop]$ 
./hadoop-common-project/hadoop-common/target/native/test_bulk_crc32
CRC 16384 bytes @ 512 bytes per checksum X 100 iterations = 1.11
CRC 16384 bytes @ 512 bytes per checksum X 100 iterations = 13.9
./hadoop-common-project/hadoop-common/target/native/test_bulk_crc32: SUCCESS.
[ed@mylittlepony hadoop]$ 
./hadoop-common-project/hadoop-common/target/native/test_bulk_crc32
CRC 16384 bytes @ 512 bytes per checksum X 100 iterations = 1.12
CRC 16384 bytes @ 512 bytes per checksum X 100 iterations = 13.92
./hadoop-common-project/hadoop-common/target/native/test_bulk_crc32: SUCCESS.
[ed@mylittlepony hadoop]$ 
./hadoop-common-project/hadoop-common/target/native/test_bulk_crc32
CRC 16384 bytes @ 512 bytes per checksum X 100 iterations = 1.12
CRC 16384 bytes @ 512 bytes per checksum X 100 iterations = 13.92
./hadoop-common-project/hadoop-common/target/native/test_bulk_crc32: SUCCESS.
{code}

Loking at the average time over 5 runs gives

BEFORE

Castagnoli average = 1.116 sec
Zlib average = 13.848 sec

AFTER

Castagnoli average = 1.116
Zlib average = 13.93

So the performance for the Castagnoli polynomial is the same. For the Zlib 
poynomial there seems to be a performance degradation of 0.6%. This may be due 
to experimental error, however this is unaccelerated in any case on x86 because 
it is not supported on x86 HW and is not used for HDFS.

For comparison, on aarch64 partner HW I get the following averages

Castagnoli = 3.586
Zlib = 3.580

Many thanks for you help with this,
Ed.


 Add support for hardware crc on ARM aarch64 architecture
 

 Key: HADOOP-11660
 URL: https://issues.apache.org/jira/browse/HADOOP-11660
 Project: Hadoop Common
  Issue Type: Improvement
  Components: native
Affects Versions: 3.0.0
 Environment: ARM aarch64 development platform
Reporter: Edward Nevill
Assignee: Edward Nevill
Priority: Minor
  Labels: performance
 Attachments: jira-11660.patch

   Original Estimate: 48h
  Remaining Estimate:

[jira] [Updated] (HADOOP-11660) Add support for hardware crc on ARM aarch64 architecture


 [ 
https://issues.apache.org/jira/browse/HADOOP-11660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Edward Nevill updated HADOOP-11660:
---
Attachment: jira-11660.patch

 Add support for hardware crc on ARM aarch64 architecture
 

 Key: HADOOP-11660
 URL: https://issues.apache.org/jira/browse/HADOOP-11660
 Project: Hadoop Common
  Issue Type: Improvement
  Components: native
Affects Versions: 3.0.0
 Environment: ARM aarch64 development platform
Reporter: Edward Nevill
Assignee: Edward Nevill
Priority: Minor
  Labels: performance
 Attachments: jira-11660.patch

   Original Estimate: 48h
  Remaining Estimate: 48h

 This patch adds support for hardware crc for ARM's new 64 bit architecture
 The patch is completely conditionalized on __aarch64__
 I have only added support for the non pipelined version as I benchmarked the 
 pipelined version on aarch64 and it showed no performance improvement.
 The aarch64 version supports both Castagnoli and Zlib CRCs as both of these 
 are supported on ARM aarch64 hardwre.
 To benchmark this I modified the test_bulk_crc32 test to print out the time 
 taken to CRC a 1MB dataset 1000 times.
 Before:
 CRC 1048576 bytes @ 512 bytes per checksum X 1000 iterations = 2.55
 CRC 1048576 bytes @ 512 bytes per checksum X 1000 iterations = 2.55
 After:
 CRC 1048576 bytes @ 512 bytes per checksum X 1000 iterations = 0.57
 CRC 1048576 bytes @ 512 bytes per checksum X 1000 iterations = 0.57
 So this represents a 5X performance improvement on raw CRC calculation.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HADOOP-11660) Add support for hardware crc on ARM aarch64 architecture


 [ 
https://issues.apache.org/jira/browse/HADOOP-11660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Edward Nevill updated HADOOP-11660:
---
Status: Open  (was: Patch Available)

 Add support for hardware crc on ARM aarch64 architecture
 

 Key: HADOOP-11660
 URL: https://issues.apache.org/jira/browse/HADOOP-11660
 Project: Hadoop Common
  Issue Type: Improvement
  Components: native
Affects Versions: 3.0.0
 Environment: ARM aarch64 development platform
Reporter: Edward Nevill
Assignee: Edward Nevill
Priority: Minor
  Labels: performance
   Original Estimate: 48h
  Remaining Estimate: 48h

 This patch adds support for hardware crc for ARM's new 64 bit architecture
 The patch is completely conditionalized on __aarch64__
 I have only added support for the non pipelined version as I benchmarked the 
 pipelined version on aarch64 and it showed no performance improvement.
 The aarch64 version supports both Castagnoli and Zlib CRCs as both of these 
 are supported on ARM aarch64 hardwre.
 To benchmark this I modified the test_bulk_crc32 test to print out the time 
 taken to CRC a 1MB dataset 1000 times.
 Before:
 CRC 1048576 bytes @ 512 bytes per checksum X 1000 iterations = 2.55
 CRC 1048576 bytes @ 512 bytes per checksum X 1000 iterations = 2.55
 After:
 CRC 1048576 bytes @ 512 bytes per checksum X 1000 iterations = 0.57
 CRC 1048576 bytes @ 512 bytes per checksum X 1000 iterations = 0.57
 So this represents a 5X performance improvement on raw CRC calculation.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HADOOP-11660) Add support for hardware crc on ARM aarch64 architecture


 [ 
https://issues.apache.org/jira/browse/HADOOP-11660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Edward Nevill updated HADOOP-11660:
---
Attachment: (was: jira-11660.patch)

 Add support for hardware crc on ARM aarch64 architecture
 

 Key: HADOOP-11660
 URL: https://issues.apache.org/jira/browse/HADOOP-11660
 Project: Hadoop Common
  Issue Type: Improvement
  Components: native
Affects Versions: 3.0.0
 Environment: ARM aarch64 development platform
Reporter: Edward Nevill
Assignee: Edward Nevill
Priority: Minor
  Labels: performance
   Original Estimate: 48h
  Remaining Estimate: 48h

 This patch adds support for hardware crc for ARM's new 64 bit architecture
 The patch is completely conditionalized on __aarch64__
 I have only added support for the non pipelined version as I benchmarked the 
 pipelined version on aarch64 and it showed no performance improvement.
 The aarch64 version supports both Castagnoli and Zlib CRCs as both of these 
 are supported on ARM aarch64 hardwre.
 To benchmark this I modified the test_bulk_crc32 test to print out the time 
 taken to CRC a 1MB dataset 1000 times.
 Before:
 CRC 1048576 bytes @ 512 bytes per checksum X 1000 iterations = 2.55
 CRC 1048576 bytes @ 512 bytes per checksum X 1000 iterations = 2.55
 After:
 CRC 1048576 bytes @ 512 bytes per checksum X 1000 iterations = 0.57
 CRC 1048576 bytes @ 512 bytes per checksum X 1000 iterations = 0.57
 So this represents a 5X performance improvement on raw CRC calculation.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-11664) Loading predefined EC schemas from configuration

2015-03-30 Thread Zhe Zhang (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-11664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14387024#comment-14387024
 ] 

Zhe Zhang commented on HADOOP-11664:


Thanks Kai for making the additional change. I merged it in the weekly rebase 
so we have a cleaner git history. 

 Loading predefined EC schemas from configuration
 

 Key: HADOOP-11664
 URL: https://issues.apache.org/jira/browse/HADOOP-11664
 Project: Hadoop Common
  Issue Type: Sub-task
Reporter: Kai Zheng
Assignee: Kai Zheng
 Fix For: HDFS-7285

 Attachments: HADOOP-11664-v2.patch, HADOOP-11664-v3.patch, 
 HDFS-7371_v1.patch


 System administrator can configure multiple EC codecs in hdfs-site.xml file, 
 and codec instances or schemas in a new configuration file named 
 ec-schema.xml in the conf folder. A codec can be referenced by its instance 
 or schema using the codec name, and a schema can be utilized and specified by 
 the schema name for a folder or EC ZONE to enforce EC. Once a schema is used 
 to define an EC ZONE, then its associated parameter values will be stored as 
 xattributes and respected thereafter.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-11775) Typos in SwiftNativeFileSystem and SwiftNativeFileSystemStore Javadoc

2015-03-30 Thread Yanjun Wang (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-11775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14387034#comment-14387034
 ] 

Yanjun Wang commented on HADOOP-11775:
--

Thank you for assigning to me, [~airbots]. I will submit the patch later today.

 Typos in SwiftNativeFileSystem and SwiftNativeFileSystemStore Javadoc
 -

 Key: HADOOP-11775
 URL: https://issues.apache.org/jira/browse/HADOOP-11775
 Project: Hadoop Common
  Issue Type: Bug
  Components: documentation
Affects Versions: 2.6.0
Reporter: Chen He
Assignee: Yanjun Wang
Priority: Trivial
  Labels: newbie

   /**
* Initalize the filesystem store -this creates the REST client binding.
*
* @param fsURI URI of the filesystem, which is used to map to the 
 filesystem-specific
*  options in the configuration file
* @param configuration configuration
* @throws IOException on any failure.
*/
   /**
* Low level method to do a deep listing of all entries, not stopping
* at the next directory entry. This is to let tests be confident that
* recursive deletes c really are working.
* @param path path to recurse down
* @param newest ask for the newest data, potentially slower than not.
* @return a potentially empty array of file status
* @throws IOException any problem
*/
   /**
* Low-level operation to also set the block size for this operation
* @param path   the file name to open
* @param bufferSize the size of the buffer to be used.
* @param readBlockSize how big should the read blockk/buffer size be?
* @return the input stream
* @throws FileNotFoundException if the file is not found
* @throws IOException any IO problem
*/



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-11754) RM fails to start in non-secure mode due to authentication filter failure


[ 
https://issues.apache.org/jira/browse/HADOOP-11754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14387111#comment-14387111
 ] 

Haohui Mai commented on HADOOP-11754:
-

The findbugs warnings are fixed in HADOOP-11761.

 RM fails to start in non-secure mode due to authentication filter failure
 -

 Key: HADOOP-11754
 URL: https://issues.apache.org/jira/browse/HADOOP-11754
 Project: Hadoop Common
  Issue Type: Bug
Affects Versions: 2.7.0
Reporter: Sangjin Lee
Assignee: Haohui Mai
Priority: Blocker
 Attachments: HADOOP-11754-v1.patch, HADOOP-11754-v2.patch, 
 HADOOP-11754.000.patch, HADOOP-11754.001.patch, HADOOP-11754.002.patch, 
 HADOOP-11754.003.patch


 RM fails to start in the non-secure mode with the following exception:
 {noformat}
 2015-03-25 22:02:42,526 WARN org.mortbay.log: failed RMAuthenticationFilter: 
 javax.servlet.ServletException: java.lang.RuntimeException: Could not read 
 signature secret file: /Users/sjlee/hadoop-http-auth-signature-secret
 2015-03-25 22:02:42,526 WARN org.mortbay.log: Failed startup of context 
 org.mortbay.jetty.webapp.WebAppContext@6de50b08{/,jar:file:/Users/sjlee/hadoop-3.0.0-SNAPSHOT/share/hadoop/yarn/hadoop-yarn-common-3.0.0-SNAPSHOT.jar!/webapps/cluster}
 javax.servlet.ServletException: java.lang.RuntimeException: Could not read 
 signature secret file: /Users/sjlee/hadoop-http-auth-signature-secret
   at 
 org.apache.hadoop.security.authentication.server.AuthenticationFilter.initializeSecretProvider(AuthenticationFilter.java:266)
   at 
 org.apache.hadoop.security.authentication.server.AuthenticationFilter.init(AuthenticationFilter.java:225)
   at 
 org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticationFilter.init(DelegationTokenAuthenticationFilter.java:161)
   at 
 org.apache.hadoop.yarn.server.security.http.RMAuthenticationFilter.init(RMAuthenticationFilter.java:53)
   at org.mortbay.jetty.servlet.FilterHolder.doStart(FilterHolder.java:97)
   at 
 org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
   at 
 org.mortbay.jetty.servlet.ServletHandler.initialize(ServletHandler.java:713)
   at org.mortbay.jetty.servlet.Context.startContext(Context.java:140)
   at 
 org.mortbay.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1282)
   at 
 org.mortbay.jetty.handler.ContextHandler.doStart(ContextHandler.java:518)
   at 
 org.mortbay.jetty.webapp.WebAppContext.doStart(WebAppContext.java:499)
   at 
 org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
   at 
 org.mortbay.jetty.handler.HandlerCollection.doStart(HandlerCollection.java:152)
   at 
 org.mortbay.jetty.handler.ContextHandlerCollection.doStart(ContextHandlerCollection.java:156)
   at 
 org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
   at 
 org.mortbay.jetty.handler.HandlerWrapper.doStart(HandlerWrapper.java:130)
   at org.mortbay.jetty.Server.doStart(Server.java:224)
   at 
 org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
   at org.apache.hadoop.http.HttpServer2.start(HttpServer2.java:773)
   at org.apache.hadoop.yarn.webapp.WebApps$Builder.start(WebApps.java:274)
   at 
 org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.startWepApp(ResourceManager.java:974)
   at 
 org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceStart(ResourceManager.java:1074)
   at 
 org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
   at 
 org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1208)
 Caused by: java.lang.RuntimeException: Could not read signature secret file: 
 /Users/sjlee/hadoop-http-auth-signature-secret
   at 
 org.apache.hadoop.security.authentication.util.FileSignerSecretProvider.init(FileSignerSecretProvider.java:59)
   at 
 org.apache.hadoop.security.authentication.server.AuthenticationFilter.initializeSecretProvider(AuthenticationFilter.java:264)
   ... 23 more
 ...
 2015-03-25 22:02:42,538 FATAL 
 org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error starting 
 ResourceManager
 org.apache.hadoop.yarn.webapp.WebAppException: Error starting http server
   at org.apache.hadoop.yarn.webapp.WebApps$Builder.start(WebApps.java:279)
   at 
 org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.startWepApp(ResourceManager.java:974)
   at 
 org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceStart(ResourceManager.java:1074)
   at 
 org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
   at

[jira] [Updated] (HADOOP-11552) Allow handoff on the server side for RPC requests

[
https://issues.apache.org/jira/browse/HADOOP-11552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Siddharth Seth updated HADOOP-11552:

Status: Open (was: Patch Available)

Allow handoff on the server side for RPC requests
-

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HADOOP-11552) Allow handoff on the server side for RPC requests

[
https://issues.apache.org/jira/browse/HADOOP-11552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Siddharth Seth updated HADOOP-11552:

Attachment: HADOOP-11552.3.txt

Updated patch to apply cleanly to trunk.

Allow handoff on the server side for RPC requests
-

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-11761) Fix findbugs warnings in org.apache.hadoop.security.authentication


[ 
https://issues.apache.org/jira/browse/HADOOP-11761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14387101#comment-14387101
 ] 

Haohui Mai commented on HADOOP-11761:
-

+1

 Fix findbugs warnings in org.apache.hadoop.security.authentication
 --

 Key: HADOOP-11761
 URL: https://issues.apache.org/jira/browse/HADOOP-11761
 Project: Hadoop Common
  Issue Type: Bug
Reporter: Li Lu
Assignee: Li Lu
Priority: Minor
  Labels: findbugs
 Attachments: HADOOP-11761-032615.patch, HADOOP-11761-032715.patch


 As discovered in HADOOP-11748, we need to fix the findbugs warnings in 
 org.apache.hadoop.security.authentication. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-11757) NFS gateway should shutdown when it can't start UDP or TCP server

2015-03-30 Thread Jing Zhao (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-11757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14387104#comment-14387104
 ] 

Jing Zhao commented on HADOOP-11757:


Thanks for working on this, Brandon. The patch looks good to me. +1

 NFS gateway should shutdown when it can't start UDP or TCP server
 -

 Key: HADOOP-11757
 URL: https://issues.apache.org/jira/browse/HADOOP-11757
 Project: Hadoop Common
  Issue Type: Bug
  Components: nfs
Affects Versions: 2.2.0
Reporter: Brandon Li
Assignee: Brandon Li
 Attachments: HDFS-7989.001.patch, HDFS-7989.002.patch


 Unlike the Portmap, Nfs3 class does shutdown when the service can't start.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HADOOP-11775) Typos in SwiftNativeFileSystem and SwiftNativeFileSystemStore Javadoc


 [ 
https://issues.apache.org/jira/browse/HADOOP-11775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chen He updated HADOOP-11775:
-
Assignee: Yanjun Wang  (was: Chen He)

 Typos in SwiftNativeFileSystem and SwiftNativeFileSystemStore Javadoc
 -

 Key: HADOOP-11775
 URL: https://issues.apache.org/jira/browse/HADOOP-11775
 Project: Hadoop Common
  Issue Type: Bug
  Components: documentation
Affects Versions: 2.6.0
Reporter: Chen He
Assignee: Yanjun Wang
Priority: Trivial
  Labels: newbie

   /**
* Initalize the filesystem store -this creates the REST client binding.
*
* @param fsURI URI of the filesystem, which is used to map to the 
 filesystem-specific
*  options in the configuration file
* @param configuration configuration
* @throws IOException on any failure.
*/
   /**
* Low level method to do a deep listing of all entries, not stopping
* at the next directory entry. This is to let tests be confident that
* recursive deletes c really are working.
* @param path path to recurse down
* @param newest ask for the newest data, potentially slower than not.
* @return a potentially empty array of file status
* @throws IOException any problem
*/
   /**
* Low-level operation to also set the block size for this operation
* @param path   the file name to open
* @param bufferSize the size of the buffer to be used.
* @param readBlockSize how big should the read blockk/buffer size be?
* @return the input stream
* @throws FileNotFoundException if the file is not found
* @throws IOException any IO problem
*/



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Moved] (HADOOP-11775) Typos in SwiftNativeFileSystem and SwiftNativeFileSystemStore Javadoc


 [ 
https://issues.apache.org/jira/browse/HADOOP-11775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chen He moved YARN-3421 to HADOOP-11775:


  Component/s: (was: documentation)
   documentation
Affects Version/s: (was: 2.6.0)
   2.6.0
  Key: HADOOP-11775  (was: YARN-3421)
  Project: Hadoop Common  (was: Hadoop YARN)

 Typos in SwiftNativeFileSystem and SwiftNativeFileSystemStore Javadoc
 -

 Key: HADOOP-11775
 URL: https://issues.apache.org/jira/browse/HADOOP-11775
 Project: Hadoop Common
  Issue Type: Bug
  Components: documentation
Affects Versions: 2.6.0
Reporter: Chen He
Assignee: Chen He
Priority: Trivial
  Labels: newbie

   /**
* Initalize the filesystem store -this creates the REST client binding.
*
* @param fsURI URI of the filesystem, which is used to map to the 
 filesystem-specific
*  options in the configuration file
* @param configuration configuration
* @throws IOException on any failure.
*/
   /**
* Low level method to do a deep listing of all entries, not stopping
* at the next directory entry. This is to let tests be confident that
* recursive deletes c really are working.
* @param path path to recurse down
* @param newest ask for the newest data, potentially slower than not.
* @return a potentially empty array of file status
* @throws IOException any problem
*/
   /**
* Low-level operation to also set the block size for this operation
* @param path   the file name to open
* @param bufferSize the size of the buffer to be used.
* @param readBlockSize how big should the read blockk/buffer size be?
* @return the input stream
* @throws FileNotFoundException if the file is not found
* @throws IOException any IO problem
*/



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HADOOP-11552) Allow handoff on the server side for RPC requests

[
https://issues.apache.org/jira/browse/HADOOP-11552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Siddharth Seth updated HADOOP-11552:

Status: Patch Available (was: Open)

Allow handoff on the server side for RPC requests
-

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HADOOP-11761) Fix findbugs warnings in org.apache.hadoop.security.authentication


 [ 
https://issues.apache.org/jira/browse/HADOOP-11761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Haohui Mai updated HADOOP-11761:

   Resolution: Fixed
Fix Version/s: 2.7.0
 Hadoop Flags: Reviewed
   Status: Resolved  (was: Patch Available)

I've committed the patch to trunk, branch-2 and branch-2.7. Thanks 
[~gtCarrera9] for the contribution.

 Fix findbugs warnings in org.apache.hadoop.security.authentication
 --

 Key: HADOOP-11761
 URL: https://issues.apache.org/jira/browse/HADOOP-11761
 Project: Hadoop Common
  Issue Type: Bug
Reporter: Li Lu
Assignee: Li Lu
Priority: Minor
  Labels: findbugs
 Fix For: 2.7.0

 Attachments: HADOOP-11761-032615.patch, HADOOP-11761-032715.patch


 As discovered in HADOOP-11748, we need to fix the findbugs warnings in 
 org.apache.hadoop.security.authentication. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-11761) Fix findbugs warnings in org.apache.hadoop.security.authentication

2015-03-30 Thread Hudson (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-11761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14387117#comment-14387117
 ] 

Hudson commented on HADOOP-11761:
-

FAILURE: Integrated in Hadoop-trunk-Commit #7464 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/7464/])
HADOOP-11761. Fix findbugs warnings in 
org.apache.hadoop.security.authentication. Contributed by Li Lu. (wheat9: rev 
82fa3adfd889700c05464345bee91f861d78ba4f)
* hadoop-common-project/hadoop-common/CHANGES.txt
* hadoop-common-project/hadoop-auth/dev-support/findbugsExcludeFile.xml


 Fix findbugs warnings in org.apache.hadoop.security.authentication
 --

 Key: HADOOP-11761
 URL: https://issues.apache.org/jira/browse/HADOOP-11761
 Project: Hadoop Common
  Issue Type: Bug
Reporter: Li Lu
Assignee: Li Lu
Priority: Minor
  Labels: findbugs
 Fix For: 2.7.0

 Attachments: HADOOP-11761-032615.patch, HADOOP-11761-032715.patch


 As discovered in HADOOP-11748, we need to fix the findbugs warnings in 
 org.apache.hadoop.security.authentication. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HADOOP-11754) RM fails to start in non-secure mode due to authentication filter failure


 [ 
https://issues.apache.org/jira/browse/HADOOP-11754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Haohui Mai updated HADOOP-11754:

   Resolution: Fixed
Fix Version/s: 2.7.0
 Hadoop Flags: Reviewed
   Status: Resolved  (was: Patch Available)

I've committed the patch to trunk, branch-2 and branch-2.7. Thanks Sangjin for 
reporting the issues, and many others for the reviews.

 RM fails to start in non-secure mode due to authentication filter failure
 -

 Key: HADOOP-11754
 URL: https://issues.apache.org/jira/browse/HADOOP-11754
 Project: Hadoop Common
  Issue Type: Bug
Affects Versions: 2.7.0
Reporter: Sangjin Lee
Assignee: Haohui Mai
Priority: Blocker
 Fix For: 2.7.0

 Attachments: HADOOP-11754-v1.patch, HADOOP-11754-v2.patch, 
 HADOOP-11754.000.patch, HADOOP-11754.001.patch, HADOOP-11754.002.patch, 
 HADOOP-11754.003.patch


 RM fails to start in the non-secure mode with the following exception:
 {noformat}
 2015-03-25 22:02:42,526 WARN org.mortbay.log: failed RMAuthenticationFilter: 
 javax.servlet.ServletException: java.lang.RuntimeException: Could not read 
 signature secret file: /Users/sjlee/hadoop-http-auth-signature-secret
 2015-03-25 22:02:42,526 WARN org.mortbay.log: Failed startup of context 
 org.mortbay.jetty.webapp.WebAppContext@6de50b08{/,jar:file:/Users/sjlee/hadoop-3.0.0-SNAPSHOT/share/hadoop/yarn/hadoop-yarn-common-3.0.0-SNAPSHOT.jar!/webapps/cluster}
 javax.servlet.ServletException: java.lang.RuntimeException: Could not read 
 signature secret file: /Users/sjlee/hadoop-http-auth-signature-secret
   at 
 org.apache.hadoop.security.authentication.server.AuthenticationFilter.initializeSecretProvider(AuthenticationFilter.java:266)
   at 
 org.apache.hadoop.security.authentication.server.AuthenticationFilter.init(AuthenticationFilter.java:225)
   at 
 org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticationFilter.init(DelegationTokenAuthenticationFilter.java:161)
   at 
 org.apache.hadoop.yarn.server.security.http.RMAuthenticationFilter.init(RMAuthenticationFilter.java:53)
   at org.mortbay.jetty.servlet.FilterHolder.doStart(FilterHolder.java:97)
   at 
 org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
   at 
 org.mortbay.jetty.servlet.ServletHandler.initialize(ServletHandler.java:713)
   at org.mortbay.jetty.servlet.Context.startContext(Context.java:140)
   at 
 org.mortbay.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1282)
   at 
 org.mortbay.jetty.handler.ContextHandler.doStart(ContextHandler.java:518)
   at 
 org.mortbay.jetty.webapp.WebAppContext.doStart(WebAppContext.java:499)
   at 
 org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
   at 
 org.mortbay.jetty.handler.HandlerCollection.doStart(HandlerCollection.java:152)
   at 
 org.mortbay.jetty.handler.ContextHandlerCollection.doStart(ContextHandlerCollection.java:156)
   at 
 org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
   at 
 org.mortbay.jetty.handler.HandlerWrapper.doStart(HandlerWrapper.java:130)
   at org.mortbay.jetty.Server.doStart(Server.java:224)
   at 
 org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
   at org.apache.hadoop.http.HttpServer2.start(HttpServer2.java:773)
   at org.apache.hadoop.yarn.webapp.WebApps$Builder.start(WebApps.java:274)
   at 
 org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.startWepApp(ResourceManager.java:974)
   at 
 org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceStart(ResourceManager.java:1074)
   at 
 org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
   at 
 org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1208)
 Caused by: java.lang.RuntimeException: Could not read signature secret file: 
 /Users/sjlee/hadoop-http-auth-signature-secret
   at 
 org.apache.hadoop.security.authentication.util.FileSignerSecretProvider.init(FileSignerSecretProvider.java:59)
   at 
 org.apache.hadoop.security.authentication.server.AuthenticationFilter.initializeSecretProvider(AuthenticationFilter.java:264)
   ... 23 more
 ...
 2015-03-25 22:02:42,538 FATAL 
 org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error starting 
 ResourceManager
 org.apache.hadoop.yarn.webapp.WebAppException: Error starting http server
   at org.apache.hadoop.yarn.webapp.WebApps$Builder.start(WebApps.java:279)
   at 
 org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.startWepApp(ResourceManager.java:974)
   at

[jira] [Commented] (HADOOP-11754) RM fails to start in non-secure mode due to authentication filter failure

2015-03-30 Thread Hudson (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-11754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14387201#comment-14387201
 ] 

Hudson commented on HADOOP-11754:
-

FAILURE: Integrated in Hadoop-trunk-Commit #7465 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/7465/])
HADOOP-11754. RM fails to start in non-secure mode due to authentication filter 
failure. Contributed by Haohui Mai. (wheat9: rev 
90e07d55ace7221081a58a90e54b360ad68fa1ef)
* 
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/AuthenticationFilterInitializer.java
* hadoop-common-project/hadoop-common/CHANGES.txt
* 
hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/server/TestAuthenticationFilter.java
* 
hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/server/AuthenticationFilter.java
* 
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/http/HttpServer2.java


 RM fails to start in non-secure mode due to authentication filter failure
 -

 Key: HADOOP-11754
 URL: https://issues.apache.org/jira/browse/HADOOP-11754
 Project: Hadoop Common
  Issue Type: Bug
Affects Versions: 2.7.0
Reporter: Sangjin Lee
Assignee: Haohui Mai
Priority: Blocker
 Fix For: 2.7.0

 Attachments: HADOOP-11754-v1.patch, HADOOP-11754-v2.patch, 
 HADOOP-11754.000.patch, HADOOP-11754.001.patch, HADOOP-11754.002.patch, 
 HADOOP-11754.003.patch


 RM fails to start in the non-secure mode with the following exception:
 {noformat}
 2015-03-25 22:02:42,526 WARN org.mortbay.log: failed RMAuthenticationFilter: 
 javax.servlet.ServletException: java.lang.RuntimeException: Could not read 
 signature secret file: /Users/sjlee/hadoop-http-auth-signature-secret
 2015-03-25 22:02:42,526 WARN org.mortbay.log: Failed startup of context 
 org.mortbay.jetty.webapp.WebAppContext@6de50b08{/,jar:file:/Users/sjlee/hadoop-3.0.0-SNAPSHOT/share/hadoop/yarn/hadoop-yarn-common-3.0.0-SNAPSHOT.jar!/webapps/cluster}
 javax.servlet.ServletException: java.lang.RuntimeException: Could not read 
 signature secret file: /Users/sjlee/hadoop-http-auth-signature-secret
   at 
 org.apache.hadoop.security.authentication.server.AuthenticationFilter.initializeSecretProvider(AuthenticationFilter.java:266)
   at 
 org.apache.hadoop.security.authentication.server.AuthenticationFilter.init(AuthenticationFilter.java:225)
   at 
 org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticationFilter.init(DelegationTokenAuthenticationFilter.java:161)
   at 
 org.apache.hadoop.yarn.server.security.http.RMAuthenticationFilter.init(RMAuthenticationFilter.java:53)
   at org.mortbay.jetty.servlet.FilterHolder.doStart(FilterHolder.java:97)
   at 
 org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
   at 
 org.mortbay.jetty.servlet.ServletHandler.initialize(ServletHandler.java:713)
   at org.mortbay.jetty.servlet.Context.startContext(Context.java:140)
   at 
 org.mortbay.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1282)
   at 
 org.mortbay.jetty.handler.ContextHandler.doStart(ContextHandler.java:518)
   at 
 org.mortbay.jetty.webapp.WebAppContext.doStart(WebAppContext.java:499)
   at 
 org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
   at 
 org.mortbay.jetty.handler.HandlerCollection.doStart(HandlerCollection.java:152)
   at 
 org.mortbay.jetty.handler.ContextHandlerCollection.doStart(ContextHandlerCollection.java:156)
   at 
 org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
   at 
 org.mortbay.jetty.handler.HandlerWrapper.doStart(HandlerWrapper.java:130)
   at org.mortbay.jetty.Server.doStart(Server.java:224)
   at 
 org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
   at org.apache.hadoop.http.HttpServer2.start(HttpServer2.java:773)
   at org.apache.hadoop.yarn.webapp.WebApps$Builder.start(WebApps.java:274)
   at 
 org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.startWepApp(ResourceManager.java:974)
   at 
 org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceStart(ResourceManager.java:1074)
   at 
 org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
   at 
 org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1208)
 Caused by: java.lang.RuntimeException: Could not read signature secret file: 
 /Users/sjlee/hadoop-http-auth-signature-secret
   at 
 org.apache.hadoop.security.authentication.util.FileSignerSecretProvider.init(FileSignerSecretProvider.java:59)
   at

[jira] [Commented] (HADOOP-11754) RM fails to start in non-secure mode due to authentication filter failure

2015-03-30 Thread Sangjin Lee (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-11754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14387184#comment-14387184
 ] 

Sangjin Lee commented on HADOOP-11754:
--

Thanks [~wheat9] for the fix, and others for the reviews!

 RM fails to start in non-secure mode due to authentication filter failure
 -

 Key: HADOOP-11754
 URL: https://issues.apache.org/jira/browse/HADOOP-11754
 Project: Hadoop Common
  Issue Type: Bug
Affects Versions: 2.7.0
Reporter: Sangjin Lee
Assignee: Haohui Mai
Priority: Blocker
 Fix For: 2.7.0

 Attachments: HADOOP-11754-v1.patch, HADOOP-11754-v2.patch, 
 HADOOP-11754.000.patch, HADOOP-11754.001.patch, HADOOP-11754.002.patch, 
 HADOOP-11754.003.patch


 RM fails to start in the non-secure mode with the following exception:
 {noformat}
 2015-03-25 22:02:42,526 WARN org.mortbay.log: failed RMAuthenticationFilter: 
 javax.servlet.ServletException: java.lang.RuntimeException: Could not read 
 signature secret file: /Users/sjlee/hadoop-http-auth-signature-secret
 2015-03-25 22:02:42,526 WARN org.mortbay.log: Failed startup of context 
 org.mortbay.jetty.webapp.WebAppContext@6de50b08{/,jar:file:/Users/sjlee/hadoop-3.0.0-SNAPSHOT/share/hadoop/yarn/hadoop-yarn-common-3.0.0-SNAPSHOT.jar!/webapps/cluster}
 javax.servlet.ServletException: java.lang.RuntimeException: Could not read 
 signature secret file: /Users/sjlee/hadoop-http-auth-signature-secret
   at 
 org.apache.hadoop.security.authentication.server.AuthenticationFilter.initializeSecretProvider(AuthenticationFilter.java:266)
   at 
 org.apache.hadoop.security.authentication.server.AuthenticationFilter.init(AuthenticationFilter.java:225)
   at 
 org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticationFilter.init(DelegationTokenAuthenticationFilter.java:161)
   at 
 org.apache.hadoop.yarn.server.security.http.RMAuthenticationFilter.init(RMAuthenticationFilter.java:53)
   at org.mortbay.jetty.servlet.FilterHolder.doStart(FilterHolder.java:97)
   at 
 org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
   at 
 org.mortbay.jetty.servlet.ServletHandler.initialize(ServletHandler.java:713)
   at org.mortbay.jetty.servlet.Context.startContext(Context.java:140)
   at 
 org.mortbay.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1282)
   at 
 org.mortbay.jetty.handler.ContextHandler.doStart(ContextHandler.java:518)
   at 
 org.mortbay.jetty.webapp.WebAppContext.doStart(WebAppContext.java:499)
   at 
 org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
   at 
 org.mortbay.jetty.handler.HandlerCollection.doStart(HandlerCollection.java:152)
   at 
 org.mortbay.jetty.handler.ContextHandlerCollection.doStart(ContextHandlerCollection.java:156)
   at 
 org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
   at 
 org.mortbay.jetty.handler.HandlerWrapper.doStart(HandlerWrapper.java:130)
   at org.mortbay.jetty.Server.doStart(Server.java:224)
   at 
 org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
   at org.apache.hadoop.http.HttpServer2.start(HttpServer2.java:773)
   at org.apache.hadoop.yarn.webapp.WebApps$Builder.start(WebApps.java:274)
   at 
 org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.startWepApp(ResourceManager.java:974)
   at 
 org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceStart(ResourceManager.java:1074)
   at 
 org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
   at 
 org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1208)
 Caused by: java.lang.RuntimeException: Could not read signature secret file: 
 /Users/sjlee/hadoop-http-auth-signature-secret
   at 
 org.apache.hadoop.security.authentication.util.FileSignerSecretProvider.init(FileSignerSecretProvider.java:59)
   at 
 org.apache.hadoop.security.authentication.server.AuthenticationFilter.initializeSecretProvider(AuthenticationFilter.java:264)
   ... 23 more
 ...
 2015-03-25 22:02:42,538 FATAL 
 org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error starting 
 ResourceManager
 org.apache.hadoop.yarn.webapp.WebAppException: Error starting http server
   at org.apache.hadoop.yarn.webapp.WebApps$Builder.start(WebApps.java:279)
   at 
 org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.startWepApp(ResourceManager.java:974)
   at 
 org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceStart(ResourceManager.java:1074)
   at 
 org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
   at

[jira] [Commented] (HADOOP-11754) RM fails to start in non-secure mode due to authentication filter failure

2015-03-30 Thread Jing Zhao (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-11754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14387164#comment-14387164
 ] 

Jing Zhao commented on HADOOP-11754:


The latest patch looks good to me. +1

You can fix the format of the following code while committing the patch.
{code}
   try {
-secretProvider.init(config, filterConfig.getServletContext(), 
validity);
+  secretProvider = constructSecretProvider(filterConfig.getServletContext()
+, config, false);
{code}

 RM fails to start in non-secure mode due to authentication filter failure
 -

 Key: HADOOP-11754
 URL: https://issues.apache.org/jira/browse/HADOOP-11754
 Project: Hadoop Common
  Issue Type: Bug
Affects Versions: 2.7.0
Reporter: Sangjin Lee
Assignee: Haohui Mai
Priority: Blocker
 Attachments: HADOOP-11754-v1.patch, HADOOP-11754-v2.patch, 
 HADOOP-11754.000.patch, HADOOP-11754.001.patch, HADOOP-11754.002.patch, 
 HADOOP-11754.003.patch


 RM fails to start in the non-secure mode with the following exception:
 {noformat}
 2015-03-25 22:02:42,526 WARN org.mortbay.log: failed RMAuthenticationFilter: 
 javax.servlet.ServletException: java.lang.RuntimeException: Could not read 
 signature secret file: /Users/sjlee/hadoop-http-auth-signature-secret
 2015-03-25 22:02:42,526 WARN org.mortbay.log: Failed startup of context 
 org.mortbay.jetty.webapp.WebAppContext@6de50b08{/,jar:file:/Users/sjlee/hadoop-3.0.0-SNAPSHOT/share/hadoop/yarn/hadoop-yarn-common-3.0.0-SNAPSHOT.jar!/webapps/cluster}
 javax.servlet.ServletException: java.lang.RuntimeException: Could not read 
 signature secret file: /Users/sjlee/hadoop-http-auth-signature-secret
   at 
 org.apache.hadoop.security.authentication.server.AuthenticationFilter.initializeSecretProvider(AuthenticationFilter.java:266)
   at 
 org.apache.hadoop.security.authentication.server.AuthenticationFilter.init(AuthenticationFilter.java:225)
   at 
 org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticationFilter.init(DelegationTokenAuthenticationFilter.java:161)
   at 
 org.apache.hadoop.yarn.server.security.http.RMAuthenticationFilter.init(RMAuthenticationFilter.java:53)
   at org.mortbay.jetty.servlet.FilterHolder.doStart(FilterHolder.java:97)
   at 
 org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
   at 
 org.mortbay.jetty.servlet.ServletHandler.initialize(ServletHandler.java:713)
   at org.mortbay.jetty.servlet.Context.startContext(Context.java:140)
   at 
 org.mortbay.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1282)
   at 
 org.mortbay.jetty.handler.ContextHandler.doStart(ContextHandler.java:518)
   at 
 org.mortbay.jetty.webapp.WebAppContext.doStart(WebAppContext.java:499)
   at 
 org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
   at 
 org.mortbay.jetty.handler.HandlerCollection.doStart(HandlerCollection.java:152)
   at 
 org.mortbay.jetty.handler.ContextHandlerCollection.doStart(ContextHandlerCollection.java:156)
   at 
 org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
   at 
 org.mortbay.jetty.handler.HandlerWrapper.doStart(HandlerWrapper.java:130)
   at org.mortbay.jetty.Server.doStart(Server.java:224)
   at 
 org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
   at org.apache.hadoop.http.HttpServer2.start(HttpServer2.java:773)
   at org.apache.hadoop.yarn.webapp.WebApps$Builder.start(WebApps.java:274)
   at 
 org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.startWepApp(ResourceManager.java:974)
   at 
 org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceStart(ResourceManager.java:1074)
   at 
 org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
   at 
 org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1208)
 Caused by: java.lang.RuntimeException: Could not read signature secret file: 
 /Users/sjlee/hadoop-http-auth-signature-secret
   at 
 org.apache.hadoop.security.authentication.util.FileSignerSecretProvider.init(FileSignerSecretProvider.java:59)
   at 
 org.apache.hadoop.security.authentication.server.AuthenticationFilter.initializeSecretProvider(AuthenticationFilter.java:264)
   ... 23 more
 ...
 2015-03-25 22:02:42,538 FATAL 
 org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error starting 
 ResourceManager
 org.apache.hadoop.yarn.webapp.WebAppException: Error starting http server
   at org.apache.hadoop.yarn.webapp.WebApps$Builder.start(WebApps.java:279)
   at 
 org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.startWepApp(ResourceManager.java:974)
   at

[jira] [Commented] (HADOOP-11754) RM fails to start in non-secure mode due to authentication filter failure

2015-03-30 Thread Vinod Kumar Vavilapalli (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-11754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14387257#comment-14387257
 ] 

Vinod Kumar Vavilapalli commented on HADOOP-11754:
--

Quick comments
 - Why is disallowFallbackToRandomSecretProvider applicable only for file-based 
signers?
 - And why is the default the file-based signer?

 RM fails to start in non-secure mode due to authentication filter failure
 -

 Key: HADOOP-11754
 URL: https://issues.apache.org/jira/browse/HADOOP-11754
 Project: Hadoop Common
  Issue Type: Bug
Affects Versions: 2.7.0
Reporter: Sangjin Lee
Assignee: Haohui Mai
Priority: Blocker
 Fix For: 2.7.0

 Attachments: HADOOP-11754-v1.patch, HADOOP-11754-v2.patch, 
 HADOOP-11754.000.patch, HADOOP-11754.001.patch, HADOOP-11754.002.patch, 
 HADOOP-11754.003.patch


 RM fails to start in the non-secure mode with the following exception:
 {noformat}
 2015-03-25 22:02:42,526 WARN org.mortbay.log: failed RMAuthenticationFilter: 
 javax.servlet.ServletException: java.lang.RuntimeException: Could not read 
 signature secret file: /Users/sjlee/hadoop-http-auth-signature-secret
 2015-03-25 22:02:42,526 WARN org.mortbay.log: Failed startup of context 
 org.mortbay.jetty.webapp.WebAppContext@6de50b08{/,jar:file:/Users/sjlee/hadoop-3.0.0-SNAPSHOT/share/hadoop/yarn/hadoop-yarn-common-3.0.0-SNAPSHOT.jar!/webapps/cluster}
 javax.servlet.ServletException: java.lang.RuntimeException: Could not read 
 signature secret file: /Users/sjlee/hadoop-http-auth-signature-secret
   at 
 org.apache.hadoop.security.authentication.server.AuthenticationFilter.initializeSecretProvider(AuthenticationFilter.java:266)
   at 
 org.apache.hadoop.security.authentication.server.AuthenticationFilter.init(AuthenticationFilter.java:225)
   at 
 org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticationFilter.init(DelegationTokenAuthenticationFilter.java:161)
   at 
 org.apache.hadoop.yarn.server.security.http.RMAuthenticationFilter.init(RMAuthenticationFilter.java:53)
   at org.mortbay.jetty.servlet.FilterHolder.doStart(FilterHolder.java:97)
   at 
 org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
   at 
 org.mortbay.jetty.servlet.ServletHandler.initialize(ServletHandler.java:713)
   at org.mortbay.jetty.servlet.Context.startContext(Context.java:140)
   at 
 org.mortbay.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1282)
   at 
 org.mortbay.jetty.handler.ContextHandler.doStart(ContextHandler.java:518)
   at 
 org.mortbay.jetty.webapp.WebAppContext.doStart(WebAppContext.java:499)
   at 
 org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
   at 
 org.mortbay.jetty.handler.HandlerCollection.doStart(HandlerCollection.java:152)
   at 
 org.mortbay.jetty.handler.ContextHandlerCollection.doStart(ContextHandlerCollection.java:156)
   at 
 org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
   at 
 org.mortbay.jetty.handler.HandlerWrapper.doStart(HandlerWrapper.java:130)
   at org.mortbay.jetty.Server.doStart(Server.java:224)
   at 
 org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
   at org.apache.hadoop.http.HttpServer2.start(HttpServer2.java:773)
   at org.apache.hadoop.yarn.webapp.WebApps$Builder.start(WebApps.java:274)
   at 
 org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.startWepApp(ResourceManager.java:974)
   at 
 org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceStart(ResourceManager.java:1074)
   at 
 org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
   at 
 org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1208)
 Caused by: java.lang.RuntimeException: Could not read signature secret file: 
 /Users/sjlee/hadoop-http-auth-signature-secret
   at 
 org.apache.hadoop.security.authentication.util.FileSignerSecretProvider.init(FileSignerSecretProvider.java:59)
   at 
 org.apache.hadoop.security.authentication.server.AuthenticationFilter.initializeSecretProvider(AuthenticationFilter.java:264)
   ... 23 more
 ...
 2015-03-25 22:02:42,538 FATAL 
 org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error starting 
 ResourceManager
 org.apache.hadoop.yarn.webapp.WebAppException: Error starting http server
   at org.apache.hadoop.yarn.webapp.WebApps$Builder.start(WebApps.java:279)
   at 
 org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.startWepApp(ResourceManager.java:974)
   at 
 org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceStart(ResourceManager.java:1074)
   at

[jira] [Commented] (HADOOP-11552) Allow handoff on the server side for RPC requests

[
https://issues.apache.org/jira/browse/HADOOP-11552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14387182#comment-14387182
]

Hadoop QA commented on HADOOP-11552:

{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12708198/HADOOP-11552.3.txt
against trunk revision 5358b83.

{color:green}+1 @author{color}. The patch does not contain any @author
tags.

{color:green}+1 tests included{color}. The patch appears to include 4 new
or modified test files.

{color:red}-1 javac{color}. The applied patch generated 1149 javac
compiler warnings (more than the trunk's current 1148 warnings).

{color:green}+1 javadoc{color}. There were no new javadoc warning messages.

{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.

{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.

{color:red}-1 release audit{color}. The applied patch generated 1
release audit warnings.

{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-common-project/hadoop-common:

org.apache.hadoop.ipc.TestCallQueueManager

Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/6026//testReport/
Release audit warnings:
https://builds.apache.org/job/PreCommit-HADOOP-Build/6026//artifact/patchprocess/patchReleaseAuditProblems.txt
Javac warnings:
https://builds.apache.org/job/PreCommit-HADOOP-Build/6026//artifact/patchprocess/diffJavacWarnings.txt
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/6026//console

This message is automatically generated.

Allow handoff on the server side for RPC requests
-

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HADOOP-11736) KMSClientProvider addDelegationToken does not notify callers when Auth failure is due to Proxy User configuration a

2015-03-30 Thread Arun Suresh (JIRA)


 [ 
https://issues.apache.org/jira/browse/HADOOP-11736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Arun Suresh updated HADOOP-11736:
-
Priority: Minor  (was: Major)

 KMSClientProvider addDelegationToken does not notify callers when Auth 
 failure is due to Proxy User configuration a 
 

 Key: HADOOP-11736
 URL: https://issues.apache.org/jira/browse/HADOOP-11736
 Project: Hadoop Common
  Issue Type: Bug
Reporter: Arun Suresh
Assignee: Arun Suresh
Priority: Minor
 Attachments: HADOOP-11736.1.patch, HDFS-7970.1.patch


 When a long running process such as YARN RM tries to create/renew a KMS 
 DelegationToken on behalf of proxy user and if the Proxy user rules are not 
 correctly configured to allow yarn to proxy the required user, then the 
 following is found in the RM logs :
 {noformat}
 Unable to add the application to the delegation token renewer.
 java.io.IOException: java.lang.reflect.UndeclaredThrowableException
 at 
 org.apache.hadoop.crypto.key.kms.KMSClientProvider.addDelegationTokens(KMSClientProvider.java:887)
 at 
 org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider$1.call(LoadBalancingKMSClientProvider.java:132)
 at 
 org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider$1.call(LoadBalancingKMSClientProvider.java:129)
 at 
 org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider.doOp(LoadBalancingKMSClientProvider.java:94)
 at 
 org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider.addDelegationTokens(LoadBalancingKMSClientProvider.java:129)
 at 
 org.apache.hadoop.crypto.key.KeyProviderDelegationTokenExtension.addDelegationTokens(KeyProviderDelegationTokenExtension.java:86)
 ..
 ..
 at 
 org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.authenticate(DelegationTokenAuthenticator.java:127)
 at 
 org.apache.hadoop.security.authentication.client.KerberosAuthenticator.authenticate(KerberosAuthenticator.java:205)
 at 
 org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.authenticate(DelegationTokenAuthenticator.java:127)
 at 
 org.apache.hadoop.security.authentication.client.AuthenticatedURL.openConnection(AuthenticatedURL.java:216)
 at 
 org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.doDelegationTokenOperation(DelegationTokenAuthenticator.java:284)
 at 
 org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.getDelegationToken(DelegationTokenAuthenticator.java:165)
 at 
 org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticatedURL.getDelegationToken(DelegationTokenAuthenticatedURL.java:371)
 at 
 org.apache.hadoop.crypto.key.kms.KMSClientProvider$2.run(KMSClientProvider.java:874)
 at 
 org.apache.hadoop.crypto.key.kms.KMSClientProvider$2.run(KMSClientProvider.java:869)
 at java.security.AccessController.doPrivileged(Native Method)
 at javax.security.auth.Subject.doAs(Subject.java:422)
 at 
 org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
 ... 21 more
 {noformat}
 This gives no information to the user as to why the call has failed, and 
 there is generally no way for an admin to know the the ProxyUser setting is 
 the issue without going thru the code.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-11731) Rework the changelog and releasenotes


[ 
https://issues.apache.org/jira/browse/HADOOP-11731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14387339#comment-14387339
 ] 

Colin Patrick McCabe commented on HADOOP-11731:
---

Thanks, Allen.  This looks good.

bq. Nope, otherwise releasedocmaker.py --version trunk-win would fail.

OK.  I guess we can revisit this later if it becomes an issue.  Do you think it 
would be helpful to have a log message about the versions that can't be parsed?

Did you confirm that the CHANGES.txt generated for 2.7 is the same as the 
current one in branch-2.7?  +1 once that is confirmed

We may have to fiddle with some things in JIRA if anything was incorrectly 
resolved (with the wrong version or something)

 Rework the changelog and releasenotes
 -

 Key: HADOOP-11731
 URL: https://issues.apache.org/jira/browse/HADOOP-11731
 Project: Hadoop Common
  Issue Type: New Feature
  Components: documentation
Affects Versions: 3.0.0
Reporter: Allen Wittenauer
 Attachments: HADOOP-11731-00.patch, HADOOP-11731-01.patch, 
 HADOOP-11731-03.patch, HADOOP-11731-04.patch, HADOOP-11731-05.patch


 The current way we generate these build artifacts is awful.  Plus they are 
 ugly and, in the case of release notes, very hard to pick out what is 
 important.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-11552) Allow handoff on the server side for RPC requests

[
https://issues.apache.org/jira/browse/HADOOP-11552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14387397#comment-14387397
]

Colin Patrick McCabe commented on HADOOP-11552:
---

I would like to see some code actually using this before we add it, to make
sure we are getting the APIs right.

Allow handoff on the server side for RPC requests
-

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-11731) Rework the changelog and releasenotes

[
https://issues.apache.org/jira/browse/HADOOP-11731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14387482#comment-14387482
]

Colin Patrick McCabe commented on HADOOP-11731:
---

I agree that recently we haven't done as many point releases as we should (or
as many releases of any kind, for that matter). But we have done them quite
often in the past... see 2.5.1, 2.5.2, 2.4.1. Code to parse point releases
doesn't seem like dead code to me. On the other hand, code to parse
branch-1-win and trunk-win does seem like deadcode since those are no
longer being developed (and really, we never do Apache releases from feature
branches anyway).

I think people would be upset if we removed CHANGES.txt and there were big
discrepancies between the existing CHANGES.txt and what this script generated.
So I guess we will have to hold off on the removal until we can verify that.
But I think committing the script makes sense, so I am +1 on that.

Rework the changelog and releasenotes
-

Key: HADOOP-11731
URL: https://issues.apache.org/jira/browse/HADOOP-11731
Project: Hadoop Common
Issue Type: New Feature
Components: documentation
Affects Versions: 3.0.0
Reporter: Allen Wittenauer
Attachments: HADOOP-11731-00.patch, HADOOP-11731-01.patch,
HADOOP-11731-03.patch, HADOOP-11731-04.patch, HADOOP-11731-05.patch

The current way we generate these build artifacts is awful. Plus they are
ugly and, in the case of release notes, very hard to pick out what is
important.

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HADOOP-11238) Update the NameNode's Group Cache in the background when possible

[
https://issues.apache.org/jira/browse/HADOOP-11238?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Allen Wittenauer updated HADOOP-11238:
--
Fix Version/s: 2.7.0

Update the NameNode's Group Cache in the background when possible
-

Key: HADOOP-11238
URL: https://issues.apache.org/jira/browse/HADOOP-11238
Project: Hadoop Common
Issue Type: Bug
Affects Versions: 2.5.1
Reporter: Chris Li
Assignee: Chris Li
Priority: Minor
Fix For: 2.7.0

Attachments: HADOOP-11238.003.patch, HADOOP-11238.003.patch,
HADOOP-11238.patch, HADOOP-11238.patch

This patch addresses an issue where the namenode pauses during group
resolution by only allowing a single group resolution query on expiry. There
are two scenarios:
1. When there is not yet a value in the cache, all threads which make a
request will block while a single thread fetches the value.
2. When there is already a value in the cache and it is expired, the new
value will be fetched in the background while the old value is used by other
threads
This is handled by guava's cache.
Negative caching is a feature built into the groups cache, and since guava's
caches don't support different expiration times, we have a separate negative
cache which masks the guava cache: if an element exists in the negative cache
and isn't expired, we return it.
In total the logic for fetching a group is:
1. If username exists in static cache, return the value (this was already
present)
2. If username exists in negative cache and negative cache is not expired,
raise an exception as usual
3. Otherwise Defer to guava cache (see two scenarios above)
Original Issue Below:

Our namenode pauses for 12-60 seconds several times every hour. During these
pauses, no new requests can come in.
Around the time of pauses, we have log messages such as:
2014-10-22 13:24:22,688 WARN org.apache.hadoop.security.Groups: Potential
performance problem: getGroups(user=x) took 34507 milliseconds.
The current theory is:
1. Groups has a cache that is refreshed periodically. Each entry has a cache
expiry.
2. When a cache entry expires, multiple threads can see this expiration and
then we have a thundering herd effect where all these threads hit the wire
and overwhelm our LDAP servers (we are using ShellBasedUnixGroupsMapping with
sssd, how this happens has yet to be established)
3. group resolution queries begin to take longer, I've observed it taking 1.2
seconds instead of the usual 0.01-0.03 seconds when measuring in the shell
`time groups myself`
4. If there is mutual exclusion somewhere along this path, a 1 second pause
could lead to a 60 second pause as all the threads compete for the resource.
The exact cause hasn't been established
Potential solutions include:
1. Increasing group cache time, which will make the issue less frequent
2. Rolling evictions of the cache so we prevent the large spike in LDAP
queries
3. Gate the cache refresh so that only one thread is responsible for
refreshing the cache

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-11776) jdiff is broken in Hadoop 2


[ 
https://issues.apache.org/jira/browse/HADOOP-11776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14387571#comment-14387571
 ] 

Li Lu commented on HADOOP-11776:


Hi [~busbey], thanks for the pointer! Will definitely look into that. 

 jdiff is broken in Hadoop 2
 ---

 Key: HADOOP-11776
 URL: https://issues.apache.org/jira/browse/HADOOP-11776
 Project: Hadoop Common
  Issue Type: Bug
Reporter: Li Lu
Assignee: Li Lu
Priority: Blocker

 Seems like we haven't touch the API files from jdiff under dev-support for a 
 while. For now we're missing the jdiff API files for hadoop 2. We're also 
 missing YARN when generating the jdiff API files. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HADOOP-11775) Fix Typos in Swift related Javadoc


 [ 
https://issues.apache.org/jira/browse/HADOOP-11775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chen He updated HADOOP-11775:
-
Labels:   (was: newbie)

 Fix Typos in Swift related Javadoc
 --

 Key: HADOOP-11775
 URL: https://issues.apache.org/jira/browse/HADOOP-11775
 Project: Hadoop Common
  Issue Type: Bug
  Components: documentation
Affects Versions: 2.6.0
Reporter: Chen He
Assignee: Yanjun Wang
Priority: Trivial

   /**
* Initalize the filesystem store -this creates the REST client binding.
*
* @param fsURI URI of the filesystem, which is used to map to the 
 filesystem-specific
*  options in the configuration file
* @param configuration configuration
* @throws IOException on any failure.
*/
   /**
* Low level method to do a deep listing of all entries, not stopping
* at the next directory entry. This is to let tests be confident that
* recursive deletes c really are working.
* @param path path to recurse down
* @param newest ask for the newest data, potentially slower than not.
* @return a potentially empty array of file status
* @throws IOException any problem
*/
   /**
* Low-level operation to also set the block size for this operation
* @param path   the file name to open
* @param bufferSize the size of the buffer to be used.
* @param readBlockSize how big should the read blockk/buffer size be?
* @return the input stream
* @throws FileNotFoundException if the file is not found
* @throws IOException any IO problem
*/



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HADOOP-11775) Fix typos in Swift related Javadoc


 [ 
https://issues.apache.org/jira/browse/HADOOP-11775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chen He updated HADOOP-11775:
-
Summary: Fix typos in Swift related Javadoc  (was: Fix Typos in Swift 
related Javadoc)

 Fix typos in Swift related Javadoc
 --

 Key: HADOOP-11775
 URL: https://issues.apache.org/jira/browse/HADOOP-11775
 Project: Hadoop Common
  Issue Type: Bug
  Components: documentation
Affects Versions: 2.6.0
Reporter: Chen He
Assignee: Yanjun Wang
Priority: Trivial

   /**
* Initalize the filesystem store -this creates the REST client binding.
*
* @param fsURI URI of the filesystem, which is used to map to the 
 filesystem-specific
*  options in the configuration file
* @param configuration configuration
* @throws IOException on any failure.
*/
   /**
* Low level method to do a deep listing of all entries, not stopping
* at the next directory entry. This is to let tests be confident that
* recursive deletes c really are working.
* @param path path to recurse down
* @param newest ask for the newest data, potentially slower than not.
* @return a potentially empty array of file status
* @throws IOException any problem
*/
   /**
* Low-level operation to also set the block size for this operation
* @param path   the file name to open
* @param bufferSize the size of the buffer to be used.
* @param readBlockSize how big should the read blockk/buffer size be?
* @return the input stream
* @throws FileNotFoundException if the file is not found
* @throws IOException any IO problem
*/



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-11552) Allow handoff on the server side for RPC requests

2015-03-30 Thread Sanjay Radia (JIRA)

[
https://issues.apache.org/jira/browse/HADOOP-11552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14387533#comment-14387533
]

Sanjay Radia commented on HADOOP-11552:
---

You mean more than tests?

Allow handoff on the server side for RPC requests
-

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HADOOP-11535) TableMapping related tests failed due to 'successful' resolving of invalid test hostname


 [ 
https://issues.apache.org/jira/browse/HADOOP-11535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Allen Wittenauer updated HADOOP-11535:
--
Fix Version/s: 2.7.0

 TableMapping related tests failed due to 'successful' resolving of invalid 
 test hostname
 

 Key: HADOOP-11535
 URL: https://issues.apache.org/jira/browse/HADOOP-11535
 Project: Hadoop Common
  Issue Type: Test
Reporter: Kai Zheng
Assignee: Kai Zheng
Priority: Minor
 Fix For: 2.7.0

 Attachments: HADOOP-11535-v1.patch


 When mvn test in my environment, it reported the following.
 {noformat}
 Failed tests: 
   TestTableMapping.testClearingCachedMappings:144 expected:/[rack1] but 
 was:/[default-rack]
   TestTableMapping.testTableCaching:79 expected:/[rack1] but 
 was:/[default-rack]
   TestTableMapping.testResolve:56 expected:/[rack1] but 
 was:/[default-rack]
 {noformat}
 It's caused by the good resolving for the 'bad test' hostname 'a.b.c' as 
 follows.
 {noformat}
 [drankye@zkdesk hadoop-common-project]$ ping a.b.c
 PING a.b.c (220.250.64.228) 56(84) bytes of data.
 {noformat}
 I understand it may happen in just my local environment, and document this 
 just in case others also meet this. We may use even worse hostname than 
 'a.b.c' to avoid such situation.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HADOOP-11535) TableMapping related tests failed due to 'successful' resolving of invalid test hostname


 [ 
https://issues.apache.org/jira/browse/HADOOP-11535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Allen Wittenauer updated HADOOP-11535:
--
Resolution: Fixed
Status: Resolved  (was: Patch Available)

 TableMapping related tests failed due to 'successful' resolving of invalid 
 test hostname
 

 Key: HADOOP-11535
 URL: https://issues.apache.org/jira/browse/HADOOP-11535
 Project: Hadoop Common
  Issue Type: Test
Reporter: Kai Zheng
Assignee: Kai Zheng
Priority: Minor
 Fix For: 2.7.0

 Attachments: HADOOP-11535-v1.patch


 When mvn test in my environment, it reported the following.
 {noformat}
 Failed tests: 
   TestTableMapping.testClearingCachedMappings:144 expected:/[rack1] but 
 was:/[default-rack]
   TestTableMapping.testTableCaching:79 expected:/[rack1] but 
 was:/[default-rack]
   TestTableMapping.testResolve:56 expected:/[rack1] but 
 was:/[default-rack]
 {noformat}
 It's caused by the good resolving for the 'bad test' hostname 'a.b.c' as 
 follows.
 {noformat}
 [drankye@zkdesk hadoop-common-project]$ ping a.b.c
 PING a.b.c (220.250.64.228) 56(84) bytes of data.
 {noformat}
 I understand it may happen in just my local environment, and document this 
 just in case others also meet this. We may use even worse hostname than 
 'a.b.c' to avoid such situation.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HADOOP-11450) Cleanup DistCpV1 not to use deprecated methods and fix javadocs


 [ 
https://issues.apache.org/jira/browse/HADOOP-11450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Allen Wittenauer updated HADOOP-11450:
--
Fix Version/s: 2.7.0

 Cleanup DistCpV1 not to use deprecated methods and fix javadocs
 ---

 Key: HADOOP-11450
 URL: https://issues.apache.org/jira/browse/HADOOP-11450
 Project: Hadoop Common
  Issue Type: Bug
  Components: tools/distcp
Affects Versions: 2.6.0
Reporter: Tsuyoshi Ozawa
Assignee: Varun Saxena
Priority: Minor
 Fix For: 2.7.0

 Attachments: HADOOP-11450.001.patch, HADOOP-11450.002.patch, 
 HADOOP-11450.003.patch, HADOOP-11450.004.patch


 Currently, DistCpV1 is using deprecated methods and having wrong javadocs. We 
 should fix them.
 1. DistCpV1.copy doesn't have dstpath, but javadoc has it.
 {code}
 /**
  * Copy a file to a destination.
  * @param srcstat src path and metadata
  * @param dstpath dst path
 {code}
 2. Removing deprecated methods.
 {code}
  SequenceFile.Writer dir_writer = SequenceFile.createWriter(jobfs, 
 jobConf, dstdirlist, Text.class, FilePair.class, 
 SequenceFile.CompressionType.NONE);
 {code}
 {code}
 basedir = args.basedir.makeQualified(basefs);
 {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-11731) Rework the changelog and releasenotes


[ 
https://issues.apache.org/jira/browse/HADOOP-11731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14387575#comment-14387575
 ] 

Allen Wittenauer commented on HADOOP-11731:
---

So, I went through 2.7. The new code doesn't list subtasks, so of course they 
weren't in there.  The other missing bits were mislabeled JIRAs.  
Interestingly, the autocode was much more correct than the HDFS and YARN 
changes.txt files after those are taken into account.  (By quite a bit, 
actually.)

cc: [~vinodkv], just in case he cares about the questionable changes.txt files 
in the 2.7.0 branch.

 Rework the changelog and releasenotes
 -

 Key: HADOOP-11731
 URL: https://issues.apache.org/jira/browse/HADOOP-11731
 Project: Hadoop Common
  Issue Type: New Feature
  Components: documentation
Affects Versions: 3.0.0
Reporter: Allen Wittenauer
 Attachments: HADOOP-11731-00.patch, HADOOP-11731-01.patch, 
 HADOOP-11731-03.patch, HADOOP-11731-04.patch, HADOOP-11731-05.patch


 The current way we generate these build artifacts is awful.  Plus they are 
 ugly and, in the case of release notes, very hard to pick out what is 
 important.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (HADOOP-11731) Rework the changelog and releasenotes

[
https://issues.apache.org/jira/browse/HADOOP-11731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14387519#comment-14387519
]

Allen Wittenauer edited comment on HADOOP-11731 at 3/30/15 10:20 PM:
-

That isn't quite how the current relnotes.py code works. When you provide a
previousVer or let it autodecrement to figure out what the previous release
was, it puts at the top of the file Changes since release [previous version]:
... and that's it. It doesn't do anything fancy like print a separate section
for every release it can calculate.

This code base doesn't even do that. It jettisons the whole since release x
thing. I take the view that our changes files is way too large on a per release
basis vs. what we did pre-1.x. Putting multiple releases in a single file is
too much information at one time.

So if that line is removed and we don't need to know what the previous release
was, there's no reason to even try to parse the version number, except for
maybe sorting purposes. It really is mostly useless code.

(FWIW: This does mean that x.0.0 releases present some special challenges for
both the old and new code. If 2.8.0 is declared dead, then when 3.0.0 is
built, it needs to get passed both 3.0.0 and 2.8.0 to merge all the changes
together. Removing 2.8.0 from JIRA doesn't work because branch-2 does have
those changes committed. The only other way to do this is to set the fix
version to include both, similarly to how one would do it for multiple,
concurrent releases. Luckily, this should only happen a few times, at the
most, and are clearly an edge case.)

Yes, I've spent too much time studying and thinking about these issues over the
past few months. lol

was (Author: aw):
That isn't quite how the current relnotes.py code works. When you provide a
previousVer or let it autodecrement to figure out what the previous release
was, it puts at the top of the file Changes since release [previous version]:
... and that's it. It doesn't do anything fancy like print a separate section
for every release it can calculate.

(FWIW: This does mean that x.0.0 releases present some special challenges for
both the old and new code. If 2.8.0 is declared dead, then when 3.0.0 is
built, it needs to get passed both 3.0.0 and 2.8.0 to merge all the changes
together. Removing 2.8.0 from JIRA doesn't work because branch-2 does have
those changes committed. The only other way to do this is to set the fix
version to conclude both, similarly to how one would do it for multiple,
concurrent releases. Luckily, this should only happen a few times, at the
most, and are clearly an edge case.)

Yes, I've spent too much time studying and thinking about these issues over the
past few months. lol

Rework the changelog and releasenotes
-

The current way we generate these build artifacts is awful. Plus they are
ugly and, in the case of release notes, very hard to pick out what is
important.

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HADOOP-11776) jdiff is broken in Hadoop 2

Li Lu created HADOOP-11776:
--

 Summary: jdiff is broken in Hadoop 2
 Key: HADOOP-11776
 URL: https://issues.apache.org/jira/browse/HADOOP-11776
 Project: Hadoop Common
  Issue Type: Bug
Reporter: Li Lu
Assignee: Li Lu
Priority: Blocker


Seems like we haven't touch the API files from jdiff under dev-support for a 
while. For now we're missing the jdiff API files for hadoop 2. We're also 
missing YARN when generating the jdiff API files. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-11775) Typos in SwiftNativeFileSystem and SwiftNativeFileSystemStore Javadoc


[ 
https://issues.apache.org/jira/browse/HADOOP-11775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14387391#comment-14387391
 ] 

Chen He commented on HADOOP-11775:
--

Per [~qwertymaniac]'s suggestion, I will generalize this ticket for all Swift 
related Javadoc typos.

 Typos in SwiftNativeFileSystem and SwiftNativeFileSystemStore Javadoc
 -

 Key: HADOOP-11775
 URL: https://issues.apache.org/jira/browse/HADOOP-11775
 Project: Hadoop Common
  Issue Type: Bug
  Components: documentation
Affects Versions: 2.6.0
Reporter: Chen He
Assignee: Yanjun Wang
Priority: Trivial
  Labels: newbie

   /**
* Initalize the filesystem store -this creates the REST client binding.
*
* @param fsURI URI of the filesystem, which is used to map to the 
 filesystem-specific
*  options in the configuration file
* @param configuration configuration
* @throws IOException on any failure.
*/
   /**
* Low level method to do a deep listing of all entries, not stopping
* at the next directory entry. This is to let tests be confident that
* recursive deletes c really are working.
* @param path path to recurse down
* @param newest ask for the newest data, potentially slower than not.
* @return a potentially empty array of file status
* @throws IOException any problem
*/
   /**
* Low-level operation to also set the block size for this operation
* @param path   the file name to open
* @param bufferSize the size of the buffer to be used.
* @param readBlockSize how big should the read blockk/buffer size be?
* @return the input stream
* @throws FileNotFoundException if the file is not found
* @throws IOException any IO problem
*/



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HADOOP-11660) Add support for hardware crc of HDFS checksums on ARM aarch64 architecture


 [ 
https://issues.apache.org/jira/browse/HADOOP-11660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Colin Patrick McCabe updated HADOOP-11660:
--
  Resolution: Fixed
   Fix Version/s: 2.8.0
Target Version/s: 3.0.0, 2.8.0  (was: 3.0.0)
  Status: Resolved  (was: Patch Available)

 Add support for hardware crc of HDFS checksums on ARM aarch64 architecture
 --

 Key: HADOOP-11660
 URL: https://issues.apache.org/jira/browse/HADOOP-11660
 Project: Hadoop Common
  Issue Type: Improvement
  Components: native
Affects Versions: 3.0.0
 Environment: ARM aarch64 development platform
Reporter: Edward Nevill
Assignee: Edward Nevill
Priority: Minor
  Labels: performance
 Fix For: 2.8.0

 Attachments: jira-11660.patch

   Original Estimate: 48h
  Remaining Estimate: 48h

 This patch adds support for hardware crc for ARM's new 64 bit architecture
 The patch is completely conditionalized on __aarch64__
 I have only added support for the non pipelined version as I benchmarked the 
 pipelined version on aarch64 and it showed no performance improvement.
 The aarch64 version supports both Castagnoli and Zlib CRCs as both of these 
 are supported on ARM aarch64 hardwre.
 To benchmark this I modified the test_bulk_crc32 test to print out the time 
 taken to CRC a 1MB dataset 1000 times.
 Before:
 CRC 1048576 bytes @ 512 bytes per checksum X 1000 iterations = 2.55
 CRC 1048576 bytes @ 512 bytes per checksum X 1000 iterations = 2.55
 After:
 CRC 1048576 bytes @ 512 bytes per checksum X 1000 iterations = 0.57
 CRC 1048576 bytes @ 512 bytes per checksum X 1000 iterations = 0.57
 So this represents a 5X performance improvement on raw CRC calculation.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HADOOP-9805) Refactor RawLocalFileSystem#rename for improved testability.

2015-03-30 Thread Jean-Pierre Matsumoto (JIRA)


 [ 
https://issues.apache.org/jira/browse/HADOOP-9805?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jean-Pierre Matsumoto updated HADOOP-9805:
--
Attachment: HADOOP-9805.002.patch

Patch 002 attached.

Additional condition {{if (Shell.WINDOWS)}} added.

All the class is not thread-safe. I assume it’s better to keep it as is. And 
reliable way to rename with Hadoop 2.7+ is to use {{NativeIO#renameTo(File, 
File)}} as noted by Steve’s comment above.


 Refactor RawLocalFileSystem#rename for improved testability.
 

 Key: HADOOP-9805
 URL: https://issues.apache.org/jira/browse/HADOOP-9805
 Project: Hadoop Common
  Issue Type: Bug
  Components: fs, test
Affects Versions: 3.0.0, 1-win, 1.3.0, 2.1.1-beta
Reporter: Chris Nauroth
Assignee: Jean-Pierre Matsumoto
Priority: Minor
  Labels: newbie
 Attachments: HADOOP-9805.001.patch, HADOOP-9805.002.patch


 {{RawLocalFileSystem#rename}} contains fallback logic to provide POSIX rename 
 behavior on platforms where {{java.io.File#renameTo}} fails.  The method 
 returns early if {{java.io.File#renameTo}} succeeds, so test runs may not 
 cover the fallback logic depending on the platform.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HADOOP-11633) Convert remaining branch-2 .apt.vm files to markdown


 [ 
https://issues.apache.org/jira/browse/HADOOP-11633?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Allen Wittenauer updated HADOOP-11633:
--
Fix Version/s: 2.7.0

 Convert remaining branch-2 .apt.vm files to markdown
 

 Key: HADOOP-11633
 URL: https://issues.apache.org/jira/browse/HADOOP-11633
 Project: Hadoop Common
  Issue Type: Improvement
  Components: documentation
Affects Versions: 2.7.0
Reporter: Colin Patrick McCabe
Assignee: Masatake Iwasaki
 Fix For: 2.7.0

 Attachments: HADOOP-11633.001.patch, HADOOP-11633.002.patch, 
 HADOOP-11633.003.patch


 We should convert the remaining branch-2 .apt.vm files to markdown.
 Excluding the yarn files, which are covered by YARN-3168, we have remaining:
 {code}
 cmccabe@keter:~/hadoop find -name '*.apt.vm'
 ./hadoop-hdfs-project/hadoop-hdfs-httpfs/src/site/apt/index.apt.vm
 ./hadoop-hdfs-project/hadoop-hdfs-httpfs/src/site/apt/UsingHttpTools.apt.vm
 ./hadoop-hdfs-project/hadoop-hdfs-httpfs/src/site/apt/ServerSetup.apt.vm
 ./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/site/apt/EncryptedShuffle.apt.vm
 ./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/site/apt/HadoopStreaming.apt.vm
 ./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/site/apt/MapredAppMasterRest.apt.vm
 ./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/site/apt/MapReduceTutorial.apt.vm
 ./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/site/apt/MapredCommands.apt.vm
 ./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/site/apt/MapReduce_Compatibility_Hadoop1_Hadoop2.apt.vm
 ./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/site/apt/DistributedCacheDeploy.apt.vm
 ./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/site/apt/PluggableShuffleAndPluggableSort.apt.vm
 ./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/site/apt/HistoryServerRest.apt.vm
 ./hadoop-common-project/hadoop-common/src/site/apt/CommandsManual.apt.vm
 ./hadoop-common-project/hadoop-common/src/site/apt/ClusterSetup.apt.vm
 ./hadoop-common-project/hadoop-auth/src/site/apt/Configuration.apt.vm
 ./hadoop-common-project/hadoop-auth/src/site/apt/index.apt.vm
 ./hadoop-common-project/hadoop-auth/src/site/apt/BuildingIt.apt.vm
 ./hadoop-common-project/hadoop-auth/src/site/apt/Examples.apt.vm
 ./hadoop-common-project/hadoop-kms/src/site/apt/index.apt.vm
 ./hadoop-tools/hadoop-openstack/src/site/apt/index.apt.vm
 ./hadoop-tools/hadoop-sls/src/site/apt/SchedulerLoadSimulator.apt.vm
 ./hadoop-project/src/site/apt/index.apt.vm
 {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HADOOP-11775) Fix Typos in Swift related Javadoc


 [ 
https://issues.apache.org/jira/browse/HADOOP-11775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chen He updated HADOOP-11775:
-
Summary: Fix Typos in Swift related Javadoc  (was: Typos in 
SwiftNativeFileSystem and SwiftNativeFileSystemStore Javadoc)

 Fix Typos in Swift related Javadoc
 --

 Key: HADOOP-11775
 URL: https://issues.apache.org/jira/browse/HADOOP-11775
 Project: Hadoop Common
  Issue Type: Bug
  Components: documentation
Affects Versions: 2.6.0
Reporter: Chen He
Assignee: Yanjun Wang
Priority: Trivial
  Labels: newbie

   /**
* Initalize the filesystem store -this creates the REST client binding.
*
* @param fsURI URI of the filesystem, which is used to map to the 
 filesystem-specific
*  options in the configuration file
* @param configuration configuration
* @throws IOException on any failure.
*/
   /**
* Low level method to do a deep listing of all entries, not stopping
* at the next directory entry. This is to let tests be confident that
* recursive deletes c really are working.
* @param path path to recurse down
* @param newest ask for the newest data, potentially slower than not.
* @return a potentially empty array of file status
* @throws IOException any problem
*/
   /**
* Low-level operation to also set the block size for this operation
* @param path   the file name to open
* @param bufferSize the size of the buffer to be used.
* @param readBlockSize how big should the read blockk/buffer size be?
* @return the input stream
* @throws FileNotFoundException if the file is not found
* @throws IOException any IO problem
*/



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HADOOP-11660) Add support for hardware crc of HDFS checksums on ARM aarch64 architecture


 [ 
https://issues.apache.org/jira/browse/HADOOP-11660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Colin Patrick McCabe updated HADOOP-11660:
--
 Target Version/s: 2.8.0  (was: 3.0.0, 2.8.0)
Affects Version/s: (was: 3.0.0)
   2.8.0

 Add support for hardware crc of HDFS checksums on ARM aarch64 architecture
 --

 Key: HADOOP-11660
 URL: https://issues.apache.org/jira/browse/HADOOP-11660
 Project: Hadoop Common
  Issue Type: Improvement
  Components: native
Affects Versions: 2.8.0
 Environment: ARM aarch64 development platform
Reporter: Edward Nevill
Assignee: Edward Nevill
Priority: Minor
  Labels: performance
 Fix For: 2.8.0

 Attachments: jira-11660.patch

   Original Estimate: 48h
  Remaining Estimate: 48h

 This patch adds support for hardware crc for ARM's new 64 bit architecture
 The patch is completely conditionalized on __aarch64__
 I have only added support for the non pipelined version as I benchmarked the 
 pipelined version on aarch64 and it showed no performance improvement.
 The aarch64 version supports both Castagnoli and Zlib CRCs as both of these 
 are supported on ARM aarch64 hardwre.
 To benchmark this I modified the test_bulk_crc32 test to print out the time 
 taken to CRC a 1MB dataset 1000 times.
 Before:
 CRC 1048576 bytes @ 512 bytes per checksum X 1000 iterations = 2.55
 CRC 1048576 bytes @ 512 bytes per checksum X 1000 iterations = 2.55
 After:
 CRC 1048576 bytes @ 512 bytes per checksum X 1000 iterations = 0.57
 CRC 1048576 bytes @ 512 bytes per checksum X 1000 iterations = 0.57
 So this represents a 5X performance improvement on raw CRC calculation.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-11731) Rework the changelog and releasenotes

[
https://issues.apache.org/jira/browse/HADOOP-11731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14387444#comment-14387444
]

Allen Wittenauer commented on HADOOP-11731:
---

bq. Do you think it would be helpful to have a log message about the versions
that can't be parsed?

Not really. In fact, that's mostly dead code now. It was previously used to
try and guess what the previous release was. Needless to say, it was a broken
concept since we rarely do point releases. When I'm less distracted with other
things, maybe I'll remove it.

bq. Did you confirm that the CHANGES.txt generated for 2.7 is the same as the
current one in branch-2.7?

I haven't, mainly because I don't spend any time in branch-2. I'm not sure I'd
trust CHANGES.TXT in any branch really, given the number of mistakes I found in
trunk's file and some of the other changes.txt files I did look at. (3-way
comparison, jira vs. git vs. file, and even then I'm sure I didn't find
everything wrong).

Rework the changelog and releasenotes
-

The current way we generate these build artifacts is awful. Plus they are
ugly and, in the case of release notes, very hard to pick out what is
important.

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-11731) Rework the changelog and releasenotes

[
https://issues.apache.org/jira/browse/HADOOP-11731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14387519#comment-14387519
]

Allen Wittenauer commented on HADOOP-11731:
---

(FWIW: This does mean that x.0.0 releases present some special challenges for
both the old and new code. If 2.8.0 is declared dead, then when 3.0.0 is
built, it needs to get passed both 3.0.0 and 2.8.0 to merge all the changes
together. Removing 2.8.0 from JIRA doesn't work because branch-2 does have
those changes committed. The only other way to do this is to set the fix
version to conclude both, similarly to how one would do it for multiple,
concurrent releases. Luckily, this should only happen a few times, at the
most, and are clearly an edge case.)

Yes, I've spent too much time studying and thinking about these issues over the
past few months. lol

Rework the changelog and releasenotes
-

The current way we generate these build artifacts is awful. Plus they are
ugly and, in the case of release notes, very hard to pick out what is
important.

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HADOOP-11317) Increment SLF4J version to 1.7.10


 [ 
https://issues.apache.org/jira/browse/HADOOP-11317?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Allen Wittenauer updated HADOOP-11317:
--
Fix Version/s: 2.7.0

 Increment SLF4J version to 1.7.10
 -

 Key: HADOOP-11317
 URL: https://issues.apache.org/jira/browse/HADOOP-11317
 Project: Hadoop Common
  Issue Type: Sub-task
  Components: build
Affects Versions: 2.6.0
Reporter: Steve Loughran
Assignee: Tim Robertson
 Fix For: 2.7.0

 Attachments: HADOOP-11317.patch


 YARN-2875 highllights some problems with SLF4J 1.7.5; needs to be pulled up 
 to 1.7.7; 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-11776) jdiff is broken in Hadoop 2

2015-03-30 Thread Sean Busbey (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-11776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14387565#comment-14387565
 ] 

Sean Busbey commented on HADOOP-11776:
--

Rather than working ot update the jdiff interfaces, you could switch to using 
the Java API Compliance Checker. That's what HBase uses, and they were kind 
enough to add support for API definitions via class annotations. So now we 
check compatibility via InterfaceAudience definitions directly. (see [our test 
script|https://github.com/apache/hbase/blob/master/dev-support/check_compatibility.sh]
 for info or HBASE-12808) 

 jdiff is broken in Hadoop 2
 ---

 Key: HADOOP-11776
 URL: https://issues.apache.org/jira/browse/HADOOP-11776
 Project: Hadoop Common
  Issue Type: Bug
Reporter: Li Lu
Assignee: Li Lu
Priority: Blocker

 Seems like we haven't touch the API files from jdiff under dev-support for a 
 while. For now we're missing the jdiff API files for hadoop 2. We're also 
 missing YARN when generating the jdiff API files. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-11731) Rework the changelog and releasenotes


[ 
https://issues.apache.org/jira/browse/HADOOP-11731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14387607#comment-14387607
 ] 

Allen Wittenauer commented on HADOOP-11731:
---

Actually, I compared against the 2.7 changes.txt entries listed in branch-2.  A 
quick manual check shows that people aren't updating branch-2 when they change 
the branch-2.7 changes.txt. Neat!  I'll run another analysis here in a bit.  
But I doubt I'm going to find much different. 

 Rework the changelog and releasenotes
 -

 Key: HADOOP-11731
 URL: https://issues.apache.org/jira/browse/HADOOP-11731
 Project: Hadoop Common
  Issue Type: New Feature
  Components: documentation
Affects Versions: 3.0.0
Reporter: Allen Wittenauer
 Attachments: HADOOP-11731-00.patch, HADOOP-11731-01.patch, 
 HADOOP-11731-03.patch, HADOOP-11731-04.patch, HADOOP-11731-05.patch


 The current way we generate these build artifacts is awful.  Plus they are 
 ugly and, in the case of release notes, very hard to pick out what is 
 important.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HADOOP-11775) Fix typos in Swift related Javadoc


 [ 
https://issues.apache.org/jira/browse/HADOOP-11775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chen He updated HADOOP-11775:
-
Description: 
Some typos are listed below but not limited to:
SwiftNativeFileSystemObject
  /**
   * Initalize the filesystem store -this creates the REST client binding.
   *
   * @param fsURI URI of the filesystem, which is used to map to the 
filesystem-specific
   *  options in the configuration file
   * @param configuration configuration
   * @throws IOException on any failure.
   */
SwiftNativeFileSystem
  /**
   * Low level method to do a deep listing of all entries, not stopping
   * at the next directory entry. This is to let tests be confident that
   * recursive deletes c really are working.
   * @param path path to recurse down
   * @param newest ask for the newest data, potentially slower than not.
   * @return a potentially empty array of file status
   * @throws IOException any problem
   */

  /**
   * Low-level operation to also set the block size for this operation
   * @param path   the file name to open
   * @param bufferSize the size of the buffer to be used.
   * @param readBlockSize how big should the read blockk/buffer size be?
   * @return the input stream
   * @throws FileNotFoundException if the file is not found
   * @throws IOException any IO problem
   */

SwiftRestClient

  /**
   * Converts Swift path to URI to make request.
   * This is public for unit testing
   *
   * @param path path to object
   * @param endpointURI damain url e.g. http://domain.com
   * @return valid URI for object
   */




  was:
  /**
   * Initalize the filesystem store -this creates the REST client binding.
   *
   * @param fsURI URI of the filesystem, which is used to map to the 
filesystem-specific
   *  options in the configuration file
   * @param configuration configuration
   * @throws IOException on any failure.
   */

  /**
   * Low level method to do a deep listing of all entries, not stopping
   * at the next directory entry. This is to let tests be confident that
   * recursive deletes c really are working.
   * @param path path to recurse down
   * @param newest ask for the newest data, potentially slower than not.
   * @return a potentially empty array of file status
   * @throws IOException any problem
   */

  /**
   * Low-level operation to also set the block size for this operation
   * @param path   the file name to open
   * @param bufferSize the size of the buffer to be used.
   * @param readBlockSize how big should the read blockk/buffer size be?
   * @return the input stream
   * @throws FileNotFoundException if the file is not found
   * @throws IOException any IO problem
   */


 Fix typos in Swift related Javadoc
 --

 Key: HADOOP-11775
 URL: https://issues.apache.org/jira/browse/HADOOP-11775
 Project: Hadoop Common
  Issue Type: Bug
  Components: documentation
Affects Versions: 2.6.0
Reporter: Chen He
Assignee: Yanjun Wang
Priority: Trivial

 Some typos are listed below but not limited to:
 SwiftNativeFileSystemObject
   /**
* Initalize the filesystem store -this creates the REST client binding.
*
* @param fsURI URI of the filesystem, which is used to map to the 
 filesystem-specific
*  options in the configuration file
* @param configuration configuration
* @throws IOException on any failure.
*/
 SwiftNativeFileSystem
   /**
* Low level method to do a deep listing of all entries, not stopping
* at the next directory entry. This is to let tests be confident that
* recursive deletes c really are working.
* @param path path to recurse down
* @param newest ask for the newest data, potentially slower than not.
* @return a potentially empty array of file status
* @throws IOException any problem
*/
   /**
* Low-level operation to also set the block size for this operation
* @param path   the file name to open
* @param bufferSize the size of the buffer to be used.
* @param readBlockSize how big should the read blockk/buffer size be?
* @return the input stream
* @throws FileNotFoundException if the file is not found
* @throws IOException any IO problem
*/
 SwiftRestClient
   /**
* Converts Swift path to URI to make request.
* This is public for unit testing
*
* @param path path to object
* @param endpointURI damain url e.g. http://domain.com
* @return valid URI for object
*/



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HADOOP-11775) Fix typos in Swift related Javadoc


 [ 
https://issues.apache.org/jira/browse/HADOOP-11775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chen He updated HADOOP-11775:
-
Description: 
Some typos are listed below but not limited to:
SwiftNativeFileSystemObject.java
  /**
   * Initalize the filesystem store -this creates the REST client binding.
   *
   * @param fsURI URI of the filesystem, which is used to map to the 
filesystem-specific
   *  options in the configuration file
   * @param configuration configuration
   * @throws IOException on any failure.
   */

SwiftNativeFileSystem.java
  /**
   * Low level method to do a deep listing of all entries, not stopping
   * at the next directory entry. This is to let tests be confident that
   * recursive deletes c really are working.
   * @param path path to recurse down
   * @param newest ask for the newest data, potentially slower than not.
   * @return a potentially empty array of file status
   * @throws IOException any problem
   */

  /**
   * Low-level operation to also set the block size for this operation
   * @param path   the file name to open
   * @param bufferSize the size of the buffer to be used.
   * @param readBlockSize how big should the read blockk/buffer size be?
   * @return the input stream
   * @throws FileNotFoundException if the file is not found
   * @throws IOException any IO problem
   */

SwiftRestClient.java

  /**
   * Converts Swift path to URI to make request.
   * This is public for unit testing
   *
   * @param path path to object
   * @param endpointURI damain url e.g. http://domain.com
   * @return valid URI for object
   */




  was:
Some typos are listed below but not limited to:
SwiftNativeFileSystemObject
  /**
   * Initalize the filesystem store -this creates the REST client binding.
   *
   * @param fsURI URI of the filesystem, which is used to map to the 
filesystem-specific
   *  options in the configuration file
   * @param configuration configuration
   * @throws IOException on any failure.
   */
SwiftNativeFileSystem
  /**
   * Low level method to do a deep listing of all entries, not stopping
   * at the next directory entry. This is to let tests be confident that
   * recursive deletes c really are working.
   * @param path path to recurse down
   * @param newest ask for the newest data, potentially slower than not.
   * @return a potentially empty array of file status
   * @throws IOException any problem
   */

  /**
   * Low-level operation to also set the block size for this operation
   * @param path   the file name to open
   * @param bufferSize the size of the buffer to be used.
   * @param readBlockSize how big should the read blockk/buffer size be?
   * @return the input stream
   * @throws FileNotFoundException if the file is not found
   * @throws IOException any IO problem
   */

SwiftRestClient

  /**
   * Converts Swift path to URI to make request.
   * This is public for unit testing
   *
   * @param path path to object
   * @param endpointURI damain url e.g. http://domain.com
   * @return valid URI for object
   */





 Fix typos in Swift related Javadoc
 --

 Key: HADOOP-11775
 URL: https://issues.apache.org/jira/browse/HADOOP-11775
 Project: Hadoop Common
  Issue Type: Bug
  Components: documentation
Affects Versions: 2.6.0
Reporter: Chen He
Assignee: Yanjun Wang
Priority: Trivial

 Some typos are listed below but not limited to:
 SwiftNativeFileSystemObject.java
   /**
* Initalize the filesystem store -this creates the REST client binding.
*
* @param fsURI URI of the filesystem, which is used to map to the 
 filesystem-specific
*  options in the configuration file
* @param configuration configuration
* @throws IOException on any failure.
*/
 SwiftNativeFileSystem.java
   /**
* Low level method to do a deep listing of all entries, not stopping
* at the next directory entry. This is to let tests be confident that
* recursive deletes c really are working.
* @param path path to recurse down
* @param newest ask for the newest data, potentially slower than not.
* @return a potentially empty array of file status
* @throws IOException any problem
*/
   /**
* Low-level operation to also set the block size for this operation
* @param path   the file name to open
* @param bufferSize the size of the buffer to be used.
* @param readBlockSize how big should the read blockk/buffer size be?
* @return the input stream
* @throws FileNotFoundException if the file is not found
* @throws IOException any IO problem
*/
 SwiftRestClient.java
   /**
* Converts Swift path to URI to make request.
* This is public for unit testing
*
* @param path path to object

[jira] [Commented] (HADOOP-9805) Refactor RawLocalFileSystem#rename for improved testability.


[ 
https://issues.apache.org/jira/browse/HADOOP-9805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14387570#comment-14387570
 ] 

Hadoop QA commented on HADOOP-9805:
---

{color:green}+1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12708249/HADOOP-9805.002.patch
  against trunk revision cc0a01c.

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 1 new 
or modified test files.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  There were no new javadoc warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 2.0.3) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 core tests{color}.  The patch passed unit tests in 
hadoop-common-project/hadoop-common.

Test results: 
https://builds.apache.org/job/PreCommit-HADOOP-Build/6027//testReport/
Console output: 
https://builds.apache.org/job/PreCommit-HADOOP-Build/6027//console

This message is automatically generated.

 Refactor RawLocalFileSystem#rename for improved testability.
 

 Key: HADOOP-9805
 URL: https://issues.apache.org/jira/browse/HADOOP-9805
 Project: Hadoop Common
  Issue Type: Bug
  Components: fs, test
Affects Versions: 3.0.0, 1-win, 1.3.0, 2.1.1-beta
Reporter: Chris Nauroth
Assignee: Jean-Pierre Matsumoto
Priority: Minor
  Labels: newbie
 Attachments: HADOOP-9805.001.patch, HADOOP-9805.002.patch


 {{RawLocalFileSystem#rename}} contains fallback logic to provide POSIX rename 
 behavior on platforms where {{java.io.File#renameTo}} fails.  The method 
 returns early if {{java.io.File#renameTo}} succeeds, so test runs may not 
 cover the fallback logic depending on the platform.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-9805) Refactor RawLocalFileSystem#rename for improved testability.


[ 
https://issues.apache.org/jira/browse/HADOOP-9805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14387356#comment-14387356
 ] 

Colin Patrick McCabe commented on HADOOP-9805:
--

{code}
366 // Enforce POSIX rename behavior that a source directory replaces an
367 // existing destination if the destination is an empty directory. 
On most
368 // platforms, this is already handled by the Java API call above. 
Some
369 // platforms (notably Windows) do not provide this behavior, so the 
Java API
370 // call renameTo(dstFile) fails. Delete destination and attempt 
rename
371 // again.
{code}
This code should not be needed on UNIX, right?  This should be inside an {{if 
(Shell.WINDOWS)}} block.

This also contains a small race condition where if the rename fails due to an 
existing file, but that file is deleted by another thread in between the rename 
and the call to {{File#exists}}, we will erroneously return {{false}}.  Of 
course this bug is Windows-specific and we could easily fix it in a follow-on 
JIRA (probably via JNI code).

 Refactor RawLocalFileSystem#rename for improved testability.
 

 Key: HADOOP-9805
 URL: https://issues.apache.org/jira/browse/HADOOP-9805
 Project: Hadoop Common
  Issue Type: Bug
  Components: fs, test
Affects Versions: 3.0.0, 1-win, 1.3.0, 2.1.1-beta
Reporter: Chris Nauroth
Assignee: Jean-Pierre Matsumoto
Priority: Minor
  Labels: newbie
 Attachments: HADOOP-9805.001.patch


 {{RawLocalFileSystem#rename}} contains fallback logic to provide POSIX rename 
 behavior on platforms where {{java.io.File#renameTo}} fails.  The method 
 returns early if {{java.io.File#renameTo}} succeeds, so test runs may not 
 cover the fallback logic depending on the platform.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HADOOP-11660) Add support for hardware crc of HDFS checksums on ARM aarch64 architecture


 [ 
https://issues.apache.org/jira/browse/HADOOP-11660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Colin Patrick McCabe updated HADOOP-11660:
--
Summary: Add support for hardware crc of HDFS checksums on ARM aarch64 
architecture  (was: Add support for hardware crc on ARM aarch64 architecture)

 Add support for hardware crc of HDFS checksums on ARM aarch64 architecture
 --

 Key: HADOOP-11660
 URL: https://issues.apache.org/jira/browse/HADOOP-11660
 Project: Hadoop Common
  Issue Type: Improvement
  Components: native
Affects Versions: 3.0.0
 Environment: ARM aarch64 development platform
Reporter: Edward Nevill
Assignee: Edward Nevill
Priority: Minor
  Labels: performance
 Attachments: jira-11660.patch

   Original Estimate: 48h
  Remaining Estimate: 48h

 This patch adds support for hardware crc for ARM's new 64 bit architecture
 The patch is completely conditionalized on __aarch64__
 I have only added support for the non pipelined version as I benchmarked the 
 pipelined version on aarch64 and it showed no performance improvement.
 The aarch64 version supports both Castagnoli and Zlib CRCs as both of these 
 are supported on ARM aarch64 hardwre.
 To benchmark this I modified the test_bulk_crc32 test to print out the time 
 taken to CRC a 1MB dataset 1000 times.
 Before:
 CRC 1048576 bytes @ 512 bytes per checksum X 1000 iterations = 2.55
 CRC 1048576 bytes @ 512 bytes per checksum X 1000 iterations = 2.55
 After:
 CRC 1048576 bytes @ 512 bytes per checksum X 1000 iterations = 0.57
 CRC 1048576 bytes @ 512 bytes per checksum X 1000 iterations = 0.57
 So this represents a 5X performance improvement on raw CRC calculation.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-11660) Add support for hardware crc of HDFS checksums on ARM aarch64 architecture

[
https://issues.apache.org/jira/browse/HADOOP-11660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14387384#comment-14387384
]

Colin Patrick McCabe commented on HADOOP-11660:
---

+1. Thanks, [~enevill].

Just a quick note for the future, we usually leave around the old patch when
attaching a new patch. Then we have files like jira-11660.001.patch,
jira-11660.002.patch, etc. attached so we can see the evolution of the patch
over time.

Committed to 2.8.

Add support for hardware crc of HDFS checksums on ARM aarch64 architecture
--

Key: HADOOP-11660
URL: https://issues.apache.org/jira/browse/HADOOP-11660
Project: Hadoop Common
Issue Type: Improvement
Components: native
Affects Versions: 3.0.0
Environment: ARM aarch64 development platform
Reporter: Edward Nevill
Assignee: Edward Nevill
Priority: Minor
Labels: performance
Attachments: jira-11660.patch

Original Estimate: 48h
Remaining Estimate: 48h

This patch adds support for hardware crc for ARM's new 64 bit architecture
The patch is completely conditionalized on __aarch64__
I have only added support for the non pipelined version as I benchmarked the
pipelined version on aarch64 and it showed no performance improvement.
The aarch64 version supports both Castagnoli and Zlib CRCs as both of these
are supported on ARM aarch64 hardwre.
To benchmark this I modified the test_bulk_crc32 test to print out the time
taken to CRC a 1MB dataset 1000 times.
Before:
CRC 1048576 bytes @ 512 bytes per checksum X 1000 iterations = 2.55
CRC 1048576 bytes @ 512 bytes per checksum X 1000 iterations = 2.55
After:
CRC 1048576 bytes @ 512 bytes per checksum X 1000 iterations = 0.57
CRC 1048576 bytes @ 512 bytes per checksum X 1000 iterations = 0.57
So this represents a 5X performance improvement on raw CRC calculation.

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-11660) Add support for hardware crc of HDFS checksums on ARM aarch64 architecture

2015-03-30 Thread Hudson (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-11660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14387420#comment-14387420
 ] 

Hudson commented on HADOOP-11660:
-

SUCCESS: Integrated in Hadoop-trunk-Commit #7469 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/7469/])
HADOOP-11660. Add support for hardware crc of HDFS checksums on ARM aarch64 
architecture (Edward Nevill via Colin P. McCabe) (cmccabe: rev 
d9ac5ee2c4dcd4a108ca892af501618caaea450c)
* 
hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/util/bulk_crc32_aarch64.c
* 
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestNativeCrc32.java
* 
hadoop-common-project/hadoop-common/src/main/native/src/test/org/apache/hadoop/util/test_bulk_crc32.c
* 
hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/util/bulk_crc32.c
* hadoop-common-project/hadoop-common/src/CMakeLists.txt
* hadoop-common-project/hadoop-common/CHANGES.txt
* 
hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/util/bulk_crc32_x86.c


 Add support for hardware crc of HDFS checksums on ARM aarch64 architecture
 --

 Key: HADOOP-11660
 URL: https://issues.apache.org/jira/browse/HADOOP-11660
 Project: Hadoop Common
  Issue Type: Improvement
  Components: native
Affects Versions: 2.8.0
 Environment: ARM aarch64 development platform
Reporter: Edward Nevill
Assignee: Edward Nevill
Priority: Minor
  Labels: performance
 Fix For: 2.8.0

 Attachments: jira-11660.patch

   Original Estimate: 48h
  Remaining Estimate: 48h

 This patch adds support for hardware crc for ARM's new 64 bit architecture
 The patch is completely conditionalized on __aarch64__
 I have only added support for the non pipelined version as I benchmarked the 
 pipelined version on aarch64 and it showed no performance improvement.
 The aarch64 version supports both Castagnoli and Zlib CRCs as both of these 
 are supported on ARM aarch64 hardwre.
 To benchmark this I modified the test_bulk_crc32 test to print out the time 
 taken to CRC a 1MB dataset 1000 times.
 Before:
 CRC 1048576 bytes @ 512 bytes per checksum X 1000 iterations = 2.55
 CRC 1048576 bytes @ 512 bytes per checksum X 1000 iterations = 2.55
 After:
 CRC 1048576 bytes @ 512 bytes per checksum X 1000 iterations = 0.57
 CRC 1048576 bytes @ 512 bytes per checksum X 1000 iterations = 0.57
 So this represents a 5X performance improvement on raw CRC calculation.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HADOOP-10542) Potential null pointer dereference in Jets3tFileSystemStore#retrieveBlock()


 [ 
https://issues.apache.org/jira/browse/HADOOP-10542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Allen Wittenauer updated HADOOP-10542:
--
Fix Version/s: 2.7.0

 Potential null pointer dereference in Jets3tFileSystemStore#retrieveBlock()
 ---

 Key: HADOOP-10542
 URL: https://issues.apache.org/jira/browse/HADOOP-10542
 Project: Hadoop Common
  Issue Type: Bug
  Components: fs/s3
Affects Versions: 2.6.0
Reporter: Ted Yu
Assignee: Ted Yu
Priority: Minor
 Fix For: 2.7.0

 Attachments: hadoop-10542-001.patch


 {code}
   in = get(blockToKey(block), byteRangeStart);
   out = new BufferedOutputStream(new FileOutputStream(fileBlock));
   byte[] buf = new byte[bufferSize];
   int numRead;
   while ((numRead = in.read(buf)) = 0) {
 {code}
 get() may return null.
 The while loop dereferences in without null check.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-6857) FsShell should report raw disk usage including replication factor


[ 
https://issues.apache.org/jira/browse/HADOOP-6857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14387557#comment-14387557
 ] 

Allen Wittenauer commented on HADOOP-6857:
--

This is still sitting in the changes.txt file for branch-2 and probably 2.7

 FsShell should report raw disk usage including replication factor
 -

 Key: HADOOP-6857
 URL: https://issues.apache.org/jira/browse/HADOOP-6857
 Project: Hadoop Common
  Issue Type: Improvement
  Components: fs
Reporter: Alex Kozlov
Assignee: Byron Wong
 Fix For: 3.0.0

 Attachments: HADOOP-6857-revert.patch, HADOOP-6857.patch, 
 HADOOP-6857.patch, HADOOP-6857.patch, revert-HADOOP-6857-from-branch-2.patch, 
 show-space-consumed.txt


 Currently FsShell report HDFS usage with hadoop fs -dus path command.  
 Since replication level is per file level, it would be nice to add raw disk 
 usage including the replication factor (maybe hadoop fs -dus -raw path?). 
  This will allow to assess resource usage more accurately.  -- Alex K



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Issue Comment Deleted] (HADOOP-11746) rewrite test-patch.sh


 [ 
https://issues.apache.org/jira/browse/HADOOP-11746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Allen Wittenauer updated HADOOP-11746:
--
Comment: was deleted

(was: -04:
* effectively same as -03, but adds a subtask section (which in some releases 
2x and 3x the changes file!) and removes the decrement version thing)

 rewrite test-patch.sh
 -

 Key: HADOOP-11746
 URL: https://issues.apache.org/jira/browse/HADOOP-11746
 Project: Hadoop Common
  Issue Type: Test
  Components: build, test
Affects Versions: 3.0.0
Reporter: Allen Wittenauer
Assignee: Allen Wittenauer
 Attachments: HADOOP-11746-00.patch, HADOOP-11746-01.patch, 
 HADOOP-11746-02.patch, HADOOP-11746-03.patch


 This code is bad and you should feel bad.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HADOOP-11746) rewrite test-patch.sh


 [ 
https://issues.apache.org/jira/browse/HADOOP-11746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Allen Wittenauer updated HADOOP-11746:
--
Attachment: HADOOP-11746-04.patch

-04:
* effectively same as -03, but adds a subtask section (which in some releases 
2x and 3x the changes file!) and removes the decrement version thing

 rewrite test-patch.sh
 -

 Key: HADOOP-11746
 URL: https://issues.apache.org/jira/browse/HADOOP-11746
 Project: Hadoop Common
  Issue Type: Test
  Components: build, test
Affects Versions: 3.0.0
Reporter: Allen Wittenauer
Assignee: Allen Wittenauer
 Attachments: HADOOP-11746-00.patch, HADOOP-11746-01.patch, 
 HADOOP-11746-02.patch, HADOOP-11746-03.patch


 This code is bad and you should feel bad.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HADOOP-11746) rewrite test-patch.sh


 [ 
https://issues.apache.org/jira/browse/HADOOP-11746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Allen Wittenauer updated HADOOP-11746:
--
Attachment: (was: HADOOP-11746-04.patch)

 rewrite test-patch.sh
 -

 Key: HADOOP-11746
 URL: https://issues.apache.org/jira/browse/HADOOP-11746
 Project: Hadoop Common
  Issue Type: Test
  Components: build, test
Affects Versions: 3.0.0
Reporter: Allen Wittenauer
Assignee: Allen Wittenauer
 Attachments: HADOOP-11746-00.patch, HADOOP-11746-01.patch, 
 HADOOP-11746-02.patch, HADOOP-11746-03.patch


 This code is bad and you should feel bad.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-6857) FsShell should report raw disk usage including replication factor

2015-03-30 Thread Konstantin Shvachko (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-6857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14387626#comment-14387626
 ] 

Konstantin Shvachko commented on HADOOP-6857:
-

Because it was committed to branch-2 and 2.7. As discussed earlier there is bug 
fix, which went everywhere, and the print format change, which was reverted 
from branch 2 by HADOOP-11666.

 FsShell should report raw disk usage including replication factor
 -

 Key: HADOOP-6857
 URL: https://issues.apache.org/jira/browse/HADOOP-6857
 Project: Hadoop Common
  Issue Type: Improvement
  Components: fs
Reporter: Alex Kozlov
Assignee: Byron Wong
 Fix For: 3.0.0

 Attachments: HADOOP-6857-revert.patch, HADOOP-6857.patch, 
 HADOOP-6857.patch, HADOOP-6857.patch, revert-HADOOP-6857-from-branch-2.patch, 
 show-space-consumed.txt


 Currently FsShell report HDFS usage with hadoop fs -dus path command.  
 Since replication level is per file level, it would be nice to add raw disk 
 usage including the replication factor (maybe hadoop fs -dus -raw path?). 
  This will allow to assess resource usage more accurately.  -- Alex K



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-11377) jdiff failing on java 7 and java 8, Null.java not found


[ 
https://issues.apache.org/jira/browse/HADOOP-11377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14387660#comment-14387660
 ] 

Li Lu commented on HADOOP-11377:


Hi [~ozawa]! Seems like the illegal package name problem also blocks us from 
getting jdiff results. Any hints/backgrounds to resolve that? Thanks! 

 jdiff failing on java 7 and java 8, Null.java not found
 -

 Key: HADOOP-11377
 URL: https://issues.apache.org/jira/browse/HADOOP-11377
 Project: Hadoop Common
  Issue Type: Sub-task
  Components: build
Affects Versions: 2.6.0, 2.7.0
 Environment: Java8 jenkins
Reporter: Steve Loughran
Assignee: Tsuyoshi Ozawa
 Attachments: HADOOP-11377.001.patch


 Jdiff is having problems on Java 8, as it cannot find a javadoc for the new 
 {{Null}} datatype
 {code}
 'https://builds.apache.org/job/Hadoop-common-trunk-Java8/ws/hadoop-common-project/hadoop-common/dev-support/jdiff/Null.java'
 The ' characters around the executable and arguments are
 not part of the command.
   [javadoc] javadoc: error - Illegal package name: 
   [javadoc] javadoc: error - File not found: 
 https://builds.apache.org/job/Hadoop-common-trunk-Java8/ws/hadoop-common-project/hadoop-common/dev-support/jdiff/Null.java;
 {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HADOOP-11778) Update checkstyle


 [ 
https://issues.apache.org/jira/browse/HADOOP-11778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Allen Wittenauer updated HADOOP-11778:
--
Attachment: HADOOP-11778.000.patch

-00:
* initial version

 Update checkstyle
 -

 Key: HADOOP-11778
 URL: https://issues.apache.org/jira/browse/HADOOP-11778
 Project: Hadoop Common
  Issue Type: Improvement
  Components: build, test
Affects Versions: 3.0.0
Reporter: Allen Wittenauer
 Attachments: HADOOP-11778.000.patch


 We should update checkstyle so that mvn checkstyle:checkstyle-aggregate 
 doesn't NPE.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HADOOP-11777) `hadoop classpath` behaviour different from previous releases

2015-03-30 Thread Swapnil Daingade (JIRA)

Swapnil Daingade created HADOOP-11777:
-

 Summary: `hadoop classpath` behaviour different from previous 
releases
 Key: HADOOP-11777
 URL: https://issues.apache.org/jira/browse/HADOOP-11777
 Project: Hadoop Common
  Issue Type: Bug
Affects Versions: 2.6.0
Reporter: Swapnil Daingade
Assignee: Swapnil Daingade


In 2.6 doing a `hadoop classpath` returns the following

classpath [--glob|--jar path|-h|--help] :
  Prints the classpath needed to get the Hadoop jar and the required
  libraries.
  Options:

  --glob   expand wildcards
  --jar path write classpath as manifest in jar named path
  -h, --help   print help

This is different from earlier releases.
However, passing any argument to the command will return the classpath as 
earlier
E.g. `hadoop classpath foo`

It would be nice to preserve the original behaviour for scripts that read the 
classpath using this command.

The fix seems to be a one line change in
hadoop-common-project/hadoop-common/src/main/bin/hadoop





--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-11758) Add options to filter out too much granular tracing spans


[ 
https://issues.apache.org/jira/browse/HADOOP-11758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14387823#comment-14387823
 ] 

Colin Patrick McCabe commented on HADOOP-11758:
---

Check out HDFS-8026... does this help reduce the number of spans in the write 
path?

 Add options to filter out too much granular tracing spans
 -

 Key: HADOOP-11758
 URL: https://issues.apache.org/jira/browse/HADOOP-11758
 Project: Hadoop Common
  Issue Type: Improvement
  Components: tracing
Reporter: Masatake Iwasaki
Assignee: Masatake Iwasaki
Priority: Minor
 Attachments: testWriteTraceHooks.html


 in order to avoid queue in span receiver spills



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-11746) rewrite test-patch.sh


[ 
https://issues.apache.org/jira/browse/HADOOP-11746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14387900#comment-14387900
 ] 

Allen Wittenauer commented on HADOOP-11746:
---

Setting HADOOP-11778 as required for checkstyle support, based upon my current 
experiments.

 rewrite test-patch.sh
 -

 Key: HADOOP-11746
 URL: https://issues.apache.org/jira/browse/HADOOP-11746
 Project: Hadoop Common
  Issue Type: Test
  Components: build, test
Affects Versions: 3.0.0
Reporter: Allen Wittenauer
Assignee: Allen Wittenauer
 Attachments: HADOOP-11746-00.patch, HADOOP-11746-01.patch, 
 HADOOP-11746-02.patch, HADOOP-11746-03.patch


 This code is bad and you should feel bad.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-11731) Rework the changelog and releasenotes


[ 
https://issues.apache.org/jira/browse/HADOOP-11731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14387843#comment-14387843
 ] 

Allen Wittenauer commented on HADOOP-11731:
---

If I turn on subtasks, then the missing JIRAs from branch-2.7 vs. the generated 
report is only HADOOP-6857 (and if you read the JIRA, you can see why).  
Meanwhile, branch-2.7's changes.txt is missing 259 JIRAs, again including 
subtasks.

So yea, this is pretty much what I expected: the changes.txt file is completely 
out of sync with JIRA, with JIRA generally being more accurate.

 Rework the changelog and releasenotes
 -

 Key: HADOOP-11731
 URL: https://issues.apache.org/jira/browse/HADOOP-11731
 Project: Hadoop Common
  Issue Type: New Feature
  Components: documentation
Affects Versions: 3.0.0
Reporter: Allen Wittenauer
 Attachments: HADOOP-11731-00.patch, HADOOP-11731-01.patch, 
 HADOOP-11731-03.patch, HADOOP-11731-04.patch, HADOOP-11731-05.patch


 The current way we generate these build artifacts is awful.  Plus they are 
 ugly and, in the case of release notes, very hard to pick out what is 
 important.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (HADOOP-11777) `hadoop classpath` behaviour different from previous releases

2015-03-30 Thread Swapnil Daingade (JIRA)


 [ 
https://issues.apache.org/jira/browse/HADOOP-11777?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Swapnil Daingade resolved HADOOP-11777.
---
   Resolution: Duplicate
Fix Version/s: 2.7.0

Fixed as part of HDFS-7591

 `hadoop classpath` behaviour different from previous releases
 -

 Key: HADOOP-11777
 URL: https://issues.apache.org/jira/browse/HADOOP-11777
 Project: Hadoop Common
  Issue Type: Bug
Affects Versions: 2.6.0
Reporter: Swapnil Daingade
Assignee: Swapnil Daingade
 Fix For: 2.7.0


 In 2.6 doing a `hadoop classpath` returns the following
 classpath [--glob|--jar path|-h|--help] :
   Prints the classpath needed to get the Hadoop jar and the required
   libraries.
   Options:
   --glob   expand wildcards
   --jar path write classpath as manifest in jar named path
   -h, --help   print help
 This is different from earlier releases.
 However, passing any argument to the command will return the classpath as 
 earlier
 E.g. `hadoop classpath foo`
 It would be nice to preserve the original behaviour for scripts that read the 
 classpath using this command.
 The fix seems to be a one line change in
 hadoop-common-project/hadoop-common/src/main/bin/hadoop



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-11500) InputStream is left unclosed in ApplicationClassLoader

2015-03-30 Thread mai shurong (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-11500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14387881#comment-14387881
 ] 

mai shurong commented on HADOOP-11500:
--

The code in the patch is not supported by java6.
After I put the patch to hadoop-2.6.0 src , the build with maven fail. 
I found reason is the code  in the patch is not supported by java6. 
To solve the problem, firstly JAVA_HOME should be set to java7 home; secondly 
modify maven-compiler-plugin to 1.7 in POM file hadoop-project/pom.xml as 
follows:

--- a/hadoop-project/pom.xml 2014-11-14 05:09:25.0 +0800
+++ b/hadoop-project/pom.xml  2015-03-26 18:59:49.0 +0800
@@ -938,8 +938,8 @@
   artifactIdmaven-compiler-plugin/artifactId
   version2.5.1/version
   configuration
-source1.6/source
-target1.6/target
+source1.7/source
+target1.7/target
   /configuration
 /plugin
 plugin
@@ -1241,8 +1241,8 @@
 artifactIdmaven-compiler-plugin/artifactId
 configuration
   forktrue/fork
-  source1.6/source
-  target1.6/target
+  source1.7/source
+  target1.7/target
   compilerArguments
 Xlint/
 Xmaxwarns/Xmaxwarns

 InputStream is left unclosed in ApplicationClassLoader
 --

 Key: HADOOP-11500
 URL: https://issues.apache.org/jira/browse/HADOOP-11500
 Project: Hadoop Common
  Issue Type: Bug
Affects Versions: 2.6.0
Reporter: Ted Yu
Assignee: Ted Yu
 Fix For: 2.7.0

 Attachments: hadoop-11500-001.patch, hadoop-11500-002.patch


 {code}
 InputStream is = null;
 try {
   is = ApplicationClassLoader.class.getClassLoader().
   getResourceAsStream(PROPERTIES_FILE);
 {code}
 The InputStream is not closed in the static block.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HADOOP-11731) Rework the changelog and releasenotes


 [ 
https://issues.apache.org/jira/browse/HADOOP-11731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Allen Wittenauer updated HADOOP-11731:
--
Attachment: HADOOP-11731-06.patch

-06:
* subtasks section which can easily 2x-3x the changes file
* removed the decrement code in versions

 Rework the changelog and releasenotes
 -

 Key: HADOOP-11731
 URL: https://issues.apache.org/jira/browse/HADOOP-11731
 Project: Hadoop Common
  Issue Type: New Feature
  Components: documentation
Affects Versions: 3.0.0
Reporter: Allen Wittenauer
 Attachments: HADOOP-11731-00.patch, HADOOP-11731-01.patch, 
 HADOOP-11731-03.patch, HADOOP-11731-04.patch, HADOOP-11731-05.patch, 
 HADOOP-11731-06.patch


 The current way we generate these build artifacts is awful.  Plus they are 
 ugly and, in the case of release notes, very hard to pick out what is 
 important.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HADOOP-11778) Update checkstyle

Allen Wittenauer created HADOOP-11778:
-

 Summary: Update checkstyle
 Key: HADOOP-11778
 URL: https://issues.apache.org/jira/browse/HADOOP-11778
 Project: Hadoop Common
  Issue Type: Improvement
  Components: build, test
Affects Versions: 3.0.0
Reporter: Allen Wittenauer


We should update checkstyle so that mvn checkstyle:checkstyle-aggregate doesn't 
NPE.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-11740) Combine erasure encoder and decoder interfaces

2015-03-30 Thread Zhe Zhang (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-11740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14387651#comment-14387651
 ] 

Zhe Zhang commented on HADOOP-11740:


I agree. The current patch does this actually. 

 Combine erasure encoder and decoder interfaces
 --

 Key: HADOOP-11740
 URL: https://issues.apache.org/jira/browse/HADOOP-11740
 Project: Hadoop Common
  Issue Type: Sub-task
  Components: io
Reporter: Zhe Zhang
Assignee: Zhe Zhang
 Attachments: HADOOP-11740-000.patch


 Rationale [discussed | 
 https://issues.apache.org/jira/browse/HDFS-7337?focusedCommentId=14376540page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14376540]
  under HDFS-7337.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HADOOP-11778) Update checkstyle


 [ 
https://issues.apache.org/jira/browse/HADOOP-11778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Allen Wittenauer updated HADOOP-11778:
--
Status: Patch Available  (was: Open)

 Update checkstyle
 -

 Key: HADOOP-11778
 URL: https://issues.apache.org/jira/browse/HADOOP-11778
 Project: Hadoop Common
  Issue Type: Improvement
  Components: build, test
Affects Versions: 3.0.0
Reporter: Allen Wittenauer
 Attachments: HADOOP-11778.000.patch


 We should update checkstyle so that mvn checkstyle:checkstyle-aggregate 
 doesn't NPE.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-11778) Update checkstyle


[ 
https://issues.apache.org/jira/browse/HADOOP-11778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14387965#comment-14387965
 ] 

Hadoop QA commented on HADOOP-11778:


{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  
http://issues.apache.org/jira/secure/attachment/12708308/HADOOP-11778.000.patch
  against trunk revision cce66ba.

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:red}-1 tests included{color}.  The patch doesn't appear to include 
any new or modified tests.
Please justify why no new tests are needed for this 
patch.
Also please list what manual steps were performed to 
verify this patch.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  There were no new javadoc warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 2.0.3) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 core tests{color}.  The patch passed unit tests in .

Test results: 
https://builds.apache.org/job/PreCommit-HADOOP-Build/6029//testReport/
Console output: 
https://builds.apache.org/job/PreCommit-HADOOP-Build/6029//console

This message is automatically generated.

 Update checkstyle
 -

 Key: HADOOP-11778
 URL: https://issues.apache.org/jira/browse/HADOOP-11778
 Project: Hadoop Common
  Issue Type: Improvement
  Components: build, test
Affects Versions: 3.0.0
Reporter: Allen Wittenauer
 Attachments: HADOOP-11778.000.patch


 We should update checkstyle so that mvn checkstyle:checkstyle-aggregate 
 doesn't NPE.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HADOOP-11746) rewrite test-patch.sh


 [ 
https://issues.apache.org/jira/browse/HADOOP-11746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Allen Wittenauer updated HADOOP-11746:
--
Release Note: 
* test-patch.sh now has new output that is different than the previous versions
* test-patch.sh is now pluggable via the test-patch.d directory, with 
checkstyle and shellcheck tests included
* JIRA comments now use much more markup to improve readability
* test-patch.sh now supports either a file name, a URL, or a JIRA issue as 
input in developer mode
Hadoop Flags: Incompatible change

 rewrite test-patch.sh
 -

 Key: HADOOP-11746
 URL: https://issues.apache.org/jira/browse/HADOOP-11746
 Project: Hadoop Common
  Issue Type: Test
  Components: build, test
Affects Versions: 3.0.0
Reporter: Allen Wittenauer
Assignee: Allen Wittenauer
 Attachments: HADOOP-11746-00.patch, HADOOP-11746-01.patch, 
 HADOOP-11746-02.patch, HADOOP-11746-03.patch, HADOOP-11746-04.patch


 This code is bad and you should feel bad.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-11746) rewrite test-patch.sh


[ 
https://issues.apache.org/jira/browse/HADOOP-11746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14388072#comment-14388072
 ] 

Allen Wittenauer commented on HADOOP-11746:
---

I'm marking this as incompatible because the output changed.

 rewrite test-patch.sh
 -

 Key: HADOOP-11746
 URL: https://issues.apache.org/jira/browse/HADOOP-11746
 Project: Hadoop Common
  Issue Type: Test
  Components: build, test
Affects Versions: 3.0.0
Reporter: Allen Wittenauer
Assignee: Allen Wittenauer
 Attachments: HADOOP-11746-00.patch, HADOOP-11746-01.patch, 
 HADOOP-11746-02.patch, HADOOP-11746-03.patch, HADOOP-11746-04.patch


 This code is bad and you should feel bad.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-11377) jdiff failing on java 7 and java 8, Null.java not found

2015-03-30 Thread Tsuyoshi Ozawa (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-11377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14388069#comment-14388069
 ] 

Tsuyoshi Ozawa commented on HADOOP-11377:
-

Li, thanks for your help! Feel free to change the assignee of the issue if 
you'll make any changes to fix the problem. I'll review it.

 jdiff failing on java 7 and java 8, Null.java not found
 -

 Key: HADOOP-11377
 URL: https://issues.apache.org/jira/browse/HADOOP-11377
 Project: Hadoop Common
  Issue Type: Sub-task
  Components: build
Affects Versions: 2.6.0, 2.7.0
 Environment: Java8 jenkins
Reporter: Steve Loughran
Assignee: Tsuyoshi Ozawa
 Attachments: HADOOP-11377.001.patch


 Jdiff is having problems on Java 8, as it cannot find a javadoc for the new 
 {{Null}} datatype
 {code}
 'https://builds.apache.org/job/Hadoop-common-trunk-Java8/ws/hadoop-common-project/hadoop-common/dev-support/jdiff/Null.java'
 The ' characters around the executable and arguments are
 not part of the command.
   [javadoc] javadoc: error - Illegal package name: 
   [javadoc] javadoc: error - File not found: 
 https://builds.apache.org/job/Hadoop-common-trunk-Java8/ws/hadoop-common-project/hadoop-common/dev-support/jdiff/Null.java;
 {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HADOOP-11746) rewrite test-patch.sh


 [ 
https://issues.apache.org/jira/browse/HADOOP-11746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Allen Wittenauer updated HADOOP-11746:
--
Attachment: HADOOP-11746-04.patch

-04:
* *MUCH* better branch and issue detection on patches
* checkstyle sort of works now!
* add missing license files
* some general code cleanup
* JIRA comments are now formatted correctly
* console comments had some display issues fixed
* Removed a lot of if jenkins this, if not jenkins this, simplifying the code 
paths
* in addition to files, test-patch.sh now supports issues and http(s) URLs 
which will automatically download patches off of JIRA 
* give some hints about execution times while running too
* fixed some issues around running tests

To-do:
* Test on Jenkins
* Do more broken patch tests
* Any other requests?

 rewrite test-patch.sh
 -

 Key: HADOOP-11746
 URL: https://issues.apache.org/jira/browse/HADOOP-11746
 Project: Hadoop Common
  Issue Type: Test
  Components: build, test
Affects Versions: 3.0.0
Reporter: Allen Wittenauer
Assignee: Allen Wittenauer
 Attachments: HADOOP-11746-00.patch, HADOOP-11746-01.patch, 
 HADOOP-11746-02.patch, HADOOP-11746-03.patch, HADOOP-11746-04.patch


 This code is bad and you should feel bad.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HADOOP-11746) rewrite test-patch.sh


 [ 
https://issues.apache.org/jira/browse/HADOOP-11746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Allen Wittenauer updated HADOOP-11746:
--
Status: Patch Available  (was: Open)

 rewrite test-patch.sh
 -

 Key: HADOOP-11746
 URL: https://issues.apache.org/jira/browse/HADOOP-11746
 Project: Hadoop Common
  Issue Type: Test
  Components: build, test
Affects Versions: 3.0.0
Reporter: Allen Wittenauer
Assignee: Allen Wittenauer
 Attachments: HADOOP-11746-00.patch, HADOOP-11746-01.patch, 
 HADOOP-11746-02.patch, HADOOP-11746-03.patch, HADOOP-11746-04.patch


 This code is bad and you should feel bad.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-11740) Combine erasure encoder and decoder interfaces

2015-03-30 Thread Kai Zheng (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-11740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14387902#comment-14387902
 ] 

Kai Zheng commented on HADOOP-11740:


Great. I will review it late today, if looks ok I will commit it. Thanks.

 Combine erasure encoder and decoder interfaces
 --

 Key: HADOOP-11740
 URL: https://issues.apache.org/jira/browse/HADOOP-11740
 Project: Hadoop Common
  Issue Type: Sub-task
  Components: io
Reporter: Zhe Zhang
Assignee: Zhe Zhang
 Attachments: HADOOP-11740-000.patch


 Rationale [discussed | 
 https://issues.apache.org/jira/browse/HDFS-7337?focusedCommentId=14376540page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14376540]
  under HDFS-7337.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-11377) jdiff failing on java 7 and java 8, Null.java not found


[ 
https://issues.apache.org/jira/browse/HADOOP-11377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14388031#comment-14388031
 ] 

Li Lu commented on HADOOP-11377:


I did some investigation and I think I know the direct cause of the {{javadoc: 
error - Illegal package name: }} now. It looks like the jdiff.compatibility 
is set to empty at the very beginning of hadoop-project-dist/pom.xml, which may 
cause ant to pass that empyt line as . Javadoc may understand that as a 
package name. After I comment that line out the error disappeared. 

 jdiff failing on java 7 and java 8, Null.java not found
 -

 Key: HADOOP-11377
 URL: https://issues.apache.org/jira/browse/HADOOP-11377
 Project: Hadoop Common
  Issue Type: Sub-task
  Components: build
Affects Versions: 2.6.0, 2.7.0
 Environment: Java8 jenkins
Reporter: Steve Loughran
Assignee: Tsuyoshi Ozawa
 Attachments: HADOOP-11377.001.patch


 Jdiff is having problems on Java 8, as it cannot find a javadoc for the new 
 {{Null}} datatype
 {code}
 'https://builds.apache.org/job/Hadoop-common-trunk-Java8/ws/hadoop-common-project/hadoop-common/dev-support/jdiff/Null.java'
 The ' characters around the executable and arguments are
 not part of the command.
   [javadoc] javadoc: error - Illegal package name: 
   [javadoc] javadoc: error - File not found: 
 https://builds.apache.org/job/Hadoop-common-trunk-Java8/ws/hadoop-common-project/hadoop-common/dev-support/jdiff/Null.java;
 {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HADOOP-11377) jdiff failing on java 7 and java 8, Null.java not found