[jira] [Commented] (HDFS-10629) Federation Router

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15537926#comment-15537926
 ] 

Hadoop QA commented on HDFS-10629:
--

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
21s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 6 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  7m 
 5s{color} | {color:green} HDFS-10467 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
45s{color} | {color:green} HDFS-10467 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
31s{color} | {color:green} HDFS-10467 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
51s{color} | {color:green} HDFS-10467 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
13s{color} | {color:green} HDFS-10467 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
41s{color} | {color:green} HDFS-10467 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
55s{color} | {color:green} HDFS-10467 passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
41s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
0m 29s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch 
generated 168 new + 390 unchanged - 0 fixed = 558 total (was 390) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
 9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  1m 
48s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs generated 7 new + 0 
unchanged - 0 fixed = 7 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
52s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 57m 29s{color} 
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
20s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 77m  7s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-hdfs-project/hadoop-hdfs |
|  |  
org.apache.hadoop.hdfs.server.federation.resolver.RemoteLocation.equals(Object) 
is unusual  At RemoteLocation.java:RemoteLocation.java:[line 71] |
|  |  
org.apache.hadoop.hdfs.server.federation.resolver.RemoteLocation.equals(Object) 
does not check for null argument  At RemoteLocation.java:null argument  At 
RemoteLocation.java:[line 71] |
|  |  Sequence of calls to java.util.concurrent.ConcurrentHashMap may not be 
atomic in 
org.apache.hadoop.hdfs.server.federation.router.ConnectionManager.getConnection(UserGroupInformation,
 String)  At ConnectionManager.java:may not be atomic in 
org.apache.hadoop.hdfs.server.federation.router.ConnectionManager.getConnection(UserGroupInformation,
 String)  At ConnectionManager.java:[line 268] |
|  |  Exception is caught when Exception is not thrown in 
org.apache.hadoop.hdfs.server.federation.router.FederationUtil.createInstance(Configuration,
 Object, Class, String, String, Class)  At FederationUtil.java:is not thrown in 
org.apache.hadoop.hdfs.server.federation.router.FederationUtil.createInstance(Configuration,
 Object, Class, String, String, Class)  At FederationUtil.java:[line 67] |
|  |  
org.apache.hadoop.hdfs.server.federation.router.Router.initAndStartRouter(Configuration,
 boolean) invokes System.exit(...), which shuts down the entire virtual machine 
 At Router.java:shuts down the entire virtual machine  At 

[jira] [Created] (HDFS-10943) rollEditLog expects empty EditsDoubleBuffer.bufCurrent which is not guaranteed

[Yongjun Zhang (JIRA)]

Yongjun Zhang created HDFS-10943:


 Summary: rollEditLog expects empty EditsDoubleBuffer.bufCurrent 
which is not guaranteed
 Key: HDFS-10943
 URL: https://issues.apache.org/jira/browse/HDFS-10943
 Project: Hadoop HDFS
  Issue Type: Bug
Reporter: Yongjun Zhang


Per the following trace stack:
{code}
FATAL org.apache.hadoop.hdfs.server.namenode.FSEditLog: Error: finalize log 
segment 10562075963, 10562174157 failed for required journal 
(JournalAndStream(mgr=QJM to [0.0.0.1:8485, 0.0.0.2:8485, 0.0.0.3:8485, 
0.0.0.4:8485, 0.0.0.5:8485], stream=QuorumOutputStream starting at txid 
10562075963))
java.io.IOException: FSEditStream has 49708 bytes still to be flushed and 
cannot be closed.
at 
org.apache.hadoop.hdfs.server.namenode.EditsDoubleBuffer.close(EditsDoubleBuffer.java:66)
at 
org.apache.hadoop.hdfs.qjournal.client.QuorumOutputStream.close(QuorumOutputStream.java:65)
at 
org.apache.hadoop.hdfs.server.namenode.JournalSet$JournalAndStream.closeStream(JournalSet.java:115)
at 
org.apache.hadoop.hdfs.server.namenode.JournalSet$4.apply(JournalSet.java:235)
at 
org.apache.hadoop.hdfs.server.namenode.JournalSet.mapJournalsAndReportErrors(JournalSet.java:393)
at 
org.apache.hadoop.hdfs.server.namenode.JournalSet.finalizeLogSegment(JournalSet.java:231)
at 
org.apache.hadoop.hdfs.server.namenode.FSEditLog.endCurrentLogSegment(FSEditLog.java:1243)
at 
org.apache.hadoop.hdfs.server.namenode.FSEditLog.rollEditLog(FSEditLog.java:1172)
at 
org.apache.hadoop.hdfs.server.namenode.FSImage.rollEditLog(FSImage.java:1243)
at 
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.rollEditLog(FSNamesystem.java:6437)
at 
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.rollEditLog(NameNodeRpcServer.java:1002)
at 
org.apache.hadoop.hdfs.protocolPB.NamenodeProtocolServerSideTranslatorPB.rollEditLog(NamenodeProtocolServerSideTranslatorPB.java:142)
at 
org.apache.hadoop.hdfs.protocol.proto.NamenodeProtocolProtos$NamenodeProtocolService$2.callBlockingMethod(NamenodeProtocolProtos.java:12025)
at 
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1060)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2086)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2082)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2080)
2016-09-23 21:40:59,618 WARN 
org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager: Aborting 
QuorumOutputStream starting at txid 10562075963
{code}

The exception is from  EditsDoubleBuffer
{code}
 public void close() throws IOException {
Preconditions.checkNotNull(bufCurrent);
Preconditions.checkNotNull(bufReady);

int bufSize = bufCurrent.size();
if (bufSize != 0) {
  throw new IOException("FSEditStream has " + bufSize
  + " bytes still to be flushed and cannot be closed.");
}

IOUtils.cleanup(null, bufCurrent, bufReady);
bufCurrent = bufReady = null;
  }
{code}

We can see that FSNamesystem.rollEditLog expects  EditsDoubleBuffer.bufCurrent 
to be empty.

Edits are recorded via FSEditLog$logSync, which does:
{code}
   * The data is double-buffered within each edit log implementation so that
   * in-memory writing can occur in parallel with the on-disk writing.
   *
   * Each sync occurs in three steps:
   *   1. synchronized, it swaps the double buffer and sets the isSyncRunning
   *  flag.
   *   2. unsynchronized, it flushes the data to storage
   *   3. synchronized, it resets the flag and notifies anyone waiting on the
   *  sync.
   *
   * The lack of synchronization on step 2 allows other threads to continue
   * to write into the memory buffer while the sync is in progress.
   * Because this step is unsynchronized, actions that need to avoid
   * concurrency with sync() should be synchronized and also call
   * waitForSyncToFinish() before assuming they are running alone.
   */
{code}

We can see that step 2 is on-purposely not synchronized to let other threads to 
write into the memory buffer, presumbaly EditsDoubleBuffer.bufCurrent. This 
means that the EditsDoubleBuffer.bufCurrent  can be non-empty when logSync is 
done.

Now if rollEditLog happens, the above exception happens.

Another interesting observation is, the size of the EditsDoubleBuffer can be as 
large as "private int outputBufferCapacity = 512 * 1024;", which means a lot of 
edits could get buffered before they are flushed to JNs. 

How can rollEdit operation expect the 

[jira] [Updated] (HDFS-10629) Federation Router

[Jason Kace (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-10629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Kace updated HDFS-10629:
--
Attachment: HDFS-10629-HDFS-10467-004.patch

Updating the patch to include:

* Added a retry policy to the Router NN RPC client to customize retries on 
network/exception failures.  Currently uses 
RetryPolicies.failoverOnNetworkException.

* Separated the Router NN RPC client (router -> NN) into a separate class.  
Refactored it a bit to be more clear.  Changed the method invocation parameters 
and flow.

* Updated the Router's implementation of the ClientProtocol to support multiple 
destinations per path.  For example, a single path such as /a in the global 
namespace can map to 1-N different namespaces.  Certain APIs, such as 
getListing, concurrently collect the listings from all of the 
namespaces/locations for the global path, and then aggregate them together.  
Other APIs such as mkdir were modfiied to check all locations to ensure the 
directory doesn't exist before permitting a create.

*  Added a few features to the NN rpc client.  The largest one was the 
separation of invocations into 3 categories (single call, multiple concurrent 
calls, multiple sequential calls until a success condition is met) to support 
multiple nameservices/paths per global path.

*  Updated the unit tests to cover a few more conditions, including better mock 
namenode resolver/discovery.

*  Removed a few unnecessary functions/components from the patch.  
Documentation improvements.

> Federation Router
> -
>
> Key: HDFS-10629
> URL: https://issues.apache.org/jira/browse/HDFS-10629
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: hdfs
>Reporter: Inigo Goiri
>Assignee: Jason Kace
> Attachments: HDFS-10629-HDFS-10467-002.patch, 
> HDFS-10629-HDFS-10467-003.patch, HDFS-10629-HDFS-10467-004.patch, 
> HDFS-10629.000.patch, HDFS-10629.001.patch
>
>
> Component that routes calls from the clients to the right Namespace. It 
> implements {{ClientProtocol}}.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10629) Federation Router

[Jason Kace (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15537824#comment-15537824
 ] 

Jason Kace commented on HDFS-10629:
---

FederationNamenodeServiceState#EXPIRED isn't used. Is it needed?

* Yes, the expired state is used to indicate NNs that haven't responded to a 
heartbeat in a long time.  This prevents them from being used on failover 
retries as they are very likely gone.  It also allows for the disabling of NNs 
and namespaces that are no longer present in the cluster.  Once expired the 
state store ignores the NN registrations until they respond to another 
heartbeat.

Is FederationNamenodeServiceState really needed, e.g. if HAServiceState can 
work?

* HAServiceState is similar and could be used, but I think it is cleaner to use 
own own state.  There is no clear mapping for the expired state, and a less 
clear mapping for unavailable.  In the non-HA case, NNs are either active, 
unavailable or expired.  In the HA-case they are either Active, Standby, 
Unavailable or Expired.

Do we need NamenodeStatusReport? Per 
https://issues.apache.org/jira/browse/HDFS-10467?focusedCommentId=15382535=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15382535,
 this might be required only for the rebalancer scenario. We can add it later 
when necessary.

* The status report is used by the NN heartbeat service to register/discover a 
NN and store it's state in the store.  It is included in this patch due to its 
role in the ActiveNamenodeResolver.  The router via heartbeats can discover NNs 
that exist in the subcluster without relying on the config files.

Router#getLocationsForPath command "Get the NN RPC client”, does it actually 
return RPC client? It seems to return the remote location.

* See the next patch (004).  I refactored the NN client into a separate class 
and renamed a number of the functions.  Each RPC client wraps a 
proxy/connection object to a single NN.

It seems better exception handling will be handled by other jiras. For example, 
RetriableException can be thrown by active NN and should be handled properly.

*  Yes, exception handling is a TODO.  In patch 004 I added a 
RetryPolicies.failoverOnNetworkException to handle simple network failures and 
to detect failover conditions.  This area needs continued work.

Router.java has several mismatch between parameter name in the javadoc and 
actual name.

*  Thanks, fixed.

ConnectionPool has clean up task to reduce the connections. But If there are 
lots of users, there will be lots of ConnectionPool objects. Maybe not a major 
issue, but wonder if someone can launch attack on router. Or we can make the 
connection max size global across different pools. Have you checked out if 
Server#ConnectionManager can be used instead?

* Yes, there is no limit on the number of connections that can be in the pool.  
I will take a look at the server connection manager to see what I can use, but 
I do not believe this can be reused in its whole.  The connection pool caches 
NN ClientProtocol proxy/connection object.

Is DFSUtil#getNamenodeWebAddr required for this jira? If yes, support for https 
is needed.

* Removed

What is the use case of blockpool based lookup? For example 
updateBlockForPipeline and some other methods calls invokeMethod based on 
blockpool. Wonder if it can use the version used by many other methods.

* Some ClientProtocol APIs, such as 
updatePipeline/updateBlockForPipeline/abandonBlock do not have a file/source 
path in their parameter list.  We use the block pool ID to map 1:1 to a 
nameservice (based on the NN heartbeat which records the block pool id for each 
nameservice in the state store).  In these cases the block pool ID is used to 
determine which nameservice to target.

reportBadBlocks mentions DatanodeProtocol in the comment. That can be removed 
given Router only serves as proxy between client and NN.

* Updated the code to remove the handler.

Given ActiveNamenodeLocator and other interfaces are likely to change, maybe 
mark it as @InterfaceStability.Evolving, also given they aren’t used by 
applications, @InterfaceAudience.Private seems more appropriate.

* This is a good idea.  I'll check all of the interfaces to make sure they are 
properly marked.  I think private is best as they are not currently used by 
other components or clients.

> Federation Router
> -
>
> Key: HDFS-10629
> URL: https://issues.apache.org/jira/browse/HDFS-10629
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: hdfs
>Reporter: Inigo Goiri
>Assignee: Jason Kace
> Attachments: HDFS-10629-HDFS-10467-002.patch, 
> HDFS-10629-HDFS-10467-003.patch, HDFS-10629.000.patch, HDFS-10629.001.patch
>
>
> Component that routes calls from the clients to the right Namespace. It 
> implements {{ClientProtocol}}.

[jira] [Commented] (HDFS-10637) Modifications to remove the assumption that FsVolumes are backed by java.io.File.

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15537761#comment-15537761
 ] 

Hadoop QA commented on HDFS-10637:
--

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
16s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 14 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  7m 
11s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
42s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
0m 36s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch 
generated 9 new + 984 unchanged - 11 fixed = 993 total (was 995) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
 9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
54s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 60m  6s{color} 
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
18s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 80m 13s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.datanode.TestDirectoryScanner |
|   | hadoop.hdfs.TestRenameWhileOpen |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:9560f25 |
| JIRA Issue | HDFS-10637 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12831203/HDFS-10637.010.patch |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  |
| uname | Linux 89dc4e911088 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 
20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / 89bd6d2 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| checkstyle | 
https://builds.apache.org/job/PreCommit-HDFS-Build/16963/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
 |
| unit | 
https://builds.apache.org/job/PreCommit-HDFS-Build/16963/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HDFS-Build/16963/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U: 
hadoop-hdfs-project/hadoop-hdfs |
| Console output | 
https://builds.apache.org/job/PreCommit-HDFS-Build/16963/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT   http://yetus.apache.org |


This message was automatically generated.



> Modifications to remove the assumption that FsVolumes are backed by 
> java.io.File.
> 

[jira] [Work started] (HDFS-10922) Adding additional unit tests for Trash

[Weiwei Yang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-10922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on HDFS-10922 started by Weiwei Yang.
--
> Adding additional unit tests for Trash
> --
>
> Key: HDFS-10922
> URL: https://issues.apache.org/jira/browse/HDFS-10922
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: test
>Reporter: Xiaoyu Yao
>Assignee: Weiwei Yang
>
> This ticket is opened to track adding unit tests for Trash.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10690) Optimize insertion/removal of replica in ShortCircuitCache.java

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15537690#comment-15537690
 ] 

Hadoop QA commented on HDFS-10690:
--

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
13s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
26s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  7m 
 2s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  3m 
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
21s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m  
7s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red}  0m 
45s{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red}  1m  
6s{color} | {color:red} hadoop-hdfs-project in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red}  1m  6s{color} 
| {color:red} hadoop-hdfs-project in the patch failed. {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
29s{color} | {color:green} hadoop-hdfs-project: The patch generated 0 new + 90 
unchanged - 4 fixed = 90 total (was 94) {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red}  0m 
44s{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  0m 
21s{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m 
58s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}  0m 47s{color} 
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
18s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 27m 21s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:9560f25 |
| JIRA Issue | HDFS-10690 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12831205/HDFS-10690.007.patch |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  |
| uname | Linux 9c586e03e52d 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 
17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / 89bd6d2 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| mvninstall | 
https://builds.apache.org/job/PreCommit-HDFS-Build/16964/artifact/patchprocess/patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs.txt
 |
| compile | 
https://builds.apache.org/job/PreCommit-HDFS-Build/16964/artifact/patchprocess/patch-compile-hadoop-hdfs-project.txt
 |
| javac | 
https://builds.apache.org/job/PreCommit-HDFS-Build/16964/artifact/patchprocess/patch-compile-hadoop-hdfs-project.txt
 |
| mvnsite | 

[jira] [Commented] (HDFS-10923) Make InstrumentedLock require ReentrantLock

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15537668#comment-15537668
 ] 

Hadoop QA commented on HDFS-10923:
--

| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
17s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 2 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  7m 
19s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
52s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
58s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 62m 
49s{color} | {color:green} hadoop-hdfs in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
17s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 83m  8s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:9560f25 |
| JIRA Issue | HDFS-10923 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12831200/HDFS-10923.02.patch |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  |
| uname | Linux 3360f0007609 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 
21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / 2549ee9 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HDFS-Build/16962/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U: 
hadoop-hdfs-project/hadoop-hdfs |
| Console output | 
https://builds.apache.org/job/PreCommit-HDFS-Build/16962/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT   http://yetus.apache.org |


This message was automatically generated.



> Make InstrumentedLock require ReentrantLock
> ---
>
> Key: HDFS-10923
> URL: https://issues.apache.org/jira/browse/HDFS-10923
> Project: Hadoop HDFS
>  Issue Type: Bug
>Reporter: Arpit Agarwal
>Assignee: Arpit Agarwal
> Attachments: HDFS-10923.01.patch, HDFS-10923.02.patch
>
>
> Make InstrumentedLock use ReentrantLock instead of Lock, so nested 
> acquire/release calls can be instrumented correctly.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-10797) Disk usage summary of snapshots causes renamed blocks to get counted twice

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15537650#comment-15537650
 ] 

Hadoop QA commented on HDFS-10797:
--

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
17s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  7m 
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
57s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  
3s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
48s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
0m 28s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch 
generated 1 new + 267 unchanged - 0 fixed = 268 total (was 267) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  2m  
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
58s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 61m 55s{color} 
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
17s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 83m  5s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | 
hadoop.hdfs.server.blockmanagement.TestPendingInvalidateBlock |
| Timed out junit tests | org.apache.hadoop.hdfs.TestSmallBlock |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:9560f25 |
| JIRA Issue | HDFS-10797 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12831199/HDFS-10797.009.patch |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  |
| uname | Linux 2925a49ab866 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed 
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / 2549ee9 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| checkstyle | 
https://builds.apache.org/job/PreCommit-HDFS-Build/16961/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
 |
| unit | 
https://builds.apache.org/job/PreCommit-HDFS-Build/16961/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HDFS-Build/16961/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U: 
hadoop-hdfs-project/hadoop-hdfs |
| Console output | 
https://builds.apache.org/job/PreCommit-HDFS-Build/16961/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT   http://yetus.apache.org |


This message was automatically generated.



> Disk usage summary of snapshots causes renamed blocks to get counted twice
> 

[jira] [Commented] (HDFS-10893) Refactor TestDFSShell by setting up MiniDFSCluser once for all commands test

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10893?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15537643#comment-15537643
 ] 

Hadoop QA commented on HDFS-10893:
--

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
18s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  7m 
54s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
40s{color} | {color:green} branch-2 passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
43s{color} | {color:green} branch-2 passed with JDK v1.7.0_111 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
29s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
51s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
16s{color} | {color:green} branch-2 passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  1m 
54s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs in branch-2 has 1 
extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
56s{color} | {color:green} branch-2 passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
39s{color} | {color:green} branch-2 passed with JDK v1.7.0_111 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
39s{color} | {color:green} the patch passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
40s{color} | {color:green} the patch passed with JDK v1.7.0_111 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
40s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
0m 26s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch 
generated 54 new + 94 unchanged - 80 fixed = 148 total (was 174) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  2m  
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
54s{color} | {color:green} the patch passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
36s{color} | {color:green} the patch passed with JDK v1.7.0_111 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 61m 22s{color} 
| {color:red} hadoop-hdfs in the patch failed with JDK v1.7.0_111. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
23s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}148m 23s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.7.0_111 Failed junit tests | hadoop.hdfs.TestEncryptionZones |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:b59b8b7 |
| JIRA Issue | HDFS-10893 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12831180/HDFS-10893-branch-2.001.patch
 |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  |
| uname | Linux 1e28675bb559 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 
17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git 

[jira] [Commented] (HDFS-10690) Optimize insertion/removal of replica in ShortCircuitCache.java

[Fenghua Hu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15537580#comment-15537580
 ] 

Fenghua Hu commented on HDFS-10690:
---

Hi [~xyao], I should had run a clean compilation, sorry for your inconvenience. 
I will update the patch soon.

> Optimize insertion/removal of replica in ShortCircuitCache.java
> ---
>
> Key: HDFS-10690
> URL: https://issues.apache.org/jira/browse/HDFS-10690
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: hdfs-client
>Affects Versions: 3.0.0-alpha2
>Reporter: Fenghua Hu
>Assignee: Fenghua Hu
> Attachments: HDFS-10690.001.patch, HDFS-10690.002.patch, 
> HDFS-10690.003.patch, HDFS-10690.004.patch, HDFS-10690.005.patch, 
> HDFS-10690.006.patch, HDFS-10690.007.patch, ShortCircuitCache_LinkedMap.patch
>
>   Original Estimate: 336h
>  Remaining Estimate: 336h
>
> Currently in ShortCircuitCache, two TreeMap objects are used to track the 
> cached replicas.
> private final TreeMap evictable = new TreeMap<>();
> private final TreeMap evictableMmapped = new 
> TreeMap<>();
> TreeMap employs Red-Black tree for sorting. This isn't an issue when using 
> traditional HDD. But when using high-performance SSD/PCIe Flash, the cost 
> inserting/removing an entry  becomes considerable.
> To mitigate it, we designed a new list-based for replica tracking.
> The list is a double-linked FIFO. FIFO is time-based, thus insertion is a 
> very low cost operation. On the other hand, list is not lookup-friendly. To 
> address this issue, we introduce two references into ShortCircuitReplica 
> object.
> ShortCircuitReplica next = null;
> ShortCircuitReplica prev = null;
> In this way, lookup is not needed when removing a replica from the list. We 
> only need to modify its predecessor's and successor's references in the lists.
> Our tests showed up to 15-50% performance improvement when using PCIe flash 
> as storage media.
> The original patch is against 2.6.4, now I am porting to Hadoop trunk, and 
> patch will be posted soon.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-10690) Optimize insertion/removal of replica in ShortCircuitCache.java

[Fenghua Hu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-10690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Fenghua Hu updated HDFS-10690:
--
Attachment: HDFS-10690.007.patch

> Optimize insertion/removal of replica in ShortCircuitCache.java
> ---
>
> Key: HDFS-10690
> URL: https://issues.apache.org/jira/browse/HDFS-10690
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: hdfs-client
>Affects Versions: 3.0.0-alpha2
>Reporter: Fenghua Hu
>Assignee: Fenghua Hu
> Attachments: HDFS-10690.001.patch, HDFS-10690.002.patch, 
> HDFS-10690.003.patch, HDFS-10690.004.patch, HDFS-10690.005.patch, 
> HDFS-10690.006.patch, HDFS-10690.007.patch, ShortCircuitCache_LinkedMap.patch
>
>   Original Estimate: 336h
>  Remaining Estimate: 336h
>
> Currently in ShortCircuitCache, two TreeMap objects are used to track the 
> cached replicas.
> private final TreeMap evictable = new TreeMap<>();
> private final TreeMap evictableMmapped = new 
> TreeMap<>();
> TreeMap employs Red-Black tree for sorting. This isn't an issue when using 
> traditional HDD. But when using high-performance SSD/PCIe Flash, the cost 
> inserting/removing an entry  becomes considerable.
> To mitigate it, we designed a new list-based for replica tracking.
> The list is a double-linked FIFO. FIFO is time-based, thus insertion is a 
> very low cost operation. On the other hand, list is not lookup-friendly. To 
> address this issue, we introduce two references into ShortCircuitReplica 
> object.
> ShortCircuitReplica next = null;
> ShortCircuitReplica prev = null;
> In this way, lookup is not needed when removing a replica from the list. We 
> only need to modify its predecessor's and successor's references in the lists.
> Our tests showed up to 15-50% performance improvement when using PCIe flash 
> as storage media.
> The original patch is against 2.6.4, now I am porting to Hadoop trunk, and 
> patch will be posted soon.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-10637) Modifications to remove the assumption that FsVolumes are backed by java.io.File.

[Virajith Jalaparti (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-10637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Virajith Jalaparti updated HDFS-10637:
--
Status: Patch Available  (was: Open)

> Modifications to remove the assumption that FsVolumes are backed by 
> java.io.File.
> -
>
> Key: HDFS-10637
> URL: https://issues.apache.org/jira/browse/HDFS-10637
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: datanode, fs
>Reporter: Virajith Jalaparti
>Assignee: Virajith Jalaparti
> Attachments: HDFS-10637.001.patch, HDFS-10637.002.patch, 
> HDFS-10637.003.patch, HDFS-10637.004.patch, HDFS-10637.005.patch, 
> HDFS-10637.006.patch, HDFS-10637.007.patch, HDFS-10637.008.patch, 
> HDFS-10637.009.patch, HDFS-10637.010.patch
>
>
> Modifications to {{FsVolumeSpi}} and {{FsVolumeImpl}} to remove references to 
> {{java.io.File}}.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-10637) Modifications to remove the assumption that FsVolumes are backed by java.io.File.

[Virajith Jalaparti (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-10637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Virajith Jalaparti updated HDFS-10637:
--
Attachment: HDFS-10637.010.patch

Attaching new patch based on [~eddyxu]'s comments. 

> Modifications to remove the assumption that FsVolumes are backed by 
> java.io.File.
> -
>
> Key: HDFS-10637
> URL: https://issues.apache.org/jira/browse/HDFS-10637
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: datanode, fs
>Reporter: Virajith Jalaparti
>Assignee: Virajith Jalaparti
> Attachments: HDFS-10637.001.patch, HDFS-10637.002.patch, 
> HDFS-10637.003.patch, HDFS-10637.004.patch, HDFS-10637.005.patch, 
> HDFS-10637.006.patch, HDFS-10637.007.patch, HDFS-10637.008.patch, 
> HDFS-10637.009.patch, HDFS-10637.010.patch
>
>
> Modifications to {{FsVolumeSpi}} and {{FsVolumeImpl}} to remove references to 
> {{java.io.File}}.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10609) Uncaught InvalidEncryptionKeyException during pipeline recovery may abort downstream applications

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15537550#comment-15537550
 ] 

Hadoop QA commented on HDFS-10609:
--

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
18s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  6m 
 1s{color} | {color:green} branch-2.7 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m  
2s{color} | {color:green} branch-2.7 passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m  
2s{color} | {color:green} branch-2.7 passed with JDK v1.7.0_111 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
31s{color} | {color:green} branch-2.7 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
59s{color} | {color:green} branch-2.7 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
15s{color} | {color:green} branch-2.7 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  2m 
59s{color} | {color:green} branch-2.7 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
58s{color} | {color:green} branch-2.7 passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
40s{color} | {color:green} branch-2.7 passed with JDK v1.7.0_111 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
55s{color} | {color:green} the patch passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
59s{color} | {color:green} the patch passed with JDK v1.7.0_111 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
59s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
0m 29s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch 
generated 3 new + 584 unchanged - 5 fixed = 587 total (was 589) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
12s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red}  0m  
0s{color} | {color:red} The patch has 3260 line(s) that end in whitespace. Use 
git apply --whitespace=fix <>. Refer 
https://git-scm.com/docs/git-apply {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red}  1m 
23s{color} | {color:red} The patch 113 line(s) with tabs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  3m 
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
55s{color} | {color:green} the patch passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
40s{color} | {color:green} the patch passed with JDK v1.7.0_111 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 47m 38s{color} 
| {color:red} hadoop-hdfs in the patch failed with JDK v1.7.0_111. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red}  0m 
19s{color} | {color:red} The patch generated 3 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black}127m 37s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_101 Failed junit tests | 
hadoop.hdfs.server.namenode.ha.TestBootstrapStandbyWithQJM |
|   | hadoop.hdfs.TestLeaseRecovery2 |
|   | hadoop.hdfs.TestReplaceDatanodeOnFailure |
|   | hadoop.hdfs.web.TestWebHdfsTokens |
|   | hadoop.hdfs.protocol.datatransfer.sasl.TestSaslDataTransfer |
|   | hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots |
|   | hadoop.hdfs.web.TestWebHdfsTimeouts |
| JDK v1.7.0_111 Failed junit tests | 

[jira] [Commented] (HDFS-10931) libhdfs++: Fix object lifecycle issues in the BlockReader

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15537542#comment-15537542
 ] 

Hadoop QA commented on HDFS-10931:
--

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 16m 
26s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red}  0m  
0s{color} | {color:red} The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m 
 5s{color} | {color:green} HDFS-8707 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  7m 
21s{color} | {color:green} HDFS-8707 passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  7m 
14s{color} | {color:green} HDFS-8707 passed with JDK v1.7.0_111 {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
24s{color} | {color:green} HDFS-8707 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
15s{color} | {color:green} HDFS-8707 passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  7m 
25s{color} | {color:green} the patch passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green}  7m 
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  7m 
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  7m  
0s{color} | {color:green} the patch passed with JDK v1.7.0_111 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green}  7m  
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  7m  
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 10m 
21s{color} | {color:green} hadoop-hdfs-native-client in the patch passed with 
JDK v1.7.0_111. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
33s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 79m 33s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:78fc6b6 |
| JIRA Issue | HDFS-10931 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12831175/HDFS-10931.HDFS-8707.001.patch
 |
| Optional Tests |  asflicense  compile  cc  mvnsite  javac  unit  |
| uname | Linux 40271a3901b6 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 
17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | HDFS-8707 / feba09b |
| Default Java | 1.7.0_111 |
| Multi-JDK versions |  /usr/lib/jvm/java-8-oracle:1.8.0_101 
/usr/lib/jvm/java-7-openjdk-amd64:1.7.0_111 |
| JDK v1.7.0_111  Test Results | 
https://builds.apache.org/job/PreCommit-HDFS-Build/16958/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-native-client U: 
hadoop-hdfs-project/hadoop-hdfs-native-client |
| Console output | 
https://builds.apache.org/job/PreCommit-HDFS-Build/16958/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT   http://yetus.apache.org |


This message was automatically generated.



> libhdfs++: Fix object lifecycle issues in the BlockReader
> -
>
> Key: HDFS-10931
> URL: https://issues.apache.org/jira/browse/HDFS-10931
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: hdfs-client
>Reporter: James Clampffer
>Assignee: James Clampffer
>

[jira] [Commented] (HDFS-10923) Make InstrumentedLock require ReentrantLock

[Chris Douglas (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15537539#comment-15537539
 ] 

Chris Douglas commented on HDFS-10923:
--

+1 lgtm

> Make InstrumentedLock require ReentrantLock
> ---
>
> Key: HDFS-10923
> URL: https://issues.apache.org/jira/browse/HDFS-10923
> Project: Hadoop HDFS
>  Issue Type: Bug
>Reporter: Arpit Agarwal
>Assignee: Arpit Agarwal
> Attachments: HDFS-10923.01.patch, HDFS-10923.02.patch
>
>
> Make InstrumentedLock use ReentrantLock instead of Lock, so nested 
> acquire/release calls can be instrumented correctly.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10619) Cache path in InodesInPath

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15537538#comment-15537538
 ] 

Hadoop QA commented on HDFS-10619:
--

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
15s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red}  0m  
0s{color} | {color:red} The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  8m 
 9s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m  
1s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  2m  
6s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
11s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
 3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
56s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 65m 55s{color} 
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
21s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 88m 31s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.qjournal.client.TestQuorumJournalManager |
|   | hadoop.hdfs.server.blockmanagement.TestRBWBlockInvalidation |
|   | hadoop.hdfs.server.datanode.TestDataNodeUUID |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:9560f25 |
| JIRA Issue | HDFS-10619 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12831173/HDFS-10619.1.patch |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  |
| uname | Linux 51868a22ad2d 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed 
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / 2549ee9 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| unit | 
https://builds.apache.org/job/PreCommit-HDFS-Build/16957/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HDFS-Build/16957/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U: 
hadoop-hdfs-project/hadoop-hdfs |
| Console output | 
https://builds.apache.org/job/PreCommit-HDFS-Build/16957/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT   http://yetus.apache.org |


This message was automatically generated.



> Cache path in InodesInPath
> --
>
> Key: HDFS-10619
> URL: 

[jira] [Commented] (HDFS-10910) HDFS Erasure Coding doc should state its currently supported erasure coding policies

[SammiChen (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15537537#comment-15537537
 ] 

SammiChen commented on HDFS-10910:
--

Holiday from Oct. 1 to Oct. 7, please expect no response. Thanks.



> HDFS Erasure Coding doc should state its currently supported erasure coding 
> policies
> 
>
> Key: HDFS-10910
> URL: https://issues.apache.org/jira/browse/HDFS-10910
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: documentation, erasure-coding
>Affects Versions: 3.0.0-alpha1
>Reporter: Wei-Chiu Chuang
>Assignee: Yiqun Lin
> Attachments: HDFS-10910.001.patch, HDFS-10910.002.patch, 
> HDFS-10910.003.patch
>
>
> While HDFS Erasure Coding doc states a variety of possible combinations of 
> algorithms, block group size and cell size, the code (as of 3.0.0-alpha1) 
> allows only three policies: RS_6_3_SCHEMA, RS_3_2_SCHEMA and 
> RS_6_3_LEGACY_SCHEMA. All with default cell size. I think this should be 
> documented.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10910) HDFS Erasure Coding doc should state its currently supported erasure coding policies

[Yiqun Lin (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15537534#comment-15537534
 ] 

Yiqun Lin commented on HDFS-10910:
--

Thanks [~jojochuang] for the commit!

> HDFS Erasure Coding doc should state its currently supported erasure coding 
> policies
> 
>
> Key: HDFS-10910
> URL: https://issues.apache.org/jira/browse/HDFS-10910
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: documentation, erasure-coding
>Affects Versions: 3.0.0-alpha1
>Reporter: Wei-Chiu Chuang
>Assignee: Yiqun Lin
> Attachments: HDFS-10910.001.patch, HDFS-10910.002.patch, 
> HDFS-10910.003.patch
>
>
> While HDFS Erasure Coding doc states a variety of possible combinations of 
> algorithms, block group size and cell size, the code (as of 3.0.0-alpha1) 
> allows only three policies: RS_6_3_SCHEMA, RS_3_2_SCHEMA and 
> RS_6_3_LEGACY_SCHEMA. All with default cell size. I think this should be 
> documented.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-10932) Ozone : fix XceiverClient slow shutdown

[Anu Engineer (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-10932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anu Engineer updated HDFS-10932:

  Resolution: Fixed
Hadoop Flags: Reviewed
  Status: Resolved  (was: Patch Available)

[~vagarychen] Thank you for finding and fixing the issue. I have committed this 
to the feature branch.

> Ozone : fix XceiverClient slow shutdown
> ---
>
> Key: HDFS-10932
> URL: https://issues.apache.org/jira/browse/HDFS-10932
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>Reporter: Chen Liang
>Assignee: Chen Liang
> Attachments: HDFS-10932-HDFS-7240.002.patch, 
> HDFS-10932-HDFS-7240.002.patch, HDFS-10932.001.patch
>
>
> Currently {{XceiverClient}} is the underlying entity of 
> {{DistributedStorageHandler.newKeyWriter()}} and 
> {{DistributedStorageHandler.newKeyReader()}}  for making call to container 
> for read/write. When {{XceiverClient}} gets closed, 
> {{group.shutdownGracefully()}} gets called, which is an asynchronous call. 
> A problem is that this asynchronous call has default quiet period of 2 
> seconds before it actually shutdown, so if we have a burst of read/write 
> calls, we would end up having threads created faster than they got 
> terminated, reaching system limit at some point.
> Ideally, this needs to be fixed with cached clients instead of creating new 
> thread each time. This JIRA only tries to give a temporary fix for the time 
> being.
> Thanks [~anu] for the offline discussion.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10932) Ozone : fix XceiverClient slow shutdown

[Anu Engineer (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15537475#comment-15537475
 ] 

Anu Engineer commented on HDFS-10932:
-

+1, I will commit this now. 

> Ozone : fix XceiverClient slow shutdown
> ---
>
> Key: HDFS-10932
> URL: https://issues.apache.org/jira/browse/HDFS-10932
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>Reporter: Chen Liang
>Assignee: Chen Liang
> Attachments: HDFS-10932-HDFS-7240.002.patch, 
> HDFS-10932-HDFS-7240.002.patch, HDFS-10932.001.patch
>
>
> Currently {{XceiverClient}} is the underlying entity of 
> {{DistributedStorageHandler.newKeyWriter()}} and 
> {{DistributedStorageHandler.newKeyReader()}}  for making call to container 
> for read/write. When {{XceiverClient}} gets closed, 
> {{group.shutdownGracefully()}} gets called, which is an asynchronous call. 
> A problem is that this asynchronous call has default quiet period of 2 
> seconds before it actually shutdown, so if we have a burst of read/write 
> calls, we would end up having threads created faster than they got 
> terminated, reaching system limit at some point.
> Ideally, this needs to be fixed with cached clients instead of creating new 
> thread each time. This JIRA only tries to give a temporary fix for the time 
> being.
> Thanks [~anu] for the offline discussion.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10932) Ozone : fix XceiverClient slow shutdown

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15537451#comment-15537451
 ] 

Hadoop QA commented on HDFS-10932:
--

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
22s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red}  0m  
0s{color} | {color:red} The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  7m 
27s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
53s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
28s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
56s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
12s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
56s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
18s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
 9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  2m  
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
17s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 61m 49s{color} 
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
21s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 83m 30s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestCrcCorruption |
|   | hadoop.hdfs.server.namenode.ha.TestEditLogsDuringFailover |
|   | hadoop.hdfs.server.datanode.TestBlockPoolManager |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:9560f25 |
| JIRA Issue | HDFS-10932 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12831170/HDFS-10932-HDFS-7240.002.patch
 |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  |
| uname | Linux cc2000c7987f 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed 
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | HDFS-7240 / 595257e |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| unit | 
https://builds.apache.org/job/PreCommit-HDFS-Build/16954/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HDFS-Build/16954/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U: 
hadoop-hdfs-project/hadoop-hdfs |
| Console output | 
https://builds.apache.org/job/PreCommit-HDFS-Build/16954/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT   http://yetus.apache.org |


This message was automatically generated.



> Ozone : fix XceiverClient slow shutdown
> ---
>
> Key: HDFS-10932
> 

[jira] [Commented] (HDFS-10940) Reduce performance penalty of block caching when not used

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15537450#comment-15537450
 ] 

Hadoop QA commented on HDFS-10940:
--

| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
18s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  6m 
59s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 60m 
45s{color} | {color:green} hadoop-hdfs in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
19s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 80m  9s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:9560f25 |
| JIRA Issue | HDFS-10940 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12831169/HDFS-10940.patch |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  |
| uname | Linux 6286851f34d4 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 
21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / 2549ee9 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HDFS-Build/16955/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U: 
hadoop-hdfs-project/hadoop-hdfs |
| Console output | 
https://builds.apache.org/job/PreCommit-HDFS-Build/16955/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT   http://yetus.apache.org |


This message was automatically generated.



> Reduce performance penalty of block caching when not used
> -
>
> Key: HDFS-10940
> URL: https://issues.apache.org/jira/browse/HDFS-10940
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: namenode
>Affects Versions: 2.7
>Reporter: Daryn Sharp
>Assignee: Daryn Sharp
> Attachments: HDFS-10940.patch
>
>
> For every block location generated, the CacheManager will create a junk 
> object for a hash lookup of cached locations.  If there are no cached blocks, 
> none of this is 

[jira] [Updated] (HDFS-10942) Incorrect handling of flushing edit logs to JN

[Yongjun Zhang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-10942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yongjun Zhang updated HDFS-10942:
-
Description: 
We use EditsDoubleBuffer to handle edit logs:
{code}
/**
 * A double-buffer for edits. New edits are written into the first buffer
 * while the second is available to be flushed. Each time the double-buffer
 * is flushed, the two internal buffers are swapped. This allows edits
 * to progress concurrently to flushes without allocating new buffers each
 * time.
 */
{code}

With the following code, that flush the ready buffer, it copy the ready buffer 
to a local copy, then flush.

{code}

QuarumOutputStream (buf in the code below is an instance of EditsDoubleBuffer):

  @Override
  protected void flushAndSync(boolean durable) throws IOException {
int numReadyBytes = buf.countReadyBytes();
if (numReadyBytes > 0) {
  int numReadyTxns = buf.countReadyTxns();
  long firstTxToFlush = buf.getFirstReadyTxId();

  assert numReadyTxns > 0;

  // Copy from our double-buffer into a new byte array. This is for
  // two reasons:
  // 1) The IPC code has no way of specifying to send only a slice of
  //a larger array.
  // 2) because the calls to the underlying nodes are asynchronous, we
  //need a defensive copy to avoid accidentally mutating the buffer
  //before it is sent.
  DataOutputBuffer bufToSend = new DataOutputBuffer(numReadyBytes);
  buf.flushTo(bufToSend);
  assert bufToSend.getLength() == numReadyBytes;
  byte[] data = bufToSend.getData();
  assert data.length == bufToSend.getLength();
{code}

 The above call doesn't seem to prevent the orginal copy of the buffer inside 
buf from being swapped by  the following method. 

{code}
EditsDoubleBuffer:

 public void setReadyToFlush() {
assert isFlushed() : "previous data not flushed yet";
TxnBuffer tmp = bufReady;
bufReady = bufCurrent;
bufCurrent = tmp;
  }

{code}

Though we have some runtime assertion in the code, the assertion is not enabled 
in production, so the expected condition in the assert may be false at runtime. 
This would possibly cause a mess.  When any condition is not as expected by the 
assertion, it seems a real exception should be thrown instead.

So two issues in summary:
- How we synchronize between the flush and the swap of the two buffers
- Whether we should throw real exception instead of the assert that's disabled 
at runtime normally.

A possible third issue:
* If NN crashes for some reason, before the edits in bufCurrent gets to 
bufReady and flushed, these edits in will be lost. 


  was:
We use EditsDoubleBuffer to handle edit logs:
{code}
/**
 * A double-buffer for edits. New edits are written into the first buffer
 * while the second is available to be flushed. Each time the double-buffer
 * is flushed, the two internal buffers are swapped. This allows edits
 * to progress concurrently to flushes without allocating new buffers each
 * time.
 */
{code}

With the following code, that flush the ready buffer, it copy the ready buffer 
to a local copy, then flush.

{code}

QuarumOutputStream (buf in the code below is an instance of EditsDoubleBuffer):

  @Override
  protected void flushAndSync(boolean durable) throws IOException {
int numReadyBytes = buf.countReadyBytes();
if (numReadyBytes > 0) {
  int numReadyTxns = buf.countReadyTxns();
  long firstTxToFlush = buf.getFirstReadyTxId();

  assert numReadyTxns > 0;

  // Copy from our double-buffer into a new byte array. This is for
  // two reasons:
  // 1) The IPC code has no way of specifying to send only a slice of
  //a larger array.
  // 2) because the calls to the underlying nodes are asynchronous, we
  //need a defensive copy to avoid accidentally mutating the buffer
  //before it is sent.
  DataOutputBuffer bufToSend = new DataOutputBuffer(numReadyBytes);
  buf.flushTo(bufToSend);
  assert bufToSend.getLength() == numReadyBytes;
  byte[] data = bufToSend.getData();
  assert data.length == bufToSend.getLength();
{code}

 The above call doesn't seem to prevent the orginal copy of the buffer inside 
buf from being swapped by  the following method. 

{code}
EditsDoubleBuffer:

 public void setReadyToFlush() {
assert isFlushed() : "previous data not flushed yet";
TxnBuffer tmp = bufReady;
bufReady = bufCurrent;
bufCurrent = tmp;
  }

{code}

Though we have some runtime assertion in the code, the assertion is not enabled 
in production, so the expected condition in the assert may be false at runtime. 
This would possibly cause a mess.  When any condition is not as expected by the 
assertion, it seems a real exception should be thrown instead.

So two issues in summary:
- How we synchronize between the flush and the swap of the two buffers
- Whether we should throw real exception 

[jira] [Updated] (HDFS-10923) Make InstrumentedLock require ReentrantLock

[Arpit Agarwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-10923?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Arpit Agarwal updated HDFS-10923:
-
Attachment: HDFS-10923.02.patch

v02 patch fixes checkstyle issues.

> Make InstrumentedLock require ReentrantLock
> ---
>
> Key: HDFS-10923
> URL: https://issues.apache.org/jira/browse/HDFS-10923
> Project: Hadoop HDFS
>  Issue Type: Bug
>Reporter: Arpit Agarwal
>Assignee: Arpit Agarwal
> Attachments: HDFS-10923.01.patch, HDFS-10923.02.patch
>
>
> Make InstrumentedLock use ReentrantLock instead of Lock, so nested 
> acquire/release calls can be instrumented correctly.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10713) Throttle FsNameSystem lock warnings

[Arpit Agarwal (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15537407#comment-15537407
 ] 

Arpit Agarwal commented on HDFS-10713:
--

Thanks for the backport [~zhz].

> Throttle FsNameSystem lock warnings
> ---
>
> Key: HDFS-10713
> URL: https://issues.apache.org/jira/browse/HDFS-10713
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: logging, namenode
>Reporter: Arpit Agarwal
>Assignee: Hanisha Koneru
> Fix For: 2.8.0, 2.7.4, 3.0.0-alpha2
>
> Attachments: HDFS-10713.000.patch, HDFS-10713.001.patch, 
> HDFS-10713.002.patch, HDFS-10713.003.patch, HDFS-10713.004.patch, 
> HDFS-10713.005.patch, HDFS-10713.006.patch, HDFS-10713.007.patch, 
> HDFS-10713.008.patch, HDFS-10713.009.patch, HDFS-10713.010.patch, 
> HDFS-10713.011.patch, HDFS-10713.012.patch, HDFS-10713.branch-2.000.patch
>
>
> The NameNode logs a message if the FSNamesystem write lock is held by a 
> thread for over 1 second. These messages can be throttled to at one most one 
> per x minutes to avoid potentially filling up NN logs. We can also log the 
> number of suppressed notices since the last log message.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-10797) Disk usage summary of snapshots causes renamed blocks to get counted twice

[Sean Mackrory (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-10797?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sean Mackrory updated HDFS-10797:
-
Attachment: HDFS-10797.009.patch

Fixing style issues, and always using getCounts() to retrieve counts for thread 
safety (which I noticed actually is important here, due to the yield() 
function).

> Disk usage summary of snapshots causes renamed blocks to get counted twice
> --
>
> Key: HDFS-10797
> URL: https://issues.apache.org/jira/browse/HDFS-10797
> Project: Hadoop HDFS
>  Issue Type: Bug
>Reporter: Sean Mackrory
>Assignee: Sean Mackrory
> Attachments: HDFS-10797.001.patch, HDFS-10797.002.patch, 
> HDFS-10797.003.patch, HDFS-10797.004.patch, HDFS-10797.005.patch, 
> HDFS-10797.006.patch, HDFS-10797.007.patch, HDFS-10797.008.patch, 
> HDFS-10797.009.patch
>
>
> DirectoryWithSnapshotFeature.computeContentSummary4Snapshot calculates how 
> much disk usage is used by a snapshot by tallying up the files in the 
> snapshot that have since been deleted (that way it won't overlap with regular 
> files whose disk usage is computed separately). However that is determined 
> from a diff that shows moved (to Trash or otherwise) or renamed files as a 
> deletion and a creation operation that may overlap with the list of blocks. 
> Only the deletion operation is taken into consideration, and this causes 
> those blocks to get represented twice in the disk usage tallying.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10896) Move lock logging logic from FSNamesystem into FSNamesystemLock

[Zhe Zhang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15537400#comment-15537400
 ] 

Zhe Zhang commented on HDFS-10896:
--

I committed to trunk~branch-2.8. But the backport the branch-2.7 is pretty 
messy. [~xkrogen] Do you mind posting a branch-2.7 patch? Or suggest additional 
backports to branch-2.7 to help resolve conflict?

> Move lock logging logic from FSNamesystem into FSNamesystemLock
> ---
>
> Key: HDFS-10896
> URL: https://issues.apache.org/jira/browse/HDFS-10896
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: namenode
>Reporter: Erik Krogen
>Assignee: Erik Krogen
>  Labels: logging, namenode
> Fix For: 2.8.0, 3.0.0-alpha2
>
> Attachments: HDFS-10896.000.patch, HDFS-10896.001.patch, 
> HDFS-10896.002.patch, HDFS-10896.003.patch, HDFS-10896.004.patch
>
>
> There are a number of tickets (HDFS-10742, HDFS-10817, HDFS-10713, this 
> subtask's story HDFS-10475) which are adding/improving logging/metrics around 
> the {{FSNamesystemLock}}. All of this is done in {{FSNamesystem}} right now, 
> which is polluting the namesystem with ThreadLocal variables, timing 
> counters, etc. which are only relevant to the lock itself and the number of 
> these increases as the logging/metrics become more sophisticated. It would be 
> best to move these all into FSNamesystemLock to keep the metrics/logging tied 
> directly to the item of interest. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-10896) Move lock logging logic from FSNamesystem into FSNamesystemLock

[Zhe Zhang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-10896?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zhe Zhang updated HDFS-10896:
-
Fix Version/s: 3.0.0-alpha2
   2.8.0

> Move lock logging logic from FSNamesystem into FSNamesystemLock
> ---
>
> Key: HDFS-10896
> URL: https://issues.apache.org/jira/browse/HDFS-10896
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: namenode
>Reporter: Erik Krogen
>Assignee: Erik Krogen
>  Labels: logging, namenode
> Fix For: 2.8.0, 3.0.0-alpha2
>
> Attachments: HDFS-10896.000.patch, HDFS-10896.001.patch, 
> HDFS-10896.002.patch, HDFS-10896.003.patch, HDFS-10896.004.patch
>
>
> There are a number of tickets (HDFS-10742, HDFS-10817, HDFS-10713, this 
> subtask's story HDFS-10475) which are adding/improving logging/metrics around 
> the {{FSNamesystemLock}}. All of this is done in {{FSNamesystem}} right now, 
> which is polluting the namesystem with ThreadLocal variables, timing 
> counters, etc. which are only relevant to the lock itself and the number of 
> these increases as the logging/metrics become more sophisticated. It would be 
> best to move these all into FSNamesystemLock to keep the metrics/logging tied 
> directly to the item of interest. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-10872) Add MutableRate metrics for FSNamesystemLock operations

[Erik Krogen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-10872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Erik Krogen updated HDFS-10872:
---
Attachment: HDFS-10872.000.patch

> Add MutableRate metrics for FSNamesystemLock operations
> ---
>
> Key: HDFS-10872
> URL: https://issues.apache.org/jira/browse/HDFS-10872
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: namenode
>Reporter: Erik Krogen
>Assignee: Erik Krogen
> Attachments: HDFS-10872.000.patch
>
>
> Add metrics for FSNamesystemLock operations to see, overall, how long each 
> operation is holding the lock for. Use MutableRate metrics for now. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Created] (HDFS-10942) Incorrect handling of flushing edit logs to JN

[Yongjun Zhang (JIRA)]

Yongjun Zhang created HDFS-10942:


 Summary: Incorrect handling of flushing edit logs to JN
 Key: HDFS-10942
 URL: https://issues.apache.org/jira/browse/HDFS-10942
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: s, hdfs
Reporter: Yongjun Zhang


We use EditsDoubleBuffer to handle edit logs:
{code}
/**
 * A double-buffer for edits. New edits are written into the first buffer
 * while the second is available to be flushed. Each time the double-buffer
 * is flushed, the two internal buffers are swapped. This allows edits
 * to progress concurrently to flushes without allocating new buffers each
 * time.
 */
{code}

With the following code, that flush the ready buffer, it copy the ready buffer 
to a local copy, then flush.

{code}

QuarumOutputStream (buf in the code below is an instance of EditsDoubleBuffer):

  @Override
  protected void flushAndSync(boolean durable) throws IOException {
int numReadyBytes = buf.countReadyBytes();
if (numReadyBytes > 0) {
  int numReadyTxns = buf.countReadyTxns();
  long firstTxToFlush = buf.getFirstReadyTxId();

  assert numReadyTxns > 0;

  // Copy from our double-buffer into a new byte array. This is for
  // two reasons:
  // 1) The IPC code has no way of specifying to send only a slice of
  //a larger array.
  // 2) because the calls to the underlying nodes are asynchronous, we
  //need a defensive copy to avoid accidentally mutating the buffer
  //before it is sent.
  DataOutputBuffer bufToSend = new DataOutputBuffer(numReadyBytes);
  buf.flushTo(bufToSend);
  assert bufToSend.getLength() == numReadyBytes;
  byte[] data = bufToSend.getData();
  assert data.length == bufToSend.getLength();
{code}

 The above call doesn't seem to prevent the orginal copy of the buffer inside 
buf from being swapped by  the following method. 

{code}
EditsDoubleBuffer:

 public void setReadyToFlush() {
assert isFlushed() : "previous data not flushed yet";
TxnBuffer tmp = bufReady;
bufReady = bufCurrent;
bufCurrent = tmp;
  }

{code}

Though we have some runtime assertion in the code, the assertion is not enabled 
in production, so the expected condition in the assert may be false at runtime. 
This would possibly cause a mess.  When any condition is not as expected by the 
assertion, it seems a real exception should be thrown instead.

So two issues in summary:
- How we synchronize between the flush and the swap of the two buffers
- Whether we should throw real exception instead of the assert that's disabled 
at runtime normally.





--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10713) Throttle FsNameSystem lock warnings

[Zhe Zhang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15537395#comment-15537395
 ] 

Zhe Zhang commented on HDFS-10713:
--

I backported this change to branch-2.7 since the original read/write lock 
logging patches are there.

> Throttle FsNameSystem lock warnings
> ---
>
> Key: HDFS-10713
> URL: https://issues.apache.org/jira/browse/HDFS-10713
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: logging, namenode
>Reporter: Arpit Agarwal
>Assignee: Hanisha Koneru
> Fix For: 2.8.0, 2.7.4, 3.0.0-alpha2
>
> Attachments: HDFS-10713.000.patch, HDFS-10713.001.patch, 
> HDFS-10713.002.patch, HDFS-10713.003.patch, HDFS-10713.004.patch, 
> HDFS-10713.005.patch, HDFS-10713.006.patch, HDFS-10713.007.patch, 
> HDFS-10713.008.patch, HDFS-10713.009.patch, HDFS-10713.010.patch, 
> HDFS-10713.011.patch, HDFS-10713.012.patch, HDFS-10713.branch-2.000.patch
>
>
> The NameNode logs a message if the FSNamesystem write lock is held by a 
> thread for over 1 second. These messages can be throttled to at one most one 
> per x minutes to avoid potentially filling up NN logs. We can also log the 
> number of suppressed notices since the last log message.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-10713) Throttle FsNameSystem lock warnings

[Zhe Zhang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-10713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zhe Zhang updated HDFS-10713:
-
Fix Version/s: 2.7.4

> Throttle FsNameSystem lock warnings
> ---
>
> Key: HDFS-10713
> URL: https://issues.apache.org/jira/browse/HDFS-10713
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: logging, namenode
>Reporter: Arpit Agarwal
>Assignee: Hanisha Koneru
> Fix For: 2.8.0, 2.7.4, 3.0.0-alpha2
>
> Attachments: HDFS-10713.000.patch, HDFS-10713.001.patch, 
> HDFS-10713.002.patch, HDFS-10713.003.patch, HDFS-10713.004.patch, 
> HDFS-10713.005.patch, HDFS-10713.006.patch, HDFS-10713.007.patch, 
> HDFS-10713.008.patch, HDFS-10713.009.patch, HDFS-10713.010.patch, 
> HDFS-10713.011.patch, HDFS-10713.012.patch, HDFS-10713.branch-2.000.patch
>
>
> The NameNode logs a message if the FSNamesystem write lock is held by a 
> thread for over 1 second. These messages can be throttled to at one most one 
> per x minutes to avoid potentially filling up NN logs. We can also log the 
> number of suppressed notices since the last log message.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10872) Add MutableRate metrics for FSNamesystemLock operations

[Erik Krogen (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15537392#comment-15537392
 ] 

Erik Krogen commented on HDFS-10872:


Attaching v000 patch.

This makes {{FSNamesystemLock}} into a {{MetricsSource}}. When the lock is 
released, the amount of time for which the lock was held is recorded as a 
metric corresponding to the specific operation which caused the lock to be 
held. This was achieved by manually specifying the operation in calls to 
{{FSNamesystemLock.read/writeLock}} since there is no easy way to tell what 
operation caused the lock's acquisition from within {{FSNamesystemLock}}. Lock 
releases which do not specify an operation are filed under "OTHER". There are a 
number of lock usages within {{BlockManager}} which have this classification 
because {{BlockManager}} only views the {{Namesystem}} API, not 
{{FSNamesystem}}, and I did not want to modify that API. I am open to 
suggestions here. 

Since this occurs along the critical path and may add overhead, it is disabled 
by default. To minimize overhead, each thread stores a local copy of the 
metrics. Every so often, set by a configurable interval, each thread will 
insert the metrics it has accumulated into a shared {{MetricsRegistry}}. I am 
doing some microbenchmark performance testing now and will post results when 
they are available. 

> Add MutableRate metrics for FSNamesystemLock operations
> ---
>
> Key: HDFS-10872
> URL: https://issues.apache.org/jira/browse/HDFS-10872
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: namenode
>Reporter: Erik Krogen
>Assignee: Erik Krogen
>
> Add metrics for FSNamesystemLock operations to see, overall, how long each 
> operation is holding the lock for. Use MutableRate metrics for now. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10429) DataStreamer interrupted warning always appears when using CLI upload file

[Wei-Chiu Chuang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15537327#comment-15537327
 ] 

Wei-Chiu Chuang commented on HDFS-10429:


Ping: [~jingzhao]'s suggestion sounds good to me. [~aplusplus] would you like 
to submit a new patch? Thanks a lot.

> DataStreamer interrupted warning  always appears when using CLI upload file
> ---
>
> Key: HDFS-10429
> URL: https://issues.apache.org/jira/browse/HDFS-10429
> Project: Hadoop HDFS
>  Issue Type: Bug
>Affects Versions: 2.7.3
>Reporter: Zhiyuan Yang
>Assignee: Zhiyuan Yang
>Priority: Minor
> Attachments: HDFS-10429.1.patch
>
>
> Every time I use 'hdfs dfs -put' upload file, this warning is printed:
> {code:java}
> 16/05/18 20:57:56 WARN hdfs.DataStreamer: Caught exception
> java.lang.InterruptedException
>   at java.lang.Object.wait(Native Method)
>   at java.lang.Thread.join(Thread.java:1245)
>   at java.lang.Thread.join(Thread.java:1319)
>   at 
> org.apache.hadoop.hdfs.DataStreamer.closeResponder(DataStreamer.java:871)
>   at org.apache.hadoop.hdfs.DataStreamer.endBlock(DataStreamer.java:519)
>   at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:696)
> {code}
> The reason is this: originally, DataStreamer::closeResponder always prints a 
> warning about InterruptedException; since HDFS-9812, 
> DFSOutputStream::closeImpl  always forces threads to close, which causes 
> InterruptedException.
> A simple fix is to use debug level log instead of warning level.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-10872) Add MutableRate metrics for FSNamesystemLock operations

[Erik Krogen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-10872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Erik Krogen updated HDFS-10872:
---
Status: Open  (was: Patch Available)

> Add MutableRate metrics for FSNamesystemLock operations
> ---
>
> Key: HDFS-10872
> URL: https://issues.apache.org/jira/browse/HDFS-10872
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: namenode
>Reporter: Erik Krogen
>Assignee: Erik Krogen
>
> Add metrics for FSNamesystemLock operations to see, overall, how long each 
> operation is holding the lock for. Use MutableRate metrics for now. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-10872) Add MutableRate metrics for FSNamesystemLock operations

[Erik Krogen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-10872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Erik Krogen updated HDFS-10872:
---
Status: Patch Available  (was: In Progress)

> Add MutableRate metrics for FSNamesystemLock operations
> ---
>
> Key: HDFS-10872
> URL: https://issues.apache.org/jira/browse/HDFS-10872
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: namenode
>Reporter: Erik Krogen
>Assignee: Erik Krogen
>
> Add metrics for FSNamesystemLock operations to see, overall, how long each 
> operation is holding the lock for. Use MutableRate metrics for now. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10637) Modifications to remove the assumption that FsVolumes are backed by java.io.File.

[Lei (Eddy) Xu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15537272#comment-15537272
 ] 

Lei (Eddy) Xu commented on HDFS-10637:
--

Yes. That's right.

Thanks a lot!

> Modifications to remove the assumption that FsVolumes are backed by 
> java.io.File.
> -
>
> Key: HDFS-10637
> URL: https://issues.apache.org/jira/browse/HDFS-10637
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: datanode, fs
>Reporter: Virajith Jalaparti
>Assignee: Virajith Jalaparti
> Attachments: HDFS-10637.001.patch, HDFS-10637.002.patch, 
> HDFS-10637.003.patch, HDFS-10637.004.patch, HDFS-10637.005.patch, 
> HDFS-10637.006.patch, HDFS-10637.007.patch, HDFS-10637.008.patch, 
> HDFS-10637.009.patch
>
>
> Modifications to {{FsVolumeSpi}} and {{FsVolumeImpl}} to remove references to 
> {{java.io.File}}.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10637) Modifications to remove the assumption that FsVolumes are backed by java.io.File.

[Virajith Jalaparti (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15537266#comment-15537266
 ] 

Virajith Jalaparti commented on HDFS-10637:
---

Hi [~eddyxu], 

bq. what about setting {{FsVolumeImpl#storageStorage}} as {{final}} ...

I assume you meant {{FsVolumeImpl#storageLocation}} here? Yes, I think this 
makes sense. So, the changes will be the following:
1) Have only one constructor for {{FsVolumeImpl}}, which takes in as input the 
{{StorageDirectory}}. This would involve moving the code in the {{private 
FsVolumeImpl(...String currentDir...StorageLocation storageLocation)}} to the 
{{FsVolumeImpl(...,StorageDirectory sd, ...)}} constructor. 

2) Declare both {{FsVolumeImpl#storageLocation}}, and 
{{FsVolumeImpl#currentDir}} as final.

The above changes will ensure that both are initialized consistently, and they 
are not modified again. 

Is this what you had in mind? 

I will post the updated patch soon. 






> Modifications to remove the assumption that FsVolumes are backed by 
> java.io.File.
> -
>
> Key: HDFS-10637
> URL: https://issues.apache.org/jira/browse/HDFS-10637
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: datanode, fs
>Reporter: Virajith Jalaparti
>Assignee: Virajith Jalaparti
> Attachments: HDFS-10637.001.patch, HDFS-10637.002.patch, 
> HDFS-10637.003.patch, HDFS-10637.004.patch, HDFS-10637.005.patch, 
> HDFS-10637.006.patch, HDFS-10637.007.patch, HDFS-10637.008.patch, 
> HDFS-10637.009.patch
>
>
> Modifications to {{FsVolumeSpi}} and {{FsVolumeImpl}} to remove references to 
> {{java.io.File}}.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10637) Modifications to remove the assumption that FsVolumes are backed by java.io.File.

[Lei (Eddy) Xu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15537214#comment-15537214
 ] 

Lei (Eddy) Xu commented on HDFS-10637:
--

Hi, [~virajith]  Thanks a lot for update the patch. It looks really good.

bq. currentDir is now just a private variable that holds this value. 

My only concerns is that, because {{currentDir == storageLocation + 
STORAGE_DIR_CURRENT}}, putting both of them into {{FsVolumeImpl}}, introduces 
the chance of inconsistent states (i.e., {{currentDir != storageLocation + 
/current}}).  Alternatively, what about setting {{FsVolumeImpl#storageStorage}} 
as {{final}}, and do not introduce {{private FsVolumeImpl(...String 
currentDir...StorageLocation storageLocation)}}, which will likely become 
{{public}}  later by other people.  So that you can consider {{currentDir}} is 
a cached value of {{storageLocation}} or vice versa. 

What do you think?



> Modifications to remove the assumption that FsVolumes are backed by 
> java.io.File.
> -
>
> Key: HDFS-10637
> URL: https://issues.apache.org/jira/browse/HDFS-10637
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: datanode, fs
>Reporter: Virajith Jalaparti
>Assignee: Virajith Jalaparti
> Attachments: HDFS-10637.001.patch, HDFS-10637.002.patch, 
> HDFS-10637.003.patch, HDFS-10637.004.patch, HDFS-10637.005.patch, 
> HDFS-10637.006.patch, HDFS-10637.007.patch, HDFS-10637.008.patch, 
> HDFS-10637.009.patch
>
>
> Modifications to {{FsVolumeSpi}} and {{FsVolumeImpl}} to remove references to 
> {{java.io.File}}.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10797) Disk usage summary of snapshots causes renamed blocks to get counted twice

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15537211#comment-15537211
 ] 

Hadoop QA commented on HDFS-10797:
--

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
13s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  6m 
59s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
52s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
43s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
0m 26s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch 
generated 3 new + 267 unchanged - 0 fixed = 270 total (was 267) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  1m 
54s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs generated 1 new + 0 
unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
53s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 58m 42s{color} 
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
18s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 78m 16s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-hdfs-project/hadoop-hdfs |
|  |  Inconsistent synchronization of 
org.apache.hadoop.hdfs.server.namenode.ContentSummaryComputationContext.counts; 
locked 55% of time  Unsynchronized access at 
ContentSummaryComputationContext.java:55% of time  Unsynchronized access at 
ContentSummaryComputationContext.java:[line 91] |
| Failed junit tests | hadoop.hdfs.TestDFSShell |
|   | hadoop.hdfs.TestFileCorruption |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:9560f25 |
| JIRA Issue | HDFS-10797 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12831164/HDFS-10797.008.patch |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  |
| uname | Linux 7ddd29c9e58a 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 
21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / 2549ee9 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| checkstyle | 
https://builds.apache.org/job/PreCommit-HDFS-Build/16953/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
 |
| findbugs | 
https://builds.apache.org/job/PreCommit-HDFS-Build/16953/artifact/patchprocess/new-findbugs-hadoop-hdfs-project_hadoop-hdfs.html
 |
| unit | 

[jira] [Commented] (HDFS-10851) FSDirStatAndListingOp: stop passing path as string

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10851?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15537195#comment-15537195
 ] 

Hadoop QA commented on HDFS-10851:
--

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 13m  
6s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  8m 
59s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
43s{color} | {color:green} branch-2 passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
46s{color} | {color:green} branch-2 passed with JDK v1.7.0_111 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
31s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
54s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
17s{color} | {color:green} branch-2 passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  2m  
0s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs in branch-2 has 1 
extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
55s{color} | {color:green} branch-2 passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
38s{color} | {color:green} branch-2 passed with JDK v1.7.0_111 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
39s{color} | {color:green} the patch passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
42s{color} | {color:green} the patch passed with JDK v1.7.0_111 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
42s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
0m 27s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch 
generated 2 new + 175 unchanged - 15 fixed = 177 total (was 190) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 1s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  2m  
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
55s{color} | {color:green} the patch passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
35s{color} | {color:green} the patch passed with JDK v1.7.0_111 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 54m  9s{color} 
| {color:red} hadoop-hdfs in the patch failed with JDK v1.7.0_111. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
23s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}146m 37s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.7.0_111 Failed junit tests | 
hadoop.hdfs.server.namenode.TestDiskspaceQuotaUpdate |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:b59b8b7 |
| JIRA Issue | HDFS-10851 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12831149/HDFS-10851.branch-2.patch
 |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  |
| uname | Linux e07911c21f60 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 
17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 

[jira] [Created] (HDFS-10941) Improve BlockManager#processMisReplicatesAsync log

[Xiaoyu Yao (JIRA)]

Xiaoyu Yao created HDFS-10941:
-

 Summary: Improve BlockManager#processMisReplicatesAsync log
 Key: HDFS-10941
 URL: https://issues.apache.org/jira/browse/HDFS-10941
 Project: Hadoop HDFS
  Issue Type: Improvement
  Components: namenode
Reporter: Xiaoyu Yao
Assignee: Xiaoyu Yao


BlockManager#processMisReplicatesAsync is the daemon thread running inside 
namenode to handle miserplicated blocks. As shown below, it has a trace log for 
each of the block in the cluster being processed (1 blocks per iteration 
after sleep 10s). 
{code}
  MisReplicationResult res = processMisReplicatedBlock(block);
  if (LOG.isTraceEnabled()) {
LOG.trace("block " + block + ": " + res);
  }
{code}
However, it is not very useful as dumping every block in the cluster will 
overwhelm the namenode log without much useful information assuming the 
majority of the blocks are not over/under replicated. This ticket is opened to 
improve the log for easy troubleshooting of block replication related issues by:
 
1) add debug log for blocks that get under/over replicated result during 
{{processMisReplicatedBlock()}} 

2) or change to trace log for only blocks that get non-OK result during 
{{processMisReplicatedBlock()}} 




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10934) TestDFSShell.testStat fails intermittently

[Mingliang Liu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15537138#comment-15537138
 ] 

Mingliang Liu commented on HDFS-10934:
--

What I proposed is some code snippet liket:
{code}
diff --git 
a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSShell.java
 
b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSShell.java
index fc90db5..558bcda 100644
--- 
a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSShell.java
+++ 
b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSShell.java
@@ -2127,11 +2127,11 @@ public void testStat() throws Exception {
   fmt.setTimeZone(TimeZone.getTimeZone("UTC"));
   final Path testDir1 = new Path("testStat", "dir1");
   dfs.mkdirs(testDir1);
-  final FileStatus status1 = dfs.getFileStatus(testDir1);
-  final String mtime1 = fmt.format(new 
Date(status1.getModificationTime()));
   final Path testFile2 = new Path(testDir1, "file2");
   DFSTestUtil.createFile(dfs, testFile2, 2 * blockSize, (short) 3, 0);
-  final FileStatus status2 = dfs.getFileStatus(testDir1);
+  final FileStatus status1 = dfs.getFileStatus(testDir1);
+  final String mtime1 = fmt.format(new 
Date(status1.getModificationTime()));
+  final FileStatus status2 = dfs.getFileStatus(testFile2);
   final String mtime2 = fmt.format(new 
Date(status2.getModificationTime()));
 
   final ByteArrayOutputStream out = new ByteArrayOutputStream();
{code}
Do you think it makes sense or do you provide a similar patch? Thanks.

> TestDFSShell.testStat fails intermittently
> --
>
> Key: HDFS-10934
> URL: https://issues.apache.org/jira/browse/HDFS-10934
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: test
>Reporter: Eric Badger
>Assignee: Eric Badger
> Attachments: HDFS-10934.001.patch
>
>
> Saw this failure in an internal build. Reran the test 30 times and it failed 
> once with the same type of failure. 
> {noformat}
> org.junit.ComparisonFailure: Unexpected -stat output: 2016-09-30 03:48:56
> 2016-09-30 03:48:57
>  expected:<...6
> 2016-09-30 03:48:5[7]
> > but was:<...6
> 2016-09-30 03:48:5[6]
> >
>   at org.junit.Assert.assertEquals(Assert.java:115)
>   at org.apache.hadoop.hdfs.TestDFSShell.testStat(TestDFSShell.java:2082)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-10893) Refactor TestDFSShell by setting up MiniDFSCluser once for all commands test

[Mingliang Liu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-10893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mingliang Liu updated HDFS-10893:
-
Attachment: HDFS-10893-branch-2.001.patch

> Refactor TestDFSShell by setting up MiniDFSCluser once for all commands test
> 
>
> Key: HDFS-10893
> URL: https://issues.apache.org/jira/browse/HDFS-10893
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: test
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
> Attachments: HDFS-10893-branch-2.000.patch, 
> HDFS-10893-branch-2.001.patch, HDFS-10893.000.patch, HDFS-10893.001.patch
>
>
> It seems that setting up MiniDFSCluser once for all commands test will reduce 
> the total time. To share a global cluster, the tests should use individual 
> test directories to avoid conflict between test cases. Meanwhile, the 
> MiniDFSCluster should not use the default root data directory; or else tests 
> are not able to launch another cluster(s) by default.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10896) Move lock logging logic from FSNamesystem into FSNamesystemLock

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15537116#comment-15537116
 ] 

Hudson commented on HDFS-10896:
---

SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10522 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/10522/])
HDFS-10896. Move lock logging logic from FSNamesystem into (zhz: rev 
434c5ea75dc3d87513e49290ac148ff5163c)
* (edit) 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
* (edit) 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystemLock.java
* (add) 
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSNamesystemLock.java
* (edit) 
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSNamesystem.java


> Move lock logging logic from FSNamesystem into FSNamesystemLock
> ---
>
> Key: HDFS-10896
> URL: https://issues.apache.org/jira/browse/HDFS-10896
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: namenode
>Reporter: Erik Krogen
>Assignee: Erik Krogen
>  Labels: logging, namenode
> Attachments: HDFS-10896.000.patch, HDFS-10896.001.patch, 
> HDFS-10896.002.patch, HDFS-10896.003.patch, HDFS-10896.004.patch
>
>
> There are a number of tickets (HDFS-10742, HDFS-10817, HDFS-10713, this 
> subtask's story HDFS-10475) which are adding/improving logging/metrics around 
> the {{FSNamesystemLock}}. All of this is done in {{FSNamesystem}} right now, 
> which is polluting the namesystem with ThreadLocal variables, timing 
> counters, etc. which are only relevant to the lock itself and the number of 
> these increases as the logging/metrics become more sophisticated. It would be 
> best to move these all into FSNamesystemLock to keep the metrics/logging tied 
> directly to the item of interest. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-10931) libhdfs++: Fix object lifecycle issues in the BlockReader

[James Clampffer (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-10931?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

James Clampffer updated HDFS-10931:
---
Attachment: HDFS-10931.HDFS-8707.001.patch

New patch.  The last patch would cause sporadic failures due to errors injected 
into the read pipeline during the mini stress test.  In that test we run the 
stress test twice: 1 with error injection and 1 without.  I've removed the 
EXPECT_ZERO on the failure count of the error injected case because while the 
RPC engine can generally retry on errors however the read path wont and is 
bound to cause false positives.  What the error injected test is really looking 
for is crashes due to unexpected states in the read case.

To make testing more robust in the future we could use separate error injection 
for the read pipeline and the RPC layer.  Then it would be possible to assert 
that the number of failed reads is <= the number of injected read pipeline 
errors.  Barring a large run of consecutive simulated failures RPC is unlikely 
to cause a false positive.

> libhdfs++: Fix object lifecycle issues in the BlockReader
> -
>
> Key: HDFS-10931
> URL: https://issues.apache.org/jira/browse/HDFS-10931
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: hdfs-client
>Reporter: James Clampffer
>Assignee: James Clampffer
>Priority: Critical
> Attachments: HDFS-10931.HDFS-8707.000.patch, 
> HDFS-10931.HDFS-8707.001.patch
>
>
> The BlockReader can work itself into a a state during AckRead (possibly other 
> stages as well) where the pipeline posts a task for asio with a pointer back 
> into itself, then promptly calls "delete this" without canceling the asio 
> request.  The asio task finishes and tries to acquire the lock at the address 
> where the DataNodeConnection used to live - but the DN connection is no 
> longer valid so it's scribbling on some arbitrary bit of memory.  On some 
> platforms the underlying address used by the mutex state will be handed out 
> to future mutexes so the scribble breaks that state and all the locks in that 
> process start misbehaving.
> This can be reproduced by using the patch from HDFS-8790 and adding more 
> worker threads + a lot more reader threads.
> I'm going to fix this in two parts:
> 1) Duct tape + superglue patch to make sure that all top level continuations 
> in the block reader pipeline hold a shared_ptr to the DataNodeConnection.  
> Nested continuations also get a copy of the shared_ptr to make sure the 
> connection is alive.  This at least keeps the connection alive so that it can 
> keep returning asio::operation_aborted.
> 2) The continuation stuff needs a lot of work to make sure this type of bug 
> doesn't keep popping up.  We've already fixed these issues in the RPC code.  
> This will most likely need to be split into a few jiras.
> - Continuation "framework" can be slimmed down quite a bit, perhaps even 
> removed.  Near zero documentation + many implied contracts = constant bug 
> chasing.
> - Add comments to actually describe what's going on in the networking code.  
> This bug took significantly longer than it should have to track down because 
> I hadn't worked on the BlockReader in a while.
> - No more "delete this".
> - Flatten out nested continuations e.g. the guts of BlockReaderImpl::AckRead. 
>  It's unclear why they were implemented like this in the first place and 
> there's no comments to indicate that this was intentional.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-10893) Refactor TestDFSShell by setting up MiniDFSCluser once for all commands test

[Mingliang Liu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-10893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mingliang Liu updated HDFS-10893:
-
Attachment: HDFS-10893.001.patch

> Refactor TestDFSShell by setting up MiniDFSCluser once for all commands test
> 
>
> Key: HDFS-10893
> URL: https://issues.apache.org/jira/browse/HDFS-10893
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: test
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
> Attachments: HDFS-10893-branch-2.000.patch, HDFS-10893.000.patch, 
> HDFS-10893.001.patch
>
>
> It seems that setting up MiniDFSCluser once for all commands test will reduce 
> the total time. To share a global cluster, the tests should use individual 
> test directories to avoid conflict between test cases. Meanwhile, the 
> MiniDFSCluster should not use the default root data directory; or else tests 
> are not able to launch another cluster(s) by default.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10893) Refactor TestDFSShell by setting up MiniDFSCluser once for all commands test

[Mingliang Liu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10893?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15537096#comment-15537096
 ] 

Mingliang Liu commented on HDFS-10893:
--

V1 patch rebased from {{trunk}} branch.

> Refactor TestDFSShell by setting up MiniDFSCluser once for all commands test
> 
>
> Key: HDFS-10893
> URL: https://issues.apache.org/jira/browse/HDFS-10893
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: test
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
> Attachments: HDFS-10893-branch-2.000.patch, HDFS-10893.000.patch, 
> HDFS-10893.001.patch
>
>
> It seems that setting up MiniDFSCluser once for all commands test will reduce 
> the total time. To share a global cluster, the tests should use individual 
> test directories to avoid conflict between test cases. Meanwhile, the 
> MiniDFSCluster should not use the default root data directory; or else tests 
> are not able to launch another cluster(s) by default.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-10619) Cache path in InodesInPath

[Daryn Sharp (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-10619?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daryn Sharp updated HDFS-10619:
---
Attachment: HDFS-10619.1.patch

Updated.

> Cache path in InodesInPath
> --
>
> Key: HDFS-10619
> URL: https://issues.apache.org/jira/browse/HDFS-10619
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: hdfs
>Reporter: Daryn Sharp
>Assignee: Daryn Sharp
> Attachments: HDFS-10619.1.patch, HDFS-10619.patch
>
>
> INodesInPath#getPath, a frequently called method, dynamically builds the 
> path.  IIP should cache the path upon construction.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10934) TestDFSShell.testStat fails intermittently

[Mingliang Liu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15537088#comment-15537088
 ] 

Mingliang Liu commented on HDFS-10934:
--

Changing the status2 from {{testDir1}} to {{testFile2}} is not enough. We have 
to also move it after creating files (or leave it open) as creating a new file 
will change the modification time of its parent directory. I assumed the 
file/directory last modification time (though never asserted) would be the same 
because of this. And it turned out to be not reliable. You're right that 
closing the file will change the file modification time but not its parent 
directory. And they may differ.

So I thought 1) mkdir, 2) create file (and close it), 3) get file and dir 
status, and 4) do the assertions against respective mtimes were preferred to 
fix this. Thanks.

> TestDFSShell.testStat fails intermittently
> --
>
> Key: HDFS-10934
> URL: https://issues.apache.org/jira/browse/HDFS-10934
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: test
>Reporter: Eric Badger
>Assignee: Eric Badger
> Attachments: HDFS-10934.001.patch
>
>
> Saw this failure in an internal build. Reran the test 30 times and it failed 
> once with the same type of failure. 
> {noformat}
> org.junit.ComparisonFailure: Unexpected -stat output: 2016-09-30 03:48:56
> 2016-09-30 03:48:57
>  expected:<...6
> 2016-09-30 03:48:5[7]
> > but was:<...6
> 2016-09-30 03:48:5[6]
> >
>   at org.junit.Assert.assertEquals(Assert.java:115)
>   at org.apache.hadoop.hdfs.TestDFSShell.testStat(TestDFSShell.java:2082)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-10609) Uncaught InvalidEncryptionKeyException during pipeline recovery may abort downstream applications

[Wei-Chiu Chuang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-10609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wei-Chiu Chuang updated HDFS-10609:
---
Attachment: HDFS-10609.branch-2.7.02.patch

Most code style warnings are unrelated. The test failure can not be reproduce 
on my local machine. Attach a slightly updated patch to trigger precommit again.

> Uncaught InvalidEncryptionKeyException during pipeline recovery may abort 
> downstream applications
> -
>
> Key: HDFS-10609
> URL: https://issues.apache.org/jira/browse/HDFS-10609
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: encryption
>Affects Versions: 2.6.0
> Environment: CDH5.8.0
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
> Fix For: 2.8.0, 3.0.0-alpha2
>
> Attachments: HDFS-10609.001.patch, HDFS-10609.002.patch, 
> HDFS-10609.003.patch, HDFS-10609.004.patch, HDFS-10609.005.patch, 
> HDFS-10609.branch-2.7.01.patch, HDFS-10609.branch-2.7.02.patch
>
>
> In normal operations, if SASL negotiation fails due to 
> {{InvalidEncryptionKeyException}}, it is typically a benign exception, which 
> is caught and retried :
> {code:title=SaslDataTransferServer#doSaslHandshake}
>   if (ioe instanceof SaslException &&
>   ioe.getCause() != null &&
>   ioe.getCause() instanceof InvalidEncryptionKeyException) {
> // This could just be because the client is long-lived and hasn't gotten
> // a new encryption key from the NN in a while. Upon receiving this
> // error, the client will get a new encryption key from the NN and retry
> // connecting to this DN.
> sendInvalidKeySaslErrorMessage(out, ioe.getCause().getMessage());
>   } 
> {code}
> {code:title=DFSOutputStream.DataStreamer#createBlockOutputStream}
> if (ie instanceof InvalidEncryptionKeyException && refetchEncryptionKey > 0) {
> DFSClient.LOG.info("Will fetch a new encryption key and retry, " 
> + "encryption key was invalid when connecting to "
> + nodes[0] + " : " + ie);
> {code}
> However, if the exception is thrown during pipeline recovery, the 
> corresponding code does not handle it properly, and the exception is spilled 
> out to downstream applications, such as SOLR, aborting its operation:
> {quote}
> 2016-07-06 12:12:51,992 ERROR org.apache.solr.update.HdfsTransactionLog: 
> Exception closing tlog.
> org.apache.hadoop.hdfs.protocol.datatransfer.InvalidEncryptionKeyException: 
> Can't re-compute encryption key for nonce, since the required block key 
> (keyID=557709482) doesn't exist. Current key: 1350592619
> at 
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.DataTransferSaslUtil.readSaslMessageAndNegotiatedCipherOption(DataTransferSaslUtil.java:417)
> at 
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.doSaslHandshake(SaslDataTransferClient.java:474)
> at 
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.getEncryptedStreams(SaslDataTransferClient.java:299)
> at 
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.send(SaslDataTransferClient.java:242)
> at 
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.checkTrustAndSend(SaslDataTransferClient.java:211)
> at 
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.socketSend(SaslDataTransferClient.java:183)
> at 
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.transfer(DFSOutputStream.java:1308)
> at 
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1272)
> at 
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1433)
> at 
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:1147)
> at 
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:632)
> 2016-07-06 12:12:51,997 ERROR org.apache.solr.update.CommitTracker: auto 
> commit error...:org.apache.solr.common.SolrException: 
> org.apache.hadoop.hdfs.protocol.datatransfer.InvalidEncryptionKeyException: 
> Can't re-compute encryption key for nonce, since the required block key 
> (keyID=557709482) doesn't exist. Current key: 1350592619
> at 
> org.apache.solr.update.HdfsTransactionLog.close(HdfsTransactionLog.java:316)
> at 
> org.apache.solr.update.TransactionLog.decref(TransactionLog.java:505)
> at org.apache.solr.update.UpdateLog.addOldLog(UpdateLog.java:380)
> at org.apache.solr.update.UpdateLog.postCommit(UpdateLog.java:676)
> at 
> org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:623)

[jira] [Commented] (HDFS-10921) TestDiskspaceQuotaUpdate doesn't wait for NN to get out of safe mode

[Eric Badger (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15537074#comment-15537074
 ] 

Eric Badger commented on HDFS-10921:


Quick update: Adding the @Before method as [~xkrogen] proposed above does not 
work. For a reason that isn't immediately obvious to me, 
{{cluster.getNamesystem().clear()}} breaks all of the tests. Even when run on 
only a single test in the test class, clearing the namesystem before the test 
completely breaks it. If I clear the namesystem in tandem with 
{{cluster.restartNameNodes()}} then all of the tests pass, but they take ~82 
seconds to run on my machine. Restarting the cluster after every test, on the 
other hand, takes under 30 seconds total.

However, by adding the else clause to the @Before method, I was able to remove 
all of the {{restartNameNodes()}} and {{restartDataNode()}} calls in the code 
and the tests still passed. So the only problem that we have is the issue that 
[~shahrs87] raised about Namesystem pollution by bad tests. This test suite 
doesn't have any of those problems, so (unless I can figure out how to 
effectively clear the namesystem in a small amount of time without corrupting 
the tests) the question will be whether we want decreased runtime or to be 
resilient against tests written in the future that use the same file names. 

> TestDiskspaceQuotaUpdate doesn't wait for NN to get out of safe mode
> 
>
> Key: HDFS-10921
> URL: https://issues.apache.org/jira/browse/HDFS-10921
> Project: Hadoop HDFS
>  Issue Type: Bug
>Reporter: Eric Badger
>Assignee: Eric Badger
> Attachments: HDFS-10921.001.patch, HDFS-10921.002.patch
>
>
> Test fails intermittently because the NN is still in safe mode. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10932) Ozone : fix XceiverClient slow shutdown

[Anu Engineer (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15537070#comment-15537070
 ] 

Anu Engineer commented on HDFS-10932:
-

scheduled a new build 
https://builds.apache.org/job/PreCommit-HDFS-Build/1​69​54/

> Ozone : fix XceiverClient slow shutdown
> ---
>
> Key: HDFS-10932
> URL: https://issues.apache.org/jira/browse/HDFS-10932
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>Reporter: Chen Liang
>Assignee: Chen Liang
> Attachments: HDFS-10932-HDFS-7240.002.patch, 
> HDFS-10932-HDFS-7240.002.patch, HDFS-10932.001.patch
>
>
> Currently {{XceiverClient}} is the underlying entity of 
> {{DistributedStorageHandler.newKeyWriter()}} and 
> {{DistributedStorageHandler.newKeyReader()}}  for making call to container 
> for read/write. When {{XceiverClient}} gets closed, 
> {{group.shutdownGracefully()}} gets called, which is an asynchronous call. 
> A problem is that this asynchronous call has default quiet period of 2 
> seconds before it actually shutdown, so if we have a burst of read/write 
> calls, we would end up having threads created faster than they got 
> terminated, reaching system limit at some point.
> Ideally, this needs to be fixed with cached clients instead of creating new 
> thread each time. This JIRA only tries to give a temporary fix for the time 
> being.
> Thanks [~anu] for the offline discussion.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-10932) Ozone : fix XceiverClient slow shutdown

[Chen Liang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-10932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chen Liang updated HDFS-10932:
--
Attachment: HDFS-10932-HDFS-7240.002.patch

> Ozone : fix XceiverClient slow shutdown
> ---
>
> Key: HDFS-10932
> URL: https://issues.apache.org/jira/browse/HDFS-10932
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>Reporter: Chen Liang
>Assignee: Chen Liang
> Attachments: HDFS-10932-HDFS-7240.002.patch, 
> HDFS-10932-HDFS-7240.002.patch, HDFS-10932.001.patch
>
>
> Currently {{XceiverClient}} is the underlying entity of 
> {{DistributedStorageHandler.newKeyWriter()}} and 
> {{DistributedStorageHandler.newKeyReader()}}  for making call to container 
> for read/write. When {{XceiverClient}} gets closed, 
> {{group.shutdownGracefully()}} gets called, which is an asynchronous call. 
> A problem is that this asynchronous call has default quiet period of 2 
> seconds before it actually shutdown, so if we have a burst of read/write 
> calls, we would end up having threads created faster than they got 
> terminated, reaching system limit at some point.
> Ideally, this needs to be fixed with cached clients instead of creating new 
> thread each time. This JIRA only tries to give a temporary fix for the time 
> being.
> Thanks [~anu] for the offline discussion.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-10851) FSDirStatAndListingOp: stop passing path as string

[Kihwal Lee (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-10851?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kihwal Lee updated HDFS-10851:
--
   Resolution: Fixed
 Hadoop Flags: Reviewed
Fix Version/s: 3.0.0-alpha2
   2.8.0
   Status: Resolved  (was: Patch Available)

Committed to trunk, branch-2 and branch-2.8.

> FSDirStatAndListingOp: stop passing path as string
> --
>
> Key: HDFS-10851
> URL: https://issues.apache.org/jira/browse/HDFS-10851
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: hdfs
>Reporter: Daryn Sharp
>Assignee: Daryn Sharp
> Fix For: 2.8.0, 3.0.0-alpha2
>
> Attachments: HDFS-10851.1.patch, HDFS-10851.branch-2.patch, 
> HDFS-10851.patch
>
>
> Path strings should be resolved once into INodesInPath.  The IIP should be 
> used extensively from that point forward.  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10932) Ozone : fix XceiverClient slow shutdown

[Chen Liang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15537065#comment-15537065
 ] 

Chen Liang commented on HDFS-10932:
---

Sure! I updated the patch a second time to trigger the Jenkins check.

> Ozone : fix XceiverClient slow shutdown
> ---
>
> Key: HDFS-10932
> URL: https://issues.apache.org/jira/browse/HDFS-10932
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>Reporter: Chen Liang
>Assignee: Chen Liang
> Attachments: HDFS-10932-HDFS-7240.002.patch, 
> HDFS-10932-HDFS-7240.002.patch, HDFS-10932.001.patch
>
>
> Currently {{XceiverClient}} is the underlying entity of 
> {{DistributedStorageHandler.newKeyWriter()}} and 
> {{DistributedStorageHandler.newKeyReader()}}  for making call to container 
> for read/write. When {{XceiverClient}} gets closed, 
> {{group.shutdownGracefully()}} gets called, which is an asynchronous call. 
> A problem is that this asynchronous call has default quiet period of 2 
> seconds before it actually shutdown, so if we have a burst of read/write 
> calls, we would end up having threads created faster than they got 
> terminated, reaching system limit at some point.
> Ideally, this needs to be fixed with cached clients instead of creating new 
> thread each time. This JIRA only tries to give a temporary fix for the time 
> being.
> Thanks [~anu] for the offline discussion.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10851) FSDirStatAndListingOp: stop passing path as string

[Kihwal Lee (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10851?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15537052#comment-15537052
 ] 

Kihwal Lee commented on HDFS-10851:
---

The branch-2 patch looks good. The only difference is the {{ecPolicy}} 
parameter that does not exit in branch-2.

> FSDirStatAndListingOp: stop passing path as string
> --
>
> Key: HDFS-10851
> URL: https://issues.apache.org/jira/browse/HDFS-10851
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: hdfs
>Reporter: Daryn Sharp
>Assignee: Daryn Sharp
> Attachments: HDFS-10851.1.patch, HDFS-10851.branch-2.patch, 
> HDFS-10851.patch
>
>
> Path strings should be resolved once into INodesInPath.  The IIP should be 
> used extensively from that point forward.  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10934) TestDFSShell.testStat fails intermittently

[Eric Badger (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15537047#comment-15537047
 ] 

Eric Badger commented on HDFS-10934:


bq. I then realized that perhaps I should not rely on the assumption mtime1 == 
mtime2 in the first place.

[~liuml07], if I'm reading the test correctly, the test never checks that 
{{mtime1 == mtime2}}. It calls {{doFsStat()}} on both the file and directory, 
but checks the directory time against mtime1 and then the file against mtime2. 
I agree that the file status and directory status are generally free to be 
different, which is why I kept the file open so that they would be equal in 
this case (file created in directory, no other operations). It was slightly 
peculiar to me that we were testing the modification time of the directory 
(mtime2) against the modification time of the file, but it works in this 
contrived case. 

The easier way to fix this test would be to change {{testDir1}} to 
{{testFile2}} in the following line 
{noformat}
final FileStatus status2 = dfs.getFileStatus(testDir1);
{noformat}

It no longer tests that the directory gets modified when a file is created, but 
I don't believe that is the goal of this test. 

> TestDFSShell.testStat fails intermittently
> --
>
> Key: HDFS-10934
> URL: https://issues.apache.org/jira/browse/HDFS-10934
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: test
>Reporter: Eric Badger
>Assignee: Eric Badger
> Attachments: HDFS-10934.001.patch
>
>
> Saw this failure in an internal build. Reran the test 30 times and it failed 
> once with the same type of failure. 
> {noformat}
> org.junit.ComparisonFailure: Unexpected -stat output: 2016-09-30 03:48:56
> 2016-09-30 03:48:57
>  expected:<...6
> 2016-09-30 03:48:5[7]
> > but was:<...6
> 2016-09-30 03:48:5[6]
> >
>   at org.junit.Assert.assertEquals(Assert.java:115)
>   at org.apache.hadoop.hdfs.TestDFSShell.testStat(TestDFSShell.java:2082)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10932) Ozone : fix XceiverClient slow shutdown

[Anu Engineer (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15537044#comment-15537044
 ] 

Anu Engineer commented on HDFS-10932:
-

[~vagarychen] Thanks for checking. Since the exception says "Exception 
connecting XceiverClient." and the change was in the XceiverClient, I will just 
get one more run from Jenkins. Just want to make sure that we are not 
introducing any regressions.

> Ozone : fix XceiverClient slow shutdown
> ---
>
> Key: HDFS-10932
> URL: https://issues.apache.org/jira/browse/HDFS-10932
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>Reporter: Chen Liang
>Assignee: Chen Liang
> Attachments: HDFS-10932-HDFS-7240.002.patch, HDFS-10932.001.patch
>
>
> Currently {{XceiverClient}} is the underlying entity of 
> {{DistributedStorageHandler.newKeyWriter()}} and 
> {{DistributedStorageHandler.newKeyReader()}}  for making call to container 
> for read/write. When {{XceiverClient}} gets closed, 
> {{group.shutdownGracefully()}} gets called, which is an asynchronous call. 
> A problem is that this asynchronous call has default quiet period of 2 
> seconds before it actually shutdown, so if we have a burst of read/write 
> calls, we would end up having threads created faster than they got 
> terminated, reaching system limit at some point.
> Ideally, this needs to be fixed with cached clients instead of creating new 
> thread each time. This JIRA only tries to give a temporary fix for the time 
> being.
> Thanks [~anu] for the offline discussion.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10932) Ozone : fix XceiverClient slow shutdown

[Chen Liang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15537032#comment-15537032
 ] 

Chen Liang commented on HDFS-10932:
---

The failed tests do not seem to be related, local runs do not have the test 
fails in {{TestOzoneRestWithMiniCluster}} and {{TestBlockTokenWithDFSStriped}}, 
but {{TestBlockPoolManager.testFederationRefresh}} has been consistently 
failing locally with the patch AND without the patch. May need to fix it 
separately.

> Ozone : fix XceiverClient slow shutdown
> ---
>
> Key: HDFS-10932
> URL: https://issues.apache.org/jira/browse/HDFS-10932
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>Reporter: Chen Liang
>Assignee: Chen Liang
> Attachments: HDFS-10932-HDFS-7240.002.patch, HDFS-10932.001.patch
>
>
> Currently {{XceiverClient}} is the underlying entity of 
> {{DistributedStorageHandler.newKeyWriter()}} and 
> {{DistributedStorageHandler.newKeyReader()}}  for making call to container 
> for read/write. When {{XceiverClient}} gets closed, 
> {{group.shutdownGracefully()}} gets called, which is an asynchronous call. 
> A problem is that this asynchronous call has default quiet period of 2 
> seconds before it actually shutdown, so if we have a burst of read/write 
> calls, we would end up having threads created faster than they got 
> terminated, reaching system limit at some point.
> Ideally, this needs to be fixed with cached clients instead of creating new 
> thread each time. This JIRA only tries to give a temporary fix for the time 
> being.
> Thanks [~anu] for the offline discussion.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10939) Reduce performance penalty of encryption zones

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15537029#comment-15537029
 ] 

Hadoop QA commented on HDFS-10939:
--

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m  
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red}  0m  5s{color} 
| {color:red} HDFS-10939 does not apply to trunk. Rebase required? Wrong 
Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HDFS-10939 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12831165/HDFS-10939.patch |
| Console output | 
https://builds.apache.org/job/PreCommit-HDFS-Build/16952/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT   http://yetus.apache.org |


This message was automatically generated.



> Reduce performance penalty of encryption zones
> --
>
> Key: HDFS-10939
> URL: https://issues.apache.org/jira/browse/HDFS-10939
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: hdfs
>Affects Versions: 2.7
>Reporter: Daryn Sharp
>Assignee: Daryn Sharp
> Attachments: HDFS-10939.patch
>
>
> The encryption zone APIs should be optimized to extensively use IIPs to 
> eliminate path resolutions.  The performance penalties incurred by common 
> operations like creation of file statuses may be reduced by more extensive 
> short-circuiting of EZ lookups when no EZs exist.  All file creates should 
> not be subjected to the multi-stage locking performance penalty required only 
> for EDEK generation.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-10891) Über-jira: Test Improvement by adding missing test cases for existing code

[Mingliang Liu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-10891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mingliang Liu updated HDFS-10891:
-
Target Version/s: 2.8.0  (was: 3.0.0-alpha2)

> Über-jira: Test Improvement by adding missing test cases for existing code
> --
>
> Key: HDFS-10891
> URL: https://issues.apache.org/jira/browse/HDFS-10891
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: test
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
>
> We have covered some test cases in our nightly run and we'd like to backport 
> those test cases to the community code branch. This work includes the efforts 
> but not limited to:
> # Add more *meaningful* test cases for existing code so it's hard to break it 
> as stability and compatibility should always be guarded if possible
> # Refactor existing tests that share common code snippet (e.g. helper 
> methods) and/or logic (e.g. set up a MiniDFSCluster). One approach that is 
> not popular (which it should) is @Parameterized tests
> # Reduce unnecessary overhead in unit tests (e.g. long interval sleep, build 
> MiniDFSCluster multiple times instead of reusing the same one). Some of the 
> cases were addressed by [HDFS-10666].
> This is not a long-term work to cover all future improvement in unit tests. 
> We'll be happy to resolve this after our internal test cases (will file 
> separate JIRAs for that) are mostly addressed/considered.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-10934) TestDFSShell.testStat fails intermittently

[Mingliang Liu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-10934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mingliang Liu updated HDFS-10934:
-
Target Version/s: 2.8.0
 Component/s: test

> TestDFSShell.testStat fails intermittently
> --
>
> Key: HDFS-10934
> URL: https://issues.apache.org/jira/browse/HDFS-10934
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: test
>Reporter: Eric Badger
>Assignee: Eric Badger
> Attachments: HDFS-10934.001.patch
>
>
> Saw this failure in an internal build. Reran the test 30 times and it failed 
> once with the same type of failure. 
> {noformat}
> org.junit.ComparisonFailure: Unexpected -stat output: 2016-09-30 03:48:56
> 2016-09-30 03:48:57
>  expected:<...6
> 2016-09-30 03:48:5[7]
> > but was:<...6
> 2016-09-30 03:48:5[6]
> >
>   at org.junit.Assert.assertEquals(Assert.java:115)
>   at org.apache.hadoop.hdfs.TestDFSShell.testStat(TestDFSShell.java:2082)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-10934) TestDFSShell.testStat fails intermittently

[Mingliang Liu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-10934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mingliang Liu updated HDFS-10934:
-
Issue Type: Sub-task  (was: Bug)
Parent: HDFS-10891

> TestDFSShell.testStat fails intermittently
> --
>
> Key: HDFS-10934
> URL: https://issues.apache.org/jira/browse/HDFS-10934
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>Reporter: Eric Badger
>Assignee: Eric Badger
> Attachments: HDFS-10934.001.patch
>
>
> Saw this failure in an internal build. Reran the test 30 times and it failed 
> once with the same type of failure. 
> {noformat}
> org.junit.ComparisonFailure: Unexpected -stat output: 2016-09-30 03:48:56
> 2016-09-30 03:48:57
>  expected:<...6
> 2016-09-30 03:48:5[7]
> > but was:<...6
> 2016-09-30 03:48:5[6]
> >
>   at org.junit.Assert.assertEquals(Assert.java:115)
>   at org.apache.hadoop.hdfs.TestDFSShell.testStat(TestDFSShell.java:2082)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10934) TestDFSShell.testStat fails intermittently

[Mingliang Liu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15537011#comment-15537011
 ] 

Mingliang Liu commented on HDFS-10934:
--

[~ebadger], it's a nice catch! I think the patch will fix this.

I then realized that perhaps I should not rely on the assumption mtime1 == 
mtime2 in the first place. We simple mkdir, create file (and close it), get 
file status, get dir status, and do the assertions against respective mtime. 
The file status and dir status are free to be different. This way, the test 
code is more straightforward and a bit easier to read/maintain. Any thoughts? 
Thanks.

> TestDFSShell.testStat fails intermittently
> --
>
> Key: HDFS-10934
> URL: https://issues.apache.org/jira/browse/HDFS-10934
> Project: Hadoop HDFS
>  Issue Type: Bug
>Reporter: Eric Badger
>Assignee: Eric Badger
> Attachments: HDFS-10934.001.patch
>
>
> Saw this failure in an internal build. Reran the test 30 times and it failed 
> once with the same type of failure. 
> {noformat}
> org.junit.ComparisonFailure: Unexpected -stat output: 2016-09-30 03:48:56
> 2016-09-30 03:48:57
>  expected:<...6
> 2016-09-30 03:48:5[7]
> > but was:<...6
> 2016-09-30 03:48:5[6]
> >
>   at org.junit.Assert.assertEquals(Assert.java:115)
>   at org.apache.hadoop.hdfs.TestDFSShell.testStat(TestDFSShell.java:2082)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-10940) Reduce performance penalty of block caching when not used

[Daryn Sharp (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-10940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daryn Sharp updated HDFS-10940:
---
Attachment: HDFS-10940.patch

Very simple patch.  Pass LocatedBlocks to CacheManager.  CM immediately returns 
if there are no cached locations.  Pushed down the call to the CM into the BM - 
reducing redundancy for all callers.

> Reduce performance penalty of block caching when not used
> -
>
> Key: HDFS-10940
> URL: https://issues.apache.org/jira/browse/HDFS-10940
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: namenode
>Affects Versions: 2.7
>Reporter: Daryn Sharp
>Assignee: Daryn Sharp
> Attachments: HDFS-10940.patch
>
>
> For every block location generated, the CacheManager will create a junk 
> object for a hash lookup of cached locations.  If there are no cached blocks, 
> none of this is required.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-10940) Reduce performance penalty of block caching when not used

[Daryn Sharp (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-10940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daryn Sharp updated HDFS-10940:
---
Status: Patch Available  (was: Open)

> Reduce performance penalty of block caching when not used
> -
>
> Key: HDFS-10940
> URL: https://issues.apache.org/jira/browse/HDFS-10940
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: namenode
>Affects Versions: 2.7
>Reporter: Daryn Sharp
>Assignee: Daryn Sharp
> Attachments: HDFS-10940.patch
>
>
> For every block location generated, the CacheManager will create a junk 
> object for a hash lookup of cached locations.  If there are no cached blocks, 
> none of this is required.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Created] (HDFS-10940) Reduce performance penalty of block caching when not used

[Daryn Sharp (JIRA)]

Daryn Sharp created HDFS-10940:
--

 Summary: Reduce performance penalty of block caching when not used
 Key: HDFS-10940
 URL: https://issues.apache.org/jira/browse/HDFS-10940
 Project: Hadoop HDFS
  Issue Type: Improvement
  Components: namenode
Affects Versions: 2.7
Reporter: Daryn Sharp
Assignee: Daryn Sharp


For every block location generated, the CacheManager will create a junk object 
for a hash lookup of cached locations.  If there are no cached blocks, none of 
this is required.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-10939) Reduce performance penalty of encryption zones

[Daryn Sharp (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-10939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daryn Sharp updated HDFS-10939:
---
Status: Patch Available  (was: Open)

> Reduce performance penalty of encryption zones
> --
>
> Key: HDFS-10939
> URL: https://issues.apache.org/jira/browse/HDFS-10939
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: hdfs
>Affects Versions: 2.7
>Reporter: Daryn Sharp
>Assignee: Daryn Sharp
> Attachments: HDFS-10939.patch
>
>
> The encryption zone APIs should be optimized to extensively use IIPs to 
> eliminate path resolutions.  The performance penalties incurred by common 
> operations like creation of file statuses may be reduced by more extensive 
> short-circuiting of EZ lookups when no EZs exist.  All file creates should 
> not be subjected to the multi-stage locking performance penalty required only 
> for EDEK generation.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-10939) Reduce performance penalty of encryption zones

[Daryn Sharp (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-10939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daryn Sharp updated HDFS-10939:
---
Attachment: HDFS-10939.patch

Full use of IIP in EncryptionZoneManager and FsDirEncryptionZoneOp required a 
few minor IIP xattr api changes with a small ripple into other classes.

Main changes are:
# a few more shorts via hasCreatedEncryptionZone()
# moved EZ specific code from FsDirWriteFileOp into FsDirEncryptionZoneOp
# significantly cleaned up startFile method
# forced multi-stage locking is removed.  Instead of read lock, resolve, 
pre-conditions, read unlock, may generate EDEK, write lock, resolve, 
pre-conditions, do create; it's now write lock, resolve, pre-conditions, (iff 
need EDEK: write unlock, generate EDEK, write lock, re-resolve, 
pre-conditions), do create
# don't allocate BlocksMapUpdateInfo until just before needed so failed creates 
don't waste allocations
# tweaked how skipSync is set to avoid nothing to sync logging in some cases 

> Reduce performance penalty of encryption zones
> --
>
> Key: HDFS-10939
> URL: https://issues.apache.org/jira/browse/HDFS-10939
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: hdfs
>Affects Versions: 2.7
>Reporter: Daryn Sharp
>Assignee: Daryn Sharp
> Attachments: HDFS-10939.patch
>
>
> The encryption zone APIs should be optimized to extensively use IIPs to 
> eliminate path resolutions.  The performance penalties incurred by common 
> operations like creation of file statuses may be reduced by more extensive 
> short-circuiting of EZ lookups when no EZs exist.  All file creates should 
> not be subjected to the multi-stage locking performance penalty required only 
> for EDEK generation.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-10797) Disk usage summary of snapshots causes renamed blocks to get counted twice

[Sean Mackrory (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-10797?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sean Mackrory updated HDFS-10797:
-
Attachment: HDFS-10797.008.patch

Attaching a patch that address the findbugs warnings.

> Disk usage summary of snapshots causes renamed blocks to get counted twice
> --
>
> Key: HDFS-10797
> URL: https://issues.apache.org/jira/browse/HDFS-10797
> Project: Hadoop HDFS
>  Issue Type: Bug
>Reporter: Sean Mackrory
>Assignee: Sean Mackrory
> Attachments: HDFS-10797.001.patch, HDFS-10797.002.patch, 
> HDFS-10797.003.patch, HDFS-10797.004.patch, HDFS-10797.005.patch, 
> HDFS-10797.006.patch, HDFS-10797.007.patch, HDFS-10797.008.patch
>
>
> DirectoryWithSnapshotFeature.computeContentSummary4Snapshot calculates how 
> much disk usage is used by a snapshot by tallying up the files in the 
> snapshot that have since been deleted (that way it won't overlap with regular 
> files whose disk usage is computed separately). However that is determined 
> from a diff that shows moved (to Trash or otherwise) or renamed files as a 
> deletion and a creation operation that may overlap with the list of blocks. 
> Only the deletion operation is taken into consideration, and this causes 
> those blocks to get represented twice in the disk usage tallying.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Assigned] (HDFS-7642) NameNode should periodically log DataNode decommissioning progress

[Sean Mackrory (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-7642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sean Mackrory reassigned HDFS-7642:
---

Assignee: Sean Mackrory

> NameNode should periodically log DataNode decommissioning progress
> --
>
> Key: HDFS-7642
> URL: https://issues.apache.org/jira/browse/HDFS-7642
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Reporter: Zhe Zhang
>Assignee: Sean Mackrory
>Priority: Minor
>
> We've see a case where the decommissioning was stuck due to some files have 
> more replicas then DNs. HDFS-5662 fixes this particular issue but there are 
> other use cases where the decommissioning process might get stuck or slow 
> down. Some monitoring / logging will help debugging those issues.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-7642) NameNode should periodically log DataNode decommissioning progress

[Zhe Zhang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-7642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zhe Zhang updated HDFS-7642:

Assignee: (was: Zhe Zhang)

> NameNode should periodically log DataNode decommissioning progress
> --
>
> Key: HDFS-7642
> URL: https://issues.apache.org/jira/browse/HDFS-7642
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Reporter: Zhe Zhang
>Priority: Minor
>
> We've see a case where the decommissioning was stuck due to some files have 
> more replicas then DNs. HDFS-5662 fixes this particular issue but there are 
> other use cases where the decommissioning process might get stuck or slow 
> down. Some monitoring / logging will help debugging those issues.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-7642) NameNode should periodically log DataNode decommissioning progress

[Zhe Zhang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-7642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15536954#comment-15536954
 ] 

Zhe Zhang commented on HDFS-7642:
-

[~mackrorysd] Sure! Thanks for the interest. Unassigning myself now.

> NameNode should periodically log DataNode decommissioning progress
> --
>
> Key: HDFS-7642
> URL: https://issues.apache.org/jira/browse/HDFS-7642
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Reporter: Zhe Zhang
>Assignee: Zhe Zhang
>Priority: Minor
>
> We've see a case where the decommissioning was stuck due to some files have 
> more replicas then DNs. HDFS-5662 fixes this particular issue but there are 
> other use cases where the decommissioning process might get stuck or slow 
> down. Some monitoring / logging will help debugging those issues.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10907) Fix Erasure Coding documentation

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15536938#comment-15536938
 ] 

Hudson commented on HDFS-10907:
---

SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10521 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/10521/])
HDFS-10907. Fix Erasure Coding documentation. Contributed by Manoj (weichiu: 
rev 7fad1221d6f35e84b320fab82174525c067ad521)
* (edit) hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HDFSCommands.md
* (edit) hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HDFSErasureCoding.md


> Fix Erasure Coding documentation
> 
>
> Key: HDFS-10907
> URL: https://issues.apache.org/jira/browse/HDFS-10907
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: documentation, erasure-coding
>Affects Versions: 3.0.0-alpha1
>Reporter: Wei-Chiu Chuang
>Assignee: Manoj Govindassamy
>Priority: Trivial
>  Labels: newbie
> Fix For: 3.0.0-alpha2
>
> Attachments: HDFS-10907.01.patch
>
>
> Found one error in HDFSErasureCoding.md, missed by HDFS-9097:
> Under 
> bq. `policyName`: The ErasureCoding policy to be used for files under this 
> directory. This is an optional parameter, specified using ‘-s’ flag.
> It should be '-p'.
> The same error also appears in HDFSCommands.md:
> {quote}
> Usage:
>hdfs erasurecode \[generic options]
>  \[-setPolicy \[-s ] ]
> {quote}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10908) Improve StripedBlockReader#createBlockReader error logging

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15536939#comment-15536939
 ] 

Hudson commented on HDFS-10908:
---

SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10521 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/10521/])
HDFS-10908. Improve StripedBlockReader#createBlockReader error logging. 
(weichiu: rev 2ab1ef15c5e0b05fed5106d6bbecb3ead2b25f9a)
* (edit) 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/erasurecode/StripedBlockReader.java


> Improve StripedBlockReader#createBlockReader error logging
> --
>
> Key: HDFS-10908
> URL: https://issues.apache.org/jira/browse/HDFS-10908
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: erasure-coding
>Affects Versions: 3.0.0-alpha2
>Reporter: Wei-Chiu Chuang
>Assignee: Manoj Govindassamy
>Priority: Minor
> Fix For: 3.0.0-alpha2
>
> Attachments: HDFS-10908.01.patch
>
>
> In {{StripedBlockReader#createBlockReader}} if any IOException is thrown, the 
> error is logged at debug level and then returns a null. This means in a 
> typical configuration an NPE may be thrown without logging a cause if the 
> StripedBlockReader object is accessed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10910) HDFS Erasure Coding doc should state its currently supported erasure coding policies

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15536934#comment-15536934
 ] 

Hudson commented on HDFS-10910:
---

SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10521 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/10521/])
HDFS-10910. HDFS Erasure Coding doc should state its currently supported 
(weichiu: rev ee33a02234511ac69c1e491fd38490a141ec907e)
* (edit) hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HDFSErasureCoding.md


> HDFS Erasure Coding doc should state its currently supported erasure coding 
> policies
> 
>
> Key: HDFS-10910
> URL: https://issues.apache.org/jira/browse/HDFS-10910
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: documentation, erasure-coding
>Affects Versions: 3.0.0-alpha1
>Reporter: Wei-Chiu Chuang
>Assignee: Yiqun Lin
> Attachments: HDFS-10910.001.patch, HDFS-10910.002.patch, 
> HDFS-10910.003.patch
>
>
> While HDFS Erasure Coding doc states a variety of possible combinations of 
> algorithms, block group size and cell size, the code (as of 3.0.0-alpha1) 
> allows only three policies: RS_6_3_SCHEMA, RS_3_2_SCHEMA and 
> RS_6_3_LEGACY_SCHEMA. All with default cell size. I think this should be 
> documented.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-10907) Fix Erasure Coding documentation

[Wei-Chiu Chuang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-10907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wei-Chiu Chuang updated HDFS-10907:
---
   Resolution: Fixed
 Hadoop Flags: Reviewed
Fix Version/s: 3.0.0-alpha2
   Status: Resolved  (was: Patch Available)

Committed to trunk. Thanks for contributing the patch!

> Fix Erasure Coding documentation
> 
>
> Key: HDFS-10907
> URL: https://issues.apache.org/jira/browse/HDFS-10907
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: documentation, erasure-coding
>Affects Versions: 3.0.0-alpha1
>Reporter: Wei-Chiu Chuang
>Assignee: Manoj Govindassamy
>Priority: Trivial
>  Labels: newbie
> Fix For: 3.0.0-alpha2
>
> Attachments: HDFS-10907.01.patch
>
>
> Found one error in HDFSErasureCoding.md, missed by HDFS-9097:
> Under 
> bq. `policyName`: The ErasureCoding policy to be used for files under this 
> directory. This is an optional parameter, specified using ‘-s’ flag.
> It should be '-p'.
> The same error also appears in HDFSCommands.md:
> {quote}
> Usage:
>hdfs erasurecode \[generic options]
>  \[-setPolicy \[-s ] ]
> {quote}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-10910) HDFS Erasure Coding doc should state its currently supported erasure coding policies

[Wei-Chiu Chuang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-10910?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wei-Chiu Chuang updated HDFS-10910:
---
Resolution: Fixed
Status: Resolved  (was: Patch Available)

Committed the patch to trunk. Thanks [~linyiqun] for contributing the patch and 
[~Sammi] for the review!

> HDFS Erasure Coding doc should state its currently supported erasure coding 
> policies
> 
>
> Key: HDFS-10910
> URL: https://issues.apache.org/jira/browse/HDFS-10910
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: documentation, erasure-coding
>Affects Versions: 3.0.0-alpha1
>Reporter: Wei-Chiu Chuang
>Assignee: Yiqun Lin
> Attachments: HDFS-10910.001.patch, HDFS-10910.002.patch, 
> HDFS-10910.003.patch
>
>
> While HDFS Erasure Coding doc states a variety of possible combinations of 
> algorithms, block group size and cell size, the code (as of 3.0.0-alpha1) 
> allows only three policies: RS_6_3_SCHEMA, RS_3_2_SCHEMA and 
> RS_6_3_LEGACY_SCHEMA. All with default cell size. I think this should be 
> documented.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-10910) HDFS Erasure Coding doc should state its currently supported erasure coding policies

[Wei-Chiu Chuang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-10910?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wei-Chiu Chuang updated HDFS-10910:
---
Target Version/s: 3.0.0-alpha2

> HDFS Erasure Coding doc should state its currently supported erasure coding 
> policies
> 
>
> Key: HDFS-10910
> URL: https://issues.apache.org/jira/browse/HDFS-10910
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: documentation, erasure-coding
>Affects Versions: 3.0.0-alpha1
>Reporter: Wei-Chiu Chuang
>Assignee: Yiqun Lin
> Attachments: HDFS-10910.001.patch, HDFS-10910.002.patch, 
> HDFS-10910.003.patch
>
>
> While HDFS Erasure Coding doc states a variety of possible combinations of 
> algorithms, block group size and cell size, the code (as of 3.0.0-alpha1) 
> allows only three policies: RS_6_3_SCHEMA, RS_3_2_SCHEMA and 
> RS_6_3_LEGACY_SCHEMA. All with default cell size. I think this should be 
> documented.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10918) Add a tool to get FileEncryptionInfo from CLI

[Xiao Chen (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15536841#comment-15536841
 ] 

Xiao Chen commented on HDFS-10918:
--

Failed tests look unrelated and passed locally.
Will wait for some earth rotations in case anyone has interests, plan to commit 
this on Monday.

> Add a tool to get FileEncryptionInfo from CLI
> -
>
> Key: HDFS-10918
> URL: https://issues.apache.org/jira/browse/HDFS-10918
> Project: Hadoop HDFS
>  Issue Type: New Feature
>  Components: encryption
>Reporter: Xiao Chen
>Assignee: Xiao Chen
> Attachments: HDFS-10918.01.patch, HDFS-10918.02.patch, 
> HDFS-10918.03.patch, HDFS-10918.04.patch, HDFS-10918.05.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-10908) Improve StripedBlockReader#createBlockReader error logging

[Wei-Chiu Chuang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-10908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wei-Chiu Chuang updated HDFS-10908:
---
   Resolution: Fixed
Fix Version/s: 3.0.0-alpha2
   Status: Resolved  (was: Patch Available)

Committed it to trunk. Thanks [~manojg] for the patch!

> Improve StripedBlockReader#createBlockReader error logging
> --
>
> Key: HDFS-10908
> URL: https://issues.apache.org/jira/browse/HDFS-10908
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: erasure-coding
>Affects Versions: 3.0.0-alpha2
>Reporter: Wei-Chiu Chuang
>Assignee: Manoj Govindassamy
>Priority: Minor
> Fix For: 3.0.0-alpha2
>
> Attachments: HDFS-10908.01.patch
>
>
> In {{StripedBlockReader#createBlockReader}} if any IOException is thrown, the 
> error is logged at debug level and then returns a null. This means in a 
> typical configuration an NPE may be thrown without logging a cause if the 
> StripedBlockReader object is accessed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10938) Ozone: SCM: Add datanode protocol

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15536819#comment-15536819
 ] 

Hadoop QA commented on HDFS-10938:
--

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
13s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red}  0m  
0s{color} | {color:red} The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m 
23s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
48s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
58s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
16s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
25s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
 8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m  
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green}  1m  
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m  
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green}  0m  
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
24s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 61m 45s{color} 
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
20s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 82m 17s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestRenameWhileOpen |
|   | hadoop.hdfs.server.datanode.TestBlockPoolManager |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:9560f25 |
| JIRA Issue | HDFS-10938 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12831134/HDFS-10938-HDFS-7240.001.patch
 |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  xml  cc  |
| uname | Linux dfb5eed107f6 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 
17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | HDFS-7240 / 595257e |
| Default Java | 1.8.0_101 |
| unit | 
https://builds.apache.org/job/PreCommit-HDFS-Build/16950/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HDFS-Build/16950/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U: 
hadoop-hdfs-project/hadoop-hdfs |
| Console output | 
https://builds.apache.org/job/PreCommit-HDFS-Build/16950/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT   http://yetus.apache.org |


This message was automatically generated.



> Ozone: SCM: Add datanode protocol
> -
>
> Key: HDFS-10938
> URL: https://issues.apache.org/jira/browse/HDFS-10938
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Anu Engineer
>Assignee: Anu Engineer
> Fix For: HDFS-7240
>
> Attachments: HDFS-10938-HDFS-7240.001.patch
>
>
> Adds the datanode protoc 

[jira] [Commented] (HDFS-10912) Ozone:SCM: Add chill mode support to NodeManager.

[Xiaoyu Yao (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15536813#comment-15536813
 ] 

Xiaoyu Yao commented on HDFS-10912:
---

Thanks @Anu for reporting the issue and posting the patch. It looks good to me 
overall. Below are some minor issues:

NodeManager.java
# NIT: extra space "*  Chill mode is" 
# Typo: "Forcefully exists ""
# clearChillModeFlag(), is this for test only? add annotation for that if it is 
the case.

SCMNodeManager.java
# NIT: comments below can be removed as it is covered in the class description
of NodeManager.java. I would suggest we summarize the usage of both safemode and
the new chillmode as discussed in previous comments and put it in the comments. 
{code}/**
107* Chill mode is the period where node manager is waiting for 
sufficient
108* nodes to report in.
109*/
{code}
# getChillModeStatus(): the logic of returning different status string can be 
consolidated to get a consistent string output for easy parsing or trouble 
shooting.
# forceExitChillMode()/forceEnterChillMode: typo "exists". Should we use "chill 
mode" instead of
"safe mode" in the comments.




> Ozone:SCM: Add chill mode support to NodeManager.
> -
>
> Key: HDFS-10912
> URL: https://issues.apache.org/jira/browse/HDFS-10912
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Anu Engineer
>Assignee: Anu Engineer
> Attachments: HDFS-10912-HDFS-7240.001.patch, 
> HDFS-10912-HDFS-7240.002.patch
>
>
> Add chill mode support : That is to add the ability to force exit or enter 
> chill mode. As well as get the current chill mode status from node manager.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10918) Add a tool to get FileEncryptionInfo from CLI

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15536810#comment-15536810
 ] 

Hadoop QA commented on HDFS-10918:
--

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
16s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  1m 
43s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  8m 
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  7m 
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  2m 
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  4m 
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  2m  
2s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
14s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  2m 
 2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  6m 
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  6m 
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  2m 
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green}  0m  
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  4m 
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  2m  
1s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}  7m 54s{color} 
| {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m 
57s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 58m 24s{color} 
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
25s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}118m  6s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.net.TestDNS |
|   | hadoop.hdfs.TestDFSShell |
|   | hadoop.hdfs.TestRenameWhileOpen |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:9560f25 |
| JIRA Issue | HDFS-10918 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12831129/HDFS-10918.05.patch |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  xml  |
| uname | Linux 4443968100ff 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 
21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / 0670149 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| unit | 
https://builds.apache.org/job/PreCommit-HDFS-Build/16947/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
 |
| unit | 

[jira] [Created] (HDFS-10939) Reduce performance penalty of encryption zones

[Daryn Sharp (JIRA)]

Daryn Sharp created HDFS-10939:
--

 Summary: Reduce performance penalty of encryption zones
 Key: HDFS-10939
 URL: https://issues.apache.org/jira/browse/HDFS-10939
 Project: Hadoop HDFS
  Issue Type: Sub-task
Affects Versions: 2.7
Reporter: Daryn Sharp
Assignee: Daryn Sharp


The encryption zone APIs should be optimized to extensively use IIPs to 
eliminate path resolutions.  The performance penalties incurred by common 
operations like creation of file statuses may be reduced by more extensive 
short-circuiting of EZ lookups when no EZs exist.  All file creates should not 
be subjected to the multi-stage locking performance penalty required only for 
EDEK generation.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10797) Disk usage summary of snapshots causes renamed blocks to get counted twice

[Sean Mackrory (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15536805#comment-15536805
 ] 

Sean Mackrory commented on HDFS-10797:
--

findbugs output is correct: I'm using synchronization to prevent any problems 
caused by the fact that we swap the counts object to tally up deleted, 
snapshotted space. We did this before, and it *should* be fine, since this is 
single-threaded everywhere I've seen, but thought it'd be wise to guard just in 
case. I'll synchronize the entire function on the context object instead and 
resubmit - other feedback welcome in the mean time, since that won't affect 
most of the patch.

> Disk usage summary of snapshots causes renamed blocks to get counted twice
> --
>
> Key: HDFS-10797
> URL: https://issues.apache.org/jira/browse/HDFS-10797
> Project: Hadoop HDFS
>  Issue Type: Bug
>Reporter: Sean Mackrory
>Assignee: Sean Mackrory
> Attachments: HDFS-10797.001.patch, HDFS-10797.002.patch, 
> HDFS-10797.003.patch, HDFS-10797.004.patch, HDFS-10797.005.patch, 
> HDFS-10797.006.patch, HDFS-10797.007.patch
>
>
> DirectoryWithSnapshotFeature.computeContentSummary4Snapshot calculates how 
> much disk usage is used by a snapshot by tallying up the files in the 
> snapshot that have since been deleted (that way it won't overlap with regular 
> files whose disk usage is computed separately). However that is determined 
> from a diff that shows moved (to Trash or otherwise) or renamed files as a 
> deletion and a creation operation that may overlap with the list of blocks. 
> Only the deletion operation is taken into consideration, and this causes 
> those blocks to get represented twice in the disk usage tallying.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10932) Ozone : fix XceiverClient slow shutdown

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15536798#comment-15536798
 ] 

Hadoop QA commented on HDFS-10932:
--

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
18s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red}  0m  
0s{color} | {color:red} The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m 
29s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
50s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
28s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
59s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
15s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  2m  
5s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
17s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
13s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 58m 32s{color} 
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
25s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 83m  7s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.ozone.web.TestOzoneRestWithMiniCluster |
|   | hadoop.hdfs.server.datanode.TestBlockPoolManager |
|   | hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:9560f25 |
| JIRA Issue | HDFS-10932 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12831132/HDFS-10932-HDFS-7240.002.patch
 |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  |
| uname | Linux 4a76345ad3b5 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 
20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | HDFS-7240 / 595257e |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| unit | 
https://builds.apache.org/job/PreCommit-HDFS-Build/16949/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HDFS-Build/16949/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U: 
hadoop-hdfs-project/hadoop-hdfs |
| Console output | 
https://builds.apache.org/job/PreCommit-HDFS-Build/16949/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT   http://yetus.apache.org |


This message was automatically generated.



> Ozone : fix XceiverClient slow shutdown
> ---
>
> Key: HDFS-10932

[jira] [Commented] (HDFS-10797) Disk usage summary of snapshots causes renamed blocks to get counted twice

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15536780#comment-15536780
 ] 

Hadoop QA commented on HDFS-10797:
--

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
16s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  8m 
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m  
3s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  
1s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
52s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
0m 28s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch 
generated 3 new + 267 unchanged - 0 fixed = 270 total (was 267) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  1m 
58s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs generated 2 new + 0 
unchanged - 0 fixed = 2 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 62m 
31s{color} | {color:green} hadoop-hdfs in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
22s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 84m 51s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-hdfs-project/hadoop-hdfs |
|  |  Synchronization on ContentSummaryComputationContext.counts in futile 
attempt to guard it  At ContentSummaryComputationContext.java:attempt to guard 
it  At ContentSummaryComputationContext.java:[line 224] |
|  |  Synchronization on ContentSummaryComputationContext.counts in futile 
attempt to guard it  At ContentSummaryComputationContext.java:attempt to guard 
it  At ContentSummaryComputationContext.java:[line 232] |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:9560f25 |
| JIRA Issue | HDFS-10797 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12831131/HDFS-10797.007.patch |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  |
| uname | Linux 27f500bb7b8c 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed 
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / 0670149 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| checkstyle | 
https://builds.apache.org/job/PreCommit-HDFS-Build/16948/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
 |
| findbugs | 
https://builds.apache.org/job/PreCommit-HDFS-Build/16948/artifact/patchprocess/new-findbugs-hadoop-hdfs-project_hadoop-hdfs.html
 |
|  Test Results | 

[jira] [Updated] (HDFS-10851) FSDirStatAndListingOp: stop passing path as string

[Daryn Sharp (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-10851?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daryn Sharp updated HDFS-10851:
---
Attachment: HDFS-10851.branch-2.patch

Just had to remove a few references to EC policy.

> FSDirStatAndListingOp: stop passing path as string
> --
>
> Key: HDFS-10851
> URL: https://issues.apache.org/jira/browse/HDFS-10851
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: hdfs
>Reporter: Daryn Sharp
>Assignee: Daryn Sharp
> Attachments: HDFS-10851.1.patch, HDFS-10851.branch-2.patch, 
> HDFS-10851.patch
>
>
> Path strings should be resolved once into INodesInPath.  The IIP should be 
> used extensively from that point forward.  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-7642) NameNode should periodically log DataNode decommissioning progress

[Sean Mackrory (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-7642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15536701#comment-15536701
 ] 

Sean Mackrory commented on HDFS-7642:
-

Hey [~zhz] - I'd like to work on this, if that's alright with you. Since it's 
been quite a few months, I assume nothing is actively in progress here?

I would probably implement this somewhere inside 
DecommissionManager.Monitor.run()

> NameNode should periodically log DataNode decommissioning progress
> --
>
> Key: HDFS-7642
> URL: https://issues.apache.org/jira/browse/HDFS-7642
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Reporter: Zhe Zhang
>Assignee: Zhe Zhang
>Priority: Minor
>
> We've see a case where the decommissioning was stuck due to some files have 
> more replicas then DNs. HDFS-5662 fixes this particular issue but there are 
> other use cases where the decommissioning process might get stuck or slow 
> down. Some monitoring / logging will help debugging those issues.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10301) BlockReport retransmissions may lead to storages falsely being declared zombie if storage report processing happens out of order

[Konstantin Shvachko (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15536676#comment-15536676
 ] 

Konstantin Shvachko commented on HDFS-10301:


I think the solution for the {{checkLease()}} problem is 
# to restore returning {{false}} in  {{checkLease()}}. That way "bad" DNs will 
not be able to send FBRs without obtaining a lease.
# Move release lease logic from per-storage {{BM.processReport()}} to 
{{NameNodeRpcServer.blockReport()}} just after all storages in the report are 
processed. That way we will not need to track the order of processing 
per-storage reports by FutureTask threads, but just release the lease once all 
storages are done, that is when {{curRpc + 1 == totalRpcs}} .

This will exactly match the current behavior of br-leases. It does not solve 
the race between a timed out BR and the repeating BR in multi-RPC BR case. But 
the race exists in current code as well, and I would prefer to address this in 
another issue.

So before going to testing this on multiple versions and configurations I would 
like to confirm with you guys that this is the only problem remaining, and we 
have a consensus on the solution. Please confirm.

> BlockReport retransmissions may lead to storages falsely being declared 
> zombie if storage report processing happens out of order
> 
>
> Key: HDFS-10301
> URL: https://issues.apache.org/jira/browse/HDFS-10301
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: namenode
>Affects Versions: 2.6.1
>Reporter: Konstantin Shvachko
>Assignee: Vinitha Reddy Gankidi
>Priority: Critical
> Attachments: HDFS-10301.002.patch, HDFS-10301.003.patch, 
> HDFS-10301.004.patch, HDFS-10301.005.patch, HDFS-10301.006.patch, 
> HDFS-10301.007.patch, HDFS-10301.008.patch, HDFS-10301.009.patch, 
> HDFS-10301.01.patch, HDFS-10301.010.patch, HDFS-10301.011.patch, 
> HDFS-10301.012.patch, HDFS-10301.013.patch, HDFS-10301.014.patch, 
> HDFS-10301.branch-2.7.patch, HDFS-10301.branch-2.patch, 
> HDFS-10301.sample.patch, zombieStorageLogs.rtf
>
>
> When NameNode is busy a DataNode can timeout sending a block report. Then it 
> sends the block report again. Then NameNode while process these two reports 
> at the same time can interleave processing storages from different reports. 
> This screws up the blockReportId field, which makes NameNode think that some 
> storages are zombie. Replicas from zombie storages are immediately removed, 
> causing missing blocks.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Comment Edited] (HDFS-10690) Optimize insertion/removal of replica in ShortCircuitCache.java

[Xiaoyu Yao (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15536651#comment-15536651
 ] 

Xiaoyu Yao edited comment on HDFS-10690 at 9/30/16 6:34 PM:


[~fenghua_hu], we will need to update the {{CacheVisistor}} instances in 
{{TestEnhancedByteBufferAccess.java}} and {{TestShortCircuitCache.java}} now 
that the {{CacheVisitor}} interface has been changed from using Map to 
LinkedMap. Otherwise, it will fail with compiler error like below. 

Not sure why Jenkins run did not catch this, but will file separate infra JIRA 
for that. 

{code}
[ERROR] 
/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/TestEnhancedByteBufferAccess.java:[291,18]
 org.apache.hadoop.fs.TestEnhancedByteBufferAccess.CountingVisitor is not 
abstract and does not override abstract method 
visit(int,java.util.Map,java.util.Map,org.apache.commons.collections.map.LinkedMap,org.apache.commons.collections.map.LinkedMap)
 in org.apache.hadoop.hdfs.shortcircuit.ShortCircuitCache.CacheVisitor
[ERROR] 
/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/TestEnhancedByteBufferAccess.java:[306,5]
 method does not override or implement a method from a supertype
{code}


was (Author: xyao):
[~fenghua_hu], we will need to update {{TestEnhancedByteBufferAccess.java}} and 
{{TestShortCircuitCache.java}} now that the {{CacheVisitor}} interface has been 
changed. Otherwise, it will fail with compiler error like below. 

Not sure why Jenkins run did not catch this, but will file separate infra JIRA 
for that. 

{code}
[ERROR] 
/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/TestEnhancedByteBufferAccess.java:[291,18]
 org.apache.hadoop.fs.TestEnhancedByteBufferAccess.CountingVisitor is not 
abstract and does not override abstract method 
visit(int,java.util.Map,java.util.Map,org.apache.commons.collections.map.LinkedMap,org.apache.commons.collections.map.LinkedMap)
 in org.apache.hadoop.hdfs.shortcircuit.ShortCircuitCache.CacheVisitor
[ERROR] 
/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/TestEnhancedByteBufferAccess.java:[306,5]
 method does not override or implement a method from a supertype
{code}

> Optimize insertion/removal of replica in ShortCircuitCache.java
> ---
>
> Key: HDFS-10690
> URL: https://issues.apache.org/jira/browse/HDFS-10690
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: hdfs-client
>Affects Versions: 3.0.0-alpha2
>Reporter: Fenghua Hu
>Assignee: Fenghua Hu
> Attachments: HDFS-10690.001.patch, HDFS-10690.002.patch, 
> HDFS-10690.003.patch, HDFS-10690.004.patch, HDFS-10690.005.patch, 
> HDFS-10690.006.patch, ShortCircuitCache_LinkedMap.patch
>
>   Original Estimate: 336h
>  Remaining Estimate: 336h
>
> Currently in ShortCircuitCache, two TreeMap objects are used to track the 
> cached replicas.
> private final TreeMap evictable = new TreeMap<>();
> private final TreeMap evictableMmapped = new 
> TreeMap<>();
> TreeMap employs Red-Black tree for sorting. This isn't an issue when using 
> traditional HDD. But when using high-performance SSD/PCIe Flash, the cost 
> inserting/removing an entry  becomes considerable.
> To mitigate it, we designed a new list-based for replica tracking.
> The list is a double-linked FIFO. FIFO is time-based, thus insertion is a 
> very low cost operation. On the other hand, list is not lookup-friendly. To 
> address this issue, we introduce two references into ShortCircuitReplica 
> object.
> ShortCircuitReplica next = null;
> ShortCircuitReplica prev = null;
> In this way, lookup is not needed when removing a replica from the list. We 
> only need to modify its predecessor's and successor's references in the lists.
> Our tests showed up to 15-50% performance improvement when using PCIe flash 
> as storage media.
> The original patch is against 2.6.4, now I am porting to Hadoop trunk, and 
> patch will be posted soon.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10690) Optimize insertion/removal of replica in ShortCircuitCache.java

[Xiaoyu Yao (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15536651#comment-15536651
 ] 

Xiaoyu Yao commented on HDFS-10690:
---

[~fenghua_hu], we will need to update {{TestEnhancedByteBufferAccess.java}} and 
{{TestShortCircuitCache.java}} now that the {{CacheVisitor}} interface has been 
changed. Otherwise, it will fail with compiler error like below. 

Not sure why Jenkins run did not catch this, but will file separate infra JIRA 
for that. 

{code}
[ERROR] 
/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/TestEnhancedByteBufferAccess.java:[291,18]
 org.apache.hadoop.fs.TestEnhancedByteBufferAccess.CountingVisitor is not 
abstract and does not override abstract method 
visit(int,java.util.Map,java.util.Map,org.apache.commons.collections.map.LinkedMap,org.apache.commons.collections.map.LinkedMap)
 in org.apache.hadoop.hdfs.shortcircuit.ShortCircuitCache.CacheVisitor
[ERROR] 
/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/TestEnhancedByteBufferAccess.java:[306,5]
 method does not override or implement a method from a supertype
{code}

> Optimize insertion/removal of replica in ShortCircuitCache.java
> ---
>
> Key: HDFS-10690
> URL: https://issues.apache.org/jira/browse/HDFS-10690
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: hdfs-client
>Affects Versions: 3.0.0-alpha2
>Reporter: Fenghua Hu
>Assignee: Fenghua Hu
> Attachments: HDFS-10690.001.patch, HDFS-10690.002.patch, 
> HDFS-10690.003.patch, HDFS-10690.004.patch, HDFS-10690.005.patch, 
> HDFS-10690.006.patch, ShortCircuitCache_LinkedMap.patch
>
>   Original Estimate: 336h
>  Remaining Estimate: 336h
>
> Currently in ShortCircuitCache, two TreeMap objects are used to track the 
> cached replicas.
> private final TreeMap evictable = new TreeMap<>();
> private final TreeMap evictableMmapped = new 
> TreeMap<>();
> TreeMap employs Red-Black tree for sorting. This isn't an issue when using 
> traditional HDD. But when using high-performance SSD/PCIe Flash, the cost 
> inserting/removing an entry  becomes considerable.
> To mitigate it, we designed a new list-based for replica tracking.
> The list is a double-linked FIFO. FIFO is time-based, thus insertion is a 
> very low cost operation. On the other hand, list is not lookup-friendly. To 
> address this issue, we introduce two references into ShortCircuitReplica 
> object.
> ShortCircuitReplica next = null;
> ShortCircuitReplica prev = null;
> In this way, lookup is not needed when removing a replica from the list. We 
> only need to modify its predecessor's and successor's references in the lists.
> Our tests showed up to 15-50% performance improvement when using PCIe flash 
> as storage media.
> The original patch is against 2.6.4, now I am porting to Hadoop trunk, and 
> patch will be posted soon.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10896) Move lock logging logic from FSNamesystem into FSNamesystemLock

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15536649#comment-15536649
 ] 

Hadoop QA commented on HDFS-10896:
--

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
19s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 2 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  6m 
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
42s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
0m 25s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch 
generated 1 new + 191 unchanged - 4 fixed = 192 total (was 195) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
53s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 56m 46s{color} 
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
21s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 75m 44s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes |
|   | hadoop.hdfs.server.namenode.TestDecommissioningStatus |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:9560f25 |
| JIRA Issue | HDFS-10896 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12831128/HDFS-10896.004.patch |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  |
| uname | Linux da3be1245d1d 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 
21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / 0670149 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| checkstyle | 
https://builds.apache.org/job/PreCommit-HDFS-Build/16946/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
 |
| unit | 
https://builds.apache.org/job/PreCommit-HDFS-Build/16946/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HDFS-Build/16946/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U: 
hadoop-hdfs-project/hadoop-hdfs |
| Console output | 
https://builds.apache.org/job/PreCommit-HDFS-Build/16946/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT   http://yetus.apache.org |


This message was automatically generated.



> Move lock logging logic from FSNamesystem into FSNamesystemLock
> 

[jira] [Commented] (HDFS-10908) Improve StripedBlockReader#createBlockReader error logging

[Wei-Chiu Chuang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15536643#comment-15536643
 ] 

Wei-Chiu Chuang commented on HDFS-10908:


LGTM. Test case not needed since it's a log message only improvement.

> Improve StripedBlockReader#createBlockReader error logging
> --
>
> Key: HDFS-10908
> URL: https://issues.apache.org/jira/browse/HDFS-10908
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: erasure-coding
>Affects Versions: 3.0.0-alpha2
>Reporter: Wei-Chiu Chuang
>Assignee: Manoj Govindassamy
>Priority: Minor
> Attachments: HDFS-10908.01.patch
>
>
> In {{StripedBlockReader#createBlockReader}} if any IOException is thrown, the 
> error is logged at debug level and then returns a null. This means in a 
> typical configuration an NPE may be thrown without logging a cause if the 
> StripedBlockReader object is accessed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10851) FSDirStatAndListingOp: stop passing path as string

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10851?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15536640#comment-15536640
 ] 

Hudson commented on HDFS-10851:
---

SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10519 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/10519/])
HDFS-10851. FSDirStatAndListingOp: stop passing path as string. (kihwal: rev 
a0730aa5ced7666a8c92f9fb830b615f5f9f477a)
* (edit) 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirAclOp.java
* (edit) 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java
* (edit) 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirStatAndListingOp.java
* (edit) 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodesInPath.java
* (edit) 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java
* (edit) 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeAttributeProvider.java
* (edit) 
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestSnapshotPathINodes.java
* (edit) 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirXAttrOp.java


> FSDirStatAndListingOp: stop passing path as string
> --
>
> Key: HDFS-10851
> URL: https://issues.apache.org/jira/browse/HDFS-10851
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: hdfs
>Reporter: Daryn Sharp
>Assignee: Daryn Sharp
> Attachments: HDFS-10851.1.patch, HDFS-10851.patch
>
>
> Path strings should be resolved once into INodesInPath.  The IIP should be 
> used extensively from that point forward.  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10921) TestDiskspaceQuotaUpdate doesn't wait for NN to get out of safe mode

[Rushabh S Shah (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15536630#comment-15536630
 ] 

Rushabh S Shah commented on HDFS-10921:
---

bq. Rushabh S Shah, do you have any concerns with Erik Krogen's approach?
+1.

> TestDiskspaceQuotaUpdate doesn't wait for NN to get out of safe mode
> 
>
> Key: HDFS-10921
> URL: https://issues.apache.org/jira/browse/HDFS-10921
> Project: Hadoop HDFS
>  Issue Type: Bug
>Reporter: Eric Badger
>Assignee: Eric Badger
> Attachments: HDFS-10921.001.patch, HDFS-10921.002.patch
>
>
> Test fails intermittently because the NN is still in safe mode. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org