[jira] [Commented] (HDFS-7337) Configurable and pluggable erasure codec and policy

[Xiao Chen (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-7337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16210622#comment-16210622
 ] 

Xiao Chen commented on HDFS-7337:
-

Hi [~drankye] and [~Sammi],
Thanks a lot for working on this cool feature!
Is there a jira for persisting EC policy to NN metadata? I searched the linked 
jiras but didn't find one.

> Configurable and pluggable erasure codec and policy
> ---
>
> Key: HDFS-7337
> URL: https://issues.apache.org/jira/browse/HDFS-7337
> Project: Hadoop HDFS
>  Issue Type: New Feature
>  Components: erasure-coding
>Reporter: Zhe Zhang
>Assignee: SammiChen
>Priority: Critical
>  Labels: hdfs-ec-3.0-nice-to-have
> Fix For: 3.0.0-beta1
>
> Attachments: HDFS-7337-prototype-v1.patch, 
> HDFS-7337-prototype-v2.zip, HDFS-7337-prototype-v3.zip, PluggableErasureCodec 
> v4.pdf, PluggableErasureCodec-v2.pdf, PluggableErasureCodec-v3.pdf, 
> PluggableErasureCodec.pdf
>
>
> According to HDFS-7285 and the design, this considers to support multiple 
> Erasure Codecs via pluggable approach. It allows to define and configure 
> multiple codec schemas with different coding algorithms and parameters. The 
> resultant codec schemas can be utilized and specified via command tool for 
> different file folders. While design and implement such pluggable framework, 
> it’s also to implement a concrete codec by default (Reed Solomon) to prove 
> the framework is useful and workable. Separate JIRA could be opened for the 
> RS codec implementation.
> Note HDFS-7353 will focus on the very low level codec API and implementation 
> to make concrete vendor libraries transparent to the upper layer. This JIRA 
> focuses on high level stuffs that interact with configuration, schema and etc.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-12521) Ozone: SCM should read all Container info into memory when booting up

[Anu Engineer (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-12521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16210591#comment-16210591
 ] 

Anu Engineer commented on HDFS-12521:
-

[~ljain], Thanks for taking care of this. Some very minor comments.

* Can we please move the changes in initializeContainerMaps to a different 
function?
if you want you can still call them via initializeContainerMaps.

* pipeline.jkava: ignorableFieldNames, can we please make sure that this indeed 
does ignore these field names. Also, it would be nice to make sure that list 
container JSON does not change before and after this patch.

* Thanks for adding the TestContainerStateManager.

> Ozone: SCM should read all Container info into memory when booting up
> -
>
> Key: HDFS-12521
> URL: https://issues.apache.org/jira/browse/HDFS-12521
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: ozone
>Reporter: Anu Engineer
>Assignee: Lokesh Jain
>  Labels: ozoneMerge
> Attachments: HDFS-12521-HDFS-7240.001.patch, 
> HDFS-12521-HDFS-7240.002.patch, HDFS-12521-HDFS-7240.003.patch
>
>
> When SCM boots up it should read all containers into memory. This is a 
> performance optimization that allows delays on SCM side. This JIRA tracks 
> that issue.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-12663) Ozone: OzoneClient: Remove protobuf classes exposed to clients through OzoneBucket

[Anu Engineer (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-12663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16210511#comment-16210511
 ] 

Anu Engineer commented on HDFS-12663:
-

bq.  If we use it as it's now, should we explain what OzoneProtos is to the 
user?

Makes sense, let us go ahead with this patch. Thanks for the explanation.

> Ozone: OzoneClient: Remove protobuf classes exposed to clients through 
> OzoneBucket
> --
>
> Key: HDFS-12663
> URL: https://issues.apache.org/jira/browse/HDFS-12663
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: ozone
>Reporter: Nanda kumar
>Assignee: Nanda kumar
>  Labels: ozoneMerge
> Attachments: HDFS-12663-HDFS-7240.000.patch, 
> HDFS-12663-HDFS-7240.001.patch
>
>
> As part of {{OzoneBucket#createKey}} we are currently exposing protobuf enums 
> {{OzoneProtos.ReplicationType}} and {{OzoneProtos.ReplicationFactor}} through 
> client, this can be replaced with client side enums and conversion can be 
> done internally.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-12680) Ozone: SCM: Lease support for container creation

[Anu Engineer (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-12680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16210503#comment-16210503
 ] 

Anu Engineer commented on HDFS-12680:
-

[~nandakumar131] Thanks for getting this done. Probably one of our final patch 
we should do before the merge. Overall, I think this looks very good, some very 
minor comments below.


* One small question: Please do treat this as a  question and not a comment. 
We seem to trigger the Begin state for the container as soon as we hand over 
the container state to the client. The earlier design was slightly different, 
that is the timer began when the client notified SCM that it is going to create 
a container. Not that it is going to make any difference, just want to make 
sure that this is a conscious decision.

* ContainerInfo:hashCode(): Since state can change all the time, this ceases to 
be a unique ID for the container. Or do you intend to define the container 
identity based on State too? Is there a use case where this is needed.

* OZONE_SCM_CONTAINER_CREATION_LEASE_TIMEOUT , should we add this to 
ozone-conf.xml? also, can we define the units? if you get a chance update the 
lease manager ctor doc too, please.

* Certainly not part of this JIRA, but what happens after the timeOut 
(OzoneProtos.LifeCycleEvent.TIMEOUT), should we file another JIRA?
* TestContainerMapping.java -- conf.set ==> conf.setInt? So we can pass the 
right data type





> Ozone: SCM: Lease support for container creation
> 
>
> Key: HDFS-12680
> URL: https://issues.apache.org/jira/browse/HDFS-12680
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: ozone
>Reporter: Nanda kumar
>Assignee: Nanda kumar
>  Labels: ozoneMerge
> Attachments: HDFS-12680-HDFS-7240.000.patch
>
>
> This brings in lease support for container creation.
> Lease should be give for a container that is moved to {{CREATING}} state when 
> {{BEGIN_CREATE}} event happens, {{LeaseException}} should be thrown if the 
> container already holds a lease. Lease must be released during 
> {{COMPLETE_CREATE}} event. If the lease times out container should be moved 
> to {{DELETING}} state, and exception should be thrown if {{COMPLETE_CREATE}} 
> event is received on that container.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-12638) NameNode exits due to ReplicationMonitor thread received Runtime exception in ReplicationWork#chooseTargets

[Weiwei Yang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-12638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16210498#comment-16210498
 ] 

Weiwei Yang commented on HDFS-12638:


Hi [~daryn]

bq. it sounds like you want to avoid the symptom (NPE) 

Yes, I think our code should bear with such orphan blocks, instead of failing 
the NN with NPE like this. At least.

bq. rather than address the orphaned block (root cause)?

The test case explains how these orphaned blocks come from

||Step#||Step Operation||Explain||
|1|Create a file| this creates a few blocks, e.g B1|
|2|Run rolling upgrade (or create a snapshot)|this is the condition to trigger 
the copy-on-truncate behavior|
|3|Truncate the file to a smaller size|"copy-on-truncate" schema is used, it 
creates a new block B2 and copy required bytes from B1 to B2, and inode 
reference updated to B2|
|4|Delete this file|this will delete inode ref and remove B2 from blocks map, 
leaving B1 behind. So when we read snapshot again, it is able to find its 
original block B1.|

Please see also {{FSDirTtruncateOp#shouldCopyOnTruncate}}. I have read the 
truncate design doc, this seems to be the designed behavior. We cannot delete 
the old block otherwise snapshot won't be able to read it anymore. I assume 
when the snapshot gets deleted, these blocks will be also removed from the 
blocks map. But before that, we need to live with such orphaned blocks. Any 
thoughts on this?

Thanks


> NameNode exits due to ReplicationMonitor thread received Runtime exception in 
> ReplicationWork#chooseTargets
> ---
>
> Key: HDFS-12638
> URL: https://issues.apache.org/jira/browse/HDFS-12638
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: hdfs
>Affects Versions: 2.8.2
>Reporter: Jiandan Yang 
> Attachments: HDFS-12638-branch-2.8.2.001.patch
>
>
> Active NamNode exit due to NPE, I can confirm that the BlockCollection passed 
> in when creating ReplicationWork is null, but I do not know why 
> BlockCollection is null, By view history I found 
> [HDFS-9754|https://issues.apache.org/jira/browse/HDFS-9754] remove judging  
> whether  BlockCollection is null.
> NN logs are as following:
> {code:java}
> 2017-10-11 16:29:06,161 ERROR [ReplicationMonitor] 
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: 
> ReplicationMonitor thread received Runtime exception.
> java.lang.NullPointerException
> at 
> org.apache.hadoop.hdfs.server.blockmanagement.ReplicationWork.chooseTargets(ReplicationWork.java:55)
> at 
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.computeReplicationWorkForBlocks(BlockManager.java:1532)
> at 
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.computeReplicationWork(BlockManager.java:1491)
> at 
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.computeDatanodeWork(BlockManager.java:3792)
> at 
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$ReplicationMonitor.run(BlockManager.java:3744)
> at java.lang.Thread.run(Thread.java:834)
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-12680) Ozone: SCM: Lease support for container creation

[Anu Engineer (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-12680?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anu Engineer updated HDFS-12680:

Labels: ozoneMerge  (was: )

> Ozone: SCM: Lease support for container creation
> 
>
> Key: HDFS-12680
> URL: https://issues.apache.org/jira/browse/HDFS-12680
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: ozone
>Reporter: Nanda kumar
>Assignee: Nanda kumar
>  Labels: ozoneMerge
> Attachments: HDFS-12680-HDFS-7240.000.patch
>
>
> This brings in lease support for container creation.
> Lease should be give for a container that is moved to {{CREATING}} state when 
> {{BEGIN_CREATE}} event happens, {{LeaseException}} should be thrown if the 
> container already holds a lease. Lease must be released during 
> {{COMPLETE_CREATE}} event. If the lease times out container should be moved 
> to {{DELETING}} state, and exception should be thrown if {{COMPLETE_CREATE}} 
> event is received on that container.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-7878) API - expose an unique file identifier

[Chris Douglas (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-7878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16210392#comment-16210392
 ] 

Chris Douglas commented on HDFS-7878:
-

Failure is due to HADOOP-14961

> API - expose an unique file identifier
> --
>
> Key: HDFS-7878
> URL: https://issues.apache.org/jira/browse/HDFS-7878
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>  Labels: BB2015-05-TBR
> Attachments: HDFS-7878.01.patch, HDFS-7878.02.patch, 
> HDFS-7878.03.patch, HDFS-7878.04.patch, HDFS-7878.05.patch, 
> HDFS-7878.06.patch, HDFS-7878.07.patch, HDFS-7878.08.patch, 
> HDFS-7878.09.patch, HDFS-7878.10.patch, HDFS-7878.11.patch, 
> HDFS-7878.12.patch, HDFS-7878.13.patch, HDFS-7878.14.patch, HDFS-7878.patch
>
>
> See HDFS-487.
> Even though that is resolved as duplicate, the ID is actually not exposed by 
> the JIRA it supposedly duplicates.
> INode ID for the file should be easy to expose; alternatively ID could be 
> derived from block IDs, to account for appends...
> This is useful e.g. for cache key by file, to make sure cache stays correct 
> when file is overwritten.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-12420) Add an option to disallow 'namenode format -force'

[Junping Du (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-12420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Junping Du updated HDFS-12420:
--
Fix Version/s: (was: 2.8.2)
   2.8.3

> Add an option to disallow 'namenode format -force'
> --
>
> Key: HDFS-12420
> URL: https://issues.apache.org/jira/browse/HDFS-12420
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Reporter: Ajay Kumar
>Assignee: Ajay Kumar
> Fix For: 2.9.0, 2.8.3, 2.7.5, 3.0.0, 3.1.0
>
> Attachments: HDFS-12420.01.patch, HDFS-12420.02.patch, 
> HDFS-12420.03.patch, HDFS-12420.04.patch, HDFS-12420.05.patch, 
> HDFS-12420.06.patch, HDFS-12420.07.patch, HDFS-12420.08.patch, 
> HDFS-12420.09.patch, HDFS-12420.10.patch, HDFS-12420.11.patch, 
> HDFS-12420.12.patch
>
>
> Support for disabling NameNode format to avoid accidental formatting of 
> Namenode in production cluster. If someone really wants to delete the 
> complete fsImage, they can first delete the metadata dir and then run {code} 
> hdfs namenode -format{code} manually.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-12420) Add an option to disallow 'namenode format -force'

[Junping Du (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-12420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16210386#comment-16210386
 ] 

Junping Du commented on HDFS-12420:
---

Drop 2.8.2 from fix version as the patch only commit to branch-2.8 instead of 
branch-2.8.2.

> Add an option to disallow 'namenode format -force'
> --
>
> Key: HDFS-12420
> URL: https://issues.apache.org/jira/browse/HDFS-12420
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Reporter: Ajay Kumar
>Assignee: Ajay Kumar
> Fix For: 2.9.0, 2.8.3, 2.7.5, 3.0.0, 3.1.0
>
> Attachments: HDFS-12420.01.patch, HDFS-12420.02.patch, 
> HDFS-12420.03.patch, HDFS-12420.04.patch, HDFS-12420.05.patch, 
> HDFS-12420.06.patch, HDFS-12420.07.patch, HDFS-12420.08.patch, 
> HDFS-12420.09.patch, HDFS-12420.10.patch, HDFS-12420.11.patch, 
> HDFS-12420.12.patch
>
>
> Support for disabling NameNode format to avoid accidental formatting of 
> Namenode in production cluster. If someone really wants to delete the 
> complete fsImage, they can first delete the metadata dir and then run {code} 
> hdfs namenode -format{code} manually.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-11902) [READ] Merge BlockFormatProvider and FileRegionProvider.

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-11902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16210362#comment-16210362
 ] 

Hadoop QA commented on HDFS-11902:
--

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m  
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} docker {color} | {color:red}  0m 
12s{color} | {color:red} Docker failed to build yetus/hadoop:71bbb86. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HDFS-11902 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12892943/HDFS-11902-HDFS-9806.009.patch
 |
| Console output | 
https://builds.apache.org/job/PreCommit-HDFS-Build/21743/console |
| Powered by | Apache Yetus 0.6.0-SNAPSHOT   http://yetus.apache.org |


This message was automatically generated.



> [READ] Merge BlockFormatProvider and FileRegionProvider.
> 
>
> Key: HDFS-11902
> URL: https://issues.apache.org/jira/browse/HDFS-11902
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>Reporter: Virajith Jalaparti
>Assignee: Virajith Jalaparti
> Attachments: HDFS-11902-HDFS-9806.001.patch, 
> HDFS-11902-HDFS-9806.002.patch, HDFS-11902-HDFS-9806.003.patch, 
> HDFS-11902-HDFS-9806.004.patch, HDFS-11902-HDFS-9806.005.patch, 
> HDFS-11902-HDFS-9806.006.patch, HDFS-11902-HDFS-9806.007.patch, 
> HDFS-11902-HDFS-9806.008.patch, HDFS-11902-HDFS-9806.009.patch
>
>
> Currently {{BlockFormatProvider}} and {{TextFileRegionProvider}} perform 
> almost the same function on the Namenode and Datanode respectively. This JIRA 
> is to merge them into one.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-12620) Backporting HDFS-10467 to branch-2

[JIRA]

 [ 
https://issues.apache.org/jira/browse/HDFS-12620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Íñigo Goiri updated HDFS-12620:
---
Attachment: HDFS-12620-branch-2.010.patch

> Backporting HDFS-10467 to branch-2
> --
>
> Key: HDFS-12620
> URL: https://issues.apache.org/jira/browse/HDFS-12620
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Reporter: Íñigo Goiri
>Assignee: Íñigo Goiri
> Attachments: HDFS-10467-branch-2.001.patch, 
> HDFS-10467-branch-2.002.patch, HDFS-10467-branch-2.003.patch, 
> HDFS-10467-branch-2.patch, HDFS-12620-branch-2.000.patch, 
> HDFS-12620-branch-2.004.patch, HDFS-12620-branch-2.005.patch, 
> HDFS-12620-branch-2.006.patch, HDFS-12620-branch-2.007.patch, 
> HDFS-12620-branch-2.008.patch, HDFS-12620-branch-2.009.patch, 
> HDFS-12620-branch-2.010.patch
>
>
> When backporting HDFS-10467, there are a few things that changed:
> * {{bin\hdfs}}
> * {{ClientProtocol}}
> * Java 7 not supporting referencing functions
> * {{org.eclipse.jetty.util.ajax.JSON}} in branch-2 is 
> {{org.mortbay.util.ajax.JSON}}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-11902) [READ] Merge BlockFormatProvider and FileRegionProvider.

[Virajith Jalaparti (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-11902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Virajith Jalaparti updated HDFS-11902:
--
Status: Patch Available  (was: Open)

> [READ] Merge BlockFormatProvider and FileRegionProvider.
> 
>
> Key: HDFS-11902
> URL: https://issues.apache.org/jira/browse/HDFS-11902
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>Reporter: Virajith Jalaparti
>Assignee: Virajith Jalaparti
> Attachments: HDFS-11902-HDFS-9806.001.patch, 
> HDFS-11902-HDFS-9806.002.patch, HDFS-11902-HDFS-9806.003.patch, 
> HDFS-11902-HDFS-9806.004.patch, HDFS-11902-HDFS-9806.005.patch, 
> HDFS-11902-HDFS-9806.006.patch, HDFS-11902-HDFS-9806.007.patch, 
> HDFS-11902-HDFS-9806.008.patch, HDFS-11902-HDFS-9806.009.patch
>
>
> Currently {{BlockFormatProvider}} and {{TextFileRegionProvider}} perform 
> almost the same function on the Namenode and Datanode respectively. This JIRA 
> is to merge them into one.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-11902) [READ] Merge BlockFormatProvider and FileRegionProvider.

[Virajith Jalaparti (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-11902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Virajith Jalaparti updated HDFS-11902:
--
Attachment: (was: HDFS-11902-HDFS-9806.009.patch)

> [READ] Merge BlockFormatProvider and FileRegionProvider.
> 
>
> Key: HDFS-11902
> URL: https://issues.apache.org/jira/browse/HDFS-11902
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>Reporter: Virajith Jalaparti
>Assignee: Virajith Jalaparti
> Attachments: HDFS-11902-HDFS-9806.001.patch, 
> HDFS-11902-HDFS-9806.002.patch, HDFS-11902-HDFS-9806.003.patch, 
> HDFS-11902-HDFS-9806.004.patch, HDFS-11902-HDFS-9806.005.patch, 
> HDFS-11902-HDFS-9806.006.patch, HDFS-11902-HDFS-9806.007.patch, 
> HDFS-11902-HDFS-9806.008.patch, HDFS-11902-HDFS-9806.009.patch
>
>
> Currently {{BlockFormatProvider}} and {{TextFileRegionProvider}} perform 
> almost the same function on the Namenode and Datanode respectively. This JIRA 
> is to merge them into one.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-11902) [READ] Merge BlockFormatProvider and FileRegionProvider.

[Virajith Jalaparti (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-11902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Virajith Jalaparti updated HDFS-11902:
--
Attachment: HDFS-11902-HDFS-9806.009.patch

> [READ] Merge BlockFormatProvider and FileRegionProvider.
> 
>
> Key: HDFS-11902
> URL: https://issues.apache.org/jira/browse/HDFS-11902
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>Reporter: Virajith Jalaparti
>Assignee: Virajith Jalaparti
> Attachments: HDFS-11902-HDFS-9806.001.patch, 
> HDFS-11902-HDFS-9806.002.patch, HDFS-11902-HDFS-9806.003.patch, 
> HDFS-11902-HDFS-9806.004.patch, HDFS-11902-HDFS-9806.005.patch, 
> HDFS-11902-HDFS-9806.006.patch, HDFS-11902-HDFS-9806.007.patch, 
> HDFS-11902-HDFS-9806.008.patch, HDFS-11902-HDFS-9806.009.patch
>
>
> Currently {{BlockFormatProvider}} and {{TextFileRegionProvider}} perform 
> almost the same function on the Namenode and Datanode respectively. This JIRA 
> is to merge them into one.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-11902) [READ] Merge BlockFormatProvider and FileRegionProvider.

[Virajith Jalaparti (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-11902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Virajith Jalaparti updated HDFS-11902:
--
Status: Open  (was: Patch Available)

> [READ] Merge BlockFormatProvider and FileRegionProvider.
> 
>
> Key: HDFS-11902
> URL: https://issues.apache.org/jira/browse/HDFS-11902
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>Reporter: Virajith Jalaparti
>Assignee: Virajith Jalaparti
> Attachments: HDFS-11902-HDFS-9806.001.patch, 
> HDFS-11902-HDFS-9806.002.patch, HDFS-11902-HDFS-9806.003.patch, 
> HDFS-11902-HDFS-9806.004.patch, HDFS-11902-HDFS-9806.005.patch, 
> HDFS-11902-HDFS-9806.006.patch, HDFS-11902-HDFS-9806.007.patch, 
> HDFS-11902-HDFS-9806.008.patch, HDFS-11902-HDFS-9806.009.patch
>
>
> Currently {{BlockFormatProvider}} and {{TextFileRegionProvider}} perform 
> almost the same function on the Namenode and Datanode respectively. This JIRA 
> is to merge them into one.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-12620) Backporting HDFS-10467 to branch-2

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-12620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16210346#comment-16210346
 ] 

Hadoop QA commented on HDFS-12620:
--

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
26s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} shelldocs {color} | {color:blue}  0m  
0s{color} | {color:blue} Shelldocs was not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 25 new or modified test 
files. {color} |
|| || || || {color:brown} branch-2 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  7m 
48s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
50s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
42s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
57s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  2m  
5s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  
9s{color} | {color:green} branch-2 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red}  0m 
49s{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red}  0m 
47s{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} cc {color} | {color:red}  0m 47s{color} | 
{color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red}  0m 47s{color} 
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
0m 36s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch 
generated 12 new + 624 unchanged - 0 fixed = 636 total (was 624) {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red}  0m 
50s{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} shellcheck {color} | {color:green}  0m 
 0s{color} | {color:green} There were no new shellcheck issues. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 1s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green}  0m  
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  0m 
25s{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
12s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}  0m 43s{color} 
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
20s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 21m 36s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:eaf5c66 |
| JIRA Issue | HDFS-12620 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12892937/HDFS-12620-branch-2.009.patch
 |
| Optional Tests |  asflicense  xml  compile  javac  javadoc  mvninstall  
mvnsite  unit  shadedclient  shellcheck  shelldocs  findbugs  checkstyle  cc  |
| uname | Linux c92d64cfcebf 3.13.0-129-generic #178-Ubuntu SMP Fri Aug 11 
12:48:20 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | branch-2 / e7e2b81 |
| Default Java | 1.7.0_151 |
| shellcheck | v0.4.6 |
| findbugs | v3.0.0 |
| mvninstall | 
https://builds.apache.org/job/PreCommit-HDFS-Build/21741/artifact/patchprocess/patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs.txt
 |
| compile | 

[jira] [Commented] (HDFS-12502) nntop should support a category based on FilesInGetListingOps

[Konstantin Shvachko (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-12502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16210343#comment-16210343
 ] 

Konstantin Shvachko commented on HDFS-12502:


+1 for the latest patch.

> nntop should support a category based on FilesInGetListingOps
> -
>
> Key: HDFS-12502
> URL: https://issues.apache.org/jira/browse/HDFS-12502
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: metrics
>Reporter: Zhe Zhang
>Assignee: Zhe Zhang
> Attachments: HDFS-12502.00.patch, HDFS-12502.01.patch, 
> HDFS-12502.02.patch, HDFS-12502.03.patch, HDFS-12502.04.patch
>
>
> Large listing ops can oftentimes be the main contributor to NameNode 
> slowness. The aggregate cost of listing ops is proportional to the 
> {{FilesInGetListingOps}} rather than the number of listing ops. Therefore 
> it'd be very useful for nntop to support this category.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-12682) ECAdmin -listPolicies will always show policy state as DISABLED

[Xiao Chen (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-12682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16210317#comment-16210317
 ] 

Xiao Chen commented on HDFS-12682:
--

Had a quick discussion with Andrew, here's the outcome:
ErasureCodingPolicyState should not be part of the ErasureCodingPolicy, since 
ECP should be immutable. This was added recently from HDFS-12258.

The concern is that ECP in part of HdfsFileStatus, so pretty impactful to 
clients doing listing. Although apps like hive and impala probably doesn't care 
about the ECP, returning a wrong ECPS will just be a bug.

I'll look at whether we can do HDFS-12258 in another way, so we HdfsFileStatus 
only contains immutable ECP, and -listPolices would get the ECPS with ECP.

Thanks Andrew for the discussion!

> ECAdmin -listPolicies will always show policy state as DISABLED
> ---
>
> Key: HDFS-12682
> URL: https://issues.apache.org/jira/browse/HDFS-12682
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: erasure-coding
>Reporter: Xiao Chen
>Assignee: Xiao Chen
>  Labels: hdfs-ec-3.0-must-do
>
> On a real cluster, {{hdfs ec -listPolicies}} will always show policy state as 
> DISABLED.
> {noformat}
> [hdfs@nightly6x-1 root]$ hdfs ec -listPolicies
> Erasure Coding Policies:
> ErasureCodingPolicy=[Name=RS-10-4-1024k, Schema=[ECSchema=[Codec=rs, 
> numDataUnits=10, numParityUnits=4]], CellSize=1048576, Id=5, State=DISABLED]
> ErasureCodingPolicy=[Name=RS-3-2-1024k, Schema=[ECSchema=[Codec=rs, 
> numDataUnits=3, numParityUnits=2]], CellSize=1048576, Id=2, State=DISABLED]
> ErasureCodingPolicy=[Name=RS-6-3-1024k, Schema=[ECSchema=[Codec=rs, 
> numDataUnits=6, numParityUnits=3]], CellSize=1048576, Id=1, State=DISABLED]
> ErasureCodingPolicy=[Name=RS-LEGACY-6-3-1024k, 
> Schema=[ECSchema=[Codec=rs-legacy, numDataUnits=6, numParityUnits=3]], 
> CellSize=1048576, Id=3, State=DISABLED]
> ErasureCodingPolicy=[Name=XOR-2-1-1024k, Schema=[ECSchema=[Codec=xor, 
> numDataUnits=2, numParityUnits=1]], CellSize=1048576, Id=4, State=DISABLED]
> [hdfs@nightly6x-1 root]$ hdfs ec -getPolicy -path /ecec
> XOR-2-1-1024k
> {noformat}
> This is because when [deserializing 
> protobuf|https://github.com/apache/hadoop/blob/branch-3.0.0-beta1/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelperClient.java#L2942],
>  the static instance of [SystemErasureCodingPolicies 
> class|https://github.com/apache/hadoop/blob/branch-3.0.0-beta1/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocol/SystemErasureCodingPolicies.java#L101]
>  is first checked, and always returns the cached policy objects, which are 
> created by default with state=DISABLED.
> All the existing unit tests pass, because that static instance that the 
> client (e.g. ECAdmin) reads in unit test is updated by NN. :)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Comment Edited] (HDFS-8921) Add an option to Balancer for specifying the k-most over-utilized DNs or all over-utilized DNs as sources.

[Plamen Jeliazkov (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-8921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16210274#comment-16210274
 ] 

Plamen Jeliazkov edited comment on HDFS-8921 at 10/18/17 11:26 PM:
---

Yes, I am interested. I have a rough first patch. Would you help review it?

I can confirm it caps over-utilized sources but we should consider whether we 
should cap above-average and the other possible sources as well.


was (Author: zero45):
Yes, I am interested. I have a rough first patch. Would you help review it with 
me?

I can confirm it caps over-utilized sources but we should consider whether we 
should cap above-average and the other possible sources as well.

> Add an option to Balancer for specifying the k-most over-utilized DNs or all 
> over-utilized DNs as sources.
> --
>
> Key: HDFS-8921
> URL: https://issues.apache.org/jira/browse/HDFS-8921
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: balancer & mover
>Reporter: Tsz Wo Nicholas Sze
>Assignee: Plamen Jeliazkov
> Attachments: HDFS-8921.patch
>
>
> Arpit suggested to add a separate option to source from the most 
> over-utilized DataNodes first so the administrator does not have to pass the 
> source DNs manually; see [this 
> comment|https://issues.apache.org/jira/browse/HDFS-8826?focusedCommentId=14700576=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14700576].
>   The new option could allow specifying the k-most over-utilized DNs or all 
> over-utilized DNs as sources.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-12495) TestPendingInvalidateBlock#testPendingDeleteUnknownBlocks fails intermittently

[Junping Du (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-12495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16210288#comment-16210288
 ] 

Junping Du commented on HDFS-12495:
---

I don't find the patch landed in branch-2.8.2. Drop 2.8.2 in fix version.

> TestPendingInvalidateBlock#testPendingDeleteUnknownBlocks fails intermittently
> --
>
> Key: HDFS-12495
> URL: https://issues.apache.org/jira/browse/HDFS-12495
> Project: Hadoop HDFS
>  Issue Type: Bug
>Affects Versions: 2.9.0, 3.0.0-beta1, 2.8.2
>Reporter: Eric Badger
>Assignee: Eric Badger
>  Labels: flaky-test
> Fix For: 2.9.0, 3.0.0-beta1, 2.8.3, 3.0.0, 3.1.0
>
> Attachments: HDFS-12495.001.patch, HDFS-12495.002.patch
>
>
> {noformat}
> java.net.BindException: Problem binding to [localhost:36701] 
> java.net.BindException: Address already in use; For more details see:  
> http://wiki.apache.org/hadoop/BindException
>   at sun.nio.ch.Net.bind0(Native Method)
>   at sun.nio.ch.Net.bind(Net.java:433)
>   at sun.nio.ch.Net.bind(Net.java:425)
>   at 
> sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
>   at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
>   at org.apache.hadoop.ipc.Server.bind(Server.java:546)
>   at org.apache.hadoop.ipc.Server$Listener.(Server.java:955)
>   at org.apache.hadoop.ipc.Server.(Server.java:2655)
>   at org.apache.hadoop.ipc.RPC$Server.(RPC.java:968)
>   at 
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server.(ProtobufRpcEngine.java:367)
>   at 
> org.apache.hadoop.ipc.ProtobufRpcEngine.getServer(ProtobufRpcEngine.java:342)
>   at org.apache.hadoop.ipc.RPC$Builder.build(RPC.java:810)
>   at 
> org.apache.hadoop.hdfs.server.datanode.DataNode.initIpcServer(DataNode.java:954)
>   at 
> org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:1314)
>   at 
> org.apache.hadoop.hdfs.server.datanode.DataNode.(DataNode.java:481)
>   at 
> org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:2611)
>   at 
> org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:2499)
>   at 
> org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:2546)
>   at 
> org.apache.hadoop.hdfs.MiniDFSCluster.restartDataNode(MiniDFSCluster.java:2152)
>   at 
> org.apache.hadoop.hdfs.server.blockmanagement.TestPendingInvalidateBlock.testPendingDeleteUnknownBlocks(TestPendingInvalidateBlock.java:175)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-12495) TestPendingInvalidateBlock#testPendingDeleteUnknownBlocks fails intermittently

[Junping Du (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-12495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Junping Du updated HDFS-12495:
--
Fix Version/s: (was: 2.8.2)

> TestPendingInvalidateBlock#testPendingDeleteUnknownBlocks fails intermittently
> --
>
> Key: HDFS-12495
> URL: https://issues.apache.org/jira/browse/HDFS-12495
> Project: Hadoop HDFS
>  Issue Type: Bug
>Affects Versions: 2.9.0, 3.0.0-beta1, 2.8.2
>Reporter: Eric Badger
>Assignee: Eric Badger
>  Labels: flaky-test
> Fix For: 2.9.0, 3.0.0-beta1, 2.8.3, 3.0.0, 3.1.0
>
> Attachments: HDFS-12495.001.patch, HDFS-12495.002.patch
>
>
> {noformat}
> java.net.BindException: Problem binding to [localhost:36701] 
> java.net.BindException: Address already in use; For more details see:  
> http://wiki.apache.org/hadoop/BindException
>   at sun.nio.ch.Net.bind0(Native Method)
>   at sun.nio.ch.Net.bind(Net.java:433)
>   at sun.nio.ch.Net.bind(Net.java:425)
>   at 
> sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
>   at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
>   at org.apache.hadoop.ipc.Server.bind(Server.java:546)
>   at org.apache.hadoop.ipc.Server$Listener.(Server.java:955)
>   at org.apache.hadoop.ipc.Server.(Server.java:2655)
>   at org.apache.hadoop.ipc.RPC$Server.(RPC.java:968)
>   at 
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server.(ProtobufRpcEngine.java:367)
>   at 
> org.apache.hadoop.ipc.ProtobufRpcEngine.getServer(ProtobufRpcEngine.java:342)
>   at org.apache.hadoop.ipc.RPC$Builder.build(RPC.java:810)
>   at 
> org.apache.hadoop.hdfs.server.datanode.DataNode.initIpcServer(DataNode.java:954)
>   at 
> org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:1314)
>   at 
> org.apache.hadoop.hdfs.server.datanode.DataNode.(DataNode.java:481)
>   at 
> org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:2611)
>   at 
> org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:2499)
>   at 
> org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:2546)
>   at 
> org.apache.hadoop.hdfs.MiniDFSCluster.restartDataNode(MiniDFSCluster.java:2152)
>   at 
> org.apache.hadoop.hdfs.server.blockmanagement.TestPendingInvalidateBlock.testPendingDeleteUnknownBlocks(TestPendingInvalidateBlock.java:175)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Assigned] (HDFS-8921) Add an option to Balancer for specifying the k-most over-utilized DNs or all over-utilized DNs as sources.

[Plamen Jeliazkov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-8921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Plamen Jeliazkov reassigned HDFS-8921:
--

Assignee: Plamen Jeliazkov

> Add an option to Balancer for specifying the k-most over-utilized DNs or all 
> over-utilized DNs as sources.
> --
>
> Key: HDFS-8921
> URL: https://issues.apache.org/jira/browse/HDFS-8921
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: balancer & mover
>Reporter: Tsz Wo Nicholas Sze
>Assignee: Plamen Jeliazkov
>
> Arpit suggested to add a separate option to source from the most 
> over-utilized DataNodes first so the administrator does not have to pass the 
> source DNs manually; see [this 
> comment|https://issues.apache.org/jira/browse/HDFS-8826?focusedCommentId=14700576=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14700576].
>   The new option could allow specifying the k-most over-utilized DNs or all 
> over-utilized DNs as sources.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-12594) SnapshotDiff - snapshotDiff fails if the snapshotDiff report exceeds the RPC response limit

[Tsz Wo Nicholas Sze (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-12594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16210293#comment-16210293
 ] 

Tsz Wo Nicholas Sze commented on HDFS-12594:


The patch seems suggesting some incompatible changes:
- SnapshotDiffReportEntryProto => SnapshotDiffReportListingEntryProto
- SnapshotDiffReportProto => SnapshotDiffReportListingProto
- changing ClientProtocol.getSnapshotDiffReport

We should keep the old API working so that the old clients will work with new 
servers.

I suggest we add a new ClientProtocol.getSnapshotDiffReportListing method and 
define new protos for the iterative snapshot diff report API.

> SnapshotDiff - snapshotDiff fails if the snapshotDiff report exceeds the RPC 
> response limit
> ---
>
> Key: HDFS-12594
> URL: https://issues.apache.org/jira/browse/HDFS-12594
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: hdfs
>Reporter: Shashikant Banerjee
>Assignee: Shashikant Banerjee
> Attachments: HDFS-12594.001.patch, HDFS-12594.002.patch, 
> SnapshotDiff_Improvemnets .pdf
>
>
> The snapshotDiff command fails if the snapshotDiff report size is larger than 
> the configuration value of ipc.maximum.response.length which is by default 
> 128 MB. 
> Worst case, with all Renames ops in sanpshots each with source and target 
> name equal to MAX_PATH_LEN which is 8k characters, this would result in at 
> 8192 renames.
>  
> SnapshotDiff is currently used by distcp to optimize copy operations and in 
> case of the the diff report exceeding the limit , it fails with the below 
> exception:
> Test set: 
> org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotDiffReport
> ---
> Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 112.095 sec 
> <<< FAILURE! - in 
> org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotDiffReport
> testDiffReportWithMillionFiles(org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotDiffReport)
>   Time elapsed: 111.906 sec  <<< ERROR!
> java.io.IOException: Failed on local exception: 
> org.apache.hadoop.ipc.RpcException: RPC response exceeds maximum data length; 
> Host Details : local host is: "hw15685.local/10.200.5.230"; destination host 
> is: "localhost":59808;
> Attached is the proposal for the changes required.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-12620) Backporting HDFS-10467 to branch-2

[JIRA]

 [ 
https://issues.apache.org/jira/browse/HDFS-12620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Íñigo Goiri updated HDFS-12620:
---
Attachment: HDFS-12620-branch-2.009.patch

> Backporting HDFS-10467 to branch-2
> --
>
> Key: HDFS-12620
> URL: https://issues.apache.org/jira/browse/HDFS-12620
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Reporter: Íñigo Goiri
>Assignee: Íñigo Goiri
> Attachments: HDFS-10467-branch-2.001.patch, 
> HDFS-10467-branch-2.002.patch, HDFS-10467-branch-2.003.patch, 
> HDFS-10467-branch-2.patch, HDFS-12620-branch-2.000.patch, 
> HDFS-12620-branch-2.004.patch, HDFS-12620-branch-2.005.patch, 
> HDFS-12620-branch-2.006.patch, HDFS-12620-branch-2.007.patch, 
> HDFS-12620-branch-2.008.patch, HDFS-12620-branch-2.009.patch
>
>
> When backporting HDFS-10467, there are a few things that changed:
> * {{bin\hdfs}}
> * {{ClientProtocol}}
> * Java 7 not supporting referencing functions
> * {{org.eclipse.jetty.util.ajax.JSON}} in branch-2 is 
> {{org.mortbay.util.ajax.JSON}}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-8921) Add an option to Balancer for specifying the k-most over-utilized DNs or all over-utilized DNs as sources.

[Plamen Jeliazkov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-8921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Plamen Jeliazkov updated HDFS-8921:
---
Attachment: HDFS-8921.patch

> Add an option to Balancer for specifying the k-most over-utilized DNs or all 
> over-utilized DNs as sources.
> --
>
> Key: HDFS-8921
> URL: https://issues.apache.org/jira/browse/HDFS-8921
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: balancer & mover
>Reporter: Tsz Wo Nicholas Sze
>Assignee: Plamen Jeliazkov
> Attachments: HDFS-8921.patch
>
>
> Arpit suggested to add a separate option to source from the most 
> over-utilized DataNodes first so the administrator does not have to pass the 
> source DNs manually; see [this 
> comment|https://issues.apache.org/jira/browse/HDFS-8826?focusedCommentId=14700576=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14700576].
>   The new option could allow specifying the k-most over-utilized DNs or all 
> over-utilized DNs as sources.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-8921) Add an option to Balancer for specifying the k-most over-utilized DNs or all over-utilized DNs as sources.

[Plamen Jeliazkov (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-8921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16210274#comment-16210274
 ] 

Plamen Jeliazkov commented on HDFS-8921:


Yes, I am interested. I have a rough first patch. Would you help review it with 
me?

I can confirm it caps over-utilized sources but we should consider whether we 
should cap above-average and the other possible sources as well.

> Add an option to Balancer for specifying the k-most over-utilized DNs or all 
> over-utilized DNs as sources.
> --
>
> Key: HDFS-8921
> URL: https://issues.apache.org/jira/browse/HDFS-8921
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: balancer & mover
>Reporter: Tsz Wo Nicholas Sze
>
> Arpit suggested to add a separate option to source from the most 
> over-utilized DataNodes first so the administrator does not have to pass the 
> source DNs manually; see [this 
> comment|https://issues.apache.org/jira/browse/HDFS-8826?focusedCommentId=14700576=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14700576].
>   The new option could allow specifying the k-most over-utilized DNs or all 
> over-utilized DNs as sources.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-12620) Backporting HDFS-10467 to branch-2

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-12620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16210235#comment-16210235
 ] 

Hadoop QA commented on HDFS-12620:
--

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 11m 
43s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} shelldocs {color} | {color:blue}  0m  
0s{color} | {color:blue} Shelldocs was not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 25 new or modified test 
files. {color} |
|| || || || {color:brown} branch-2 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  7m 
40s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
46s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
41s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
56s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  2m  
1s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  
1s{color} | {color:green} branch-2 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red}  0m 
25s{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red}  0m 
24s{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} cc {color} | {color:red}  0m 24s{color} | 
{color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red}  0m 24s{color} 
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
0m 38s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch 
generated 12 new + 624 unchanged - 0 fixed = 636 total (was 624) {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red}  0m 
27s{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} shellcheck {color} | {color:green}  0m 
 1s{color} | {color:green} There were no new shellcheck issues. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 1s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green}  0m  
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  0m 
26s{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red}  1m  
9s{color} | {color:red} hadoop-hdfs-project_hadoop-hdfs generated 3 new + 9 
unchanged - 0 fixed = 12 total (was 9) {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}  0m 27s{color} 
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
21s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 31m  5s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:eaf5c66 |
| JIRA Issue | HDFS-12620 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12892922/HDFS-12620-branch-2.008.patch
 |
| Optional Tests |  asflicense  xml  compile  javac  javadoc  mvninstall  
mvnsite  unit  shadedclient  shellcheck  shelldocs  findbugs  checkstyle  cc  |
| uname | Linux bf5711e7f0bc 3.13.0-129-generic #178-Ubuntu SMP Fri Aug 11 
12:48:20 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | branch-2 / e7e2b81 |
| Default Java | 1.7.0_151 |
| shellcheck | v0.4.6 |
| findbugs | v3.0.0 |
| mvninstall | 
https://builds.apache.org/job/PreCommit-HDFS-Build/21740/artifact/patchprocess/patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs.txt
 |
| compile | 

[jira] [Commented] (HDFS-12682) ECAdmin -listPolicies will always show policy state as DISABLED

[Andrew Wang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-12682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16210230#comment-16210230
 ] 

Andrew Wang commented on HDFS-12682:


My concern was actually for the clients, since there are apps (Hive, Impala) 
that do listing of thousands or millions of files.

I assume we can do a hybrid approach, where we get most of the fields from the 
static class, but get the enabled/disabled state from the PB?

> ECAdmin -listPolicies will always show policy state as DISABLED
> ---
>
> Key: HDFS-12682
> URL: https://issues.apache.org/jira/browse/HDFS-12682
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: erasure-coding
>Reporter: Xiao Chen
>Assignee: Xiao Chen
>  Labels: hdfs-ec-3.0-must-do
>
> On a real cluster, {{hdfs ec -listPolicies}} will always show policy state as 
> DISABLED.
> {noformat}
> [hdfs@nightly6x-1 root]$ hdfs ec -listPolicies
> Erasure Coding Policies:
> ErasureCodingPolicy=[Name=RS-10-4-1024k, Schema=[ECSchema=[Codec=rs, 
> numDataUnits=10, numParityUnits=4]], CellSize=1048576, Id=5, State=DISABLED]
> ErasureCodingPolicy=[Name=RS-3-2-1024k, Schema=[ECSchema=[Codec=rs, 
> numDataUnits=3, numParityUnits=2]], CellSize=1048576, Id=2, State=DISABLED]
> ErasureCodingPolicy=[Name=RS-6-3-1024k, Schema=[ECSchema=[Codec=rs, 
> numDataUnits=6, numParityUnits=3]], CellSize=1048576, Id=1, State=DISABLED]
> ErasureCodingPolicy=[Name=RS-LEGACY-6-3-1024k, 
> Schema=[ECSchema=[Codec=rs-legacy, numDataUnits=6, numParityUnits=3]], 
> CellSize=1048576, Id=3, State=DISABLED]
> ErasureCodingPolicy=[Name=XOR-2-1-1024k, Schema=[ECSchema=[Codec=xor, 
> numDataUnits=2, numParityUnits=1]], CellSize=1048576, Id=4, State=DISABLED]
> [hdfs@nightly6x-1 root]$ hdfs ec -getPolicy -path /ecec
> XOR-2-1-1024k
> {noformat}
> This is because when [deserializing 
> protobuf|https://github.com/apache/hadoop/blob/branch-3.0.0-beta1/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelperClient.java#L2942],
>  the static instance of [SystemErasureCodingPolicies 
> class|https://github.com/apache/hadoop/blob/branch-3.0.0-beta1/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocol/SystemErasureCodingPolicies.java#L101]
>  is first checked, and always returns the cached policy objects, which are 
> created by default with state=DISABLED.
> All the existing unit tests pass, because that static instance that the 
> client (e.g. ECAdmin) reads in unit test is updated by NN. :)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-12683) DFSZKFailOverController re-order logic for logging Exception

[Bharat Viswanadham (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-12683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bharat Viswanadham updated HDFS-12683:
--
Description: 
Log the exception before closing the connections and terminating server.

As some times occasionally we have seen DFSZKFailOver shutdown, but no 
exception or no error being logged.

  was:Log the exception before closing the connections and terminating server.


> DFSZKFailOverController re-order logic for logging Exception
> 
>
> Key: HDFS-12683
> URL: https://issues.apache.org/jira/browse/HDFS-12683
> Project: Hadoop HDFS
>  Issue Type: Bug
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>
> Log the exception before closing the connections and terminating server.
> As some times occasionally we have seen DFSZKFailOver shutdown, but no 
> exception or no error being logged.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Created] (HDFS-12683) DFSZKFailOverController re-order logic for logging Exception

[Bharat Viswanadham (JIRA)]

Bharat Viswanadham created HDFS-12683:
-

 Summary: DFSZKFailOverController re-order logic for logging 
Exception
 Key: HDFS-12683
 URL: https://issues.apache.org/jira/browse/HDFS-12683
 Project: Hadoop HDFS
  Issue Type: Bug
Reporter: Bharat Viswanadham


Log the exception before closing the connections and terminating server.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Work started] (HDFS-12683) DFSZKFailOverController re-order logic for logging Exception

[Bharat Viswanadham (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-12683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on HDFS-12683 started by Bharat Viswanadham.
-
> DFSZKFailOverController re-order logic for logging Exception
> 
>
> Key: HDFS-12683
> URL: https://issues.apache.org/jira/browse/HDFS-12683
> Project: Hadoop HDFS
>  Issue Type: Bug
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>
> Log the exception before closing the connections and terminating server.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Assigned] (HDFS-12683) DFSZKFailOverController re-order logic for logging Exception

[Bharat Viswanadham (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-12683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bharat Viswanadham reassigned HDFS-12683:
-

Assignee: Bharat Viswanadham

> DFSZKFailOverController re-order logic for logging Exception
> 
>
> Key: HDFS-12683
> URL: https://issues.apache.org/jira/browse/HDFS-12683
> Project: Hadoop HDFS
>  Issue Type: Bug
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>
> Log the exception before closing the connections and terminating server.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-12682) ECAdmin -listPolicies will always show policy state as DISABLED

[Xiao Chen (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-12682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16210220#comment-16210220
 ] 

Xiao Chen commented on HDFS-12682:
--

Looking at the history of HDFS-11565, I understand this is done to save the 
redundant object constructions in NN.

I'm thinking we should keep the same for the NN, but construct every time for 
the client. 
Don't think this overhead will matter for a downstream long-running dfsclient 
that calls {{DFS#getAllErasureCodingPolicies}} repeatedly, since there will be 
numerous other objects created during the RPC. There will also be the headache 
to update these objects without the FSN lock.

Would this be okay with the intent of HDFS-11565, [~andrew.wang]?

> ECAdmin -listPolicies will always show policy state as DISABLED
> ---
>
> Key: HDFS-12682
> URL: https://issues.apache.org/jira/browse/HDFS-12682
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: erasure-coding
>Reporter: Xiao Chen
>Assignee: Xiao Chen
>  Labels: hdfs-ec-3.0-must-do
>
> On a real cluster, {{hdfs ec -listPolicies}} will always show policy state as 
> DISABLED.
> {noformat}
> [hdfs@nightly6x-1 root]$ hdfs ec -listPolicies
> Erasure Coding Policies:
> ErasureCodingPolicy=[Name=RS-10-4-1024k, Schema=[ECSchema=[Codec=rs, 
> numDataUnits=10, numParityUnits=4]], CellSize=1048576, Id=5, State=DISABLED]
> ErasureCodingPolicy=[Name=RS-3-2-1024k, Schema=[ECSchema=[Codec=rs, 
> numDataUnits=3, numParityUnits=2]], CellSize=1048576, Id=2, State=DISABLED]
> ErasureCodingPolicy=[Name=RS-6-3-1024k, Schema=[ECSchema=[Codec=rs, 
> numDataUnits=6, numParityUnits=3]], CellSize=1048576, Id=1, State=DISABLED]
> ErasureCodingPolicy=[Name=RS-LEGACY-6-3-1024k, 
> Schema=[ECSchema=[Codec=rs-legacy, numDataUnits=6, numParityUnits=3]], 
> CellSize=1048576, Id=3, State=DISABLED]
> ErasureCodingPolicy=[Name=XOR-2-1-1024k, Schema=[ECSchema=[Codec=xor, 
> numDataUnits=2, numParityUnits=1]], CellSize=1048576, Id=4, State=DISABLED]
> [hdfs@nightly6x-1 root]$ hdfs ec -getPolicy -path /ecec
> XOR-2-1-1024k
> {noformat}
> This is because when [deserializing 
> protobuf|https://github.com/apache/hadoop/blob/branch-3.0.0-beta1/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelperClient.java#L2942],
>  the static instance of [SystemErasureCodingPolicies 
> class|https://github.com/apache/hadoop/blob/branch-3.0.0-beta1/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocol/SystemErasureCodingPolicies.java#L101]
>  is first checked, and always returns the cached policy objects, which are 
> created by default with state=DISABLED.
> All the existing unit tests pass, because that static instance that the 
> client (e.g. ECAdmin) reads in unit test is updated by NN. :)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-12682) ECAdmin -listPolicies will always show policy state as DISABLED

[Xiao Chen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-12682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xiao Chen updated HDFS-12682:
-
Description: 
On a real cluster, {{hdfs ec -listPolicies}} will always show policy state as 
DISABLED.

{noformat}
[hdfs@nightly6x-1 root]$ hdfs ec -listPolicies
Erasure Coding Policies:
ErasureCodingPolicy=[Name=RS-10-4-1024k, Schema=[ECSchema=[Codec=rs, 
numDataUnits=10, numParityUnits=4]], CellSize=1048576, Id=5, State=DISABLED]
ErasureCodingPolicy=[Name=RS-3-2-1024k, Schema=[ECSchema=[Codec=rs, 
numDataUnits=3, numParityUnits=2]], CellSize=1048576, Id=2, State=DISABLED]
ErasureCodingPolicy=[Name=RS-6-3-1024k, Schema=[ECSchema=[Codec=rs, 
numDataUnits=6, numParityUnits=3]], CellSize=1048576, Id=1, State=DISABLED]
ErasureCodingPolicy=[Name=RS-LEGACY-6-3-1024k, 
Schema=[ECSchema=[Codec=rs-legacy, numDataUnits=6, numParityUnits=3]], 
CellSize=1048576, Id=3, State=DISABLED]
ErasureCodingPolicy=[Name=XOR-2-1-1024k, Schema=[ECSchema=[Codec=xor, 
numDataUnits=2, numParityUnits=1]], CellSize=1048576, Id=4, State=DISABLED]
[hdfs@nightly6x-1 root]$ hdfs ec -getPolicy -path /ecec
XOR-2-1-1024k
{noformat}

This is because when [deserializing 
protobuf|https://github.com/apache/hadoop/blob/branch-3.0.0-beta1/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelperClient.java#L2942],
 the static instance of [SystemErasureCodingPolicies 
class|https://github.com/apache/hadoop/blob/branch-3.0.0-beta1/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocol/SystemErasureCodingPolicies.java#L101]
 is first checked, and always returns the cached policy objects, which are 
created by default with state=DISABLED.

All the existing unit tests pass, because that static instance that the client 
(e.g. ECAdmin) reads in unit test is updated by NN. :)

  was:
On a real cluster, {{hdfs ec -listPolicies}} will always show policy state as 
DISABLED.

{noformat}
[hdfs@nightly6x-1 root]$ hdfs ec -listPolicies
Erasure Coding Policies:
ErasureCodingPolicy=[Name=RS-10-4-1024k, Schema=[ECSchema=[Codec=rs, 
numDataUnits=10, numParityUnits=4]], CellSize=1048576, Id=5, State=DISABLED]
ErasureCodingPolicy=[Name=RS-3-2-1024k, Schema=[ECSchema=[Codec=rs, 
numDataUnits=3, numParityUnits=2]], CellSize=1048576, Id=2, State=DISABLED]
ErasureCodingPolicy=[Name=RS-6-3-1024k, Schema=[ECSchema=[Codec=rs, 
numDataUnits=6, numParityUnits=3]], CellSize=1048576, Id=1, State=DISABLED]
ErasureCodingPolicy=[Name=RS-LEGACY-6-3-1024k, 
Schema=[ECSchema=[Codec=rs-legacy, numDataUnits=6, numParityUnits=3]], 
CellSize=1048576, Id=3, State=DISABLED]
ErasureCodingPolicy=[Name=XOR-2-1-1024k, Schema=[ECSchema=[Codec=xor, 
numDataUnits=2, numParityUnits=1]], CellSize=1048576, Id=4, State=DISABLED]
[hdfs@nightly6x-1 root]$ hdfs ec -getPolicy -path /ecec
XOR-2-1-1024k
{noformat}

This is because when [deserializing 
protobuf|https://github.com/apache/hadoop/blob/branch-3.0.0-beta1/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelperClient.java#L2942],
 the static instance of [SystemErasureCodingPolicies 
class|https://github.com/apache/hadoop/blob/branch-3.0.0-beta1/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocol/SystemErasureCodingPolicies.java#L101]
 is first checked, and always returns the cached policy objects, which are 
created by default with state=DISABLED.


> ECAdmin -listPolicies will always show policy state as DISABLED
> ---
>
> Key: HDFS-12682
> URL: https://issues.apache.org/jira/browse/HDFS-12682
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: erasure-coding
>Reporter: Xiao Chen
>Assignee: Xiao Chen
>  Labels: hdfs-ec-3.0-must-do
>
> On a real cluster, {{hdfs ec -listPolicies}} will always show policy state as 
> DISABLED.
> {noformat}
> [hdfs@nightly6x-1 root]$ hdfs ec -listPolicies
> Erasure Coding Policies:
> ErasureCodingPolicy=[Name=RS-10-4-1024k, Schema=[ECSchema=[Codec=rs, 
> numDataUnits=10, numParityUnits=4]], CellSize=1048576, Id=5, State=DISABLED]
> ErasureCodingPolicy=[Name=RS-3-2-1024k, Schema=[ECSchema=[Codec=rs, 
> numDataUnits=3, numParityUnits=2]], CellSize=1048576, Id=2, State=DISABLED]
> ErasureCodingPolicy=[Name=RS-6-3-1024k, Schema=[ECSchema=[Codec=rs, 
> numDataUnits=6, numParityUnits=3]], CellSize=1048576, Id=1, State=DISABLED]
> ErasureCodingPolicy=[Name=RS-LEGACY-6-3-1024k, 
> Schema=[ECSchema=[Codec=rs-legacy, numDataUnits=6, numParityUnits=3]], 
> CellSize=1048576, Id=3, State=DISABLED]
> ErasureCodingPolicy=[Name=XOR-2-1-1024k, Schema=[ECSchema=[Codec=xor, 
> numDataUnits=2, numParityUnits=1]], CellSize=1048576, Id=4, State=DISABLED]
> [hdfs@nightly6x-1 root]$ hdfs ec -getPolicy -path /ecec
> XOR-2-1-1024k
> {noformat}
> This is 

[jira] [Created] (HDFS-12682) ECAdmin -listPolicies will always show policy state as DISABLED

[Xiao Chen (JIRA)]

Xiao Chen created HDFS-12682:


 Summary: ECAdmin -listPolicies will always show policy state as 
DISABLED
 Key: HDFS-12682
 URL: https://issues.apache.org/jira/browse/HDFS-12682
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: erasure-coding
Reporter: Xiao Chen
Assignee: Xiao Chen


On a real cluster, {{hdfs ec -listPolicies}} will always show policy state as 
DISABLED.

{noformat}
[hdfs@nightly6x-1 root]$ hdfs ec -listPolicies
Erasure Coding Policies:
ErasureCodingPolicy=[Name=RS-10-4-1024k, Schema=[ECSchema=[Codec=rs, 
numDataUnits=10, numParityUnits=4]], CellSize=1048576, Id=5, State=DISABLED]
ErasureCodingPolicy=[Name=RS-3-2-1024k, Schema=[ECSchema=[Codec=rs, 
numDataUnits=3, numParityUnits=2]], CellSize=1048576, Id=2, State=DISABLED]
ErasureCodingPolicy=[Name=RS-6-3-1024k, Schema=[ECSchema=[Codec=rs, 
numDataUnits=6, numParityUnits=3]], CellSize=1048576, Id=1, State=DISABLED]
ErasureCodingPolicy=[Name=RS-LEGACY-6-3-1024k, 
Schema=[ECSchema=[Codec=rs-legacy, numDataUnits=6, numParityUnits=3]], 
CellSize=1048576, Id=3, State=DISABLED]
ErasureCodingPolicy=[Name=XOR-2-1-1024k, Schema=[ECSchema=[Codec=xor, 
numDataUnits=2, numParityUnits=1]], CellSize=1048576, Id=4, State=DISABLED]
[hdfs@nightly6x-1 root]$ hdfs ec -getPolicy -path /ecec
XOR-2-1-1024k
{noformat}

This is because when [deserializing 
protobuf|https://github.com/apache/hadoop/blob/branch-3.0.0-beta1/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelperClient.java#L2942],
 the static instance of [SystemErasureCodingPolicies 
class|https://github.com/apache/hadoop/blob/branch-3.0.0-beta1/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocol/SystemErasureCodingPolicies.java#L101]
 is first checked, and always returns the cached policy objects, which are 
created by default with state=DISABLED.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-12620) Backporting HDFS-10467 to branch-2

[JIRA]

 [ 
https://issues.apache.org/jira/browse/HDFS-12620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Íñigo Goiri updated HDFS-12620:
---
Attachment: HDFS-12620-branch-2.008.patch

> Backporting HDFS-10467 to branch-2
> --
>
> Key: HDFS-12620
> URL: https://issues.apache.org/jira/browse/HDFS-12620
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Reporter: Íñigo Goiri
>Assignee: Íñigo Goiri
> Attachments: HDFS-10467-branch-2.001.patch, 
> HDFS-10467-branch-2.002.patch, HDFS-10467-branch-2.003.patch, 
> HDFS-10467-branch-2.patch, HDFS-12620-branch-2.000.patch, 
> HDFS-12620-branch-2.004.patch, HDFS-12620-branch-2.005.patch, 
> HDFS-12620-branch-2.006.patch, HDFS-12620-branch-2.007.patch, 
> HDFS-12620-branch-2.008.patch
>
>
> When backporting HDFS-10467, there are a few things that changed:
> * {{bin\hdfs}}
> * {{ClientProtocol}}
> * Java 7 not supporting referencing functions
> * {{org.eclipse.jetty.util.ajax.JSON}} in branch-2 is 
> {{org.mortbay.util.ajax.JSON}}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-7878) API - expose an unique file identifier

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-7878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16210116#comment-16210116
 ] 

Hadoop QA commented on HDFS-7878:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m  
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} docker {color} | {color:red}  2m 
50s{color} | {color:red} Docker failed to build yetus/hadoop:0de40f0. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HDFS-7878 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12892907/HDFS-7878.14.patch |
| Console output | 
https://builds.apache.org/job/PreCommit-HDFS-Build/21739/console |
| Powered by | Apache Yetus 0.6.0-SNAPSHOT   http://yetus.apache.org |


This message was automatically generated.



> API - expose an unique file identifier
> --
>
> Key: HDFS-7878
> URL: https://issues.apache.org/jira/browse/HDFS-7878
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>  Labels: BB2015-05-TBR
> Attachments: HDFS-7878.01.patch, HDFS-7878.02.patch, 
> HDFS-7878.03.patch, HDFS-7878.04.patch, HDFS-7878.05.patch, 
> HDFS-7878.06.patch, HDFS-7878.07.patch, HDFS-7878.08.patch, 
> HDFS-7878.09.patch, HDFS-7878.10.patch, HDFS-7878.11.patch, 
> HDFS-7878.12.patch, HDFS-7878.13.patch, HDFS-7878.14.patch, HDFS-7878.patch
>
>
> See HDFS-487.
> Even though that is resolved as duplicate, the ID is actually not exposed by 
> the JIRA it supposedly duplicates.
> INode ID for the file should be easy to expose; alternatively ID could be 
> derived from block IDs, to account for appends...
> This is useful e.g. for cache key by file, to make sure cache stays correct 
> when file is overwritten.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-12310) [SPS]: Provide an option to track the status of in progress requests

[Lei (Eddy) Xu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-12310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16210106#comment-16210106
 ] 

Lei (Eddy) Xu commented on HDFS-12310:
--

Hi, [~surendrasingh]

Thanks for working on it. Please see the following reviews:

* Please fix the comments:
 {code}
 /**
   * @return ture if all the files satisfy policy in given path.
   * @throws IOException
   */
  public StoragePolicySatisfyPathStatus checkStoragePolicySatisfyPathStatus(
{code}

It should put {{spsStatusInfo}} into the map?
{code}
 StoragePolicySatisfyPathStatusInfo spsStatusInfo =
  spsStatus.get(startInode.getId());
  if (spsStatusInfo == null) {
spsStatusInfo = new StoragePolicySatisfyPathStatusInfo();
  }
  spsStatusInfo.setSuccessStatus();
{code}
The above code is in muitple places, i.e., {{removeItemTrackInfo}}, looks 
suspicious. 

* Would it make sense to put {{cleanSpsStatus();}} into {{ if 
(!namesystem.isInSafeMode()) }}, and avoid run it for every processed inode.

* {{StoragePolicySatisfyPathStatusInfo#setSuccessStatus()/setInProgress() / 
canRemoveStatus()}} can be project-private.  Also it'd be nice to be consistent 
with the names, like {{setSuccess() / canRemove()}},{{setInProgress()}} is 
a good example. 
* Could you add comment to {{DFSClient# checkStoragePolicySatisfyPathStatus()}} 
as well?

Thanks!


> [SPS]: Provide an option to track the status of in progress requests
> 
>
> Key: HDFS-12310
> URL: https://issues.apache.org/jira/browse/HDFS-12310
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: datanode, namenode
>Reporter: Uma Maheswara Rao G
>Assignee: Surendra Singh Lilhore
> Attachments: HDFS-12310-HDFS-10285-01.patch, 
> HDFS-12310-HDFS-10285-02.patch, HDFS-12310-HDFS-10285-03.patch
>
>
> As per the [~andrew.wang] 's review comments in HDFS-10285, This is the JIRA 
> for tracking about the options how we track the progress of SPS requests.
>  



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-12681) Fold HdfsLocatedFileStatus into HdfsFileStatus

[Chris Douglas (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-12681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16210095#comment-16210095
 ] 

Chris Douglas commented on HDFS-12681:
--

Applications using {{instanceof}} to determine if the {{FileStatus}} instance 
contains locations may be affected. Most should be distinguished by the caller.

> Fold HdfsLocatedFileStatus into HdfsFileStatus
> --
>
> Key: HDFS-12681
> URL: https://issues.apache.org/jira/browse/HDFS-12681
> Project: Hadoop HDFS
>  Issue Type: Bug
>Reporter: Chris Douglas
>Priority: Minor
>
> {{HdfsLocatedFileStatus}} is a subtype of {{HdfsFileStatus}}, but not of 
> {{LocatedFileStatus}}. Conversion requires copying common fields and shedding 
> unknown data. It would be cleaner and sufficient for {{HdfsFileStatus}} to 
> extend {{LocatedFileStatus}}.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-7878) API - expose an unique file identifier

[Chris Douglas (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-7878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chris Douglas updated HDFS-7878:

Attachment: HDFS-7878.14.patch

Fix some javadoc

> API - expose an unique file identifier
> --
>
> Key: HDFS-7878
> URL: https://issues.apache.org/jira/browse/HDFS-7878
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>  Labels: BB2015-05-TBR
> Attachments: HDFS-7878.01.patch, HDFS-7878.02.patch, 
> HDFS-7878.03.patch, HDFS-7878.04.patch, HDFS-7878.05.patch, 
> HDFS-7878.06.patch, HDFS-7878.07.patch, HDFS-7878.08.patch, 
> HDFS-7878.09.patch, HDFS-7878.10.patch, HDFS-7878.11.patch, 
> HDFS-7878.12.patch, HDFS-7878.13.patch, HDFS-7878.14.patch, HDFS-7878.patch
>
>
> See HDFS-487.
> Even though that is resolved as duplicate, the ID is actually not exposed by 
> the JIRA it supposedly duplicates.
> INode ID for the file should be easy to expose; alternatively ID could be 
> derived from block IDs, to account for appends...
> This is useful e.g. for cache key by file, to make sure cache stays correct 
> when file is overwritten.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Created] (HDFS-12681) Fold HdfsLocatedFileStatus into HdfsFileStatus

[Chris Douglas (JIRA)]

Chris Douglas created HDFS-12681:


 Summary: Fold HdfsLocatedFileStatus into HdfsFileStatus
 Key: HDFS-12681
 URL: https://issues.apache.org/jira/browse/HDFS-12681
 Project: Hadoop HDFS
  Issue Type: Bug
Reporter: Chris Douglas
Priority: Minor


{{HdfsLocatedFileStatus}} is a subtype of {{HdfsFileStatus}}, but not of 
{{LocatedFileStatus}}. Conversion requires copying common fields and shedding 
unknown data. It would be cleaner and sufficient for {{HdfsFileStatus}} to 
extend {{LocatedFileStatus}}.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-7878) API - expose an unique file identifier

[Chris Douglas (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-7878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chris Douglas updated HDFS-7878:

Attachment: HDFS-7878.13.patch

> API - expose an unique file identifier
> --
>
> Key: HDFS-7878
> URL: https://issues.apache.org/jira/browse/HDFS-7878
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>  Labels: BB2015-05-TBR
> Attachments: HDFS-7878.01.patch, HDFS-7878.02.patch, 
> HDFS-7878.03.patch, HDFS-7878.04.patch, HDFS-7878.05.patch, 
> HDFS-7878.06.patch, HDFS-7878.07.patch, HDFS-7878.08.patch, 
> HDFS-7878.09.patch, HDFS-7878.10.patch, HDFS-7878.11.patch, 
> HDFS-7878.12.patch, HDFS-7878.13.patch, HDFS-7878.patch
>
>
> See HDFS-487.
> Even though that is resolved as duplicate, the ID is actually not exposed by 
> the JIRA it supposedly duplicates.
> INode ID for the file should be easy to expose; alternatively ID could be 
> derived from block IDs, to account for appends...
> This is useful e.g. for cache key by file, to make sure cache stays correct 
> when file is overwritten.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-7878) API - expose an unique file identifier

[Chris Douglas (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-7878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chris Douglas updated HDFS-7878:

Attachment: (was: HDFS-7878.13.patch)

> API - expose an unique file identifier
> --
>
> Key: HDFS-7878
> URL: https://issues.apache.org/jira/browse/HDFS-7878
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>  Labels: BB2015-05-TBR
> Attachments: HDFS-7878.01.patch, HDFS-7878.02.patch, 
> HDFS-7878.03.patch, HDFS-7878.04.patch, HDFS-7878.05.patch, 
> HDFS-7878.06.patch, HDFS-7878.07.patch, HDFS-7878.08.patch, 
> HDFS-7878.09.patch, HDFS-7878.10.patch, HDFS-7878.11.patch, 
> HDFS-7878.12.patch, HDFS-7878.patch
>
>
> See HDFS-487.
> Even though that is resolved as duplicate, the ID is actually not exposed by 
> the JIRA it supposedly duplicates.
> INode ID for the file should be easy to expose; alternatively ID could be 
> derived from block IDs, to account for appends...
> This is useful e.g. for cache key by file, to make sure cache stays correct 
> when file is overwritten.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-7878) API - expose an unique file identifier

[Chris Douglas (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-7878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chris Douglas updated HDFS-7878:

Attachment: HDFS-7878.13.patch

Trying another iteration. I think this addresses the outstanding feedback and 
reflects the consensus.

This does not extend {{FileStatus}} and should be compatible with 3.0.0-beta1. 
Rather than adding {{open(FileStatus)}} this adds {{open(PathHandle)}} with an 
{{getPathHandle(FileStatus, Option...)}} pattern for handle parameters. The 
standard set of options is a mix of two flags, allowing the caller to specify 
whether the {{PathHandle}} remains valid after the file is modified or if the 
file is moved. The implementation defines a few utility methods to make the 
semantics clearer and to support {{Stream}} APIs easing bulk conversion. 
Implementations can extend the set of options with bespoke semantics.

This patch updates the specification and contract tests. The current 
implementation in HDFS does not detect content changes, so it only passes the 
{{reference()}} check that resolves the {{PathHandle}} even if the file is 
moved and/or changed (i.e., open by inodeid).

[~mackrorysd], [~andrew.wang], [~ste...@apache.org], [~anu] Could you take a 
look?

To support creating a {{PathHandle}} from a {{LocatedFileStatus}}, I'll open a 
followup JIRA to implement one of [~ste...@apache.org]'s ideas, to make 
{{HdfsFileStatus}} extend {{LocatedFileStatus}} and fold the functionality of 
{{HdfsLocatedFileStatus}} into its supertype.

> API - expose an unique file identifier
> --
>
> Key: HDFS-7878
> URL: https://issues.apache.org/jira/browse/HDFS-7878
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>  Labels: BB2015-05-TBR
> Attachments: HDFS-7878.01.patch, HDFS-7878.02.patch, 
> HDFS-7878.03.patch, HDFS-7878.04.patch, HDFS-7878.05.patch, 
> HDFS-7878.06.patch, HDFS-7878.07.patch, HDFS-7878.08.patch, 
> HDFS-7878.09.patch, HDFS-7878.10.patch, HDFS-7878.11.patch, 
> HDFS-7878.12.patch, HDFS-7878.13.patch, HDFS-7878.patch
>
>
> See HDFS-487.
> Even though that is resolved as duplicate, the ID is actually not exposed by 
> the JIRA it supposedly duplicates.
> INode ID for the file should be easy to expose; alternatively ID could be 
> derived from block IDs, to account for appends...
> This is useful e.g. for cache key by file, to make sure cache stays correct 
> when file is overwritten.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-12605) [READ] TestNameNodeProvidedImplementation#testProvidedDatanodeFailures fails after rebase

[Virajith Jalaparti (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-12605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Virajith Jalaparti updated HDFS-12605:
--
Resolution: Fixed
Status: Resolved  (was: Patch Available)

> [READ] TestNameNodeProvidedImplementation#testProvidedDatanodeFailures fails 
> after rebase
> -
>
> Key: HDFS-12605
> URL: https://issues.apache.org/jira/browse/HDFS-12605
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>Reporter: Virajith Jalaparti
>Assignee: Virajith Jalaparti
> Attachments: HDFS-12605-HDFS-9806.001.patch
>
>
> {{TestNameNodeProvidedImplementation#testProvidedDatanodeFailures}} fails 
> after rebase with the following error:
> {code}
> java.lang.NullPointerException: null
>   at 
> org.apache.hadoop.hdfs.net.DFSTopologyNodeImpl.decStorageTypeCount(DFSTopologyNodeImpl.java:127)
>   at 
> org.apache.hadoop.hdfs.net.DFSTopologyNodeImpl.remove(DFSTopologyNodeImpl.java:318)
>   at 
> org.apache.hadoop.hdfs.net.DFSTopologyNodeImpl.remove(DFSTopologyNodeImpl.java:336)
>   at 
> org.apache.hadoop.net.NetworkTopology.remove(NetworkTopology.java:222)
>   at 
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager.removeDatanode(DatanodeManager.java:712)
>   at 
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager.removeDeadDatanode(DatanodeManager.java:755)
>   at 
> org.apache.hadoop.hdfs.server.blockmanagement.HeartbeatManager.heartbeatCheck(HeartbeatManager.java:407)
>   at 
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManagerTestUtil.noticeDeadDatanode(BlockManagerTestUtil.java:213)
>   at 
> org.apache.hadoop.hdfs.server.namenode.TestNameNodeProvidedImplementation.testProvidedDatanodeFailures(TestNameNodeProvidedImplementation.java:471)
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-12605) [READ] TestNameNodeProvidedImplementation#testProvidedDatanodeFailures fails after rebase

[Virajith Jalaparti (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-12605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16210013#comment-16210013
 ] 

Virajith Jalaparti commented on HDFS-12605:
---

Thanks for testing this out [~ehiggs]. Committed the patch to the HDFS-9806 
feature branch.

> [READ] TestNameNodeProvidedImplementation#testProvidedDatanodeFailures fails 
> after rebase
> -
>
> Key: HDFS-12605
> URL: https://issues.apache.org/jira/browse/HDFS-12605
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>Reporter: Virajith Jalaparti
>Assignee: Virajith Jalaparti
> Attachments: HDFS-12605-HDFS-9806.001.patch
>
>
> {{TestNameNodeProvidedImplementation#testProvidedDatanodeFailures}} fails 
> after rebase with the following error:
> {code}
> java.lang.NullPointerException: null
>   at 
> org.apache.hadoop.hdfs.net.DFSTopologyNodeImpl.decStorageTypeCount(DFSTopologyNodeImpl.java:127)
>   at 
> org.apache.hadoop.hdfs.net.DFSTopologyNodeImpl.remove(DFSTopologyNodeImpl.java:318)
>   at 
> org.apache.hadoop.hdfs.net.DFSTopologyNodeImpl.remove(DFSTopologyNodeImpl.java:336)
>   at 
> org.apache.hadoop.net.NetworkTopology.remove(NetworkTopology.java:222)
>   at 
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager.removeDatanode(DatanodeManager.java:712)
>   at 
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager.removeDeadDatanode(DatanodeManager.java:755)
>   at 
> org.apache.hadoop.hdfs.server.blockmanagement.HeartbeatManager.heartbeatCheck(HeartbeatManager.java:407)
>   at 
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManagerTestUtil.noticeDeadDatanode(BlockManagerTestUtil.java:213)
>   at 
> org.apache.hadoop.hdfs.server.namenode.TestNameNodeProvidedImplementation.testProvidedDatanodeFailures(TestNameNodeProvidedImplementation.java:471)
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-11902) [READ] Merge BlockFormatProvider and FileRegionProvider.

[Virajith Jalaparti (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-11902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Virajith Jalaparti updated HDFS-11902:
--
Status: Patch Available  (was: Open)

> [READ] Merge BlockFormatProvider and FileRegionProvider.
> 
>
> Key: HDFS-11902
> URL: https://issues.apache.org/jira/browse/HDFS-11902
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>Reporter: Virajith Jalaparti
>Assignee: Virajith Jalaparti
> Attachments: HDFS-11902-HDFS-9806.001.patch, 
> HDFS-11902-HDFS-9806.002.patch, HDFS-11902-HDFS-9806.003.patch, 
> HDFS-11902-HDFS-9806.004.patch, HDFS-11902-HDFS-9806.005.patch, 
> HDFS-11902-HDFS-9806.006.patch, HDFS-11902-HDFS-9806.007.patch, 
> HDFS-11902-HDFS-9806.008.patch, HDFS-11902-HDFS-9806.009.patch
>
>
> Currently {{BlockFormatProvider}} and {{TextFileRegionProvider}} perform 
> almost the same function on the Namenode and Datanode respectively. This JIRA 
> is to merge them into one.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-11902) [READ] Merge BlockFormatProvider and FileRegionProvider.

[Virajith Jalaparti (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-11902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Virajith Jalaparti updated HDFS-11902:
--
Status: Open  (was: Patch Available)

> [READ] Merge BlockFormatProvider and FileRegionProvider.
> 
>
> Key: HDFS-11902
> URL: https://issues.apache.org/jira/browse/HDFS-11902
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>Reporter: Virajith Jalaparti
>Assignee: Virajith Jalaparti
> Attachments: HDFS-11902-HDFS-9806.001.patch, 
> HDFS-11902-HDFS-9806.002.patch, HDFS-11902-HDFS-9806.003.patch, 
> HDFS-11902-HDFS-9806.004.patch, HDFS-11902-HDFS-9806.005.patch, 
> HDFS-11902-HDFS-9806.006.patch, HDFS-11902-HDFS-9806.007.patch, 
> HDFS-11902-HDFS-9806.008.patch, HDFS-11902-HDFS-9806.009.patch
>
>
> Currently {{BlockFormatProvider}} and {{TextFileRegionProvider}} perform 
> almost the same function on the Namenode and Datanode respectively. This JIRA 
> is to merge them into one.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-11902) [READ] Merge BlockFormatProvider and FileRegionProvider.

[Virajith Jalaparti (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-11902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16209962#comment-16209962
 ] 

Virajith Jalaparti commented on HDFS-11902:
---

Thanks for taking a look at it [~ehiggs]! I am waiting for jenkins to come back 
on v009 of the patch. I will commit to the feature branch once that happens.

> [READ] Merge BlockFormatProvider and FileRegionProvider.
> 
>
> Key: HDFS-11902
> URL: https://issues.apache.org/jira/browse/HDFS-11902
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>Reporter: Virajith Jalaparti
>Assignee: Virajith Jalaparti
> Attachments: HDFS-11902-HDFS-9806.001.patch, 
> HDFS-11902-HDFS-9806.002.patch, HDFS-11902-HDFS-9806.003.patch, 
> HDFS-11902-HDFS-9806.004.patch, HDFS-11902-HDFS-9806.005.patch, 
> HDFS-11902-HDFS-9806.006.patch, HDFS-11902-HDFS-9806.007.patch, 
> HDFS-11902-HDFS-9806.008.patch, HDFS-11902-HDFS-9806.009.patch
>
>
> Currently {{BlockFormatProvider}} and {{TextFileRegionProvider}} perform 
> almost the same function on the Namenode and Datanode respectively. This JIRA 
> is to merge them into one.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-11902) [READ] Merge BlockFormatProvider and FileRegionProvider.

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-11902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16209957#comment-16209957
 ] 

Hadoop QA commented on HDFS-11902:
--

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m  
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} docker {color} | {color:red}  2m 
58s{color} | {color:red} Docker failed to build yetus/hadoop:71bbb86. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HDFS-11902 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12892878/HDFS-11902-HDFS-9806.009.patch
 |
| Console output | 
https://builds.apache.org/job/PreCommit-HDFS-Build/21738/console |
| Powered by | Apache Yetus 0.6.0-SNAPSHOT   http://yetus.apache.org |


This message was automatically generated.



> [READ] Merge BlockFormatProvider and FileRegionProvider.
> 
>
> Key: HDFS-11902
> URL: https://issues.apache.org/jira/browse/HDFS-11902
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>Reporter: Virajith Jalaparti
>Assignee: Virajith Jalaparti
> Attachments: HDFS-11902-HDFS-9806.001.patch, 
> HDFS-11902-HDFS-9806.002.patch, HDFS-11902-HDFS-9806.003.patch, 
> HDFS-11902-HDFS-9806.004.patch, HDFS-11902-HDFS-9806.005.patch, 
> HDFS-11902-HDFS-9806.006.patch, HDFS-11902-HDFS-9806.007.patch, 
> HDFS-11902-HDFS-9806.008.patch, HDFS-11902-HDFS-9806.009.patch
>
>
> Currently {{BlockFormatProvider}} and {{TextFileRegionProvider}} perform 
> almost the same function on the Namenode and Datanode respectively. This JIRA 
> is to merge them into one.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-12607) [READ] Even one dead datanode with PROVIDED storage results in ProvidedStorageInfo being marked as FAILED

[Virajith Jalaparti (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-12607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Virajith Jalaparti updated HDFS-12607:
--
Status: Patch Available  (was: Open)

> [READ] Even one dead datanode with PROVIDED storage results in 
> ProvidedStorageInfo being marked as FAILED
> -
>
> Key: HDFS-12607
> URL: https://issues.apache.org/jira/browse/HDFS-12607
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>Reporter: Virajith Jalaparti
>Assignee: Virajith Jalaparti
> Attachments: HDFS-12607-HDFS-9806.001.patch, HDFS-12607.repro.patch
>
>
> When a DN configured with PROVIDED storage is marked as dead by the NN, the 
> state of {{providedStorageInfo}} in {{ProvidedStorageMap}} is set to FAILED, 
> and never becomes NORMAL. The state should change to FAILED only if all 
> datanodes with PROVIDED storage are dead, and should be restored back to 
> NORMAL when a Datanode with NORMAL DatanodeStorage reports in.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-12607) [READ] Even one dead datanode with PROVIDED storage results in ProvidedStorageInfo being marked as FAILED

[Virajith Jalaparti (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-12607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Virajith Jalaparti updated HDFS-12607:
--
Description: When a DN configured with PROVIDED storage is marked as dead 
by the NN, the state of {{providedStorageInfo}} in {{ProvidedStorageMap}} is 
set to FAILED, and never becomes NORMAL. The state should change to FAILED only 
if all datanodes with PROVIDED storage are dead, and should be restored back to 
NORMAL when a Datanode with NORMAL DatanodeStorage reports in.  (was: When a DN 
configured with PROVIDED storage is marked as dead by the NN, the state of 
{{providedStorageInfo}} in {{ProvidedStorageMap}} is set to FAILED, and never 
becomes NORMAL. The state should change to FAILED only if all datanodes with 
PROVIDED storage are dead, and should be restored back to NORMAL when a 
Datanode to NORMAL DatanodeStorage reports in.)

> [READ] Even one dead datanode with PROVIDED storage results in 
> ProvidedStorageInfo being marked as FAILED
> -
>
> Key: HDFS-12607
> URL: https://issues.apache.org/jira/browse/HDFS-12607
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>Reporter: Virajith Jalaparti
>Assignee: Virajith Jalaparti
> Attachments: HDFS-12607-HDFS-9806.001.patch, HDFS-12607.repro.patch
>
>
> When a DN configured with PROVIDED storage is marked as dead by the NN, the 
> state of {{providedStorageInfo}} in {{ProvidedStorageMap}} is set to FAILED, 
> and never becomes NORMAL. The state should change to FAILED only if all 
> datanodes with PROVIDED storage are dead, and should be restored back to 
> NORMAL when a Datanode with NORMAL DatanodeStorage reports in.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-12607) [READ] Even one dead datanode with PROVIDED storage results in ProvidedStorageInfo being marked as FAILED

[Virajith Jalaparti (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-12607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Virajith Jalaparti updated HDFS-12607:
--
Status: Open  (was: Patch Available)

> [READ] Even one dead datanode with PROVIDED storage results in 
> ProvidedStorageInfo being marked as FAILED
> -
>
> Key: HDFS-12607
> URL: https://issues.apache.org/jira/browse/HDFS-12607
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>Reporter: Virajith Jalaparti
>Assignee: Virajith Jalaparti
> Attachments: HDFS-12607-HDFS-9806.001.patch, HDFS-12607.repro.patch
>
>
> When a DN configured with PROVIDED storage is marked as dead by the NN, the 
> state of {{providedStorageInfo}} in {{ProvidedStorageMap}} is set to FAILED, 
> and never becomes NORMAL. The state should change to FAILED only if all 
> datanodes with PROVIDED storage are dead, and should be restored back to 
> NORMAL when a Datanode to NORMAL DatanodeStorage reports in.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-11902) [READ] Merge BlockFormatProvider and FileRegionProvider.

[Virajith Jalaparti (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-11902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Virajith Jalaparti updated HDFS-11902:
--
Attachment: (was: HDFS-11902-HDFS-9806.009.patch)

> [READ] Merge BlockFormatProvider and FileRegionProvider.
> 
>
> Key: HDFS-11902
> URL: https://issues.apache.org/jira/browse/HDFS-11902
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>Reporter: Virajith Jalaparti
>Assignee: Virajith Jalaparti
> Attachments: HDFS-11902-HDFS-9806.001.patch, 
> HDFS-11902-HDFS-9806.002.patch, HDFS-11902-HDFS-9806.003.patch, 
> HDFS-11902-HDFS-9806.004.patch, HDFS-11902-HDFS-9806.005.patch, 
> HDFS-11902-HDFS-9806.006.patch, HDFS-11902-HDFS-9806.007.patch, 
> HDFS-11902-HDFS-9806.008.patch, HDFS-11902-HDFS-9806.009.patch
>
>
> Currently {{BlockFormatProvider}} and {{TextFileRegionProvider}} perform 
> almost the same function on the Namenode and Datanode respectively. This JIRA 
> is to merge them into one.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-11902) [READ] Merge BlockFormatProvider and FileRegionProvider.

[Virajith Jalaparti (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-11902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Virajith Jalaparti updated HDFS-11902:
--
Status: Patch Available  (was: Open)

> [READ] Merge BlockFormatProvider and FileRegionProvider.
> 
>
> Key: HDFS-11902
> URL: https://issues.apache.org/jira/browse/HDFS-11902
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>Reporter: Virajith Jalaparti
>Assignee: Virajith Jalaparti
> Attachments: HDFS-11902-HDFS-9806.001.patch, 
> HDFS-11902-HDFS-9806.002.patch, HDFS-11902-HDFS-9806.003.patch, 
> HDFS-11902-HDFS-9806.004.patch, HDFS-11902-HDFS-9806.005.patch, 
> HDFS-11902-HDFS-9806.006.patch, HDFS-11902-HDFS-9806.007.patch, 
> HDFS-11902-HDFS-9806.008.patch, HDFS-11902-HDFS-9806.009.patch
>
>
> Currently {{BlockFormatProvider}} and {{TextFileRegionProvider}} perform 
> almost the same function on the Namenode and Datanode respectively. This JIRA 
> is to merge them into one.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-11902) [READ] Merge BlockFormatProvider and FileRegionProvider.

[Virajith Jalaparti (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-11902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Virajith Jalaparti updated HDFS-11902:
--
Attachment: HDFS-11902-HDFS-9806.009.patch

> [READ] Merge BlockFormatProvider and FileRegionProvider.
> 
>
> Key: HDFS-11902
> URL: https://issues.apache.org/jira/browse/HDFS-11902
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>Reporter: Virajith Jalaparti
>Assignee: Virajith Jalaparti
> Attachments: HDFS-11902-HDFS-9806.001.patch, 
> HDFS-11902-HDFS-9806.002.patch, HDFS-11902-HDFS-9806.003.patch, 
> HDFS-11902-HDFS-9806.004.patch, HDFS-11902-HDFS-9806.005.patch, 
> HDFS-11902-HDFS-9806.006.patch, HDFS-11902-HDFS-9806.007.patch, 
> HDFS-11902-HDFS-9806.008.patch, HDFS-11902-HDFS-9806.009.patch
>
>
> Currently {{BlockFormatProvider}} and {{TextFileRegionProvider}} perform 
> almost the same function on the Namenode and Datanode respectively. This JIRA 
> is to merge them into one.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-12605) [READ] TestNameNodeProvidedImplementation#testProvidedDatanodeFailures fails after rebase

[Ewan Higgs (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-12605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16209894#comment-16209894
 ] 

Ewan Higgs commented on HDFS-12605:
---

I tested this with a NN and DN on a local system and it worked. I previously 
had the exception locally when starting DNs but applying the patch fixed it 
when I tried to bounce a DN.

+1

> [READ] TestNameNodeProvidedImplementation#testProvidedDatanodeFailures fails 
> after rebase
> -
>
> Key: HDFS-12605
> URL: https://issues.apache.org/jira/browse/HDFS-12605
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>Reporter: Virajith Jalaparti
>Assignee: Virajith Jalaparti
> Attachments: HDFS-12605-HDFS-9806.001.patch
>
>
> {{TestNameNodeProvidedImplementation#testProvidedDatanodeFailures}} fails 
> after rebase with the following error:
> {code}
> java.lang.NullPointerException: null
>   at 
> org.apache.hadoop.hdfs.net.DFSTopologyNodeImpl.decStorageTypeCount(DFSTopologyNodeImpl.java:127)
>   at 
> org.apache.hadoop.hdfs.net.DFSTopologyNodeImpl.remove(DFSTopologyNodeImpl.java:318)
>   at 
> org.apache.hadoop.hdfs.net.DFSTopologyNodeImpl.remove(DFSTopologyNodeImpl.java:336)
>   at 
> org.apache.hadoop.net.NetworkTopology.remove(NetworkTopology.java:222)
>   at 
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager.removeDatanode(DatanodeManager.java:712)
>   at 
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager.removeDeadDatanode(DatanodeManager.java:755)
>   at 
> org.apache.hadoop.hdfs.server.blockmanagement.HeartbeatManager.heartbeatCheck(HeartbeatManager.java:407)
>   at 
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManagerTestUtil.noticeDeadDatanode(BlockManagerTestUtil.java:213)
>   at 
> org.apache.hadoop.hdfs.server.namenode.TestNameNodeProvidedImplementation.testProvidedDatanodeFailures(TestNameNodeProvidedImplementation.java:471)
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-8921) Add an option to Balancer for specifying the k-most over-utilized DNs or all over-utilized DNs as sources.

[Tsz Wo Nicholas Sze (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-8921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16209885#comment-16209885
 ] 

Tsz Wo Nicholas Sze commented on HDFS-8921:
---

It sounds like a good idea.  Are you interested in working on it?

> Add an option to Balancer for specifying the k-most over-utilized DNs or all 
> over-utilized DNs as sources.
> --
>
> Key: HDFS-8921
> URL: https://issues.apache.org/jira/browse/HDFS-8921
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: balancer & mover
>Reporter: Tsz Wo Nicholas Sze
>Assignee: Tsz Wo Nicholas Sze
>
> Arpit suggested to add a separate option to source from the most 
> over-utilized DataNodes first so the administrator does not have to pass the 
> source DNs manually; see [this 
> comment|https://issues.apache.org/jira/browse/HDFS-8826?focusedCommentId=14700576=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14700576].
>   The new option could allow specifying the k-most over-utilized DNs or all 
> over-utilized DNs as sources.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Assigned] (HDFS-8921) Add an option to Balancer for specifying the k-most over-utilized DNs or all over-utilized DNs as sources.

[Tsz Wo Nicholas Sze (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-8921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tsz Wo Nicholas Sze reassigned HDFS-8921:
-

Assignee: (was: Tsz Wo Nicholas Sze)

> Add an option to Balancer for specifying the k-most over-utilized DNs or all 
> over-utilized DNs as sources.
> --
>
> Key: HDFS-8921
> URL: https://issues.apache.org/jira/browse/HDFS-8921
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: balancer & mover
>Reporter: Tsz Wo Nicholas Sze
>
> Arpit suggested to add a separate option to source from the most 
> over-utilized DataNodes first so the administrator does not have to pass the 
> source DNs manually; see [this 
> comment|https://issues.apache.org/jira/browse/HDFS-8826?focusedCommentId=14700576=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14700576].
>   The new option could allow specifying the k-most over-utilized DNs or all 
> over-utilized DNs as sources.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Issue Comment Deleted] (HDFS-12605) [READ] TestNameNodeProvidedImplementation#testProvidedDatanodeFailures fails after rebase

[Ewan Higgs (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-12605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ewan Higgs updated HDFS-12605:
--
Comment: was deleted

(was: +1

I'm running into this on the branch and this should fix the issue.)

> [READ] TestNameNodeProvidedImplementation#testProvidedDatanodeFailures fails 
> after rebase
> -
>
> Key: HDFS-12605
> URL: https://issues.apache.org/jira/browse/HDFS-12605
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>Reporter: Virajith Jalaparti
>Assignee: Virajith Jalaparti
> Attachments: HDFS-12605-HDFS-9806.001.patch
>
>
> {{TestNameNodeProvidedImplementation#testProvidedDatanodeFailures}} fails 
> after rebase with the following error:
> {code}
> java.lang.NullPointerException: null
>   at 
> org.apache.hadoop.hdfs.net.DFSTopologyNodeImpl.decStorageTypeCount(DFSTopologyNodeImpl.java:127)
>   at 
> org.apache.hadoop.hdfs.net.DFSTopologyNodeImpl.remove(DFSTopologyNodeImpl.java:318)
>   at 
> org.apache.hadoop.hdfs.net.DFSTopologyNodeImpl.remove(DFSTopologyNodeImpl.java:336)
>   at 
> org.apache.hadoop.net.NetworkTopology.remove(NetworkTopology.java:222)
>   at 
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager.removeDatanode(DatanodeManager.java:712)
>   at 
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager.removeDeadDatanode(DatanodeManager.java:755)
>   at 
> org.apache.hadoop.hdfs.server.blockmanagement.HeartbeatManager.heartbeatCheck(HeartbeatManager.java:407)
>   at 
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManagerTestUtil.noticeDeadDatanode(BlockManagerTestUtil.java:213)
>   at 
> org.apache.hadoop.hdfs.server.namenode.TestNameNodeProvidedImplementation.testProvidedDatanodeFailures(TestNameNodeProvidedImplementation.java:471)
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-11902) [READ] Merge BlockFormatProvider and FileRegionProvider.

[Virajith Jalaparti (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-11902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Virajith Jalaparti updated HDFS-11902:
--
Status: Open  (was: Patch Available)

> [READ] Merge BlockFormatProvider and FileRegionProvider.
> 
>
> Key: HDFS-11902
> URL: https://issues.apache.org/jira/browse/HDFS-11902
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>Reporter: Virajith Jalaparti
>Assignee: Virajith Jalaparti
> Attachments: HDFS-11902-HDFS-9806.001.patch, 
> HDFS-11902-HDFS-9806.002.patch, HDFS-11902-HDFS-9806.003.patch, 
> HDFS-11902-HDFS-9806.004.patch, HDFS-11902-HDFS-9806.005.patch, 
> HDFS-11902-HDFS-9806.006.patch, HDFS-11902-HDFS-9806.007.patch, 
> HDFS-11902-HDFS-9806.008.patch, HDFS-11902-HDFS-9806.009.patch
>
>
> Currently {{BlockFormatProvider}} and {{TextFileRegionProvider}} perform 
> almost the same function on the Namenode and Datanode respectively. This JIRA 
> is to merge them into one.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-11639) [WRITE] Encode the BlockAlias in the client protocol

[Virajith Jalaparti (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-11639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Virajith Jalaparti updated HDFS-11639:
--
Summary: [WRITE] Encode the BlockAlias in the client protocol  (was: [READ] 
Encode the BlockAlias in the client protocol)

> [WRITE] Encode the BlockAlias in the client protocol
> 
>
> Key: HDFS-11639
> URL: https://issues.apache.org/jira/browse/HDFS-11639
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: hdfs
>Reporter: Ewan Higgs
>Assignee: Ewan Higgs
> Attachments: HDFS-11639-HDFS-9806.001.patch, 
> HDFS-11639-HDFS-9806.002.patch, HDFS-11639-HDFS-9806.003.patch, 
> HDFS-11639-HDFS-9806.004.patch, HDFS-11639-HDFS-9806.005.patch
>
>
> As part of the {{PROVIDED}} storage type, we have a {{BlockAlias}} type which 
> encodes information about where the data comes from. i.e. URI, offset, 
> length, and nonce value. This data should be encoded in the protocol 
> ({{LocatedBlockProto}} and the {{BlockTokenIdentifier}}) when a block is 
> available using the PROVIDED storage type.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-11828) [WRITE] Refactor FsDatasetImpl to use the BlockAlias from ClientProtocol for PROVIDED blocks.

[Virajith Jalaparti (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-11828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Virajith Jalaparti updated HDFS-11828:
--
Summary: [WRITE] Refactor FsDatasetImpl to use the BlockAlias from 
ClientProtocol for PROVIDED blocks.  (was: [READ] Refactor FsDatasetImpl to use 
the BlockAlias from ClientProtocol for PROVIDED blocks.)

> [WRITE] Refactor FsDatasetImpl to use the BlockAlias from ClientProtocol for 
> PROVIDED blocks.
> -
>
> Key: HDFS-11828
> URL: https://issues.apache.org/jira/browse/HDFS-11828
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>Reporter: Ewan Higgs
>Assignee: Ewan Higgs
>
> From HDFS-11639:
> {quote}[~virajith]
> Looking over this patch, one thing that occurred to me is if it makes sense 
> to unify FileRegionProvider with BlockProvider? They both have very close 
> functionality.
> I like the use of BlockProvider#resolve(). If we unify FileRegionProvider 
> with BlockProvider, then resolve can return null if the block map is 
> accessible from the Datanodes also. If it is accessible only from the 
> Namenode, then a non-null value can be propagated to the Datanode.
> One of the motivations for adding the BlockAlias to the client protocol was 
> to have the blocks map only on the Namenode. In this scenario, the ReplicaMap 
> in FsDatasetImpl of will not have any replicas apriori. Thus, one way to 
> ensure that the FsDatasetImpl interface continues to function as today is to 
> create a FinalizedProvidedReplica in FsDatasetImpl#getBlockInputStream when 
> BlockAlias is not null.
> {quote}
> {quote}[~ehiggs]
> With the pending refactoring of the FsDatasetImpl which won't have replicas a 
> priori, I wonder if it makes sense for the Datanode to have a 
> FileRegionProvider or BlockProvider at all. They are given the appropriate 
> block ID and block alias in the readBlock or writeBlock message. Maybe I'm 
> overlooking what's still being provided.{quote}
> {quote}[~virajith]
> I was trying to reconcile the existing design (FsDatasetImpl knows about 
> provided blocks apriori) with the new design where FsDatasetImpl will not 
> know about these before but just constructs them on-the-fly using the 
> BlockAlias from readBlock or writeBlock. Using BlockProvider#resolve() allows 
> us to have both designs exist in parallel. I was wondering if we should still 
> retain the earlier given the latter design.
> {quote}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-12605) [READ] TestNameNodeProvidedImplementation#testProvidedDatanodeFailures fails after rebase

[Ewan Higgs (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-12605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16209864#comment-16209864
 ] 

Ewan Higgs commented on HDFS-12605:
---

+1

I'm running into this on the branch and this should fix the issue.

> [READ] TestNameNodeProvidedImplementation#testProvidedDatanodeFailures fails 
> after rebase
> -
>
> Key: HDFS-12605
> URL: https://issues.apache.org/jira/browse/HDFS-12605
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>Reporter: Virajith Jalaparti
>Assignee: Virajith Jalaparti
> Attachments: HDFS-12605-HDFS-9806.001.patch
>
>
> {{TestNameNodeProvidedImplementation#testProvidedDatanodeFailures}} fails 
> after rebase with the following error:
> {code}
> java.lang.NullPointerException: null
>   at 
> org.apache.hadoop.hdfs.net.DFSTopologyNodeImpl.decStorageTypeCount(DFSTopologyNodeImpl.java:127)
>   at 
> org.apache.hadoop.hdfs.net.DFSTopologyNodeImpl.remove(DFSTopologyNodeImpl.java:318)
>   at 
> org.apache.hadoop.hdfs.net.DFSTopologyNodeImpl.remove(DFSTopologyNodeImpl.java:336)
>   at 
> org.apache.hadoop.net.NetworkTopology.remove(NetworkTopology.java:222)
>   at 
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager.removeDatanode(DatanodeManager.java:712)
>   at 
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager.removeDeadDatanode(DatanodeManager.java:755)
>   at 
> org.apache.hadoop.hdfs.server.blockmanagement.HeartbeatManager.heartbeatCheck(HeartbeatManager.java:407)
>   at 
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManagerTestUtil.noticeDeadDatanode(BlockManagerTestUtil.java:213)
>   at 
> org.apache.hadoop.hdfs.server.namenode.TestNameNodeProvidedImplementation.testProvidedDatanodeFailures(TestNameNodeProvidedImplementation.java:471)
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-12680) Ozone: SCM: Lease support for container creation

[Nanda kumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-12680?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nanda kumar updated HDFS-12680:
---
Status: Patch Available  (was: Open)

> Ozone: SCM: Lease support for container creation
> 
>
> Key: HDFS-12680
> URL: https://issues.apache.org/jira/browse/HDFS-12680
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: ozone
>Reporter: Nanda kumar
>Assignee: Nanda kumar
> Attachments: HDFS-12680-HDFS-7240.000.patch
>
>
> This brings in lease support for container creation.
> Lease should be give for a container that is moved to {{CREATING}} state when 
> {{BEGIN_CREATE}} event happens, {{LeaseException}} should be thrown if the 
> container already holds a lease. Lease must be released during 
> {{COMPLETE_CREATE}} event. If the lease times out container should be moved 
> to {{DELETING}} state, and exception should be thrown if {{COMPLETE_CREATE}} 
> event is received on that container.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-12680) Ozone: SCM: Lease support for container creation

[Nanda kumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-12680?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nanda kumar updated HDFS-12680:
---
Attachment: HDFS-12680-HDFS-7240.000.patch

> Ozone: SCM: Lease support for container creation
> 
>
> Key: HDFS-12680
> URL: https://issues.apache.org/jira/browse/HDFS-12680
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: ozone
>Reporter: Nanda kumar
>Assignee: Nanda kumar
> Attachments: HDFS-12680-HDFS-7240.000.patch
>
>
> This brings in lease support for container creation.
> Lease should be give for a container that is moved to {{CREATING}} state when 
> {{BEGIN_CREATE}} event happens, {{LeaseException}} should be thrown if the 
> container already holds a lease. Lease must be released during 
> {{COMPLETE_CREATE}} event. If the lease times out container should be moved 
> to {{DELETING}} state, and exception should be thrown if {{COMPLETE_CREATE}} 
> event is received on that container.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-12680) Ozone: SCM: Lease support for container creation

[Nanda kumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-12680?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nanda kumar updated HDFS-12680:
---
Description: 
This brings in lease support for container creation.
Lease should be give for a container that is moved to {{CREATING}} state when 
{{BEGIN_CREATE}} event happens, {{LeaseException}} should be thrown if the 
container already holds a lease. Lease must be released during 
{{COMPLETE_CREATE}} event. If the lease times out container should be moved to 
{{DELETING}} state, and exception should be thrown if {{COMPLETE_CREATE}} event 
is received on that container.

  was:
This brings in lease support for container creation.
Lease should be give for a container that is moved to {{CREATING}} state when 
{{BEGIN_CREATE}} event happens, {{LeaseException}} should be thrown if the 
container already holds a lease. Lease must be released during 
{{COMPLETE_CREATE}} event. If the lease times out container should be moved to 
{{DELETING}} state, and exception is thrown if {{COMPLETE_CREATE}} event is 
received on that container.


> Ozone: SCM: Lease support for container creation
> 
>
> Key: HDFS-12680
> URL: https://issues.apache.org/jira/browse/HDFS-12680
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: ozone
>Reporter: Nanda kumar
>Assignee: Nanda kumar
>
> This brings in lease support for container creation.
> Lease should be give for a container that is moved to {{CREATING}} state when 
> {{BEGIN_CREATE}} event happens, {{LeaseException}} should be thrown if the 
> container already holds a lease. Lease must be released during 
> {{COMPLETE_CREATE}} event. If the lease times out container should be moved 
> to {{DELETING}} state, and exception should be thrown if {{COMPLETE_CREATE}} 
> event is received on that container.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-12680) Ozone: SCM: Lease support for container creation

[Nanda kumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-12680?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nanda kumar updated HDFS-12680:
---
Description: 
This brings in lease support for container creation.
Lease should be give for a container that is moved to {{CREATING}} state when 
{{BEGIN_CREATE}} event happens, {{LeaseException}} should be thrown if the 
container already holds a lease. Lease must be released during 
{{COMPLETE_CREATE}} event. If the lease times out container should be moved to 
{{DELETING}} state, and exception is thrown if {{COMPLETE_CREATE}} event is 
received on that container.

  was:
This brings in lease support for container creation.
Lease should be give for a container that is moved to {{CREATING}} state while 
{{BEGIN_CREATE}} event happens, {{LeaseException}} should be thrown if the 
container already holds a lease. Lease must be released during 
{{COMPLETE_CREATE}} event. If the lease times out container should be moved to 
{{DELETING}} state, and exception is thrown if {{COMPLETE_CREATE}} event is 
received on that container.


> Ozone: SCM: Lease support for container creation
> 
>
> Key: HDFS-12680
> URL: https://issues.apache.org/jira/browse/HDFS-12680
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: ozone
>Reporter: Nanda kumar
>Assignee: Nanda kumar
>
> This brings in lease support for container creation.
> Lease should be give for a container that is moved to {{CREATING}} state when 
> {{BEGIN_CREATE}} event happens, {{LeaseException}} should be thrown if the 
> container already holds a lease. Lease must be released during 
> {{COMPLETE_CREATE}} event. If the lease times out container should be moved 
> to {{DELETING}} state, and exception is thrown if {{COMPLETE_CREATE}} event 
> is received on that container.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Created] (HDFS-12680) Ozone: SCM: Lease support for container creation

[Nanda kumar (JIRA)]

Nanda kumar created HDFS-12680:
--

 Summary: Ozone: SCM: Lease support for container creation
 Key: HDFS-12680
 URL: https://issues.apache.org/jira/browse/HDFS-12680
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: ozone
Reporter: Nanda kumar
Assignee: Nanda kumar


This brings in lease support for container creation.
Lease should be give for a container that is moved to {{CREATING}} state while 
{{BEGIN_CREATE}} event happens, {{LeaseException}} should be thrown if the 
container already holds a lease. Lease must be released during 
{{COMPLETE_CREATE}} event. If the lease times out container should be moved to 
{{DELETING}} state, and exception is thrown if {{COMPLETE_CREATE}} event is 
received on that container.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-12090) Handling writes from HDFS to Provided storages

[Ewan Higgs (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-12090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ewan Higgs updated HDFS-12090:
--
Attachment: HDFS-12090-Functional-Specification.002.pdf

Attaching updated version of Functional Specification with some cleanups by 
[~virajith].

> Handling writes from HDFS to Provided storages
> --
>
> Key: HDFS-12090
> URL: https://issues.apache.org/jira/browse/HDFS-12090
> Project: Hadoop HDFS
>  Issue Type: New Feature
>Reporter: Virajith Jalaparti
> Attachments: HDFS-12090-Functional-Specification.001.pdf, 
> HDFS-12090-Functional-Specification.002.pdf, HDFS-12090-design.001.pdf
>
>
> HDFS-9806 introduces the concept of {{PROVIDED}} storage, which makes data in 
> external storage systems accessible through HDFS. However, HDFS-9806 is 
> limited to data being read through HDFS. This JIRA will deal with how data 
> can be written to such {{PROVIDED}} storages from HDFS.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Comment Edited] (HDFS-8921) Add an option to Balancer for specifying the k-most over-utilized DNs or all over-utilized DNs as sources.

[Plamen Jeliazkov (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-8921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16209798#comment-16209798
 ] 

Plamen Jeliazkov edited comment on HDFS-8921 at 10/18/17 6:40 PM:
--

Would it be acceptable to say something "k-most over-utilized DNs per 
iteration"? This would be very trivial to implement but would not be equivalent 
to the include file example as the set would change per iteration.

Example: [dn1: 75%, dn2: 60%, dn3: 25%], if max-sources was 1, then after on 
first iteration dn1 would get chosen and we might see: [dn1:55%, dn2:60%, 
dn3:45%], and now the next iteration would pick dn2.


was (Author: zero45):
Would it be acceptable to say something "k-most over-utilized DNs per 
iteration"? This would be very trivial to implement but would not be equivalent 
to the include file example as the set would change per iteration.

Example: {dn1: 75%, dn2: 60%, dn3: 25%}, if max-sources was 1, then after on 
first iteration dn1 would get chosen and we might see: {dn1:55%, dn2:60%, 
dn3:45%}, and now the next iteration would pick dn2.

> Add an option to Balancer for specifying the k-most over-utilized DNs or all 
> over-utilized DNs as sources.
> --
>
> Key: HDFS-8921
> URL: https://issues.apache.org/jira/browse/HDFS-8921
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: balancer & mover
>Reporter: Tsz Wo Nicholas Sze
>Assignee: Tsz Wo Nicholas Sze
>
> Arpit suggested to add a separate option to source from the most 
> over-utilized DataNodes first so the administrator does not have to pass the 
> source DNs manually; see [this 
> comment|https://issues.apache.org/jira/browse/HDFS-8826?focusedCommentId=14700576=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14700576].
>   The new option could allow specifying the k-most over-utilized DNs or all 
> over-utilized DNs as sources.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-8921) Add an option to Balancer for specifying the k-most over-utilized DNs or all over-utilized DNs as sources.

[Plamen Jeliazkov (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-8921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16209798#comment-16209798
 ] 

Plamen Jeliazkov commented on HDFS-8921:


Would it be acceptable to say something "k-most over-utilized DNs per 
iteration"? This would be very trivial to implement but would not be equivalent 
to the include file example as the set would change per iteration.

Example: {dn1: 75%, dn2: 60%, dn3: 25%}, if max-sources was 1, then after on 
first iteration dn1 would get chosen and we might see: {dn1:55%, dn2:60%, 
dn3:45%}, and now the next iteration would pick dn2.

> Add an option to Balancer for specifying the k-most over-utilized DNs or all 
> over-utilized DNs as sources.
> --
>
> Key: HDFS-8921
> URL: https://issues.apache.org/jira/browse/HDFS-8921
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: balancer & mover
>Reporter: Tsz Wo Nicholas Sze
>Assignee: Tsz Wo Nicholas Sze
>
> Arpit suggested to add a separate option to source from the most 
> over-utilized DataNodes first so the administrator does not have to pass the 
> source DNs manually; see [this 
> comment|https://issues.apache.org/jira/browse/HDFS-8826?focusedCommentId=14700576=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14700576].
>   The new option could allow specifying the k-most over-utilized DNs or all 
> over-utilized DNs as sources.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Resolved] (HDFS-12676) when blocks has corrupted replicas，throws Exception

[Wei-Chiu Chuang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-12676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wei-Chiu Chuang resolved HDFS-12676.

Resolution: Duplicate

Resolving it as a dup. Thanks @lynn for reporting it.

> when blocks has corrupted replicas，throws Exception
> ---
>
> Key: HDFS-12676
> URL: https://issues.apache.org/jira/browse/HDFS-12676
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: hdfs
>Reporter: lynn
>
> when blocks has corrupted replicas，throws Exception as follows:
> Exception 1:
> 2017-10-18 15:24:55,858 WARN  blockmanagement.BlockManager 
> (BlockManager.java:createLocatedBlock(938)) - Inconsistent number of corrupt 
> replicas for blk_1073750384_504374 blockMap has 0 but corrupt replicas map 
> has 1
> 2017-10-18 15:24:55,859 WARN  ipc.Server (Server.java:logException(2433)) - 
> IPC Server handler 116 on 8020, call 
> org.apache.hadoop.hdfs.protocol.ClientProtocol.getBlockLocations from 
> 10.43.160.18:56313 Call#2 Retry#-1
> java.lang.ArrayIndexOutOfBoundsException: 1
>   at 
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.createLocatedBlock(BlockManager.java:972)
>   at 
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.createLocatedBlock(BlockManager.java:911)
>   at 
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.createLocatedBlockList(BlockManager.java:884)
>   at 
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.createLocatedBlocks(BlockManager.java:1011)
>   at 
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:2010)
>   at 
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1960)
>   at 
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1873)
>   at 
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:693)
>   at 
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:373)
>   at 
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
>   at 
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:640)
>   at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982)
>   at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2351)
>   at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2347)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at javax.security.auth.Subject.doAs(Subject.java:422)
>   at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1865)
>   at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2345)
> Exception 2:
> 2017-10-12 16:59:36,591 INFO  blockmanagement.BlockManager 
> (BlockManager.java:computeReplicationWorkForBlocks(1649)) - Blocks chosen but 
> could not be replicated = 4; of which 0 have no target, 4 have no source, 0 
> are UC, 0 are abandoned, 0 already have enough replicas.
> 2017-10-12 16:59:36,809 WARN  blockmanagement.BlockManager 
> (BlockManager.java:createLocatedBlock(938)) - Inconsistent number of corrupt 
> replicas for blk_1073789106_2278702 blockMap has 0 but corrupt replicas map 
> has 2
> 2017-10-12 16:59:36,810 WARN  ipc.Server (Server.java:logException(2433)) - 
> IPC Server handler 123 on 8020, call 
> org.apache.hadoop.hdfs.protocol.ClientProtocol.getBlockLocations from 
> 10.46.230.12:47974 Call#2 Retry#-1
> java.lang.NegativeArraySizeException
>   at 
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.createLocatedBlock(BlockManager.java:946)
>   at 
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.createLocatedBlock(BlockManager.java:911)
>   at 
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.createLocatedBlockList(BlockManager.java:884)
>   at 
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.createLocatedBlocks(BlockManager.java:997)
>   at 
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:2010)
>   at 
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1960)
>   at 
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1873)
>   at 
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:693)
>   at 
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:373)
>   at 
> 

[jira] [Created] (HDFS-12679) Namenode stale memory Datanode gets Wipes out

[Vivek Ghatala (JIRA)]

Vivek Ghatala created HDFS-12679:


 Summary: Namenode stale memory Datanode gets Wipes out
 Key: HDFS-12679
 URL: https://issues.apache.org/jira/browse/HDFS-12679
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: namenode
Affects Versions: 2.8.1
Reporter: Vivek Ghatala


Namenode stale mapping information about datanodes commands wiping out datanode.

LOG LINES:

2017-10-11 23:01:45,969 DEBUG 
org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: remove datanode 
XXX.XX.XX.1:Y

2017-10-11 23:01:45,969 DEBUG 
org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: 
DatanodeManager.wipeDatanode(XXX.XX.XX.1:Y): storage ##STORAGE_ID1## is 
removed from datanodeMap.

Scenario: Our environment uses shared storage. Whenever some datanode restarts, 
some other node comes up with that storage with different Ip address. It is 
possible that multiple datanodes restarts at same time. 

Case: the node 1 serving some storage X is now serving storage Y. node 2 comes 
up and is serving storage X now. Namenode here commands storage Y to get wiped 
out.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-12620) Backporting HDFS-10467 to branch-2

[JIRA]

 [ 
https://issues.apache.org/jira/browse/HDFS-12620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Íñigo Goiri updated HDFS-12620:
---
Attachment: HDFS-12620-branch-2.007.patch

> Backporting HDFS-10467 to branch-2
> --
>
> Key: HDFS-12620
> URL: https://issues.apache.org/jira/browse/HDFS-12620
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Reporter: Íñigo Goiri
>Assignee: Íñigo Goiri
> Attachments: HDFS-10467-branch-2.001.patch, 
> HDFS-10467-branch-2.002.patch, HDFS-10467-branch-2.003.patch, 
> HDFS-10467-branch-2.patch, HDFS-12620-branch-2.000.patch, 
> HDFS-12620-branch-2.004.patch, HDFS-12620-branch-2.005.patch, 
> HDFS-12620-branch-2.006.patch, HDFS-12620-branch-2.007.patch
>
>
> When backporting HDFS-10467, there are a few things that changed:
> * {{bin\hdfs}}
> * {{ClientProtocol}}
> * Java 7 not supporting referencing functions
> * {{org.eclipse.jetty.util.ajax.JSON}} in branch-2 is 
> {{org.mortbay.util.ajax.JSON}}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-11807) libhdfs++: Get minidfscluster tests running under valgrind

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-11807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16209678#comment-16209678
 ] 

Hadoop QA commented on HDFS-11807:
--

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m  
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} docker {color} | {color:red}  7m  
6s{color} | {color:red} Docker failed to build yetus/hadoop:3117e2a. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HDFS-11807 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12892802/HDFS-11807.HDFS-8707.000.patch
 |
| Console output | 
https://builds.apache.org/job/PreCommit-HDFS-Build/21737/console |
| Powered by | Apache Yetus 0.6.0-SNAPSHOT   http://yetus.apache.org |


This message was automatically generated.



> libhdfs++: Get minidfscluster tests running under valgrind
> --
>
> Key: HDFS-11807
> URL: https://issues.apache.org/jira/browse/HDFS-11807
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: hdfs-client
>Reporter: James Clampffer
>Assignee: Anatoli Shein
> Attachments: HDFS-11807.HDFS-8707.000.patch
>
>
> The gmock based unit tests generally don't expose race conditions and memory 
> stomps.  A good way to expose these is running libhdfs++ stress tests and 
> tools under valgrind and pointing them at a real cluster.  Right now the CI 
> tools don't do that so bugs occasionally slip in and aren't caught until they 
> cause trouble in applications that use libhdfs++ for HDFS access.
> The reason the minidfscluster tests don't run under valgrind is because the 
> GC and JIT compiler in the embedded JVM do things that look like errors to 
> valgrind.  I'd like to have these tests do some basic setup and then fork 
> into two processes: one for the minidfscluster stuff and one for the 
> libhdfs++ client test.  A small amount of shared memory can be used to 
> provide a place for the minidfscluster to stick the hdfsBuilder object that 
> the client needs to get info about which port to connect to.  Can also stick 
> a condition variable there to let the minidfscluster know when it can shut 
> down.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-12663) Ozone: OzoneClient: Remove protobuf classes exposed to clients through OzoneBucket

[Nanda kumar (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-12663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16209657#comment-16209657
 ] 

Nanda kumar commented on HDFS-12663:


Thanks for the review [~anu]
bq. I am wondering why? Does the users or developers have any specific benefit 
from this change ?
There is no benefit as such, but implementation of {{ClientProtocol}} like 
{{RestClient}} doesn't need to know about protobuf classes. Also as a user why 
should they need to know/use {{OzoneProtos.ReplicationFactor}}. If we use it as 
it's now, should we explain what {{OzoneProtos}} is to the user?

bq. OzoneBucket.java:219: The default Replication Type and Replication Factor 
should be read from the configs. You can always use a sane default like this.
Good suggestion, will update the patch.

> Ozone: OzoneClient: Remove protobuf classes exposed to clients through 
> OzoneBucket
> --
>
> Key: HDFS-12663
> URL: https://issues.apache.org/jira/browse/HDFS-12663
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: ozone
>Reporter: Nanda kumar
>Assignee: Nanda kumar
>  Labels: ozoneMerge
> Attachments: HDFS-12663-HDFS-7240.000.patch, 
> HDFS-12663-HDFS-7240.001.patch
>
>
> As part of {{OzoneBucket#createKey}} we are currently exposing protobuf enums 
> {{OzoneProtos.ReplicationType}} and {{OzoneProtos.ReplicationFactor}} through 
> client, this can be replaced with client side enums and conversion can be 
> done internally.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-11807) libhdfs++: Get minidfscluster tests running under valgrind

[Anatoli Shein (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-11807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anatoli Shein updated HDFS-11807:
-
Attachment: HDFS-11807.HDFS-8707.000.patch

First stab at running the mini dfs stress test under valgrind.

> libhdfs++: Get minidfscluster tests running under valgrind
> --
>
> Key: HDFS-11807
> URL: https://issues.apache.org/jira/browse/HDFS-11807
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: hdfs-client
>Reporter: James Clampffer
>Assignee: Anatoli Shein
> Attachments: HDFS-11807.HDFS-8707.000.patch
>
>
> The gmock based unit tests generally don't expose race conditions and memory 
> stomps.  A good way to expose these is running libhdfs++ stress tests and 
> tools under valgrind and pointing them at a real cluster.  Right now the CI 
> tools don't do that so bugs occasionally slip in and aren't caught until they 
> cause trouble in applications that use libhdfs++ for HDFS access.
> The reason the minidfscluster tests don't run under valgrind is because the 
> GC and JIT compiler in the embedded JVM do things that look like errors to 
> valgrind.  I'd like to have these tests do some basic setup and then fork 
> into two processes: one for the minidfscluster stuff and one for the 
> libhdfs++ client test.  A small amount of shared memory can be used to 
> provide a place for the minidfscluster to stick the hdfsBuilder object that 
> the client needs to get info about which port to connect to.  Can also stick 
> a condition variable there to let the minidfscluster know when it can shut 
> down.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-11807) libhdfs++: Get minidfscluster tests running under valgrind

[Anatoli Shein (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-11807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anatoli Shein updated HDFS-11807:
-
Status: Patch Available  (was: In Progress)

libhdfs_mini_stress now runs under valgrind

> libhdfs++: Get minidfscluster tests running under valgrind
> --
>
> Key: HDFS-11807
> URL: https://issues.apache.org/jira/browse/HDFS-11807
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: hdfs-client
>Reporter: James Clampffer
>Assignee: Anatoli Shein
>
> The gmock based unit tests generally don't expose race conditions and memory 
> stomps.  A good way to expose these is running libhdfs++ stress tests and 
> tools under valgrind and pointing them at a real cluster.  Right now the CI 
> tools don't do that so bugs occasionally slip in and aren't caught until they 
> cause trouble in applications that use libhdfs++ for HDFS access.
> The reason the minidfscluster tests don't run under valgrind is because the 
> GC and JIT compiler in the embedded JVM do things that look like errors to 
> valgrind.  I'd like to have these tests do some basic setup and then fork 
> into two processes: one for the minidfscluster stuff and one for the 
> libhdfs++ client test.  A small amount of shared memory can be used to 
> provide a place for the minidfscluster to stick the hdfsBuilder object that 
> the client needs to get info about which port to connect to.  Can also stick 
> a condition variable there to let the minidfscluster know when it can shut 
> down.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Assigned] (HDFS-11807) libhdfs++: Get minidfscluster tests running under valgrind

[Anatoli Shein (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-11807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anatoli Shein reassigned HDFS-11807:


Assignee: Anatoli Shein  (was: James Clampffer)

> libhdfs++: Get minidfscluster tests running under valgrind
> --
>
> Key: HDFS-11807
> URL: https://issues.apache.org/jira/browse/HDFS-11807
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: hdfs-client
>Reporter: James Clampffer
>Assignee: Anatoli Shein
>
> The gmock based unit tests generally don't expose race conditions and memory 
> stomps.  A good way to expose these is running libhdfs++ stress tests and 
> tools under valgrind and pointing them at a real cluster.  Right now the CI 
> tools don't do that so bugs occasionally slip in and aren't caught until they 
> cause trouble in applications that use libhdfs++ for HDFS access.
> The reason the minidfscluster tests don't run under valgrind is because the 
> GC and JIT compiler in the embedded JVM do things that look like errors to 
> valgrind.  I'd like to have these tests do some basic setup and then fork 
> into two processes: one for the minidfscluster stuff and one for the 
> libhdfs++ client test.  A small amount of shared memory can be used to 
> provide a place for the minidfscluster to stick the hdfsBuilder object that 
> the client needs to get info about which port to connect to.  Can also stick 
> a condition variable there to let the minidfscluster know when it can shut 
> down.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Work started] (HDFS-11807) libhdfs++: Get minidfscluster tests running under valgrind

[Anatoli Shein (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-11807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on HDFS-11807 started by Anatoli Shein.

> libhdfs++: Get minidfscluster tests running under valgrind
> --
>
> Key: HDFS-11807
> URL: https://issues.apache.org/jira/browse/HDFS-11807
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: hdfs-client
>Reporter: James Clampffer
>Assignee: Anatoli Shein
>
> The gmock based unit tests generally don't expose race conditions and memory 
> stomps.  A good way to expose these is running libhdfs++ stress tests and 
> tools under valgrind and pointing them at a real cluster.  Right now the CI 
> tools don't do that so bugs occasionally slip in and aren't caught until they 
> cause trouble in applications that use libhdfs++ for HDFS access.
> The reason the minidfscluster tests don't run under valgrind is because the 
> GC and JIT compiler in the embedded JVM do things that look like errors to 
> valgrind.  I'd like to have these tests do some basic setup and then fork 
> into two processes: one for the minidfscluster stuff and one for the 
> libhdfs++ client test.  A small amount of shared memory can be used to 
> provide a place for the minidfscluster to stick the hdfsBuilder object that 
> the client needs to get info about which port to connect to.  Can also stick 
> a condition variable there to let the minidfscluster know when it can shut 
> down.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-12638) NameNode exits due to ReplicationMonitor thread received Runtime exception in ReplicationWork#chooseTargets

[Daryn Sharp (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-12638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16209604#comment-16209604
 ] 

Daryn Sharp commented on HDFS-12638:


[~cheersyang], maybe I'm misinterpreting the proposal but it sounds like you 
want to avoid the symptom (NPE) rather than address the orphaned block (root 
cause)?

I haven't really studied the truncate code.  For "copy-on-truncate" it sounds 
reasonable for both blocks to be in the map.  But both should be referenced 
somewhere.  Presumably this is just for snapshots, which means one of the 
blocks is in a diff.  If delete misses one of the blocks, that's the root cause 
we must fix.  Be warned the block reclamation code is not a fun place.

> NameNode exits due to ReplicationMonitor thread received Runtime exception in 
> ReplicationWork#chooseTargets
> ---
>
> Key: HDFS-12638
> URL: https://issues.apache.org/jira/browse/HDFS-12638
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: hdfs
>Affects Versions: 2.8.2
>Reporter: Jiandan Yang 
> Attachments: HDFS-12638-branch-2.8.2.001.patch
>
>
> Active NamNode exit due to NPE, I can confirm that the BlockCollection passed 
> in when creating ReplicationWork is null, but I do not know why 
> BlockCollection is null, By view history I found 
> [HDFS-9754|https://issues.apache.org/jira/browse/HDFS-9754] remove judging  
> whether  BlockCollection is null.
> NN logs are as following:
> {code:java}
> 2017-10-11 16:29:06,161 ERROR [ReplicationMonitor] 
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: 
> ReplicationMonitor thread received Runtime exception.
> java.lang.NullPointerException
> at 
> org.apache.hadoop.hdfs.server.blockmanagement.ReplicationWork.chooseTargets(ReplicationWork.java:55)
> at 
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.computeReplicationWorkForBlocks(BlockManager.java:1532)
> at 
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.computeReplicationWork(BlockManager.java:1491)
> at 
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.computeDatanodeWork(BlockManager.java:3792)
> at 
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$ReplicationMonitor.run(BlockManager.java:3744)
> at java.lang.Thread.run(Thread.java:834)
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-12574) Add CryptoInputStream to WebHdfsFileSystem read call.

[Daryn Sharp (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-12574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16209570#comment-16209570
 ] 

Daryn Sharp commented on HDFS-12574:


This is bad: {{CryptoProtocolVersion.values()}}.  The values method always 
allocates a new garbage array for every invocation.  I forget where else I made 
a change to have a static array assignment of the values and created a static 
{{valueOf}} to return the item from the static array.  I can't find it, looks 
like it might have been undone...  Note that protobufs actually do this.

{{WebHdfsFileSystem#open}} contains a copy-n-paste of the same code in 
{{DFSClient#createWrappedInputStream}}.  {{CryptoInputStream}} can work with 
any general stream so let's make a general wrapping method.  Maybe create an 
interface something like {{EncryptableInputStream}} for the 
{{getFileEncryptionInfo}} which {{DFSInputStream}} and {{WebHdfsInputStream}} 
implements.  Pass an encryptable stream and it returns a wrapped stream if 
necessary.

I'm not thrilled with stream construction always calling file info but I 
understand the stream is lazily opened which creates a chicken and egg problem 
for determining whether to return a crypto stream.  Double check that failing 
in the {{ReadRunner}} ctor doesn't cause any retry loop issues or partial 
stream leakage.  I'll scrutinize too.

I think using the cached file status at open in 
{{ReadRunner#initializeInputStream}} subtly changes semantics.  The the file is 
actively being appended, this change may prevent the client from reading the 
additional data.  Or if the file is truncated it might cause the client to 
incorrectly get an EOF.  Double check and I will too.  Granted, it would be 
great if opening a file doesn't cause double file status calls.

Why the change to {{MiniDFSCluster}}?

> Add CryptoInputStream to WebHdfsFileSystem read call.
> -
>
> Key: HDFS-12574
> URL: https://issues.apache.org/jira/browse/HDFS-12574
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: encryption, kms, webhdfs
>Reporter: Rushabh S Shah
>Assignee: Rushabh S Shah
> Attachments: HDFS-12574.001.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-12624) Ozone: number of keys/values/buckets to KSMMetrics

[Elek, Marton (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-12624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16209529#comment-16209529
 ] 

Elek, Marton commented on HDFS-12624:
-

I just did a quick test. I have about 5 000 000 entries in the ksm database. 
The full scan without parsing took about 40 seconds.

{code}
long count = this.store.getRangeKVs(null, Integer.MAX_VALUE,
  MetadataKeyFilters.getNormalKeyFilter()).parallelStream().count();
{code}

(Note: currently it's not possible the scan through the database without 
storing everying in the database. I propose to modify the MetadataStore 
interface with adding new methods with returns Stream.)

So it's possible (even with sync operation) to count all the keys during the 
startup.



> Ozone: number of keys/values/buckets to KSMMetrics
> --
>
> Key: HDFS-12624
> URL: https://issues.apache.org/jira/browse/HDFS-12624
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Elek, Marton
>Assignee: Elek, Marton
>
> During my last ozone test with 100 node ozone cluster I see a problem to 
> track how many keys/volumes/buckets do I have.
> I opened this jira to start a discussion about extending KSM metrics (but let 
> me know if this is already planned somewhere else) and add number of 
> keys/volumes/buckets to the metrics interface.
> These counters could be added to anywhere else (for example as a client call) 
> but I think it is an important number and would be worth to monitor it.
> I see multiple ways to achieve it:
> 1. Extend the `org.apache.hadoop.utils.MetadaStore` class with an additional 
> count() method. As I know there is no easy way to implement it with leveldb 
> but with rocksdb there is a posibility to get the _estimated_ number of keys.
> On the other hand KSM stores volumes/buckets/keys in the same db, so we can't 
> use it without splitting the ksm.db to separated dbs.
> 2. Create a background task to iterate over all the keys and count ozone 
> key/volume/bucket numbers:
> pro: it would be independent from the existing program flow
> con: doesn't provided up-to-date information.
> con: it uses more resources to scan the whole db frequently
> 3. During the startup we can iterate over the whole ksm.db and count the 
> current metrics, and later we can update the numbers in case of new 
> create/delete calls. It uses additional resources during the startup (should 
> be checked how much time is to parse a db with millions of keys) but after 
> that it would be fast. Also we can introduce new confguration variables to 
> skip the initial scan. In that case the numbers will be valid only from the 
> last restart but the startup would be fast.
> I suggest to use the 3rd approach, could you please comment about your 
> opinion? 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-12396) Webhdfs file system should get delegation token from kms provider.

[Daryn Sharp (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-12396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16209463#comment-16209463
 ] 

Daryn Sharp commented on HDFS-12396:


In {{KMSUtil}}, I'm not fond of returning null when passed null.  In general, 
it bothers me to see methods short out and return null for what would/should be 
invalid input.  It tends to create landmines for other changes and/or mask 
bugs.  It's the caller's responsibility to decide to invoke a method, and the 
method should do exactly what it's designed to do.

The moved/new methods seem like they should be in {{KMSUtil}}, rather than 
{{DFSUtilClient}}, with private/unstable annotations in case we need to make 
further modifications.

Minor, {{KeyProviderHelper}} is rather generic and doesn't convey what it does. 
 I'd consider something more like {{KeyProviderTokenAdapter}}, 
{{KeyProviderTokenIssuer}}, etc.

Does {{WebHdfsFileSystem#keyProvider}} really need to exist and only be set for 
tests?  Could a spy be used instead?

> Webhdfs file system should get delegation token from kms provider.
> --
>
> Key: HDFS-12396
> URL: https://issues.apache.org/jira/browse/HDFS-12396
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: encryption, kms, webhdfs
>Reporter: Rushabh S Shah
>Assignee: Rushabh S Shah
> Attachments: HDFS-12396.001.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-12674) hadoop fs -get does not throw permission denied in some cases

[Daryn Sharp (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-12674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16209450#comment-16209450
 ] 

Daryn Sharp commented on HDFS-12674:


We can't yet discount the jvm.  The problem we debugged is when cwd is 
unreadable.  Java does the right thing when an absolute path is given because 
it's just doing system calls.  Relative paths are the problem.  Per the strace 
we did, the jvm appears to be doing a fchdir to the hsperf directory during 
startup.  It now can't chdir back to the unreadable cwd at startup so relative 
paths are resolved from the hsperf dir.

> hadoop fs -get does not throw permission denied in some cases
> -
>
> Key: HDFS-12674
> URL: https://issues.apache.org/jira/browse/HDFS-12674
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: hdfs
>Affects Versions: 2.8.1
>Reporter: Kuhu Shukla
>
> whoami : mysuperuser
> pwd : /tmp/testKuhu with no write permissions,
> {code}
> $ ls -ld .
> dr-xr-xr-x 2 mysuperuser users 4096 Oct 16 20:36 .
> $ hadoop fs -get testFile
> get: /tmp/testKuhu/testFile.COPYING (Permission denied)
> {code}
> But, when 
> whoami : mysuperuser
> pwd : /tmp/testKuhu this time with no read permissions,
> {code}
> $ ls -ld .
> d-xx-x 2 mysuperuser users 4096 Oct 16 20:36 .
> $ hadoop fs -get testFile
> $
> {code}
> passes and copies it to /tmp/hsperfdata_mysuperuser/
> The JIRA tracks this inconsistent behavior.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Comment Edited] (HDFS-11468) Ozone: SCM: Add Node Metrics for SCM

[Weiwei Yang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-11468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16209291#comment-16209291
 ] 

Weiwei Yang edited comment on HDFS-11468 at 10/18/17 1:06 PM:
--

Hi [~linyiqun]

bq. Since metric is the data of aggregated statistics. I'd like to update this 
as ConainerSize. This will look simplify and also good understanding.

First time I read this metric name, I don't know what exact it is, that was why 
I suggested to use a long name. After all we have a lot of long names already 
as metric names. E.g {{LoadingFsImageCount}}, {{LoadingFsImageElapsedTime}}, 
and that explains what they are better (not every one looks the docs).

bq. There will be some more type scm metrics added into SCMMetrics in the 
future. Why we merge these metrics into that class? SCMMXBean is just the JMX 
interface for exposing scm information.

These are SCM level metrics right? Is it a bit overlapping with SCMMXBean?

Overall I am +0.5 to the patch, just want to get upon parts clear before 
getting this in. Hope that makes sense thanks.


was (Author: cheersyang):
Hi [~linyiqun]

bq. Since metric is the data of aggregated statistics. I'd like to update this 
as ConainerSize. This will look simplify and also good understanding.

First time I read this metric name, I don't know what exact it is, that was why 
I suggested to use a long name. After all we have a lot of long names already 
as metric names. E.g {{LoadingFsImageCount}}, {{LoadingFsImageElapsedTime}}, 
and that explains what they are better (not every one looks the docs).

bq. There will be some more type scm metrics added into SCMMetrics in the 
future. Why we merge these metrics into that class? SCMMXBean is just the JMX 
interface for exposing scm information.

These are SCM level metrics right? Is it a bit overlapping with SCMMXBean?

Overall I am +0.5 to the patch, just want to upon parts clear before getting 
this in. Hope that makes sense thanks.

> Ozone: SCM: Add Node Metrics for SCM
> 
>
> Key: HDFS-11468
> URL: https://issues.apache.org/jira/browse/HDFS-11468
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: ozone
>Reporter: Xiaoyu Yao
>Assignee: Yiqun Lin
>Priority: Critical
>  Labels: OzonePostMerge
> Attachments: HDFS-11468-HDFS-7240.001.patch, 
> HDFS-11468-HDFS-7240.002.patch, HDFS-11468-HDFS-7240.003.patch
>
>
> This ticket is opened to add node metrics in SCM based on heartbeat, node 
> report and container report from datanodes. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-11468) Ozone: SCM: Add Node Metrics for SCM

[Weiwei Yang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-11468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16209291#comment-16209291
 ] 

Weiwei Yang commented on HDFS-11468:


Hi [~linyiqun]

bq. Since metric is the data of aggregated statistics. I'd like to update this 
as ConainerSize. This will look simplify and also good understanding.

First time I read this metric name, I don't know what exact it is, that was why 
I suggested to use a long name. After all we have a lot of long names already 
as metric names. E.g {{LoadingFsImageCount}}, {{LoadingFsImageElapsedTime}}, 
and that explains what they are better (not every one looks the docs).

bq. There will be some more type scm metrics added into SCMMetrics in the 
future. Why we merge these metrics into that class? SCMMXBean is just the JMX 
interface for exposing scm information.

These are SCM level metrics right? Is it a bit overlapping with SCMMXBean?

Overall I am +0.5 to the patch, just want to upon parts clear before getting 
this in. Hope that makes sense thanks.

> Ozone: SCM: Add Node Metrics for SCM
> 
>
> Key: HDFS-11468
> URL: https://issues.apache.org/jira/browse/HDFS-11468
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: ozone
>Reporter: Xiaoyu Yao
>Assignee: Yiqun Lin
>Priority: Critical
>  Labels: OzonePostMerge
> Attachments: HDFS-11468-HDFS-7240.001.patch, 
> HDFS-11468-HDFS-7240.002.patch, HDFS-11468-HDFS-7240.003.patch
>
>
> This ticket is opened to add node metrics in SCM based on heartbeat, node 
> report and container report from datanodes. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-12675) Ozone: TestLeaseManager#testLeaseCallbackWithMultipleLeases fails

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-12675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16209249#comment-16209249
 ] 

Hadoop QA commented on HDFS-12675:
--

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
12s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
|| || || || {color:brown} HDFS-7240 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m 
11s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
55s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
41s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m  
1s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
11m 54s{color} | {color:green} branch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  2m  
5s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  
3s{color} | {color:green} HDFS-7240 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
 1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m  
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
11m 28s{color} | {color:green} patch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  2m 
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  
0s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 89m 21s{color} 
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
22s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}140m 34s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.cblock.TestCBlockCLI |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:71bbb86 |
| JIRA Issue | HDFS-12675 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12892764/HDFS-12675-HDFS-7240.001.patch
 |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  shadedclient  findbugs  checkstyle  |
| uname | Linux c0b72d07d926 3.13.0-129-generic #178-Ubuntu SMP Fri Aug 11 
12:48:20 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | HDFS-7240 / 9bf86df |
| Default Java | 1.8.0_144 |
| findbugs | v3.1.0-RC1 |
| unit | 
https://builds.apache.org/job/PreCommit-HDFS-Build/21735/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HDFS-Build/21735/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U: 
hadoop-hdfs-project/hadoop-hdfs |
| Console output | 
https://builds.apache.org/job/PreCommit-HDFS-Build/21735/console |
| Powered by | Apache Yetus 0.6.0-SNAPSHOT   http://yetus.apache.org |


This message was automatically generated.



> Ozone: 

[jira] [Updated] (HDFS-12678) Ozone: Corona: Add statistical information to json output

[Lokesh Jain (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-12678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lokesh Jain updated HDFS-12678:
---
Component/s: ozone

> Ozone: Corona: Add statistical information to json output
> -
>
> Key: HDFS-12678
> URL: https://issues.apache.org/jira/browse/HDFS-12678
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: ozone
>Reporter: Lokesh Jain
>Assignee: Lokesh Jain
>
> The Corona json output should contain statistical information like mean, 
> quantiles and standard deviation of time taken by ozone operations.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-12678) Ozone: Corona: Add statistical information to json output

[Lokesh Jain (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-12678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lokesh Jain updated HDFS-12678:
---
Description: The Corona json output should contain statistical information 
like mean, quantiles and standard deviation of time taken by ozone operations.

> Ozone: Corona: Add statistical information to json output
> -
>
> Key: HDFS-12678
> URL: https://issues.apache.org/jira/browse/HDFS-12678
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: ozone
>Reporter: Lokesh Jain
>Assignee: Lokesh Jain
>
> The Corona json output should contain statistical information like mean, 
> quantiles and standard deviation of time taken by ozone operations.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-12619) Do not catch and throw unchecked exceptions if IBRs fail to process

[Wei-Chiu Chuang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-12619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16209165#comment-16209165
 ] 

Wei-Chiu Chuang commented on HDFS-12619:


Committing the patch based on [~xiaochen] and [~hanishakoneru]'s +1.

> Do not catch and throw unchecked exceptions if IBRs fail to process
> ---
>
> Key: HDFS-12619
> URL: https://issues.apache.org/jira/browse/HDFS-12619
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: namenode
>Affects Versions: 2.8.0, 2.7.3, 3.0.0-alpha1
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
>Priority: Minor
> Attachments: HDFS-12619.001.patch
>
>
> HDFS-9198 added the following code
> {code:title=BlockManager#processIncrementalBlockReport}
> public void processIncrementalBlockReport(final DatanodeID nodeID,
>   final StorageReceivedDeletedBlocks srdb) throws IOException {
> ...
> try {
>   processIncrementalBlockReport(node, srdb);
> } catch (Exception ex) {
>   node.setForceRegistration(true);
>   throw ex;
> }
>   }
> {code}
> In Apache Hadoop 2.7.x ~ 3.0, the code snippet is accepted by Java compiler. 
> However, when I attempted to backport it to a CDH5.3 release (based on Apache 
> Hadoop 2.5.0), the compiler complains the exception is unhandled, because the 
> method defines it throws IOException instead of Exception.
> While the code compiles for Apache Hadoop 2.7.x ~ 3.0, I feel it is not a 
> good practice to catch an unchecked exception and then rethrow it. How about 
> rewriting it with a finally block and a conditional variable?



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Assigned] (HDFS-12678) Ozone: Corona: Add statistical information to json output

[Lokesh Jain (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-12678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lokesh Jain reassigned HDFS-12678:
--

Assignee: Lokesh Jain

> Ozone: Corona: Add statistical information to json output
> -
>
> Key: HDFS-12678
> URL: https://issues.apache.org/jira/browse/HDFS-12678
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>Reporter: Lokesh Jain
>Assignee: Lokesh Jain
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Created] (HDFS-12678) Ozone: Corona: Add statistical information to json output

[Lokesh Jain (JIRA)]

Lokesh Jain created HDFS-12678:
--

 Summary: Ozone: Corona: Add statistical information to json output
 Key: HDFS-12678
 URL: https://issues.apache.org/jira/browse/HDFS-12678
 Project: Hadoop HDFS
  Issue Type: Sub-task
Reporter: Lokesh Jain






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-12521) Ozone: SCM should read all Container info into memory when booting up

[Lokesh Jain (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-12521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16209151#comment-16209151
 ] 

Lokesh Jain commented on HDFS-12521:


Thanks for the review [~linyiqun]. The v3 patch addresses the issues pointed 
out by you.
Regarding the comment :-
_line204: We can just print the error info instead of checking message then log 
error. The error message contained in exception can be showed in logging. 
Another point, please use LOG.error here.
_ 
I did this so that unnecessary exception information is not printed in the 
logs. Because on start the container db is empty and listContainer would raise 
an exception when used for initializing ContainerStateManager.

> Ozone: SCM should read all Container info into memory when booting up
> -
>
> Key: HDFS-12521
> URL: https://issues.apache.org/jira/browse/HDFS-12521
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: ozone
>Reporter: Anu Engineer
>Assignee: Lokesh Jain
>  Labels: ozoneMerge
> Attachments: HDFS-12521-HDFS-7240.001.patch, 
> HDFS-12521-HDFS-7240.002.patch, HDFS-12521-HDFS-7240.003.patch
>
>
> When SCM boots up it should read all containers into memory. This is a 
> performance optimization that allows delays on SCM side. This JIRA tracks 
> that issue.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-12676) when blocks has corrupted replicas，throws Exception

[Wei-Chiu Chuang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-12676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16209146#comment-16209146
 ] 

Wei-Chiu Chuang commented on HDFS-12676:


Correct. Please check out the attached source code patch at HDFS-11445, 
HDFS-11755.

> when blocks has corrupted replicas，throws Exception
> ---
>
> Key: HDFS-12676
> URL: https://issues.apache.org/jira/browse/HDFS-12676
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: hdfs
>Reporter: lynn
>
> when blocks has corrupted replicas，throws Exception as follows:
> Exception 1:
> 2017-10-18 15:24:55,858 WARN  blockmanagement.BlockManager 
> (BlockManager.java:createLocatedBlock(938)) - Inconsistent number of corrupt 
> replicas for blk_1073750384_504374 blockMap has 0 but corrupt replicas map 
> has 1
> 2017-10-18 15:24:55,859 WARN  ipc.Server (Server.java:logException(2433)) - 
> IPC Server handler 116 on 8020, call 
> org.apache.hadoop.hdfs.protocol.ClientProtocol.getBlockLocations from 
> 10.43.160.18:56313 Call#2 Retry#-1
> java.lang.ArrayIndexOutOfBoundsException: 1
>   at 
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.createLocatedBlock(BlockManager.java:972)
>   at 
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.createLocatedBlock(BlockManager.java:911)
>   at 
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.createLocatedBlockList(BlockManager.java:884)
>   at 
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.createLocatedBlocks(BlockManager.java:1011)
>   at 
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:2010)
>   at 
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1960)
>   at 
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1873)
>   at 
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:693)
>   at 
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:373)
>   at 
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
>   at 
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:640)
>   at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982)
>   at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2351)
>   at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2347)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at javax.security.auth.Subject.doAs(Subject.java:422)
>   at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1865)
>   at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2345)
> Exception 2:
> 2017-10-12 16:59:36,591 INFO  blockmanagement.BlockManager 
> (BlockManager.java:computeReplicationWorkForBlocks(1649)) - Blocks chosen but 
> could not be replicated = 4; of which 0 have no target, 4 have no source, 0 
> are UC, 0 are abandoned, 0 already have enough replicas.
> 2017-10-12 16:59:36,809 WARN  blockmanagement.BlockManager 
> (BlockManager.java:createLocatedBlock(938)) - Inconsistent number of corrupt 
> replicas for blk_1073789106_2278702 blockMap has 0 but corrupt replicas map 
> has 2
> 2017-10-12 16:59:36,810 WARN  ipc.Server (Server.java:logException(2433)) - 
> IPC Server handler 123 on 8020, call 
> org.apache.hadoop.hdfs.protocol.ClientProtocol.getBlockLocations from 
> 10.46.230.12:47974 Call#2 Retry#-1
> java.lang.NegativeArraySizeException
>   at 
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.createLocatedBlock(BlockManager.java:946)
>   at 
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.createLocatedBlock(BlockManager.java:911)
>   at 
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.createLocatedBlockList(BlockManager.java:884)
>   at 
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.createLocatedBlocks(BlockManager.java:997)
>   at 
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:2010)
>   at 
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1960)
>   at 
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1873)
>   at 
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:693)
>   at 
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:373)
>   at 
> 

[jira] [Commented] (HDFS-12521) Ozone: SCM should read all Container info into memory when booting up

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-12521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16209135#comment-16209135
 ] 

Hadoop QA commented on HDFS-12521:
--

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m  
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} docker {color} | {color:red}  0m 
21s{color} | {color:red} Docker failed to build yetus/hadoop:71bbb86. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HDFS-12521 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12892768/HDFS-12521-HDFS-7240.003.patch
 |
| Console output | 
https://builds.apache.org/job/PreCommit-HDFS-Build/21736/console |
| Powered by | Apache Yetus 0.6.0-SNAPSHOT   http://yetus.apache.org |


This message was automatically generated.



> Ozone: SCM should read all Container info into memory when booting up
> -
>
> Key: HDFS-12521
> URL: https://issues.apache.org/jira/browse/HDFS-12521
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: ozone
>Reporter: Anu Engineer
>Assignee: Lokesh Jain
>  Labels: ozoneMerge
> Attachments: HDFS-12521-HDFS-7240.001.patch, 
> HDFS-12521-HDFS-7240.002.patch, HDFS-12521-HDFS-7240.003.patch
>
>
> When SCM boots up it should read all containers into memory. This is a 
> performance optimization that allows delays on SCM side. This JIRA tracks 
> that issue.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-12521) Ozone: SCM should read all Container info into memory when booting up

[Lokesh Jain (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-12521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lokesh Jain updated HDFS-12521:
---
Attachment: HDFS-12521-HDFS-7240.003.patch

> Ozone: SCM should read all Container info into memory when booting up
> -
>
> Key: HDFS-12521
> URL: https://issues.apache.org/jira/browse/HDFS-12521
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: ozone
>Reporter: Anu Engineer
>Assignee: Lokesh Jain
>  Labels: ozoneMerge
> Attachments: HDFS-12521-HDFS-7240.001.patch, 
> HDFS-12521-HDFS-7240.002.patch, HDFS-12521-HDFS-7240.003.patch
>
>
> When SCM boots up it should read all containers into memory. This is a 
> performance optimization that allows delays on SCM side. This JIRA tracks 
> that issue.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Comment Edited] (HDFS-12675) Ozone: TestLeaseManager#testLeaseCallbackWithMultipleLeases fails

[Yiqun Lin (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-12675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16208869#comment-16208869
 ] 

Yiqun Lin edited comment on HDFS-12675 at 10/18/17 10:17 AM:
-

I just debugged the test. The root cause of this is that {{Thread#sleep}} in 
{{LeaseMonitor#run}} slowly ran than sleep operation executed in test.  During 
the test, I found  {{LeaseMonitor#run}} would run one or two empty iterations 
then prepare sleep for release lease4 and lease5. But here we only increase a 
little delay time for desired sleep time. So there is a chance this delay is 
smaller than empty-iterations running time.
One simple way is to increase the sleep in test to ensure the lease is released 
and its callback be done.


was (Author: linyiqun):
I just debugged the test. The root cause of this is that {{Thread#sleep}} in 
{{LeaseMonitor#run}} slowly ran than sleep operation executed in test.  During 
the test, I found  {{LeaseMonitor#run}} would run one or two empty iterations 
then prepare sleep for release lease4 and lease5. But here we only increase a 
little delay time for desired sleep time. So there is a chance this delay is 
smaller than empty-iterations running time.
One simple way is to increase the sleep in test to ensure the lease is released 
in loop.

> Ozone: TestLeaseManager#testLeaseCallbackWithMultipleLeases fails 
> --
>
> Key: HDFS-12675
> URL: https://issues.apache.org/jira/browse/HDFS-12675
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Yiqun Lin
>Assignee: Yiqun Lin
> Attachments: HDFS-12675-HDFS-7240.001.patch
>
>
> Caught one UT failure 
> {{TestLeaseManager#testLeaseCallbackWithMultipleLeases}}:
> {noformat}
> jcava.lang.AssertionError: null
>   at org.junit.Assert.fail(Assert.java:86)
>   at org.junit.Assert.assertTrue(Assert.java:41)
>   at org.junit.Assert.assertTrue(Assert.java:52)
>   at 
> org.apache.hadoop.ozone.lease.TestLeaseManager.testLeaseCallbackWithMultipleLeases(TestLeaseManager.java:293)
> {noformat}
> The reason of this error is  lease {{leaseFive}} didn't expire in test.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Comment Edited] (HDFS-12675) Ozone: TestLeaseManager#testLeaseCallbackWithMultipleLeases fails

[Yiqun Lin (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-12675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16208869#comment-16208869
 ] 

Yiqun Lin edited comment on HDFS-12675 at 10/18/17 10:14 AM:
-

I just debugged the test. The root cause of this is that {{Thread#sleep}} in 
{{LeaseMonitor#run}} slowly ran than sleep operation executed in test.  During 
the test, I found  {{LeaseMonitor#run}} would run one or two empty iterations 
then prepare sleep for release lease4 and lease5. But here we only increase a 
little delay time for desired sleep time. So there is a chance this delay is 
smaller than empty-iterations running time.
One simple way is to increase the sleep in test to ensure the lease is released 
in loop.


was (Author: linyiqun):
I am thinking one corner case leading for this:

# {{LeaseNotFoundException}} or {{LeaseExpiredException}} is thrown in loop 
{{for (T resource : activeLeases.keySet()) {}}.
# Then the {{sleepTime}} used in {{while}} loop doesn't get a correct value, 
then program sleep for {{Long.MAX_VALUE}} time.Remaining leases cannot be 
released.

> Ozone: TestLeaseManager#testLeaseCallbackWithMultipleLeases fails 
> --
>
> Key: HDFS-12675
> URL: https://issues.apache.org/jira/browse/HDFS-12675
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Yiqun Lin
>Assignee: Yiqun Lin
> Attachments: HDFS-12675-HDFS-7240.001.patch
>
>
> Caught one UT failure 
> {{TestLeaseManager#testLeaseCallbackWithMultipleLeases}}:
> {noformat}
> jcava.lang.AssertionError: null
>   at org.junit.Assert.fail(Assert.java:86)
>   at org.junit.Assert.assertTrue(Assert.java:41)
>   at org.junit.Assert.assertTrue(Assert.java:52)
>   at 
> org.apache.hadoop.ozone.lease.TestLeaseManager.testLeaseCallbackWithMultipleLeases(TestLeaseManager.java:293)
> {noformat}
> The reason of this error is  lease {{leaseFive}} didn't expire in test.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org