[jira] [Commented] (HDFS-6991) Notify NN of evicted block before deleting it from RAM disk

2014-10-02 Thread Hudson (JIRA) 

[ 
https://issues.apache.org/jira/browse/HDFS-6991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14156353#comment-14156353
 ] 

Hudson commented on HDFS-6991:
--

FAILURE: Integrated in Hadoop-Yarn-trunk #698 (See 
[https://builds.apache.org/job/Hadoop-Yarn-trunk/698/])
HDFS-6991. Notify NN of evicted block before deleting it from RAM disk. (Arpit 
Agarwal) (arp: rev a18caf7753623a94a7cdb1c607cda79586de08dc)
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-6581.txt
* 
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/MiniDFSCluster.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPOfferService.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestLazyPersistFiles.java


 Notify NN of evicted block before deleting it from RAM disk
 ---

 Key: HDFS-6991
 URL: https://issues.apache.org/jira/browse/HDFS-6991
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: datanode, namenode
Affects Versions: HDFS-6581
Reporter: Arpit Agarwal
Assignee: Arpit Agarwal
 Fix For: 3.0.0

 Attachments: HDFS-6991.01.patch, HDFS-6991.02.patch, 
 HDFS-6991.03.patch


 Couple of bug fixes required around eviction:
 # When evicting a block from RAM disk to persistent storage, the DN should 
 schedule an incremental block report for a 'received' replica on persistent 
 storage.
 # {{BlockManager.processReportedBlock}} needs a fix to correctly update the 
 storage ID to reflect the block moving from RAM_DISK to DISK.
 Found by [~xyao] via HDFS-6950.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-6991) Notify NN of evicted block before deleting it from RAM disk

2014-10-02 Thread Hudson (JIRA) 

[ 
https://issues.apache.org/jira/browse/HDFS-6991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14156431#comment-14156431
 ] 

Hudson commented on HDFS-6991:
--

FAILURE: Integrated in Hadoop-Hdfs-trunk #1889 (See 
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/1889/])
HDFS-6991. Notify NN of evicted block before deleting it from RAM disk. (Arpit 
Agarwal) (arp: rev a18caf7753623a94a7cdb1c607cda79586de08dc)
* hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-6581.txt
* 
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/MiniDFSCluster.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestLazyPersistFiles.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPOfferService.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java


 Notify NN of evicted block before deleting it from RAM disk
 ---

 Key: HDFS-6991
 URL: https://issues.apache.org/jira/browse/HDFS-6991
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: datanode, namenode
Affects Versions: HDFS-6581
Reporter: Arpit Agarwal
Assignee: Arpit Agarwal
 Fix For: 3.0.0

 Attachments: HDFS-6991.01.patch, HDFS-6991.02.patch, 
 HDFS-6991.03.patch


 Couple of bug fixes required around eviction:
 # When evicting a block from RAM disk to persistent storage, the DN should 
 schedule an incremental block report for a 'received' replica on persistent 
 storage.
 # {{BlockManager.processReportedBlock}} needs a fix to correctly update the 
 storage ID to reflect the block moving from RAM_DISK to DISK.
 Found by [~xyao] via HDFS-6950.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-6991) Notify NN of evicted block before deleting it from RAM disk

2014-10-02 Thread Hudson (JIRA) 

[ 
https://issues.apache.org/jira/browse/HDFS-6991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14156548#comment-14156548
 ] 

Hudson commented on HDFS-6991:
--

FAILURE: Integrated in Hadoop-Mapreduce-trunk #1914 (See 
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1914/])
HDFS-6991. Notify NN of evicted block before deleting it from RAM disk. (Arpit 
Agarwal) (arp: rev a18caf7753623a94a7cdb1c607cda79586de08dc)
* hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-6581.txt
* 
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestLazyPersistFiles.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/MiniDFSCluster.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPOfferService.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java


 Notify NN of evicted block before deleting it from RAM disk
 ---

 Key: HDFS-6991
 URL: https://issues.apache.org/jira/browse/HDFS-6991
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: datanode, namenode
Affects Versions: HDFS-6581
Reporter: Arpit Agarwal
Assignee: Arpit Agarwal
 Fix For: 3.0.0

 Attachments: HDFS-6991.01.patch, HDFS-6991.02.patch, 
 HDFS-6991.03.patch


 Couple of bug fixes required around eviction:
 # When evicting a block from RAM disk to persistent storage, the DN should 
 schedule an incremental block report for a 'received' replica on persistent 
 storage.
 # {{BlockManager.processReportedBlock}} needs a fix to correctly update the 
 storage ID to reflect the block moving from RAM_DISK to DISK.
 Found by [~xyao] via HDFS-6950.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-6991) Notify NN of evicted block before deleting it from RAM disk

2014-10-01 Thread Hudson (JIRA) 

[ 
https://issues.apache.org/jira/browse/HDFS-6991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14155027#comment-14155027
 ] 

Hudson commented on HDFS-6991:
--

FAILURE: Integrated in Hadoop-trunk-Commit #6163 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/6163/])
HDFS-6991. Notify NN of evicted block before deleting it from RAM disk. (Arpit 
Agarwal) (arp: rev a18caf7753623a94a7cdb1c607cda79586de08dc)
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestLazyPersistFiles.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/MiniDFSCluster.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPOfferService.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-6581.txt
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java


 Notify NN of evicted block before deleting it from RAM disk
 ---

 Key: HDFS-6991
 URL: https://issues.apache.org/jira/browse/HDFS-6991
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: datanode, namenode
Affects Versions: HDFS-6581
Reporter: Arpit Agarwal
Assignee: Arpit Agarwal
 Fix For: HDFS-6581

 Attachments: HDFS-6991.01.patch, HDFS-6991.02.patch, 
 HDFS-6991.03.patch


 Couple of bug fixes required around eviction:
 # When evicting a block from RAM disk to persistent storage, the DN should 
 schedule an incremental block report for a 'received' replica on persistent 
 storage.
 # {{BlockManager.processReportedBlock}} needs a fix to correctly update the 
 storage ID to reflect the block moving from RAM_DISK to DISK.
 Found by [~xyao] via HDFS-6950.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-6991) Notify NN of evicted block before deleting it from RAM disk

2014-09-08 Thread Jitendra Nath Pandey (JIRA) 

[ 
https://issues.apache.org/jira/browse/HDFS-6991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14126102#comment-14126102
 ] 

Jitendra Nath Pandey commented on HDFS-6991:


+1

 Notify NN of evicted block before deleting it from RAM disk
 ---

 Key: HDFS-6991
 URL: https://issues.apache.org/jira/browse/HDFS-6991
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: datanode, namenode
Affects Versions: HDFS-6581
Reporter: Arpit Agarwal
Assignee: Arpit Agarwal
 Attachments: HDFS-6991.01.patch, HDFS-6991.02.patch, 
 HDFS-6991.03.patch


 Couple of bug fixes required around eviction:
 # When evicting a block from RAM disk to persistent storage, the DN should 
 schedule an incremental block report for a 'received' replica on persistent 
 storage.
 # {{BlockManager.processReportedBlock}} needs a fix to correctly update the 
 storage ID to reflect the block moving from RAM_DISK to DISK.
 Found by [~xyao] via HDFS-6950.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-6991) Notify NN of evicted block before deleting it from RAM disk

2014-09-04 Thread Xiaoyu Yao (JIRA) 

[ 
https://issues.apache.org/jira/browse/HDFS-6991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14121933#comment-14121933
 ] 

Xiaoyu Yao commented on HDFS-6991:
--

+1 with some minor issues in FsDataSetImpl.java 

line 2298:  should be  
{code}
while (iterations++  MAX_BLOCK_EVICTIONS_PER_ITERATION 
 transientFreeSpaceBelowThreshold())
{code}

line:2303: Log.info should be LOG.debug
{code}
if (LOG.isDebugEnabled()) {
  LOG.info(Evicting block  + replicaState);
}
{code}

 Notify NN of evicted block before deleting it from RAM disk
 ---

 Key: HDFS-6991
 URL: https://issues.apache.org/jira/browse/HDFS-6991
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: datanode, namenode
Affects Versions: HDFS-6581
Reporter: Arpit Agarwal
Assignee: Arpit Agarwal
 Attachments: HDFS-6991.01.patch, HDFS-6991.02.patch


 When evicting a block from RAM disk to persistent storage, the DN should 
 notify the NN of the persistent replica before deleting the replica from RAM 
 disk. Else there can be a window of time during which the block is considered 
 'missing' by the NN.
 Found by [~xyao] via HDFS-6950.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-6991) Notify NN of evicted block before deleting it from RAM disk

2014-09-03 Thread Arpit Agarwal (JIRA) 

[ 
https://issues.apache.org/jira/browse/HDFS-6991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14119458#comment-14119458
 ] 

Arpit Agarwal commented on HDFS-6991:
-

The patch requires the patches for HDFS-6930, HDFS-6977 and HDFS-6950 to be 
applied in that order.

 Notify NN of evicted block before deleting it from RAM disk
 ---

 Key: HDFS-6991
 URL: https://issues.apache.org/jira/browse/HDFS-6991
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: datanode, namenode
Affects Versions: HDFS-6581
Reporter: Arpit Agarwal
Assignee: Arpit Agarwal
 Attachments: HDFS-6991.01.patch


 When evicting a block from RAM disk to persistent storage, the DN should 
 notify the NN of the persistent replica before deleting the replica from RAM 
 disk. Else there can be a window of time during which the block is considered 
 'missing' by the NN.
 Found by [~xyao] via HDFS-6950.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)