[jira] [Commented] (HDFS-6783) Fix not really wait for rescan in HDFS CacheReplicationModitor

2014-07-31 Thread Yi Liu (JIRA) 

[ 
https://issues.apache.org/jira/browse/HDFS-6783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14080614#comment-14080614
 ] 

Yi Liu commented on HDFS-6783:
--

How about we distinguish these two kinds of rescan (triggered by needsRescan, 
or interval time up), and in {{waitForRescanIfNeeded}} we only check the rescan 
triggered by {{needsRescan}}.

 Fix not really wait for rescan in HDFS CacheReplicationModitor
 --

 Key: HDFS-6783
 URL: https://issues.apache.org/jira/browse/HDFS-6783
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: caching
Affects Versions: 3.0.0
Reporter: Yi Liu
Assignee: Yi Liu
 Attachments: HDFS-6783.001.patch


 In monitor thread, needsRescan is set to false before real scan starts, so 
 for {{waitForRescanIfNeeded}} will return for the first condition:
 {code}
 if (!needsRescan) {
   return;
 }
 {code}



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HDFS-6783) Fix not really wait for rescan in HDFS CacheReplicationModitor

2014-07-31 Thread Hadoop QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/HDFS-6783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14080777#comment-14080777
 ] 

Hadoop QA commented on HDFS-6783:
-

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12658868/HDFS-6783.002.patch
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:red}-1 tests included{color}.  The patch doesn't appear to include 
any new or modified tests.
Please justify why no new tests are needed for this 
patch.
Also please list what manual steps were performed to 
verify this patch.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  There were no new javadoc warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 2.0.3) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:red}-1 core tests{color}.  The patch failed these unit tests in 
hadoop-hdfs-project/hadoop-hdfs:

  
org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover

{color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: 
https://builds.apache.org/job/PreCommit-HDFS-Build/7507//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7507//console

This message is automatically generated.

 Fix not really wait for rescan in HDFS CacheReplicationModitor
 --

 Key: HDFS-6783
 URL: https://issues.apache.org/jira/browse/HDFS-6783
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: caching
Affects Versions: 3.0.0
Reporter: Yi Liu
Assignee: Yi Liu
 Attachments: HDFS-6783.001.patch, HDFS-6783.002.patch


 In monitor thread, needsRescan is set to false before real scan starts, so 
 for {{waitForRescanIfNeeded}} will return for the first condition:
 {code}
 if (!needsRescan) {
   return;
 }
 {code}



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HDFS-6783) Fix not really wait for rescan in HDFS CacheReplicationModitor

2014-07-30 Thread Hadoop QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/HDFS-6783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14079400#comment-14079400
 ] 

Hadoop QA commented on HDFS-6783:
-

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12658639/HDFS-6783.001.patch
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:red}-1 tests included{color}.  The patch doesn't appear to include 
any new or modified tests.
Please justify why no new tests are needed for this 
patch.
Also please list what manual steps were performed to 
verify this patch.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  There were no new javadoc warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 2.0.3) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:red}-1 core tests{color}.  The patch failed these unit tests in 
hadoop-hdfs-project/hadoop-hdfs:

  
org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover
  org.apache.hadoop.hdfs.server.datanode.TestBlockRecovery

{color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: 
https://builds.apache.org/job/PreCommit-HDFS-Build/7496//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7496//console

This message is automatically generated.

 Fix not really wait for rescan in HDFS CacheReplicationModitor
 --

 Key: HDFS-6783
 URL: https://issues.apache.org/jira/browse/HDFS-6783
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: caching
Affects Versions: 3.0.0
Reporter: Yi Liu
Assignee: Yi Liu
 Attachments: HDFS-6783.001.patch


 In monitor thread, needsRescan is set to false before real scan starts, so 
 for {{waitForRescanIfNeeded}} will return for the first condition:
 {code}
 if (!needsRescan) {
   return;
 }
 {code}



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HDFS-6783) Fix not really wait for rescan in HDFS CacheReplicationModitor

2014-07-30 Thread Colin Patrick McCabe (JIRA) 

[ 
https://issues.apache.org/jira/browse/HDFS-6783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14080370#comment-14080370
 ] 

Colin Patrick McCabe commented on HDFS-6783:


OK.  So the concern is a race between 
{code}
  isScanning = true;
  needsRescan = false;
} finally {
  lock.unlock();
}
{code}
And code that calls {{setNeedsRescan}} followed by {{waitForRescanIfNeeded}}.  
Since there are so many code paths that drop the lock in between those two 
calls, this could certainly happen, I think.

This patch would cause us to wait for even rescans that were not triggered by 
{{needsRescan}}, though.

We probably need some kind of scan epoch system to do this right...

 Fix not really wait for rescan in HDFS CacheReplicationModitor
 --

 Key: HDFS-6783
 URL: https://issues.apache.org/jira/browse/HDFS-6783
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: caching
Affects Versions: 3.0.0
Reporter: Yi Liu
Assignee: Yi Liu
 Attachments: HDFS-6783.001.patch


 In monitor thread, needsRescan is set to false before real scan starts, so 
 for {{waitForRescanIfNeeded}} will return for the first condition:
 {code}
 if (!needsRescan) {
   return;
 }
 {code}



--
This message was sent by Atlassian JIRA
(v6.2#6252)