date:20171128

[jira] [Created] (HDFS-12867) Ozone: TestOzoneConfigurationFields fails consitently

2017-11-28 Thread Yiqun Lin (JIRA)

Yiqun Lin created HDFS-12867:


 Summary: Ozone: TestOzoneConfigurationFields fails consitently
 Key: HDFS-12867
 URL: https://issues.apache.org/jira/browse/HDFS-12867
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: ozone, test
Affects Versions: HDFS-7240
Reporter: Yiqun Lin
Assignee: Yiqun Lin


The unit test TestOzoneConfigurationFields fails consistently because of 2 
config entries missing in ozone-default file. The stack trace:
{noformat}
java.lang.AssertionError: class org.apache.hadoop.ozone.OzoneConfigKeys class 
org.apache.hadoop.scm.ScmConfigKeys class 
org.apache.hadoop.ozone.ksm.KSMConfigKeys class 
org.apache.hadoop.cblock.CBlockConfigKeys has 2 variables missing in 
ozone-default.xml Entries:   ozone.rest.client.port  ozone.rest.servers 
expected:<0> but was:<2>
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
at 
org.apache.hadoop.conf.TestConfigurationFieldsBase.testCompareConfigurationClassAgainstXml(TestConfigurationFieldsBase.java:493)
{noformat}
The config {{ozone.rest.client.port}}, {{ozone.rest.servers}} were introduced 
in HDFS-12549
but missing to documented. This leads the failure.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-12866) Recursive delete of a large directory or snapshot makes namenode unresponsive

2017-11-28 Thread Yongjun Zhang (JIRA)

Yongjun Zhang created HDFS-12866:


 Summary: Recursive delete of a large directory or snapshot makes 
namenode unresponsive
 Key: HDFS-12866
 URL: https://issues.apache.org/jira/browse/HDFS-12866
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: hdfs
Reporter: Yongjun Zhang


Currently file/directory deletion happens in two steps (see 
{{FSNamesystem#delete(String src, boolean recursive, boolean logRetryCache)}}:

# Do the following under fsn write lock and release the lock afterwards
** 1.1  recursively traverse the target, collect INodes and all blocks to be 
deleted
** 1.2  delete all INodes
# Delete the blocks to be deleted incrementally, chunk by chunk. That is, in a 
loop, do:   
** acquire fsn write lock,
** delete chunk of blocks
** release fsn write lock

Breaking the deletion to two steps is to not hold the fsn write lock for too 
long thus making NN not responsive. However, even with this, for deleting large 
directory, or deleting snapshot that has a lot of contents, step 1 itself would 
takes long time thus still hold the fsn write lock for too long and make NN not 
responsive.

A possible solution would be to add one more sub step in step 1, and only hold 
fsn write lock in sub step 1.1:

* 1.1. hold the fsn write lock, disconnect the target to be deleted from its 
parent dir, release the lock
* 1.2 recursively traverse the target, collect INodes and all blocks to be 
deleted
* 1.3  delete all INodes

Then do step 2.

This means, any operations on any file/dir need to check if its ancestor is 
deleted (ancestor is disconnected), similar to what's done in 
FSNamesystem#isFileDeleted method.

I'm throwing the thought here for further discussion. Welcome comments and 
inputs.








--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Reopened] (HDFS-9754) Avoid unnecessary getBlockCollection calls in BlockManager

2017-11-28 Thread Konstantin Shvachko (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-9754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Konstantin Shvachko reopened HDFS-9754:
---

There is clearly value in the work done here.
I would rather revert the entire thing in order to unblock 2.8 release, and 
then let people modify the patch.
Reopening this for now.

> Avoid unnecessary getBlockCollection calls in BlockManager
> --
>
> Key: HDFS-9754
> URL: https://issues.apache.org/jira/browse/HDFS-9754
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: namenode
>Reporter: Jing Zhao
>Assignee: Jing Zhao
> Fix For: 2.9.0, 3.0.0-alpha1, 2.8.2
>
> Attachments: HDFS-9754.000.patch, HDFS-9754.001.patch, 
> HDFS-9754.002.patch
>
>
> Currently BlockManager calls {{Namesystem#getBlockCollection}} in order to:
> 1. check if the block has already been abandoned
> 2. identify the storage policy of the block
> 3. meta save
> For #1 we can use BlockInfo's internal state instead of checking if the 
> corresponding file still exists.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86

2017-11-28 Thread Apache Jenkins Server

For more details, see 
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/607/

[Nov 27, 2017 6:19:58 PM] (jianhe) YARN-6168. Restarted RM may not inform AM 
about all existing containers.
[Nov 27, 2017 10:31:52 PM] (yufei) YARN-7363. ContainerLocalizer don't have a 
valid log4j config in case of
[Nov 28, 2017 3:48:55 AM] (yqlin) HDFS-12858. Add router admin commands usage 
in HDFS commands reference
[Nov 28, 2017 11:52:59 AM] (stevel) HADOOP-15042. Azure 
PageBlobInputStream.skip() can return negative value
[Nov 28, 2017 1:07:11 PM] (sunilg) YARN-7499. Layout changes to Application 
details page in new YARN UI.




-1 overall


The following subsystems voted -1:
asflicense findbugs unit


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace


The following subsystems are considered long running:
(runtime bigger than 1h  0m  0s)
unit


Specific tests:

FindBugs :

   module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api 
   org.apache.hadoop.yarn.api.records.Resource.getResources() may expose 
internal representation by returning Resource.resources At Resource.java:by 
returning Resource.resources At Resource.java:[line 213] 

Failed junit tests :

   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure000 
   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure150 
   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure 
   hadoop.fs.viewfs.TestViewFileSystemLinkFallback 
   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure070 
   hadoop.fs.viewfs.TestViewFsWithXAttrs 
   hadoop.hdfs.TestQuota 
   hadoop.hdfs.TestMaintenanceState 
   hadoop.hdfs.TestErasureCodingPoliciesWithRandomECPolicy 
   hadoop.hdfs.TestSetrepIncreasing 
   hadoop.hdfs.TestDFSStripedInputStream 
   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure210 
   hadoop.fs.viewfs.TestViewFileSystemHdfs 
   hadoop.hdfs.TestDFSStripedOutputStream 
   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure200 
   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure190 
   hadoop.hdfs.TestClientProtocolForPipelineRecovery 
   hadoop.hdfs.server.balancer.TestBalancerRPCDelay 
   hadoop.hdfs.TestUnsetAndChangeDirectoryEcPolicy 
   hadoop.fs.viewfs.TestViewFileSystemLinkMergeSlash 
   hadoop.hdfs.web.TestWebHdfsTimeouts 
   hadoop.hdfs.TestErasureCodingPolicies 
   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure060 
   hadoop.fs.TestUnbuffer 
   
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestNodeLabelContainerAllocation
 
   hadoop.mapreduce.v2.hs.webapp.TestHsWebServicesAttempts 
   hadoop.mapreduce.v2.hs.webapp.TestHsWebServicesJobs 
   hadoop.yarn.service.TestServiceAM 
   hadoop.yarn.service.TestYarnNativeServices 
   hadoop.yarn.sls.nodemanager.TestNMSimulator 
  

   cc:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/607/artifact/out/diff-compile-cc-root.txt
  [4.0K]

   javac:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/607/artifact/out/diff-compile-javac-root.txt
  [276K]

   checkstyle:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/607/artifact/out/diff-checkstyle-root.txt
  [17M]

   pylint:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/607/artifact/out/diff-patch-pylint.txt
  [20K]

   shellcheck:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/607/artifact/out/diff-patch-shellcheck.txt
  [20K]

   shelldocs:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/607/artifact/out/diff-patch-shelldocs.txt
  [12K]

   whitespace:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/607/artifact/out/whitespace-eol.txt
  [8.8M]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/607/artifact/out/whitespace-tabs.txt
  [288K]

   findbugs:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/607/artifact/out/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api-warnings.html
  [8.0K]

   javadoc:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/607/artifact/out/diff-javadoc-javadoc-root.txt
  [760K]

   unit:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/607/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
  [1.8M]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/607/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
  [80K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/607/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-hs.txt
  [104K]

Re: [DISCUSS] Merge Absolute resource configuration support in Capacity Scheduler (YARN-5881) to trunk

2017-11-28 Thread Eric Payne

Thanks Sunil for the great work on this feature.
I looked through the design document, reviewed the code, and tested out branch
YARN-5881. The design makes sense and the code looks like it is implementing
the desing in a sensible way. However, I have encountered a couple of bugs. I
opened https://issues.apache.org/jira/browse/YARN-7575 to track my findings.
Basically, here's a summary:

The design document from YARN-5881 says that for max-capacity:
3) For each queue, we require: a) if max-resource not set, it automatically
set to parent.max-resource

When I try not setting
anyyarn.scheduler.capacity..maximum-capacity, the RMUI scheduler
page refuses to render. It looks like it's in
CapacitySchedulerPage$LeafQueueInfoBlock.

Also... A job will run in the leaf queue with no max capacity set and it will
grow to the max capacity of the cluster, but if I add resources to the node,
the job won't grow any more even though it has pending resources.

Thanks,Eric

From: Sunil G
To: "yarn-...@hadoop.apache.org" ; Hadoop Common
; Hdfs-dev ;
"mapreduce-...@hadoop.apache.org"
Sent: Friday, November 24, 2017 11:49 AM
Subject: [DISCUSS] Merge Absolute resource configuration support in Capacity
Scheduler (YARN-5881) to trunk

Hi All,

We would like to bring up the discussion of merging “absolute min/max
resources support in capacity scheduler” branch (YARN-5881) [2] into trunk
in a few weeks. The goal is to get it in for Hadoop 3.1.

*Major work happened in this branch*

- YARN-6471. Support to add min/max resource configuration for a queue
- YARN-7332. Compute effectiveCapacity per each resource vector
- YARN-7411. Inter-Queue preemption's computeFixpointAllocation need to
handle absolute resources.

*Regarding design details*

Please refer [1] for detailed design document.

*Regarding to testing:*

We did extensive tests for the feature in the last couple of months.
Comparing to latest trunk.

- For SLS benchmark: We didn't see observable performance gap from
simulated test based on 8K nodes SLS traces (1 PB memory). We got 3k+
containers allocated per second.

- For microbenchmark: We use performance test cases added by YARN 6775, it
did not show much performance regression comparing to trunk.

*YARN-5881*

#ResourceTypes = 2. Avg of fastest 20: 55294.52
#ResourceTypes = 2. Avg of fastest 20: 55401.66

*trunk*
#ResourceTypes = 2. Avg of fastest 20: 55865.92
#ResourceTypes = 2. Avg of fastest 20: 55096.418

*Regarding to API stability:*

All newly added @Public APIs are @Unstable.

Documentation jira [3] could help to provide detailed configuration
details. This feature works from end-to-end and we are running this in our
development cluster for last couple of months and undergone good amount of
testing. Branch code is run against trunk and tracked via [4].

We would love to get your thoughts before opening a voting thread.

Special thanks to a team of folks who worked hard and contributed towards
this efforts including design discussion / patch / reviews, etc.: Wangda
Tan, Vinod Kumar Vavilappali, Rohith Sharma K S.

[1] :
https://issues.apache.org/jira/secure/attachment/12855984/YARN-5881.Support.Absolute.Min.Max.Resource.In.Capacity.Scheduler.design-doc.v1.pdf
[2] : https://issues.apache.org/jira/browse/YARN-5881

[3] : https://issues.apache.org/jira/browse/YARN-7533

[4] : https://issues.apache.org/jira/browse/YARN-7510

Thanks,

Sunil G and Wangda Tan

[jira] [Created] (HDFS-12865) RequestHedgingProxyProvider should handle case when none of the proxies are available

2017-11-28 Thread Mukul Kumar Singh (JIRA)

Mukul Kumar Singh created HDFS-12865:


 Summary: RequestHedgingProxyProvider should handle case when none 
of the proxies are available
 Key: HDFS-12865
 URL: https://issues.apache.org/jira/browse/HDFS-12865
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: ha
Reporter: Mukul Kumar Singh
Assignee: Mukul Kumar Singh


RequestHedgingProxyProvider when all the targets have failover'ed will throw a 
MultiException as expected. But this MultiException will not have the 
corresponding lower level exceptions.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-12864) Ozone: Turn off full synced write in RocksDB MetadataStore

2017-11-28 Thread Elek, Marton (JIRA)

Elek, Marton created HDFS-12864:
---

 Summary: Ozone: Turn off full synced write in RocksDB MetadataStore
 Key: HDFS-12864
 URL: https://issues.apache.org/jira/browse/HDFS-12864
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: ozone
Reporter: Elek, Marton
Assignee: Elek, Marton


Based on the suggestion from [~xyao] I created the patch to turn off fully 
synced write in the rocksdb.

With this patch we measured about 10x performance gain (200 nodes cluster + 30 
corona instance).

1. We don't need fully synced rocksdb writing, as HA will cover the few cases 
when a node would be lost

2. Long term (IMHO) we need to add a possibility to configure any settings in 
rocksdb, so it could be turned off, but the setSync(false) is a reasonable 
default.




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-12863) Avoid displaying configuration deprecation messages many times

2017-11-28 Thread Akira Ajisaka (JIRA)

Akira Ajisaka created HDFS-12863:


 Summary: Avoid displaying configuration deprecation messages many 
times
 Key: HDFS-12863
 URL: https://issues.apache.org/jira/browse/HDFS-12863
 Project: Hadoop HDFS
  Issue Type: Sub-task
Reporter: Akira Ajisaka
Priority: Minor


In DataNode daemon log, the following info message is displayed every 30 
seconds:
{noformat}
2017-11-28 12:19:52,127 INFO org.apache.hadoop.conf.Configuration.deprecation: 
No unit for ozone.scm.heartbeat.interval.seconds(30) assuming SECONDS
2017-11-28 12:20:22,127 INFO org.apache.hadoop.conf.Configuration.deprecation: 
No unit for ozone.scm.heartbeat.interval.seconds(30) assuming SECONDS
2017-11-28 12:20:52,128 INFO org.apache.hadoop.conf.Configuration.deprecation: 
No unit for ozone.scm.heartbeat.interval.seconds(30) assuming SECONDS
{noformat}
The message should be displayed once.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Resolved] (HDFS-9477) namenode starts failed:FSEditLogLoader: Encountered exception on operation TimesOp

2017-11-28 Thread Yuanbo Liu (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-9477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yuanbo Liu resolved HDFS-9477.
--
Resolution: Fixed

I believe that HDFS-8269 has solved this issue, please check it.

> namenode starts failed:FSEditLogLoader: Encountered exception on operation 
> TimesOp
> --
>
> Key: HDFS-9477
> URL: https://issues.apache.org/jira/browse/HDFS-9477
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: namenode
>Affects Versions: 2.7.0
> Environment: Ubuntu 12.04.1 LTS, java version "1.7.0_79"
>Reporter: aplee
>Assignee: aplee
>
> backup namenode start failed, log below:
> 2015-11-28 14:09:13,462 ERROR 
> org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader: Encountered exception 
> on operation TimesOp [length=0, path=/.reserved/.inodes/2346114, mtime=-1, 
> atime=1448692924700, opCode=OP_TIMES, txid=14774180]
> java.lang.NullPointerException
>   at 
> org.apache.hadoop.hdfs.server.namenode.FSDirAttrOp.unprotectedSetTimes(FSDirAttrOp.java:473)
>   at 
> org.apache.hadoop.hdfs.server.namenode.FSDirAttrOp.unprotectedSetTimes(FSDirAttrOp.java:299)
>   at 
> org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.applyEditLogOp(FSEditLogLoader.java:629)
>   at 
> org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:234)
>   at 
> org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:143)
>   at 
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:832)
>   at 
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:813)
>   at 
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer.doTailEdits(EditLogTailer.java:232)
>   at 
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.doWork(EditLogTailer.java:331)
>   at 
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.access$200(EditLogTailer.java:284)
>   at 
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread$1.run(EditLogTailer.java:301)
>   at 
> org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.java:415)
>   at 
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.run(EditLogTailer.java:297)
> 2015-11-28 14:09:13,572 FATAL 
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer: Unknown error 
> encountered while tailing edits. Shutting down standby NN.
> java.io.IOException: java.lang.NullPointerException
>   at 
> org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:244)
>   at 
> org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:143)
>   at 
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:832)
>   at 
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:813)
>   at 
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer.doTailEdits(EditLogTailer.java:232)
>   at 
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.doWork(EditLogTailer.java:331)
>   at 
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.access$200(EditLogTailer.java:284)
>   at 
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread$1.run(EditLogTailer.java:301)
>   at 
> org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.java:415)
>   at 
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.run(EditLogTailer.java:297)
> Caused by: java.lang.NullPointerException
>   at 
> org.apache.hadoop.hdfs.server.namenode.FSDirAttrOp.unprotectedSetTimes(FSDirAttrOp.java:473)
>   at 
> org.apache.hadoop.hdfs.server.namenode.FSDirAttrOp.unprotectedSetTimes(FSDirAttrOp.java:299)
>   at 
> org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.applyEditLogOp(FSEditLogLoader.java:629)
>   at 
> org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:234)
>   ... 9 more
> 2015-11-28 14:09:13,574 INFO org.apache.hadoop.util.ExitUtil: Exiting with 
> status 1
> 2015-11-28 14:09:13,575 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: 
> SHUTDOWN_MSG: 
> I found record in Edits, but I don't know how this record generated
> 
> OP_TIMES
> 
>   14774180
>   0
>   /.reserved/.inodes/2346114
>   -1
>   1448692924700
> 
>   



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-12862) When modify cacheDirective ,editLog myq serial relative expiryTime

2017-11-28 Thread Wang XL (JIRA)

Wang XL created HDFS-12862:
--

 Summary: When modify cacheDirective ,editLog myq serial relative 
expiryTime
 Key: HDFS-12862
 URL: https://issues.apache.org/jira/browse/HDFS-12862
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: caching, hdfs
Affects Versions: 2.7.1
 Environment: 

Reporter: Wang XL


The logic in FSNDNCacheOp#modifyCacheDirective is not correct.  when modify 
cacheDirective,the expiration in directive may be a relative expiryTime, and 
EditLog will 
 serial a relative expiry time.
{code:java}
// Some comments here
static void modifyCacheDirective(
  FSNamesystem fsn, CacheManager cacheManager, CacheDirectiveInfo directive,
  EnumSet flags, boolean logRetryCache) throws IOException {
final FSPermissionChecker pc = getFsPermissionChecker(fsn);

cacheManager.modifyDirective(directive, pc, flags);
fsn.getEditLog().logModifyCacheDirectiveInfo(directive, logRetryCache);
  }
{code}

But when standBy NN reload the log ,it will call 
FSImageSerialization#readCacheDirectiveInfo  as a absolute expiryTime.It will 
result in the inconsistency .

{code:java}
  public static CacheDirectiveInfo readCacheDirectiveInfo(DataInput in)
  throws IOException {
CacheDirectiveInfo.Builder builder =
new CacheDirectiveInfo.Builder();
builder.setId(readLong(in));
int flags = in.readInt();
if ((flags & 0x1) != 0) {
  builder.setPath(new Path(readString(in)));
}
if ((flags & 0x2) != 0) {
  builder.setReplication(readShort(in));
}
if ((flags & 0x4) != 0) {
  builder.setPool(readString(in));
}
if ((flags & 0x8) != 0) {
  builder.setExpiration(
  CacheDirectiveInfo.Expiration.newAbsolute(readLong(in)));
}
if ((flags & ~0xF) != 0) {
  throw new IOException("unknown flags set in " +
  "ModifyCacheDirectiveInfoOp: " + flags);
}
return builder.build();
  }
{code}




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org