[jira] [Updated] (HDFS-8234) DistributedFileSystem and Globber should apply PathFilter early
[
https://issues.apache.org/jira/browse/HDFS-8234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
J.Andreina updated HDFS-8234:
-
Attachment: HDFS-8234.3.patch
Attaching rebased patch.
Please review.
> DistributedFileSystem and Globber should apply PathFilter early
> ---
>
> Key: HDFS-8234
> URL: https://issues.apache.org/jira/browse/HDFS-8234
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Rohini Palaniswamy
>Assignee: J.Andreina
> Labels: newbie
> Attachments: HDFS-8234.1.patch, HDFS-8234.2.patch, HDFS-8234.3.patch
>
>
> HDFS-985 added partial listing in listStatus to avoid listing entries of
> large directory in one go. If listStatus(Path p, PathFilter f) call is made,
> filter is applied after fetching all the entries resulting in a big list
> being constructed on the client side. If the
> DistributedFileSystem.listStatusInternal() applied the PathFilter it would be
> more efficient. So DistributedFileSystem should override listStatus(Path f,
> PathFilter filter) and apply PathFilter early.
> Globber.java also applies filter after calling listStatus. It should call
> listStatus with the PathFilter.
> {code}
> FileStatus[] children = listStatus(candidate.getPath());
>.
> for (FileStatus child : children) {
> // Set the child path based on the parent path.
> child.setPath(new Path(candidate.getPath(),
> child.getPath().getName()));
> if (globFilter.accept(child.getPath())) {
> newCandidates.add(child);
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-10191) [NNBench] OP_DELETE Operation is'nt working
J.Andreina created HDFS-10191: - Summary: [NNBench] OP_DELETE Operation is'nt working Key: HDFS-10191 URL: https://issues.apache.org/jira/browse/HDFS-10191 Project: Hadoop HDFS Issue Type: Bug Reporter: J.Andreina Assignee: J.Andreina After the fix of MAPREDUCE-6363 , in NNBench OP_DELETE Operation is'nt working -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8935) Erasure Coding: createErasureCodingZone api should accept the policyname as argument instead of ErasureCodingPolicy
[ https://issues.apache.org/jira/browse/HDFS-8935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] J.Andreina updated HDFS-8935: - Status: Patch Available (was: Open) > Erasure Coding: createErasureCodingZone api should accept the policyname as > argument instead of ErasureCodingPolicy > --- > > Key: HDFS-8935 > URL: https://issues.apache.org/jira/browse/HDFS-8935 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: J.Andreina >Assignee: J.Andreina > Attachments: HDFS-8935.1.patch > > > Current behavior : User has to specify ErasureCodingPolicy as an argument for > createErasureCodingZone api . > This can be made in sync with creation of EC zone through CLI , where user > need to specify only the policy name. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8935) Erasure Coding: createErasureCodingZone api should accept the policyname as argument instead of ErasureCodingPolicy
[ https://issues.apache.org/jira/browse/HDFS-8935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] J.Andreina updated HDFS-8935: - Attachment: HDFS-8935.1.patch Attaching an initial patch. Please review. > Erasure Coding: createErasureCodingZone api should accept the policyname as > argument instead of ErasureCodingPolicy > --- > > Key: HDFS-8935 > URL: https://issues.apache.org/jira/browse/HDFS-8935 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: J.Andreina >Assignee: J.Andreina > Attachments: HDFS-8935.1.patch > > > Current behavior : User has to specify ErasureCodingPolicy as an argument for > createErasureCodingZone api . > This can be made in sync with creation of EC zone through CLI , where user > need to specify only the policy name. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-9393) After choosing favored nodes, choosing nodes for remaining replicas should go through BlockPlacementPolicy
[ https://issues.apache.org/jira/browse/HDFS-9393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] J.Andreina updated HDFS-9393: - Attachment: HDFS-9393.3.patch Updated the patch fixing whitespace. Testcase failures are unrelated Please review. > After choosing favored nodes, choosing nodes for remaining replicas should go > through BlockPlacementPolicy > -- > > Key: HDFS-9393 > URL: https://issues.apache.org/jira/browse/HDFS-9393 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: J.Andreina >Assignee: J.Andreina > Attachments: HDFS-9393.1.patch, HDFS-9393.2.patch, HDFS-9393.3.patch > > > Current Behavior is : > After choosing replicas from passed favored nodes , choosing nodes for > remaining replica does not go through BlockPlacementPolicy. > Hence eventhough there is a local client datanode is available and not passed > as part of favored nodes , probability for choosing local datanode is less. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7447) Number of maximum Acl entries on a File/Folder should be made user configurable than hardcoding .
[
https://issues.apache.org/jira/browse/HDFS-7447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15059466#comment-15059466
]
J.Andreina commented on HDFS-7447:
--
Thanks [~qwertymaniac] for the clarifications.
I'll close this jira.
> Number of maximum Acl entries on a File/Folder should be made user
> configurable than hardcoding .
> -
>
> Key: HDFS-7447
> URL: https://issues.apache.org/jira/browse/HDFS-7447
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: security
>Reporter: J.Andreina
>
> By default on creating a folder1 will have 6 acl entries . On top of that
> assigning acl on a folder1 exceeds 32 , then unable to assign acls for a
> group/user to folder1.
> {noformat}
> 2014-11-20 18:55:06,553 ERROR [qtp1279235236-17 - /rolexml/role/modrole]
> Error occured while setting permissions for Resource:[
> hdfs://hacluster/folder1 ] and Error message is : Invalid ACL: ACL has 33
> entries, which exceeds maximum of 32.
> at
> org.apache.hadoop.hdfs.server.namenode.AclTransformation.buildAndValidateAcl(AclTransformation.java:274)
> at
> org.apache.hadoop.hdfs.server.namenode.AclTransformation.mergeAclEntries(AclTransformation.java:181)
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.unprotectedModifyAclEntries(FSDirectory.java:2771)
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.modifyAclEntries(FSDirectory.java:2757)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.modifyAclEntries(FSNamesystem.java:7734)
> {noformat}
> Here value 32 is hardcoded , which can be made user configurable.
> {noformat}
> private static List buildAndValidateAcl(ArrayList aclBuilder)
> throws AclException
> {
> if(aclBuilder.size() > 32)
> throw new AclException((new StringBuilder()).append("Invalid ACL:
> ACL has ").append(aclBuilder.size()).append(" entries, which exceeds maximum
> of ").append(32).append(".").toString());
> :
> :
> }
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HDFS-7447) Number of maximum Acl entries on a File/Folder should be made user configurable than hardcoding .
[
https://issues.apache.org/jira/browse/HDFS-7447?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
J.Andreina resolved HDFS-7447.
--
Resolution: Not A Problem
> Number of maximum Acl entries on a File/Folder should be made user
> configurable than hardcoding .
> -
>
> Key: HDFS-7447
> URL: https://issues.apache.org/jira/browse/HDFS-7447
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: security
>Reporter: J.Andreina
>
> By default on creating a folder1 will have 6 acl entries . On top of that
> assigning acl on a folder1 exceeds 32 , then unable to assign acls for a
> group/user to folder1.
> {noformat}
> 2014-11-20 18:55:06,553 ERROR [qtp1279235236-17 - /rolexml/role/modrole]
> Error occured while setting permissions for Resource:[
> hdfs://hacluster/folder1 ] and Error message is : Invalid ACL: ACL has 33
> entries, which exceeds maximum of 32.
> at
> org.apache.hadoop.hdfs.server.namenode.AclTransformation.buildAndValidateAcl(AclTransformation.java:274)
> at
> org.apache.hadoop.hdfs.server.namenode.AclTransformation.mergeAclEntries(AclTransformation.java:181)
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.unprotectedModifyAclEntries(FSDirectory.java:2771)
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.modifyAclEntries(FSDirectory.java:2757)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.modifyAclEntries(FSNamesystem.java:7734)
> {noformat}
> Here value 32 is hardcoded , which can be made user configurable.
> {noformat}
> private static List buildAndValidateAcl(ArrayList aclBuilder)
> throws AclException
> {
> if(aclBuilder.size() > 32)
> throw new AclException((new StringBuilder()).append("Invalid ACL:
> ACL has ").append(aclBuilder.size()).append(" entries, which exceeds maximum
> of ").append(32).append(".").toString());
> :
> :
> }
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9393) After choosing favored nodes, choosing nodes for remaining replicas should go through BlockPlacementPolicy
[ https://issues.apache.org/jira/browse/HDFS-9393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] J.Andreina updated HDFS-9393: - Attachment: HDFS-9393.2.patch Updated the patch with latest code. Please review. > After choosing favored nodes, choosing nodes for remaining replicas should go > through BlockPlacementPolicy > -- > > Key: HDFS-9393 > URL: https://issues.apache.org/jira/browse/HDFS-9393 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: J.Andreina >Assignee: J.Andreina > Attachments: HDFS-9393.1.patch, HDFS-9393.2.patch > > > Current Behavior is : > After choosing replicas from passed favored nodes , choosing nodes for > remaining replica does not go through BlockPlacementPolicy. > Hence eventhough there is a local client datanode is available and not passed > as part of favored nodes , probability for choosing local datanode is less. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9393) After choosing favored nodes, choosing nodes for remaining replicas should go through BlockPlacementPolicy
[ https://issues.apache.org/jira/browse/HDFS-9393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15035635#comment-15035635 ] J.Andreina commented on HDFS-9393: -- Testcase failures are unrelated. Please review. > After choosing favored nodes, choosing nodes for remaining replicas should go > through BlockPlacementPolicy > -- > > Key: HDFS-9393 > URL: https://issues.apache.org/jira/browse/HDFS-9393 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: J.Andreina >Assignee: J.Andreina > Attachments: HDFS-9393.1.patch > > > Current Behavior is : > After choosing replicas from passed favored nodes , choosing nodes for > remaining replica does not go through BlockPlacementPolicy. > Hence eventhough there is a local client datanode is available and not passed > as part of favored nodes , probability for choosing local datanode is less. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9495) Data node opens random port for HTTPServer, not configurable
[
https://issues.apache.org/jira/browse/HDFS-9495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15035334#comment-15035334
]
J.Andreina commented on HDFS-9495:
--
Currently in DatanodeHttpServer, we are passing the port as 0 (which means
random port ) for endpoint that HTTP server would listen to.
{code}
HttpServer2.Builder builder = new HttpServer2.Builder()
.setName("datanode")
.setConf(confForInfoServer)
.setACL(new AccessControlList(conf.get(DFS_ADMIN, " ")))
.hostName(getHostnameForSpnegoPrincipal(confForInfoServer))
.addEndpoint(URI.create("http://localhost:0;))
.setFindPort(true);
{code}
We can make it configurable and provide a default port instead of taking up a
random.
Please provide your feedback.
> Data node opens random port for HTTPServer, not configurable
>
>
> Key: HDFS-9495
> URL: https://issues.apache.org/jira/browse/HDFS-9495
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: neha
>
> Data node opens random port for HTTP Server which is not configurable
> currently. Better to make it configurable.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9393) After choosing favored nodes, choosing nodes for remaining replicas should go through BlockPlacementPolicy
[
https://issues.apache.org/jira/browse/HDFS-9393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
J.Andreina updated HDFS-9393:
-
Attachment: HDFS-9393.1.patch
uploaded a patch , where changes are done by passing a fresh {{chosenNodes}} to
choose targets if no enough favored nodes.
This would help in not avoiding to choose local node as a target if not part
of favored nodes (will go through block placement policy if there is no enough
favored nodes).
Please review.
> After choosing favored nodes, choosing nodes for remaining replicas should go
> through BlockPlacementPolicy
> --
>
> Key: HDFS-9393
> URL: https://issues.apache.org/jira/browse/HDFS-9393
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: J.Andreina
>Assignee: J.Andreina
> Attachments: HDFS-9393.1.patch
>
>
> Current Behavior is :
> After choosing replicas from passed favored nodes , choosing nodes for
> remaining replica does not go through BlockPlacementPolicy.
> Hence eventhough there is a local client datanode is available and not passed
> as part of favored nodes , probability for choosing local datanode is less.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9393) After choosing favored nodes, choosing nodes for remaining replicas should go through BlockPlacementPolicy
[ https://issues.apache.org/jira/browse/HDFS-9393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] J.Andreina updated HDFS-9393: - Status: Patch Available (was: Open) > After choosing favored nodes, choosing nodes for remaining replicas should go > through BlockPlacementPolicy > -- > > Key: HDFS-9393 > URL: https://issues.apache.org/jira/browse/HDFS-9393 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: J.Andreina >Assignee: J.Andreina > Attachments: HDFS-9393.1.patch > > > Current Behavior is : > After choosing replicas from passed favored nodes , choosing nodes for > remaining replica does not go through BlockPlacementPolicy. > Hence eventhough there is a local client datanode is available and not passed > as part of favored nodes , probability for choosing local datanode is less. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-9393) After choosing favored nodes, choosing nodes for remaining replicas should go through BlockPlacementPolicy
J.Andreina created HDFS-9393: Summary: After choosing favored nodes, choosing nodes for remaining replicas should go through BlockPlacementPolicy Key: HDFS-9393 URL: https://issues.apache.org/jira/browse/HDFS-9393 Project: Hadoop HDFS Issue Type: Bug Reporter: J.Andreina Assignee: J.Andreina Current Behavior is : After choosing replicas from passed favored nodes , choosing nodes for remaining replica does not go through BlockPlacementPolicy. Hence eventhough there is a local client datanode is available and not passed as part of favored nodes , probability for choosing local datanode is less. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9364) Unnecessary DNS resolution attempts when creating NameNodeProxies
[
https://issues.apache.org/jira/browse/HDFS-9364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14989117#comment-14989117
]
J.Andreina commented on HDFS-9364:
--
bq. I think it's theoretically possible for a misconfigured cluster to reach
there and silently try to resolve in said methodsAnother option is to do a
precondition check here instead of handling it, but I feel that may be too
aggressive... What do you think?
[~xiaochen], As u have mentioned, there is already a check for
failoverProxyProvider as below
{code}
if (failoverProxyProvider == null) {
return createNonHAProxy(conf, DFSUtilClient.getNNAddress(nameNodeUri),
xface, UserGroupInformation.getCurrentUser(), true,
fallbackToSimpleAuth);
}
{code}
Here for any Non-HA cluster/for HA cluster where NameNode URI is configured to
an ip:port(illogical URI), there is no necessity for configuring
failoverProxyProvider(hence failoverProxyProvider would be null )
So i feel, here it is not required to handle cases with misconfigurations :).
> Unnecessary DNS resolution attempts when creating NameNodeProxies
> -
>
> Key: HDFS-9364
> URL: https://issues.apache.org/jira/browse/HDFS-9364
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Xiao Chen
>Assignee: Xiao Chen
> Attachments: HDFS-9364.001.patch, HDFS-9364.002.patch
>
>
> When creating NameNodeProxies, we always try to DNS-resolve namenode URIs.
> This is unnecessary if the URI is logical, and may be significantly slow if
> the DNS is having problems.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9364) Unnecessary DNS resolution attempts when creating NameNodeProxies
[
https://issues.apache.org/jira/browse/HDFS-9364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14988984#comment-14988984
]
J.Andreina commented on HDFS-9364:
--
[~xiaochen] ,thanks for raising this issue.Patch looks good to me.
I have few review comments.
1.
Following place also requires a check on whether NNURI is logical or not.
{{DFSUtilClient#getNNAddress(Configuration conf)}} --> should
internally call {{DFSUtilClient#getNNAddressCheckLogical()}}
2.
As the check on NNURI is logical, is not necessary for Non-HA cluster,in patch
checks at following places is not required
{{NameNodeProxies#createProxy}}
{{NameNodeProxiesClient#createProxyWithClientProtocol}}
> Unnecessary DNS resolution attempts when creating NameNodeProxies
> -
>
> Key: HDFS-9364
> URL: https://issues.apache.org/jira/browse/HDFS-9364
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Xiao Chen
>Assignee: Xiao Chen
> Attachments: HDFS-9364.001.patch
>
>
> When creating NameNodeProxies, we always try to DNS-resolve namenode URIs.
> This is unnecessary if the URI is logical, and may be significantly slow if
> the DNS is having problems.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9219) Even if permission is enabled in an environment, while resolving reserved paths there is no check on permission.
[
https://issues.apache.org/jira/browse/HDFS-9219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
J.Andreina updated HDFS-9219:
-
Attachment: HDFS-9219.3.patch
Updated the patch with GenericTestUtils.assertExceptionContains(..) instead of
using assertTrue(e.getMessage().contains(..))
Failures are unrelated to this patch.
> Even if permission is enabled in an environment, while resolving reserved
> paths there is no check on permission.
>
>
> Key: HDFS-9219
> URL: https://issues.apache.org/jira/browse/HDFS-9219
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: J.Andreina
>Assignee: J.Andreina
> Attachments: HDFS-9219.1.patch, HDFS-9219.2.patch, HDFS-9219.3.patch
>
>
> Currently at few instances , reserved paths are resolved without checking for
> permission, even if "dfs.permissions.enabled" is set to true.
> {code}
> FSPermissionChecker pc = fsd.getPermissionChecker();
> byte[][] pathComponents =
> FSDirectory.getPathComponentsForReservedPath(src);
> INodesInPath iip;
> fsd.writeLock();
> try {
> src = *FSDirectory.resolvePath(src, pathComponents, fsd);*
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9219) Even if permission is enabled in an environment, while resolving reserved paths there is no check on permission.
[
https://issues.apache.org/jira/browse/HDFS-9219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
J.Andreina updated HDFS-9219:
-
Attachment: HDFS-9219.2.patch
Updated the patch , including a testcase.
Please review.
> Even if permission is enabled in an environment, while resolving reserved
> paths there is no check on permission.
>
>
> Key: HDFS-9219
> URL: https://issues.apache.org/jira/browse/HDFS-9219
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: J.Andreina
>Assignee: J.Andreina
> Attachments: HDFS-9219.1.patch, HDFS-9219.2.patch
>
>
> Currently at few instances , reserved paths are resolved without checking for
> permission, even if "dfs.permissions.enabled" is set to true.
> {code}
> FSPermissionChecker pc = fsd.getPermissionChecker();
> byte[][] pathComponents =
> FSDirectory.getPathComponentsForReservedPath(src);
> INodesInPath iip;
> fsd.writeLock();
> try {
> src = *FSDirectory.resolvePath(src, pathComponents, fsd);*
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9322) Keep order of addresses to nameservice mapping from configuration file
[
https://issues.apache.org/jira/browse/HDFS-9322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14980102#comment-14980102
]
J.Andreina commented on HDFS-9322:
--
That was good catch [~ravwojdyla].
At nameservice level order is maintained, but within each nameservice, namenode
id's order is not maintained.
Some nits:
1. Need to name the patch as following.
--.patch
ex: HDFS-1000-trunk-01.patch
2. Add a testcase for HA case in TestGetConf
> Keep order of addresses to nameservice mapping from configuration file
> --
>
> Key: HDFS-9322
> URL: https://issues.apache.org/jira/browse/HDFS-9322
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.7.1
>Reporter: Rafal Wojdyla
>Assignee: Rafal Wojdyla
> Labels: has-patch
> Attachments: HDFS-9322.patch_1
>
>
> getAddressesForNameserviceId which is used by ConfiguredFailoverProxyProvider
> does not keep order of namenodes/addresses from configuration file - instead
> it relays on order given by HashMap (key is service id) which is misaligned
> with comment/doc in ConfiguredFailoverProxyProvider that says:
> {code}
> /**
> * A FailoverProxyProvider implementation which allows one to configure two
> URIs
> * to connect to during fail-over. The first configured address is tried
> first,
> * and on a fail-over event the other address is tried.
> */
> {code}
> One solution is to use LinkedHashMap which is insertion-ordered.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8545) Refactor FS#getUsed() to use ContentSummary and add an API to fetch the total file length from a specific path
[ https://issues.apache.org/jira/browse/HDFS-8545?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] J.Andreina updated HDFS-8545: - Attachment: HDFS-8545.6.patch Thanks [~vinayrpet] for the review comments. I have updated the patch. Please review. > Refactor FS#getUsed() to use ContentSummary and add an API to fetch the total > file length from a specific path > -- > > Key: HDFS-8545 > URL: https://issues.apache.org/jira/browse/HDFS-8545 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: J.Andreina >Assignee: J.Andreina >Priority: Minor > Attachments: HDFS-8545.1.patch, HDFS-8545.2.patch, HDFS-8545.3.patch, > HDFS-8545.4.patch, HDFS-8545.5.patch, HDFS-8545.6.patch > > > Currently by default in FileSystem#getUsed() returns the total file size from > root. > It is good to have an api to return the total file size from specified path > ,same as we specify the path in "./hdfs dfs -du -s /path" . -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-9044) Give Priority to FavouredNodes , before selecting nodes from FavouredNode's Node Group
[
https://issues.apache.org/jira/browse/HDFS-9044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
J.Andreina updated HDFS-9044:
-
Attachment: HDFS-9044.6.patch
Thanks [~vinayrpet] for the review comments.
Updated the patch please review
> Give Priority to FavouredNodes , before selecting nodes from FavouredNode's
> Node Group
> --
>
> Key: HDFS-9044
> URL: https://issues.apache.org/jira/browse/HDFS-9044
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: J.Andreina
>Assignee: J.Andreina
> Attachments: HDFS-9044.1.patch, HDFS-9044.2.patch, HDFS-9044.3.patch,
> HDFS-9044.4.patch, HDFS-9044.5.patch, HDFS-9044.6.patch
>
>
> Passing Favored nodes intention is to place replica among the favored node
> Current behavior in Node group is
> If favored node is not available it goes to one among favored
> nodegroup.
> {noformat}
> Say for example:
> 1)I need 3 replicas and passed 5 favored nodes.
> 2)Out of 5 favored nodes 3 favored nodes are not good.
> 3)Then based on BlockPlacementPolicyWithNodeGroup out of 5 targets node
> returned , 3 will be random node from 3 bad FavoredNode's nodegroup.
> 4)Then there is a probability that all my 3 replicas are placed on
> Random node from FavoredNodes's nodegroup , instead of giving priority to 2
> favored nodes returned as target.
> {noformat}
> *Instead of returning 5 targets on 3rd step above , we can return 2 good
> favored nodes as target*
> *And remaining 1 needed replica can be chosen from Random node of bad
> FavoredNodes's nodegroup.*
> This will make sure that the FavoredNodes are given priority.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8545) Refactor FS#getUsed() to use ContentSummary and add an API to fetch the total file length from a specific path
[ https://issues.apache.org/jira/browse/HDFS-8545?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] J.Andreina updated HDFS-8545: - Attachment: HDFS-8545.7.patch Attaching a patch, fixing checkstyle errors. Please review. > Refactor FS#getUsed() to use ContentSummary and add an API to fetch the total > file length from a specific path > -- > > Key: HDFS-8545 > URL: https://issues.apache.org/jira/browse/HDFS-8545 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: J.Andreina >Assignee: J.Andreina >Priority: Minor > Attachments: HDFS-8545.1.patch, HDFS-8545.2.patch, HDFS-8545.3.patch, > HDFS-8545.4.patch, HDFS-8545.5.patch, HDFS-8545.6.patch, HDFS-8545.7.patch > > > Currently by default in FileSystem#getUsed() returns the total file size from > root. > It is good to have an api to return the total file size from specified path > ,same as we specify the path in "./hdfs dfs -du -s /path" . -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9044) Give Priority to FavouredNodes , before selecting nodes from FavouredNode's Node Group
[
https://issues.apache.org/jira/browse/HDFS-9044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14979772#comment-14979772
]
J.Andreina commented on HDFS-9044:
--
Failures are unrelated to this patch.
Please review.
> Give Priority to FavouredNodes , before selecting nodes from FavouredNode's
> Node Group
> --
>
> Key: HDFS-9044
> URL: https://issues.apache.org/jira/browse/HDFS-9044
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: J.Andreina
>Assignee: J.Andreina
> Attachments: HDFS-9044.1.patch, HDFS-9044.2.patch, HDFS-9044.3.patch,
> HDFS-9044.4.patch, HDFS-9044.5.patch, HDFS-9044.6.patch
>
>
> Passing Favored nodes intention is to place replica among the favored node
> Current behavior in Node group is
> If favored node is not available it goes to one among favored
> nodegroup.
> {noformat}
> Say for example:
> 1)I need 3 replicas and passed 5 favored nodes.
> 2)Out of 5 favored nodes 3 favored nodes are not good.
> 3)Then based on BlockPlacementPolicyWithNodeGroup out of 5 targets node
> returned , 3 will be random node from 3 bad FavoredNode's nodegroup.
> 4)Then there is a probability that all my 3 replicas are placed on
> Random node from FavoredNodes's nodegroup , instead of giving priority to 2
> favored nodes returned as target.
> {noformat}
> *Instead of returning 5 targets on 3rd step above , we can return 2 good
> favored nodes as target*
> *And remaining 1 needed replica can be chosen from Random node of bad
> FavoredNodes's nodegroup.*
> This will make sure that the FavoredNodes are given priority.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9044) Give Priority to FavouredNodes , before selecting nodes from FavouredNode's Node Group
[
https://issues.apache.org/jira/browse/HDFS-9044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
J.Andreina updated HDFS-9044:
-
Attachment: HDFS-9044.6.patch
> Give Priority to FavouredNodes , before selecting nodes from FavouredNode's
> Node Group
> --
>
> Key: HDFS-9044
> URL: https://issues.apache.org/jira/browse/HDFS-9044
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: J.Andreina
>Assignee: J.Andreina
> Attachments: HDFS-9044.1.patch, HDFS-9044.2.patch, HDFS-9044.3.patch,
> HDFS-9044.4.patch, HDFS-9044.5.patch, HDFS-9044.6.patch
>
>
> Passing Favored nodes intention is to place replica among the favored node
> Current behavior in Node group is
> If favored node is not available it goes to one among favored
> nodegroup.
> {noformat}
> Say for example:
> 1)I need 3 replicas and passed 5 favored nodes.
> 2)Out of 5 favored nodes 3 favored nodes are not good.
> 3)Then based on BlockPlacementPolicyWithNodeGroup out of 5 targets node
> returned , 3 will be random node from 3 bad FavoredNode's nodegroup.
> 4)Then there is a probability that all my 3 replicas are placed on
> Random node from FavoredNodes's nodegroup , instead of giving priority to 2
> favored nodes returned as target.
> {noformat}
> *Instead of returning 5 targets on 3rd step above , we can return 2 good
> favored nodes as target*
> *And remaining 1 needed replica can be chosen from Random node of bad
> FavoredNodes's nodegroup.*
> This will make sure that the FavoredNodes are given priority.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9044) Give Priority to FavouredNodes , before selecting nodes from FavouredNode's Node Group
[
https://issues.apache.org/jira/browse/HDFS-9044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
J.Andreina updated HDFS-9044:
-
Attachment: (was: HDFS-9044.6.patch)
> Give Priority to FavouredNodes , before selecting nodes from FavouredNode's
> Node Group
> --
>
> Key: HDFS-9044
> URL: https://issues.apache.org/jira/browse/HDFS-9044
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: J.Andreina
>Assignee: J.Andreina
> Attachments: HDFS-9044.1.patch, HDFS-9044.2.patch, HDFS-9044.3.patch,
> HDFS-9044.4.patch, HDFS-9044.5.patch
>
>
> Passing Favored nodes intention is to place replica among the favored node
> Current behavior in Node group is
> If favored node is not available it goes to one among favored
> nodegroup.
> {noformat}
> Say for example:
> 1)I need 3 replicas and passed 5 favored nodes.
> 2)Out of 5 favored nodes 3 favored nodes are not good.
> 3)Then based on BlockPlacementPolicyWithNodeGroup out of 5 targets node
> returned , 3 will be random node from 3 bad FavoredNode's nodegroup.
> 4)Then there is a probability that all my 3 replicas are placed on
> Random node from FavoredNodes's nodegroup , instead of giving priority to 2
> favored nodes returned as target.
> {noformat}
> *Instead of returning 5 targets on 3rd step above , we can return 2 good
> favored nodes as target*
> *And remaining 1 needed replica can be chosen from Random node of bad
> FavoredNodes's nodegroup.*
> This will make sure that the FavoredNodes are given priority.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8631) WebHDFS : Support list/setQuota
[ https://issues.apache.org/jira/browse/HDFS-8631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14968574#comment-14968574 ] J.Andreina commented on HDFS-8631: -- bq.So I used "9223372036854775807" as default value and this is same as QUOTA_DONT_SET and Long.MAX_VALUE. You are correct [~surendrasingh] , annotation members expect constant expressions. We can ignore review comment #1. > WebHDFS : Support list/setQuota > --- > > Key: HDFS-8631 > URL: https://issues.apache.org/jira/browse/HDFS-8631 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: nijel >Assignee: Surendra Singh Lilhore > Attachments: HDFS-8631-001.patch > > > User is able do quota management from filesystem object. Same operation can > be allowed trough REST API. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9070) Allow fsck display pending replica location information for being-written blocks
[ https://issues.apache.org/jira/browse/HDFS-9070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14966196#comment-14966196 ] J.Andreina commented on HDFS-9070: -- Thanks [~demongaorui]. Updated patch looks good to me. > Allow fsck display pending replica location information for being-written > blocks > > > Key: HDFS-9070 > URL: https://issues.apache.org/jira/browse/HDFS-9070 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: GAO Rui >Assignee: GAO Rui > Attachments: HDFS-9070--HDFS-7285.00.patch, > HDFS-9070-HDFS-7285.00.patch, HDFS-9070-HDFS-7285.01.patch, > HDFS-9070-HDFS-7285.02.patch, HDFS-9070-trunk.03.patch, > HDFS-9070-trunk.04.patch, HDFS-9070-trunk.05.patch, HDFS-9070-trunk.06.patch, > HDFS-9070-trunk.07.patch > > > When a EC file is being written, it can be helpful to allow fsck display > datanode information of the being-written EC file block group. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8631) WebHDFS : Support list/setQuota
[
https://issues.apache.org/jira/browse/HDFS-8631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14963013#comment-14963013
]
J.Andreina commented on HDFS-8631:
--
Thanks [~surendrasingh] for the patch.
It looks great.
I have few comments :
1.
Both in NamespaceQuotaParam and in StoragespaceQuotaParam, can update below as
default value:
{code}
public static final String DEFAULT =
String.valueOf(HdfsConstants.QUOTA_DONT_SET);
instead of
public static final String DEFAULT = String.valueOf(Long.MAX_VALUE);
{code}
2.
Both in NamespaceQuotaParam and in StoragespaceQuotaParam, can update the
constructor code as below for more readability
{code}
super(DOMAIN, value, HdfsConstants.QUOTA_RESET, null);
instead of
super(DOMAIN, value, -1L, null);
{code}
3.
Inorder to make test run faster, start 0 datanodes in both testcases, as there
is no file is been written in tests.
4.
Can update the webhdfs document for supporting list/setQuota.
> WebHDFS : Support list/setQuota
> ---
>
> Key: HDFS-8631
> URL: https://issues.apache.org/jira/browse/HDFS-8631
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: nijel
>Assignee: Surendra Singh Lilhore
> Attachments: HDFS-8631-001.patch
>
>
> User is able do quota management from filesystem object. Same operation can
> be allowed trough REST API.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9234) WebHdfs : getContentSummary() should give quota for storage types
[
https://issues.apache.org/jira/browse/HDFS-9234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14962173#comment-14962173
]
J.Andreina commented on HDFS-9234:
--
Thanks [~surendrasingh] for the patch . It looks overall good.
One small nit:
We can make test run little faster by skipping start of datanode, as we are
not creating any files.
You can start 0 datanode, this will help test run faster.
{code}
try {
cluster = new MiniDFSCluster.Builder(conf).numDataNodes(0).build();
{code}
> WebHdfs : getContentSummary() should give quota for storage types
> -
>
> Key: HDFS-9234
> URL: https://issues.apache.org/jira/browse/HDFS-9234
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: webhdfs
>Affects Versions: 2.7.1
>Reporter: Surendra Singh Lilhore
>Assignee: Surendra Singh Lilhore
> Attachments: HDFS-9234-001.patch, HDFS-9234-002.patch,
> HDFS-9234-003.patch, HDFS-9234-004.patch
>
>
> Currently webhdfs API for ContentSummary give only namequota and spacequota
> but it will not give storage types quota.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8673) HDFS reports file already exists if there is a file/dir name end with ._COPYING_
[ https://issues.apache.org/jira/browse/HDFS-8673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14962168#comment-14962168 ] J.Andreina commented on HDFS-8673: -- I am just worried that, it would break existing behavior of overwriting those temp files, [~airbots]. If everyone agrees, then I am okay for this changes, provided Jira is marked as incompatible change. > HDFS reports file already exists if there is a file/dir name end with > ._COPYING_ > > > Key: HDFS-8673 > URL: https://issues.apache.org/jira/browse/HDFS-8673 > Project: Hadoop HDFS > Issue Type: Bug > Components: fs >Affects Versions: 2.7.0 >Reporter: Chen He >Assignee: Chen He > Attachments: HDFS-8673.000-WIP.patch, HDFS-8673.000.patch, > HDFS-8673.001.patch, HDFS-8673.002.patch, HDFS-8673.003.patch, > HDFS-8673.003.patch > > > Because CLI is using CommandWithDestination.java which add "._COPYING_" to > the tail of file name when it does the copy. It will cause problem if there > is a file/dir already called *._COPYING_ on HDFS. > For file: > -bash-4.1$ hadoop fs -put 5M /user/occ/ > -bash-4.1$ hadoop fs -mv /user/occ/5M /user/occ/5M._COPYING_ > -bash-4.1$ hadoop fs -ls /user/occ/ > Found 1 items > -rw-r--r-- 1 occ supergroup5242880 2015-06-26 05:16 > /user/occ/5M._COPYING_ > -bash-4.1$ hadoop fs -put 128K /user/occ/5M > -bash-4.1$ hadoop fs -ls /user/occ/ > Found 1 items > -rw-r--r-- 1 occ supergroup 131072 2015-06-26 05:19 /user/occ/5M > For dir: > -bash-4.1$ hadoop fs -mkdir /user/occ/5M._COPYING_ > -bash-4.1$ hadoop fs -ls /user/occ/ > Found 1 items > drwxr-xr-x - occ supergroup 0 2015-06-26 05:24 > /user/occ/5M._COPYING_ > -bash-4.1$ hadoop fs -put 128K /user/occ/5M > put: /user/occ/5M._COPYING_ already exists as a directory > -bash-4.1$ hadoop fs -ls /user/occ/ > (/user/occ/5M._COPYING_ is gone) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9070) Allow fsck display pending replica location information for being-written blocks
[
https://issues.apache.org/jira/browse/HDFS-9070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14962176#comment-14962176
]
J.Andreina commented on HDFS-9070:
--
Thanks [~demongaorui] ,recent changes Looks good.
Some nits:
1.
{code}
+} else if (isComplete && corruptReplicas != null
+&& corruptReplicas.contains(dnDesc)) {
+ sb.append("CORRUPT)");
+} else if (isComplete && blocksExcess != null
+&& blocksExcess.contains(storedBlock)) {
+ sb.append("EXCESS)");
{code}
I don't think *isComplete()* check is required in above code .
Block can be CORRUPT or EXCESS, logically possible only if it is complete. So
explicit check might not be required.
2. Test {{testFsckOpenECFiles}} is written for EC files, but changes are not
specific to EC file, and there is nothing asserted against EC related. Hence
when the test is run, it takes lot of time, since it involves 10 DNs.
IMO, its okay to test samething with normal non-EC file.
> Allow fsck display pending replica location information for being-written
> blocks
>
>
> Key: HDFS-9070
> URL: https://issues.apache.org/jira/browse/HDFS-9070
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: GAO Rui
>Assignee: GAO Rui
> Attachments: HDFS-9070--HDFS-7285.00.patch,
> HDFS-9070-HDFS-7285.00.patch, HDFS-9070-HDFS-7285.01.patch,
> HDFS-9070-HDFS-7285.02.patch, HDFS-9070-trunk.03.patch,
> HDFS-9070-trunk.04.patch, HDFS-9070-trunk.05.patch, HDFS-9070-trunk.06.patch
>
>
> When a EC file is being written, it can be helpful to allow fsck display
> datanode information of the being-written EC file block group.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9219) Even if permission is enabled in an environment, while resolving reserved paths there is no check on permission.
[
https://issues.apache.org/jira/browse/HDFS-9219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
J.Andreina updated HDFS-9219:
-
Status: Patch Available (was: Open)
> Even if permission is enabled in an environment, while resolving reserved
> paths there is no check on permission.
>
>
> Key: HDFS-9219
> URL: https://issues.apache.org/jira/browse/HDFS-9219
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: J.Andreina
>Assignee: J.Andreina
> Attachments: HDFS-9219.1.patch
>
>
> Currently at few instances , reserved paths are resolved without checking for
> permission, even if "dfs.permissions.enabled" is set to true.
> {code}
> FSPermissionChecker pc = fsd.getPermissionChecker();
> byte[][] pathComponents =
> FSDirectory.getPathComponentsForReservedPath(src);
> INodesInPath iip;
> fsd.writeLock();
> try {
> src = *FSDirectory.resolvePath(src, pathComponents, fsd);*
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9219) Even if permission is enabled in an environment, while resolving reserved paths there is no check on permission.
[
https://issues.apache.org/jira/browse/HDFS-9219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
J.Andreina updated HDFS-9219:
-
Attachment: HDFS-9219.1.patch
Attached an initial patch .
Please review.
> Even if permission is enabled in an environment, while resolving reserved
> paths there is no check on permission.
>
>
> Key: HDFS-9219
> URL: https://issues.apache.org/jira/browse/HDFS-9219
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: J.Andreina
>Assignee: J.Andreina
> Attachments: HDFS-9219.1.patch
>
>
> Currently at few instances , reserved paths are resolved without checking for
> permission, even if "dfs.permissions.enabled" is set to true.
> {code}
> FSPermissionChecker pc = fsd.getPermissionChecker();
> byte[][] pathComponents =
> FSDirectory.getPathComponentsForReservedPath(src);
> INodesInPath iip;
> fsd.writeLock();
> try {
> src = *FSDirectory.resolvePath(src, pathComponents, fsd);*
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8545) Add an API to fetch the total file length from a specific path, apart from getting by default from root
[ https://issues.apache.org/jira/browse/HDFS-8545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14958576#comment-14958576 ] J.Andreina commented on HDFS-8545: -- bq. On a second thought, along with new overloaded API, how about refactoring current getUsed() also to get the usage from the contentsummary than iterating through the list of files recursively. I agree with you [~vinayrpet] , Updated the patch .please review. > Add an API to fetch the total file length from a specific path, apart from > getting by default from root > --- > > Key: HDFS-8545 > URL: https://issues.apache.org/jira/browse/HDFS-8545 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: J.Andreina >Assignee: J.Andreina >Priority: Minor > Attachments: HDFS-8545.1.patch, HDFS-8545.2.patch, HDFS-8545.3.patch, > HDFS-8545.4.patch, HDFS-8545.5.patch > > > Currently by default in FileSystem#getUsed() returns the total file size from > root. > It is good to have an api to return the total file size from specified path > ,same as we specify the path in "./hdfs dfs -du -s /path" . -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8545) Add an API to fetch the total file length from a specific path, apart from getting by default from root
[ https://issues.apache.org/jira/browse/HDFS-8545?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] J.Andreina updated HDFS-8545: - Attachment: HDFS-8545.5.patch > Add an API to fetch the total file length from a specific path, apart from > getting by default from root > --- > > Key: HDFS-8545 > URL: https://issues.apache.org/jira/browse/HDFS-8545 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: J.Andreina >Assignee: J.Andreina >Priority: Minor > Attachments: HDFS-8545.1.patch, HDFS-8545.2.patch, HDFS-8545.3.patch, > HDFS-8545.4.patch, HDFS-8545.5.patch > > > Currently by default in FileSystem#getUsed() returns the total file size from > root. > It is good to have an api to return the total file size from specified path > ,same as we specify the path in "./hdfs dfs -du -s /path" . -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8545) Refactor FS#getUsed() to use ContentSummary and add an API to fetch the total file length from a specific path
[ https://issues.apache.org/jira/browse/HDFS-8545?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] J.Andreina updated HDFS-8545: - Summary: Refactor FS#getUsed() to use ContentSummary and add an API to fetch the total file length from a specific path (was: Add an API to fetch the total file length from a specific path, apart from getting by default from root) > Refactor FS#getUsed() to use ContentSummary and add an API to fetch the total > file length from a specific path > -- > > Key: HDFS-8545 > URL: https://issues.apache.org/jira/browse/HDFS-8545 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: J.Andreina >Assignee: J.Andreina >Priority: Minor > Attachments: HDFS-8545.1.patch, HDFS-8545.2.patch, HDFS-8545.3.patch, > HDFS-8545.4.patch, HDFS-8545.5.patch > > > Currently by default in FileSystem#getUsed() returns the total file size from > root. > It is good to have an api to return the total file size from specified path > ,same as we specify the path in "./hdfs dfs -du -s /path" . -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-9044) Give Priority to FavouredNodes , before selecting nodes from FavouredNode's Node Group
[
https://issues.apache.org/jira/browse/HDFS-9044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
J.Andreina updated HDFS-9044:
-
Attachment: HDFS-9044.5.patch
Thanks [~vinayrpet] for the review comments.
Have updated the patch. Please review.
> Give Priority to FavouredNodes , before selecting nodes from FavouredNode's
> Node Group
> --
>
> Key: HDFS-9044
> URL: https://issues.apache.org/jira/browse/HDFS-9044
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: J.Andreina
>Assignee: J.Andreina
> Attachments: HDFS-9044.1.patch, HDFS-9044.2.patch, HDFS-9044.3.patch,
> HDFS-9044.4.patch, HDFS-9044.5.patch
>
>
> Passing Favored nodes intention is to place replica among the favored node
> Current behavior in Node group is
> If favored node is not available it goes to one among favored
> nodegroup.
> {noformat}
> Say for example:
> 1)I need 3 replicas and passed 5 favored nodes.
> 2)Out of 5 favored nodes 3 favored nodes are not good.
> 3)Then based on BlockPlacementPolicyWithNodeGroup out of 5 targets node
> returned , 3 will be random node from 3 bad FavoredNode's nodegroup.
> 4)Then there is a probability that all my 3 replicas are placed on
> Random node from FavoredNodes's nodegroup , instead of giving priority to 2
> favored nodes returned as target.
> {noformat}
> *Instead of returning 5 targets on 3rd step above , we can return 2 good
> favored nodes as target*
> *And remaining 1 needed replica can be chosen from Random node of bad
> FavoredNodes's nodegroup.*
> This will make sure that the FavoredNodes are given priority.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9219) Even if permission is enabled in an environment, while resolving reserved paths there is no check on permission.
[
https://issues.apache.org/jira/browse/HDFS-9219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14954480#comment-14954480
]
J.Andreina commented on HDFS-9219:
--
Thanks [~liuml07] for the comments.
Check on SuperUserPrivilege is required even before resolving reserved path.
There are 2 resolvePath methods( static and non-static ) in
FsDirectory#resolvePath(..)
Only in non-static method we have the check on permissions.
Some places invokes static resolvePath method (which does not have a permission
check ) eventhough its required.
> Even if permission is enabled in an environment, while resolving reserved
> paths there is no check on permission.
>
>
> Key: HDFS-9219
> URL: https://issues.apache.org/jira/browse/HDFS-9219
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: J.Andreina
>Assignee: J.Andreina
>
> Currently at few instances , reserved paths are resolved without checking for
> permission, even if "dfs.permissions.enabled" is set to true.
> {code}
> FSPermissionChecker pc = fsd.getPermissionChecker();
> byte[][] pathComponents =
> FSDirectory.getPathComponentsForReservedPath(src);
> INodesInPath iip;
> fsd.writeLock();
> try {
> src = *FSDirectory.resolvePath(src, pathComponents, fsd);*
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9070) Allow fsck display pending replica location information for being-written blocks
[
https://issues.apache.org/jira/browse/HDFS-9070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14950132#comment-14950132
]
J.Andreina commented on HDFS-9070:
--
Hi [~demongaorui] ,thanks for the patch. It looks good overall . I have few
comments.
1.
In last patch , we are printing the under construction block information
only if the user sets "showLocations" to true.
We can remove the check for showLocations in below code, to be consistent
with the behavior in finalized blocks.
{code}
if (!blocks.isLastBlockComplete() && lastBlock != null && showLocations)
{
report.append('\n');
report.append("Under Construction Block:\n");
{code}
2.
There are possibilities that while the last block is under-construction , the
node is getting into decommissioning state.
So the possible replica state , when the block is under-construction could be :
*Decommissioning*
*STALE_NODE*
*STALE_BLOCK_CONTENT*
*LIVE*
So i feel it would be good , if we can show the rackinfo and replica
informations as well.
We can extract the existing finalized-block's code for the check on
(showLocations || showRacks || showReplicaDetails ) to a method and can reuse
the code for under-construction block also.
> Allow fsck display pending replica location information for being-written
> blocks
>
>
> Key: HDFS-9070
> URL: https://issues.apache.org/jira/browse/HDFS-9070
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: GAO Rui
>Assignee: GAO Rui
> Attachments: HDFS-9070--HDFS-7285.00.patch,
> HDFS-9070-HDFS-7285.00.patch, HDFS-9070-HDFS-7285.01.patch,
> HDFS-9070-HDFS-7285.02.patch, HDFS-9070-trunk.03.patch,
> HDFS-9070-trunk.04.patch
>
>
> When a EC file is being written, it can be helpful to allow fsck display
> datanode information of the being-written EC file block group.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9219) Even if permission is enabled in an environment, while resolving reserved paths there is no check on permission.
[
https://issues.apache.org/jira/browse/HDFS-9219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14950148#comment-14950148
]
J.Andreina commented on HDFS-9219:
--
Thanks [~liuml07] for checking this issue.
I forgot to mention the location of the code occurrence in description .
Mentioned code snippet is from FSDirAttrOp#setStoragePolicy(..)
bq. Is checkOwner() called in those cases?
Currently there is no check on Owner , if permission is enabled.
> Even if permission is enabled in an environment, while resolving reserved
> paths there is no check on permission.
>
>
> Key: HDFS-9219
> URL: https://issues.apache.org/jira/browse/HDFS-9219
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: J.Andreina
>Assignee: J.Andreina
>
> Currently at few instances , reserved paths are resolved without checking for
> permission, even if "dfs.permissions.enabled" is set to true.
> {code}
> FSPermissionChecker pc = fsd.getPermissionChecker();
> byte[][] pathComponents =
> FSDirectory.getPathComponentsForReservedPath(src);
> INodesInPath iip;
> fsd.writeLock();
> try {
> src = *FSDirectory.resolvePath(src, pathComponents, fsd);*
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HDFS-9219) Even if permission is enabled in an environment, while resolving reserved paths there is no check on permission.
J.Andreina created HDFS-9219:
Summary: Even if permission is enabled in an environment, while
resolving reserved paths there is no check on permission.
Key: HDFS-9219
URL: https://issues.apache.org/jira/browse/HDFS-9219
Project: Hadoop HDFS
Issue Type: Bug
Reporter: J.Andreina
Assignee: J.Andreina
Currently at few instances , reserved paths are resolved without checking for
permission, even if "dfs.permissions.enabled" is set to true.
{code}
FSPermissionChecker pc = fsd.getPermissionChecker();
byte[][] pathComponents = FSDirectory.getPathComponentsForReservedPath(src);
INodesInPath iip;
fsd.writeLock();
try {
src = *FSDirectory.resolvePath(src, pathComponents, fsd);*
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Moved] (HDFS-9218) BlockPlacementPolicy related logs should contain the details about the filename and blockid
[
https://issues.apache.org/jira/browse/HDFS-9218?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
J.Andreina moved YARN-4244 to HDFS-9218:
Key: HDFS-9218 (was: YARN-4244)
Project: Hadoop HDFS (was: Hadoop YARN)
> BlockPlacementPolicy related logs should contain the details about the
> filename and blockid
> ---
>
> Key: HDFS-9218
> URL: https://issues.apache.org/jira/browse/HDFS-9218
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: J.Andreina
>Assignee: J.Andreina
>
> Currently the user will not get the details about which file/block , the
> BlockPlacementPolicy is not able to find a replica node , if there is a huge
> client write operation is going on.
> For example consider below failure message , which does'nt have details about
> file/block , which will be difficult to track later.
> {noformat}
> final String message =
> "Failed to place enough replicas, still in need of "
> + (totalReplicasExpected - results.size()) + " to reach " +
> totalReplicasExpected
> + " (unavailableStorages=" + unavailableStorages + ",
> storagePolicy="
> + storagePolicy + ", newBlock=" + newBlock + ")";
> String msg = "All required storage types are unavailable: "
> + " unavailableStorages=" + unavailableStorages
> + ", storagePolicy=" + storagePolicy.getName();
> {noformat}
> It is better to provide the file/block information in the logs for better
> debugability .
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9044) Give Priority to FavouredNodes , before selecting nodes from FavouredNode's Node Group
[
https://issues.apache.org/jira/browse/HDFS-9044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14908123#comment-14908123
]
J.Andreina commented on HDFS-9044:
--
[~vinayrpet] and [~szetszwo] , can you please have a look at this jira and
provide your feedback
> Give Priority to FavouredNodes , before selecting nodes from FavouredNode's
> Node Group
> --
>
> Key: HDFS-9044
> URL: https://issues.apache.org/jira/browse/HDFS-9044
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: J.Andreina
>Assignee: J.Andreina
> Attachments: HDFS-9044.1.patch, HDFS-9044.2.patch, HDFS-9044.3.patch,
> HDFS-9044.4.patch
>
>
> Passing Favored nodes intention is to place replica among the favored node
> Current behavior in Node group is
> If favored node is not available it goes to one among favored
> nodegroup.
> {noformat}
> Say for example:
> 1)I need 3 replicas and passed 5 favored nodes.
> 2)Out of 5 favored nodes 3 favored nodes are not good.
> 3)Then based on BlockPlacementPolicyWithNodeGroup out of 5 targets node
> returned , 3 will be random node from 3 bad FavoredNode's nodegroup.
> 4)Then there is a probability that all my 3 replicas are placed on
> Random node from FavoredNodes's nodegroup , instead of giving priority to 2
> favored nodes returned as target.
> {noformat}
> *Instead of returning 5 targets on 3rd step above , we can return 2 good
> favored nodes as target*
> *And remaining 1 needed replica can be chosen from Random node of bad
> FavoredNodes's nodegroup.*
> This will make sure that the FavoredNodes are given priority.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8673) HDFS reports file already exists if there is a file/dir name end with ._COPYING_
[
https://issues.apache.org/jira/browse/HDFS-8673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14907731#comment-14907731
]
J.Andreina commented on HDFS-8673:
--
bq.if there is exception during uploading .copying file, HDFS will remove
failed leftover file.
{noformat}
I agree with you [~airbots] , incase of any exception while writing "File1" ,
then temporary "File1._COPYING_" file will be deleted.
But in case of JVM crash temporary file: "File1._COPYING_" will be retained
in the Filesystem.
{noformat}
> HDFS reports file already exists if there is a file/dir name end with
> ._COPYING_
>
>
> Key: HDFS-8673
> URL: https://issues.apache.org/jira/browse/HDFS-8673
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: fs
>Affects Versions: 2.7.0
>Reporter: Chen He
>Assignee: Chen He
> Attachments: HDFS-8673.000-WIP.patch, HDFS-8673.000.patch,
> HDFS-8673.001.patch, HDFS-8673.002.patch, HDFS-8673.003.patch,
> HDFS-8673.003.patch
>
>
> Because CLI is using CommandWithDestination.java which add "._COPYING_" to
> the tail of file name when it does the copy. It will cause problem if there
> is a file/dir already called *._COPYING_ on HDFS.
> For file:
> -bash-4.1$ hadoop fs -put 5M /user/occ/
> -bash-4.1$ hadoop fs -mv /user/occ/5M /user/occ/5M._COPYING_
> -bash-4.1$ hadoop fs -ls /user/occ/
> Found 1 items
> -rw-r--r-- 1 occ supergroup5242880 2015-06-26 05:16
> /user/occ/5M._COPYING_
> -bash-4.1$ hadoop fs -put 128K /user/occ/5M
> -bash-4.1$ hadoop fs -ls /user/occ/
> Found 1 items
> -rw-r--r-- 1 occ supergroup 131072 2015-06-26 05:19 /user/occ/5M
> For dir:
> -bash-4.1$ hadoop fs -mkdir /user/occ/5M._COPYING_
> -bash-4.1$ hadoop fs -ls /user/occ/
> Found 1 items
> drwxr-xr-x - occ supergroup 0 2015-06-26 05:24
> /user/occ/5M._COPYING_
> -bash-4.1$ hadoop fs -put 128K /user/occ/5M
> put: /user/occ/5M._COPYING_ already exists as a directory
> -bash-4.1$ hadoop fs -ls /user/occ/
> (/user/occ/5M._COPYING_ is gone)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8673) HDFS reports file already exists if there is a file/dir name end with ._COPYING_
[ https://issues.apache.org/jira/browse/HDFS-8673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14906296#comment-14906296 ] J.Andreina commented on HDFS-8673: -- [~airbots], , Thanks for the patch. The patch looks good to me , but i feel the current behavior would be broken. *Before patch :* FileAlreadyExistsException will be thrown , only if "Directory" with the "File1._COPYING_" exist. But not for a file with the same name. *After patch:* Exception will be thrown , for both file and directory. *This is a kind of behavior change for end-users.* *For example:* Say while user is writing a 10GB file ( "File1" ) and if the write operation is interrupted and if "File1._COPYING_" file is retained in Filesystem, then user might re-try to write the same "File1". Write will success , as we overwrite the "File1._COPYING_" *But after patch:* User re-try to write "File1" will fail with exception that the "File1._COPYING_" already exist. [~ste...@apache.org], can you provide your feedback on this / correct me if iam wrong. > HDFS reports file already exists if there is a file/dir name end with > ._COPYING_ > > > Key: HDFS-8673 > URL: https://issues.apache.org/jira/browse/HDFS-8673 > Project: Hadoop HDFS > Issue Type: Bug > Components: fs >Affects Versions: 2.7.0 >Reporter: Chen He >Assignee: Chen He > Attachments: HDFS-8673.000-WIP.patch, HDFS-8673.000.patch, > HDFS-8673.001.patch, HDFS-8673.002.patch, HDFS-8673.003.patch, > HDFS-8673.003.patch > > > Because CLI is using CommandWithDestination.java which add "._COPYING_" to > the tail of file name when it does the copy. It will cause problem if there > is a file/dir already called *._COPYING_ on HDFS. > For file: > -bash-4.1$ hadoop fs -put 5M /user/occ/ > -bash-4.1$ hadoop fs -mv /user/occ/5M /user/occ/5M._COPYING_ > -bash-4.1$ hadoop fs -ls /user/occ/ > Found 1 items > -rw-r--r-- 1 occ supergroup5242880 2015-06-26 05:16 > /user/occ/5M._COPYING_ > -bash-4.1$ hadoop fs -put 128K /user/occ/5M > -bash-4.1$ hadoop fs -ls /user/occ/ > Found 1 items > -rw-r--r-- 1 occ supergroup 131072 2015-06-26 05:19 /user/occ/5M > For dir: > -bash-4.1$ hadoop fs -mkdir /user/occ/5M._COPYING_ > -bash-4.1$ hadoop fs -ls /user/occ/ > Found 1 items > drwxr-xr-x - occ supergroup 0 2015-06-26 05:24 > /user/occ/5M._COPYING_ > -bash-4.1$ hadoop fs -put 128K /user/occ/5M > put: /user/occ/5M._COPYING_ already exists as a directory > -bash-4.1$ hadoop fs -ls /user/occ/ > (/user/occ/5M._COPYING_ is gone) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9044) Give Priority to FavouredNodes , before selecting nodes from FavouredNode's Node Group
[
https://issues.apache.org/jira/browse/HDFS-9044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14904366#comment-14904366
]
J.Andreina commented on HDFS-9044:
--
Checkstyle comments can be ignored , as all the 9 parameters of the method are
been used.
Please review.
> Give Priority to FavouredNodes , before selecting nodes from FavouredNode's
> Node Group
> --
>
> Key: HDFS-9044
> URL: https://issues.apache.org/jira/browse/HDFS-9044
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: J.Andreina
>Assignee: J.Andreina
> Attachments: HDFS-9044.1.patch, HDFS-9044.2.patch, HDFS-9044.3.patch,
> HDFS-9044.4.patch
>
>
> Passing Favored nodes intention is to place replica among the favored node
> Current behavior in Node group is
> If favored node is not available it goes to one among favored
> nodegroup.
> {noformat}
> Say for example:
> 1)I need 3 replicas and passed 5 favored nodes.
> 2)Out of 5 favored nodes 3 favored nodes are not good.
> 3)Then based on BlockPlacementPolicyWithNodeGroup out of 5 targets node
> returned , 3 will be random node from 3 bad FavoredNode's nodegroup.
> 4)Then there is a probability that all my 3 replicas are placed on
> Random node from FavoredNodes's nodegroup , instead of giving priority to 2
> favored nodes returned as target.
> {noformat}
> *Instead of returning 5 targets on 3rd step above , we can return 2 good
> favored nodes as target*
> *And remaining 1 needed replica can be chosen from Random node of bad
> FavoredNodes's nodegroup.*
> This will make sure that the FavoredNodes are given priority.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9071) chooseTargets in ReplicationWork may pass incomplete srcPath
[ https://issues.apache.org/jira/browse/HDFS-9071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14904511#comment-14904511 ] J.Andreina commented on HDFS-9071: -- Thanks [~He Tianyi] for reporting this issue, that was a good catch. bq. I've observed that chooseTargets in ReplicationWork may pass incomplete srcPath (not starting with '/') to block placement policy. I agree . But i doubt , is there any specific reason for going ahead to choose target (in ReplicationWork), after finding that srcpath is invalid/null , in the patch ? > chooseTargets in ReplicationWork may pass incomplete srcPath > > > Key: HDFS-9071 > URL: https://issues.apache.org/jira/browse/HDFS-9071 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.6.0 >Reporter: He Tianyi >Assignee: He Tianyi > Attachments: HDFS-9071.0001.patch > > > I've observed that chooseTargets in ReplicationWork may pass incomplete > srcPath (not starting with '/') to block placement policy. > It is possible that srcPath is extensively used in custom placement policy. > In this case, the incomplete srcPath may further cause AssertionError if try > to get INode with it inside placement policy. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9123) Validation of a path ended with a '/'
[
https://issues.apache.org/jira/browse/HDFS-9123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14904042#comment-14904042
]
J.Andreina commented on HDFS-9123:
--
Thanks [~jojochuang] . That was a good catch. I have few comments .
bq.if the source path is ended with a '/' path separator, the existing
validation for sub-directories fails
As I have verified, issue of copying indefinitely, exist only if src is just
root ('/"). It will not happen for any other paths ending with '/'.
{noformat}
> ./hdfs dfs -cp /abc/ /abc/abcdef
cp: `/abc' to `/abc/abcdef/abc': is a subdirectory of itself
{noformat}
*So, I feel since this is only for src is root, validation can be updated
addressing this special case.*
*I think below code would fix it correctly*
{code}
- String srcPath = src.fs.makeQualified(src.path).toString();
- String dstPath = dst.fs.makeQualified(target.path).toString();
+ Path srcPath = src.fs.makeQualified(src.path);
+ Path dstPath = dst.fs.makeQualified(target.path);
if (dstPath.equals(srcPath)) {
PathIOException e = new PathIOException(src.toString(),
"are identical");
e.setTargetPath(dstPath.toString());
throw e;
}
- if (dstPath.startsWith(srcPath+Path.SEPARATOR)) {
+ if (srcPath.isRoot() || dstPath.toString().startsWith(srcPath.toString()
+ Path.SEPARATOR)) {
{code}
*Also it would be good if you can update the jira title and summary to reflect
the special case*
> Validation of a path ended with a '/'
> -
>
> Key: HDFS-9123
> URL: https://issues.apache.org/jira/browse/HDFS-9123
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: fs
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
>Priority: Minor
> Attachments: HDFS-9123.001.patch, HDFS-9123.002.patch
>
>
> HDFS forbids copying from a directory to its subdirectory (e.g. hdfs dfs -cp
> /abc /abc/xyz) as otherwise it could cause infinite copying (/abc/xyz/xyz,
> /abc/xyz/xyz, /abc/xyz/xyz/xyz,... etc)
> However, if the source path is ended with a '/' path separator, the existing
> validation for sub-directories fails. For example, copying from / to /abc
> would cause infinite copying, until the disk space is filled up.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9044) Give Priority to FavouredNodes , before selecting nodes from FavouredNode's Node Group
[
https://issues.apache.org/jira/browse/HDFS-9044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
J.Andreina updated HDFS-9044:
-
Attachment: HDFS-9044.4.patch
Updated the patch.
Please review.
> Give Priority to FavouredNodes , before selecting nodes from FavouredNode's
> Node Group
> --
>
> Key: HDFS-9044
> URL: https://issues.apache.org/jira/browse/HDFS-9044
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: J.Andreina
>Assignee: J.Andreina
> Attachments: HDFS-9044.1.patch, HDFS-9044.2.patch, HDFS-9044.3.patch,
> HDFS-9044.4.patch
>
>
> Passing Favored nodes intention is to place replica among the favored node
> Current behavior in Node group is
> If favored node is not available it goes to one among favored
> nodegroup.
> {noformat}
> Say for example:
> 1)I need 3 replicas and passed 5 favored nodes.
> 2)Out of 5 favored nodes 3 favored nodes are not good.
> 3)Then based on BlockPlacementPolicyWithNodeGroup out of 5 targets node
> returned , 3 will be random node from 3 bad FavoredNode's nodegroup.
> 4)Then there is a probability that all my 3 replicas are placed on
> Random node from FavoredNodes's nodegroup , instead of giving priority to 2
> favored nodes returned as target.
> {noformat}
> *Instead of returning 5 targets on 3rd step above , we can return 2 good
> favored nodes as target*
> *And remaining 1 needed replica can be chosen from Random node of bad
> FavoredNodes's nodegroup.*
> This will make sure that the FavoredNodes are given priority.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9123) Validation of a path ended with a '/'
[
https://issues.apache.org/jira/browse/HDFS-9123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14904229#comment-14904229
]
J.Andreina commented on HDFS-9123:
--
@Steve Loughran, that was a good point.
In this testcase, test will fail by the next assertion on return code if
exception is not thrown.
{code}
try {
val = shell.run(args1);
} catch (Exception e) {
System.err.println("Exception raised from DFSShell.run " +
e.getLocalizedMessage());
}
assertEquals(1, val);
{code}
And when i checked , other similar tests also uses the same pattern in this
test class.
> Validation of a path ended with a '/'
> -
>
> Key: HDFS-9123
> URL: https://issues.apache.org/jira/browse/HDFS-9123
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: fs
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
>Priority: Minor
> Attachments: HDFS-9123.001.patch, HDFS-9123.002.patch
>
>
> HDFS forbids copying from a directory to its subdirectory (e.g. hdfs dfs -cp
> /abc /abc/xyz) as otherwise it could cause infinite copying (/abc/xyz/xyz,
> /abc/xyz/xyz, /abc/xyz/xyz/xyz,... etc)
> However, if the source path is ended with a '/' path separator, the existing
> validation for sub-directories fails. For example, copying from / to /abc
> would cause infinite copying, until the disk space is filled up.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9044) Give Priority to FavouredNodes , before selecting nodes from FavouredNode's Node Group
[
https://issues.apache.org/jira/browse/HDFS-9044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
J.Andreina updated HDFS-9044:
-
Attachment: HDFS-9044.3.patch
Fixed checkstyle and findbug errors.
Updated the patch.
Please review.
> Give Priority to FavouredNodes , before selecting nodes from FavouredNode's
> Node Group
> --
>
> Key: HDFS-9044
> URL: https://issues.apache.org/jira/browse/HDFS-9044
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: J.Andreina
>Assignee: J.Andreina
> Attachments: HDFS-9044.1.patch, HDFS-9044.2.patch, HDFS-9044.3.patch
>
>
> Passing Favored nodes intention is to place replica among the favored node
> Current behavior in Node group is
> If favored node is not available it goes to one among favored
> nodegroup.
> {noformat}
> Say for example:
> 1)I need 3 replicas and passed 5 favored nodes.
> 2)Out of 5 favored nodes 3 favored nodes are not good.
> 3)Then based on BlockPlacementPolicyWithNodeGroup out of 5 targets node
> returned , 3 will be random node from 3 bad FavoredNode's nodegroup.
> 4)Then there is a probability that all my 3 replicas are placed on
> Random node from FavoredNodes's nodegroup , instead of giving priority to 2
> favored nodes returned as target.
> {noformat}
> *Instead of returning 5 targets on 3rd step above , we can return 2 good
> favored nodes as target*
> *And remaining 1 needed replica can be chosen from Random node of bad
> FavoredNodes's nodegroup.*
> This will make sure that the FavoredNodes are given priority.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9044) Give Priority to FavouredNodes , before selecting nodes from FavouredNode's Node Group
[
https://issues.apache.org/jira/browse/HDFS-9044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14902475#comment-14902475
]
J.Andreina commented on HDFS-9044:
--
Testcase failures are unrelated.
Checkstyle comments can be ignored , as all the 9 parameters of the method are
been used.
Please review.
> Give Priority to FavouredNodes , before selecting nodes from FavouredNode's
> Node Group
> --
>
> Key: HDFS-9044
> URL: https://issues.apache.org/jira/browse/HDFS-9044
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: J.Andreina
>Assignee: J.Andreina
> Attachments: HDFS-9044.1.patch, HDFS-9044.2.patch, HDFS-9044.3.patch
>
>
> Passing Favored nodes intention is to place replica among the favored node
> Current behavior in Node group is
> If favored node is not available it goes to one among favored
> nodegroup.
> {noformat}
> Say for example:
> 1)I need 3 replicas and passed 5 favored nodes.
> 2)Out of 5 favored nodes 3 favored nodes are not good.
> 3)Then based on BlockPlacementPolicyWithNodeGroup out of 5 targets node
> returned , 3 will be random node from 3 bad FavoredNode's nodegroup.
> 4)Then there is a probability that all my 3 replicas are placed on
> Random node from FavoredNodes's nodegroup , instead of giving priority to 2
> favored nodes returned as target.
> {noformat}
> *Instead of returning 5 targets on 3rd step above , we can return 2 good
> favored nodes as target*
> *And remaining 1 needed replica can be chosen from Random node of bad
> FavoredNodes's nodegroup.*
> This will make sure that the FavoredNodes are given priority.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9044) Give Priority to FavouredNodes , before selecting nodes from FavouredNode's Node Group
[
https://issues.apache.org/jira/browse/HDFS-9044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
J.Andreina updated HDFS-9044:
-
Attachment: HDFS-9044.2.patch
Updated the patch fixing the checkstyle and javadoc warnings.
Testcase failures are unrelated.
Please review.
> Give Priority to FavouredNodes , before selecting nodes from FavouredNode's
> Node Group
> --
>
> Key: HDFS-9044
> URL: https://issues.apache.org/jira/browse/HDFS-9044
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: J.Andreina
>Assignee: J.Andreina
> Attachments: HDFS-9044.1.patch, HDFS-9044.2.patch
>
>
> Passing Favored nodes intention is to place replica among the favored node
> Current behavior in Node group is
> If favored node is not available it goes to one among favored
> nodegroup.
> {noformat}
> Say for example:
> 1)I need 3 replicas and passed 5 favored nodes.
> 2)Out of 5 favored nodes 3 favored nodes are not good.
> 3)Then based on BlockPlacementPolicyWithNodeGroup out of 5 targets node
> returned , 3 will be random node from 3 bad FavoredNode's nodegroup.
> 4)Then there is a probability that all my 3 replicas are placed on
> Random node from FavoredNodes's nodegroup , instead of giving priority to 2
> favored nodes returned as target.
> {noformat}
> *Instead of returning 5 targets on 3rd step above , we can return 2 good
> favored nodes as target*
> *And remaining 1 needed replica can be chosen from Random node of bad
> FavoredNodes's nodegroup.*
> This will make sure that the FavoredNodes are given priority.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9044) Give Priority to FavouredNodes , before selecting nodes from FavouredNode's Node Group
[
https://issues.apache.org/jira/browse/HDFS-9044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
J.Andreina updated HDFS-9044:
-
Status: Patch Available (was: Open)
> Give Priority to FavouredNodes , before selecting nodes from FavouredNode's
> Node Group
> --
>
> Key: HDFS-9044
> URL: https://issues.apache.org/jira/browse/HDFS-9044
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: J.Andreina
>Assignee: J.Andreina
> Attachments: HDFS-9044.1.patch
>
>
> Passing Favored nodes intention is to place replica among the favored node
> Current behavior in Node group is
> If favored node is not available it goes to one among favored
> nodegroup.
> {noformat}
> Say for example:
> 1)I need 3 replicas and passed 5 favored nodes.
> 2)Out of 5 favored nodes 3 favored nodes are not good.
> 3)Then based on BlockPlacementPolicyWithNodeGroup out of 5 targets node
> returned , 3 will be random node from 3 bad FavoredNode's nodegroup.
> 4)Then there is a probability that all my 3 replicas are placed on
> Random node from FavoredNodes's nodegroup , instead of giving priority to 2
> favored nodes returned as target.
> {noformat}
> *Instead of returning 5 targets on 3rd step above , we can return 2 good
> favored nodes as target*
> *And remaining 1 needed replica can be chosen from Random node of bad
> FavoredNodes's nodegroup.*
> This will make sure that the FavoredNodes are given priority.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9044) Give Priority to FavouredNodes , before selecting nodes from FavouredNode's Node Group
[
https://issues.apache.org/jira/browse/HDFS-9044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
J.Andreina updated HDFS-9044:
-
Attachment: HDFS-9044.1.patch
Updated an initial patch ( for choosing good favored nodes as target with
higher priority , before choosing bad FavoredNode's node group)
Please review the patch and give your feedback.
> Give Priority to FavouredNodes , before selecting nodes from FavouredNode's
> Node Group
> --
>
> Key: HDFS-9044
> URL: https://issues.apache.org/jira/browse/HDFS-9044
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: J.Andreina
>Assignee: J.Andreina
> Attachments: HDFS-9044.1.patch
>
>
> Passing Favored nodes intention is to place replica among the favored node
> Current behavior in Node group is
> If favored node is not available it goes to one among favored
> nodegroup.
> {noformat}
> Say for example:
> 1)I need 3 replicas and passed 5 favored nodes.
> 2)Out of 5 favored nodes 3 favored nodes are not good.
> 3)Then based on BlockPlacementPolicyWithNodeGroup out of 5 targets node
> returned , 3 will be random node from 3 bad FavoredNode's nodegroup.
> 4)Then there is a probability that all my 3 replicas are placed on
> Random node from FavoredNodes's nodegroup , instead of giving priority to 2
> favored nodes returned as target.
> {noformat}
> *Instead of returning 5 targets on 3rd step above , we can return 2 good
> favored nodes as target*
> *And remaining 1 needed replica can be chosen from Random node of bad
> FavoredNodes's nodegroup.*
> This will make sure that the FavoredNodes are given priority.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9073) Fix failures in TestLazyPersistLockedMemory#testReleaseOnEviction
[
https://issues.apache.org/jira/browse/HDFS-9073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14746939#comment-14746939
]
J.Andreina commented on HDFS-9073:
--
Testcase failures are unrelated.
Please review.
> Fix failures in TestLazyPersistLockedMemory#testReleaseOnEviction
> -
>
> Key: HDFS-9073
> URL: https://issues.apache.org/jira/browse/HDFS-9073
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Affects Versions: 3.0.0
> Environment: jenkins
>Reporter: J.Andreina
>Assignee: J.Andreina
>Priority: Critical
> Attachments: HDFS-9073.1.patch, HDFS-9073.2.patch
>
>
> Testcase ,TestLazyPersistLockedMemory#testReleaseOnEviction fails with below
> exception
> {noformat}
> java.lang.AssertionError: expected:<1> but was:<0>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at org.junit.Assert.assertEquals(Assert.java:542)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.LazyPersistTestCase.verifyRamDiskJMXMetric(LazyPersistTestCase.java:483)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistLockedMemory.testReleaseOnEviction(TestLazyPersistLockedMemory.java:122)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9072) Fix random failures in TestJMXGet
[ https://issues.apache.org/jira/browse/HDFS-9072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14746937#comment-14746937 ] J.Andreina commented on HDFS-9072: -- Testcases failures in TestLazyPersistLockedMemory , TestLazyPersistReplicaPlacement class will be fixed in HDFS-9073. Rest of the testcase failures are unrelated. Please review. > Fix random failures in TestJMXGet > - > > Key: HDFS-9072 > URL: https://issues.apache.org/jira/browse/HDFS-9072 > Project: Hadoop HDFS > Issue Type: Bug > Components: test >Affects Versions: 3.0.0 >Reporter: J.Andreina >Assignee: J.Andreina >Priority: Critical > Attachments: HDFS-9072-01.patch > > > There is a slight delay in updating the jmx values after HDFS operations. But > assertions are done even that . Hence following testcases fails. > TestJMXGet#testNameNode() > TestJMXGet#testDataNode() -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9073) Fix failures in TestLazyPersistLockedMemory#testReleaseOnEviction
[
https://issues.apache.org/jira/browse/HDFS-9073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14747196#comment-14747196
]
J.Andreina commented on HDFS-9073:
--
Thanks [~ste...@apache.org] for the commit
> Fix failures in TestLazyPersistLockedMemory#testReleaseOnEviction
> -
>
> Key: HDFS-9073
> URL: https://issues.apache.org/jira/browse/HDFS-9073
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Affects Versions: 3.0.0
> Environment: jenkins
>Reporter: J.Andreina
>Assignee: J.Andreina
>Priority: Critical
> Fix For: 2.8.0
>
> Attachments: HDFS-9073.1.patch, HDFS-9073.2.patch
>
>
> Testcase ,TestLazyPersistLockedMemory#testReleaseOnEviction fails with below
> exception
> {noformat}
> java.lang.AssertionError: expected:<1> but was:<0>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at org.junit.Assert.assertEquals(Assert.java:542)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.LazyPersistTestCase.verifyRamDiskJMXMetric(LazyPersistTestCase.java:483)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistLockedMemory.testReleaseOnEviction(TestLazyPersistLockedMemory.java:122)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9044) Give Priority to FavouredNodes , before selecting nodes from FavouredNode's Node Group
[
https://issues.apache.org/jira/browse/HDFS-9044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
J.Andreina updated HDFS-9044:
-
Description:
Passing Favored nodes intention is to place replica among the favored node
Current behavior in Node group is
If favored node is not available it goes to one among favored
nodegroup.
{noformat}
Say for example:
1)I need 3 replicas and passed 5 favored nodes.
2)Out of 5 favored nodes 3 favored nodes are not good.
3)Then based on BlockPlacementPolicyWithNodeGroup out of 5 targets node
returned , 3 will be random node from 3 bad FavoredNode's nodegroup.
4)Then there is a probability that all my 3 replicas are placed on
Random node from FavoredNodes's nodegroup , instead of giving priority to 2
favored nodes returned as target.
{noformat}
*Instead of returning 5 targets on 3rd step above , we can return 2 good
favored nodes as target*
*And remaining 1 needed replica can be chosen from Random node of bad
FavoredNodes's nodegroup.*
This will make sure that the FavoredNodes are given priority.
was:
Current behavior:
While choosing targets for replicas if favored nodes are not available , then
it goes for choosing the random node from the favored node group instead of
falling back to regular BlockPlacementPolicyWithNodeGroup
> Give Priority to FavouredNodes , before selecting nodes from FavouredNode's
> Node Group
> --
>
> Key: HDFS-9044
> URL: https://issues.apache.org/jira/browse/HDFS-9044
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: J.Andreina
>Assignee: J.Andreina
>
> Passing Favored nodes intention is to place replica among the favored node
> Current behavior in Node group is
> If favored node is not available it goes to one among favored
> nodegroup.
> {noformat}
> Say for example:
> 1)I need 3 replicas and passed 5 favored nodes.
> 2)Out of 5 favored nodes 3 favored nodes are not good.
> 3)Then based on BlockPlacementPolicyWithNodeGroup out of 5 targets node
> returned , 3 will be random node from 3 bad FavoredNode's nodegroup.
> 4)Then there is a probability that all my 3 replicas are placed on
> Random node from FavoredNodes's nodegroup , instead of giving priority to 2
> favored nodes returned as target.
> {noformat}
> *Instead of returning 5 targets on 3rd step above , we can return 2 good
> favored nodes as target*
> *And remaining 1 needed replica can be chosen from Random node of bad
> FavoredNodes's nodegroup.*
> This will make sure that the FavoredNodes are given priority.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9072) Fix random failures in TestJMXGet
[ https://issues.apache.org/jira/browse/HDFS-9072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14747198#comment-14747198 ] J.Andreina commented on HDFS-9072: -- Thanks [~ste...@apache.org] for the commit > Fix random failures in TestJMXGet > - > > Key: HDFS-9072 > URL: https://issues.apache.org/jira/browse/HDFS-9072 > Project: Hadoop HDFS > Issue Type: Bug > Components: test >Affects Versions: 3.0.0, 2.8.0 >Reporter: J.Andreina >Assignee: J.Andreina >Priority: Critical > Fix For: 2.8.0 > > Attachments: HDFS-9072-01.patch > > > There is a slight delay in updating the jmx values after HDFS operations. But > assertions are done even that . Hence following testcases fails. > TestJMXGet#testNameNode() > TestJMXGet#testDataNode() -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-9044) Give Priority to FavouredNodes , before selecting nodes from FavouredNode's Node Group
[ https://issues.apache.org/jira/browse/HDFS-9044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] J.Andreina updated HDFS-9044: - Summary: Give Priority to FavouredNodes , before selecting nodes from FavouredNode's Node Group (was: If favored nodes are not available, then should fall back to regular BlockPlacementPolicyWithNodeGroup) > Give Priority to FavouredNodes , before selecting nodes from FavouredNode's > Node Group > -- > > Key: HDFS-9044 > URL: https://issues.apache.org/jira/browse/HDFS-9044 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: J.Andreina >Assignee: J.Andreina > > Current behavior: > While choosing targets for replicas if favored nodes are not available , then > it goes for choosing the random node from the favored node group instead of > falling back to regular BlockPlacementPolicyWithNodeGroup -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-9074) Fix failures in TestLazyPersistReplicaPlacement
[
https://issues.apache.org/jira/browse/HDFS-9074?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
J.Andreina updated HDFS-9074:
-
Resolution: Implemented
Status: Resolved (was: Patch Available)
Fixed as part of HDFS-9073. Hence resolving this issue.
> Fix failures in TestLazyPersistReplicaPlacement
> ---
>
> Key: HDFS-9074
> URL: https://issues.apache.org/jira/browse/HDFS-9074
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Affects Versions: 3.0.0
> Environment: jenkins
>Reporter: J.Andreina
>Assignee: J.Andreina
>Priority: Critical
> Attachments: HDFS-9074.1.patch
>
>
> Testcases TestLazyPersistReplicaPlacement#testSynchronousEviction,
> TestLazyPersistReplicaPlacement#testFallbackToDiskFull fails with below
> excepiton
> {noformat}
> java.lang.AssertionError: expected:<1> but was:<0>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at org.junit.Assert.assertEquals(Assert.java:542)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.LazyPersistTestCase.verifyRamDiskJMXMetric(LazyPersistTestCase.java:483)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaPlacement.testSynchronousEviction(TestLazyPersistReplicaPlacement.java:92)
> {noformat}
> {noformat}
> java.lang.AssertionError: expected:<1> but was:<0>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at org.junit.Assert.assertEquals(Assert.java:542)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.LazyPersistTestCase.verifyRamDiskJMXMetric(LazyPersistTestCase.java:483)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaPlacement.testFallbackToDiskFull(TestLazyPersistReplicaPlacement.java:107)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9072) Fix random failures in TestJMXGet
[ https://issues.apache.org/jira/browse/HDFS-9072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] J.Andreina updated HDFS-9072: - Attachment: HDFS-9072-01.patch Updated the patch. Please review. > Fix random failures in TestJMXGet > - > > Key: HDFS-9072 > URL: https://issues.apache.org/jira/browse/HDFS-9072 > Project: Hadoop HDFS > Issue Type: Bug > Components: test >Affects Versions: 3.0.0 >Reporter: J.Andreina >Assignee: J.Andreina >Priority: Critical > Attachments: HDFS-9072-01.patch > > > There is a slight delay in updating the jmx values after HDFS operations. But > assertions are done even that . Hence following testcases fails. > TestJMXGet#testNameNode() > TestJMXGet#testDataNode() -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-9072) Fix random failures in TestJMXGet
[ https://issues.apache.org/jira/browse/HDFS-9072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] J.Andreina updated HDFS-9072: - Status: Patch Available (was: Open) > Fix random failures in TestJMXGet > - > > Key: HDFS-9072 > URL: https://issues.apache.org/jira/browse/HDFS-9072 > Project: Hadoop HDFS > Issue Type: Bug > Components: test >Affects Versions: 3.0.0 >Reporter: J.Andreina >Assignee: J.Andreina >Priority: Critical > Attachments: HDFS-9072-01.patch > > > There is a slight delay in updating the jmx values after HDFS operations. But > assertions are done even that . Hence following testcases fails. > TestJMXGet#testNameNode() > TestJMXGet#testDataNode() -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-9073) Fix failures in TestLazyPersistLockedMemory#testReleaseOnEviction
[
https://issues.apache.org/jira/browse/HDFS-9073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
J.Andreina updated HDFS-9073:
-
Attachment: HDFS-9073.1.patch
Thanks [~vinayrpet]
Updated the patch , after the fix of HDFS-9067.
Please review.
> Fix failures in TestLazyPersistLockedMemory#testReleaseOnEviction
> -
>
> Key: HDFS-9073
> URL: https://issues.apache.org/jira/browse/HDFS-9073
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Affects Versions: 3.0.0
> Environment: jenkins
>Reporter: J.Andreina
>Assignee: J.Andreina
>Priority: Critical
> Attachments: HDFS-9073.1.patch
>
>
> Testcase ,TestLazyPersistLockedMemory#testReleaseOnEviction fails with below
> exception
> {noformat}
> java.lang.AssertionError: expected:<1> but was:<0>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at org.junit.Assert.assertEquals(Assert.java:542)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.LazyPersistTestCase.verifyRamDiskJMXMetric(LazyPersistTestCase.java:483)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistLockedMemory.testReleaseOnEviction(TestLazyPersistLockedMemory.java:122)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9073) Fix failures in TestLazyPersistLockedMemory#testReleaseOnEviction
[
https://issues.apache.org/jira/browse/HDFS-9073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
J.Andreina updated HDFS-9073:
-
Attachment: HDFS-9073.2.patch
Updated the patch.
Please review.
> Fix failures in TestLazyPersistLockedMemory#testReleaseOnEviction
> -
>
> Key: HDFS-9073
> URL: https://issues.apache.org/jira/browse/HDFS-9073
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Affects Versions: 3.0.0
> Environment: jenkins
>Reporter: J.Andreina
>Assignee: J.Andreina
>Priority: Critical
> Attachments: HDFS-9073.1.patch, HDFS-9073.2.patch
>
>
> Testcase ,TestLazyPersistLockedMemory#testReleaseOnEviction fails with below
> exception
> {noformat}
> java.lang.AssertionError: expected:<1> but was:<0>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at org.junit.Assert.assertEquals(Assert.java:542)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.LazyPersistTestCase.verifyRamDiskJMXMetric(LazyPersistTestCase.java:483)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistLockedMemory.testReleaseOnEviction(TestLazyPersistLockedMemory.java:122)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9073) Fix failures in TestLazyPersistLockedMemory#testReleaseOnEviction
[
https://issues.apache.org/jira/browse/HDFS-9073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
J.Andreina updated HDFS-9073:
-
Status: Patch Available (was: Reopened)
> Fix failures in TestLazyPersistLockedMemory#testReleaseOnEviction
> -
>
> Key: HDFS-9073
> URL: https://issues.apache.org/jira/browse/HDFS-9073
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Affects Versions: 3.0.0
> Environment: jenkins
>Reporter: J.Andreina
>Assignee: J.Andreina
>Priority: Critical
> Attachments: HDFS-9073.1.patch
>
>
> Testcase ,TestLazyPersistLockedMemory#testReleaseOnEviction fails with below
> exception
> {noformat}
> java.lang.AssertionError: expected:<1> but was:<0>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at org.junit.Assert.assertEquals(Assert.java:542)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.LazyPersistTestCase.verifyRamDiskJMXMetric(LazyPersistTestCase.java:483)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistLockedMemory.testReleaseOnEviction(TestLazyPersistLockedMemory.java:122)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9074) Fix failures in TestLazyPersistReplicaPlacement
[
https://issues.apache.org/jira/browse/HDFS-9074?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
J.Andreina updated HDFS-9074:
-
Status: Patch Available (was: Open)
> Fix failures in TestLazyPersistReplicaPlacement
> ---
>
> Key: HDFS-9074
> URL: https://issues.apache.org/jira/browse/HDFS-9074
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Affects Versions: 3.0.0
> Environment: jenkins
>Reporter: J.Andreina
>Assignee: J.Andreina
>Priority: Critical
> Attachments: HDFS-9074.1.patch
>
>
> Testcases TestLazyPersistReplicaPlacement#testSynchronousEviction,
> TestLazyPersistReplicaPlacement#testFallbackToDiskFull fails with below
> excepiton
> {noformat}
> java.lang.AssertionError: expected:<1> but was:<0>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at org.junit.Assert.assertEquals(Assert.java:542)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.LazyPersistTestCase.verifyRamDiskJMXMetric(LazyPersistTestCase.java:483)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaPlacement.testSynchronousEviction(TestLazyPersistReplicaPlacement.java:92)
> {noformat}
> {noformat}
> java.lang.AssertionError: expected:<1> but was:<0>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at org.junit.Assert.assertEquals(Assert.java:542)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.LazyPersistTestCase.verifyRamDiskJMXMetric(LazyPersistTestCase.java:483)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaPlacement.testFallbackToDiskFull(TestLazyPersistReplicaPlacement.java:107)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9074) Fix failures in TestLazyPersistReplicaPlacement
[
https://issues.apache.org/jira/browse/HDFS-9074?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
J.Andreina updated HDFS-9074:
-
Attachment: HDFS-9074.1.patch
Thanks [~vinayrpet]
Updated the patch , after the fix of HDFS-9067.
Please review.
> Fix failures in TestLazyPersistReplicaPlacement
> ---
>
> Key: HDFS-9074
> URL: https://issues.apache.org/jira/browse/HDFS-9074
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Affects Versions: 3.0.0
> Environment: jenkins
>Reporter: J.Andreina
>Assignee: J.Andreina
>Priority: Critical
> Attachments: HDFS-9074.1.patch
>
>
> Testcases TestLazyPersistReplicaPlacement#testSynchronousEviction,
> TestLazyPersistReplicaPlacement#testFallbackToDiskFull fails with below
> excepiton
> {noformat}
> java.lang.AssertionError: expected:<1> but was:<0>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at org.junit.Assert.assertEquals(Assert.java:542)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.LazyPersistTestCase.verifyRamDiskJMXMetric(LazyPersistTestCase.java:483)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaPlacement.testSynchronousEviction(TestLazyPersistReplicaPlacement.java:92)
> {noformat}
> {noformat}
> java.lang.AssertionError: expected:<1> but was:<0>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at org.junit.Assert.assertEquals(Assert.java:542)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.LazyPersistTestCase.verifyRamDiskJMXMetric(LazyPersistTestCase.java:483)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaPlacement.testFallbackToDiskFull(TestLazyPersistReplicaPlacement.java:107)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9043) Doc updation for commands in HDFS Federation
[
https://issues.apache.org/jira/browse/HDFS-9043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14742983#comment-14742983
]
J.Andreina commented on HDFS-9043:
--
Thanks [~aw] for the comments.
bq. The given trunk patch appears to be missing the same fixes as the branch-2
patch
In trunk already command is updated correctly using hdfs as :
{noformat}
[hdfs]$ $HADOOP_PREFIX/bin/hdfs --daemon start balancer [-policy ]
{noformat}
bq. Also, if we're going to bother with this, let's make it complete: all of
the sbin commands should get replaced with their equivalent bin versions since
sbin is mostly deprecated.
Currently the sbin commands used are :
{noformat}
sbin/refresh-namenodes.sh
sbin/distribute-exclude.sh
sbin/stop-dfs.sh
sbin/start-dfs.sh
{noformat}
And for above commands there is no alternative yet.
Please correct me if iam wrong .
> Doc updation for commands in HDFS Federation
>
>
> Key: HDFS-9043
> URL: https://issues.apache.org/jira/browse/HDFS-9043
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: documentation
>Reporter: J.Andreina
>Assignee: J.Andreina
>Priority: Minor
> Attachments: HDFS-9043-1.patch, HDFS-9043-branch-2-1.patch,
> HDFS-9043-branch-2.7.0-1.patch
>
>
> 1. command is wrong
> {noformat}
> $HADOOP_PREFIX/bin/hdfs dfsadmin -refreshNameNode
> :
> {noformat}
> Correct command is : hdfs dfsadmin -refreshNameNode's'
> 2.command is wrong
> {noformat}
> $HADOOP_PREFIX/sbin/hadoop-daemon.sh --config $HADOOP_CONF_DIR --script
> $HADOOP_PREFIX/bin/hdfs start balancer
> {noformat}
> Correct command is : *start-balancer.sh -policy*
> 3. Reference link to balancer for further details is wrong
> {noformat}
> Note that Balancer only balances the data and does not balance the namespace.
> For the complete command usage, see balancer.
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9074) Fix failures in TestLazyPersistReplicaPlacement
[
https://issues.apache.org/jira/browse/HDFS-9074?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14743226#comment-14743226
]
J.Andreina commented on HDFS-9074:
--
Thanks [~ste...@apache.org] for looking at this issue.
> Fix failures in TestLazyPersistReplicaPlacement
> ---
>
> Key: HDFS-9074
> URL: https://issues.apache.org/jira/browse/HDFS-9074
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Affects Versions: 3.0.0
> Environment: jenkins
>Reporter: J.Andreina
>Assignee: J.Andreina
>Priority: Critical
>
> Testcases TestLazyPersistReplicaPlacement#testSynchronousEviction,
> TestLazyPersistReplicaPlacement#testFallbackToDiskFull fails with below
> excepiton
> {noformat}
> java.lang.AssertionError: expected:<1> but was:<0>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at org.junit.Assert.assertEquals(Assert.java:542)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.LazyPersistTestCase.verifyRamDiskJMXMetric(LazyPersistTestCase.java:483)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaPlacement.testSynchronousEviction(TestLazyPersistReplicaPlacement.java:92)
> {noformat}
> {noformat}
> java.lang.AssertionError: expected:<1> but was:<0>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at org.junit.Assert.assertEquals(Assert.java:542)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.LazyPersistTestCase.verifyRamDiskJMXMetric(LazyPersistTestCase.java:483)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaPlacement.testFallbackToDiskFull(TestLazyPersistReplicaPlacement.java:107)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9074) Fix failures in TestLazyPersistReplicaPlacement
[
https://issues.apache.org/jira/browse/HDFS-9074?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14743228#comment-14743228
]
J.Andreina commented on HDFS-9074:
--
Current patch Updated in HDFS-9067 seems to fix all jmx related issues. Locally
i verified the same.
Ill resolve this issue.
> Fix failures in TestLazyPersistReplicaPlacement
> ---
>
> Key: HDFS-9074
> URL: https://issues.apache.org/jira/browse/HDFS-9074
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Affects Versions: 3.0.0
> Environment: jenkins
>Reporter: J.Andreina
>Assignee: J.Andreina
>Priority: Critical
>
> Testcases TestLazyPersistReplicaPlacement#testSynchronousEviction,
> TestLazyPersistReplicaPlacement#testFallbackToDiskFull fails with below
> excepiton
> {noformat}
> java.lang.AssertionError: expected:<1> but was:<0>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at org.junit.Assert.assertEquals(Assert.java:542)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.LazyPersistTestCase.verifyRamDiskJMXMetric(LazyPersistTestCase.java:483)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaPlacement.testSynchronousEviction(TestLazyPersistReplicaPlacement.java:92)
> {noformat}
> {noformat}
> java.lang.AssertionError: expected:<1> but was:<0>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at org.junit.Assert.assertEquals(Assert.java:542)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.LazyPersistTestCase.verifyRamDiskJMXMetric(LazyPersistTestCase.java:483)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaPlacement.testFallbackToDiskFull(TestLazyPersistReplicaPlacement.java:107)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HDFS-9072) Fix random failures in TestJMXGet
J.Andreina created HDFS-9072: Summary: Fix random failures in TestJMXGet Key: HDFS-9072 URL: https://issues.apache.org/jira/browse/HDFS-9072 Project: Hadoop HDFS Issue Type: Bug Reporter: J.Andreina Assignee: J.Andreina There is a slight delay in updating the jmx values after HDFS operations. But assertions are done even that . Hence following testcases fails. TestJMXGet#testNameNode() TestJMXGet#testDataNode() -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8545) Add an API to fetch the total file length from a specific path, apart from getting by default from root
[ https://issues.apache.org/jira/browse/HDFS-8545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14742956#comment-14742956 ] J.Andreina commented on HDFS-8545: -- Thanks [~wheat9] for the review comments. bq.My question is whether there are any use cases where the command line tool is insufficient and you absolutely need a new API for this? Command line tool is useful for admins to check manually. API will help to get the same detail programmatically by the user applications. > Add an API to fetch the total file length from a specific path, apart from > getting by default from root > --- > > Key: HDFS-8545 > URL: https://issues.apache.org/jira/browse/HDFS-8545 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: J.Andreina >Assignee: J.Andreina >Priority: Minor > Attachments: HDFS-8545.1.patch, HDFS-8545.2.patch, HDFS-8545.3.patch, > HDFS-8545.4.patch > > > Currently by default in FileSystem#getUsed() returns the total file size from > root. > It is good to have an api to return the total file size from specified path > ,same as we specify the path in "./hdfs dfs -du -s /path" . -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Reopened] (HDFS-9073) Fix failures in TestLazyPersistLockedMemory#testReleaseOnEviction
[
https://issues.apache.org/jira/browse/HDFS-9073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
J.Andreina reopened HDFS-9073:
--
Reopening as the fix for HDFS-9067 does not apply.
> Fix failures in TestLazyPersistLockedMemory#testReleaseOnEviction
> -
>
> Key: HDFS-9073
> URL: https://issues.apache.org/jira/browse/HDFS-9073
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: J.Andreina
>Assignee: J.Andreina
>
> Testcase ,TestLazyPersistLockedMemory#testReleaseOnEviction fails with below
> exception
> {noformat}
> java.lang.AssertionError: expected:<1> but was:<0>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at org.junit.Assert.assertEquals(Assert.java:542)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.LazyPersistTestCase.verifyRamDiskJMXMetric(LazyPersistTestCase.java:483)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistLockedMemory.testReleaseOnEviction(TestLazyPersistLockedMemory.java:122)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HDFS-9073) Fix failures in TestLazyPersistLockedMemory#testReleaseOnEviction
J.Andreina created HDFS-9073:
Summary: Fix failures in
TestLazyPersistLockedMemory#testReleaseOnEviction
Key: HDFS-9073
URL: https://issues.apache.org/jira/browse/HDFS-9073
Project: Hadoop HDFS
Issue Type: Bug
Reporter: J.Andreina
Assignee: J.Andreina
Testcase ,TestLazyPersistLockedMemory#testReleaseOnEviction fails with below
exception
{noformat}
java.lang.AssertionError: expected:<1> but was:<0>
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
at org.junit.Assert.assertEquals(Assert.java:542)
at
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.LazyPersistTestCase.verifyRamDiskJMXMetric(LazyPersistTestCase.java:483)
at
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistLockedMemory.testReleaseOnEviction(TestLazyPersistLockedMemory.java:122)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8780) Fetching live/dead datanode list with arg true for removeDecommissionNode,returns list with decom node.
[
https://issues.apache.org/jira/browse/HDFS-8780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14742881#comment-14742881
]
J.Andreina commented on HDFS-8780:
--
TestCase failures are unrelated.
Please review.
> Fetching live/dead datanode list with arg true for
> removeDecommissionNode,returns list with decom node.
> ---
>
> Key: HDFS-8780
> URL: https://issues.apache.org/jira/browse/HDFS-8780
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: J.Andreina
>Assignee: J.Andreina
>Priority: Critical
> Attachments: HDFS-8780.1.patch, HDFS-8780.2.patch, HDFS-8780.3.patch
>
>
> Current implementation:
> ==
> DatanodeManager#removeDecomNodeFromList() , Decommissioned node will be
> removed from dead/live node list only if below conditions are met
> I . If the Include list is not empty.
> II. If include and exclude list does not have decommissioned node and node
> state is decommissioned.
> {code}
> if (!hostFileManager.hasIncludes()) {
> return;
>}
> if ((!hostFileManager.isIncluded(node)) &&
> (!hostFileManager.isExcluded(node))
> && node.isDecommissioned()) {
> // Include list is not empty, an existing datanode does not appear
> // in both include or exclude lists and it has been decommissioned.
> // Remove it from the node list.
> it.remove();
> }
> {code}
> As mentioned in javadoc a datanode cannot be in "already decommissioned
> datanode state".
> Following the steps mentioned in javadoc datanode state is "dead" and not
> decommissioned.
> *Can we avoid the unnecessary checks and have check for the node is in
> decommissioned state then remove from node list. ?*
> Please provide your feedback.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HDFS-9073) Fix failures in TestLazyPersistLockedMemory#testReleaseOnEviction
[
https://issues.apache.org/jira/browse/HDFS-9073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
J.Andreina resolved HDFS-9073.
--
Resolution: Duplicate
Resolving this as duplicate to HDFS-9067 . As the fix will be on same utility
class.
> Fix failures in TestLazyPersistLockedMemory#testReleaseOnEviction
> -
>
> Key: HDFS-9073
> URL: https://issues.apache.org/jira/browse/HDFS-9073
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: J.Andreina
>Assignee: J.Andreina
>
> Testcase ,TestLazyPersistLockedMemory#testReleaseOnEviction fails with below
> exception
> {noformat}
> java.lang.AssertionError: expected:<1> but was:<0>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at org.junit.Assert.assertEquals(Assert.java:542)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.LazyPersistTestCase.verifyRamDiskJMXMetric(LazyPersistTestCase.java:483)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistLockedMemory.testReleaseOnEviction(TestLazyPersistLockedMemory.java:122)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Moved] (HDFS-9074) Fix failures in TestLazyPersistReplicaPlacement
[
https://issues.apache.org/jira/browse/HDFS-9074?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
J.Andreina moved HADOOP-12410 to HDFS-9074:
---
Key: HDFS-9074 (was: HADOOP-12410)
Project: Hadoop HDFS (was: Hadoop Common)
> Fix failures in TestLazyPersistReplicaPlacement
> ---
>
> Key: HDFS-9074
> URL: https://issues.apache.org/jira/browse/HDFS-9074
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: J.Andreina
>Assignee: J.Andreina
>
> Testcases TestLazyPersistReplicaPlacement#testSynchronousEviction,
> TestLazyPersistReplicaPlacement#testFallbackToDiskFull fails with below
> excepiton
> {noformat}
> java.lang.AssertionError: expected:<1> but was:<0>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at org.junit.Assert.assertEquals(Assert.java:542)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.LazyPersistTestCase.verifyRamDiskJMXMetric(LazyPersistTestCase.java:483)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaPlacement.testSynchronousEviction(TestLazyPersistReplicaPlacement.java:92)
> {noformat}
> {noformat}
> java.lang.AssertionError: expected:<1> but was:<0>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at org.junit.Assert.assertEquals(Assert.java:542)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.LazyPersistTestCase.verifyRamDiskJMXMetric(LazyPersistTestCase.java:483)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaPlacement.testFallbackToDiskFull(TestLazyPersistReplicaPlacement.java:107)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8780) Fetching live/dead datanode list with arg true for removeDecommissionNode,returns list with decom node.
[
https://issues.apache.org/jira/browse/HDFS-8780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
J.Andreina updated HDFS-8780:
-
Attachment: HDFS-8780.3.patch
Updated the patch fixing checkstyle comments.
Testcase failures are unrelated
Please review.
> Fetching live/dead datanode list with arg true for
> removeDecommissionNode,returns list with decom node.
> ---
>
> Key: HDFS-8780
> URL: https://issues.apache.org/jira/browse/HDFS-8780
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: J.Andreina
>Assignee: J.Andreina
>Priority: Critical
> Attachments: HDFS-8780.1.patch, HDFS-8780.2.patch, HDFS-8780.3.patch
>
>
> Current implementation:
> ==
> DatanodeManager#removeDecomNodeFromList() , Decommissioned node will be
> removed from dead/live node list only if below conditions are met
> I . If the Include list is not empty.
> II. If include and exclude list does not have decommissioned node and node
> state is decommissioned.
> {code}
> if (!hostFileManager.hasIncludes()) {
> return;
>}
> if ((!hostFileManager.isIncluded(node)) &&
> (!hostFileManager.isExcluded(node))
> && node.isDecommissioned()) {
> // Include list is not empty, an existing datanode does not appear
> // in both include or exclude lists and it has been decommissioned.
> // Remove it from the node list.
> it.remove();
> }
> {code}
> As mentioned in javadoc a datanode cannot be in "already decommissioned
> datanode state".
> Following the steps mentioned in javadoc datanode state is "dead" and not
> decommissioned.
> *Can we avoid the unnecessary checks and have check for the node is in
> decommissioned state then remove from node list. ?*
> Please provide your feedback.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9036) In BlockPlacementPolicyWithNodeGroup#chooseLocalStorage , random node is selected eventhough fallbackToLocalRack is true.
[
https://issues.apache.org/jira/browse/HDFS-9036?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
J.Andreina updated HDFS-9036:
-
Attachment: HDFS-9036.2.patch
Thanks [~vinayrpet] for the review comments.
Updated the patch.
Please review.
> In BlockPlacementPolicyWithNodeGroup#chooseLocalStorage , random node is
> selected eventhough fallbackToLocalRack is true.
> -
>
> Key: HDFS-9036
> URL: https://issues.apache.org/jira/browse/HDFS-9036
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: J.Andreina
>Assignee: J.Andreina
> Attachments: HDFS-9036.1.patch, HDFS-9036.2.patch
>
>
> For example in below clusterMap
> Writer = "/d2/r4/n8" which does not have local node.
> Available nodes rack = {"/d2/r4/n7","/d1/r1/n1","/d1/r2/n3","/d2/r3/n6"}
> Current hierarchy in choosing node for first local storage replica is
> 1. Choose local machine (Not available)
> 2. Choose Local node group machine (Not available)
> 3. choose random
> *But instead of choosing random , should choose localrack node first(if
> fallbackToLocalRack is true, in example it is "/d2/r4/n7" ) else should go
> for radom node*
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9042) Update document for the Storage policy name
[ https://issues.apache.org/jira/browse/HDFS-9042?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] J.Andreina updated HDFS-9042: - Attachment: HDFS-9042.1.patch Attached an initial patch. Please review. > Update document for the Storage policy name > --- > > Key: HDFS-9042 > URL: https://issues.apache.org/jira/browse/HDFS-9042 > Project: Hadoop HDFS > Issue Type: Bug > Components: documentation >Reporter: J.Andreina >Assignee: J.Andreina >Priority: Minor > Attachments: HDFS-9042.1.patch > > > Storage policy name : > Incorrect : "Lasy_Persist" > Correct : "Lazy_Persist" -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-9042) Update document for the Storage policy name
[ https://issues.apache.org/jira/browse/HDFS-9042?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] J.Andreina updated HDFS-9042: - Status: Patch Available (was: Open) > Update document for the Storage policy name > --- > > Key: HDFS-9042 > URL: https://issues.apache.org/jira/browse/HDFS-9042 > Project: Hadoop HDFS > Issue Type: Bug > Components: documentation >Reporter: J.Andreina >Assignee: J.Andreina >Priority: Minor > Attachments: HDFS-9042.1.patch > > > Storage policy name : > Incorrect : "Lasy_Persist" > Correct : "Lazy_Persist" -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8975) Erasure coding : Fix random failure in TestSafeModeWithStripedFile
[
https://issues.apache.org/jira/browse/HDFS-8975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14740430#comment-14740430
]
J.Andreina commented on HDFS-8975:
--
Failures are unrelated
Please review.
> Erasure coding : Fix random failure in TestSafeModeWithStripedFile
> --
>
> Key: HDFS-8975
> URL: https://issues.apache.org/jira/browse/HDFS-8975
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: J.Andreina
>Assignee: J.Andreina
> Attachments: HDFS-8975-HDFS-7285-01.patch,
> HDFS-8975-HDFS-7285-02.patch
>
>
> TestSafeModeWithStripedFile#testStripedFile0 fails randomly because:
> In below code they are restarting DN and even before DN registration happens
> to NN, block report is triggered and following operation fails
> i) Safemode safe block is counted (which is coming as 0 randomly ,
> eventhough one block is safe )
> ii) Check for NN in safemode.
> {code}
> cluster.restartDataNode(dnprops.remove(0));
> cluster.triggerBlockReports();
> assertEquals(1, NameNodeAdapter.getSafeModeSafeBlocks(nn));
> {code}
> {code}
> dnProperty = dnprops.remove(0);
> restartDN(dnProperty,nameNodeAddress);
> assertFalse(nn.isInSafeMode());
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9043) Doc updation for commands in HDFS Federation
[
https://issues.apache.org/jira/browse/HDFS-9043?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
J.Andreina updated HDFS-9043:
-
Attachment: HDFS-9043-branch-2-1.patch
HDFS-9043-branch-2.7.0-1.patch
HDFS-9043-1.patch
Thanks [~aw] for the comments.
Attached separate patches for trunk , branch-2 and branch-2.7.0 (as the fixes
are different in each branch).
Please review.
> Doc updation for commands in HDFS Federation
>
>
> Key: HDFS-9043
> URL: https://issues.apache.org/jira/browse/HDFS-9043
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: documentation
>Reporter: J.Andreina
>Assignee: J.Andreina
>Priority: Minor
> Attachments: HDFS-9043-1.patch, HDFS-9043-branch-2-1.patch,
> HDFS-9043-branch-2.7.0-1.patch
>
>
> 1. command is wrong
> {noformat}
> $HADOOP_PREFIX/bin/hdfs dfsadmin -refreshNameNode
> :
> {noformat}
> Correct command is : hdfs dfsadmin -refreshNameNode's'
> 2.command is wrong
> {noformat}
> $HADOOP_PREFIX/sbin/hadoop-daemon.sh --config $HADOOP_CONF_DIR --script
> $HADOOP_PREFIX/bin/hdfs start balancer
> {noformat}
> Correct command is : *start-balancer.sh -policy*
> 3. Reference link to balancer for further details is wrong
> {noformat}
> Note that Balancer only balances the data and does not balance the namespace.
> For the complete command usage, see balancer.
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9043) Doc updation for commands in HDFS Federation
[
https://issues.apache.org/jira/browse/HDFS-9043?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
J.Andreina updated HDFS-9043:
-
Status: Patch Available (was: Open)
> Doc updation for commands in HDFS Federation
>
>
> Key: HDFS-9043
> URL: https://issues.apache.org/jira/browse/HDFS-9043
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: documentation
>Reporter: J.Andreina
>Assignee: J.Andreina
>Priority: Minor
> Attachments: HDFS-9043-1.patch, HDFS-9043-branch-2-1.patch,
> HDFS-9043-branch-2.7.0-1.patch
>
>
> 1. command is wrong
> {noformat}
> $HADOOP_PREFIX/bin/hdfs dfsadmin -refreshNameNode
> :
> {noformat}
> Correct command is : hdfs dfsadmin -refreshNameNode's'
> 2.command is wrong
> {noformat}
> $HADOOP_PREFIX/sbin/hadoop-daemon.sh --config $HADOOP_CONF_DIR --script
> $HADOOP_PREFIX/bin/hdfs start balancer
> {noformat}
> Correct command is : *start-balancer.sh -policy*
> 3. Reference link to balancer for further details is wrong
> {noformat}
> Note that Balancer only balances the data and does not balance the namespace.
> For the complete command usage, see balancer.
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8780) Fetching live/dead datanode list with arg true for removeDecommissionNode,returns list with decom node.
[
https://issues.apache.org/jira/browse/HDFS-8780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
J.Andreina updated HDFS-8780:
-
Attachment: HDFS-8780.1.patch
Updated an initial patch.
Please review.
> Fetching live/dead datanode list with arg true for
> removeDecommissionNode,returns list with decom node.
> ---
>
> Key: HDFS-8780
> URL: https://issues.apache.org/jira/browse/HDFS-8780
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: J.Andreina
>Assignee: J.Andreina
>Priority: Critical
> Attachments: HDFS-8780.1.patch
>
>
> Current implementation:
> ==
> DatanodeManager#removeDecomNodeFromList() , Decommissioned node will be
> removed from dead/live node list only if below conditions are met
> I . If the Include list is not empty.
> II. If include and exclude list does not have decommissioned node and node
> state is decommissioned.
> {code}
> if (!hostFileManager.hasIncludes()) {
> return;
>}
> if ((!hostFileManager.isIncluded(node)) &&
> (!hostFileManager.isExcluded(node))
> && node.isDecommissioned()) {
> // Include list is not empty, an existing datanode does not appear
> // in both include or exclude lists and it has been decommissioned.
> // Remove it from the node list.
> it.remove();
> }
> {code}
> As mentioned in javadoc a datanode cannot be in "already decommissioned
> datanode state".
> Following the steps mentioned in javadoc datanode state is "dead" and not
> decommissioned.
> *Can we avoid the unnecessary checks and have check for the node is in
> decommissioned state then remove from node list. ?*
> Please provide your feedback.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8780) Fetching live/dead datanode list with arg true for removeDecommissionNode,returns list with decom node.
[
https://issues.apache.org/jira/browse/HDFS-8780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
J.Andreina updated HDFS-8780:
-
Status: Patch Available (was: Open)
> Fetching live/dead datanode list with arg true for
> removeDecommissionNode,returns list with decom node.
> ---
>
> Key: HDFS-8780
> URL: https://issues.apache.org/jira/browse/HDFS-8780
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: J.Andreina
>Assignee: J.Andreina
>Priority: Critical
> Attachments: HDFS-8780.1.patch
>
>
> Current implementation:
> ==
> DatanodeManager#removeDecomNodeFromList() , Decommissioned node will be
> removed from dead/live node list only if below conditions are met
> I . If the Include list is not empty.
> II. If include and exclude list does not have decommissioned node and node
> state is decommissioned.
> {code}
> if (!hostFileManager.hasIncludes()) {
> return;
>}
> if ((!hostFileManager.isIncluded(node)) &&
> (!hostFileManager.isExcluded(node))
> && node.isDecommissioned()) {
> // Include list is not empty, an existing datanode does not appear
> // in both include or exclude lists and it has been decommissioned.
> // Remove it from the node list.
> it.remove();
> }
> {code}
> As mentioned in javadoc a datanode cannot be in "already decommissioned
> datanode state".
> Following the steps mentioned in javadoc datanode state is "dead" and not
> decommissioned.
> *Can we avoid the unnecessary checks and have check for the node is in
> decommissioned state then remove from node list. ?*
> Please provide your feedback.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8780) Fetching live/dead datanode list with arg true for removeDecommissionNode,returns list with decom node.
[
https://issues.apache.org/jira/browse/HDFS-8780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
J.Andreina updated HDFS-8780:
-
Attachment: HDFS-8780.2.patch
Updated the patch fixing checkstyle comments.
Testcase failures are unrelated.
Please review.
> Fetching live/dead datanode list with arg true for
> removeDecommissionNode,returns list with decom node.
> ---
>
> Key: HDFS-8780
> URL: https://issues.apache.org/jira/browse/HDFS-8780
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: J.Andreina
>Assignee: J.Andreina
>Priority: Critical
> Attachments: HDFS-8780.1.patch, HDFS-8780.2.patch
>
>
> Current implementation:
> ==
> DatanodeManager#removeDecomNodeFromList() , Decommissioned node will be
> removed from dead/live node list only if below conditions are met
> I . If the Include list is not empty.
> II. If include and exclude list does not have decommissioned node and node
> state is decommissioned.
> {code}
> if (!hostFileManager.hasIncludes()) {
> return;
>}
> if ((!hostFileManager.isIncluded(node)) &&
> (!hostFileManager.isExcluded(node))
> && node.isDecommissioned()) {
> // Include list is not empty, an existing datanode does not appear
> // in both include or exclude lists and it has been decommissioned.
> // Remove it from the node list.
> it.remove();
> }
> {code}
> As mentioned in javadoc a datanode cannot be in "already decommissioned
> datanode state".
> Following the steps mentioned in javadoc datanode state is "dead" and not
> decommissioned.
> *Can we avoid the unnecessary checks and have check for the node is in
> decommissioned state then remove from node list. ?*
> Please provide your feedback.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8853) Erasure Coding: Provide ECSchema validation when creating ECZone
[
https://issues.apache.org/jira/browse/HDFS-8853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
J.Andreina updated HDFS-8853:
-
Attachment: HDFS-8853-HDFS-7285-06.patch
Sorry for the delay [~zhz] .
Testcase failures are unrelated to this issue and will be handled as part of
HDFS-8975 , HDFS-8704.
Finbug warnings are unrelated.
Rebased the patch . Please review.
> Erasure Coding: Provide ECSchema validation when creating ECZone
>
>
> Key: HDFS-8853
> URL: https://issues.apache.org/jira/browse/HDFS-8853
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Rakesh R
>Assignee: J.Andreina
> Attachments: HDFS-8853-HDFS-7285-01.patch,
> HDFS-8853-HDFS-7285-05.patch, HDFS-8853-HDFS-7285-06.patch,
> HDFS-8853-HDFS-7285-merge-02.patch, HDFS-8853-HDFS-7285-merge-03.patch,
> HDFS-8853-HDFS-7285-merge-04.patch
>
>
> Presently the {{DFS#createErasureCodingZone(path, ecSchema, cellSize)}}
> doesn't have any validation that the given {{ecSchema}} is available in
> {{ErasureCodingSchemaManager#activeSchemas}} list. Now, if it doesn't exists
> then will create the ECZone with {{null}} schema. IMHO we could improve this
> by doing necessary basic sanity checks.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HDFS-9043) Doc updation for commands in HDFS Federation
J.Andreina created HDFS-9043:
Summary: Doc updation for commands in HDFS Federation
Key: HDFS-9043
URL: https://issues.apache.org/jira/browse/HDFS-9043
Project: Hadoop HDFS
Issue Type: Bug
Components: documentation
Reporter: J.Andreina
Assignee: J.Andreina
Priority: Minor
1. command is wrong
{noformat}
$HADOOP_PREFIX/bin/hdfs dfsadmin -refreshNameNode
:
{noformat}
Correct command is : hdfs dfsadmin -refreshNameNode's'
2.command is wrong
{noformat}
$HADOOP_PREFIX/sbin/hadoop-daemon.sh --config $HADOOP_CONF_DIR --script
$HADOOP_PREFIX/bin/hdfs start balancer
{noformat}
Correct command is : *hdfs balancer -policy*
3. Reference link to balancer for further details is wrong
{noformat}
Note that Balancer only balances the data and does not balance the namespace.
For the complete command usage, see balancer.
{noformat}
4. URL is not proper
{noformat}
Similar to the Namenode status web page, when using federation a Cluster Web
Console is available to monitor the federated cluster at
http:///dfsclusterhealth.jsp.
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HDFS-9042) Update document for the Storage policy name
J.Andreina created HDFS-9042: Summary: Update document for the Storage policy name Key: HDFS-9042 URL: https://issues.apache.org/jira/browse/HDFS-9042 Project: Hadoop HDFS Issue Type: Bug Components: documentation Reporter: J.Andreina Assignee: J.Andreina Priority: Minor Storage policy name : Incorrect : "Lasy_Persist" Correct : "Lazy_Persist" -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8975) Erasure coding : Fix random failure in TestSafeModeWithStripedFile
[
https://issues.apache.org/jira/browse/HDFS-8975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
J.Andreina updated HDFS-8975:
-
Attachment: HDFS-8975-HDFS-7285-02.patch
Thanks [~walter.k.su] for the review comments.
Udpated the patch .
Please review.
> Erasure coding : Fix random failure in TestSafeModeWithStripedFile
> --
>
> Key: HDFS-8975
> URL: https://issues.apache.org/jira/browse/HDFS-8975
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: J.Andreina
>Assignee: J.Andreina
> Attachments: HDFS-8975-HDFS-7285-01.patch,
> HDFS-8975-HDFS-7285-02.patch
>
>
> TestSafeModeWithStripedFile#testStripedFile0 fails randomly because:
> In below code they are restarting DN and even before DN registration happens
> to NN, block report is triggered and following operation fails
> i) Safemode safe block is counted (which is coming as 0 randomly ,
> eventhough one block is safe )
> ii) Check for NN in safemode.
> {code}
> cluster.restartDataNode(dnprops.remove(0));
> cluster.triggerBlockReports();
> assertEquals(1, NameNodeAdapter.getSafeModeSafeBlocks(nn));
> {code}
> {code}
> dnProperty = dnprops.remove(0);
> restartDN(dnProperty,nameNodeAddress);
> assertFalse(nn.isInSafeMode());
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9036) In BlockPlacementPolicyWithNodeGroup#chooseLocalStorage , random node is selected eventhough fallbackToLocalRack is true.
[
https://issues.apache.org/jira/browse/HDFS-9036?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
J.Andreina updated HDFS-9036:
-
Status: Patch Available (was: Open)
> In BlockPlacementPolicyWithNodeGroup#chooseLocalStorage , random node is
> selected eventhough fallbackToLocalRack is true.
> -
>
> Key: HDFS-9036
> URL: https://issues.apache.org/jira/browse/HDFS-9036
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: J.Andreina
>Assignee: J.Andreina
> Attachments: HDFS-9036.1.patch
>
>
> For example in below clusterMap
> Writer = "/d2/r4/n8" which does not have local node.
> Available nodes rack = {"/d2/r4/n7","/d1/r1/n1","/d1/r2/n3","/d2/r3/n6"}
> Current hierarchy in choosing node for first local storage replica is
> 1. Choose local machine (Not available)
> 2. Choose Local node group machine (Not available)
> 3. choose random
> *But instead of choosing random , should choose localrack node first(if
> fallbackToLocalRack is true, in example it is "/d2/r4/n7" ) else should go
> for radom node*
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9036) In BlockPlacementPolicyWithNodeGroup#chooseLocalStorage , random node is selected eventhough fallbackToLocalRack is true.
[
https://issues.apache.org/jira/browse/HDFS-9036?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
J.Andreina updated HDFS-9036:
-
Attachment: HDFS-9036.1.patch
Attached an initial patch .
Please review.
> In BlockPlacementPolicyWithNodeGroup#chooseLocalStorage , random node is
> selected eventhough fallbackToLocalRack is true.
> -
>
> Key: HDFS-9036
> URL: https://issues.apache.org/jira/browse/HDFS-9036
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: J.Andreina
>Assignee: J.Andreina
> Attachments: HDFS-9036.1.patch
>
>
> For example in below clusterMap
> Writer = "/d2/r4/n8" which does not have local node.
> Available nodes rack = {"/d2/r4/n7","/d1/r1/n1","/d1/r2/n3","/d2/r3/n6"}
> Current hierarchy in choosing node for first local storage replica is
> 1. Choose local machine (Not available)
> 2. Choose Local node group machine (Not available)
> 3. choose random
> *But instead of choosing random , should choose localrack node first(if
> fallbackToLocalRack is true, in example it is "/d2/r4/n7" ) else should go
> for radom node*
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HDFS-9044) If favored nodes are not available, then should fall back to regular BlockPlacementPolicyWithNodeGroup
J.Andreina created HDFS-9044: Summary: If favored nodes are not available, then should fall back to regular BlockPlacementPolicyWithNodeGroup Key: HDFS-9044 URL: https://issues.apache.org/jira/browse/HDFS-9044 Project: Hadoop HDFS Issue Type: Bug Reporter: J.Andreina Assignee: J.Andreina Current behavior: While choosing targets for replicas if favored nodes are not available , then it goes for choosing the random node from the favored node group instead of falling back to regular BlockPlacementPolicyWithNodeGroup -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-9043) Doc updation for commands in HDFS Federation
[
https://issues.apache.org/jira/browse/HDFS-9043?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
J.Andreina updated HDFS-9043:
-
Description:
1. command is wrong
{noformat}
$HADOOP_PREFIX/bin/hdfs dfsadmin -refreshNameNode
:
{noformat}
Correct command is : hdfs dfsadmin -refreshNameNode's'
2.command is wrong
{noformat}
$HADOOP_PREFIX/sbin/hadoop-daemon.sh --config $HADOOP_CONF_DIR --script
$HADOOP_PREFIX/bin/hdfs start balancer
{noformat}
Correct command is : *start-balancer.sh -policy*
3. Reference link to balancer for further details is wrong
{noformat}
Note that Balancer only balances the data and does not balance the namespace.
For the complete command usage, see balancer.
{noformat}
was:
1. command is wrong
{noformat}
$HADOOP_PREFIX/bin/hdfs dfsadmin -refreshNameNode
:
{noformat}
Correct command is : hdfs dfsadmin -refreshNameNode's'
2.command is wrong
{noformat}
$HADOOP_PREFIX/sbin/hadoop-daemon.sh --config $HADOOP_CONF_DIR --script
$HADOOP_PREFIX/bin/hdfs start balancer
{noformat}
Correct command is : *hdfs balancer -policy*
3. Reference link to balancer for further details is wrong
{noformat}
Note that Balancer only balances the data and does not balance the namespace.
For the complete command usage, see balancer.
{noformat}
4. URL is not proper
{noformat}
Similar to the Namenode status web page, when using federation a Cluster Web
Console is available to monitor the federated cluster at
http:///dfsclusterhealth.jsp.
{noformat}
> Doc updation for commands in HDFS Federation
>
>
> Key: HDFS-9043
> URL: https://issues.apache.org/jira/browse/HDFS-9043
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: documentation
>Reporter: J.Andreina
>Assignee: J.Andreina
>Priority: Minor
>
> 1. command is wrong
> {noformat}
> $HADOOP_PREFIX/bin/hdfs dfsadmin -refreshNameNode
> :
> {noformat}
> Correct command is : hdfs dfsadmin -refreshNameNode's'
> 2.command is wrong
> {noformat}
> $HADOOP_PREFIX/sbin/hadoop-daemon.sh --config $HADOOP_CONF_DIR --script
> $HADOOP_PREFIX/bin/hdfs start balancer
> {noformat}
> Correct command is : *start-balancer.sh -policy*
> 3. Reference link to balancer for further details is wrong
> {noformat}
> Note that Balancer only balances the data and does not balance the namespace.
> For the complete command usage, see balancer.
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9043) Doc updation for commands in HDFS Federation
[
https://issues.apache.org/jira/browse/HDFS-9043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14738797#comment-14738797
]
J.Andreina commented on HDFS-9043:
--
Thanks [~aw] for the comments.
I agree with your point .
Shall we update the command to *start-balancer.sh -policy* which would be
common to both branch-2 and trunk
> Doc updation for commands in HDFS Federation
>
>
> Key: HDFS-9043
> URL: https://issues.apache.org/jira/browse/HDFS-9043
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: documentation
>Reporter: J.Andreina
>Assignee: J.Andreina
>Priority: Minor
>
> 1. command is wrong
> {noformat}
> $HADOOP_PREFIX/bin/hdfs dfsadmin -refreshNameNode
> :
> {noformat}
> Correct command is : hdfs dfsadmin -refreshNameNode's'
> 2.command is wrong
> {noformat}
> $HADOOP_PREFIX/sbin/hadoop-daemon.sh --config $HADOOP_CONF_DIR --script
> $HADOOP_PREFIX/bin/hdfs start balancer
> {noformat}
> Correct command is : *start-balancer.sh -policy*
> 3. Reference link to balancer for further details is wrong
> {noformat}
> Note that Balancer only balances the data and does not balance the namespace.
> For the complete command usage, see balancer.
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8853) Erasure Coding: Provide ECSchema validation when setting EC policy
[
https://issues.apache.org/jira/browse/HDFS-8853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14740135#comment-14740135
]
J.Andreina commented on HDFS-8853:
--
Thanks [~zhz] for the commit .
> Erasure Coding: Provide ECSchema validation when setting EC policy
> --
>
> Key: HDFS-8853
> URL: https://issues.apache.org/jira/browse/HDFS-8853
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Rakesh R
>Assignee: J.Andreina
> Fix For: HDFS-7285
>
> Attachments: HDFS-8853-HDFS-7285-01.patch,
> HDFS-8853-HDFS-7285-05.patch, HDFS-8853-HDFS-7285-06.patch,
> HDFS-8853-HDFS-7285-merge-02.patch, HDFS-8853-HDFS-7285-merge-03.patch,
> HDFS-8853-HDFS-7285-merge-04.patch
>
>
> Presently the {{DFS#createErasureCodingZone(path, ecSchema, cellSize)}}
> doesn't have any validation that the given {{ecSchema}} is available in
> {{ErasureCodingSchemaManager#activeSchemas}} list. Now, if it doesn't exists
> then will create the ECZone with {{null}} schema. IMHO we could improve this
> by doing necessary basic sanity checks.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9036) In BlockPlacementPolicyWithNodeGroup#chooseLocalStorage , random node is selected eventhough fallbackToLocalRack is true.
[
https://issues.apache.org/jira/browse/HDFS-9036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14740131#comment-14740131
]
J.Andreina commented on HDFS-9036:
--
Testcase failures are unrelated .
Please review.
> In BlockPlacementPolicyWithNodeGroup#chooseLocalStorage , random node is
> selected eventhough fallbackToLocalRack is true.
> -
>
> Key: HDFS-9036
> URL: https://issues.apache.org/jira/browse/HDFS-9036
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: J.Andreina
>Assignee: J.Andreina
> Attachments: HDFS-9036.1.patch
>
>
> For example in below clusterMap
> Writer = "/d2/r4/n8" which does not have local node.
> Available nodes rack = {"/d2/r4/n7","/d1/r1/n1","/d1/r2/n3","/d2/r3/n6"}
> Current hierarchy in choosing node for first local storage replica is
> 1. Choose local machine (Not available)
> 2. Choose Local node group machine (Not available)
> 3. choose random
> *But instead of choosing random , should choose localrack node first(if
> fallbackToLocalRack is true, in example it is "/d2/r4/n7" ) else should go
> for radom node*
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8545) Add an API to fetch the total file length from a specific path, apart from getting by default from root
[ https://issues.apache.org/jira/browse/HDFS-8545?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] J.Andreina updated HDFS-8545: - Attachment: HDFS-8545.4.patch Thanks [~vinayrpet] for the review comments. I have updated the patch .Please review > Add an API to fetch the total file length from a specific path, apart from > getting by default from root > --- > > Key: HDFS-8545 > URL: https://issues.apache.org/jira/browse/HDFS-8545 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: J.Andreina >Assignee: J.Andreina >Priority: Minor > Attachments: HDFS-8545.1.patch, HDFS-8545.2.patch, HDFS-8545.3.patch, > HDFS-8545.4.patch > > > Currently by default in FileSystem#getUsed() returns the total file size from > root. > It is good to have an api to return the total file size from specified path > ,same as we specify the path in "./hdfs dfs -du -s /path" . -- This message was sent by Atlassian JIRA (v6.3.4#6332)
