[jira] [Updated] (HDFS-12352) [branch-2] Use HDFS specific network topology to choose datanode in BlockPlacementPolicyDefault
[ https://issues.apache.org/jira/browse/HDFS-12352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chen Liang updated HDFS-12352: -- Status: Patch Available (was: Open) > [branch-2] Use HDFS specific network topology to choose datanode in > BlockPlacementPolicyDefault > --- > > Key: HDFS-12352 > URL: https://issues.apache.org/jira/browse/HDFS-12352 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Chen Liang >Assignee: Chen Liang > Attachments: HDFS-12352-branch-2.001.patch > > > This JIRA is to backport HDFS-11530 to branch-2 -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-12352) [branch-2] Use HDFS specific network topology to choose datanode in BlockPlacementPolicyDefault
[ https://issues.apache.org/jira/browse/HDFS-12352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16140834#comment-16140834 ] Chen Liang edited comment on HDFS-12352 at 8/24/17 10:15 PM: - I manually fixed several conflicts when cherry-picking from trunk. So, [~linyiqun] any chance you could take a look if you get a chance? It would be appreciated. Thank you very much in advance. was (Author: vagarychen): I manually fixed several conflicts when cherry-picking from trunk. So [~linyiqun] it would be appreciated if you could take a look if you get a chance. Thanks in advance. > [branch-2] Use HDFS specific network topology to choose datanode in > BlockPlacementPolicyDefault > --- > > Key: HDFS-12352 > URL: https://issues.apache.org/jira/browse/HDFS-12352 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Chen Liang >Assignee: Chen Liang > Attachments: HDFS-12352-branch-2.001.patch > > > This JIRA is to backport HDFS-11530 to branch-2 -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12321) Ozone : debug cli: add support to load user-provided SQL query
[
https://issues.apache.org/jira/browse/HDFS-12321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chen Liang updated HDFS-12321:
--
Attachment: HDFS-12321-HDFS-7240.009.patch
Thanks [~anu] for the comments, addressed in v009 patch. Also fixed another
bug.
Tested from command line in my local run. Worked fine. Only one thing, the
command needs to be like:
{{./bin/hdfs ozsqlquery -p ~/out_sql.db --countVolumeBucketSkippingBucket
"volumeName=volumeTest0 skipBucketName=bucketTest2"}}
namely, the argument of the user provided command must be put in one single
string by quoting in {{""}}, otherwise only the first one gets taken.
> Ozone : debug cli: add support to load user-provided SQL query
> --
>
> Key: HDFS-12321
> URL: https://issues.apache.org/jira/browse/HDFS-12321
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Chen Liang
>Assignee: Chen Liang
> Fix For: ozone
>
> Attachments: HDFS-12321-HDFS-7240.001.patch,
> HDFS-12321-HDFS-7240.002.patch, HDFS-12321-HDFS-7240.003.patch,
> HDFS-12321-HDFS-7240.004.patch, HDFS-12321-HDFS-7240.005.patch,
> HDFS-12321-HDFS-7240.006.patch, HDFS-12321-HDFS-7240.007.patch,
> HDFS-12321-HDFS-7240.008.patch, HDFS-12321-HDFS-7240.009.patch
>
>
> This JIRA extends SQL CLI to support loading a user-provided file that
> includes any sql query the user wants to run on the SQLite db obtained by
> converting Ozone metadata db.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-12334) [branch-2] Add storage type demand to into DFSNetworkTopology#chooseRandom
Chen Liang created HDFS-12334: - Summary: [branch-2] Add storage type demand to into DFSNetworkTopology#chooseRandom Key: HDFS-12334 URL: https://issues.apache.org/jira/browse/HDFS-12334 Project: Hadoop HDFS Issue Type: Bug Reporter: Chen Liang Assignee: Chen Liang This JIRA is to backport HDFS-11514 to branch-2. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12321) Ozone : debug cli: add support to load user-provided SQL query
[ https://issues.apache.org/jira/browse/HDFS-12321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chen Liang updated HDFS-12321: -- Attachment: HDFS-12321-HDFS-7240.005.patch Thanks [~anu] for the review and the comments! Post v005 patch to display a hint message when missing the command file config instead of throwing an exception. Also, previous patches were missing the change to hdfs script, added in v005 patch as well. bq. to support passing arguments to queries This sounds a very good extension, thanks! Will track this in another JIRA bq. read the json file and add the commands to the Options I thought of this too, but it seemed a bit tricky with the Options class. Will need to continue looking closer on how to do this. > Ozone : debug cli: add support to load user-provided SQL query > -- > > Key: HDFS-12321 > URL: https://issues.apache.org/jira/browse/HDFS-12321 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Chen Liang >Assignee: Chen Liang > Fix For: ozone > > Attachments: HDFS-12321-HDFS-7240.001.patch, > HDFS-12321-HDFS-7240.002.patch, HDFS-12321-HDFS-7240.003.patch, > HDFS-12321-HDFS-7240.004.patch, HDFS-12321-HDFS-7240.005.patch > > > This JIRA extends SQL CLI to support loading a user-provided file that > includes any sql query the user wants to run on the SQLite db obtained by > converting Ozone metadata db. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12321) Ozone : debug cli: add support to load user-provided SQL query
[ https://issues.apache.org/jira/browse/HDFS-12321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chen Liang updated HDFS-12321: -- Attachment: HDFS-12321-HDFS-7240.004.patch Fix the javac warning in v004 patch. > Ozone : debug cli: add support to load user-provided SQL query > -- > > Key: HDFS-12321 > URL: https://issues.apache.org/jira/browse/HDFS-12321 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Chen Liang >Assignee: Chen Liang > Fix For: ozone > > Attachments: HDFS-12321-HDFS-7240.001.patch, > HDFS-12321-HDFS-7240.002.patch, HDFS-12321-HDFS-7240.003.patch, > HDFS-12321-HDFS-7240.004.patch > > > This JIRA extends SQL CLI to support loading a user-provided file that > includes any sql query the user wants to run on the SQLite db obtained by > converting Ozone metadata db. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12334) [branch-2] Add storage type demand to into DFSNetworkTopology#chooseRandom
[ https://issues.apache.org/jira/browse/HDFS-12334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chen Liang updated HDFS-12334: -- Status: Patch Available (was: Open) > [branch-2] Add storage type demand to into DFSNetworkTopology#chooseRandom > -- > > Key: HDFS-12334 > URL: https://issues.apache.org/jira/browse/HDFS-12334 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Chen Liang >Assignee: Chen Liang > Attachments: HDFS-12334-branch-2.001.patch > > > This JIRA is to backport HDFS-11514 to branch-2. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12334) [branch-2] Add storage type demand to into DFSNetworkTopology#chooseRandom
[ https://issues.apache.org/jira/browse/HDFS-12334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chen Liang updated HDFS-12334: -- Attachment: HDFS-12334-branch-2.001.patch > [branch-2] Add storage type demand to into DFSNetworkTopology#chooseRandom > -- > > Key: HDFS-12334 > URL: https://issues.apache.org/jira/browse/HDFS-12334 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Chen Liang >Assignee: Chen Liang > Attachments: HDFS-12334-branch-2.001.patch > > > This JIRA is to backport HDFS-11514 to branch-2. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12365) Ozone: ListVolume displays incorrect createdOn time when the volume was created by OzoneRpcClient
[
https://issues.apache.org/jira/browse/HDFS-12365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16144093#comment-16144093
]
Chen Liang commented on HDFS-12365:
---
Thanks [~cheersyang] for the catch! The v001 patch LGTM, I've committed to the
feature branch.
> Ozone: ListVolume displays incorrect createdOn time when the volume was
> created by OzoneRpcClient
> -
>
> Key: HDFS-12365
> URL: https://issues.apache.org/jira/browse/HDFS-12365
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
> Attachments: HDFS-12365-HDFS-7240.001.patch
>
>
> Reproduce steps
> 1. Create a key in ozone with corona (this delegates the call to
> OzoneRpcClient), e.g
> {code}
> [wwei@ozone1 hadoop-3.0.0-beta1-SNAPSHOT]$ ./bin/hdfs corona -numOfThreads 1
> -numOfVolumes 1 -numOfBuckets 1 -numOfKeys 1
> {code}
> 2. Run listVolume
> {code}
> [wwei@ozone1 hadoop-3.0.0-beta1-SNAPSHOT]$ ./bin/hdfs oz -listVolume
> http://localhost:9864 -user wwei
> {
> "owner" : {
> "name" : "wwei"
> },
> "quota" : {
> "unit" : "TB",
> "size" : 1048576
> },
> "volumeName" : "vol-0-31437",
> "createdOn" : "Thu, 01 Jan 1970 00:00:00 GMT",
> "createdBy" : null
> }
> {
> "owner" : {
> "name" : "wwei"
> },
> "quota" : {
> "unit" : "TB",
> "size" : 1048576
> },
> "volumeName" : "vol-0-38900",
> "createdOn" : "Thu, 01 Jan 1970 00:00:00 GMT",
> "createdBy" : null
> }
> {code}
> Note, the time displayed in {{createdOn}} are both incorrect {{Thu, 01 Jan
> 1970 00:00:00 GMT}}.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12365) Ozone: ListVolume displays incorrect createdOn time when the volume was created by OzoneRpcClient
[
https://issues.apache.org/jira/browse/HDFS-12365?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chen Liang updated HDFS-12365:
--
Resolution: Fixed
Status: Resolved (was: Patch Available)
> Ozone: ListVolume displays incorrect createdOn time when the volume was
> created by OzoneRpcClient
> -
>
> Key: HDFS-12365
> URL: https://issues.apache.org/jira/browse/HDFS-12365
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
> Attachments: HDFS-12365-HDFS-7240.001.patch
>
>
> Reproduce steps
> 1. Create a key in ozone with corona (this delegates the call to
> OzoneRpcClient), e.g
> {code}
> [wwei@ozone1 hadoop-3.0.0-beta1-SNAPSHOT]$ ./bin/hdfs corona -numOfThreads 1
> -numOfVolumes 1 -numOfBuckets 1 -numOfKeys 1
> {code}
> 2. Run listVolume
> {code}
> [wwei@ozone1 hadoop-3.0.0-beta1-SNAPSHOT]$ ./bin/hdfs oz -listVolume
> http://localhost:9864 -user wwei
> {
> "owner" : {
> "name" : "wwei"
> },
> "quota" : {
> "unit" : "TB",
> "size" : 1048576
> },
> "volumeName" : "vol-0-31437",
> "createdOn" : "Thu, 01 Jan 1970 00:00:00 GMT",
> "createdBy" : null
> }
> {
> "owner" : {
> "name" : "wwei"
> },
> "quota" : {
> "unit" : "TB",
> "size" : 1048576
> },
> "volumeName" : "vol-0-38900",
> "createdOn" : "Thu, 01 Jan 1970 00:00:00 GMT",
> "createdBy" : null
> }
> {code}
> Note, the time displayed in {{createdOn}} are both incorrect {{Thu, 01 Jan
> 1970 00:00:00 GMT}}.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12321) Ozone : debug cli: add support to load user-provided SQL query
[
https://issues.apache.org/jira/browse/HDFS-12321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16144144#comment-16144144
]
Chen Liang commented on HDFS-12321:
---
The failed tests all passed in my local run. {{TestSQLQuery}} seems to complain
not able to find ozoneQuery.json file when Jenkins was running it. But
interestingly the other test {{TestKSMSQLCli}} that also uses this file seem to
have passed.
> Ozone : debug cli: add support to load user-provided SQL query
> --
>
> Key: HDFS-12321
> URL: https://issues.apache.org/jira/browse/HDFS-12321
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Chen Liang
>Assignee: Chen Liang
> Fix For: ozone
>
> Attachments: HDFS-12321-HDFS-7240.001.patch,
> HDFS-12321-HDFS-7240.002.patch, HDFS-12321-HDFS-7240.003.patch,
> HDFS-12321-HDFS-7240.004.patch, HDFS-12321-HDFS-7240.005.patch,
> HDFS-12321-HDFS-7240.006.patch, HDFS-12321-HDFS-7240.007.patch,
> HDFS-12321-HDFS-7240.008.patch, HDFS-12321-HDFS-7240.009.patch,
> HDFS-12321-HDFS-7240.010.patch
>
>
> This JIRA extends SQL CLI to support loading a user-provided file that
> includes any sql query the user wants to run on the SQLite db obtained by
> converting Ozone metadata db.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12368) [branch-2] Enable DFSNetworkTopology as default
[ https://issues.apache.org/jira/browse/HDFS-12368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chen Liang updated HDFS-12368: -- Attachment: HDFS-12368-branch-2.001.patch > [branch-2] Enable DFSNetworkTopology as default > --- > > Key: HDFS-12368 > URL: https://issues.apache.org/jira/browse/HDFS-12368 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Chen Liang >Assignee: Chen Liang > Attachments: HDFS-12368-branch-2.001.patch > > > This JIRA is to backport HDFS-11998 to branch-2. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12368) [branch-2] Enable DFSNetworkTopology as default
[ https://issues.apache.org/jira/browse/HDFS-12368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chen Liang updated HDFS-12368: -- Status: Patch Available (was: Open) > [branch-2] Enable DFSNetworkTopology as default > --- > > Key: HDFS-12368 > URL: https://issues.apache.org/jira/browse/HDFS-12368 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Chen Liang >Assignee: Chen Liang > Attachments: HDFS-12368-branch-2.001.patch > > > This JIRA is to backport HDFS-11998 to branch-2. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-12368) [branch-2] Enable DFSNetworkTopology as default
Chen Liang created HDFS-12368: - Summary: [branch-2] Enable DFSNetworkTopology as default Key: HDFS-12368 URL: https://issues.apache.org/jira/browse/HDFS-12368 Project: Hadoop HDFS Issue Type: Improvement Reporter: Chen Liang Assignee: Chen Liang This JIRA is to backport HDFS-11998 to branch-2. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-12321) Ozone : debug cli: add support to load user-provided SQL query
[
https://issues.apache.org/jira/browse/HDFS-12321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16139316#comment-16139316
]
Chen Liang edited comment on HDFS-12321 at 8/23/17 11:18 PM:
-
Thanks Anu for the patch, I thought about some of these changes also, but was
planning for another patch. One thing I found tricky was what's the best way to
take arguments from command line for the sql queries. For example if I want to
get count of bucket from volume X, but ignoring volumes starts with Y. So we
want to take X and Y from command line. I think ideally we specify in json file
a sql like "select count(\*) as count from bucketInfo where volumeName=X and
bucketName!=Y" and then take X and Y from commandline.
Now the issue is, given two argument from command, say M and N, what is the
best way to assign the arguments, i.e.whether it is X=M,Y=N or the other way.
Ideally, I think they should be specified with options. Such as writing on
command line with something like -x M -y N, then we know it is X=M and Y=N. But
this means json command file must also provide -x and -y options and how to
parse them. This adds some complexity to the code which I was targeting for a
separate patch.
Seems v007 is taking arguments in order. Namely, if in sql string X comes
before Y and in command line M comes before N, then X=M,Y=N, right? I guess I'm
fine with this. But still, I'm not quite sure about this part:
{code}
command.getArguments().forEach((String argument) -> {
if (org.apache.commons.lang3.StringUtils.isNoneBlank(argument)) {
//TODO check if the Argument already exists, if so don't add it
// and just use it.
commandBuilder.withLongOpt(argument);
}
});
{code}
I might be totally wrong here. But I as far as I remember, {{withLongOpt}} is
specifying the longer version name of the command. For example,
{{x.withLongOpt("createContainer").create("c")}} means you can type either -c
or -createContainer, they both recognized as same createContainer command. If
this is correct, then this code snippet is basically keeping overwriting
withLongOpt with each of the arguments. In the end, the arguments do not seem
to be correctly taken and passed from command line to command instance.
Additionally, that change to {{TestPlanner}} seems unnecessary.
was (Author: vagarychen):
Thanks Anu for the patch, I thought about some of these changes also, but was
planning for another patch. One thing I found tricky was what's the best way to
take arguments from command line for the sql queries. For example if I want to
get count of bucket from volume X, but ignoring volumes starts with Y. So we
want to take X and Y from command line. I think ideally we specify in json file
a sql like "select count(*) as count from bucketInfo where volumeName=X and
bucketName!=Y" and then take X and Y from commandline.
Now the issue is, given two argument from command, say M and N, what is the
best way to assign the arguments, i.e.whether it is X=M,Y=N or the other way.
Ideally, I think they should be specified with options. Such as writing on
command line with something like -x M -y N, then we know it is X=M and Y=N. But
this means json command file must also provide -x and -y options and how to
parse them. This adds some complexity to the code which I was targeting for a
separate patch.
Seems v007 is taking arguments in order. Namely, if in sql string X comes
before Y and in command line M comes before N, then X=M,Y=N, right? I guess I'm
fine with this. But still, I'm not quite sure about this part:
{code}
command.getArguments().forEach((String argument) -> {
if (org.apache.commons.lang3.StringUtils.isNoneBlank(argument)) {
//TODO check if the Argument already exists, if so don't add it
// and just use it.
commandBuilder.withLongOpt(argument);
}
});
{code}
I might be totally wrong here. But I as far as I remember, {{withLongOpt}} is
specifying the longer version name of the command. For example,
{{x.withLongOpt("createContainer").create("c")}} means you can type either -c
or -createContainer, they both recognized as same createContainer command. If
this is correct, then this code snippet is basically keeping overwriting
withLongOpt with each of the arguments. In the end, the arguments do not seem
to be correctly taken and passed from command line to command instance.
Additionally, that change to {{TestPlanner}} seems unnecessary.
> Ozone : debug cli: add support to load user-provided SQL query
> --
>
> Key: HDFS-12321
> URL: https://issues.apache.org/jira/browse/HDFS-12321
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Chen Liang
>
[jira] [Commented] (HDFS-12321) Ozone : debug cli: add support to load user-provided SQL query
[
https://issues.apache.org/jira/browse/HDFS-12321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16139316#comment-16139316
]
Chen Liang commented on HDFS-12321:
---
Thanks Anu for the patch, I thought about some of these changes also, but was
planning for another patch. One thing I found tricky was what's the best way to
take arguments from command line for the sql queries. For example if I want to
get count of bucket from volume X, but ignoring volumes starts with Y. So we
want to take X and Y from command line. I think ideally we specify in json file
a sql like "select count(*) as count from bucketInfo where volumeName=X and
bucketName!=Y" and then take X and Y from commandline.
Now the issue is, given two argument from command, say M and N, what is the
best way to assign the arguments, i.e.whether it is X=M,Y=N or the other way.
Ideally, I think they should be specified with options. Such as writing on
command line with something like -x M -y N, then we know it is X=M and Y=N. But
this means json command file must also provide -x and -y options and how to
parse them. This adds some complexity to the code which I was targeting for a
separate patch.
Seems v007 is taking arguments in order. Namely, if in sql string X comes
before Y and in command line M comes before N, then X=M,Y=N, right? I guess I'm
fine with this. But still, I'm not quite sure about this part:
{code}
command.getArguments().forEach((String argument) -> {
if (org.apache.commons.lang3.StringUtils.isNoneBlank(argument)) {
//TODO check if the Argument already exists, if so don't add it
// and just use it.
commandBuilder.withLongOpt(argument);
}
});
{code}
I might be totally wrong here. But I as far as I remember, {{withLongOpt}} is
specifying the longer version name of the command. For example,
{{x.withLongOpt("createContainer").create("c")}} means you can type either -c
or -createContainer, they both recognized as same createContainer command. If
this is correct, then this code snippet is basically keeping overwriting
withLongOpt with each of the arguments. In the end, the arguments do not seem
to be correctly taken and passed from command line to command instance.
Additionally, that change to {{TestPlanner}} seems unnecessary.
> Ozone : debug cli: add support to load user-provided SQL query
> --
>
> Key: HDFS-12321
> URL: https://issues.apache.org/jira/browse/HDFS-12321
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Chen Liang
>Assignee: Chen Liang
> Fix For: ozone
>
> Attachments: HDFS-12321-HDFS-7240.001.patch,
> HDFS-12321-HDFS-7240.002.patch, HDFS-12321-HDFS-7240.003.patch,
> HDFS-12321-HDFS-7240.004.patch, HDFS-12321-HDFS-7240.005.patch,
> HDFS-12321-HDFS-7240.006.patch, HDFS-12321-HDFS-7240.007.patch
>
>
> This JIRA extends SQL CLI to support loading a user-provided file that
> includes any sql query the user wants to run on the SQLite db obtained by
> converting Ozone metadata db.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12321) Ozone : debug cli: add support to load user-provided SQL query
[
https://issues.apache.org/jira/browse/HDFS-12321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16139451#comment-16139451
]
Chen Liang commented on HDFS-12321:
---
bq. Now we need to add these two as two options of the command. Before we run
the SQL, we have to bind the arguments in the right location (JDBC limitations
where this is done via position, so we already remember the arguments
positions. So when user runs the command he/she will do it like this-- "hdfs
sqlquery commandName -volumeName myVolume -bucketName myBucket" – inside code
we will do a getOption("volumeName") which will return the myVolume to us,
which will become part of SQL query. Same thing for bucketName, and sql will
contain myBucket.
Yes, this is exactly I was talking about, the part that requires code is: the
CLI Parser *must be able to recognize these two options* "-volumeName" and
"-bucketName", such that they can be parsed by Parser class. i.e. we call
{{CommandLine cmd = BasicParser#parse(options)}} followed by
{{cmd.getOptionValue("volumeName")}}, then we obtain the corresponding command
line values of volumeName. But how is this going to happen? i.e. how can the
parser recognize this option?
Take another example, there is command like {{hdfs scm -container -create -c
Contaienr0}}. Take {{-create}} for instance. The way it worked is to explicitly
code {{-create}} as an option that can be recognized by CLI. So similarly here
"-volumeName" and "-bucketName" must also be coded I guess. But these two
options come with the user-provided sql query and are only meaningful for this
user-provided query, thus they can only be provided along with this sql query,
probably in the json file I suppose. Then CLI takes these options and adds them
as recognizable options. This is the whole point I was try to make.
Maybe there is a way for {{BasicParser}} or something to parse options by their
given order (which would be a bit odd to me...). In this case, things might
become easier.
> Ozone : debug cli: add support to load user-provided SQL query
> --
>
> Key: HDFS-12321
> URL: https://issues.apache.org/jira/browse/HDFS-12321
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Chen Liang
>Assignee: Chen Liang
> Fix For: ozone
>
> Attachments: HDFS-12321-HDFS-7240.001.patch,
> HDFS-12321-HDFS-7240.002.patch, HDFS-12321-HDFS-7240.003.patch,
> HDFS-12321-HDFS-7240.004.patch, HDFS-12321-HDFS-7240.005.patch,
> HDFS-12321-HDFS-7240.006.patch, HDFS-12321-HDFS-7240.007.patch
>
>
> This JIRA extends SQL CLI to support loading a user-provided file that
> includes any sql query the user wants to run on the SQLite db obtained by
> converting Ozone metadata db.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12346) [branch-2] Combine the old and the new chooseRandom for better performance
[ https://issues.apache.org/jira/browse/HDFS-12346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chen Liang updated HDFS-12346: -- Attachment: HDFS-12346-branch-2.001.patch v001 patch was cleanly cherry-picked from trunk. > [branch-2] Combine the old and the new chooseRandom for better performance > -- > > Key: HDFS-12346 > URL: https://issues.apache.org/jira/browse/HDFS-12346 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Chen Liang >Assignee: Chen Liang > Attachments: HDFS-12346-branch-2.001.patch > > > This JIRA is to backport HDFS-11577 back to branch-2. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12346) [branch-2] Combine the old and the new chooseRandom for better performance
[ https://issues.apache.org/jira/browse/HDFS-12346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chen Liang updated HDFS-12346: -- Status: Patch Available (was: Open) > [branch-2] Combine the old and the new chooseRandom for better performance > -- > > Key: HDFS-12346 > URL: https://issues.apache.org/jira/browse/HDFS-12346 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Chen Liang >Assignee: Chen Liang > Attachments: HDFS-12346-branch-2.001.patch > > > This JIRA is to backport HDFS-11577 back to branch-2. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12334) [branch-2] Add storage type demand to into DFSNetworkTopology#chooseRandom
[ https://issues.apache.org/jira/browse/HDFS-12334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16138883#comment-16138883 ] Chen Liang commented on HDFS-12334: --- The failed tests are unrelated. [~arpitagarwal] could you please take a look on this? > [branch-2] Add storage type demand to into DFSNetworkTopology#chooseRandom > -- > > Key: HDFS-12334 > URL: https://issues.apache.org/jira/browse/HDFS-12334 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Chen Liang >Assignee: Chen Liang > Attachments: HDFS-12334-branch-2.001.patch, > HDFS-12334-branch-2.002.patch > > > This JIRA is to backport HDFS-11514 to branch-2. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-12346) [branch-2] Combine the old and the new chooseRandom for better performance
Chen Liang created HDFS-12346: - Summary: [branch-2] Combine the old and the new chooseRandom for better performance Key: HDFS-12346 URL: https://issues.apache.org/jira/browse/HDFS-12346 Project: Hadoop HDFS Issue Type: Improvement Reporter: Chen Liang Assignee: Chen Liang This JIRA is to backport HDFS-11577 back to branch-2. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12321) Ozone : debug cli: add support to load user-provided SQL query
[ https://issues.apache.org/jira/browse/HDFS-12321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chen Liang updated HDFS-12321: -- Attachment: HDFS-12321-HDFS-7240.010.patch Fix checkstyle and findbugs in v010 patch > Ozone : debug cli: add support to load user-provided SQL query > -- > > Key: HDFS-12321 > URL: https://issues.apache.org/jira/browse/HDFS-12321 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Chen Liang >Assignee: Chen Liang > Fix For: ozone > > Attachments: HDFS-12321-HDFS-7240.001.patch, > HDFS-12321-HDFS-7240.002.patch, HDFS-12321-HDFS-7240.003.patch, > HDFS-12321-HDFS-7240.004.patch, HDFS-12321-HDFS-7240.005.patch, > HDFS-12321-HDFS-7240.006.patch, HDFS-12321-HDFS-7240.007.patch, > HDFS-12321-HDFS-7240.008.patch, HDFS-12321-HDFS-7240.009.patch, > HDFS-12321-HDFS-7240.010.patch > > > This JIRA extends SQL CLI to support loading a user-provided file that > includes any sql query the user wants to run on the SQLite db obtained by > converting Ozone metadata db. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12332) Logging improvement suggestion for SampleStat function MinMax.add
[
https://issues.apache.org/jira/browse/HDFS-12332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16135483#comment-16135483
]
Chen Liang commented on HDFS-12332:
---
Thanks [~xzhao9] for the improvement. One thought, "Updating max value as X"
might not be that useful because it does not say what this value is about. i.e.
a more informative output might be something like "Updating the max value of
metric X from Y to Z". Not sure whether this can be easily done though because
SampleStat seems to be a generic utility class and may not be able to know what
is the "metric X" part.
Also it's possible this can be quite noisy, in which we may want to add some
throttling checks. Lastly, seems to me that {{LOG.trace}} is probably more
suitable than {{LOG.debug}} here.
> Logging improvement suggestion for SampleStat function MinMax.add
> -
>
> Key: HDFS-12332
> URL: https://issues.apache.org/jira/browse/HDFS-12332
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: metrics
>Affects Versions: 3.0.0-alpha4
>Reporter: Xu Zhao
>Priority: Minor
> Attachments: hdfs-12332.patch
>
>
> In order to debug performance anomlies, it is better to see when the max and
> min value get updated in metrics.
> For example, in our workload we would like to see the longest latency of a
> write block request on DataNode.
> The following metric updating code could possibly update the min/max value in
> SampleStat class:
> {code:java}
> datanode.getMetrics().addWriteBlockOp(elapsed());
> datanode.getMetrics().incrWritesFromClient(peer.isLocal(), size);
> {code}
> Here I am attaching an attempt to patch it.
> The patch just adds additional DEBUG level logging in the MinMax class and
> prints the new value when it gets updated.
> Please let me know what you think. For example, if this patch would introduce
> too much noises, what is the best way to trace the read/writeBlock that has
> the minimum/maximum latency?
> Any comments are appreciated!
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12315) Use Path instead of String in the TestHdfsAdmin.verifyOpenFiles()
[
https://issues.apache.org/jira/browse/HDFS-12315?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16130909#comment-16130909
]
Chen Liang commented on HDFS-12315:
---
Thanks [~olegd] for the catch! v001 patch LGTM.
> Use Path instead of String in the TestHdfsAdmin.verifyOpenFiles()
> -
>
> Key: HDFS-12315
> URL: https://issues.apache.org/jira/browse/HDFS-12315
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Oleg Danilov
>Priority: Trivial
> Attachments: HDFS-12315.patch
>
>
> closedFiles is a set of Path, therefor closedFiles.contains(String) doesn't
> make sense.
> lines 252-261:
> {code:java}
> private void verifyOpenFiles(HashSet closedFiles,
> HashMap openFileMap) throws IOException {
> HdfsAdmin hdfsAdmin = new HdfsAdmin(FileSystem.getDefaultUri(conf), conf);
> HashSet openFiles = new HashSet<>(openFileMap.keySet());
> RemoteIterator openFilesRemoteItr =
> hdfsAdmin.listOpenFiles();
> while (openFilesRemoteItr.hasNext()) {
> String filePath = openFilesRemoteItr.next().getFilePath();
> assertFalse(filePath + " should not be listed under open files!",
> closedFiles.contains(filePath));
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12292) Federation: Support viewfs:// schema path for DfsAdmin commands
[
https://issues.apache.org/jira/browse/HDFS-12292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16130945#comment-16130945
]
Chen Liang commented on HDFS-12292:
---
Thanks [~erofeev] for the patch! Curious though, is it possible to check that
only when viewfs is being used we perform this path resolving? Because it seems
to me that in the non-viewfs case, this check doesn't really do anything and we
end up just spending extra unnecessary time for each of certain operations.
(Please correct me if I'm wrong.)
> Federation: Support viewfs:// schema path for DfsAdmin commands
> ---
>
> Key: HDFS-12292
> URL: https://issues.apache.org/jira/browse/HDFS-12292
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: federation
>Reporter: Mikhail Erofeev
>Assignee: Mikhail Erofeev
> Attachments: HDFS-12292-002.patch, HDFS-12292-003.patch,
> HDFS-12292.patch
>
>
> Motivation:
> As of now, clients need to specify a nameservice when a cluster is federated,
> otherwise, the exception is fired:
> {code}
> hdfs dfsadmin -setQuota 10 viewfs://vfs-root/user/uname
> setQuota: FileSystem viewfs://vfs-root/ is not an HDFS file system
> # with fs.defaultFS = viewfs://vfs-root/
> hdfs dfsadmin -setQuota 10 vfs-root/user/uname
> setQuota: FileSystem viewfs://vfs-root/ is not an HDFS file system
> # works fine thanks to https://issues.apache.org/jira/browse/HDFS-11432
> hdfs dfsadmin -setQuota 10 hdfs://users-fs/user/uname
> {code}
> This creates inconvenience, inability to rely on fs.defaultFS and forces to
> create client-side mappings for management scripts
> Implementation:
> PathData that is passed to commands should be resolved to its actual
> FileSystem
> Result:
> ViewFS will be resolved to the actual HDFS file system
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12321) Ozone : debug cli: add support to load user-provided SQL query
[ https://issues.apache.org/jira/browse/HDFS-12321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chen Liang updated HDFS-12321: -- Attachment: HDFS-12321-HDFS-7240.003.patch Fix the various Jenkins warnings in v003 patch. > Ozone : debug cli: add support to load user-provided SQL query > -- > > Key: HDFS-12321 > URL: https://issues.apache.org/jira/browse/HDFS-12321 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Chen Liang >Assignee: Chen Liang > Fix For: ozone > > Attachments: HDFS-12321-HDFS-7240.001.patch, > HDFS-12321-HDFS-7240.002.patch, HDFS-12321-HDFS-7240.003.patch > > > This JIRA extends SQL CLI to support loading a user-provided file that > includes any sql query the user wants to run on the SQLite db obtained by > converting Ozone metadata db. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12325) SFTPFileSystem operations should restore cwd
[
https://issues.apache.org/jira/browse/HDFS-12325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16135670#comment-16135670
]
Chen Liang commented on HDFS-12325:
---
Thanks [~elgoiri] for the review, this is good point!
But just like [~arpitagarwal] mentioned, the exception handling of the current
code has other issue. Specifically, it may leak client connections. For
example, in {{SFTPFileSystem#create}}:
{code}
if (parent == null || !mkdirs(client, parent, FsPermission.getDefault())) {
parent = (parent == null) ? new Path("/") : parent;
disconnect(client);
throw new IOException(String.format(E_CREATE_DIR, parent));
}
{code}
This code is making the syntax implication that before throwing the exception
to the caller, the client should be disconnected. However if mkdirs does not
even return but simply throws exception, then there is no catch clause in this
method and the exception gets thrown to the caller, in which case nowhere in
the code the client would be disconnected and thus is leaked. So like Arpit
suggested, we are planning to revisit all of this class's exception handling
and will file another JIRA to fix all of them. In that JIRA we will reexamine
how to handle the exception caused by {{cd}} part here as you pointed out.
> SFTPFileSystem operations should restore cwd
>
>
> Key: HDFS-12325
> URL: https://issues.apache.org/jira/browse/HDFS-12325
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Namit Maheshwari
>Assignee: Chen Liang
> Fix For: 2.9.0, 3.0.0-beta1
>
> Attachments: HDFS-12325.001.patch
>
>
> We've seen a case where writing to {{SFTPFileSystem}} led to unexpected
> behaviour:
> Given a directory ./data with more than one files in it, the steps it took to
> get this error was simply:
> {code}
> hdfs dfs -fs sftp://x.y.z -mkdir dir0
> hdfs dfs -fs sftp://x.y.z -copyFromLocal data dir0
> hdfs dfs -fs sftp://x.y.z -ls -R dir0
> {code}
> But not all files show up as in the ls output, in fact more often just one
> single file shows up in that path...
> Digging deeper, we found that rename, mkdirs and create operations in
> {{SFTPFileSystem}} are changing the current working directory during it's
> execution. For example in create there are:
> {code}
> client.cd(parent.toUri().getPath());
> os = client.put(f.getName());
> {code}
> The issue here is {{SFTPConnectionPool}} is caching SFTP sessions (in
> {{idleConnections}}), which contains their current working directory. So
> after these operations, the sessions will be put back to cache with a changed
> working directory. This accumulates in each call and ends up causing
> unexpected weird behaviour. Basically this error happens when processing
> multiple file system objects in one operation, and relative path is being
> used.
> The fix here is to restore the current working directory of the SFTP sessions.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12235) Ozone: DeleteKey-3: KSM SCM block deletion message and ACK interactions
[
https://issues.apache.org/jira/browse/HDFS-12235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16151296#comment-16151296
]
Chen Liang commented on HDFS-12235:
---
Thanks [~yuanbo] for the patch, looks good to me overall, some comments:
in {{BlockManagerImpl.java}} deleteBlocks
- seems to silently handles the exception when deleteBlockLog.addTransactions
went wrong. By which I mean, the rollback gets done and messages get logged,
but the caller of the function seems to have no clue to know whether this
delete is done, or exception happened and things got rolled back, in which case
delete is not done. Maybe throw the exception after the rollback is done?
- the writeBatch(rollbackBatch) may throw IOException and fail, in which case
rollback failed…not sure whether there is even a way to handle such rollback
failure at all…but maybe we should at least log a LOG.error message or
something here…
- This may not be a valid case, but since deleteBlocks does not have
synchronized, I wonder is it possible to have multiple threads doing deletion
here? if yes, then is there any chance multiple threads trying to delete same
blockID? (e.g. two duplicate key deletion gets to the server at the same time)
will we potentially run into any issue? (e.g. any issue if two threads run into
exactly same blockStore.writeBatch(batch); call?)
- some TODO comments get removed, it appears to me some of them are still
things to be done in the future?
in {{KeyManagerImpl.java}} deletePendingDeletionKey
{code}
byte[] pendingDelKey = DFSUtil.string2Bytes(objectKeyName);
if (pendingDelKey == null) {
throw new IOException("Failed to delete key " + objectKeyName
+ " because it is not found in DB");
}
{code}
it says “not found in DB” but this does not seem to be reading DB but checking
the key bytes? is it missing a db.get() call or something?
> Ozone: DeleteKey-3: KSM SCM block deletion message and ACK interactions
> ---
>
> Key: HDFS-12235
> URL: https://issues.apache.org/jira/browse/HDFS-12235
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
> Attachments: HDFS-12235-HDFS-7240.001.patch,
> HDFS-12235-HDFS-7240.002.patch, HDFS-12235-HDFS-7240.003.patch,
> HDFS-12235-HDFS-7240.004.patch, HDFS-12235-HDFS-7240.005.patch
>
>
> KSM and SCM interaction for delete key operation, both KSM and SCM stores key
> state info in a backlog, KSM needs to scan this log and send block-deletion
> command to SCM, once SCM is fully aware of the message, KSM removes the key
> completely from namespace. See more from the design doc under HDFS-11922,
> this is task break down 2.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11921) Ozone: KSM: Unable to put keys with zero length
[ https://issues.apache.org/jira/browse/HDFS-11921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chen Liang updated HDFS-11921: -- Attachment: HDFS-11921-HDFS-7240.001.patch Right now, the behaviour is that if a length of 0 is given, key allocation will succeed but following writes will throw exception due to writing out of range. Post a patch with just a unit test for illustration. [~anu] are we expecting a different behaviour? if current behaviour works, then maybe we can resolve this jira. > Ozone: KSM: Unable to put keys with zero length > --- > > Key: HDFS-11921 > URL: https://issues.apache.org/jira/browse/HDFS-11921 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Anu Engineer >Assignee: Anu Engineer >Priority: Minor > Attachments: HDFS-11921-HDFS-7240.001.patch > > > As part of working on HDFS-11909, I was trying to put zero length keys. I > found that put key refuses to do that. Here is the call trace, > bq. at ScmBlockLocationProtocolClientSideTranslatorPB.allocateBlock > we check if the block size is greater than 0, which makes sense since we > should not call into SCM to allocate a block of zero size. > However these 2 calls are invoked for creating the key, so that metadata for > key can be created, we should probably take care of this behavior here. > bq. ksm.KeyManagerImpl.allocateKey > bq. ksm.KeySpaceManager.allocateKey(KeySpaceManager.java:428) > Another way to fix this might be to just allocate a block with at least 1 > byte always, which might be easier than special casing code. > [~vagarychen] Would you like to fix this in the next patch you are working on > ? -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12268) Ozone: Add metrics for pending storage container requests
[
https://issues.apache.org/jira/browse/HDFS-12268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16149612#comment-16149612
]
Chen Liang commented on HDFS-12268:
---
Thanks [~linyiqun] for the patch. v004 patch looks good to me overall. Some
minor comments:
1. I found this sentence in {{OzoneMetrics.md}} a bit difficult to understand
"This is the metric for the client that container operations haven't been
processed by container server." Maybe rephrase this a little bit?
2. Seems to me that {{TestXceiverClientMetrics#testMetrics}} L117 and L121 two
assertions may not always hold. Because it seems to me that it is possible that
if the machine is fast enough, async calls could have finished at this point.
Maybe having another thread that keeps making async calls and got killed after
the assertion?
> Ozone: Add metrics for pending storage container requests
> -
>
> Key: HDFS-12268
> URL: https://issues.apache.org/jira/browse/HDFS-12268
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Yiqun Lin
>Assignee: Yiqun Lin
> Attachments: HDFS-12268-HDFS-7240.001.patch,
> HDFS-12268-HDFS-7240.002.patch, HDFS-12268-HDFS-7240.003.patch,
> HDFS-12268-HDFS-7240.004.patch
>
>
> As storage container async interface has been supported after HDFS-11580, we
> need to keep an eye on the queue depth of pending container requests. It can
> help us better found if there are some performance problems.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12572) Ozone: OzoneFileSystem: delete/list status/rename/mkdir APIs
[ https://issues.apache.org/jira/browse/HDFS-12572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16197680#comment-16197680 ] Chen Liang commented on HDFS-12572: --- Thanks [~msingh] for the update. v005 patch LGTM, pending Jenkins. > Ozone: OzoneFileSystem: delete/list status/rename/mkdir APIs > > > Key: HDFS-12572 > URL: https://issues.apache.org/jira/browse/HDFS-12572 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Mukul Kumar Singh >Assignee: Mukul Kumar Singh > Labels: ozoneMerge > Fix For: HDFS-7240 > > Attachments: HDFS-12572-HDFS-7240.001.patch, > HDFS-12572-HDFS-7240.002.patch, HDFS-12572-HDFS-7240.003.patch, > HDFS-12572-HDFS-7240.004.patch, HDFS-12572-HDFS-7240.005.patch > > > This jira will add the delete/list status/rename/mkdir APIs -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12546) Ozone: DB listing operation performance improvement
[
https://issues.apache.org/jira/browse/HDFS-12546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chen Liang updated HDFS-12546:
--
Resolution: Fixed
Status: Resolved (was: Patch Available)
v001 patch LGTM, and the failed tests passed in my local run. I've committed to
the feature branch, thanks [~cheersyang] for the contribution!
> Ozone: DB listing operation performance improvement
> ---
>
> Key: HDFS-12546
> URL: https://issues.apache.org/jira/browse/HDFS-12546
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
> Labels: performance
> Attachments: HDFS-12546-HDFS-7240.001.patch
>
>
> While investigating HDFS-12506, I found there are several {{getRangeKVs}} can
> be replaced by {{getSequentialRangeKVs}} to improve the performance. This
> JIRA is to track these improvements with sufficient tests.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12543) Ozone : allow create key without specifying size
[ https://issues.apache.org/jira/browse/HDFS-12543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16191806#comment-16191806 ] Chen Liang commented on HDFS-12543: --- [~xyao], working on it, will submit a patch later today. Thanks! > Ozone : allow create key without specifying size > > > Key: HDFS-12543 > URL: https://issues.apache.org/jira/browse/HDFS-12543 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Chen Liang >Assignee: Chen Liang > Labels: ozoneMerge > Attachments: HDFS-12543-HDFS-7240.001.patch, > HDFS-12543-HDFS-7240.002.patch, HDFS-12543-HDFS-7240.003.patch, > HDFS-12543-HDFS-7240.004.patch, HDFS-12543-HDFS-7240.005.patch, > HDFS-12543-HDFS-7240.006.patch, HDFS-12543-HDFS-7240.007.patch, > HDFS-12543-HDFS-7240.008.patch, HDFS-12543-HDFS-7240.009.patch > > > Currently when creating a key, it is required to specify the total size of > the key. This makes it inconvenient for the case where a key is created and > data keeps coming and being appended. This JIRA is remove the requirement of > specifying the size on key creation, and allows appending to the key > indefinitely. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12582) Replace HdfsFileStatus constructor with a builder pattern.
[ https://issues.apache.org/jira/browse/HDFS-12582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16192060#comment-16192060 ] Chen Liang commented on HDFS-12582: --- Thanks [~bharatviswa] for working on this! Could you please take a look and verify that the test failures are unrelated? > Replace HdfsFileStatus constructor with a builder pattern. > -- > > Key: HDFS-12582 > URL: https://issues.apache.org/jira/browse/HDFS-12582 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham > Attachments: HDFS-12582.01.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12572) Ozone: OzoneFileSystem: delete/list status/rename/mkdir APIs
[
https://issues.apache.org/jira/browse/HDFS-12572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16193760#comment-16193760
]
Chen Liang commented on HDFS-12572:
---
Thanks [~msingh] for updating the patch! v004 patch looks good to me overall,
some comments below:
*OzoneFileSystem.java*
1. Looks like in the rename, it is after all subdirectories and files are
renamed, the src path gets removed? Seems this means the rename can potentially
generates doubled size of entries of the directory being renamed. Would it be
better (if possible) to rename one key at a time, deletes it, the move to next
one? I'm thinking of adding {{bucket.deleteKey}} around line 267-268, and
remove {{delete(src, true);}} on line 358. Not sure if this is enough though.
2. {{OzoneListingIterator}} make it an inner class of OzoneFileSystem? Because
it seems file system calls delete, list status and rename are and will be the
only use of this class.
3. in mkdir, seems when an inode in the path is a file, mkdir gets rollback
(line 463). Should we also roll back if createDir fails (line 473)
4. "Directory is represented using empty key with no value.” did you mean root
directory?
5. OzoneListingIterator#iterate and processKey() both return a boolean but
seems nowhere depends on this return value. So this seems a bit unnecessary to
me. Do we need to have processKey and iterate to return anything at all?
*BucketProcessTemplate.java*
Just log a word "Success" does not seem to be very useful, can expand this
message a little bit? Similarly for {{KeyProcessTemplate}}
> Ozone: OzoneFileSystem: delete/list status/rename/mkdir APIs
>
>
> Key: HDFS-12572
> URL: https://issues.apache.org/jira/browse/HDFS-12572
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Mukul Kumar Singh
>Assignee: Mukul Kumar Singh
> Labels: ozoneMerge
> Fix For: HDFS-7240
>
> Attachments: HDFS-12572-HDFS-7240.001.patch,
> HDFS-12572-HDFS-7240.002.patch, HDFS-12572-HDFS-7240.003.patch,
> HDFS-12572-HDFS-7240.004.patch
>
>
> This jira will add the delete/list status/rename/mkdir APIs
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12543) Ozone : allow create key without specifying size
[ https://issues.apache.org/jira/browse/HDFS-12543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chen Liang updated HDFS-12543: -- Attachment: HDFS-12543-HDFS-7240.010.patch Thanks [~nandakumar131] for the review and the comments! Post v010 patch to rebase. Also addressed [~nandakumar131]'s comments, all the other comments are addressed. bq. I understand the optimization done with key size, but if we are going to remove it later why depend on it now? In general case, client opens a key, then start writing to block. So in original design, when a key is opened, a single "pre-allocated block" is also allocated to it, such that client does not need to issue another allocate block call after open key. But turns out, the tricky part is that, it is not that clear how many of such "pre-allocated" blocks should actually be allocated. It could be 0, when client tries to write an empty data array (as in some test cases). It could be some X > 1 blocks, if client already knows more than one will be written. So this is size is used as a "hint" from client to tell how many such "pre-allocated" blocks should be allocated. If client is about to write 0 length data, or client does not know how much will be written, it sets the hint as 0 and no "pre-allocated" happens at open key. This makes "size" only as an optimization that is optional. I would consider "pre-allocated" blocks as potentially very helpful optimization. So I'm not entirely sure whether key size here is completely redundant and should be removed. That's why I wanted to resolve this in another JIRA, and follow up when we have a better idea of how useful this turns out to be, or when we have a better approach to do this. > Ozone : allow create key without specifying size > > > Key: HDFS-12543 > URL: https://issues.apache.org/jira/browse/HDFS-12543 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Chen Liang >Assignee: Chen Liang > Labels: ozoneMerge > Attachments: HDFS-12543-HDFS-7240.001.patch, > HDFS-12543-HDFS-7240.002.patch, HDFS-12543-HDFS-7240.003.patch, > HDFS-12543-HDFS-7240.004.patch, HDFS-12543-HDFS-7240.005.patch, > HDFS-12543-HDFS-7240.006.patch, HDFS-12543-HDFS-7240.007.patch, > HDFS-12543-HDFS-7240.008.patch, HDFS-12543-HDFS-7240.009.patch, > HDFS-12543-HDFS-7240.010.patch > > > Currently when creating a key, it is required to specify the total size of > the key. This makes it inconvenient for the case where a key is created and > data keeps coming and being appended. This JIRA is remove the requirement of > specifying the size on key creation, and allows appending to the key > indefinitely. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12543) Ozone : allow create key without specifying size
[ https://issues.apache.org/jira/browse/HDFS-12543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chen Liang updated HDFS-12543: -- Attachment: HDFS-12543-HDFS-7240.012.patch Update v012 patch to fix checkstyle warning. > Ozone : allow create key without specifying size > > > Key: HDFS-12543 > URL: https://issues.apache.org/jira/browse/HDFS-12543 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Chen Liang >Assignee: Chen Liang > Labels: ozoneMerge > Attachments: HDFS-12543-HDFS-7240.001.patch, > HDFS-12543-HDFS-7240.002.patch, HDFS-12543-HDFS-7240.003.patch, > HDFS-12543-HDFS-7240.004.patch, HDFS-12543-HDFS-7240.005.patch, > HDFS-12543-HDFS-7240.006.patch, HDFS-12543-HDFS-7240.007.patch, > HDFS-12543-HDFS-7240.008.patch, HDFS-12543-HDFS-7240.009.patch, > HDFS-12543-HDFS-7240.010.patch, HDFS-12543-HDFS-7240.011.patch, > HDFS-12543-HDFS-7240.012.patch > > > Currently when creating a key, it is required to specify the total size of > the key. This makes it inconvenient for the case where a key is created and > data keeps coming and being appended. This JIRA is remove the requirement of > specifying the size on key creation, and allows appending to the key > indefinitely. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12558) Ozone: Clarify the meaning of rpc.metrics.percentiles.intervals on KSM/SCM web ui
[ https://issues.apache.org/jira/browse/HDFS-12558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16200982#comment-16200982 ] Chen Liang commented on HDFS-12558: --- Thanks [~elek]! The patch LGTM, could you please just re-submit the patch to see if Jenkins runs? > Ozone: Clarify the meaning of rpc.metrics.percentiles.intervals on KSM/SCM > web ui > - > > Key: HDFS-12558 > URL: https://issues.apache.org/jira/browse/HDFS-12558 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: HDFS-7240 >Reporter: Elek, Marton >Assignee: Elek, Marton > Attachments: HDFS-12558-HDFS-7240.001.patch, after.png, before.png > > > In Ozone (SCM/KSM) web ui we have additional visualization if > rpc.metrics.percentiles.intervals are enabled. > But according to the feedbacks it's a little bit confusing what is it exactly. > I would like to improve it and clarify how does it work. > 1. I will to add a footnote about these are not rolling windows but just > display of the last fixed window. > 2. I would like to rearrange the layout. As the different windows are > independent, I would show them in different lines and group by the intervals > and not by RpcQueueTime/RpcProcessingTime. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12585) Add description for config in Ozone config UI
[
https://issues.apache.org/jira/browse/HDFS-12585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16200977#comment-16200977
]
Chen Liang commented on HDFS-12585:
---
Thanks [~ajayydv] for working on this! Just one comment:
In {{ConfServlet.java}}, seems {{propertyMap}} is only initialized in
{{loadDescriptions()}} which is only called when {{loadDescriptionFromXml}} is
true. So seems to me that if {{loadDescriptionFromXml}} is false, then
{{propertyMap}} remains null, and this line
{{propertyMap.get(key).setValue(config.get(key));}} will then fail, right?
> Add description for config in Ozone config UI
> -
>
> Key: HDFS-12585
> URL: https://issues.apache.org/jira/browse/HDFS-12585
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Affects Versions: HDFS-7240
>Reporter: Ajay Kumar
>Assignee: Ajay Kumar
> Fix For: HDFS-7240
>
> Attachments: HDFS-12585-HDFS-7240.01.patch,
> HDFS-12585-HDFS-7240.02.patch, HDFS-12585-HDFS-7240.03.patch
>
>
> Add description for each config in Ozone config UI
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12504) Ozone: Improve SQLCLI performance
[
https://issues.apache.org/jira/browse/HDFS-12504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16201118#comment-16201118
]
Chen Liang commented on HDFS-12504:
---
Thanks [~yuanbo] for working on this! v001 patch looks pretty good to me. Just
some minor comments:
1. {{void accept(T item) throws IOException;}}, rename accept to something like
batchConsume?
2. "This class is used to batch operate kv" ==> "This class is used to batch
kv operations"
3. Change the log "Insert to sql container db, for container" to something like
"Insert to sql batch for container", and add some log to {{batchIterateStore}}
such that we can see the progress from log.
Also it would be ideal if we can have some simple benchmark results to see the
performance improvement, I will be looking into this too.
> Ozone: Improve SQLCLI performance
> -
>
> Key: HDFS-12504
> URL: https://issues.apache.org/jira/browse/HDFS-12504
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Reporter: Weiwei Yang
>Assignee: Yuanbo Liu
> Labels: performance
> Attachments: HDFS-12504-HDFS-7240.001.patch
>
>
> In my test, my {{ksm.db}} has *3017660* entries with total size of *128mb*,
> SQLCLI tool runs over *2 hours* but still not finish exporting the DB. This
> is because it iterates each entry and inserts that to another sqllite DB
> file, which is not efficient. We need to improve this to be running more
> efficiently on large DB files.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12626) Ozone : delete open key entries that will no longer be closed
[ https://issues.apache.org/jira/browse/HDFS-12626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chen Liang updated HDFS-12626: -- Attachment: HDFS-12626-HDFS-7240.001.patch Post v001 patch. > Ozone : delete open key entries that will no longer be closed > - > > Key: HDFS-12626 > URL: https://issues.apache.org/jira/browse/HDFS-12626 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Chen Liang >Assignee: Chen Liang > Attachments: HDFS-12626-HDFS-7240.001.patch > > > HDFS-12543 introduced the notion of "open key" where when a key is opened, an > open key entry gets persisted, only after client calls a close will this > entry be made visible. One issue is that if the client does not call close > (e.g. failed), then that open key entry will never be deleted from meta data. > This JIRA tracks this issue. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12415) Ozone: TestXceiverClientManager and TestAllocateContainer occasionally fails
[
https://issues.apache.org/jira/browse/HDFS-12415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16202341#comment-16202341
]
Chen Liang commented on HDFS-12415:
---
I looked in this a little bit too. What was happening seems to be that
{{SCMCommonPolicy#chooseDatanodes}} calls
{{nodeManager.getNodes(OzoneProtos.NodeState.HEALTHY);}}, but the returned list
contains a {{null}} datanode id entry. So the {{hasEnoughSpace(d,
sizeRequired)}} call on the null d will fail with NPE. And the returned list
with a null entry is returned by {{SCMNodeManager#getNodes}}, where seems there
is some datanode id in {{healthyNodes}} but not present in {{nodes}} map.
I don't see how could a datanode id be present in {{healthyNodes}} but not in
{{nodes}}, because the first thing of register is to always add that datanode
to {{nodes}}, before {{healthyNodes}}. I can only think of the issue being just
like [~msingh] mentioned, that it is probably due to some unexpected race
condition behaviour when two register calls happen and change the HashMap
{{nodes}} at the same time. So I would +1 on Mukul's change. Additionally, I
ran {{TestXceiverClientManager}} several ten times with v005 patch applied. The
test did not fail.
> Ozone: TestXceiverClientManager and TestAllocateContainer occasionally fails
>
>
> Key: HDFS-12415
> URL: https://issues.apache.org/jira/browse/HDFS-12415
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Affects Versions: HDFS-7240
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
> Attachments: HDFS-12415-HDFS-7240.001.patch,
> HDFS-12415-HDFS-7240.002.patch, HDFS-12415-HDFS-7240.003.patch,
> HDFS-12415-HDFS-7240.004.patch, HDFS-12415-HDFS-7240.005.patch
>
>
> TestXceiverClientManager seems to be occasionally failing in some jenkins
> jobs,
> {noformat}
> java.lang.NullPointerException
> at
> org.apache.hadoop.ozone.scm.node.SCMNodeManager.getNodeStat(SCMNodeManager.java:828)
> at
> org.apache.hadoop.ozone.scm.container.placement.algorithms.SCMCommonPolicy.hasEnoughSpace(SCMCommonPolicy.java:147)
> at
> org.apache.hadoop.ozone.scm.container.placement.algorithms.SCMCommonPolicy.lambda$chooseDatanodes$0(SCMCommonPolicy.java:125)
> {noformat}
> see more from [this
> report|https://builds.apache.org/job/PreCommit-HDFS-Build/21065/testReport/]
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12626) Ozone : delete open key entries that will no longer be closed
[ https://issues.apache.org/jira/browse/HDFS-12626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chen Liang updated HDFS-12626: -- Status: Patch Available (was: Open) > Ozone : delete open key entries that will no longer be closed > - > > Key: HDFS-12626 > URL: https://issues.apache.org/jira/browse/HDFS-12626 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Chen Liang >Assignee: Chen Liang > Attachments: HDFS-12626-HDFS-7240.001.patch > > > HDFS-12543 introduced the notion of "open key" where when a key is opened, an > open key entry gets persisted, only after client calls a close will this > entry be made visible. One issue is that if the client does not call close > (e.g. failed), then that open key entry will never be deleted from meta data. > This JIRA tracks this issue. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12626) Ozone : delete open key entries that will no longer be closed
[ https://issues.apache.org/jira/browse/HDFS-12626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chen Liang updated HDFS-12626: -- Attachment: HDFS-12626-HDFS-7240.002.patch 002 patch to rebase. > Ozone : delete open key entries that will no longer be closed > - > > Key: HDFS-12626 > URL: https://issues.apache.org/jira/browse/HDFS-12626 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Chen Liang >Assignee: Chen Liang > Attachments: HDFS-12626-HDFS-7240.001.patch, > HDFS-12626-HDFS-7240.002.patch > > > HDFS-12543 introduced the notion of "open key" where when a key is opened, an > open key entry gets persisted, only after client calls a close will this > entry be made visible. One issue is that if the client does not call close > (e.g. failed), then that open key entry will never be deleted from meta data. > This JIRA tracks this issue. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12632) Ozone: OzoneFileSystem: Add contract tests to OzoneFileSystem
[ https://issues.apache.org/jira/browse/HDFS-12632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16202460#comment-16202460 ] Chen Liang commented on HDFS-12632: --- Thanks [~msingh] for adding the tests. +1 > Ozone: OzoneFileSystem: Add contract tests to OzoneFileSystem > - > > Key: HDFS-12632 > URL: https://issues.apache.org/jira/browse/HDFS-12632 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Mukul Kumar Singh >Assignee: Mukul Kumar Singh > Labels: ozoneMerge > Fix For: HDFS-7240 > > Attachments: HDFS-12632-HDFS-7240.001.patch > > > HDFS-11704 adds OzoneFileSytem aka (o3) to ozone. This jira will be used to > add ContractTest for the filesystem to Ozone. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12632) Ozone: OzoneFileSystem: Add contract tests to OzoneFileSystem
[ https://issues.apache.org/jira/browse/HDFS-12632?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chen Liang updated HDFS-12632: -- Resolution: Fixed Status: Resolved (was: Patch Available) The failed tests are unrelated. I've committed this to the feature branch, thanks [~msingh] for the contribution and [~anu] for the review! > Ozone: OzoneFileSystem: Add contract tests to OzoneFileSystem > - > > Key: HDFS-12632 > URL: https://issues.apache.org/jira/browse/HDFS-12632 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Mukul Kumar Singh >Assignee: Mukul Kumar Singh > Labels: ozoneMerge > Fix For: HDFS-7240 > > Attachments: HDFS-12632-HDFS-7240.001.patch > > > HDFS-11704 adds OzoneFileSytem aka (o3) to ozone. This jira will be used to > add ContractTest for the filesystem to Ozone. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12585) Add description for config in Ozone config UI
[
https://issues.apache.org/jira/browse/HDFS-12585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16202608#comment-16202608
]
Chen Liang commented on HDFS-12585:
---
Thanks [~ajayydv] for the clarification. Then I think maybe it's better to
either rename loadDescriptionFromXml to something such as descriptionLoaded or
remove this flag and just check if {{propertyMap}} is null.
> Add description for config in Ozone config UI
> -
>
> Key: HDFS-12585
> URL: https://issues.apache.org/jira/browse/HDFS-12585
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Affects Versions: HDFS-7240
>Reporter: Ajay Kumar
>Assignee: Ajay Kumar
> Fix For: HDFS-7240
>
> Attachments: HDFS-12585-HDFS-7240.01.patch,
> HDFS-12585-HDFS-7240.02.patch, HDFS-12585-HDFS-7240.03.patch
>
>
> Add description for each config in Ozone config UI
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12626) Ozone : delete open key entries that will no longer be closed
[ https://issues.apache.org/jira/browse/HDFS-12626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chen Liang updated HDFS-12626: -- Attachment: HDFS-12626-HDFS-7240.003.patch fix asf license header missing in v003 patch. failed tests are unrelated > Ozone : delete open key entries that will no longer be closed > - > > Key: HDFS-12626 > URL: https://issues.apache.org/jira/browse/HDFS-12626 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Chen Liang >Assignee: Chen Liang > Attachments: HDFS-12626-HDFS-7240.001.patch, > HDFS-12626-HDFS-7240.002.patch, HDFS-12626-HDFS-7240.003.patch > > > HDFS-12543 introduced the notion of "open key" where when a key is opened, an > open key entry gets persisted, only after client calls a close will this > entry be made visible. One issue is that if the client does not call close > (e.g. failed), then that open key entry will never be deleted from meta data. > This JIRA tracks this issue. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12626) Ozone : delete open key entries that will no longer be closed
[ https://issues.apache.org/jira/browse/HDFS-12626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chen Liang updated HDFS-12626: -- Attachment: HDFS-12626-HDFS-7240.004.patch Somehow Jenkins ran on v002 patch again, resubmit v003 patch as v004 to trigger another run. > Ozone : delete open key entries that will no longer be closed > - > > Key: HDFS-12626 > URL: https://issues.apache.org/jira/browse/HDFS-12626 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Chen Liang >Assignee: Chen Liang > Attachments: HDFS-12626-HDFS-7240.001.patch, > HDFS-12626-HDFS-7240.002.patch, HDFS-12626-HDFS-7240.003.patch, > HDFS-12626-HDFS-7240.004.patch > > > HDFS-12543 introduced the notion of "open key" where when a key is opened, an > open key entry gets persisted, only after client calls a close will this > entry be made visible. One issue is that if the client does not call close > (e.g. failed), then that open key entry will never be deleted from meta data. > This JIRA tracks this issue. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12804) Use slf4j instead of log4j in FSEditLog
[
https://issues.apache.org/jira/browse/HDFS-12804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16252875#comment-16252875
]
Chen Liang commented on HDFS-12804:
---
Thanks for working on this [~msingh]! The patch LGTM, just one thing. Could you
please verify the failed tests are unrelated? Because I see {{TestUnbuffer}}
failed because it expects some error message.
> Use slf4j instead of log4j in FSEditLog
> ---
>
> Key: HDFS-12804
> URL: https://issues.apache.org/jira/browse/HDFS-12804
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs
>Reporter: Mukul Kumar Singh
>Assignee: Mukul Kumar Singh
> Attachments: HDFS-12804.001.patch, HDFS-12804.002.patch,
> HDFS-12804.003.patch
>
>
> FSEditLog uses log4j, this jira will update the logging to use sl4j.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12000) Ozone: Container : Add key versioning support-1
[ https://issues.apache.org/jira/browse/HDFS-12000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chen Liang updated HDFS-12000: -- Attachment: HDFS-12000-HDFS-7240.007.patch update v007 patch. rewrote many places as lots of code has changed since previous patch. post v007 patch to trigger Jenkins, still doing more testing locally. Please note that this is still work in progress. This patch only focuses on server side. Specifically, this patch adds versioning to blocks and persistent the information of the older versions to meta store. But from client side, the read and write are hidden from versioning i.e. write still always rewrites the whole key, and read still always reads only the most recently committed version of the key. Will follow up with another JIRA for more read/write change. > Ozone: Container : Add key versioning support-1 > --- > > Key: HDFS-12000 > URL: https://issues.apache.org/jira/browse/HDFS-12000 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Anu Engineer >Assignee: Chen Liang > Labels: OzonePostMerge > Attachments: HDFS-12000-HDFS-7240.001.patch, > HDFS-12000-HDFS-7240.002.patch, HDFS-12000-HDFS-7240.003.patch, > HDFS-12000-HDFS-7240.004.patch, HDFS-12000-HDFS-7240.005.patch, > HDFS-12000-HDFS-7240.007.patch, OzoneVersion.001.pdf > > > The rest interface of ozone supports versioning of keys. This support comes > from the containers and how chunks are managed to support this feature. This > JIRA tracks that feature. Will post a detailed design doc so that we can talk > about this feature. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12877) Add open(PathHandle) with default buffersize
[
https://issues.apache.org/jira/browse/HDFS-12877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16273190#comment-16273190
]
Chen Liang commented on HDFS-12877:
---
The changes looks to me, but the failed tests seem related. Could you please
take a look?
> Add open(PathHandle) with default buffersize
>
>
> Key: HDFS-12877
> URL: https://issues.apache.org/jira/browse/HDFS-12877
> Project: Hadoop HDFS
> Issue Type: Improvement
>Affects Versions: 3.1.0
>Reporter: Chris Douglas
>Assignee: Chris Douglas
>Priority: Trivial
> Attachments: HDFS-12877.00.patch, HDFS-12877.01.patch
>
>
> HDFS-7878 added an overload for {{FileSystem::open}} that requires the user
> to provide a buffer size when opening by {{PathHandle}}. Similar to
> {{open(Path)}}, it'd be convenient to have another overload that takes the
> default from the config.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12838) Ozone: Optimize number of allocated block rpc by aggregating multiple block allocation requests
[
https://issues.apache.org/jira/browse/HDFS-12838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16273354#comment-16273354
]
Chen Liang commented on HDFS-12838:
---
Thanks for working on this [~msingh]! I think this is a very good improvement.
Looks pretty good to me overall. Just one comment, seems to me when
{{KeyManagerImpl#openKey}} passes in a {{requestedSize}} of 0, then it ends up
making an unnecessary call that does nothing. Maybe we should check and skip
this case, also for better clearance of code.
> Ozone: Optimize number of allocated block rpc by aggregating multiple block
> allocation requests
> ---
>
> Key: HDFS-12838
> URL: https://issues.apache.org/jira/browse/HDFS-12838
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Mukul Kumar Singh
>Assignee: Mukul Kumar Singh
> Fix For: HDFS-7240
>
> Attachments: HDFS-12838-HDFS-7240.001.patch,
> HDFS-12838-HDFS-7240.002.patch, HDFS-12838-HDFS-7240.003.patch
>
>
> Currently KeySpaceManager allocates multiple blocks by sending multiple block
> allocation requests over the RPC. This can be optimized to aggregate multiple
> block allocation request over one rpc.
> {code}
> while (requestedSize > 0) {
> long allocateSize = Math.min(scmBlockSize, requestedSize);
> AllocatedBlock allocatedBlock =
> scmBlockClient.allocateBlock(allocateSize, type, factor);
> KsmKeyLocationInfo subKeyInfo = new KsmKeyLocationInfo.Builder()
> .setContainerName(allocatedBlock.getPipeline().getContainerName())
> .setBlockID(allocatedBlock.getKey())
> .setShouldCreateContainer(allocatedBlock.getCreateContainer())
> .setIndex(idx++)
> .setLength(allocateSize)
> .setOffset(0)
> .build();
> locations.add(subKeyInfo);
> requestedSize -= allocateSize;
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12879) Ozone : add scm init command to document.
[
https://issues.apache.org/jira/browse/HDFS-12879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chen Liang updated HDFS-12879:
--
Labels: newbie (was: )
> Ozone : add scm init command to document.
> -
>
> Key: HDFS-12879
> URL: https://issues.apache.org/jira/browse/HDFS-12879
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: ozone
>Reporter: Chen Liang
>Priority: Minor
> Labels: newbie
>
> When an Ozone cluster is initialized, before starting SCM through {{hdfs
> --daemon start scm}}, the command {{hdfs scm -init}} needs to be called
> first. But seems this command is not being documented. We should add this
> note to document.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12879) Ozone : add scm init command to document.
[
https://issues.apache.org/jira/browse/HDFS-12879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chen Liang updated HDFS-12879:
--
Issue Type: Sub-task (was: Improvement)
Parent: HDFS-7240
> Ozone : add scm init command to document.
> -
>
> Key: HDFS-12879
> URL: https://issues.apache.org/jira/browse/HDFS-12879
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Reporter: Chen Liang
>Priority: Minor
> Labels: newbie
>
> When an Ozone cluster is initialized, before starting SCM through {{hdfs
> --daemon start scm}}, the command {{hdfs scm -init}} needs to be called
> first. But seems this command is not being documented. We should add this
> note to document.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12879) Ozone : add scm init command to document.
[
https://issues.apache.org/jira/browse/HDFS-12879?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16273609#comment-16273609
]
Chen Liang commented on HDFS-12879:
---
There is another typo in {{OzoneCommandShell.md}}, which I think can be just
part of this fix : there is {{-listtBucket}} which should be {{-listBucket}}
instead.
> Ozone : add scm init command to document.
> -
>
> Key: HDFS-12879
> URL: https://issues.apache.org/jira/browse/HDFS-12879
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Reporter: Chen Liang
>Priority: Minor
> Labels: newbie
>
> When an Ozone cluster is initialized, before starting SCM through {{hdfs
> --daemon start scm}}, the command {{hdfs scm -init}} needs to be called
> first. But seems this command is not being documented. We should add this
> note to document.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12877) Add open(PathHandle) with default buffersize
[
https://issues.apache.org/jira/browse/HDFS-12877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16273618#comment-16273618
]
Chen Liang commented on HDFS-12877:
---
Thanks [~chris.douglas] for the quick update! Makes sense to me. +1 on v02
patch.
> Add open(PathHandle) with default buffersize
>
>
> Key: HDFS-12877
> URL: https://issues.apache.org/jira/browse/HDFS-12877
> Project: Hadoop HDFS
> Issue Type: Improvement
>Affects Versions: 3.1.0
>Reporter: Chris Douglas
>Assignee: Chris Douglas
>Priority: Trivial
> Attachments: HDFS-12877.00.patch, HDFS-12877.01.patch,
> HDFS-12877.02.patch
>
>
> HDFS-7878 added an overload for {{FileSystem::open}} that requires the user
> to provide a buffer size when opening by {{PathHandle}}. Similar to
> {{open(Path)}}, it'd be convenient to have another overload that takes the
> default from the config.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12799) Ozone: SCM: Close containers: extend SCMCommandResponseProto with SCMCloseContainerCmdResponseProto
[
https://issues.apache.org/jira/browse/HDFS-12799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16269775#comment-16269775
]
Chen Liang commented on HDFS-12799:
---
Thanks [~elek] for the update! Looks like v002 patch needs to be rebased again.
Also just one minor comment, could you please add some log.debug log to the new
branch in {{HeartbeatEndpointTask#processResponse}}? right before
this.context.addCommand.
> Ozone: SCM: Close containers: extend SCMCommandResponseProto with
> SCMCloseContainerCmdResponseProto
> ---
>
> Key: HDFS-12799
> URL: https://issues.apache.org/jira/browse/HDFS-12799
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Elek, Marton
>Assignee: Elek, Marton
> Attachments: HDFS-12799-HDFS-7240.001.patch,
> HDFS-12799-HDFS-7240.002.patch
>
>
> This issue is about extending the HB response protocol between SCM and DN
> with a command to ask the datanode to close a container. (This is just about
> extending the protocol not about fixing the implementation of SCM tto handle
> the state transitions).
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12799) Ozone: SCM: Close containers: extend SCMCommandResponseProto with SCMCloseContainerCmdResponseProto
[ https://issues.apache.org/jira/browse/HDFS-12799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16274836#comment-16274836 ] Chen Liang commented on HDFS-12799: --- Thanks [~elek] for the update! Could you please fix the checkstyle warnings? +1 after it's fixed. > Ozone: SCM: Close containers: extend SCMCommandResponseProto with > SCMCloseContainerCmdResponseProto > --- > > Key: HDFS-12799 > URL: https://issues.apache.org/jira/browse/HDFS-12799 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Elek, Marton >Assignee: Elek, Marton > Attachments: HDFS-12799-HDFS-7240.001.patch, > HDFS-12799-HDFS-7240.002.patch, HDFS-12799-HDFS-7240.003.patch > > > This issue is about extending the HB response protocol between SCM and DN > with a command to ask the datanode to close a container. (This is just about > extending the protocol not about fixing the implementation of SCM tto handle > the state transitions). -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12925) Ozone: Container : Add key versioning support-2
[ https://issues.apache.org/jira/browse/HDFS-12925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chen Liang updated HDFS-12925: -- Attachment: HDFS-12925-HDFS-7240.004.patch Latest Jenkins build failed again, resubmit v003 patch as v004 patch to trigger another run. > Ozone: Container : Add key versioning support-2 > --- > > Key: HDFS-12925 > URL: https://issues.apache.org/jira/browse/HDFS-12925 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Chen Liang >Assignee: Chen Liang > Attachments: HDFS-12925-HDFS-7240.001.patch, > HDFS-12925-HDFS-7240.002.patch, HDFS-12925-HDFS-7240.003.patch, > HDFS-12925-HDFS-7240.004.patch > > > One component for versioning is assembling read IO vector, (please see 4.2 > section of the [versioning design > doc|https://issues.apache.org/jira/secure/attachment/12877154/OzoneVersion.001.pdf] > under HDFS-12000 for the detail). This JIRA adds the util functions that > takes a list with blocks from different versions and properly generate the > read vector for the requested version. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12925) Ozone: Container : Add key versioning support-2
[ https://issues.apache.org/jira/browse/HDFS-12925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chen Liang updated HDFS-12925: -- Attachment: HDFS-12925-HDFS-7240.001.patch Post v001 patch. > Ozone: Container : Add key versioning support-2 > --- > > Key: HDFS-12925 > URL: https://issues.apache.org/jira/browse/HDFS-12925 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Chen Liang >Assignee: Chen Liang > Attachments: HDFS-12925-HDFS-7240.001.patch > > > One component for versioning is assembling read IO vector, (please see 4.2 > section of the versioning design doc HDFS-12000 for the detail). This JIRA > adds the util functions that takes a list with blocks from different versions > and properly generate the read vector for the requested version. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12925) Ozone: Container : Add key versioning support-2
[ https://issues.apache.org/jira/browse/HDFS-12925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chen Liang updated HDFS-12925: -- Status: Patch Available (was: Open) > Ozone: Container : Add key versioning support-2 > --- > > Key: HDFS-12925 > URL: https://issues.apache.org/jira/browse/HDFS-12925 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Chen Liang >Assignee: Chen Liang > Attachments: HDFS-12925-HDFS-7240.001.patch > > > One component for versioning is assembling read IO vector, (please see 4.2 > section of the versioning design doc HDFS-12000 for the detail). This JIRA > adds the util functions that takes a list with blocks from different versions > and properly generate the read vector for the requested version. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12751) Ozone: SCM: update container allocated size to container db for all the open containers in ContainerStateManager#close
[
https://issues.apache.org/jira/browse/HDFS-12751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16288193#comment-16288193
]
Chen Liang commented on HDFS-12751:
---
Thanks [~nandakumar131] for the clarification! I totally missed {{usedBytes}}.
Then this makes sense. I will upload a patch soon.
> Ozone: SCM: update container allocated size to container db for all the open
> containers in ContainerStateManager#close
> --
>
> Key: HDFS-12751
> URL: https://issues.apache.org/jira/browse/HDFS-12751
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Reporter: Nanda kumar
>Assignee: Chen Liang
>
> Container allocated size is maintained in memory by
> {{ContainerStateManager}}, this has to be updated in container db when we
> shutdown SCM. {{ContainerStateManager#close}} will be called during SCM
> shutdown, so updating allocated size for all the open containers should be
> done here.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12000) Ozone: Container : Add key versioning support-1
[
https://issues.apache.org/jira/browse/HDFS-12000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chen Liang updated HDFS-12000:
--
Attachment: HDFS-12000-HDFS-7240.011.patch
Fix the checkstyle warning. The javadoc and the asf license issues seem
unrelated, the failed tests also seem unrelated. All passed locally except for
the two consistently failing tests {{TestUnbuffer}} and {{TestBalancerRPCDelay}}
> Ozone: Container : Add key versioning support-1
> ---
>
> Key: HDFS-12000
> URL: https://issues.apache.org/jira/browse/HDFS-12000
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Anu Engineer
>Assignee: Chen Liang
> Labels: OzonePostMerge
> Attachments: HDFS-12000-HDFS-7240.001.patch,
> HDFS-12000-HDFS-7240.002.patch, HDFS-12000-HDFS-7240.003.patch,
> HDFS-12000-HDFS-7240.004.patch, HDFS-12000-HDFS-7240.005.patch,
> HDFS-12000-HDFS-7240.007.patch, HDFS-12000-HDFS-7240.008.patch,
> HDFS-12000-HDFS-7240.009.patch, HDFS-12000-HDFS-7240.010.patch,
> HDFS-12000-HDFS-7240.011.patch, OzoneVersion.001.pdf
>
>
> The rest interface of ozone supports versioning of keys. This support comes
> from the containers and how chunks are managed to support this feature. This
> JIRA tracks that feature. Will post a detailed design doc so that we can talk
> about this feature.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12751) Ozone: SCM: update container allocated size to container db for all the open containers in ContainerStateManager#close
[
https://issues.apache.org/jira/browse/HDFS-12751?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chen Liang updated HDFS-12751:
--
Attachment: HDFS-12751-HDFS-7240.001.patch
Post v001 patch.
> Ozone: SCM: update container allocated size to container db for all the open
> containers in ContainerStateManager#close
> --
>
> Key: HDFS-12751
> URL: https://issues.apache.org/jira/browse/HDFS-12751
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Reporter: Nanda kumar
>Assignee: Chen Liang
> Attachments: HDFS-12751-HDFS-7240.001.patch
>
>
> Container allocated size is maintained in memory by
> {{ContainerStateManager}}, this has to be updated in container db when we
> shutdown SCM. {{ContainerStateManager#close}} will be called during SCM
> shutdown, so updating allocated size for all the open containers should be
> done here.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12751) Ozone: SCM: update container allocated size to container db for all the open containers in ContainerStateManager#close
[
https://issues.apache.org/jira/browse/HDFS-12751?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chen Liang updated HDFS-12751:
--
Status: Patch Available (was: In Progress)
> Ozone: SCM: update container allocated size to container db for all the open
> containers in ContainerStateManager#close
> --
>
> Key: HDFS-12751
> URL: https://issues.apache.org/jira/browse/HDFS-12751
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Reporter: Nanda kumar
>Assignee: Chen Liang
> Attachments: HDFS-12751-HDFS-7240.001.patch
>
>
> Container allocated size is maintained in memory by
> {{ContainerStateManager}}, this has to be updated in container db when we
> shutdown SCM. {{ContainerStateManager#close}} will be called during SCM
> shutdown, so updating allocated size for all the open containers should be
> done here.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12000) Ozone: Container : Add key versioning support-1
[
https://issues.apache.org/jira/browse/HDFS-12000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16288500#comment-16288500
]
Chen Liang commented on HDFS-12000:
---
Thansk [~xyao] for checking the tests! I ran locally the tests you mentioned
and the failed tests in the latest Jenkins run. All tests passed except for
{{TestOzoneRpcClient.testPutKeyRatisThreeNodes}} which fails even without the
patch.
> Ozone: Container : Add key versioning support-1
> ---
>
> Key: HDFS-12000
> URL: https://issues.apache.org/jira/browse/HDFS-12000
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Anu Engineer
>Assignee: Chen Liang
> Labels: OzonePostMerge
> Attachments: HDFS-12000-HDFS-7240.001.patch,
> HDFS-12000-HDFS-7240.002.patch, HDFS-12000-HDFS-7240.003.patch,
> HDFS-12000-HDFS-7240.004.patch, HDFS-12000-HDFS-7240.005.patch,
> HDFS-12000-HDFS-7240.007.patch, HDFS-12000-HDFS-7240.008.patch,
> HDFS-12000-HDFS-7240.009.patch, HDFS-12000-HDFS-7240.010.patch,
> HDFS-12000-HDFS-7240.011.patch, OzoneVersion.001.pdf
>
>
> The rest interface of ozone supports versioning of keys. This support comes
> from the containers and how chunks are managed to support this feature. This
> JIRA tracks that feature. Will post a detailed design doc so that we can talk
> about this feature.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12626) Ozone : delete open key entries that will no longer be closed
[ https://issues.apache.org/jira/browse/HDFS-12626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chen Liang updated HDFS-12626: -- Attachment: HDFS-12626-HDFS-7240.006.patch Thanks [~xyao] for the review! Post v006 patch to fix the test. > Ozone : delete open key entries that will no longer be closed > - > > Key: HDFS-12626 > URL: https://issues.apache.org/jira/browse/HDFS-12626 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Chen Liang >Assignee: Chen Liang > Attachments: HDFS-12626-HDFS-7240.001.patch, > HDFS-12626-HDFS-7240.002.patch, HDFS-12626-HDFS-7240.003.patch, > HDFS-12626-HDFS-7240.004.patch, HDFS-12626-HDFS-7240.005.patch, > HDFS-12626-HDFS-7240.006.patch > > > HDFS-12543 introduced the notion of "open key" where when a key is opened, an > open key entry gets persisted, only after client calls a close will this > entry be made visible. One issue is that if the client does not call close > (e.g. failed), then that open key entry will never be deleted from meta data. > This JIRA tracks this issue. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-12925) Ozone: Container : Add key versioning support-2
Chen Liang created HDFS-12925: - Summary: Ozone: Container : Add key versioning support-2 Key: HDFS-12925 URL: https://issues.apache.org/jira/browse/HDFS-12925 Project: Hadoop HDFS Issue Type: Sub-task Components: ozone Affects Versions: HDFS-7240 Reporter: Chen Liang Assignee: Chen Liang One component for versioning is assembling read IO vector, (please see 4.2 section of the versioning design doc HDFS-12000 for the detail). This JIRA adds the util functions that takes a list with blocks from different versions and properly generate the read vector for the requested version. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12881) Output streams closed with IOUtils suppressing write errors
[
https://issues.apache.org/jira/browse/HDFS-12881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chen Liang updated HDFS-12881:
--
Fix Version/s: (was: 3.0.1)
> Output streams closed with IOUtils suppressing write errors
> ---
>
> Key: HDFS-12881
> URL: https://issues.apache.org/jira/browse/HDFS-12881
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Jason Lowe
>Assignee: Ajay Kumar
> Fix For: 3.1.0
>
> Attachments: HDFS-12881.001.patch, HDFS-12881.002.patch,
> HDFS-12881.003.patch, HDFS-12881.004.patch
>
>
> There are a few places in HDFS code that are closing an output stream with
> IOUtils.cleanupWithLogger like this:
> {code}
> try {
> ...write to outStream...
> } finally {
> IOUtils.cleanupWithLogger(LOG, outStream);
> }
> {code}
> This suppresses any IOException that occurs during the close() method which
> could lead to partial/corrupted output without throwing a corresponding
> exception. The code should either use try-with-resources or explicitly close
> the stream within the try block so the exception thrown during close() is
> properly propagated as exceptions during write operations are.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12881) Output streams closed with IOUtils suppressing write errors
[
https://issues.apache.org/jira/browse/HDFS-12881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chen Liang updated HDFS-12881:
--
Fix Version/s: 3.0.1
> Output streams closed with IOUtils suppressing write errors
> ---
>
> Key: HDFS-12881
> URL: https://issues.apache.org/jira/browse/HDFS-12881
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Jason Lowe
>Assignee: Ajay Kumar
> Fix For: 3.1.0, 3.0.1
>
> Attachments: HDFS-12881.001.patch, HDFS-12881.002.patch,
> HDFS-12881.003.patch, HDFS-12881.004.patch
>
>
> There are a few places in HDFS code that are closing an output stream with
> IOUtils.cleanupWithLogger like this:
> {code}
> try {
> ...write to outStream...
> } finally {
> IOUtils.cleanupWithLogger(LOG, outStream);
> }
> {code}
> This suppresses any IOException that occurs during the close() method which
> could lead to partial/corrupted output without throwing a corresponding
> exception. The code should either use try-with-resources or explicitly close
> the stream within the try block so the exception thrown during close() is
> properly propagated as exceptions during write operations are.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12925) Ozone: Container : Add key versioning support-2
[ https://issues.apache.org/jira/browse/HDFS-12925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chen Liang updated HDFS-12925: -- Attachment: HDFS-12925-HDFS-7240.002.patch The output link from Jenkins seem dead, not sure why the build failed, my build worked fine, resubmitting as v002 patch. > Ozone: Container : Add key versioning support-2 > --- > > Key: HDFS-12925 > URL: https://issues.apache.org/jira/browse/HDFS-12925 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Chen Liang >Assignee: Chen Liang > Attachments: HDFS-12925-HDFS-7240.001.patch, > HDFS-12925-HDFS-7240.002.patch > > > One component for versioning is assembling read IO vector, (please see 4.2 > section of the versioning design doc HDFS-12000 for the detail). This JIRA > adds the util functions that takes a list with blocks from different versions > and properly generate the read vector for the requested version. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12917) Fix description errors in testErasureCodingConf.xml
[ https://issues.apache.org/jira/browse/HDFS-12917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16291742#comment-16291742 ] Chen Liang commented on HDFS-12917: --- Thanks [~candychencan] for the catch, just one NIT, "which have an..." to "which has an..."? > Fix description errors in testErasureCodingConf.xml > --- > > Key: HDFS-12917 > URL: https://issues.apache.org/jira/browse/HDFS-12917 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: chencan > Attachments: HADOOP-12917.patch > > > In testErasureCodingConf.xml,there are two case's description may be > "getPolicy : get EC policy information at specified path, whick have an EC > Policy". -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12925) Ozone: Container : Add key versioning support-2
[ https://issues.apache.org/jira/browse/HDFS-12925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chen Liang updated HDFS-12925: -- Description: One component for versioning is assembling read IO vector, (please see 4.2 section of the [versioning design doc|https://issues.apache.org/jira/secure/attachment/12877154/OzoneVersion.001.pdf] under HDFS-12000 for the detail). This JIRA adds the util functions that takes a list with blocks from different versions and properly generate the read vector for the requested version. (was: One component for versioning is assembling read IO vector, (please see 4.2 section of the versioning design doc HDFS-12000 for the detail). This JIRA adds the util functions that takes a list with blocks from different versions and properly generate the read vector for the requested version.) > Ozone: Container : Add key versioning support-2 > --- > > Key: HDFS-12925 > URL: https://issues.apache.org/jira/browse/HDFS-12925 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Chen Liang >Assignee: Chen Liang > Attachments: HDFS-12925-HDFS-7240.001.patch, > HDFS-12925-HDFS-7240.002.patch > > > One component for versioning is assembling read IO vector, (please see 4.2 > section of the [versioning design > doc|https://issues.apache.org/jira/secure/attachment/12877154/OzoneVersion.001.pdf] > under HDFS-12000 for the detail). This JIRA adds the util functions that > takes a list with blocks from different versions and properly generate the > read vector for the requested version. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12932) Confusing LOG message for block replication
[
https://issues.apache.org/jira/browse/HDFS-12932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16295772#comment-16295772
]
Chen Liang commented on HDFS-12932:
---
Thanks [~csun] for the catch! I think maybe it's better to add a third branch
to catch the {{==}} case. Just log a message saying it remains unchanged at
that value. Because the current code always out a message for all three cases
of {{=}} {{<}} and {{>}}. I think it's probably better we don't change the
syntax here.
> Confusing LOG message for block replication
> ---
>
> Key: HDFS-12932
> URL: https://issues.apache.org/jira/browse/HDFS-12932
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs
>Affects Versions: 2.8.3
>Reporter: Chao Sun
>Assignee: Chao Sun
>Priority: Minor
> Attachments: HDFS-12932.0.patch
>
>
> In our cluster we see large number of log messages such as the following:
> {code}
> 2017-12-15 22:55:54,603 INFO
> org.apache.hadoop.hdfs.server.namenode.FSDirectory: Increasing replication
> from 3 to 3 for
> {code}
> This is a little confusing since "from 3 to 3" is not "increasing". Digging
> into it, it seems related to this piece of code:
> {code}
> if (oldBR != -1) {
> if (oldBR > targetReplication) {
> FSDirectory.LOG.info("Decreasing replication from {} to {} for {}",
> oldBR, targetReplication, iip.getPath());
> } else {
> FSDirectory.LOG.info("Increasing replication from {} to {} for {}",
> oldBR, targetReplication, iip.getPath());
> }
> }
> {code}
> Perhaps a {{oldBR == targetReplication}} case is missing?
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12925) Ozone: Container : Add key versioning support-2
[
https://issues.apache.org/jira/browse/HDFS-12925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chen Liang updated HDFS-12925:
--
Attachment: HDFS-12925-HDFS-7240.003.patch
v003 patch fixes the checkstyle and javadoc issue, findbug warnings are not
introduced by this patch. The failed tests all passed locally, except for the
consistently failing test {{TestOzoneRpcClient}}
> Ozone: Container : Add key versioning support-2
> ---
>
> Key: HDFS-12925
> URL: https://issues.apache.org/jira/browse/HDFS-12925
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Chen Liang
>Assignee: Chen Liang
> Attachments: HDFS-12925-HDFS-7240.001.patch,
> HDFS-12925-HDFS-7240.002.patch, HDFS-12925-HDFS-7240.003.patch
>
>
> One component for versioning is assembling read IO vector, (please see 4.2
> section of the [versioning design
> doc|https://issues.apache.org/jira/secure/attachment/12877154/OzoneVersion.001.pdf]
> under HDFS-12000 for the detail). This JIRA adds the util functions that
> takes a list with blocks from different versions and properly generate the
> read vector for the requested version.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-12917) Fix description errors in testErasureCodingConf.xml
[ https://issues.apache.org/jira/browse/HDFS-12917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chen Liang reassigned HDFS-12917: - Assignee: chencan > Fix description errors in testErasureCodingConf.xml > --- > > Key: HDFS-12917 > URL: https://issues.apache.org/jira/browse/HDFS-12917 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: chencan >Assignee: chencan > Attachments: HADOOP-12917.002.patch, HADOOP-12917.patch > > > In testErasureCodingConf.xml,there are two case's description may be > "getPolicy : get EC policy information at specified path, whick have an EC > Policy". -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12917) Fix description errors in testErasureCodingConf.xml
[ https://issues.apache.org/jira/browse/HDFS-12917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16293301#comment-16293301 ] Chen Liang commented on HDFS-12917: --- Thanks [~candychencan] for the updated patch! +1 on v002 patch, I've committed to trunk, (and I've changed the assignee of this JIRA to you). Thanks for your contribution! > Fix description errors in testErasureCodingConf.xml > --- > > Key: HDFS-12917 > URL: https://issues.apache.org/jira/browse/HDFS-12917 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: chencan >Assignee: chencan > Attachments: HADOOP-12917.002.patch, HADOOP-12917.patch > > > In testErasureCodingConf.xml,there are two case's description may be > "getPolicy : get EC policy information at specified path, whick have an EC > Policy". -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12917) Fix description errors in testErasureCodingConf.xml
[ https://issues.apache.org/jira/browse/HDFS-12917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chen Liang updated HDFS-12917: -- Resolution: Fixed Target Version/s: 3.1.0 Status: Resolved (was: Patch Available) > Fix description errors in testErasureCodingConf.xml > --- > > Key: HDFS-12917 > URL: https://issues.apache.org/jira/browse/HDFS-12917 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: chencan >Assignee: chencan > Attachments: HADOOP-12917.002.patch, HADOOP-12917.patch > > > In testErasureCodingConf.xml,there are two case's description may be > "getPolicy : get EC policy information at specified path, whick have an EC > Policy". -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12799) Ozone: SCM: Close containers: extend SCMCommandResponseProto with SCMCloseContainerCmdResponseProto
[ https://issues.apache.org/jira/browse/HDFS-12799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16293035#comment-16293035 ] Chen Liang commented on HDFS-12799: --- Thanks [~elek] for the update! Looks like there were compilation failures, might need to be rebased, would you mind taking a look? Thanks > Ozone: SCM: Close containers: extend SCMCommandResponseProto with > SCMCloseContainerCmdResponseProto > --- > > Key: HDFS-12799 > URL: https://issues.apache.org/jira/browse/HDFS-12799 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Elek, Marton >Assignee: Elek, Marton > Attachments: HDFS-12799-HDFS-7240.001.patch, > HDFS-12799-HDFS-7240.002.patch, HDFS-12799-HDFS-7240.003.patch, > HDFS-12799-HDFS-7240.004.patch > > > This issue is about extending the HB response protocol between SCM and DN > with a command to ask the datanode to close a container. (This is just about > extending the protocol not about fixing the implementation of SCM tto handle > the state transitions). -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-12925) Ozone: Container : Add key versioning support-2
[
https://issues.apache.org/jira/browse/HDFS-12925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16293273#comment-16293273
]
Chen Liang edited comment on HDFS-12925 at 12/15/17 9:31 PM:
-
v003 patch fixes the checkstyle and javadoc issue, findbug warnings are not
introduced by this patch. The failed tests all passed locally, except for
{{TestOzoneRpcClient}} which fails consistently even without the patch
was (Author: vagarychen):
v003 patch fixes the checkstyle and javadoc issue, findbug warnings are not
introduced by this patch. The failed tests all passed locally, except for the
consistently failing test {{TestOzoneRpcClient}}
> Ozone: Container : Add key versioning support-2
> ---
>
> Key: HDFS-12925
> URL: https://issues.apache.org/jira/browse/HDFS-12925
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Chen Liang
>Assignee: Chen Liang
> Attachments: HDFS-12925-HDFS-7240.001.patch,
> HDFS-12925-HDFS-7240.002.patch, HDFS-12925-HDFS-7240.003.patch
>
>
> One component for versioning is assembling read IO vector, (please see 4.2
> section of the [versioning design
> doc|https://issues.apache.org/jira/secure/attachment/12877154/OzoneVersion.001.pdf]
> under HDFS-12000 for the detail). This JIRA adds the util functions that
> takes a list with blocks from different versions and properly generate the
> read vector for the requested version.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12751) Ozone: SCM: update container allocated size to container db for all the open containers in ContainerStateManager#close
[
https://issues.apache.org/jira/browse/HDFS-12751?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chen Liang updated HDFS-12751:
--
Attachment: HDFS-12751-HDFS-7240.002.patch
Fix checkstyle, the failed tests are unrelated, and passed in local run
> Ozone: SCM: update container allocated size to container db for all the open
> containers in ContainerStateManager#close
> --
>
> Key: HDFS-12751
> URL: https://issues.apache.org/jira/browse/HDFS-12751
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Reporter: Nanda kumar
>Assignee: Chen Liang
> Attachments: HDFS-12751-HDFS-7240.001.patch,
> HDFS-12751-HDFS-7240.002.patch
>
>
> Container allocated size is maintained in memory by
> {{ContainerStateManager}}, this has to be updated in container db when we
> shutdown SCM. {{ContainerStateManager#close}} will be called during SCM
> shutdown, so updating allocated size for all the open containers should be
> done here.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12265) Ozone : better handling of operation fail due to chill mode
[ https://issues.apache.org/jira/browse/HDFS-12265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chen Liang updated HDFS-12265: -- Release Note: (was: Looks like this has been handled as part of HDFS-12387, close this JIRA.) > Ozone : better handling of operation fail due to chill mode > --- > > Key: HDFS-12265 > URL: https://issues.apache.org/jira/browse/HDFS-12265 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Reporter: Chen Liang >Priority: Minor > Labels: OzonePostMerge > > Currently if someone tries to create a container while SCM is in chill mode, > there will be exception of INTERNAL_ERROR, which is not very informative and > can be confusing for debugging. > We should make it easier to identify problems caused by chill mode. For > example, we may detect if SCM is in chill mode and report back to client in > some way, such that the client can backup and try again later. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-12265) Ozone : better handling of operation fail due to chill mode
[ https://issues.apache.org/jira/browse/HDFS-12265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chen Liang resolved HDFS-12265. --- Resolution: Fixed Release Note: Looks like this has been handled as part of HDFS-12387, close this JIRA. > Ozone : better handling of operation fail due to chill mode > --- > > Key: HDFS-12265 > URL: https://issues.apache.org/jira/browse/HDFS-12265 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Reporter: Chen Liang >Priority: Minor > Labels: OzonePostMerge > > Currently if someone tries to create a container while SCM is in chill mode, > there will be exception of INTERNAL_ERROR, which is not very informative and > can be confusing for debugging. > We should make it easier to identify problems caused by chill mode. For > example, we may detect if SCM is in chill mode and report back to client in > some way, such that the client can backup and try again later. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12265) Ozone : better handling of operation fail due to chill mode
[ https://issues.apache.org/jira/browse/HDFS-12265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16289691#comment-16289691 ] Chen Liang commented on HDFS-12265: --- Looks like this has been handled as part of HDFS-12387, close this JIRA. > Ozone : better handling of operation fail due to chill mode > --- > > Key: HDFS-12265 > URL: https://issues.apache.org/jira/browse/HDFS-12265 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Reporter: Chen Liang >Priority: Minor > Labels: OzonePostMerge > > Currently if someone tries to create a container while SCM is in chill mode, > there will be exception of INTERNAL_ERROR, which is not very informative and > can be confusing for debugging. > We should make it easier to identify problems caused by chill mode. For > example, we may detect if SCM is in chill mode and report back to client in > some way, such that the client can backup and try again later. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12000) Ozone: Container : Add key versioning support-1
[ https://issues.apache.org/jira/browse/HDFS-12000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chen Liang updated HDFS-12000: -- Attachment: HDFS-12000-HDFS-7240.012.patch v012 patch to rebase. > Ozone: Container : Add key versioning support-1 > --- > > Key: HDFS-12000 > URL: https://issues.apache.org/jira/browse/HDFS-12000 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Anu Engineer >Assignee: Chen Liang > Labels: OzonePostMerge > Attachments: HDFS-12000-HDFS-7240.001.patch, > HDFS-12000-HDFS-7240.002.patch, HDFS-12000-HDFS-7240.003.patch, > HDFS-12000-HDFS-7240.004.patch, HDFS-12000-HDFS-7240.005.patch, > HDFS-12000-HDFS-7240.007.patch, HDFS-12000-HDFS-7240.008.patch, > HDFS-12000-HDFS-7240.009.patch, HDFS-12000-HDFS-7240.010.patch, > HDFS-12000-HDFS-7240.011.patch, HDFS-12000-HDFS-7240.012.patch, > OzoneVersion.001.pdf > > > The rest interface of ozone supports versioning of keys. This support comes > from the containers and how chunks are managed to support this feature. This > JIRA tracks that feature. Will post a detailed design doc so that we can talk > about this feature. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-12940) Ozone: KSM: TestKeySpaceManager#testExpiredOpenKey fails occasionally
[
https://issues.apache.org/jira/browse/HDFS-12940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16297507#comment-16297507
]
Chen Liang edited comment on HDFS-12940 at 12/19/17 10:06 PM:
--
Thanks [~nandakumar131] for the catch! But when I ran test locally, I still
occasionally got assertion on line 1140, which implies that a key that should
have been deleted still exists. I suspect that in addition to the issue you
fixed, there can be another issue due to other tests: since the cleanup service
got started at the start of minicluster, as multiple tests went on, the exact
time of clean up check can be arbitrary. So it's possible that the second
{{Thread.sleep(2000);}} may not actually include a cleanup check, given that
the clean up check interval is set to 3 sec.
If this is true, then I think there are two ways to fix:
1. instead of sleeping for 2000, first sleep 2000, then create key 5, then
sleep another 2000. Then for key1~4, the sleep is 4000, which is guaranteed to
include a check that cleans up key 1~4. While for key 5, the sleep is 2000, so
even if there is a check after its creation, it guarantees the check won't
remove key because key 5 is guaranteed to have lived <2000 by the checking
time. We may also want to decrease the check interval and cleanup threshold to
make this test run faster.
2. make testExpiredOpenKey a separate unit test class, then we avoid the impact
from the other tests.
Either way looks good to me.
was (Author: vagarychen):
Thanks [~nandakumar131] for the catch! But when I ran test locally, I still
occasionally got assertion on line 1140, which implies that a key that should
have been deleted still exists. I suspect that in addition to the issue you
fixed, there can be another issue due to other tests: since the cleanup service
got started at the start of minicluster, as multiple tests went on, the exact
time of clean up check can be arbitrary. So it's possible that the second
{{Thread.sleep(2000);}} may not actually include a cleanup check, given that
the clean up check interval is set to 3 sec.
If this is true, then I think there are two ways to fix:
1. instead of sleeping for 2000, first sleep 2000, then create key 5, then
sleep another 2000. Then for key1~4, the sleep is 4000, which is guaranteed to
have included a check. While for key 5, the sleep is 2000, so even if there is
a check after its creation, it guarantees the check won't remove key because
key 5 is guaranteed to have lived <2000 by the checking time. We may also want
to decrease the check interval and cleanup threshold to make this test run
faster.
2. make testExpiredOpenKey a separate unit test class, then we avoid the impact
from the other tests.
Either way looks good to me.
> Ozone: KSM: TestKeySpaceManager#testExpiredOpenKey fails occasionally
> -
>
> Key: HDFS-12940
> URL: https://issues.apache.org/jira/browse/HDFS-12940
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Nanda kumar
>Assignee: Nanda kumar
> Attachments: HDFS-12940-HDFS-7240.000.patch
>
>
> {{TestKeySpaceManager#testExpiredOpenKey}} is flaky.
> In {{testExpiredOpenKey}} we are opening four keys for writing and wait for
> them to expire (without committing). Verification/Assertion is done by
> querying {{MiniOzoneCluster}} and matching the count. Since the {{cluster}}
> instance of {{MiniOzoneCluster}} is shared between test-cases in
> {{TestKeySpaceManager}}, we should not rely on the count. The verification
> should only happen by matching the keyNames and not with the count.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12799) Ozone: SCM: Close containers: extend SCMCommandResponseProto with SCMCloseContainerCmdResponseProto
[ https://issues.apache.org/jira/browse/HDFS-12799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16297297#comment-16297297 ] Chen Liang commented on HDFS-12799: --- Committed to the feature branch, thanks [~elek] for the contribution! > Ozone: SCM: Close containers: extend SCMCommandResponseProto with > SCMCloseContainerCmdResponseProto > --- > > Key: HDFS-12799 > URL: https://issues.apache.org/jira/browse/HDFS-12799 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Elek, Marton >Assignee: Elek, Marton > Fix For: ozone > > Attachments: HDFS-12799-HDFS-7240.001.patch, > HDFS-12799-HDFS-7240.002.patch, HDFS-12799-HDFS-7240.003.patch, > HDFS-12799-HDFS-7240.004.patch, HDFS-12799-HDFS-7240.005.patch > > > This issue is about extending the HB response protocol between SCM and DN > with a command to ask the datanode to close a container. (This is just about > extending the protocol not about fixing the implementation of SCM tto handle > the state transitions). -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12799) Ozone: SCM: Close containers: extend SCMCommandResponseProto with SCMCloseContainerCmdResponseProto
[ https://issues.apache.org/jira/browse/HDFS-12799?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chen Liang updated HDFS-12799: -- Resolution: Fixed Fix Version/s: ozone Status: Resolved (was: Patch Available) > Ozone: SCM: Close containers: extend SCMCommandResponseProto with > SCMCloseContainerCmdResponseProto > --- > > Key: HDFS-12799 > URL: https://issues.apache.org/jira/browse/HDFS-12799 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Elek, Marton >Assignee: Elek, Marton > Fix For: ozone > > Attachments: HDFS-12799-HDFS-7240.001.patch, > HDFS-12799-HDFS-7240.002.patch, HDFS-12799-HDFS-7240.003.patch, > HDFS-12799-HDFS-7240.004.patch, HDFS-12799-HDFS-7240.005.patch > > > This issue is about extending the HB response protocol between SCM and DN > with a command to ask the datanode to close a container. (This is just about > extending the protocol not about fixing the implementation of SCM tto handle > the state transitions). -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12799) Ozone: SCM: Close containers: extend SCMCommandResponseProto with SCMCloseContainerCmdResponseProto
[ https://issues.apache.org/jira/browse/HDFS-12799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16297290#comment-16297290 ] Chen Liang commented on HDFS-12799: --- Thanks [~elek] for the update! +1 on v005 patch, the failed tests and the findbug warning are unrelated. Will commit it shortly. > Ozone: SCM: Close containers: extend SCMCommandResponseProto with > SCMCloseContainerCmdResponseProto > --- > > Key: HDFS-12799 > URL: https://issues.apache.org/jira/browse/HDFS-12799 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Elek, Marton >Assignee: Elek, Marton > Attachments: HDFS-12799-HDFS-7240.001.patch, > HDFS-12799-HDFS-7240.002.patch, HDFS-12799-HDFS-7240.003.patch, > HDFS-12799-HDFS-7240.004.patch, HDFS-12799-HDFS-7240.005.patch > > > This issue is about extending the HB response protocol between SCM and DN > with a command to ask the datanode to close a container. (This is just about > extending the protocol not about fixing the implementation of SCM tto handle > the state transitions). -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12940) Ozone: KSM: TestKeySpaceManager#testExpiredOpenKey fails occasionally
[
https://issues.apache.org/jira/browse/HDFS-12940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16297507#comment-16297507
]
Chen Liang commented on HDFS-12940:
---
Thanks [~nandakumar131] for the catch! But when I ran test locally, I still
occasionally got assertion on line 1140, which implies that a key that should
have been deleted still exists. I suspect that in addition to the issue you
fixed, there can be another issue due to other tests: since the cleanup service
got started at the start of minicluster, as multiple tests went on, the exact
time of clean up check can be arbitrary. So it's possible that the second
{{Thread.sleep(2000);}} may not actually include a cleanup check, given that
the clean up check interval is set to 3 sec.
If this is true, then I think there are two ways to fix:
1. instead of sleeping for 2000, first sleep 2000, then create key 5, then
sleep another 2000. Then for key1~4, the sleep is 4000, which is guaranteed to
have included a check. While for key 5, the sleep is 2000, so even if there is
a check after its creation, it guarantees the check won't remove key because
key 5 is guaranteed to have lived <2000 by the checking time. We may also want
to decrease the check interval and cleanup threshold to make this test run
faster.
2. make testExpiredOpenKey a separate unit test class, then we avoid the impact
from the other tests.
Either way looks good to me.
> Ozone: KSM: TestKeySpaceManager#testExpiredOpenKey fails occasionally
> -
>
> Key: HDFS-12940
> URL: https://issues.apache.org/jira/browse/HDFS-12940
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Nanda kumar
>Assignee: Nanda kumar
> Attachments: HDFS-12940-HDFS-7240.000.patch
>
>
> {{TestKeySpaceManager#testExpiredOpenKey}} is flaky.
> In {{testExpiredOpenKey}} we are opening four keys for writing and wait for
> them to expire (without committing). Verification/Assertion is done by
> querying {{MiniOzoneCluster}} and matching the count. Since the {{cluster}}
> instance of {{MiniOzoneCluster}} is shared between test-cases in
> {{TestKeySpaceManager}}, we should not rely on the count. The verification
> should only happen by matching the keyNames and not with the count.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12799) Ozone: SCM: Close containers: extend SCMCommandResponseProto with SCMCloseContainerCmdResponseProto
[
https://issues.apache.org/jira/browse/HDFS-12799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16259729#comment-16259729
]
Chen Liang commented on HDFS-12799:
---
Thanks for working on this [~elek]. Returning current state does seem to be a
bug and was causing test failing. So I fixed it in HDFS-12793. As for creating
container, have you tried to call
{{cluster.getStorageContainerManager().allocateContainer}}, followed with two
{{mapping.updateContainerState}}, just like what
{{TestContainerMapping#createContainer}} is doing?
> Ozone: SCM: Close containers: extend SCMCommandResponseProto with
> SCMCloseContainerCmdResponseProto
> ---
>
> Key: HDFS-12799
> URL: https://issues.apache.org/jira/browse/HDFS-12799
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Elek, Marton
>Assignee: Elek, Marton
> Attachments: HDFS-12799-HDFS-7240.001.patch
>
>
> This issue is about extending the HB response protocol between SCM and DN
> with a command to ask the datanode to close a container. (This is just about
> extending the protocol not about fixing the implementation of SCM tto handle
> the state transitions).
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12826) Document Saying the RPC port, But it's required IPC port in Balancer Document.
[
https://issues.apache.org/jira/browse/HDFS-12826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16259763#comment-16259763
]
Chen Liang commented on HDFS-12826:
---
Thanks [~peruguusha] for the patch! Would it be a bit more precise to take the
other way: changing {{ipc}} to {{rpc}} instead?
> Document Saying the RPC port, But it's required IPC port in Balancer Document.
> --
>
> Key: HDFS-12826
> URL: https://issues.apache.org/jira/browse/HDFS-12826
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: balancer & mover, documentation
>Affects Versions: 3.0.0-beta1
>Reporter: Harshakiran Reddy
>Assignee: usharani
>Priority: Minor
> Attachments: HDFS-12826.patch
>
>
> In {{Adding a new Namenode to an existing HDFS cluster}} , refreshNamenodes
> command required IPC port but in Documentation it's saying the RPC port.
> http://hadoop.apache.org/docs/r3.0.0-beta1/hadoop-project-dist/hadoop-hdfs/Federation.html#Balancer
> {noformat}
> bin>:~/hdfsdata/HA/install/hadoop/datanode/bin> ./hdfs dfsadmin
> -refreshNamenodes host-name:65110
> refreshNamenodes: Unknown protocol:
> org.apache.hadoop.hdfs.protocol.ClientDatanodeProtocol
> bin.:~/hdfsdata/HA/install/hadoop/datanode/bin> ./hdfs dfsadmin
> -refreshNamenodes
> Usage: hdfs dfsadmin [-refreshNamenodes datanode-host:ipc_port]
> bin>:~/hdfsdata/HA/install/hadoop/datanode/bin> ./hdfs dfsadmin
> -refreshNamenodes host-name:50077
> bin>:~/hdfsdata/HA/install/hadoop/datanode/bin>
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12804) Use slf4j instead of log4j in FSEditLog
[ https://issues.apache.org/jira/browse/HDFS-12804?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chen Liang updated HDFS-12804: -- Resolution: Fixed Status: Resolved (was: Patch Available) > Use slf4j instead of log4j in FSEditLog > --- > > Key: HDFS-12804 > URL: https://issues.apache.org/jira/browse/HDFS-12804 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs >Reporter: Mukul Kumar Singh >Assignee: Mukul Kumar Singh > Attachments: HDFS-12804.001.patch, HDFS-12804.002.patch, > HDFS-12804.003.patch > > > FSEditLog uses log4j, this jira will update the logging to use sl4j. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12804) Use slf4j instead of log4j in FSEditLog
[
https://issues.apache.org/jira/browse/HDFS-12804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16259818#comment-16259818
]
Chen Liang commented on HDFS-12804:
---
Thanks [~msingh] for the update, I tested locally also, {{TestUnbuffer}} and
{{TestBalancerRPCDelay}} did fail even without the patch. I've committed v003
patch to trunk. Thanks Mukul for the contribution!
> Use slf4j instead of log4j in FSEditLog
> ---
>
> Key: HDFS-12804
> URL: https://issues.apache.org/jira/browse/HDFS-12804
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs
>Reporter: Mukul Kumar Singh
>Assignee: Mukul Kumar Singh
> Attachments: HDFS-12804.001.patch, HDFS-12804.002.patch,
> HDFS-12804.003.patch
>
>
> FSEditLog uses log4j, this jira will update the logging to use sl4j.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12791) NameNode Fsck http Connection can timeout for directories with multiple levels
[
https://issues.apache.org/jira/browse/HDFS-12791?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chen Liang updated HDFS-12791:
--
Fix Version/s: 3.1.0
> NameNode Fsck http Connection can timeout for directories with multiple levels
> --
>
> Key: HDFS-12791
> URL: https://issues.apache.org/jira/browse/HDFS-12791
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: tools
>Reporter: Mukul Kumar Singh
>Assignee: Mukul Kumar Singh
> Fix For: 3.1.0
>
> Attachments: HDFS-12791.001.patch
>
>
> Currently the http connections are flushed for every 100 files, however if
> there are multiple levels of directories in the namespace then flushing will
> be postponed till multiple directories levels have been traversed. This
> connection timeout can be avoided if both files and directories are
> considered for the flushing query.
> {code}
> if (showprogress && (replRes.totalFiles + ecRes.totalFiles) % 100 == 0) {
> out.println();
> out.flush();
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12791) NameNode Fsck http Connection can timeout for directories with multiple levels
[
https://issues.apache.org/jira/browse/HDFS-12791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16261474#comment-16261474
]
Chen Liang commented on HDFS-12791:
---
[~zhz] Sorry I should have updated it. I believe it should be 3.1.0, I have
updated the JIRA. ([~msingh] Please correct me if I'm wrong).
> NameNode Fsck http Connection can timeout for directories with multiple levels
> --
>
> Key: HDFS-12791
> URL: https://issues.apache.org/jira/browse/HDFS-12791
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: tools
>Reporter: Mukul Kumar Singh
>Assignee: Mukul Kumar Singh
> Fix For: 3.1.0
>
> Attachments: HDFS-12791.001.patch
>
>
> Currently the http connections are flushed for every 100 files, however if
> there are multiple levels of directories in the namespace then flushing will
> be postponed till multiple directories levels have been traversed. This
> connection timeout can be avoided if both files and directories are
> considered for the flushing query.
> {code}
> if (showprogress && (replRes.totalFiles + ecRes.totalFiles) % 100 == 0) {
> out.println();
> out.flush();
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11419) BlockPlacementPolicyDefault is choosing datanode in an inefficient way
[
https://issues.apache.org/jira/browse/HDFS-11419?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16241538#comment-16241538
]
Chen Liang commented on HDFS-11419:
---
Hi [~cheersyang],
Thanks for sharing and sorry for the delay on response as I am on vacation.
So just to make sure we are on the same page: you have 2 SSD on each of the 500
DNs, for each block, due to ALL_SSD policy, the NN will always try to use the 2
SSDs on each of the DNs, but when most of the SSDs are full, NN will spend lot
of time on trying to find available SSDs among the 500 DNs. Is this what you
saw?
If this is the case, I'm afraid the sub-tasks here will not be enough. Because
this JIRA was to address a different scenario such as, there are only 10 nodes
with SSD among the 500 nodes, and NN is trying to locate these 10 nodes.
Namely, here it is to *locate nodes with certain storage types*. But in your
case it seems NN is doing a bit more by trying to *locate nodes with certain
storage type _and enough space_*.
I think this can done by just adding another sub-task to this JIRA that tracks
the available space of different storage types, then the rest should be
straitforward. Any other comments? [~linyiqun], [~arpitagarwal], [~szetszwo].
> BlockPlacementPolicyDefault is choosing datanode in an inefficient way
> --
>
> Key: HDFS-11419
> URL: https://issues.apache.org/jira/browse/HDFS-11419
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Reporter: Chen Liang
>Assignee: Chen Liang
>
> Currently in {{BlockPlacementPolicyDefault}}, {{chooseTarget}} will end up
> calling into {{chooseRandom}}, which will first find a random datanode by
> calling
> {code}DatanodeDescriptor chosenNode = chooseDataNode(scope,
> excludedNodes);{code}, then it checks whether that returned datanode
> satisfies storage type requirement
> {code}storage = chooseStorage4Block(
> chosenNode, blocksize, results, entry.getKey());{code}
> If yes, {{numOfReplicas--;}}, otherwise, the node is added to excluded nodes,
> and runs the loop again until {{numOfReplicas}} is down to 0.
> A problem here is that, storage type is not being considered only until after
> a random node is already returned. We've seen a case where a cluster has a
> large number of datanodes, while only a few satisfy the storage type
> condition. So, for the most part, this code blindly picks random datanodes
> that do not satisfy the storage type requirement.
> To make matters worse, the way {{NetworkTopology#chooseRandom}} works is
> that, given a set of excluded nodes, it first finds a random datanodes, then
> if it is in excluded nodes set, try find another random nodes. So the more
> excluded nodes there are, the more likely a random node will be in the
> excluded set, in which case we basically wasted one iteration.
> Therefore, this JIRA proposes to augment/modify the relevant classes in a way
> that datanodes can be found more efficiently. There are currently two
> different high level solutions we are considering:
> 1. add some field to Node base types to describe the storage type info, and
> when searching for a node, we take into account such field(s), and do not
> return node that does not meet the storage type requirement.
> 2. change {{NetworkTopology}} class to be aware of storage types, e.g. for
> one storage type, there is one tree subset that connects all the nodes with
> that type. And one search happens on only one such subset. So unexpected
> storage types are simply not in the search space.
> Thanks [~szetszwo] for the offline discussion, and thanks [~linyiqun] for
> pointing out a wrong statement (corrected now) in the description. Any
> further comments are more than welcome.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12791) NameNode Fsck http Connection can timeout for directories with multiple levels
[
https://issues.apache.org/jira/browse/HDFS-12791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16246944#comment-16246944
]
Chen Liang commented on HDFS-12791:
---
Thanks for working on this [~msingh]! +1 on v001 patch, the failed tests all
passed in my local run. Will commit this shortly.
> NameNode Fsck http Connection can timeout for directories with multiple levels
> --
>
> Key: HDFS-12791
> URL: https://issues.apache.org/jira/browse/HDFS-12791
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: tools
>Reporter: Mukul Kumar Singh
>Assignee: Mukul Kumar Singh
> Attachments: HDFS-12791.001.patch
>
>
> Currently the http connections are flushed for every 100 files, however if
> there are multiple levels of directories in the namespace then flushing will
> be postponed till multiple directories levels have been traversed. This
> connection timeout can be avoided if both files and directories are
> considered for the flushing query.
> {code}
> if (showprogress && (replRes.totalFiles + ecRes.totalFiles) % 100 == 0) {
> out.println();
> out.flush();
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12791) NameNode Fsck http Connection can timeout for directories with multiple levels
[
https://issues.apache.org/jira/browse/HDFS-12791?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chen Liang updated HDFS-12791:
--
Resolution: Fixed
Status: Resolved (was: Patch Available)
> NameNode Fsck http Connection can timeout for directories with multiple levels
> --
>
> Key: HDFS-12791
> URL: https://issues.apache.org/jira/browse/HDFS-12791
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: tools
>Reporter: Mukul Kumar Singh
>Assignee: Mukul Kumar Singh
> Attachments: HDFS-12791.001.patch
>
>
> Currently the http connections are flushed for every 100 files, however if
> there are multiple levels of directories in the namespace then flushing will
> be postponed till multiple directories levels have been traversed. This
> connection timeout can be avoided if both files and directories are
> considered for the flushing query.
> {code}
> if (showprogress && (replRes.totalFiles + ecRes.totalFiles) % 100 == 0) {
> out.println();
> out.flush();
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
