[jira] [Commented] (HDFS-15412) Add options to set different block scan period for diffrent StorageType
[
https://issues.apache.org/jira/browse/HDFS-15412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17256862#comment-17256862
]
Ayush Saxena commented on HDFS-15412:
-
Well changes are looking fine, I would need to go through them once more though.
{code:java}
317if (this.scanPeriodMs == 0) {
318 this.scanPeriodMs = conf.scanPeriodMs;
319}
{code}
can we make the check as <=, in order to handle mis configuration, and may
Secondly :
{code:java}
313this.scanPeriodMs = TimeUnit.MILLISECONDS.convert(
314StorageType.getConfLong(datanode.getConf(),
315ref.getVolume().getStorageType(), "scan.period.hours", 0),
316TimeUnit.HOURS);
{code}
Why are we passing 0 here as default, Why not {{conf.scanPeriodMs}}?
> Add options to set different block scan period for diffrent StorageType
> ---
>
> Key: HDFS-15412
> URL: https://issues.apache.org/jira/browse/HDFS-15412
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Reporter: Yang Yun
>Assignee: Yang Yun
>Priority: Minor
> Attachments: HDFS-15412.001.patch, HDFS-15412.002.patch,
> HDFS-15412.003.patch
>
>
> For some cold data, sometime, we don't want to scan cold data as often as
> hot data. Add options that we can set the scan period time according to
> StorageType.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15756) [RBF] Cannot get updated delegation token from zookeeper
[
https://issues.apache.org/jira/browse/HDFS-15756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17256860#comment-17256860
]
Yuxuan Wang commented on HDFS-15756:
I modify the code let router throw StandbyException, so the client can retry
and failover to other routers.
{code:title=RouterSecurityManager.java|java}
renewDelegationToken(...){
...
catch (SecretManager.InvalidToken e) {
throw new StandbyException(e.getMessage());
}
}
{code}
You can have a try.
> [RBF] Cannot get updated delegation token from zookeeper
>
>
> Key: HDFS-15756
> URL: https://issues.apache.org/jira/browse/HDFS-15756
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: rbf
>Affects Versions: 3.0.0
>Reporter: hbprotoss
>Priority: Major
>
> Affected version: all version with rbf
> When RBF work with spark 2.4 client mode, there will be a chance that token
> is missing across different nodes in RBF cluster. The root cause is that
> spark renew the token(via resource manager) immediately after got one, as
> zookeeper don't have a strong consistency guarantee after an update in
> cluster, zookeeper client may read a stale value in some followers not synced
> with other nodes.
>
> We apply a patch in spark, but it is still the problem of RBF. Is it possible
> for RBF to replace the delegation token store using some other
> datasource(redis for example)?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15756) [RBF] Cannot get updated delegation token from zookeeper
[ https://issues.apache.org/jira/browse/HDFS-15756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated HDFS-15756: Summary: [RBF] Cannot get updated delegation token from zookeeper (was: [RBF]Cannot get updated delegation token from zookeeper) > [RBF] Cannot get updated delegation token from zookeeper > > > Key: HDFS-15756 > URL: https://issues.apache.org/jira/browse/HDFS-15756 > Project: Hadoop HDFS > Issue Type: Bug > Components: rbf >Affects Versions: 3.0.0 >Reporter: hbprotoss >Priority: Major > > Affected version: all version with rbf > When RBF work with spark 2.4 client mode, there will be a chance that token > is missing across different nodes in RBF cluster. The root cause is that > spark renew the token(via resource manager) immediately after got one, as > zookeeper don't have a strong consistency guarantee after an update in > cluster, zookeeper client may read a stale value in some followers not synced > with other nodes. > > We apply a patch in spark, but it is still the problem of RBF. Is it possible > for RBF to replace the delegation token store using some other > datasource(redis for example)? -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15748) RBF: Move the router related part from hadoop-federation-balance module to hadoop-hdfs-rbf.
[
https://issues.apache.org/jira/browse/HDFS-15748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17256859#comment-17256859
]
Ayush Saxena commented on HDFS-15748:
-
{quote}I add a check in preCheck phase. If the dfs.permissions.enabled=false
then throw an IOException
{quote}
This would not help, this should be true in the Namenode conf, no use of
validating in the client conf, In UT it might work but not in actual prod.
{quote}About the super user I don't know how to restrict him. Can we depend on
admin's experience?
{quote}
I don't think there is a way to restrict a super-user, The best we can do is as
of now mark a TODO and document it that in case of non RBF usage super-user
shouldn't bother the directory.
{code:java}
+srcFs.setPermission(src, FsPermission.createImmutable((short) 0));
{code}
Will this not disable read as well? Secondly while restoring we should ensure
the actual permissions which were set are restored.
TBH. This disableWrite() I don't think would be a very strong in disabling
write, May be if possible, can we do something with snapshots here? Take a
snapshot after disabling and before enabling confirm nothing changed, if
changed we redo this logic or fail or something like that?
> RBF: Move the router related part from hadoop-federation-balance module to
> hadoop-hdfs-rbf.
> ---
>
> Key: HDFS-15748
> URL: https://issues.apache.org/jira/browse/HDFS-15748
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Jinglun
>Assignee: Jinglun
>Priority: Major
> Attachments: HDFS-15748.001.patch, HDFS-15748.002.patch,
> HDFS-15748.003.patch
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15755) Add option to set retry times of checking failure volume and Import powermock for mock static method
[
https://issues.apache.org/jira/browse/HDFS-15755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17256851#comment-17256851
]
Hadoop QA commented on HDFS-15755:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
36s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} No case conflicting files
found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch does not contain any
@author tags. {color} |
| {color:green}+1{color} | {color:green} {color} | {color:green} 0m 0s{color}
| {color:green}test4tests{color} | {color:green} The patch appears to include 1
new or modified test files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
58s{color} | {color:blue}{color} | {color:blue} Maven dependency ordering for
branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m
7s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 20m
6s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 17m
20s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
35s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
4s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
21m 23s{color} | {color:green}{color} | {color:green} branch has no errors when
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
9s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
12s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 3m
15s{color} | {color:blue}{color} | {color:blue} Used deprecated FindBugs
config; considering switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
28s{color} | {color:green}{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
37s{color} | {color:blue}{color} | {color:blue} Maven dependency ordering for
patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
6s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 19m
26s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 19m
26s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 17m
20s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 17m
20s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
2m 35s{color} |
{color:orange}https://ci-hadoop.apache.org/job/PreCommit-HDFS-Build/390/artifact/out/diff-checkstyle-root.txt{color}
| {color:orange} root: The patch generated 1 new + 18 unchanged - 0 fixed = 19
total (was 18) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
58s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch has no whitespace
issues. {color} |
| {color:green}+1{color} | {color:green} xml
[jira] [Updated] (HDFS-15756) [RBF]Cannot get updated delegation token from zookeeper
[ https://issues.apache.org/jira/browse/HDFS-15756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] hbprotoss updated HDFS-15756: - Description: Affected version: all version with rbf When RBF work with spark 2.4 client mode, there will be a chance that token is missing across different nodes in RBF cluster. The root cause is that spark renew the token(via resource manager) immediately after got one, as zookeeper don't have a strong consistency guarantee after an update in cluster, zookeeper client may read a stale value in some followers not synced with other nodes. We apply a patch in spark, but it is still the problem of RBF. Is it possible for RBF to replace the delegation token store using some other datasource(redis for example)? was: When RBF work with spark 2.4 client mode, there will be a chance that token is missing across different nodes in RBF cluster. The root cause is that spark renew the token(via resource manager) immediately after got one, as zookeeper don't have a strong consistency guarantee after an update in cluster, zookeeper client may read a stale value in some followers not synced with other nodes. We apply a patch in spark, but it is still the problem of RBF. Is it possible for RBF to replace the delegation token store using some other datasource(redis for example)? > [RBF]Cannot get updated delegation token from zookeeper > --- > > Key: HDFS-15756 > URL: https://issues.apache.org/jira/browse/HDFS-15756 > Project: Hadoop HDFS > Issue Type: Bug > Components: rbf >Affects Versions: 3.0.0 >Reporter: hbprotoss >Priority: Major > > Affected version: all version with rbf > When RBF work with spark 2.4 client mode, there will be a chance that token > is missing across different nodes in RBF cluster. The root cause is that > spark renew the token(via resource manager) immediately after got one, as > zookeeper don't have a strong consistency guarantee after an update in > cluster, zookeeper client may read a stale value in some followers not synced > with other nodes. > > We apply a patch in spark, but it is still the problem of RBF. Is it possible > for RBF to replace the delegation token store using some other > datasource(redis for example)? -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-15756) [RBF]Cannot get updated delegation token from zookeeper
hbprotoss created HDFS-15756: Summary: [RBF]Cannot get updated delegation token from zookeeper Key: HDFS-15756 URL: https://issues.apache.org/jira/browse/HDFS-15756 Project: Hadoop HDFS Issue Type: Bug Components: rbf Affects Versions: 3.0.0 Reporter: hbprotoss When RBF work with spark 2.4 client mode, there will be a chance that token is missing across different nodes in RBF cluster. The root cause is that spark renew the token(via resource manager) immediately after got one, as zookeeper don't have a strong consistency guarantee after an update in cluster, zookeeper client may read a stale value in some followers not synced with other nodes. We apply a patch in spark, but it is still the problem of RBF. Is it possible for RBF to replace the delegation token store using some other datasource(redis for example)? -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15755) Add option to set retry times of checking failure volume and Import powermock for mock static method
[ https://issues.apache.org/jira/browse/HDFS-15755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Yun updated HDFS-15755: Status: Open (was: Patch Available) > Add option to set retry times of checking failure volume and Import powermock > for mock static method > > > Key: HDFS-15755 > URL: https://issues.apache.org/jira/browse/HDFS-15755 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs, test >Reporter: Yang Yun >Assignee: Yang Yun >Priority: Minor > Attachments: HDFS-15755.001.patch, HDFS-15755.002.patch, > HDFS-15755.003.patch > > > For some mounted remote volume, small network shaking may causes the failure > of volume checking and the the volume will removed out from datanode. Add an > option to set retry times and interval according to StorageType. > For mocking static methods in unit test, import PowerMock to > hadoop-hdfs-project. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15755) Add option to set retry times of checking failure volume and Import powermock for mock static method
[ https://issues.apache.org/jira/browse/HDFS-15755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Yun updated HDFS-15755: Attachment: HDFS-15755.003.patch Status: Patch Available (was: Open) > Add option to set retry times of checking failure volume and Import powermock > for mock static method > > > Key: HDFS-15755 > URL: https://issues.apache.org/jira/browse/HDFS-15755 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs, test >Reporter: Yang Yun >Assignee: Yang Yun >Priority: Minor > Attachments: HDFS-15755.001.patch, HDFS-15755.002.patch, > HDFS-15755.003.patch > > > For some mounted remote volume, small network shaking may causes the failure > of volume checking and the the volume will removed out from datanode. Add an > option to set retry times and interval according to StorageType. > For mocking static methods in unit test, import PowerMock to > hadoop-hdfs-project. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15412) Add options to set different block scan period for diffrent StorageType
[
https://issues.apache.org/jira/browse/HDFS-15412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17256763#comment-17256763
]
Hadoop QA commented on HDFS-15412:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
2s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} No case conflicting files
found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch does not contain any
@author tags. {color} |
| {color:green}+1{color} | {color:green} {color} | {color:green} 0m 0s{color}
| {color:green}test4tests{color} | {color:green} The patch appears to include 1
new or modified test files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
45s{color} | {color:blue}{color} | {color:blue} Maven dependency ordering for
branch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 25m
59s{color} |
{color:red}https://ci-hadoop.apache.org/job/PreCommit-HDFS-Build/389/artifact/out/branch-mvninstall-root.txt{color}
| {color:red} root in trunk failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
28s{color} |
{color:red}https://ci-hadoop.apache.org/job/PreCommit-HDFS-Build/389/artifact/out/branch-compile-root-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04.txt{color}
| {color:red} root in trunk failed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
31s{color} |
{color:red}https://ci-hadoop.apache.org/job/PreCommit-HDFS-Build/389/artifact/out/branch-compile-root-jdkPrivateBuild-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01.txt{color}
| {color:red} root in trunk failed with JDK Private
Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01. {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 26s{color} |
{color:orange}https://ci-hadoop.apache.org/job/PreCommit-HDFS-Build/389/artifact/out/buildtool-branch-checkstyle-root.txt{color}
| {color:orange} The patch fails to run checkstyle in root {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
28s{color} |
{color:red}https://ci-hadoop.apache.org/job/PreCommit-HDFS-Build/389/artifact/out/branch-mvnsite-hadoop-common-project_hadoop-common.txt{color}
| {color:red} hadoop-common in trunk failed. {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
31s{color} |
{color:red}https://ci-hadoop.apache.org/job/PreCommit-HDFS-Build/389/artifact/out/branch-mvnsite-hadoop-hdfs-project_hadoop-hdfs.txt{color}
| {color:red} hadoop-hdfs in trunk failed. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
2m 9s{color} | {color:green}{color} | {color:green} branch has no errors when
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
1s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
26s{color} |
{color:red}https://ci-hadoop.apache.org/job/PreCommit-HDFS-Build/389/artifact/out/branch-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkPrivateBuild-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01.txt{color}
| {color:red} hadoop-hdfs in trunk failed with JDK Private
Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01. {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 6m
55s{color} | {color:blue}{color} | {color:blue} Used deprecated FindBugs
config; considering switching to SpotBugs. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
20s{color} |
{color:red}https://ci-hadoop.apache.org/job/PreCommit-HDFS-Build/389/artifact/out/branch-findbugs-hadoop-common-project_hadoop-common.txt{color}
| {color:red} hadoop-common in trunk failed. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
27s{color} |
{color:red}https://ci-hadoop.apache.org/job/PreCommit-HDFS-Build/389/artifact/out/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs.txt{color}
| {color:red} hadoop-hdfs in trunk failed. {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
29s{color} | {color:blue}{color} | {color:blue} Maven dependency ordering for
patch {color} |
| {color:red}-1{color} | {color:red}
[jira] [Updated] (HDFS-15412) Add options to set different block scan period for diffrent StorageType
[ https://issues.apache.org/jira/browse/HDFS-15412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Yun updated HDFS-15412: Status: Open (was: Patch Available) > Add options to set different block scan period for diffrent StorageType > --- > > Key: HDFS-15412 > URL: https://issues.apache.org/jira/browse/HDFS-15412 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Reporter: Yang Yun >Assignee: Yang Yun >Priority: Minor > Attachments: HDFS-15412.001.patch, HDFS-15412.002.patch, > HDFS-15412.003.patch > > > For some cold data, sometime, we don't want to scan cold data as often as > hot data. Add options that we can set the scan period time according to > StorageType. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15412) Add options to set different block scan period for diffrent StorageType
[ https://issues.apache.org/jira/browse/HDFS-15412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Yun updated HDFS-15412: Attachment: HDFS-15412.003.patch Status: Patch Available (was: Open) > Add options to set different block scan period for diffrent StorageType > --- > > Key: HDFS-15412 > URL: https://issues.apache.org/jira/browse/HDFS-15412 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Reporter: Yang Yun >Assignee: Yang Yun >Priority: Minor > Attachments: HDFS-15412.001.patch, HDFS-15412.002.patch, > HDFS-15412.003.patch > > > For some cold data, sometime, we don't want to scan cold data as often as > hot data. Add options that we can set the scan period time according to > StorageType. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15412) Add options to set different block scan period for diffrent StorageType
[ https://issues.apache.org/jira/browse/HDFS-15412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Yun updated HDFS-15412: Attachment: (was: HDFS-15412.003.patch) > Add options to set different block scan period for diffrent StorageType > --- > > Key: HDFS-15412 > URL: https://issues.apache.org/jira/browse/HDFS-15412 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Reporter: Yang Yun >Assignee: Yang Yun >Priority: Minor > Attachments: HDFS-15412.001.patch, HDFS-15412.002.patch, > HDFS-15412.003.patch > > > For some cold data, sometime, we don't want to scan cold data as often as > hot data. Add options that we can set the scan period time according to > StorageType. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15751) Add documentation for msync() API to filesystem.md
[
https://issues.apache.org/jira/browse/HDFS-15751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17256656#comment-17256656
]
Chao Sun commented on HDFS-15751:
-
I agree with [~hexiaoqiao] and think we can mention that this is currently only
implemented for HDFS and others will just throw
{{UnsupportedOperationException}}, similar to what we're doing for {{concat}},
{{truncate}} etc.
Otherwise I'm +1 on this. Thanks.
> Add documentation for msync() API to filesystem.md
> --
>
> Key: HDFS-15751
> URL: https://issues.apache.org/jira/browse/HDFS-15751
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: documentation
>Reporter: Konstantin Shvachko
>Assignee: Konstantin Shvachko
>Priority: Major
> Attachments: HDFS-15751-01.patch
>
>
> HDFS-15567 introduced new {{FileSystem}} call {{msync()}}. Should add it to
> the API definitions.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15412) Add options to set different block scan period for diffrent StorageType
[
https://issues.apache.org/jira/browse/HDFS-15412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17256644#comment-17256644
]
Hadoop QA commented on HDFS-15412:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 2m
29s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} No case conflicting files
found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch does not contain any
@author tags. {color} |
| {color:green}+1{color} | {color:green} {color} | {color:green} 0m 0s{color}
| {color:green}test4tests{color} | {color:green} The patch appears to include 1
new or modified test files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 14m
0s{color} | {color:blue}{color} | {color:blue} Maven dependency ordering for
branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 24m
12s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 21m
47s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 20m
38s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
47s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
6s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
23m 38s{color} | {color:green}{color} | {color:green} branch has no errors when
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
8s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
28s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 3m
25s{color} | {color:blue}{color} | {color:blue} Used deprecated FindBugs
config; considering switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
46s{color} | {color:green}{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
24s{color} | {color:blue}{color} | {color:blue} Maven dependency ordering for
patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
8s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 23m
11s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 23m
11s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 20m
1s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 20m
1s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
54s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
8s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch has no whitespace
issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
19m 2s{color} | {color:green}{color} | {color:green} patch has no errors when
building and testing our client artifacts. {color} |
|
[jira] [Work logged] (HDFS-15754) Create packet metrics for DataNode
[
https://issues.apache.org/jira/browse/HDFS-15754?focusedWorklogId=529581=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-529581
]
ASF GitHub Bot logged work on HDFS-15754:
-
Author: ASF GitHub Bot
Created on: 30/Dec/20 17:57
Start Date: 30/Dec/20 17:57
Worklog Time Spent: 10m
Work Description: goiri commented on a change in pull request #2578:
URL: https://github.com/apache/hadoop/pull/2578#discussion_r550278063
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeMetrics.java
##
@@ -161,6 +163,65 @@ public void testReceivePacketMetrics() throws Exception {
}
}
+ @Test
+ public void testReceivePacketSlowMetrics() throws Exception {
+Configuration conf = new HdfsConfiguration();
+final int interval = 1;
+conf.set(DFSConfigKeys.DFS_METRICS_PERCENTILES_INTERVALS_KEY, "" +
interval);
Review comment:
setInt
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 529581)
Time Spent: 1h (was: 50m)
> Create packet metrics for DataNode
> --
>
> Key: HDFS-15754
> URL: https://issues.apache.org/jira/browse/HDFS-15754
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Reporter: Fengnan Li
>Assignee: Fengnan Li
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 1h
> Remaining Estimate: 0h
>
> In BlockReceiver, right now when there is slowness in writeToMirror,
> writeToDisk and writeToOsCache, it is dumped in the debug log. In practice we
> have found these are quite useful signal to detect issues in DataNode, so it
> will be great these metrics can be exposed by JMX.
> Also we introduced totalPacket received to use a percentage as a signal to
> detect the potentially underperforming datanode since datanodes across one
> HDFS cluster may received different numbers of packets totally.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15754) Create packet metrics for DataNode
[
https://issues.apache.org/jira/browse/HDFS-15754?focusedWorklogId=529582=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-529582
]
ASF GitHub Bot logged work on HDFS-15754:
-
Author: ASF GitHub Bot
Created on: 30/Dec/20 17:57
Start Date: 30/Dec/20 17:57
Worklog Time Spent: 10m
Work Description: goiri commented on a change in pull request #2578:
URL: https://github.com/apache/hadoop/pull/2578#discussion_r550278246
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeMetrics.java
##
@@ -161,6 +163,65 @@ public void testReceivePacketMetrics() throws Exception {
}
}
+ @Test
+ public void testReceivePacketSlowMetrics() throws Exception {
+Configuration conf = new HdfsConfiguration();
+final int interval = 1;
+conf.set(DFSConfigKeys.DFS_METRICS_PERCENTILES_INTERVALS_KEY, "" +
interval);
+MiniDFSCluster cluster = new MiniDFSCluster.Builder(conf)
+.numDataNodes(3).build();
+try {
+ cluster.waitActive();
+ DistributedFileSystem fs = cluster.getFileSystem();
+ final DataNodeFaultInjector injector =
+ Mockito.mock(DataNodeFaultInjector.class);
+ Mockito.doAnswer(new Answer() {
+@Override
+public Object answer(InvocationOnMock invocationOnMock)
Review comment:
Extract the sleeping answer and return for each?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 529582)
Time Spent: 1h 10m (was: 1h)
> Create packet metrics for DataNode
> --
>
> Key: HDFS-15754
> URL: https://issues.apache.org/jira/browse/HDFS-15754
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Reporter: Fengnan Li
>Assignee: Fengnan Li
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> In BlockReceiver, right now when there is slowness in writeToMirror,
> writeToDisk and writeToOsCache, it is dumped in the debug log. In practice we
> have found these are quite useful signal to detect issues in DataNode, so it
> will be great these metrics can be exposed by JMX.
> Also we introduced totalPacket received to use a percentage as a signal to
> detect the potentially underperforming datanode since datanodes across one
> HDFS cluster may received different numbers of packets totally.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15754) Create packet metrics for DataNode
[
https://issues.apache.org/jira/browse/HDFS-15754?focusedWorklogId=529580=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-529580
]
ASF GitHub Bot logged work on HDFS-15754:
-
Author: ASF GitHub Bot
Created on: 30/Dec/20 17:55
Start Date: 30/Dec/20 17:55
Worklog Time Spent: 10m
Work Description: goiri commented on a change in pull request #2578:
URL: https://github.com/apache/hadoop/pull/2578#discussion_r550277730
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java
##
@@ -603,12 +604,15 @@ private int receivePacket() throws IOException {
mirrorAddr,
duration);
trackSendPacketToLastNodeInPipeline(duration);
-if (duration > datanodeSlowLogThresholdMs && LOG.isWarnEnabled()) {
- LOG.warn("Slow BlockReceiver write packet to mirror took " + duration
- + "ms (threshold=" + datanodeSlowLogThresholdMs + "ms), "
- + "downstream DNs=" + Arrays.toString(downstreamDNs)
- + ", blockId=" + replicaInfo.getBlockId()
- + ", seqno=" + seqno);
+if (duration > datanodeSlowLogThresholdMs) {
+ datanode.metrics.incrPacketSlowWriteToMirror();
+ if (LOG.isWarnEnabled()) {
+LOG.warn("Slow BlockReceiver write packet to mirror took " +
duration
++ "ms (threshold=" + datanodeSlowLogThresholdMs + "ms), "
++ "downstream DNs=" + Arrays.toString(downstreamDNs)
++ ", blockId=" + replicaInfo.getBlockId()
Review comment:
As we are at it, let's use the logger format. We still need the Arrays
toString so we need the isWarnEnabled though.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 529580)
Time Spent: 50m (was: 40m)
> Create packet metrics for DataNode
> --
>
> Key: HDFS-15754
> URL: https://issues.apache.org/jira/browse/HDFS-15754
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Reporter: Fengnan Li
>Assignee: Fengnan Li
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 50m
> Remaining Estimate: 0h
>
> In BlockReceiver, right now when there is slowness in writeToMirror,
> writeToDisk and writeToOsCache, it is dumped in the debug log. In practice we
> have found these are quite useful signal to detect issues in DataNode, so it
> will be great these metrics can be exposed by JMX.
> Also we introduced totalPacket received to use a percentage as a signal to
> detect the potentially underperforming datanode since datanodes across one
> HDFS cluster may received different numbers of packets totally.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15748) RBF: Move the router related part from hadoop-federation-balance module to hadoop-hdfs-rbf.
[ https://issues.apache.org/jira/browse/HDFS-15748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17256632#comment-17256632 ] Íñigo Goiri commented on HDFS-15748: As we are at it, let's fix the javadocs and go with [^HDFS-15748.003.patch]. > RBF: Move the router related part from hadoop-federation-balance module to > hadoop-hdfs-rbf. > --- > > Key: HDFS-15748 > URL: https://issues.apache.org/jira/browse/HDFS-15748 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Jinglun >Assignee: Jinglun >Priority: Major > Attachments: HDFS-15748.001.patch, HDFS-15748.002.patch, > HDFS-15748.003.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15755) Add option to set retry times of checking failure volume and Import powermock for mock static method
[
https://issues.apache.org/jira/browse/HDFS-15755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17256630#comment-17256630
]
Hadoop QA commented on HDFS-15755:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
44s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
1s{color} | {color:green}{color} | {color:green} No case conflicting files
found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch does not contain any
@author tags. {color} |
| {color:green}+1{color} | {color:green} {color} | {color:green} 0m 0s{color}
| {color:green}test4tests{color} | {color:green} The patch appears to include 1
new or modified test files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m
2s{color} | {color:blue}{color} | {color:blue} Maven dependency ordering for
branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 23m
24s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 21m
34s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 18m
17s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
34s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
25s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
21m 46s{color} | {color:green}{color} | {color:green} branch has no errors when
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
31s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
39s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 3m
21s{color} | {color:blue}{color} | {color:blue} Used deprecated FindBugs
config; considering switching to SpotBugs. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
27s{color} | {color:blue}{color} | {color:blue} branch/hadoop-project no
findbugs output file (findbugsXml.xml) {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
38s{color} | {color:blue}{color} | {color:blue} Maven dependency ordering for
patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
19s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 13m
21s{color} |
{color:red}https://ci-hadoop.apache.org/job/PreCommit-HDFS-Build/387/artifact/out/patch-compile-root-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04.txt{color}
| {color:red} root in the patch failed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 13m 21s{color}
|
{color:red}https://ci-hadoop.apache.org/job/PreCommit-HDFS-Build/387/artifact/out/patch-compile-root-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04.txt{color}
| {color:red} root in the patch failed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 10m
49s{color} |
{color:red}https://ci-hadoop.apache.org/job/PreCommit-HDFS-Build/387/artifact/out/patch-compile-root-jdkPrivateBuild-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01.txt{color}
| {color:red} root in the patch failed with JDK Private
Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 10m 49s{color}
|
{color:red}https://ci-hadoop.apache.org/job/PreCommit-HDFS-Build/387/artifact/out/patch-compile-root-jdkPrivateBuild-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01.txt{color}
| {color:red} root in the patch failed with JDK Private
[jira] [Commented] (HDFS-15748) RBF: Move the router related part from hadoop-federation-balance module to hadoop-hdfs-rbf.
[
https://issues.apache.org/jira/browse/HDFS-15748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17256603#comment-17256603
]
Hadoop QA commented on HDFS-15748:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
39s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} No case conflicting files
found. {color} |
| {color:blue}0{color} | {color:blue} markdownlint {color} | {color:blue} 0m
0s{color} | {color:blue}{color} | {color:blue} markdownlint was not available.
{color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch does not contain any
@author tags. {color} |
| {color:green}+1{color} | {color:green} {color} | {color:green} 0m 0s{color}
| {color:green}test4tests{color} | {color:green} The patch appears to include 3
new or modified test files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
31s{color} | {color:blue}{color} | {color:blue} Maven dependency ordering for
branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m
24s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 21m
11s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 18m
29s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
36s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
14s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
16m 29s{color} | {color:green}{color} | {color:green} branch has no errors when
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
40s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
39s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 0m
59s{color} | {color:blue}{color} | {color:blue} Used deprecated FindBugs
config; considering switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
58s{color} | {color:green}{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
37s{color} | {color:blue}{color} | {color:blue} Maven dependency ordering for
patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
59s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 20m
44s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 20m 44s{color}
|
{color:red}https://ci-hadoop.apache.org/job/PreCommit-HDFS-Build/386/artifact/out/diff-compile-javac-root-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04.txt{color}
| {color:red} root-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 generated 139 new + 1902 unchanged -
139 fixed = 2041 total (was 2041) {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 18m
49s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 18m 49s{color}
|
{color:red}https://ci-hadoop.apache.org/job/PreCommit-HDFS-Build/386/artifact/out/diff-compile-javac-root-jdkPrivateBuild-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01.txt{color}
| {color:red} root-jdkPrivateBuild-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 with
JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
[jira] [Commented] (HDFS-15412) Add options to set different block scan period for diffrent StorageType
[ https://issues.apache.org/jira/browse/HDFS-15412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17256541#comment-17256541 ] Yang Yun commented on HDFS-15412: - retrigger a build for some tests faled with OOM. > Add options to set different block scan period for diffrent StorageType > --- > > Key: HDFS-15412 > URL: https://issues.apache.org/jira/browse/HDFS-15412 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Reporter: Yang Yun >Assignee: Yang Yun >Priority: Minor > Attachments: HDFS-15412.001.patch, HDFS-15412.002.patch, > HDFS-15412.003.patch > > > For some cold data, sometime, we don't want to scan cold data as often as > hot data. Add options that we can set the scan period time according to > StorageType. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15412) Add options to set different block scan period for diffrent StorageType
[ https://issues.apache.org/jira/browse/HDFS-15412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Yun updated HDFS-15412: Attachment: HDFS-15412.003.patch Status: Patch Available (was: Open) > Add options to set different block scan period for diffrent StorageType > --- > > Key: HDFS-15412 > URL: https://issues.apache.org/jira/browse/HDFS-15412 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Reporter: Yang Yun >Assignee: Yang Yun >Priority: Minor > Attachments: HDFS-15412.001.patch, HDFS-15412.002.patch, > HDFS-15412.003.patch > > > For some cold data, sometime, we don't want to scan cold data as often as > hot data. Add options that we can set the scan period time according to > StorageType. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15412) Add options to set different block scan period for diffrent StorageType
[ https://issues.apache.org/jira/browse/HDFS-15412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Yun updated HDFS-15412: Attachment: (was: HDFS-15412.003.patch) > Add options to set different block scan period for diffrent StorageType > --- > > Key: HDFS-15412 > URL: https://issues.apache.org/jira/browse/HDFS-15412 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Reporter: Yang Yun >Assignee: Yang Yun >Priority: Minor > Attachments: HDFS-15412.001.patch, HDFS-15412.002.patch, > HDFS-15412.003.patch > > > For some cold data, sometime, we don't want to scan cold data as often as > hot data. Add options that we can set the scan period time according to > StorageType. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15412) Add options to set different block scan period for diffrent StorageType
[ https://issues.apache.org/jira/browse/HDFS-15412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Yun updated HDFS-15412: Status: Open (was: Patch Available) > Add options to set different block scan period for diffrent StorageType > --- > > Key: HDFS-15412 > URL: https://issues.apache.org/jira/browse/HDFS-15412 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Reporter: Yang Yun >Assignee: Yang Yun >Priority: Minor > Attachments: HDFS-15412.001.patch, HDFS-15412.002.patch, > HDFS-15412.003.patch > > > For some cold data, sometime, we don't want to scan cold data as often as > hot data. Add options that we can set the scan period time according to > StorageType. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15755) Add option to set retry times of checking failure volume and Import powermock for mock static method
[ https://issues.apache.org/jira/browse/HDFS-15755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17256540#comment-17256540 ] Yang Yun commented on HDFS-15755: - Remove a file that should not be modified and trigger a new build. > Add option to set retry times of checking failure volume and Import powermock > for mock static method > > > Key: HDFS-15755 > URL: https://issues.apache.org/jira/browse/HDFS-15755 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs, test >Reporter: Yang Yun >Assignee: Yang Yun >Priority: Minor > Attachments: HDFS-15755.001.patch, HDFS-15755.002.patch > > > For some mounted remote volume, small network shaking may causes the failure > of volume checking and the the volume will removed out from datanode. Add an > option to set retry times and interval according to StorageType. > For mocking static methods in unit test, import PowerMock to > hadoop-hdfs-project. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15755) Add option to set retry times of checking failure volume and Import powermock for mock static method
[ https://issues.apache.org/jira/browse/HDFS-15755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Yun updated HDFS-15755: Attachment: HDFS-15755.002.patch Status: Patch Available (was: Open) > Add option to set retry times of checking failure volume and Import powermock > for mock static method > > > Key: HDFS-15755 > URL: https://issues.apache.org/jira/browse/HDFS-15755 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs, test >Reporter: Yang Yun >Assignee: Yang Yun >Priority: Minor > Attachments: HDFS-15755.001.patch, HDFS-15755.002.patch > > > For some mounted remote volume, small network shaking may causes the failure > of volume checking and the the volume will removed out from datanode. Add an > option to set retry times and interval according to StorageType. > For mocking static methods in unit test, import PowerMock to > hadoop-hdfs-project. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15755) Add option to set retry times of checking failure volume and Import powermock for mock static method
[ https://issues.apache.org/jira/browse/HDFS-15755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Yun updated HDFS-15755: Status: Open (was: Patch Available) > Add option to set retry times of checking failure volume and Import powermock > for mock static method > > > Key: HDFS-15755 > URL: https://issues.apache.org/jira/browse/HDFS-15755 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs, test >Reporter: Yang Yun >Assignee: Yang Yun >Priority: Minor > Attachments: HDFS-15755.001.patch, HDFS-15755.002.patch > > > For some mounted remote volume, small network shaking may causes the failure > of volume checking and the the volume will removed out from datanode. Add an > option to set retry times and interval according to StorageType. > For mocking static methods in unit test, import PowerMock to > hadoop-hdfs-project. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15755) Add option to set retry times of checking failure volume and Import powermock for mock static method
[ https://issues.apache.org/jira/browse/HDFS-15755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Yun updated HDFS-15755: Attachment: (was: HDFS-15755.002.patch) > Add option to set retry times of checking failure volume and Import powermock > for mock static method > > > Key: HDFS-15755 > URL: https://issues.apache.org/jira/browse/HDFS-15755 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs, test >Reporter: Yang Yun >Assignee: Yang Yun >Priority: Minor > Attachments: HDFS-15755.001.patch, HDFS-15755.002.patch > > > For some mounted remote volume, small network shaking may causes the failure > of volume checking and the the volume will removed out from datanode. Add an > option to set retry times and interval according to StorageType. > For mocking static methods in unit test, import PowerMock to > hadoop-hdfs-project. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15748) RBF: Move the router related part from hadoop-federation-balance module to hadoop-hdfs-rbf.
[ https://issues.apache.org/jira/browse/HDFS-15748?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jinglun updated HDFS-15748: --- Attachment: HDFS-15748.003.patch > RBF: Move the router related part from hadoop-federation-balance module to > hadoop-hdfs-rbf. > --- > > Key: HDFS-15748 > URL: https://issues.apache.org/jira/browse/HDFS-15748 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Jinglun >Assignee: Jinglun >Priority: Major > Attachments: HDFS-15748.001.patch, HDFS-15748.002.patch, > HDFS-15748.003.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15748) RBF: Move the router related part from hadoop-federation-balance module to hadoop-hdfs-rbf.
[
https://issues.apache.org/jira/browse/HDFS-15748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17256523#comment-17256523
]
Jinglun commented on HDFS-15748:
Hi [~elgoiri] [~ayushtkn], thanks your comments !
{quote}Setting permissions to 0, If the dfs.persmissions is set to false, or
the call is from super user, this won't help us?
{quote}
I add a check in preCheck phase. If the dfs.permissions.enabled=false then
throw an IOException. About the super user I don't know how to restrict him.
Can we depend on admin's experience?:)
{quote}Do let me know if you changed some logic anywhere.
{quote}
Logic is not changed.
{quote} this would need a test for RBF-Balance
{quote}
I have tested it in my cluster. I also checked the CLASSPATH when executing
command rbfbalance.
Upload v03. Fix checkstyle, check dfs.permissions.enabled and fix
HDFSFederationBalance.md.
> RBF: Move the router related part from hadoop-federation-balance module to
> hadoop-hdfs-rbf.
> ---
>
> Key: HDFS-15748
> URL: https://issues.apache.org/jira/browse/HDFS-15748
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Jinglun
>Assignee: Jinglun
>Priority: Major
> Attachments: HDFS-15748.001.patch, HDFS-15748.002.patch,
> HDFS-15748.003.patch
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15755) Add option to set retry times of checking failure volume and Import powermock for mock static method
[
https://issues.apache.org/jira/browse/HDFS-15755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17256490#comment-17256490
]
Hadoop QA commented on HDFS-15755:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
56s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} No case conflicting files
found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch does not contain any
@author tags. {color} |
| {color:green}+1{color} | {color:green} {color} | {color:green} 0m 0s{color}
| {color:green}test4tests{color} | {color:green} The patch appears to include 1
new or modified test files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
59s{color} | {color:blue}{color} | {color:blue} Maven dependency ordering for
branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 27m
17s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 27m
30s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 23m
7s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
18s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 4m
19s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
23m 6s{color} | {color:green}{color} | {color:green} branch has no errors when
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
26s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 4m
26s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 0m
52s{color} | {color:blue}{color} | {color:blue} Used deprecated FindBugs
config; considering switching to SpotBugs. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
29s{color} | {color:blue}{color} | {color:blue} branch/hadoop-project no
findbugs output file (findbugsXml.xml) {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
19s{color} | {color:blue}{color} | {color:blue} Maven dependency ordering for
patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
12s{color} |
{color:red}https://ci-hadoop.apache.org/job/PreCommit-HDFS-Build/385/artifact/out/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timelineservice-documentstore.txt{color}
| {color:red} hadoop-yarn-server-timelineservice-documentstore in the patch
failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
15s{color} |
{color:red}https://ci-hadoop.apache.org/job/PreCommit-HDFS-Build/385/artifact/out/patch-compile-root-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04.txt{color}
| {color:red} root in the patch failed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 15s{color}
|
{color:red}https://ci-hadoop.apache.org/job/PreCommit-HDFS-Build/385/artifact/out/patch-compile-root-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04.txt{color}
| {color:red} root in the patch failed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
15s{color} |
{color:red}https://ci-hadoop.apache.org/job/PreCommit-HDFS-Build/385/artifact/out/patch-compile-root-jdkPrivateBuild-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01.txt{color}
| {color:red} root in the patch failed with JDK Private
Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 15s{color}
|
[jira] [Commented] (HDFS-15745) Make DataNodePeerMetrics#LOW_THRESHOLD_MS and MIN_OUTLIER_DETECTION_NODES configurable
[ https://issues.apache.org/jira/browse/HDFS-15745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17256481#comment-17256481 ] Ayush Saxena commented on HDFS-15745: - Thanx [~huanghaibin] for the patch. Had a quick look, Minor comments * Making configurable is Ok, but we should keep the default the value as is. * You need to add the new conf in hdfs-defaults * Checkstyle needs to be fixed. > Make DataNodePeerMetrics#LOW_THRESHOLD_MS and MIN_OUTLIER_DETECTION_NODES > configurable > -- > > Key: HDFS-15745 > URL: https://issues.apache.org/jira/browse/HDFS-15745 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Haibin Huang >Assignee: Haibin Huang >Priority: Major > Attachments: HDFS-15745-001.patch, image-2020-12-22-17-00-50-796.png > > > When i enable DataNodePeerMetrics to find slow slow peer in cluster, i found > there is a lot of slow peer but ReportingNodes's averageDelay is very low, > and these slow peer node are normal. I think the reason of why generating so > many slow peer is that the value of DataNodePeerMetrics#LOW_THRESHOLD_MS is > too small (only 5ms) and it is not configurable. The default value of slow io > warning log threshold is 300ms, i.e. > DFSConfigKeys.DFS_DATANODE_SLOW_IO_WARNING_THRESHOLD_DEFAULT = 300, so > DataNodePeerMetrics#LOW_THRESHOLD_MS should not be less than 300ms, otherwise > namenode will get a lot of invalid slow peer information. > !image-2020-12-22-17-00-50-796.png! -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15748) RBF: Move the router related part from hadoop-federation-balance module to hadoop-hdfs-rbf.
[
https://issues.apache.org/jira/browse/HDFS-15748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17256478#comment-17256478
]
Ayush Saxena commented on HDFS-15748:
-
Thanx [~LiJinglun] for the changes
In this new one, I am not sure if your {{disableWrite}} would work, Setting
permissions to 0, If the dfs.persmissions is set to false, or the call is from
super user, this won't help us? Would need some tweak there, Though in case of
RBF that is still sorted through readonly mount entry stuff.
Other changes were looking quite direct. Do let me know if you changed some
logic anywhere, though I didn't find anything bothering. But this would need
a test for RBF-Balance
I am not sure what is the best approach, The first one or now the new one.
[~elgoiri] any preference here?
> RBF: Move the router related part from hadoop-federation-balance module to
> hadoop-hdfs-rbf.
> ---
>
> Key: HDFS-15748
> URL: https://issues.apache.org/jira/browse/HDFS-15748
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Jinglun
>Assignee: Jinglun
>Priority: Major
> Attachments: HDFS-15748.001.patch, HDFS-15748.002.patch
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15755) Add option to set retry times of checking failure volume and Import powermock for mock static method
[ https://issues.apache.org/jira/browse/HDFS-15755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17256397#comment-17256397 ] Yang Yun commented on HDFS-15755: - Hi [~ayushtkn] I update the description add code(HDFS-15755.002.patch) with new changes, * Add options to set retry time and interval for volume checking according to StorageType. * Move the version to hadoop-project/pom.xml. the default verson of mockito-core is 2.28.2, Powermock-2.0.4 is compatible with it. > Add option to set retry times of checking failure volume and Import powermock > for mock static method > > > Key: HDFS-15755 > URL: https://issues.apache.org/jira/browse/HDFS-15755 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs, test >Reporter: Yang Yun >Assignee: Yang Yun >Priority: Minor > Attachments: HDFS-15755.001.patch, HDFS-15755.002.patch > > > For some mounted remote volume, small network shaking may causes the failure > of volume checking and the the volume will removed out from datanode. Add an > option to set retry times and interval according to StorageType. > For mocking static methods in unit test, import PowerMock to > hadoop-hdfs-project. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15755) Add option to set retry times of checking failure volume and Import powermock for mock static method
[ https://issues.apache.org/jira/browse/HDFS-15755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Yun updated HDFS-15755: Attachment: HDFS-15755.002.patch Status: Patch Available (was: Open) > Add option to set retry times of checking failure volume and Import powermock > for mock static method > > > Key: HDFS-15755 > URL: https://issues.apache.org/jira/browse/HDFS-15755 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs, test >Reporter: Yang Yun >Assignee: Yang Yun >Priority: Minor > Attachments: HDFS-15755.001.patch, HDFS-15755.002.patch > > > For some mounted remote volume, small network shaking may causes the failure > of volume checking and the the volume will removed out from datanode. Add an > option to set retry times and interval according to StorageType. > For mocking static methods in unit test, import PowerMock to > hadoop-hdfs-project. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15755) Add option to set retry times of checking failure volume and Import powermock for mock static method
[ https://issues.apache.org/jira/browse/HDFS-15755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Yun updated HDFS-15755: Status: Open (was: Patch Available) > Add option to set retry times of checking failure volume and Import powermock > for mock static method > > > Key: HDFS-15755 > URL: https://issues.apache.org/jira/browse/HDFS-15755 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs, test >Reporter: Yang Yun >Assignee: Yang Yun >Priority: Minor > Attachments: HDFS-15755.001.patch > > > For some mounted remote volume, small network shaking may causes the failure > of volume checking and the the volume will removed out from datanode. Add an > option to set retry times and interval according to StorageType. > For mocking static methods in unit test, import PowerMock to > hadoop-hdfs-project. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15755) Add option to set retry times of checking failure volume and Import powermock for mock static method
[ https://issues.apache.org/jira/browse/HDFS-15755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Yun updated HDFS-15755: Description: For some mounted remote volume, small network shaking may causes the failure of volume checking and the the volume will removed out from datanode. Add an option to set retry times and interval according to StorageType. For mocking static methods in unit test, import PowerMock to hadoop-hdfs-project. was: For some mounted remote volume, small network shaking may causes the f For mocking static methods in unit test, import PowerMock to hadoop-hdfs-project. The same version PowerMock has been imported in part of hadoop-yarn-project. > Add option to set retry times of checking failure volume and Import powermock > for mock static method > > > Key: HDFS-15755 > URL: https://issues.apache.org/jira/browse/HDFS-15755 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs, test >Reporter: Yang Yun >Assignee: Yang Yun >Priority: Minor > Attachments: HDFS-15755.001.patch > > > For some mounted remote volume, small network shaking may causes the failure > of volume checking and the the volume will removed out from datanode. Add an > option to set retry times and interval according to StorageType. > For mocking static methods in unit test, import PowerMock to > hadoop-hdfs-project. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15755) Add option to set retry times of checking failure volume and Import powermock for mock static method
[ https://issues.apache.org/jira/browse/HDFS-15755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Yun updated HDFS-15755: Description: For some mounted remote volume, small network shaking may causes the f For mocking static methods in unit test, import PowerMock to hadoop-hdfs-project. The same version PowerMock has been imported in part of hadoop-yarn-project. was: For mocking static methods in unit test, import PowerMock to hadoop-hdfs-project. The same version PowerMock has been imported in part of hadoop-yarn-project. > Add option to set retry times of checking failure volume and Import powermock > for mock static method > > > Key: HDFS-15755 > URL: https://issues.apache.org/jira/browse/HDFS-15755 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs, test >Reporter: Yang Yun >Assignee: Yang Yun >Priority: Minor > Attachments: HDFS-15755.001.patch > > > For some mounted remote volume, small network shaking may causes the f > For mocking static methods in unit test, import PowerMock to > hadoop-hdfs-project. > The same version PowerMock has been imported in part of hadoop-yarn-project. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15755) Add option to set retry times of checking failure volume and Import powermock for mock static method
[ https://issues.apache.org/jira/browse/HDFS-15755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Yun updated HDFS-15755: Summary: Add option to set retry times of checking failure volume and Import powermock for mock static method (was: Import powermock to test of hadoop-hdfs-project) > Add option to set retry times of checking failure volume and Import powermock > for mock static method > > > Key: HDFS-15755 > URL: https://issues.apache.org/jira/browse/HDFS-15755 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs, test >Reporter: Yang Yun >Assignee: Yang Yun >Priority: Minor > Attachments: HDFS-15755.001.patch > > > For mocking static methods in unit test, import PowerMock to > hadoop-hdfs-project. > The same version PowerMock has been imported in part of hadoop-yarn-project. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
