[jira] [Commented] (HDFS-11242) Add refresh cluster network topology operation to dfs admin
[ https://issues.apache.org/jira/browse/HDFS-11242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17782198#comment-17782198 ] Srinivasu Majeti commented on HDFS-11242: - Hi [~weichiu] , Requst you to reivew and progress it futher. Thank you! > Add refresh cluster network topology operation to dfs admin > --- > > Key: HDFS-11242 > URL: https://issues.apache.org/jira/browse/HDFS-11242 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 3.0.0-alpha1 >Reporter: Reid Chan >Priority: Minor > Labels: pull-request-available > Attachments: HDFS-11242.002.patch, HDFS-11242.patch > > Time Spent: 50m > Remaining Estimate: 0h > > The network topology and dns to switch mapping are initialized at the start > of the namenode. > If admin wants to change the topology because of new datanodes added, he has > to stop and restart namenode(s), otherwise those new added datanodes are > squeezed under /default-rack. > It is a low frequency operation, but it should be operated appropriately, so > dfs admin should take the responsibility. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11242) Add refresh cluster network topology operation to dfs admin
[ https://issues.apache.org/jira/browse/HDFS-11242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17624202#comment-17624202 ] ASF GitHub Bot commented on HDFS-11242: --- hfutatzhanghb commented on PR #3790: URL: https://github.com/apache/hadoop/pull/3790#issuecomment-1291543296 > > hi, @secfree , why was this PR closed ? > > @hfutatzhanghb I closed it because no one feedback/review in 6 months.The PR itself works well in our clusters. Very Nice, thks @secfree > Add refresh cluster network topology operation to dfs admin > --- > > Key: HDFS-11242 > URL: https://issues.apache.org/jira/browse/HDFS-11242 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 3.0.0-alpha1 >Reporter: Reid Chan >Priority: Minor > Labels: pull-request-available > Attachments: HDFS-11242.002.patch, HDFS-11242.patch > > Time Spent: 50m > Remaining Estimate: 0h > > The network topology and dns to switch mapping are initialized at the start > of the namenode. > If admin wants to change the topology because of new datanodes added, he has > to stop and restart namenode(s), otherwise those new added datanodes are > squeezed under /default-rack. > It is a low frequency operation, but it should be operated appropriately, so > dfs admin should take the responsibility. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11242) Add refresh cluster network topology operation to dfs admin
[ https://issues.apache.org/jira/browse/HDFS-11242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17624178#comment-17624178 ] ASF GitHub Bot commented on HDFS-11242: --- secfree commented on PR #3790: URL: https://github.com/apache/hadoop/pull/3790#issuecomment-1291490442 > hi, @secfree , why was this PR closed ? @hfutatzhanghb I closed it because no one feedback/review in 6 months.The PR itself works well in our clusters. > Add refresh cluster network topology operation to dfs admin > --- > > Key: HDFS-11242 > URL: https://issues.apache.org/jira/browse/HDFS-11242 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 3.0.0-alpha1 >Reporter: Reid Chan >Priority: Minor > Labels: pull-request-available > Attachments: HDFS-11242.002.patch, HDFS-11242.patch > > Time Spent: 50m > Remaining Estimate: 0h > > The network topology and dns to switch mapping are initialized at the start > of the namenode. > If admin wants to change the topology because of new datanodes added, he has > to stop and restart namenode(s), otherwise those new added datanodes are > squeezed under /default-rack. > It is a low frequency operation, but it should be operated appropriately, so > dfs admin should take the responsibility. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11242) Add refresh cluster network topology operation to dfs admin
[ https://issues.apache.org/jira/browse/HDFS-11242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17624170#comment-17624170 ] ASF GitHub Bot commented on HDFS-11242: --- hfutatzhanghb commented on PR #3790: URL: https://github.com/apache/hadoop/pull/3790#issuecomment-1291465337 hi, @secfree , why was this PR closed ? > Add refresh cluster network topology operation to dfs admin > --- > > Key: HDFS-11242 > URL: https://issues.apache.org/jira/browse/HDFS-11242 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 3.0.0-alpha1 >Reporter: Reid Chan >Priority: Minor > Labels: pull-request-available > Attachments: HDFS-11242.002.patch, HDFS-11242.patch > > Time Spent: 50m > Remaining Estimate: 0h > > The network topology and dns to switch mapping are initialized at the start > of the namenode. > If admin wants to change the topology because of new datanodes added, he has > to stop and restart namenode(s), otherwise those new added datanodes are > squeezed under /default-rack. > It is a low frequency operation, but it should be operated appropriately, so > dfs admin should take the responsibility. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11242) Add refresh cluster network topology operation to dfs admin
[ https://issues.apache.org/jira/browse/HDFS-11242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17461788#comment-17461788 ] secfree.teng commented on HDFS-11242: - + [~ferhui] please help review > Add refresh cluster network topology operation to dfs admin > --- > > Key: HDFS-11242 > URL: https://issues.apache.org/jira/browse/HDFS-11242 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 3.0.0-alpha1 >Reporter: Reid Chan >Priority: Minor > Labels: pull-request-available > Attachments: HDFS-11242.002.patch, HDFS-11242.patch > > Time Spent: 40m > Remaining Estimate: 0h > > The network topology and dns to switch mapping are initialized at the start > of the namenode. > If admin wants to change the topology because of new datanodes added, he has > to stop and restart namenode(s), otherwise those new added datanodes are > squeezed under /default-rack. > It is a low frequency operation, but it should be operated appropriately, so > dfs admin should take the responsibility. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11242) Add refresh cluster network topology operation to dfs admin
[ https://issues.apache.org/jira/browse/HDFS-11242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17461787#comment-17461787 ] secfree.teng commented on HDFS-11242: - Confirmed with [~reidchan] that I can follow this based on his original PR. I have done rebase and changed to refresh one host. Here is the PR: [https://github.com/apache/hadoop/pull/3790] [~larsfrancke] [~hexiaoqiao] can you help review? > Add refresh cluster network topology operation to dfs admin > --- > > Key: HDFS-11242 > URL: https://issues.apache.org/jira/browse/HDFS-11242 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 3.0.0-alpha1 >Reporter: Reid Chan >Priority: Minor > Labels: pull-request-available > Attachments: HDFS-11242.002.patch, HDFS-11242.patch > > Time Spent: 40m > Remaining Estimate: 0h > > The network topology and dns to switch mapping are initialized at the start > of the namenode. > If admin wants to change the topology because of new datanodes added, he has > to stop and restart namenode(s), otherwise those new added datanodes are > squeezed under /default-rack. > It is a low frequency operation, but it should be operated appropriately, so > dfs admin should take the responsibility. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11242) Add refresh cluster network topology operation to dfs admin
[
https://issues.apache.org/jira/browse/HDFS-11242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17457815#comment-17457815
]
Hadoop QA commented on HDFS-11242:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 12s{color}
| {color:red}{color} | {color:red} HDFS-11242 does not apply to trunk. Rebase
required? Wrong Branch? See
https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute for help.
{color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HDFS-11242 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12843182/HDFS-11242.002.patch |
| Console output |
https://ci-hadoop.apache.org/job/PreCommit-HDFS-Build/750/console |
| versions | git=2.17.1 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
> Add refresh cluster network topology operation to dfs admin
> ---
>
> Key: HDFS-11242
> URL: https://issues.apache.org/jira/browse/HDFS-11242
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Affects Versions: 3.0.0-alpha1
>Reporter: Reid Chan
>Priority: Minor
> Labels: pull-request-available
> Attachments: HDFS-11242.002.patch, HDFS-11242.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> The network topology and dns to switch mapping are initialized at the start
> of the namenode.
> If admin wants to change the topology because of new datanodes added, he has
> to stop and restart namenode(s), otherwise those new added datanodes are
> squeezed under /default-rack.
> It is a low frequency operation, but it should be operated appropriately, so
> dfs admin should take the responsibility.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11242) Add refresh cluster network topology operation to dfs admin
[ https://issues.apache.org/jira/browse/HDFS-11242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16870634#comment-16870634 ] Lars Francke commented on HDFS-11242: - I'm afraid I still don't fully understand. Either way [~reidchan] I think this is a good change and desperately needed. Are you willing to rebase and bring up-to-date? If you like I can ask on the mailing list about some review support before you spent time on this. > Add refresh cluster network topology operation to dfs admin > --- > > Key: HDFS-11242 > URL: https://issues.apache.org/jira/browse/HDFS-11242 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 3.0.0-alpha1 >Reporter: Reid Chan >Priority: Minor > Attachments: HDFS-11242.002.patch, HDFS-11242.patch > > > The network topology and dns to switch mapping are initialized at the start > of the namenode. > If admin wants to change the topology because of new datanodes added, he has > to stop and restart namenode(s), otherwise those new added datanodes are > squeezed under /default-rack. > It is a low frequency operation, but it should be operated appropriately, so > dfs admin should take the responsibility. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11242) Add refresh cluster network topology operation to dfs admin
[ https://issues.apache.org/jira/browse/HDFS-11242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16870574#comment-16870574 ] He Xiaoqiao commented on HDFS-11242: Thanks [~larsfrancke] for your doubt and the detailed comments. For the situation you said above, I think the proper way is: a. clean CachedDNSToSwitchMapping#cache complete or just clean one host. (which do not support currently). b. fix the bug via Consul (sorry I am not familiar with that). c. stop DataNode and better to wait for proper time (630s by default which namenode consider datanode dead) then start DataNode. Please correct me if something wrong. I want to state that change network topology / rack aware script is very risky. We should change it cautiously. especially for large cluster. FYI. > Add refresh cluster network topology operation to dfs admin > --- > > Key: HDFS-11242 > URL: https://issues.apache.org/jira/browse/HDFS-11242 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 3.0.0-alpha1 >Reporter: Reid Chan >Priority: Minor > Attachments: HDFS-11242.002.patch, HDFS-11242.patch > > > The network topology and dns to switch mapping are initialized at the start > of the namenode. > If admin wants to change the topology because of new datanodes added, he has > to stop and restart namenode(s), otherwise those new added datanodes are > squeezed under /default-rack. > It is a low frequency operation, but it should be operated appropriately, so > dfs admin should take the responsibility. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11242) Add refresh cluster network topology operation to dfs admin
[ https://issues.apache.org/jira/browse/HDFS-11242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16870566#comment-16870566 ] Lars Francke commented on HDFS-11242: - I think I'm getting a clearer picture now but (and there's always a chance I'm mistaken) I believe what you're suggesting does not work. Focusing on your second point: This is what's not currently possible! Because the script is only ever called _once_ and the result is then cached. Look at CachedDNSToSwitchMapping. So if you have a flexible rack aware script (which we do) but have a bug in it (e.g. reading racks from Consul but there's an error in the mapping) you have no way of fixing that error without restarting the NameNode because the bad result is cached. Regarding your first point: Thanks for the pointer. After a restart a fixed rack awareness script would be called anyway so this wouldn't make any difference. In short: I believe the patch & solution that [~reidchan] proposed is a very valid use-case. > Add refresh cluster network topology operation to dfs admin > --- > > Key: HDFS-11242 > URL: https://issues.apache.org/jira/browse/HDFS-11242 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 3.0.0-alpha1 >Reporter: Reid Chan >Priority: Minor > Attachments: HDFS-11242.002.patch, HDFS-11242.patch > > > The network topology and dns to switch mapping are initialized at the start > of the namenode. > If admin wants to change the topology because of new datanodes added, he has > to stop and restart namenode(s), otherwise those new added datanodes are > squeezed under /default-rack. > It is a low frequency operation, but it should be operated appropriately, so > dfs admin should take the responsibility. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11242) Add refresh cluster network topology operation to dfs admin
[
https://issues.apache.org/jira/browse/HDFS-11242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16870557#comment-16870557
]
He Xiaoqiao commented on HDFS-11242:
Thanks [~larsfrancke] for your quick response. IIUC, namenode will check if
replications of one block has enough racks based on PlacementPolicy when
namenode restart or failover. {{BlockManager#postponedMisreplicatedBlocks}}
could be one data structure which is related. Please double check and correct
me if something wrong.
As mentioned above, I prefer to design flexible rack aware script (refer to
https://hadoop.apache.org/docs/r3.2.0/hadoop-project-dist/hadoop-common/RackAwareness.html),
rather than have to specify proper rack when add new nodes.
Thanks again.
> Add refresh cluster network topology operation to dfs admin
> ---
>
> Key: HDFS-11242
> URL: https://issues.apache.org/jira/browse/HDFS-11242
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Affects Versions: 3.0.0-alpha1
>Reporter: Reid Chan
>Priority: Minor
> Attachments: HDFS-11242.002.patch, HDFS-11242.patch
>
>
> The network topology and dns to switch mapping are initialized at the start
> of the namenode.
> If admin wants to change the topology because of new datanodes added, he has
> to stop and restart namenode(s), otherwise those new added datanodes are
> squeezed under /default-rack.
> It is a low frequency operation, but it should be operated appropriately, so
> dfs admin should take the responsibility.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11242) Add refresh cluster network topology operation to dfs admin
[ https://issues.apache.org/jira/browse/HDFS-11242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16870546#comment-16870546 ] Lars Francke commented on HDFS-11242: - Specifically: What [~reidchan] can currently already be done but requires a restart, so the same "risk" already exists. Does HDFS start moving data when the topolgy changes? I always was under the assumption that it only happens on a balance operation. > Add refresh cluster network topology operation to dfs admin > --- > > Key: HDFS-11242 > URL: https://issues.apache.org/jira/browse/HDFS-11242 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 3.0.0-alpha1 >Reporter: Reid Chan >Priority: Minor > Attachments: HDFS-11242.002.patch, HDFS-11242.patch > > > The network topology and dns to switch mapping are initialized at the start > of the namenode. > If admin wants to change the topology because of new datanodes added, he has > to stop and restart namenode(s), otherwise those new added datanodes are > squeezed under /default-rack. > It is a low frequency operation, but it should be operated appropriately, so > dfs admin should take the responsibility. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11242) Add refresh cluster network topology operation to dfs admin
[ https://issues.apache.org/jira/browse/HDFS-11242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16870544#comment-16870544 ] Lars Francke commented on HDFS-11242: - [~hexiaoqiao] I don't understand. Can you be more specific? The usecase I see is simple: You add new nodes but forget to specify the proper rack, the only way to fix it now is to restart the NameNodes which is not a good experience. I haven't looked at the patch in detail but all I need is a way to call reloadCachedMappings on the DNSToSwitchMapping. We can change the ScriptBasedMapping to not extend CachedDNSToSwitchMapping or to change CachedDNSToSwitchMapping to periodically invalidate all entries or something like that. > Add refresh cluster network topology operation to dfs admin > --- > > Key: HDFS-11242 > URL: https://issues.apache.org/jira/browse/HDFS-11242 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 3.0.0-alpha1 >Reporter: Reid Chan >Priority: Minor > Attachments: HDFS-11242.002.patch, HDFS-11242.patch > > > The network topology and dns to switch mapping are initialized at the start > of the namenode. > If admin wants to change the topology because of new datanodes added, he has > to stop and restart namenode(s), otherwise those new added datanodes are > squeezed under /default-rack. > It is a low frequency operation, but it should be operated appropriately, so > dfs admin should take the responsibility. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11242) Add refresh cluster network topology operation to dfs admin
[ https://issues.apache.org/jira/browse/HDFS-11242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16870542#comment-16870542 ] He Xiaoqiao commented on HDFS-11242: It looks like a good idea in some ways. But I do not think we should do that, since it could bring huge data transfer risk and destroy stability of cluster if we provide tools to refresh network topology dynamically. IMO, we should design more flexible rack aware script and avoid to change network topology. > Add refresh cluster network topology operation to dfs admin > --- > > Key: HDFS-11242 > URL: https://issues.apache.org/jira/browse/HDFS-11242 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 3.0.0-alpha1 >Reporter: Reid Chan >Priority: Minor > Attachments: HDFS-11242.002.patch, HDFS-11242.patch > > > The network topology and dns to switch mapping are initialized at the start > of the namenode. > If admin wants to change the topology because of new datanodes added, he has > to stop and restart namenode(s), otherwise those new added datanodes are > squeezed under /default-rack. > It is a low frequency operation, but it should be operated appropriately, so > dfs admin should take the responsibility. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11242) Add refresh cluster network topology operation to dfs admin
[ https://issues.apache.org/jira/browse/HDFS-11242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16870532#comment-16870532 ] Reid Chan commented on HDFS-11242: -- Yes, only if any committer or member can review or comment, most importantly willing to have it in HDFS, otherwise it will just get no notice and set aside as it was. > Add refresh cluster network topology operation to dfs admin > --- > > Key: HDFS-11242 > URL: https://issues.apache.org/jira/browse/HDFS-11242 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 3.0.0-alpha1 >Reporter: Reid Chan >Priority: Minor > Attachments: HDFS-11242.002.patch, HDFS-11242.patch > > > The network topology and dns to switch mapping are initialized at the start > of the namenode. > If admin wants to change the topology because of new datanodes added, he has > to stop and restart namenode(s), otherwise those new added datanodes are > squeezed under /default-rack. > It is a low frequency operation, but it should be operated appropriately, so > dfs admin should take the responsibility. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11242) Add refresh cluster network topology operation to dfs admin
[
https://issues.apache.org/jira/browse/HDFS-11242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16870486#comment-16870486
]
Hadoop QA commented on HDFS-11242:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 6s{color}
| {color:red} HDFS-11242 does not apply to trunk. Rebase required? Wrong
Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HDFS-11242 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12843182/HDFS-11242.002.patch |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/27043/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> Add refresh cluster network topology operation to dfs admin
> ---
>
> Key: HDFS-11242
> URL: https://issues.apache.org/jira/browse/HDFS-11242
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Affects Versions: 3.0.0-alpha1
>Reporter: Reid Chan
>Priority: Minor
> Attachments: HDFS-11242.002.patch, HDFS-11242.patch
>
>
> The network topology and dns to switch mapping are initialized at the start
> of the namenode.
> If admin wants to change the topology because of new datanodes added, he has
> to stop and restart namenode(s), otherwise those new added datanodes are
> squeezed under /default-rack.
> It is a low frequency operation, but it should be operated appropriately, so
> dfs admin should take the responsibility.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11242) Add refresh cluster network topology operation to dfs admin
[ https://issues.apache.org/jira/browse/HDFS-11242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16870483#comment-16870483 ] Lars Francke commented on HDFS-11242: - This would be super useful to have. [~reidchan] are you by any chance looking to revive this patch? > Add refresh cluster network topology operation to dfs admin > --- > > Key: HDFS-11242 > URL: https://issues.apache.org/jira/browse/HDFS-11242 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 3.0.0-alpha1 >Reporter: Reid Chan >Priority: Minor > Attachments: HDFS-11242.002.patch, HDFS-11242.patch > > > The network topology and dns to switch mapping are initialized at the start > of the namenode. > If admin wants to change the topology because of new datanodes added, he has > to stop and restart namenode(s), otherwise those new added datanodes are > squeezed under /default-rack. > It is a low frequency operation, but it should be operated appropriately, so > dfs admin should take the responsibility. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11242) Add refresh cluster network topology operation to dfs admin
[
https://issues.apache.org/jira/browse/HDFS-11242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15747984#comment-15747984
]
Hadoop QA commented on HDFS-11242:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
21s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
25s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
2s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
8s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 1m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
23s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 36s{color} | {color:orange} hadoop-hdfs-project: The patch generated 8 new +
640 unchanged - 1 fixed = 648 total (was 641) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
53s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 95m 10s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}125m 25s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency |
| | hadoop.hdfs.TestReadStripedFileWithMissingBlocks |
| | hadoop.hdfs.TestDistributedFileSystem |
| | hadoop.hdfs.TestSecureEncryptionZoneWithKMS |
| | hadoop.hdfs.TestTrashWithSecureEncryptionZones |
| | hadoop.hdfs.server.namenode.TestRefreshTopology |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HDFS-11242 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12843182/HDFS-11242.002.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle cc |
| uname | Linux 6a6d435e2963 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29
20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 1f14f6d |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
|
[jira] [Commented] (HDFS-11242) Add refresh cluster network topology operation to dfs admin
[
https://issues.apache.org/jira/browse/HDFS-11242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15747564#comment-15747564
]
Hadoop QA commented on HDFS-11242:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
23s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
29s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
11s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
59s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
7s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
26s{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
47s{color} | {color:red} hadoop-hdfs-project in the patch failed. {color} |
| {color:red}-1{color} | {color:red} cc {color} | {color:red} 0m 47s{color} |
{color:red} hadoop-hdfs-project in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 47s{color}
| {color:red} hadoop-hdfs-project in the patch failed. {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 33s{color} | {color:orange} hadoop-hdfs-project: The patch generated 26 new
+ 576 unchanged - 1 fixed = 602 total (was 577) {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
33s{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
16s{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
2s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 33s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
21s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 27m 30s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HDFS-11242 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12843164/HDFS-11242.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle cc |
| uname | Linux f9f3d4c91937 3.13.0-103-generic #150-Ubuntu SMP Thu Nov 24
10:34:17 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 1f14f6d |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| mvninstall |
https://builds.apache.org/job/PreCommit-HDFS-Build/17854/artifact/patchprocess/patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs.txt
|
| compile |
