[jira] [Created] (HDFS-14344) Erasure Coding: Miss EC block after decommission and restart NN
maobaolong created HDFS-14344: - Summary: Erasure Coding: Miss EC block after decommission and restart NN Key: HDFS-14344 URL: https://issues.apache.org/jira/browse/HDFS-14344 Project: Hadoop HDFS Issue Type: Sub-task Components: ec, erasure-coding, namenode Affects Versions: 3.3.0 Reporter: maobaolong -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-14353) Erasure Coding: metrics xmitsInProgress become to negative.
maobaolong created HDFS-14353: - Summary: Erasure Coding: metrics xmitsInProgress become to negative. Key: HDFS-14353 URL: https://issues.apache.org/jira/browse/HDFS-14353 Project: Hadoop HDFS Issue Type: Sub-task Components: datanode, erasure-coding Affects Versions: 3.3.0 Reporter: maobaolong -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14353) Erasure Coding: metrics xmitsInProgress become to negative.
[ https://issues.apache.org/jira/browse/HDFS-14353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16789194#comment-16789194 ] maobaolong commented on HDFS-14353: --- !screenshot-1.png! > Erasure Coding: metrics xmitsInProgress become to negative. > --- > > Key: HDFS-14353 > URL: https://issues.apache.org/jira/browse/HDFS-14353 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, erasure-coding >Affects Versions: 3.3.0 >Reporter: maobaolong >Priority: Major > Attachments: screenshot-1.png > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14353) Erasure Coding: metrics xmitsInProgress become to negative.
[ https://issues.apache.org/jira/browse/HDFS-14353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] maobaolong updated HDFS-14353: -- Attachment: screenshot-1.png > Erasure Coding: metrics xmitsInProgress become to negative. > --- > > Key: HDFS-14353 > URL: https://issues.apache.org/jira/browse/HDFS-14353 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, erasure-coding >Affects Versions: 3.3.0 >Reporter: maobaolong >Priority: Major > Attachments: screenshot-1.png > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14353) Erasure Coding: metrics xmitsInProgress become to negative.
[ https://issues.apache.org/jira/browse/HDFS-14353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16789208#comment-16789208 ] maobaolong commented on HDFS-14353: --- The suspect code i think is around the class ErasureCodingWorker and StripedBlockReconstructor. > Erasure Coding: metrics xmitsInProgress become to negative. > --- > > Key: HDFS-14353 > URL: https://issues.apache.org/jira/browse/HDFS-14353 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, erasure-coding >Affects Versions: 3.3.0 >Reporter: maobaolong >Priority: Major > Attachments: screenshot-1.png > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-14353) Erasure Coding: metrics xmitsInProgress become to negative.
[
https://issues.apache.org/jira/browse/HDFS-14353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16789208#comment-16789208
]
maobaolong edited comment on HDFS-14353 at 3/11/19 6:47 AM:
The suspect code i think is around the class ErasureCodingWorker and
StripedBlockReconstructor.
{code:java}
public void processErasureCodingTasks(
Collection ecTasks) {
for (BlockECReconstructionInfo reconInfo : ecTasks) {
int xmitsSubmitted = 0;
try {
StripedReconstructionInfo stripedReconInfo =
new StripedReconstructionInfo(
reconInfo.getExtendedBlock(), reconInfo.getErasureCodingPolicy(),
reconInfo.getLiveBlockIndices(), reconInfo.getSourceDnInfos(),
reconInfo.getTargetDnInfos(), reconInfo.getTargetStorageTypes(),
reconInfo.getTargetStorageIDs());
// It may throw IllegalArgumentException from task#stripedReader
// constructor.
final StripedBlockReconstructor task =
new StripedBlockReconstructor(this, stripedReconInfo);
if (task.hasValidTargets()) {
// See HDFS-12044. We increase xmitsInProgress even the task is only
// enqueued, so that
// 1) NN will not send more tasks than what DN can execute and
// 2) DN will not throw away reconstruction tasks, and instead keeps
// an unbounded number of tasks in the executor's task queue.
xmitsSubmitted = Math.max((int)(task.getXmits() * xmitWeight), 1);
getDatanode().incrementXmitsInProcess(xmitsSubmitted);
stripedReconstructionPool.submit(task);
} else {
LOG.warn("No missing internal block. Skip reconstruction for task:{}",
reconInfo);
}
} catch (Throwable e) {
getDatanode().decrementXmitsInProgress(xmitsSubmitted);
LOG.warn("Failed to reconstruct striped block {}",
reconInfo.getExtendedBlock().getLocalBlock(), e);
}
}
}
{code}
was (Author: maobaolong):
The suspect code i think is around the class ErasureCodingWorker and
StripedBlockReconstructor.
> Erasure Coding: metrics xmitsInProgress become to negative.
> ---
>
> Key: HDFS-14353
> URL: https://issues.apache.org/jira/browse/HDFS-14353
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, erasure-coding
>Affects Versions: 3.3.0
>Reporter: maobaolong
>Priority: Major
> Attachments: screenshot-1.png
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14353) Erasure Coding: metrics xmitsInProgress become to negative.
[ https://issues.apache.org/jira/browse/HDFS-14353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16789407#comment-16789407 ] maobaolong commented on HDFS-14353: --- I think the problem caused by the following logic: increase: xmitsSubmitted = Math.max((int)(task.getXmits() * xmitWeight), 1); getDatanode().incrementXmitsInProcess(xmitsSubmitted); descrease: In StripedBlockReconstructor#run: getDatanode().decrementXmitsInProgress(getXmits()); In my case, increase 1, decrease 3. My operation is append 3 datanode host name into the exclude_datanode_hosts file, execute “hdfs dfsadmin -refreshNodes”. PS. the 3 exclude nodes are stored the EC blocks for the RS(3,2) file. > Erasure Coding: metrics xmitsInProgress become to negative. > --- > > Key: HDFS-14353 > URL: https://issues.apache.org/jira/browse/HDFS-14353 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, erasure-coding >Affects Versions: 3.3.0 >Reporter: maobaolong >Priority: Major > Attachments: screenshot-1.png > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14353) Erasure Coding: metrics xmitsInProgress become to negative.
[
https://issues.apache.org/jira/browse/HDFS-14353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16789445#comment-16789445
]
maobaolong commented on HDFS-14353:
---
[~eddyxu] Please take a look. I think the StripedBlockReconstructor#run should
be:
{code:java}
@Override
public void run() {
try {
x
} catch (Throwable e) {
x
} finally {
getDatanode().decrementXmitsInProgress(Math.max((int)(getXmits() *
xmitWeight), 1));
x
}
}
{code}
> Erasure Coding: metrics xmitsInProgress become to negative.
> ---
>
> Key: HDFS-14353
> URL: https://issues.apache.org/jira/browse/HDFS-14353
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, erasure-coding
>Affects Versions: 3.3.0
>Reporter: maobaolong
>Priority: Major
> Attachments: screenshot-1.png
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12482) Provide a configuration to adjust the weight of EC recovery tasks to adjust the speed of recovery
[
https://issues.apache.org/jira/browse/HDFS-12482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16789448#comment-16789448
]
maobaolong commented on HDFS-12482:
---
[~eddyxu] Please take a look HDFS-14353. I think the
StripedBlockReconstructor#run should be:
{code:java}
@Override
public void run() {
try {
x
} catch (Throwable e) {
x
} finally {
getDatanode().decrementXmitsInProgress(Math.max((int)(getXmits() *
xmitWeight), 1));
x
}
}
{code}
> Provide a configuration to adjust the weight of EC recovery tasks to adjust
> the speed of recovery
> -
>
> Key: HDFS-12482
> URL: https://issues.apache.org/jira/browse/HDFS-12482
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: erasure-coding
>Affects Versions: 3.0.0-alpha4
>Reporter: Lei (Eddy) Xu
>Assignee: Lei (Eddy) Xu
>Priority: Minor
> Labels: hdfs-ec-3.0-nice-to-have
> Fix For: 3.0.0
>
> Attachments: HDFS-12482.00.patch, HDFS-12482.01.patch,
> HDFS-12482.02.patch, HDFS-12482.03.patch, HDFS-12482.04.patch,
> HDFS-12482.05.patch
>
>
> The relative speed of EC recovery comparing to 3x replica recovery is a
> function of (EC codec, number of sources, NIC speed, and CPU speed, and etc).
> Currently the EC recovery has a fixed {{xmitsInProgress}} of {{max(# of
> sources, # of targets)}} comparing to {{1}} for 3x replica recovery, and NN
> uses {{xmitsInProgress}} to decide how much recovery tasks to schedule to the
> DataNode this we can add a coefficient for user to tune the weight of EC
> recovery tasks.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-14353) Erasure Coding: metrics xmitsInProgress become to negative.
[ https://issues.apache.org/jira/browse/HDFS-14353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] maobaolong reassigned HDFS-14353: - Assignee: maobaolong > Erasure Coding: metrics xmitsInProgress become to negative. > --- > > Key: HDFS-14353 > URL: https://issues.apache.org/jira/browse/HDFS-14353 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, erasure-coding >Affects Versions: 3.3.0 >Reporter: maobaolong >Assignee: maobaolong >Priority: Major > Attachments: screenshot-1.png > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14353) Erasure Coding: metrics xmitsInProgress become to negative.
[ https://issues.apache.org/jira/browse/HDFS-14353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] maobaolong updated HDFS-14353: -- Attachment: HDFS-14353.001 > Erasure Coding: metrics xmitsInProgress become to negative. > --- > > Key: HDFS-14353 > URL: https://issues.apache.org/jira/browse/HDFS-14353 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, erasure-coding >Affects Versions: 3.3.0 >Reporter: maobaolong >Assignee: maobaolong >Priority: Major > Attachments: HDFS-14353.001, screenshot-1.png > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14353) Erasure Coding: metrics xmitsInProgress become to negative.
[ https://issues.apache.org/jira/browse/HDFS-14353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] maobaolong updated HDFS-14353: -- Fix Version/s: 3.3.0 Status: Patch Available (was: Open) > Erasure Coding: metrics xmitsInProgress become to negative. > --- > > Key: HDFS-14353 > URL: https://issues.apache.org/jira/browse/HDFS-14353 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, erasure-coding >Affects Versions: 3.3.0 >Reporter: maobaolong >Assignee: maobaolong >Priority: Major > Fix For: 3.3.0 > > Attachments: HDFS-14353.001, screenshot-1.png > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14353) Erasure Coding: metrics xmitsInProgress become to negative.
[ https://issues.apache.org/jira/browse/HDFS-14353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16790129#comment-16790129 ] maobaolong commented on HDFS-14353: --- [~linyiqun] Would you like to take a look at this patch?. After we use Hadoop3.x with EC feature, we met this issue, thank you advance. > Erasure Coding: metrics xmitsInProgress become to negative. > --- > > Key: HDFS-14353 > URL: https://issues.apache.org/jira/browse/HDFS-14353 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, erasure-coding >Affects Versions: 3.3.0 >Reporter: maobaolong >Assignee: maobaolong >Priority: Major > Fix For: 3.3.0 > > Attachments: HDFS-14353.001, screenshot-1.png > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14353) Erasure Coding: metrics xmitsInProgress become to negative.
[ https://issues.apache.org/jira/browse/HDFS-14353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16794038#comment-16794038 ] maobaolong commented on HDFS-14353: --- [~ayushtkn]Please take a look at this issue, maybe it can lead to missing block. > Erasure Coding: metrics xmitsInProgress become to negative. > --- > > Key: HDFS-14353 > URL: https://issues.apache.org/jira/browse/HDFS-14353 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, erasure-coding >Affects Versions: 3.3.0 >Reporter: maobaolong >Assignee: maobaolong >Priority: Major > Fix For: 3.3.0 > > Attachments: HDFS-14353.001, screenshot-1.png > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14344) Erasure Coding: Miss EC block after decommission and restart NN
[ https://issues.apache.org/jira/browse/HDFS-14344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16800432#comment-16800432 ] maobaolong commented on HDFS-14344: --- [~ayushtkn] Do you think HDFS-14353 can cause this issue? > Erasure Coding: Miss EC block after decommission and restart NN > --- > > Key: HDFS-14344 > URL: https://issues.apache.org/jira/browse/HDFS-14344 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ec, erasure-coding, namenode >Affects Versions: 3.3.0 >Reporter: maobaolong >Priority: Critical > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-13881) Export or Import a dirImage
maobaolong created HDFS-13881: - Summary: Export or Import a dirImage Key: HDFS-13881 URL: https://issues.apache.org/jira/browse/HDFS-13881 Project: Hadoop HDFS Issue Type: New Feature Components: namenode Affects Versions: 3.1.1 Reporter: maobaolong Assignee: maobaolong -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13881) Export or Import a dirImage
[ https://issues.apache.org/jira/browse/HDFS-13881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] maobaolong updated HDFS-13881: -- Description: Now, we want to copy a directory meta tree from a cluster's namenode to another cluster's namenode. So, i suggest that namenode can support export a directory to a image file, and another namenode can import it. > Export or Import a dirImage > --- > > Key: HDFS-13881 > URL: https://issues.apache.org/jira/browse/HDFS-13881 > Project: Hadoop HDFS > Issue Type: New Feature > Components: namenode >Affects Versions: 3.1.1 >Reporter: maobaolong >Assignee: maobaolong >Priority: Major > > Now, we want to copy a directory meta tree from a cluster's namenode to > another cluster's namenode. > So, i suggest that namenode can support export a directory to a image file, > and another namenode can import it. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-7663) Erasure Coding: Append on striped file
[ https://issues.apache.org/jira/browse/HDFS-7663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16604149#comment-16604149 ] maobaolong commented on HDFS-7663: -- [~walter.k.su] Is there any update? It is a important feature. > Erasure Coding: Append on striped file > -- > > Key: HDFS-7663 > URL: https://issues.apache.org/jira/browse/HDFS-7663 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: 3.0.0-alpha1 >Reporter: Jing Zhao >Assignee: Walter Su >Priority: Major > Attachments: HDFS-7663.00.txt, HDFS-7663.01.patch > > > Append should be easy if we have variable length block support from > HDFS-3689, i.e., the new data will be appended to a new block. We need to > revisit whether and how to support appending data to the original last block. > 1. Append to a closed striped file, with NEW_BLOCK flag enabled (this) > 2. Append to a under-construction striped file, with NEW_BLOCK flag enabled > (HDFS-9173) > 3. Append to a striped file, by appending to last block group (follow-on) > This jira attempts to implement the #1, and also track #2, #3. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13902) Add jmx conf and stacks menus to the datanode page
[ https://issues.apache.org/jira/browse/HDFS-13902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16606667#comment-16606667 ] maobaolong commented on HDFS-13902: --- [~fengchuang] Look good, we can easy to open the stacks jmx and conf page link. thank you. > Add jmx conf and stacks menus to the datanode page > --- > > Key: HDFS-13902 > URL: https://issues.apache.org/jira/browse/HDFS-13902 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Affects Versions: 3.0.3 >Reporter: fengchuang >Priority: Minor > Attachments: HDFS-13902.001.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13034) The DFSUsed value bigger than the Capacity
[
https://issues.apache.org/jira/browse/HDFS-13034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16331588#comment-16331588
]
maobaolong commented on HDFS-13034:
---
Thank you [~brahmareddy], I paste a part of the datanode jmx response here, it
shown this issue.
{code}
// code json
{
"name": "Hadoop:service=DataNode,name=FSDatasetState-null",
"modelerType":
"org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl",
"Remaining": 0,
"StorageInfo": "FSDataset{dirpath='[/data0/dfs/current, /data1/dfs/current,
/data2/dfs/current, /data3/dfs/current, /data4/dfs/current, /data5/dfs/current,
/data6/dfs/current, /data7/dfs/current, /data8/dfs/current, /data9/dfs/current,
/data10/dfs/current, /data11/dfs/current]'}",
"Capacity": 6019039002624,
"DfsUsed": 13556514861845,
"CacheCapacity": 0,
"CacheUsed": 0,
"NumFailedVolumes": 0,
"FailedStorageLocations": [],
"LastVolumeFailureDate": 0,
"EstimatedCapacityLostTotal": 0,
"NumBlocksCached": 0,
"NumBlocksFailedToCache": 0,
"NumBlocksFailedToUncache": 25679
}
{code}
> The DFSUsed value bigger than the Capacity
> --
>
> Key: HDFS-13034
> URL: https://issues.apache.org/jira/browse/HDFS-13034
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Affects Versions: 2.7.1
>Reporter: maobaolong
>Priority: Minor
>
> ||Node||Last contact||Admin State||Capacity||Used||Non DFS
> Used||Remaining||Blocks||Block pool used||Failed Volumes||Version||
> |A|0|In Service|20.65 TB|18.26 TB|0 B|1.27 TB|24330|2.57 TB (12.42%)|0|2.7.1|
> |B|2|In Service|5.47 TB|12.78 TB|0 B|1.46 TB|27657|2.65 TB (48.37%)|0|2.7.1|
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13034) The DFSUsed value bigger than the Capacity
[ https://issues.apache.org/jira/browse/HDFS-13034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16331614#comment-16331614 ] maobaolong commented on HDFS-13034: --- The reason is we reset the reserved to a very big value, so the capacity decrease, and lower than the used value. > The DFSUsed value bigger than the Capacity > -- > > Key: HDFS-13034 > URL: https://issues.apache.org/jira/browse/HDFS-13034 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs >Affects Versions: 2.7.1 >Reporter: maobaolong >Priority: Minor > > ||Node||Last contact||Admin State||Capacity||Used||Non DFS > Used||Remaining||Blocks||Block pool used||Failed Volumes||Version|| > |A|0|In Service|20.65 TB|18.26 TB|0 B|1.27 TB|24330|2.57 TB (12.42%)|0|2.7.1| > |B|2|In Service|5.47 TB|12.78 TB|0 B|1.46 TB|27657|2.65 TB (48.37%)|0|2.7.1| -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-13034) The DFSUsed value bigger than the Capacity
[ https://issues.apache.org/jira/browse/HDFS-13034?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] maobaolong resolved HDFS-13034. --- Resolution: Won't Fix This situation may be amazing, but it is not a issue. > The DFSUsed value bigger than the Capacity > -- > > Key: HDFS-13034 > URL: https://issues.apache.org/jira/browse/HDFS-13034 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs >Affects Versions: 2.7.1 >Reporter: maobaolong >Priority: Minor > > ||Node||Last contact||Admin State||Capacity||Used||Non DFS > Used||Remaining||Blocks||Block pool used||Failed Volumes||Version|| > |A|0|In Service|20.65 TB|18.26 TB|0 B|1.27 TB|24330|2.57 TB (12.42%)|0|2.7.1| > |B|2|In Service|5.47 TB|12.78 TB|0 B|1.46 TB|27657|2.65 TB (48.37%)|0|2.7.1| -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13083) Fix HDFS Router Federation doc error
[
https://issues.apache.org/jira/browse/HDFS-13083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16344692#comment-16344692
]
maobaolong commented on HDFS-13083:
---
What a great contribution, this issue trouble me many day.
Thank you for contribution. 👍
> Fix HDFS Router Federation doc error
>
>
> Key: HDFS-13083
> URL: https://issues.apache.org/jira/browse/HDFS-13083
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: federation
>Affects Versions: 2.9.0, 3.0.0
> Environment: CentOS6.5
> Hadoop 3.0.0
>Reporter: tartarus
>Priority: Major
> Labels: Router
> Fix For: 3.0.0
>
> Attachments: HDFS-13083.patch
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> HDFS Router Federation doc error
> Configuration instructions
> {quote}
> dfs.namenodes.ns-fed
> r1,r2
>
> {quote}
> then execute : *hdfs dfs -ls hdfs://ns-fed/tmp/* failed with
> *{{Couldn't create proxy provider class
> org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider}}*
> The correct configuration is
> {quote}
> dfs.{color:#FF}ha{color}.namenodes.ns-fed
> r1,r2
>
> {quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-13195) DataNode conf page cannot display the current value after reconfig
maobaolong created HDFS-13195:
-
Summary: DataNode conf page cannot display the current value
after reconfig
Key: HDFS-13195
URL: https://issues.apache.org/jira/browse/HDFS-13195
Project: Hadoop HDFS
Issue Type: Bug
Components: datanode
Affects Versions: 2.7.1
Reporter: maobaolong
Now the branch-2.7 support dfs.datanode.data.dir reconfig, but after i reconfig
this key, the conf page's value is still the old config value.
The reason is that:
{code:java}
public DatanodeHttpServer(final Configuration conf,
final DataNode datanode,
final ServerSocketChannel externalHttpChannel)
throws IOException {
this.conf = conf;
Configuration confForInfoServer = new Configuration(conf);
confForInfoServer.setInt(HttpServer2.HTTP_MAX_THREADS, 10);
HttpServer2.Builder builder = new HttpServer2.Builder()
.setName("datanode")
.setConf(confForInfoServer)
.setACL(new AccessControlList(conf.get(DFS_ADMIN, " ")))
.hostName(getHostnameForSpnegoPrincipal(confForInfoServer))
.addEndpoint(URI.create("http://localhost:0";))
.setFindPort(true);
this.infoServer = builder.build();
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13195) DataNode conf page cannot display the current value after reconfig
[
https://issues.apache.org/jira/browse/HDFS-13195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
maobaolong updated HDFS-13195:
--
Description:
Now the branch-2.7 support dfs.datanode.data.dir reconfig, but after i reconfig
this key, the conf page's value is still the old config value.
The reason is that:
{code:java}
public DatanodeHttpServer(final Configuration conf,
final DataNode datanode,
final ServerSocketChannel externalHttpChannel)
throws IOException {
this.conf = conf;
Configuration confForInfoServer = new Configuration(conf);
confForInfoServer.setInt(HttpServer2.HTTP_MAX_THREADS, 10);
HttpServer2.Builder builder = new HttpServer2.Builder()
.setName("datanode")
.setConf(confForInfoServer)
.setACL(new AccessControlList(conf.get(DFS_ADMIN, " ")))
.hostName(getHostnameForSpnegoPrincipal(confForInfoServer))
.addEndpoint(URI.create("http://localhost:0";))
.setFindPort(true);
this.infoServer = builder.build();
{code}
The confForInfoServer is a new configuration instance, while the dfsadmin
reconfig the datanode's config, the config result cannot reflect to
confForInfoServer, so we should use the datanode's conf.
was:
Now the branch-2.7 support dfs.datanode.data.dir reconfig, but after i reconfig
this key, the conf page's value is still the old config value.
The reason is that:
{code:java}
public DatanodeHttpServer(final Configuration conf,
final DataNode datanode,
final ServerSocketChannel externalHttpChannel)
throws IOException {
this.conf = conf;
Configuration confForInfoServer = new Configuration(conf);
confForInfoServer.setInt(HttpServer2.HTTP_MAX_THREADS, 10);
HttpServer2.Builder builder = new HttpServer2.Builder()
.setName("datanode")
.setConf(confForInfoServer)
.setACL(new AccessControlList(conf.get(DFS_ADMIN, " ")))
.hostName(getHostnameForSpnegoPrincipal(confForInfoServer))
.addEndpoint(URI.create("http://localhost:0";))
.setFindPort(true);
this.infoServer = builder.build();
{code}
> DataNode conf page cannot display the current value after reconfig
> ---
>
> Key: HDFS-13195
> URL: https://issues.apache.org/jira/browse/HDFS-13195
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Affects Versions: 2.7.1
>Reporter: maobaolong
>Priority: Minor
>
> Now the branch-2.7 support dfs.datanode.data.dir reconfig, but after i
> reconfig this key, the conf page's value is still the old config value.
> The reason is that:
> {code:java}
> public DatanodeHttpServer(final Configuration conf,
> final DataNode datanode,
> final ServerSocketChannel externalHttpChannel)
> throws IOException {
> this.conf = conf;
> Configuration confForInfoServer = new Configuration(conf);
> confForInfoServer.setInt(HttpServer2.HTTP_MAX_THREADS, 10);
> HttpServer2.Builder builder = new HttpServer2.Builder()
> .setName("datanode")
> .setConf(confForInfoServer)
> .setACL(new AccessControlList(conf.get(DFS_ADMIN, " ")))
> .hostName(getHostnameForSpnegoPrincipal(confForInfoServer))
> .addEndpoint(URI.create("http://localhost:0";))
> .setFindPort(true);
> this.infoServer = builder.build();
> {code}
> The confForInfoServer is a new configuration instance, while the dfsadmin
> reconfig the datanode's config, the config result cannot reflect to
> confForInfoServer, so we should use the datanode's conf.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13195) DataNode conf page cannot display the current value after reconfig
[
https://issues.apache.org/jira/browse/HDFS-13195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
maobaolong updated HDFS-13195:
--
Attachment: HDFS-13195-branch2.7.patch
> DataNode conf page cannot display the current value after reconfig
> ---
>
> Key: HDFS-13195
> URL: https://issues.apache.org/jira/browse/HDFS-13195
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Affects Versions: 2.7.1
>Reporter: maobaolong
>Priority: Minor
> Attachments: HDFS-13195-branch2.7.patch
>
>
> Now the branch-2.7 support dfs.datanode.data.dir reconfig, but after i
> reconfig this key, the conf page's value is still the old config value.
> The reason is that:
> {code:java}
> public DatanodeHttpServer(final Configuration conf,
> final DataNode datanode,
> final ServerSocketChannel externalHttpChannel)
> throws IOException {
> this.conf = conf;
> Configuration confForInfoServer = new Configuration(conf);
> confForInfoServer.setInt(HttpServer2.HTTP_MAX_THREADS, 10);
> HttpServer2.Builder builder = new HttpServer2.Builder()
> .setName("datanode")
> .setConf(confForInfoServer)
> .setACL(new AccessControlList(conf.get(DFS_ADMIN, " ")))
> .hostName(getHostnameForSpnegoPrincipal(confForInfoServer))
> .addEndpoint(URI.create("http://localhost:0";))
> .setFindPort(true);
> this.infoServer = builder.build();
> {code}
> The confForInfoServer is a new configuration instance, while the dfsadmin
> reconfig the datanode's config, the config result cannot reflect to
> confForInfoServer, so we should use the datanode's conf.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13195) DataNode conf page cannot display the current value after reconfig
[
https://issues.apache.org/jira/browse/HDFS-13195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
maobaolong updated HDFS-13195:
--
Fix Version/s: 2.7.1
Target Version/s: 2.7.1
Status: Patch Available (was: Open)
> DataNode conf page cannot display the current value after reconfig
> ---
>
> Key: HDFS-13195
> URL: https://issues.apache.org/jira/browse/HDFS-13195
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Affects Versions: 2.7.1
>Reporter: maobaolong
>Priority: Minor
> Fix For: 2.7.1
>
> Attachments: HDFS-13195-branch2.7.patch
>
>
> Now the branch-2.7 support dfs.datanode.data.dir reconfig, but after i
> reconfig this key, the conf page's value is still the old config value.
> The reason is that:
> {code:java}
> public DatanodeHttpServer(final Configuration conf,
> final DataNode datanode,
> final ServerSocketChannel externalHttpChannel)
> throws IOException {
> this.conf = conf;
> Configuration confForInfoServer = new Configuration(conf);
> confForInfoServer.setInt(HttpServer2.HTTP_MAX_THREADS, 10);
> HttpServer2.Builder builder = new HttpServer2.Builder()
> .setName("datanode")
> .setConf(confForInfoServer)
> .setACL(new AccessControlList(conf.get(DFS_ADMIN, " ")))
> .hostName(getHostnameForSpnegoPrincipal(confForInfoServer))
> .addEndpoint(URI.create("http://localhost:0";))
> .setFindPort(true);
> this.infoServer = builder.build();
> {code}
> The confForInfoServer is a new configuration instance, while the dfsadmin
> reconfig the datanode's config, the config result cannot reflect to
> confForInfoServer, so we should use the datanode's conf.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13195) DataNode conf page cannot display the current value after reconfig
[
https://issues.apache.org/jira/browse/HDFS-13195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16376582#comment-16376582
]
maobaolong commented on HDFS-13195:
---
[~vinayrpet] Please take a look at this issue. Without this patch, we cannot
let others know what the special key changed to, except the dfsadmin.
> DataNode conf page cannot display the current value after reconfig
> ---
>
> Key: HDFS-13195
> URL: https://issues.apache.org/jira/browse/HDFS-13195
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Affects Versions: 2.7.1
>Reporter: maobaolong
>Priority: Minor
> Fix For: 2.7.1
>
> Attachments: HDFS-13195-branch2.7.patch
>
>
> Now the branch-2.7 support dfs.datanode.data.dir reconfig, but after i
> reconfig this key, the conf page's value is still the old config value.
> The reason is that:
> {code:java}
> public DatanodeHttpServer(final Configuration conf,
> final DataNode datanode,
> final ServerSocketChannel externalHttpChannel)
> throws IOException {
> this.conf = conf;
> Configuration confForInfoServer = new Configuration(conf);
> confForInfoServer.setInt(HttpServer2.HTTP_MAX_THREADS, 10);
> HttpServer2.Builder builder = new HttpServer2.Builder()
> .setName("datanode")
> .setConf(confForInfoServer)
> .setACL(new AccessControlList(conf.get(DFS_ADMIN, " ")))
> .hostName(getHostnameForSpnegoPrincipal(confForInfoServer))
> .addEndpoint(URI.create("http://localhost:0";))
> .setFindPort(true);
> this.infoServer = builder.build();
> {code}
> The confForInfoServer is a new configuration instance, while the dfsadmin
> reconfig the datanode's config, the config result cannot reflect to
> confForInfoServer, so we should use the datanode's conf.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13195) DataNode conf page cannot display the current value after reconfig
[
https://issues.apache.org/jira/browse/HDFS-13195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
maobaolong updated HDFS-13195:
--
Attachment: HDFS-13195.001.patch
> DataNode conf page cannot display the current value after reconfig
> ---
>
> Key: HDFS-13195
> URL: https://issues.apache.org/jira/browse/HDFS-13195
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Affects Versions: 2.7.1
>Reporter: maobaolong
>Priority: Minor
> Fix For: 2.7.1
>
> Attachments: HDFS-13195-branch2.7.patch, HDFS-13195.001.patch
>
>
> Now the branch-2.7 support dfs.datanode.data.dir reconfig, but after i
> reconfig this key, the conf page's value is still the old config value.
> The reason is that:
> {code:java}
> public DatanodeHttpServer(final Configuration conf,
> final DataNode datanode,
> final ServerSocketChannel externalHttpChannel)
> throws IOException {
> this.conf = conf;
> Configuration confForInfoServer = new Configuration(conf);
> confForInfoServer.setInt(HttpServer2.HTTP_MAX_THREADS, 10);
> HttpServer2.Builder builder = new HttpServer2.Builder()
> .setName("datanode")
> .setConf(confForInfoServer)
> .setACL(new AccessControlList(conf.get(DFS_ADMIN, " ")))
> .hostName(getHostnameForSpnegoPrincipal(confForInfoServer))
> .addEndpoint(URI.create("http://localhost:0";))
> .setFindPort(true);
> this.infoServer = builder.build();
> {code}
> The confForInfoServer is a new configuration instance, while the dfsadmin
> reconfig the datanode's config, the config result cannot reflect to
> confForInfoServer, so we should use the datanode's conf.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13195) DataNode conf page cannot display the current value after reconfig
[
https://issues.apache.org/jira/browse/HDFS-13195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
maobaolong updated HDFS-13195:
--
Attachment: HDFS-13195.001-branch2.7.patch
> DataNode conf page cannot display the current value after reconfig
> ---
>
> Key: HDFS-13195
> URL: https://issues.apache.org/jira/browse/HDFS-13195
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Affects Versions: 2.7.1
>Reporter: maobaolong
>Priority: Minor
> Fix For: 2.7.1
>
> Attachments: HDFS-13195.001-branch2.7.patch, HDFS-13195.001.patch
>
>
> Now the branch-2.7 support dfs.datanode.data.dir reconfig, but after i
> reconfig this key, the conf page's value is still the old config value.
> The reason is that:
> {code:java}
> public DatanodeHttpServer(final Configuration conf,
> final DataNode datanode,
> final ServerSocketChannel externalHttpChannel)
> throws IOException {
> this.conf = conf;
> Configuration confForInfoServer = new Configuration(conf);
> confForInfoServer.setInt(HttpServer2.HTTP_MAX_THREADS, 10);
> HttpServer2.Builder builder = new HttpServer2.Builder()
> .setName("datanode")
> .setConf(confForInfoServer)
> .setACL(new AccessControlList(conf.get(DFS_ADMIN, " ")))
> .hostName(getHostnameForSpnegoPrincipal(confForInfoServer))
> .addEndpoint(URI.create("http://localhost:0";))
> .setFindPort(true);
> this.infoServer = builder.build();
> {code}
> The confForInfoServer is a new configuration instance, while the dfsadmin
> reconfig the datanode's config, the config result cannot reflect to
> confForInfoServer, so we should use the datanode's conf.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13195) DataNode conf page cannot display the current value after reconfig
[
https://issues.apache.org/jira/browse/HDFS-13195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
maobaolong updated HDFS-13195:
--
Attachment: (was: HDFS-13195-branch2.7.patch)
> DataNode conf page cannot display the current value after reconfig
> ---
>
> Key: HDFS-13195
> URL: https://issues.apache.org/jira/browse/HDFS-13195
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Affects Versions: 2.7.1
>Reporter: maobaolong
>Priority: Minor
> Fix For: 2.7.1
>
> Attachments: HDFS-13195.001-branch2.7.patch, HDFS-13195.001.patch
>
>
> Now the branch-2.7 support dfs.datanode.data.dir reconfig, but after i
> reconfig this key, the conf page's value is still the old config value.
> The reason is that:
> {code:java}
> public DatanodeHttpServer(final Configuration conf,
> final DataNode datanode,
> final ServerSocketChannel externalHttpChannel)
> throws IOException {
> this.conf = conf;
> Configuration confForInfoServer = new Configuration(conf);
> confForInfoServer.setInt(HttpServer2.HTTP_MAX_THREADS, 10);
> HttpServer2.Builder builder = new HttpServer2.Builder()
> .setName("datanode")
> .setConf(confForInfoServer)
> .setACL(new AccessControlList(conf.get(DFS_ADMIN, " ")))
> .hostName(getHostnameForSpnegoPrincipal(confForInfoServer))
> .addEndpoint(URI.create("http://localhost:0";))
> .setFindPort(true);
> this.infoServer = builder.build();
> {code}
> The confForInfoServer is a new configuration instance, while the dfsadmin
> reconfig the datanode's config, the config result cannot reflect to
> confForInfoServer, so we should use the datanode's conf.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13195) DataNode conf page cannot display the current value after reconfig
[
https://issues.apache.org/jira/browse/HDFS-13195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
maobaolong updated HDFS-13195:
--
Attachment: (was: HDFS-13195.001-branch2.7.patch)
> DataNode conf page cannot display the current value after reconfig
> ---
>
> Key: HDFS-13195
> URL: https://issues.apache.org/jira/browse/HDFS-13195
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Affects Versions: 2.7.1
>Reporter: maobaolong
>Priority: Minor
> Fix For: 2.7.1
>
> Attachments: HDFS-13195-branch-2.7.001.patch, HDFS-13195.001.patch
>
>
> Now the branch-2.7 support dfs.datanode.data.dir reconfig, but after i
> reconfig this key, the conf page's value is still the old config value.
> The reason is that:
> {code:java}
> public DatanodeHttpServer(final Configuration conf,
> final DataNode datanode,
> final ServerSocketChannel externalHttpChannel)
> throws IOException {
> this.conf = conf;
> Configuration confForInfoServer = new Configuration(conf);
> confForInfoServer.setInt(HttpServer2.HTTP_MAX_THREADS, 10);
> HttpServer2.Builder builder = new HttpServer2.Builder()
> .setName("datanode")
> .setConf(confForInfoServer)
> .setACL(new AccessControlList(conf.get(DFS_ADMIN, " ")))
> .hostName(getHostnameForSpnegoPrincipal(confForInfoServer))
> .addEndpoint(URI.create("http://localhost:0";))
> .setFindPort(true);
> this.infoServer = builder.build();
> {code}
> The confForInfoServer is a new configuration instance, while the dfsadmin
> reconfig the datanode's config, the config result cannot reflect to
> confForInfoServer, so we should use the datanode's conf.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13195) DataNode conf page cannot display the current value after reconfig
[
https://issues.apache.org/jira/browse/HDFS-13195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
maobaolong updated HDFS-13195:
--
Attachment: HDFS-13195-branch-2.7.001.patch
> DataNode conf page cannot display the current value after reconfig
> ---
>
> Key: HDFS-13195
> URL: https://issues.apache.org/jira/browse/HDFS-13195
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Affects Versions: 2.7.1
>Reporter: maobaolong
>Priority: Minor
> Fix For: 2.7.1
>
> Attachments: HDFS-13195-branch-2.7.001.patch, HDFS-13195.001.patch
>
>
> Now the branch-2.7 support dfs.datanode.data.dir reconfig, but after i
> reconfig this key, the conf page's value is still the old config value.
> The reason is that:
> {code:java}
> public DatanodeHttpServer(final Configuration conf,
> final DataNode datanode,
> final ServerSocketChannel externalHttpChannel)
> throws IOException {
> this.conf = conf;
> Configuration confForInfoServer = new Configuration(conf);
> confForInfoServer.setInt(HttpServer2.HTTP_MAX_THREADS, 10);
> HttpServer2.Builder builder = new HttpServer2.Builder()
> .setName("datanode")
> .setConf(confForInfoServer)
> .setACL(new AccessControlList(conf.get(DFS_ADMIN, " ")))
> .hostName(getHostnameForSpnegoPrincipal(confForInfoServer))
> .addEndpoint(URI.create("http://localhost:0";))
> .setFindPort(true);
> this.infoServer = builder.build();
> {code}
> The confForInfoServer is a new configuration instance, while the dfsadmin
> reconfig the datanode's config, the config result cannot reflect to
> confForInfoServer, so we should use the datanode's conf.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-13199) Fix the hdfs router page missing label icon issue
maobaolong created HDFS-13199: - Summary: Fix the hdfs router page missing label icon issue Key: HDFS-13199 URL: https://issues.apache.org/jira/browse/HDFS-13199 Project: Hadoop HDFS Issue Type: Bug Components: federation, hdfs Affects Versions: 3.0.0, 3.2.0 Reporter: maobaolong This bug is a typo error. decommisioned should be decommissioned -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13199) Fix the hdfs router page missing label icon issue
[ https://issues.apache.org/jira/browse/HDFS-13199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] maobaolong updated HDFS-13199: -- Status: Patch Available (was: Open) > Fix the hdfs router page missing label icon issue > - > > Key: HDFS-13199 > URL: https://issues.apache.org/jira/browse/HDFS-13199 > Project: Hadoop HDFS > Issue Type: Bug > Components: federation, hdfs >Affects Versions: 3.0.0, 3.2.0 >Reporter: maobaolong >Priority: Major > Attachments: HDFS-13199.001.patch > > > This bug is a typo error. > decommisioned should be decommissioned -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13199) Fix the hdfs router page missing label icon issue
[ https://issues.apache.org/jira/browse/HDFS-13199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] maobaolong updated HDFS-13199: -- Attachment: HDFS-13199.001.patch > Fix the hdfs router page missing label icon issue > - > > Key: HDFS-13199 > URL: https://issues.apache.org/jira/browse/HDFS-13199 > Project: Hadoop HDFS > Issue Type: Bug > Components: federation, hdfs >Affects Versions: 3.0.0, 3.2.0 >Reporter: maobaolong >Priority: Major > Attachments: HDFS-13199.001.patch > > > This bug is a typo error. > decommisioned should be decommissioned -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13199) Fix the hdfs router page missing label icon issue
[ https://issues.apache.org/jira/browse/HDFS-13199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] maobaolong updated HDFS-13199: -- Issue Type: Sub-task (was: Bug) Parent: HDFS-12615 > Fix the hdfs router page missing label icon issue > - > > Key: HDFS-13199 > URL: https://issues.apache.org/jira/browse/HDFS-13199 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: federation, hdfs >Affects Versions: 3.0.0, 3.2.0 >Reporter: maobaolong >Priority: Major > Attachments: HDFS-13199.001.patch > > > This bug is a typo error. > decommisioned should be decommissioned -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13199) Fix the hdfs router page missing label icon issue
[ https://issues.apache.org/jira/browse/HDFS-13199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16378982#comment-16378982 ] maobaolong commented on HDFS-13199: --- How can assign this issue to myself? > Fix the hdfs router page missing label icon issue > - > > Key: HDFS-13199 > URL: https://issues.apache.org/jira/browse/HDFS-13199 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: federation, hdfs >Affects Versions: 3.0.0, 3.2.0 >Reporter: maobaolong >Priority: Major > Attachments: HDFS-13199.001.patch > > > This bug is a typo error. > decommisioned should be decommissioned -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13199) RBF: Fix the hdfs router page missing label icon issue
[ https://issues.apache.org/jira/browse/HDFS-13199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16379622#comment-16379622 ] maobaolong commented on HDFS-13199: --- [~goiri] Thank you for your apply. I have great honor to become a contributor of hadoop hdfs. This issue is very similar to HDFS-9357. The difference is that HDFS-9357 is about NN, this issue is about router. Yeah, the lower case of my name is better. > RBF: Fix the hdfs router page missing label icon issue > -- > > Key: HDFS-13199 > URL: https://issues.apache.org/jira/browse/HDFS-13199 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: federation, hdfs >Affects Versions: 3.0.0, 3.2.0 >Reporter: maobaolong >Assignee: maobaolong >Priority: Major > Attachments: HDFS-13199.001.patch > > > This bug is a typo error. > decommisioned should be decommissioned -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13195) DataNode conf page cannot display the current value after reconfig
[
https://issues.apache.org/jira/browse/HDFS-13195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16379820#comment-16379820
]
maobaolong commented on HDFS-13195:
---
[~bharatviswa] I don't think this failed tests is related to me.
> DataNode conf page cannot display the current value after reconfig
> ---
>
> Key: HDFS-13195
> URL: https://issues.apache.org/jira/browse/HDFS-13195
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Affects Versions: 2.7.1
>Reporter: maobaolong
>Priority: Minor
> Fix For: 2.7.1
>
> Attachments: HDFS-13195-branch-2.7.001.patch, HDFS-13195.001.patch
>
>
> Now the branch-2.7 support dfs.datanode.data.dir reconfig, but after i
> reconfig this key, the conf page's value is still the old config value.
> The reason is that:
> {code:java}
> public DatanodeHttpServer(final Configuration conf,
> final DataNode datanode,
> final ServerSocketChannel externalHttpChannel)
> throws IOException {
> this.conf = conf;
> Configuration confForInfoServer = new Configuration(conf);
> confForInfoServer.setInt(HttpServer2.HTTP_MAX_THREADS, 10);
> HttpServer2.Builder builder = new HttpServer2.Builder()
> .setName("datanode")
> .setConf(confForInfoServer)
> .setACL(new AccessControlList(conf.get(DFS_ADMIN, " ")))
> .hostName(getHostnameForSpnegoPrincipal(confForInfoServer))
> .addEndpoint(URI.create("http://localhost:0";))
> .setFindPort(true);
> this.infoServer = builder.build();
> {code}
> The confForInfoServer is a new configuration instance, while the dfsadmin
> reconfig the datanode's config, the config result cannot reflect to
> confForInfoServer, so we should use the datanode's conf.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13195) DataNode conf page cannot display the current value after reconfig
[
https://issues.apache.org/jira/browse/HDFS-13195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16381378#comment-16381378
]
maobaolong commented on HDFS-13195:
---
[~bharatviswa] Thank you for clarify for me very much. And what should i do
next?
> DataNode conf page cannot display the current value after reconfig
> ---
>
> Key: HDFS-13195
> URL: https://issues.apache.org/jira/browse/HDFS-13195
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Affects Versions: 2.7.1
>Reporter: maobaolong
>Assignee: maobaolong
>Priority: Minor
> Fix For: 2.7.1
>
> Attachments: HDFS-13195-branch-2.7.001.patch, HDFS-13195.001.patch
>
>
> Now the branch-2.7 support dfs.datanode.data.dir reconfig, but after i
> reconfig this key, the conf page's value is still the old config value.
> The reason is that:
> {code:java}
> public DatanodeHttpServer(final Configuration conf,
> final DataNode datanode,
> final ServerSocketChannel externalHttpChannel)
> throws IOException {
> this.conf = conf;
> Configuration confForInfoServer = new Configuration(conf);
> confForInfoServer.setInt(HttpServer2.HTTP_MAX_THREADS, 10);
> HttpServer2.Builder builder = new HttpServer2.Builder()
> .setName("datanode")
> .setConf(confForInfoServer)
> .setACL(new AccessControlList(conf.get(DFS_ADMIN, " ")))
> .hostName(getHostnameForSpnegoPrincipal(confForInfoServer))
> .addEndpoint(URI.create("http://localhost:0";))
> .setFindPort(true);
> this.infoServer = builder.build();
> {code}
> The confForInfoServer is a new configuration instance, while the dfsadmin
> reconfig the datanode's config, the config result cannot reflect to
> confForInfoServer, so we should use the datanode's conf.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13204) RBF: Optimize name service safe mode icon
[ https://issues.apache.org/jira/browse/HDFS-13204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16381385#comment-16381385 ] maobaolong commented on HDFS-13204: --- I think the icon can be replace to others, but it is another thing we should think about. The core problem i think is that we should not use the node state style class in the Subclusters page, we should use the new state style class and we can replace these icons to new icons to make much sense every time. > RBF: Optimize name service safe mode icon > - > > Key: HDFS-13204 > URL: https://issues.apache.org/jira/browse/HDFS-13204 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: liuhongtong >Priority: Minor > Attachments: HDFS-13204.001.patch, image-2018-02-28-18-33-09-972.png, > image-2018-02-28-18-33-47-661.png, image-2018-02-28-18-35-35-708.png > > > In federation health webpage, the safe mode icons of Subclusters and Routers > are inconsistent. > The safe mode icon of Subclusters may induce users the name service is > maintaining. > !image-2018-02-28-18-33-09-972.png! > The safe mode icon of Routers: > !image-2018-02-28-18-33-47-661.png! > In fact, if the name service is in safe mode, users can't do writing related > operations. So I think the safe mode icon in Subclusters should be modified, > which may be more reasonable. > !image-2018-02-28-18-35-35-708.png! -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13195) DataNode conf page cannot display the current value after reconfig
[
https://issues.apache.org/jira/browse/HDFS-13195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16381586#comment-16381586
]
maobaolong commented on HDFS-13195:
---
[~bharatviswa] Whatever, thank for your kindly review.
[~elgoiri] Would you like to take a look at this issue? i think this fix is
useful.
> DataNode conf page cannot display the current value after reconfig
> ---
>
> Key: HDFS-13195
> URL: https://issues.apache.org/jira/browse/HDFS-13195
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Affects Versions: 2.7.1
>Reporter: maobaolong
>Assignee: maobaolong
>Priority: Minor
> Fix For: 2.7.1
>
> Attachments: HDFS-13195-branch-2.7.001.patch, HDFS-13195.001.patch
>
>
> Now the branch-2.7 support dfs.datanode.data.dir reconfig, but after i
> reconfig this key, the conf page's value is still the old config value.
> The reason is that:
> {code:java}
> public DatanodeHttpServer(final Configuration conf,
> final DataNode datanode,
> final ServerSocketChannel externalHttpChannel)
> throws IOException {
> this.conf = conf;
> Configuration confForInfoServer = new Configuration(conf);
> confForInfoServer.setInt(HttpServer2.HTTP_MAX_THREADS, 10);
> HttpServer2.Builder builder = new HttpServer2.Builder()
> .setName("datanode")
> .setConf(confForInfoServer)
> .setACL(new AccessControlList(conf.get(DFS_ADMIN, " ")))
> .hostName(getHostnameForSpnegoPrincipal(confForInfoServer))
> .addEndpoint(URI.create("http://localhost:0";))
> .setFindPort(true);
> this.infoServer = builder.build();
> {code}
> The confForInfoServer is a new configuration instance, while the dfsadmin
> reconfig the datanode's config, the config result cannot reflect to
> confForInfoServer, so we should use the datanode's conf.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13212) RBF: Fix router location cache issue
[ https://issues.apache.org/jira/browse/HDFS-13212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16381621#comment-16381621 ] maobaolong commented on HDFS-13212: --- [~wuweiwei] Shouldn't the MountTableResolver clean the locationCache periodically? > RBF: Fix router location cache issue > > > Key: HDFS-13212 > URL: https://issues.apache.org/jira/browse/HDFS-13212 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: federation, hdfs >Reporter: weiwei wu >Priority: Major > Attachments: HDFS-13212-001.patch > > > The MountTableResolver refreshEntries function have a bug when add a new > mount table entry which already have location cache. The old location cache > will never be invalid until this mount point change again. > Need to invalid the location cache when add the mount table entries. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13204) RBF: Optimize name service safe mode icon
[ https://issues.apache.org/jira/browse/HDFS-13204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16383039#comment-16383039 ] maobaolong commented on HDFS-13204: --- [~elgoiri] Do you think the following make sense? - Routers - Active: federationdfshealth-router-alive - Safe mode:federationdfshealth-router-safemode - Unavailable: federationdfshealth-router-unavailable > RBF: Optimize name service safe mode icon > - > > Key: HDFS-13204 > URL: https://issues.apache.org/jira/browse/HDFS-13204 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: liuhongtong >Priority: Minor > Attachments: HDFS-13204.001.patch, image-2018-02-28-18-33-09-972.png, > image-2018-02-28-18-33-47-661.png, image-2018-02-28-18-35-35-708.png > > > In federation health webpage, the safe mode icons of Subclusters and Routers > are inconsistent. > The safe mode icon of Subclusters may induce users the name service is > maintaining. > !image-2018-02-28-18-33-09-972.png! > The safe mode icon of Routers: > !image-2018-02-28-18-33-47-661.png! > In fact, if the name service is in safe mode, users can't do writing related > operations. So I think the safe mode icon in Subclusters should be modified, > which may be more reasonable. > !image-2018-02-28-18-35-35-708.png! -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13215) RBF: Move Router to its own module
[ https://issues.apache.org/jira/browse/HDFS-13215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16383206#comment-16383206 ] maobaolong commented on HDFS-13215: --- +1, good idea. 👍 > RBF: Move Router to its own module > -- > > Key: HDFS-13215 > URL: https://issues.apache.org/jira/browse/HDFS-13215 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Íñigo Goiri >Priority: Major > > We are splitting the HDFS client code base and potentially Router-based > Federation is also independent enough to be in its own package. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10880) Federation Mount Table State Store internal API
[
https://issues.apache.org/jira/browse/HDFS-10880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16385482#comment-16385482
]
maobaolong commented on HDFS-10880:
---
[~elgoiri] Thank you to contribute this feature. But i don't see any use about
the DestinationOrder.
{code:java}
public enum DestinationOrder {
HASH, // Follow consistent hashing
LOCAL, // Local first
RANDOM // Random order
}
{code}
I've seen the use of multiTarget in RouterRpcClient like invokeConcurrent and
invokeSequential method, I know the use of multiTarget there seems to ensure FT.
Please show more use case about the multiTarget and destinationOrder, I want to
use and test those cool features. Thank you.
> Federation Mount Table State Store internal API
> ---
>
> Key: HDFS-10880
> URL: https://issues.apache.org/jira/browse/HDFS-10880
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: fs
>Reporter: Jason Kace
>Assignee: Íñigo Goiri
>Priority: Major
> Fix For: 2.9.0, 3.0.0
>
> Attachments: HDFS-10880-HDFS-10467-000.patch,
> HDFS-10880-HDFS-10467-001.patch, HDFS-10880-HDFS-10467-002.patch,
> HDFS-10880-HDFS-10467-003.patch, HDFS-10880-HDFS-10467-004.patch,
> HDFS-10880-HDFS-10467-005.patch, HDFS-10880-HDFS-10467-006.patch,
> HDFS-10880-HDFS-10467-007.patch, HDFS-10880-HDFS-10467-008.patch
>
>
> The Federation Mount Table State encapsulates the mapping of file paths in
> the global namespace to a specific NN(nameservice) and local NN path. The
> mount table is shared by all router instances and represents a unified view
> of the global namespace. The state store API for the mount table allows the
> related records to be queried, updated and deleted.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13204) RBF: Optimize name service safe mode icon
[ https://issues.apache.org/jira/browse/HDFS-13204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16385627#comment-16385627 ] maobaolong commented on HDFS-13204: --- [~liuhongtong] [~elgoiri] Yeah i mean to add federationdfshealth-router-* to hadoop.css, and discuss the icon in the next jira issue. > RBF: Optimize name service safe mode icon > - > > Key: HDFS-13204 > URL: https://issues.apache.org/jira/browse/HDFS-13204 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: liuhongtong >Priority: Minor > Attachments: HDFS-13204.001.patch, image-2018-02-28-18-33-09-972.png, > image-2018-02-28-18-33-47-661.png, image-2018-02-28-18-35-35-708.png > > > In federation health webpage, the safe mode icons of Subclusters and Routers > are inconsistent. > The safe mode icon of Subclusters may induce users the name service is > maintaining. > !image-2018-02-28-18-33-09-972.png! > The safe mode icon of Routers: > !image-2018-02-28-18-33-47-661.png! > In fact, if the name service is in safe mode, users can't do writing related > operations. So I think the safe mode icon in Subclusters should be modified, > which may be more reasonable. > !image-2018-02-28-18-35-35-708.png! -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13215) RBF: Move Router to its own module
[ https://issues.apache.org/jira/browse/HDFS-13215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16385870#comment-16385870 ] maobaolong commented on HDFS-13215: --- Any update here? expecting. > RBF: Move Router to its own module > -- > > Key: HDFS-13215 > URL: https://issues.apache.org/jira/browse/HDFS-13215 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Íñigo Goiri >Priority: Major > > We are splitting the HDFS client code base and potentially Router-based > Federation is also independent enough to be in its own package. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-13226) RBF: We should throw the failure validate and refuse this mount entry
maobaolong created HDFS-13226:
-
Summary: RBF: We should throw the failure validate and refuse this
mount entry
Key: HDFS-13226
URL: https://issues.apache.org/jira/browse/HDFS-13226
Project: Hadoop HDFS
Issue Type: Sub-task
Components: hdfs
Affects Versions: 3.2.0
Reporter: maobaolong
one of the mount entry source path rule is that the source path must start with
'\', somebody didn't follow the rule and execute the following command:
{code:bash}
$ hdfs dfsrouteradmin -add addnode/ ns1 /addnode/
{code}
But, the console show we are successful add this entry.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-13226) RBF: We should throw the failure validate and refuse this mount entry
[
https://issues.apache.org/jira/browse/HDFS-13226?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
maobaolong reassigned HDFS-13226:
-
Assignee: maobaolong
> RBF: We should throw the failure validate and refuse this mount entry
> -
>
> Key: HDFS-13226
> URL: https://issues.apache.org/jira/browse/HDFS-13226
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs
>Affects Versions: 3.2.0
>Reporter: maobaolong
>Assignee: maobaolong
>Priority: Major
>
> one of the mount entry source path rule is that the source path must start
> with '\', somebody didn't follow the rule and execute the following command:
> {code:bash}
> $ hdfs dfsrouteradmin -add addnode/ ns1 /addnode/
> {code}
> But, the console show we are successful add this entry.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13226) RBF: We should throw the failure validate and refuse this mount entry
[
https://issues.apache.org/jira/browse/HDFS-13226?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
maobaolong updated HDFS-13226:
--
Labels: RBF (was: )
Fix Version/s: 3.2.0
Target Version/s: 3.2.0
Status: Patch Available (was: Open)
> RBF: We should throw the failure validate and refuse this mount entry
> -
>
> Key: HDFS-13226
> URL: https://issues.apache.org/jira/browse/HDFS-13226
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs
>Affects Versions: 3.2.0
>Reporter: maobaolong
>Assignee: maobaolong
>Priority: Major
> Labels: RBF
> Fix For: 3.2.0
>
>
> one of the mount entry source path rule is that the source path must start
> with '\', somebody didn't follow the rule and execute the following command:
> {code:bash}
> $ hdfs dfsrouteradmin -add addnode/ ns1 /addnode/
> {code}
> But, the console show we are successful add this entry.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13226) RBF: We should throw the failure validate and refuse this mount entry
[
https://issues.apache.org/jira/browse/HDFS-13226?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
maobaolong updated HDFS-13226:
--
Attachment: HDFS-13226.001.patch
> RBF: We should throw the failure validate and refuse this mount entry
> -
>
> Key: HDFS-13226
> URL: https://issues.apache.org/jira/browse/HDFS-13226
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs
>Affects Versions: 3.2.0
>Reporter: maobaolong
>Assignee: maobaolong
>Priority: Major
> Labels: RBF
> Fix For: 3.2.0
>
> Attachments: HDFS-13226.001.patch
>
>
> one of the mount entry source path rule is that the source path must start
> with '\', somebody didn't follow the rule and execute the following command:
> {code:bash}
> $ hdfs dfsrouteradmin -add addnode/ ns1 /addnode/
> {code}
> But, the console show we are successful add this entry.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13226) RBF: We should throw the failure validate and refuse this mount entry
[
https://issues.apache.org/jira/browse/HDFS-13226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16385915#comment-16385915
]
maobaolong commented on HDFS-13226:
---
[~elgoiri] Do you think this is a bug? Please take a look
> RBF: We should throw the failure validate and refuse this mount entry
> -
>
> Key: HDFS-13226
> URL: https://issues.apache.org/jira/browse/HDFS-13226
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs
>Affects Versions: 3.2.0
>Reporter: maobaolong
>Assignee: maobaolong
>Priority: Major
> Labels: RBF
> Fix For: 3.2.0
>
> Attachments: HDFS-13226.001.patch
>
>
> one of the mount entry source path rule is that the source path must start
> with '\', somebody didn't follow the rule and execute the following command:
> {code:bash}
> $ hdfs dfsrouteradmin -add addnode/ ns1 /addnode/
> {code}
> But, the console show we are successful add this entry.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13226) RBF: We should throw the failure validate and refuse this mount entry
[
https://issues.apache.org/jira/browse/HDFS-13226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16387242#comment-16387242
]
maobaolong commented on HDFS-13226:
---
[~elgoiri] If we change the log error to throw IOException to user, is it make
sense?
> RBF: We should throw the failure validate and refuse this mount entry
> -
>
> Key: HDFS-13226
> URL: https://issues.apache.org/jira/browse/HDFS-13226
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs
>Affects Versions: 3.2.0
>Reporter: maobaolong
>Assignee: maobaolong
>Priority: Major
> Labels: RBF
> Fix For: 3.2.0
>
> Attachments: HDFS-13226.001.patch
>
>
> one of the mount entry source path rule is that the source path must start
> with '\', somebody didn't follow the rule and execute the following command:
> {code:bash}
> $ hdfs dfsrouteradmin -add addnode/ ns1 /addnode/
> {code}
> But, the console show we are successful add this entry.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13195) DataNode conf page cannot display the current value after reconfig
[
https://issues.apache.org/jira/browse/HDFS-13195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16387244#comment-16387244
]
maobaolong commented on HDFS-13195:
---
[~bharatviswa] [~wheat9] Please take a look at this patch
> DataNode conf page cannot display the current value after reconfig
> ---
>
> Key: HDFS-13195
> URL: https://issues.apache.org/jira/browse/HDFS-13195
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Affects Versions: 2.7.1
>Reporter: maobaolong
>Assignee: maobaolong
>Priority: Minor
> Fix For: 2.7.1
>
> Attachments: HDFS-13195-branch-2.7.001.patch, HDFS-13195.001.patch
>
>
> Now the branch-2.7 support dfs.datanode.data.dir reconfig, but after i
> reconfig this key, the conf page's value is still the old config value.
> The reason is that:
> {code:java}
> public DatanodeHttpServer(final Configuration conf,
> final DataNode datanode,
> final ServerSocketChannel externalHttpChannel)
> throws IOException {
> this.conf = conf;
> Configuration confForInfoServer = new Configuration(conf);
> confForInfoServer.setInt(HttpServer2.HTTP_MAX_THREADS, 10);
> HttpServer2.Builder builder = new HttpServer2.Builder()
> .setName("datanode")
> .setConf(confForInfoServer)
> .setACL(new AccessControlList(conf.get(DFS_ADMIN, " ")))
> .hostName(getHostnameForSpnegoPrincipal(confForInfoServer))
> .addEndpoint(URI.create("http://localhost:0";))
> .setFindPort(true);
> this.infoServer = builder.build();
> {code}
> The confForInfoServer is a new configuration instance, while the dfsadmin
> reconfig the datanode's config, the config result cannot reflect to
> confForInfoServer, so we should use the datanode's conf.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13204) RBF: Optimize name service safe mode icon
[ https://issues.apache.org/jira/browse/HDFS-13204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16387249#comment-16387249 ] maobaolong commented on HDFS-13204: --- [~elgoiri] Okay, may be we can have more choice about federationdfshealth-router-* icon, we can choose the icon from bootstrap. [~liuhongtong] Would you like to show your icon plan? > RBF: Optimize name service safe mode icon > - > > Key: HDFS-13204 > URL: https://issues.apache.org/jira/browse/HDFS-13204 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: liuhongtong >Priority: Minor > Attachments: HDFS-13204.001.patch, image-2018-02-28-18-33-09-972.png, > image-2018-02-28-18-33-47-661.png, image-2018-02-28-18-35-35-708.png > > > In federation health webpage, the safe mode icons of Subclusters and Routers > are inconsistent. > The safe mode icon of Subclusters may induce users the name service is > maintaining. > !image-2018-02-28-18-33-09-972.png! > The safe mode icon of Routers: > !image-2018-02-28-18-33-47-661.png! > In fact, if the name service is in safe mode, users can't do writing related > operations. So I think the safe mode icon in Subclusters should be modified, > which may be more reasonable. > !image-2018-02-28-18-35-35-708.png! -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12615) Router-based HDFS federation phase 2
[ https://issues.apache.org/jira/browse/HDFS-12615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16387250#comment-16387250 ] maobaolong commented on HDFS-12615: --- [~elgoiri] Should us impl a StateStore of mysql? > Router-based HDFS federation phase 2 > > > Key: HDFS-12615 > URL: https://issues.apache.org/jira/browse/HDFS-12615 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Íñigo Goiri >Assignee: Íñigo Goiri >Priority: Major > Labels: RBF > > This umbrella JIRA tracks set of improvements over the Router-based HDFS > federation (HDFS-10467). -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13233) RBF：getMountPoint doesn't return the correct mount point of the mount table
[ https://issues.apache.org/jira/browse/HDFS-13233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16387345#comment-16387345 ] maobaolong commented on HDFS-13233: --- [~striver.wang] make sense. > RBF：getMountPoint doesn't return the correct mount point of the mount table > --- > > Key: HDFS-13233 > URL: https://issues.apache.org/jira/browse/HDFS-13233 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs >Affects Versions: 3.2.0 >Reporter: wangzhiyuan >Priority: Major > Attachments: HDFS-13233.001.patch, HDFS-13233.002.patch > > > Method MountTableResolver->getMountPoint will traverse mount table and return > the first mount point which the input path starts with, but the condition is > not sufficient. > Suppose the mount point table contains: "/user" "/user/test" "/user/test1", > if the input path is "/user/test111", the return mount point is > "/user/test1", but the correct one should be "/user". -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13226) RBF: We should throw the failure validate and refuse this mount entry
[
https://issues.apache.org/jira/browse/HDFS-13226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16388924#comment-16388924
]
maobaolong commented on HDFS-13226:
---
{code:java}
public boolean validate() {
boolean ret = super.validate();
if (this.getSourcePath() == null || this.getSourcePath().length() == 0) {
LOG.error("Invalid entry, no source path specified ", this);
ret = false;
}
if (!this.getSourcePath().startsWith("/")) {
LOG.error("Invalid entry, all mount points must start with / ", this);
ret = false;
}
if (this.getDestinations() == null || this.getDestinations().size() == 0) {
LOG.error("Invalid entry, no destination paths specified ", this);
ret = false;
}
for (RemoteLocation loc : getDestinations()) {
String nsId = loc.getNameserviceId();
if (nsId == null || nsId.length() == 0) {
LOG.error("Invalid entry, invalid destination nameservice ", this);
ret = false;
}
if (loc.getDest() == null || loc.getDest().length() == 0) {
LOG.error("Invalid entry, invalid destination path ", this);
ret = false;
}
if (!loc.getDest().startsWith("/")) {
LOG.error("Invalid entry, all destination must start with / ", this);
ret = false;
}
}
return ret;
}
{code}
Let's discuss about this method. I think this method have the following problem:
- if this.getSourcePath() return null, the this.getSourcePath().startsWith("/")
will throw NPE.
- if the sourcePath is null, the validate method will not be invoked, instead,
the NPE will be taken place as follow stack.
{code:java}
java.lang.NullPointerException
at
org.apache.hadoop.hdfs.federation.protocol.proto.HdfsServerFederationProtos$GetMountTableEntriesRequestProto$Builder.setSrcPath(HdfsServerFederationProtos.java:15937)
at
org.apache.hadoop.hdfs.server.federation.store.protocol.impl.pb.GetMountTableEntriesRequestPBImpl.setSrcPath(GetMountTableEntriesRequestPBImpl.java:73)
at
org.apache.hadoop.hdfs.server.federation.store.protocol.GetMountTableEntriesRequest.newInstance(GetMountTableEntriesRequest.java:38)
at
org.apache.hadoop.hdfs.tools.federation.RouterAdmin.addMount(RouterAdmin.java:280)
at
org.apache.hadoop.hdfs.tools.federation.RouterAdmin.addMount(RouterAdmin.java:258)
at
org.apache.hadoop.hdfs.tools.federation.RouterAdmin.run(RouterAdmin.java:154)
and normalizeFileSystemPath method found the null or empty src or dest first.
{code}
- if the nsId is null,
{code:bash}
java.lang.NullPointerException
at
org.apache.hadoop.hdfs.tools.federation.RouterAdmin.addMount(RouterAdmin.java:224)
at
org.apache.hadoop.hdfs.tools.federation.RouterAdmin.run(RouterAdmin.java:154)
{code}
- if the source start with more than one '/', the entry will be create
successfully as a different mount entry as the single '/' version.
for example
{code:bash}
hdfs dfsrouteradmin -add //addnode/ ns1 //addnode/
hdfs dfsrouteradmin -add /addnode/ ns1 /addnode/
Mount Table Entries:
SourceDestinations Owner
Group Mode Quota/Usage
//addnode/ns1->//addnode/ hadp
hadp rwxr-xr-x [NsQuota: -/-, SsQuota: -/-]
/addnode ns1->/addnode hadp
hadp rwxr-xr-x [NsQuota: -/-, SsQuota: -/-]
{code}
- if an error have occurred, should we check the following problem?
So, I want to modify to the following version of validate:
{code:java}
public boolean validate() {
if (!super.validate()) {
return false;
}
if (!this.getSourcePath().startsWith("/")
|| this.getSourcePath().startsWith("//")) {
LOG.error("Invalid entry, all mount points must start with a single / ",
this);
return false;
}
for (RemoteLocation loc : getDestinations()) {
String nsId = loc.getNameserviceId();
if (nsId.length() == 0) {
LOG.error("Invalid entry, invalid destination nameservice ", this);
return false;
} else if (!loc.getDest().startsWith("/") ||
this.getSourcePath().startsWith("//")) {
LOG.error("Invalid entry, all destination must start with a single / ",
this);
return false;
}
}
return true;
}
{code}
Or
{code:java}
@Override
public boolean validate() {
if (!super.validate()) {
return false;
}
if (!this.getSourcePath().startsWith("/")
|| this.getSourcePath().startsWith("//")) {
throw new IllegalArgumentException("Invalid entry, all mount points must
start with a single / ");
}
for (RemoteLocation loc : getDestinations()) {
String nsId = loc.getNameserviceId();
if (nsId.length() == 0) {
thro
[jira] [Updated] (HDFS-13195) DataNode conf page cannot display the current value after reconfig
[
https://issues.apache.org/jira/browse/HDFS-13195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
maobaolong updated HDFS-13195:
--
Attachment: HDFS-13195.002.patch
> DataNode conf page cannot display the current value after reconfig
> ---
>
> Key: HDFS-13195
> URL: https://issues.apache.org/jira/browse/HDFS-13195
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Affects Versions: 2.7.1
>Reporter: maobaolong
>Assignee: maobaolong
>Priority: Minor
> Fix For: 2.7.1
>
> Attachments: HDFS-13195-branch-2.7.001.patch, HDFS-13195.001.patch,
> HDFS-13195.002.patch
>
>
> Now the branch-2.7 support dfs.datanode.data.dir reconfig, but after i
> reconfig this key, the conf page's value is still the old config value.
> The reason is that:
> {code:java}
> public DatanodeHttpServer(final Configuration conf,
> final DataNode datanode,
> final ServerSocketChannel externalHttpChannel)
> throws IOException {
> this.conf = conf;
> Configuration confForInfoServer = new Configuration(conf);
> confForInfoServer.setInt(HttpServer2.HTTP_MAX_THREADS, 10);
> HttpServer2.Builder builder = new HttpServer2.Builder()
> .setName("datanode")
> .setConf(confForInfoServer)
> .setACL(new AccessControlList(conf.get(DFS_ADMIN, " ")))
> .hostName(getHostnameForSpnegoPrincipal(confForInfoServer))
> .addEndpoint(URI.create("http://localhost:0";))
> .setFindPort(true);
> this.infoServer = builder.build();
> {code}
> The confForInfoServer is a new configuration instance, while the dfsadmin
> reconfig the datanode's config, the config result cannot reflect to
> confForInfoServer, so we should use the datanode's conf.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-13241) RBF: TestRouterSafemode failed if the port 8888 in use
maobaolong created HDFS-13241: - Summary: RBF: TestRouterSafemode failed if the port in use Key: HDFS-13241 URL: https://issues.apache.org/jira/browse/HDFS-13241 Project: Hadoop HDFS Issue Type: Sub-task Components: hdfs, test Affects Versions: 3.2.0 Reporter: maobaolong Assignee: maobaolong -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13241) RBF: TestRouterSafemode failed if the port 8888 in use
[ https://issues.apache.org/jira/browse/HDFS-13241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] maobaolong updated HDFS-13241: -- Attachment: HDFS-13241.001.patch > RBF: TestRouterSafemode failed if the port in use > -- > > Key: HDFS-13241 > URL: https://issues.apache.org/jira/browse/HDFS-13241 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs, test >Affects Versions: 3.2.0 >Reporter: maobaolong >Assignee: maobaolong >Priority: Major > Attachments: HDFS-13241.001.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13195) DataNode conf page cannot display the current value after reconfig
[
https://issues.apache.org/jira/browse/HDFS-13195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
maobaolong updated HDFS-13195:
--
Attachment: HDFS-13195-branch-2.7.002.patch
> DataNode conf page cannot display the current value after reconfig
> ---
>
> Key: HDFS-13195
> URL: https://issues.apache.org/jira/browse/HDFS-13195
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Affects Versions: 2.7.1
>Reporter: maobaolong
>Assignee: maobaolong
>Priority: Minor
> Fix For: 2.7.1
>
> Attachments: HDFS-13195-branch-2.7.001.patch,
> HDFS-13195-branch-2.7.002.patch, HDFS-13195.001.patch, HDFS-13195.002.patch
>
>
> Now the branch-2.7 support dfs.datanode.data.dir reconfig, but after i
> reconfig this key, the conf page's value is still the old config value.
> The reason is that:
> {code:java}
> public DatanodeHttpServer(final Configuration conf,
> final DataNode datanode,
> final ServerSocketChannel externalHttpChannel)
> throws IOException {
> this.conf = conf;
> Configuration confForInfoServer = new Configuration(conf);
> confForInfoServer.setInt(HttpServer2.HTTP_MAX_THREADS, 10);
> HttpServer2.Builder builder = new HttpServer2.Builder()
> .setName("datanode")
> .setConf(confForInfoServer)
> .setACL(new AccessControlList(conf.get(DFS_ADMIN, " ")))
> .hostName(getHostnameForSpnegoPrincipal(confForInfoServer))
> .addEndpoint(URI.create("http://localhost:0";))
> .setFindPort(true);
> this.infoServer = builder.build();
> {code}
> The confForInfoServer is a new configuration instance, while the dfsadmin
> reconfig the datanode's config, the config result cannot reflect to
> confForInfoServer, so we should use the datanode's conf.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13195) DataNode conf page cannot display the current value after reconfig
[
https://issues.apache.org/jira/browse/HDFS-13195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16389308#comment-16389308
]
maobaolong commented on HDFS-13195:
---
[~kihwal] make sense. I use another way to fix this issue. PTAL.
> DataNode conf page cannot display the current value after reconfig
> ---
>
> Key: HDFS-13195
> URL: https://issues.apache.org/jira/browse/HDFS-13195
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Affects Versions: 2.7.1
>Reporter: maobaolong
>Assignee: maobaolong
>Priority: Minor
> Fix For: 2.7.1
>
> Attachments: HDFS-13195-branch-2.7.001.patch,
> HDFS-13195-branch-2.7.002.patch, HDFS-13195.001.patch, HDFS-13195.002.patch
>
>
> Now the branch-2.7 support dfs.datanode.data.dir reconfig, but after i
> reconfig this key, the conf page's value is still the old config value.
> The reason is that:
> {code:java}
> public DatanodeHttpServer(final Configuration conf,
> final DataNode datanode,
> final ServerSocketChannel externalHttpChannel)
> throws IOException {
> this.conf = conf;
> Configuration confForInfoServer = new Configuration(conf);
> confForInfoServer.setInt(HttpServer2.HTTP_MAX_THREADS, 10);
> HttpServer2.Builder builder = new HttpServer2.Builder()
> .setName("datanode")
> .setConf(confForInfoServer)
> .setACL(new AccessControlList(conf.get(DFS_ADMIN, " ")))
> .hostName(getHostnameForSpnegoPrincipal(confForInfoServer))
> .addEndpoint(URI.create("http://localhost:0";))
> .setFindPort(true);
> this.infoServer = builder.build();
> {code}
> The confForInfoServer is a new configuration instance, while the dfsadmin
> reconfig the datanode's config, the config result cannot reflect to
> confForInfoServer, so we should use the datanode's conf.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13226) RBF: We should throw the failure validate and refuse this mount entry
[
https://issues.apache.org/jira/browse/HDFS-13226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16389436#comment-16389436
]
maobaolong commented on HDFS-13226:
---
[~linyiqun] Would you like to take another look?
> RBF: We should throw the failure validate and refuse this mount entry
> -
>
> Key: HDFS-13226
> URL: https://issues.apache.org/jira/browse/HDFS-13226
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs
>Affects Versions: 3.2.0
>Reporter: maobaolong
>Assignee: maobaolong
>Priority: Major
> Labels: RBF
> Fix For: 3.2.0
>
> Attachments: HDFS-13226.001.patch
>
>
> one of the mount entry source path rule is that the source path must start
> with '\', somebody didn't follow the rule and execute the following command:
> {code:bash}
> $ hdfs dfsrouteradmin -add addnode/ ns1 /addnode/
> {code}
> But, the console show we are successful add this entry.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-13226) RBF: We should throw the failure validate and refuse this mount entry
[
https://issues.apache.org/jira/browse/HDFS-13226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16389436#comment-16389436
]
maobaolong edited comment on HDFS-13226 at 3/7/18 11:46 AM:
[~linyiqun] Would you like to take another look?
[~elgoiri] This check is useless under my test, if we set the dest to null, the
IllegalArgumentException will be throw from the normalizeFileSystemPath method.
was (Author: maobaolong):
[~linyiqun] Would you like to take another look?
> RBF: We should throw the failure validate and refuse this mount entry
> -
>
> Key: HDFS-13226
> URL: https://issues.apache.org/jira/browse/HDFS-13226
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs
>Affects Versions: 3.2.0
>Reporter: maobaolong
>Assignee: maobaolong
>Priority: Major
> Labels: RBF
> Fix For: 3.2.0
>
> Attachments: HDFS-13226.001.patch
>
>
> one of the mount entry source path rule is that the source path must start
> with '\', somebody didn't follow the rule and execute the following command:
> {code:bash}
> $ hdfs dfsrouteradmin -add addnode/ ns1 /addnode/
> {code}
> But, the console show we are successful add this entry.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13226) RBF: We should throw the failure validate and refuse this mount entry
[
https://issues.apache.org/jira/browse/HDFS-13226?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
maobaolong updated HDFS-13226:
--
Attachment: HDFS-13226.002.patch
> RBF: We should throw the failure validate and refuse this mount entry
> -
>
> Key: HDFS-13226
> URL: https://issues.apache.org/jira/browse/HDFS-13226
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs
>Affects Versions: 3.2.0
>Reporter: maobaolong
>Assignee: maobaolong
>Priority: Major
> Labels: RBF
> Fix For: 3.2.0
>
> Attachments: HDFS-13226.001.patch, HDFS-13226.002.patch
>
>
> one of the mount entry source path rule is that the source path must start
> with '\', somebody didn't follow the rule and execute the following command:
> {code:bash}
> $ hdfs dfsrouteradmin -add addnode/ ns1 /addnode/
> {code}
> But, the console show we are successful add this entry.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13241) RBF: TestRouterSafemode failed if the port 8888 in use
[ https://issues.apache.org/jira/browse/HDFS-13241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16389467#comment-16389467 ] maobaolong commented on HDFS-13241: --- [~elgoiri] Please check this patch, a minor fix. > RBF: TestRouterSafemode failed if the port in use > -- > > Key: HDFS-13241 > URL: https://issues.apache.org/jira/browse/HDFS-13241 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs, test >Affects Versions: 3.2.0 >Reporter: maobaolong >Assignee: maobaolong >Priority: Major > Attachments: HDFS-13241.001.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13226) RBF: We should throw the failure validate and refuse this mount entry
[
https://issues.apache.org/jira/browse/HDFS-13226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16390537#comment-16390537
]
maobaolong commented on HDFS-13226:
---
[~elgoiri] Yeah, i think throw the IllegalArgumentException is the wise choose,
indeed, user want to see the first mistake, and the message should be display
to the user directly.
So, my suggestion is that we should change the return type of validate to void.
@Override public void validate() {
super.validate();if (!this.getSourcePath().startsWith("/")
|| this.getSourcePath().startsWith("//")) { throw new
IllegalArgumentException("Invalid entry, all mount points must start with a
single / ");
}for (RemoteLocation loc : getDestinations()) { String nsId =
loc.getNameserviceId(); if (nsId.length() == 0) {throw new
IllegalArgumentException("Invalid entry, invalid destination nameservice ");
} else if (!loc.getDest().startsWith("/") ||
this.getSourcePath().startsWith("//")) {throw new
IllegalArgumentException("Invalid entry, all destination must start with a
single / ");
}
}
}
And we have to change the BaseRecord validate method return type and other
subclass too. As this a big deal, we must think twice.
> RBF: We should throw the failure validate and refuse this mount entry
> -
>
> Key: HDFS-13226
> URL: https://issues.apache.org/jira/browse/HDFS-13226
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs
>Affects Versions: 3.2.0
>Reporter: maobaolong
>Assignee: maobaolong
>Priority: Major
> Labels: RBF
> Fix For: 3.2.0
>
> Attachments: HDFS-13226.001.patch, HDFS-13226.002.patch
>
>
> one of the mount entry source path rule is that the source path must start
> with '\', somebody didn't follow the rule and execute the following command:
> {code:bash}
> $ hdfs dfsrouteradmin -add addnode/ ns1 /addnode/
> {code}
> But, the console show we are successful add this entry.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-13226) RBF: We should throw the failure validate and refuse this mount entry
[
https://issues.apache.org/jira/browse/HDFS-13226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16390537#comment-16390537
]
maobaolong edited comment on HDFS-13226 at 3/8/18 1:03 AM:
---
[~elgoiri] Yeah, i think throw the IllegalArgumentException is the wise choose,
indeed, user want to see the first mistake, and the message should be display
to the user directly.
So, my suggestion is that we should change the return type of validate to void.
{code:java}
@Override public void validate() {
super.validate();if (!this.getSourcePath().startsWith("/")
|| this.getSourcePath().startsWith("//")) { throw new
IllegalArgumentException("Invalid entry, all mount points must start with a
single / ");
}for (RemoteLocation loc : getDestinations()) { String nsId =
loc.getNameserviceId(); if (nsId.length() == 0) {throw new
IllegalArgumentException("Invalid entry, invalid destination nameservice ");
} else if (!loc.getDest().startsWith("/") ||
this.getSourcePath().startsWith("//")) {throw new
IllegalArgumentException("Invalid entry, all destination must start with a
single / ");
}
}
}
{code}
And we have to change the BaseRecord validate method return type and other
subclass too, or for compromise, we can keep the return value to boolean now?
As this a big deal, we must think twice.
was (Author: maobaolong):
[~elgoiri] Yeah, i think throw the IllegalArgumentException is the wise choose,
indeed, user want to see the first mistake, and the message should be display
to the user directly.
So, my suggestion is that we should change the return type of validate to void.
@Override public void validate() {
super.validate();if (!this.getSourcePath().startsWith("/")
|| this.getSourcePath().startsWith("//")) { throw new
IllegalArgumentException("Invalid entry, all mount points must start with a
single / ");
}for (RemoteLocation loc : getDestinations()) { String nsId =
loc.getNameserviceId(); if (nsId.length() == 0) {throw new
IllegalArgumentException("Invalid entry, invalid destination nameservice ");
} else if (!loc.getDest().startsWith("/") ||
this.getSourcePath().startsWith("//")) {throw new
IllegalArgumentException("Invalid entry, all destination must start with a
single / ");
}
}
}
And we have to change the BaseRecord validate method return type and other
subclass too. As this a big deal, we must think twice.
> RBF: We should throw the failure validate and refuse this mount entry
> -
>
> Key: HDFS-13226
> URL: https://issues.apache.org/jira/browse/HDFS-13226
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs
>Affects Versions: 3.2.0
>Reporter: maobaolong
>Assignee: maobaolong
>Priority: Major
> Labels: RBF
> Fix For: 3.2.0
>
> Attachments: HDFS-13226.001.patch, HDFS-13226.002.patch
>
>
> one of the mount entry source path rule is that the source path must start
> with '\', somebody didn't follow the rule and execute the following command:
> {code:bash}
> $ hdfs dfsrouteradmin -add addnode/ ns1 /addnode/
> {code}
> But, the console show we are successful add this entry.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-13226) RBF: We should throw the failure validate and refuse this mount entry
[
https://issues.apache.org/jira/browse/HDFS-13226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16390537#comment-16390537
]
maobaolong edited comment on HDFS-13226 at 3/8/18 1:04 AM:
---
[~elgoiri] Yeah, i think throw the IllegalArgumentException is the wise choose,
indeed, user want to see the first mistake, and the message should be display
to the user directly.
So, my suggestion is that we should change the return type of validate to void.
{code:java}
@Override
public void validate() {
super.validate();
if (!this.getSourcePath().startsWith("/")
|| this.getSourcePath().startsWith("//")) {
throw new IllegalArgumentException("Invalid entry, all mount points must
start with a single / ");
}
for (RemoteLocation loc : getDestinations()) {
String nsId = loc.getNameserviceId();
if (nsId.length() == 0) {
throw new IllegalArgumentException("Invalid entry, invalid destination
nameservice ");
} else if (!loc.getDest().startsWith("/") ||
this.getSourcePath().startsWith("//")) {
throw new IllegalArgumentException("Invalid entry, all destination must
start with a single / ");
}
}
}
{code}
And we have to change the BaseRecord validate method return type and other
subclass too, or for compromise, we can keep the return value to boolean now?
As this a big deal, we must think twice.
was (Author: maobaolong):
[~elgoiri] Yeah, i think throw the IllegalArgumentException is the wise choose,
indeed, user want to see the first mistake, and the message should be display
to the user directly.
So, my suggestion is that we should change the return type of validate to void.
{code:java}
@Override
public boolean validate() {
if (!super.validate()) {
return false;
}
if (!this.getSourcePath().startsWith("/")
|| this.getSourcePath().startsWith("//")) {
throw new IllegalArgumentException("Invalid entry, all mount points must
start with a single / ");
}
for (RemoteLocation loc : getDestinations()) {
String nsId = loc.getNameserviceId();
if (nsId.length() == 0) {
throw new IllegalArgumentException("Invalid entry, invalid destination
nameservice ");
} else if (!loc.getDest().startsWith("/") ||
this.getSourcePath().startsWith("//")) {
throw new IllegalArgumentException("Invalid entry, all destination must
start with a single / ");
}
}
return true;
}
{code}
And we have to change the BaseRecord validate method return type and other
subclass too, or for compromise, we can keep the return value to boolean now?
As this a big deal, we must think twice.
> RBF: We should throw the failure validate and refuse this mount entry
> -
>
> Key: HDFS-13226
> URL: https://issues.apache.org/jira/browse/HDFS-13226
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs
>Affects Versions: 3.2.0
>Reporter: maobaolong
>Assignee: maobaolong
>Priority: Major
> Labels: RBF
> Fix For: 3.2.0
>
> Attachments: HDFS-13226.001.patch, HDFS-13226.002.patch
>
>
> one of the mount entry source path rule is that the source path must start
> with '\', somebody didn't follow the rule and execute the following command:
> {code:bash}
> $ hdfs dfsrouteradmin -add addnode/ ns1 /addnode/
> {code}
> But, the console show we are successful add this entry.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-13226) RBF: We should throw the failure validate and refuse this mount entry
[
https://issues.apache.org/jira/browse/HDFS-13226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16390537#comment-16390537
]
maobaolong edited comment on HDFS-13226 at 3/8/18 1:04 AM:
---
[~elgoiri] Yeah, i think throw the IllegalArgumentException is the wise choose,
indeed, user want to see the first mistake, and the message should be display
to the user directly.
So, my suggestion is that we should change the return type of validate to void.
{code:java}
@Override
public boolean validate() {
if (!super.validate()) {
return false;
}
if (!this.getSourcePath().startsWith("/")
|| this.getSourcePath().startsWith("//")) {
throw new IllegalArgumentException("Invalid entry, all mount points must
start with a single / ");
}
for (RemoteLocation loc : getDestinations()) {
String nsId = loc.getNameserviceId();
if (nsId.length() == 0) {
throw new IllegalArgumentException("Invalid entry, invalid destination
nameservice ");
} else if (!loc.getDest().startsWith("/") ||
this.getSourcePath().startsWith("//")) {
throw new IllegalArgumentException("Invalid entry, all destination must
start with a single / ");
}
}
return true;
}
{code}
And we have to change the BaseRecord validate method return type and other
subclass too, or for compromise, we can keep the return value to boolean now?
As this a big deal, we must think twice.
was (Author: maobaolong):
[~elgoiri] Yeah, i think throw the IllegalArgumentException is the wise choose,
indeed, user want to see the first mistake, and the message should be display
to the user directly.
So, my suggestion is that we should change the return type of validate to void.
{code:java}
@Override public void validate() {
super.validate();if (!this.getSourcePath().startsWith("/")
|| this.getSourcePath().startsWith("//")) { throw new
IllegalArgumentException("Invalid entry, all mount points must start with a
single / ");
}for (RemoteLocation loc : getDestinations()) { String nsId =
loc.getNameserviceId(); if (nsId.length() == 0) {throw new
IllegalArgumentException("Invalid entry, invalid destination nameservice ");
} else if (!loc.getDest().startsWith("/") ||
this.getSourcePath().startsWith("//")) {throw new
IllegalArgumentException("Invalid entry, all destination must start with a
single / ");
}
}
}
{code}
And we have to change the BaseRecord validate method return type and other
subclass too, or for compromise, we can keep the return value to boolean now?
As this a big deal, we must think twice.
> RBF: We should throw the failure validate and refuse this mount entry
> -
>
> Key: HDFS-13226
> URL: https://issues.apache.org/jira/browse/HDFS-13226
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs
>Affects Versions: 3.2.0
>Reporter: maobaolong
>Assignee: maobaolong
>Priority: Major
> Labels: RBF
> Fix For: 3.2.0
>
> Attachments: HDFS-13226.001.patch, HDFS-13226.002.patch
>
>
> one of the mount entry source path rule is that the source path must start
> with '\', somebody didn't follow the rule and execute the following command:
> {code:bash}
> $ hdfs dfsrouteradmin -add addnode/ ns1 /addnode/
> {code}
> But, the console show we are successful add this entry.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-13226) RBF: We should throw the failure validate and refuse this mount entry
[
https://issues.apache.org/jira/browse/HDFS-13226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16390537#comment-16390537
]
maobaolong edited comment on HDFS-13226 at 3/8/18 1:06 AM:
---
[~elgoiri] Yeah, i think throw the IllegalArgumentException is the wise choose,
indeed, user want to see the first mistake, and the message should be display
to the user directly.
So, my suggestion is that we should change the return type of validate to void.
{code:java}
@Override
public void validate() {
super.validate();
if (!this.getSourcePath().startsWith("/")
|| this.getSourcePath().startsWith("//")) {
throw new IllegalArgumentException("Invalid entry, all mount points must
start with a single / ");
}
for (RemoteLocation loc : getDestinations()) {
String nsId = loc.getNameserviceId();
if (nsId.length() == 0) {
throw new IllegalArgumentException("Invalid entry, invalid destination
nameservice ");
} else if (!loc.getDest().startsWith("/") ||
this.getSourcePath().startsWith("//")) {
throw new IllegalArgumentException("Invalid entry, all destination must
start with a single / ");
}
}
}
{code}
And we have to change the BaseRecord validate method return type and other
subclass too, or for compromise, we can keep the return value to boolean now?
As this a big deal, we must think twice.
Maybe we can reference the HAState.checkOperation
was (Author: maobaolong):
[~elgoiri] Yeah, i think throw the IllegalArgumentException is the wise choose,
indeed, user want to see the first mistake, and the message should be display
to the user directly.
So, my suggestion is that we should change the return type of validate to void.
{code:java}
@Override
public void validate() {
super.validate();
if (!this.getSourcePath().startsWith("/")
|| this.getSourcePath().startsWith("//")) {
throw new IllegalArgumentException("Invalid entry, all mount points must
start with a single / ");
}
for (RemoteLocation loc : getDestinations()) {
String nsId = loc.getNameserviceId();
if (nsId.length() == 0) {
throw new IllegalArgumentException("Invalid entry, invalid destination
nameservice ");
} else if (!loc.getDest().startsWith("/") ||
this.getSourcePath().startsWith("//")) {
throw new IllegalArgumentException("Invalid entry, all destination must
start with a single / ");
}
}
}
{code}
And we have to change the BaseRecord validate method return type and other
subclass too, or for compromise, we can keep the return value to boolean now?
As this a big deal, we must think twice.
> RBF: We should throw the failure validate and refuse this mount entry
> -
>
> Key: HDFS-13226
> URL: https://issues.apache.org/jira/browse/HDFS-13226
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs
>Affects Versions: 3.2.0
>Reporter: maobaolong
>Assignee: maobaolong
>Priority: Major
> Labels: RBF
> Fix For: 3.2.0
>
> Attachments: HDFS-13226.001.patch, HDFS-13226.002.patch
>
>
> one of the mount entry source path rule is that the source path must start
> with '\', somebody didn't follow the rule and execute the following command:
> {code:bash}
> $ hdfs dfsrouteradmin -add addnode/ ns1 /addnode/
> {code}
> But, the console show we are successful add this entry.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13226) RBF: We should throw the failure validate and refuse this mount entry
[
https://issues.apache.org/jira/browse/HDFS-13226?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
maobaolong updated HDFS-13226:
--
Attachment: HDFS-13226.004.patch
HDFS-13226.003.patch
> RBF: We should throw the failure validate and refuse this mount entry
> -
>
> Key: HDFS-13226
> URL: https://issues.apache.org/jira/browse/HDFS-13226
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs
>Affects Versions: 3.2.0
>Reporter: maobaolong
>Assignee: maobaolong
>Priority: Major
> Labels: RBF
> Fix For: 3.2.0
>
> Attachments: HDFS-13226.001.patch, HDFS-13226.002.patch,
> HDFS-13226.003.patch, HDFS-13226.004.patch
>
>
> one of the mount entry source path rule is that the source path must start
> with '\', somebody didn't follow the rule and execute the following command:
> {code:bash}
> $ hdfs dfsrouteradmin -add addnode/ ns1 /addnode/
> {code}
> But, the console show we are successful add this entry.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13226) RBF: We should throw the failure validate and refuse this mount entry
[
https://issues.apache.org/jira/browse/HDFS-13226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16390567#comment-16390567
]
maobaolong commented on HDFS-13226:
---
As the Router has not been used widely, we maybe can correct this completely,
If so, Please take a look at HDFS-13226.003.patch.
Otherwise, we can use the HDFS-13226.004.patch, just a little strange for a
boolean method return a useless return value.
Please feel free for that, we can discuss more.
> RBF: We should throw the failure validate and refuse this mount entry
> -
>
> Key: HDFS-13226
> URL: https://issues.apache.org/jira/browse/HDFS-13226
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs
>Affects Versions: 3.2.0
>Reporter: maobaolong
>Assignee: maobaolong
>Priority: Major
> Labels: RBF
> Fix For: 3.2.0
>
> Attachments: HDFS-13226.001.patch, HDFS-13226.002.patch,
> HDFS-13226.003.patch, HDFS-13226.004.patch
>
>
> one of the mount entry source path rule is that the source path must start
> with '\', somebody didn't follow the rule and execute the following command:
> {code:bash}
> $ hdfs dfsrouteradmin -add addnode/ ns1 /addnode/
> {code}
> But, the console show we are successful add this entry.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13226) RBF: We should throw the failure validate and refuse this mount entry
[
https://issues.apache.org/jira/browse/HDFS-13226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16390570#comment-16390570
]
maobaolong commented on HDFS-13226:
---
[~elgoiri] I think we should check the exception for the unit test, because the
exception has been catch by RouterAdmin.
> RBF: We should throw the failure validate and refuse this mount entry
> -
>
> Key: HDFS-13226
> URL: https://issues.apache.org/jira/browse/HDFS-13226
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs
>Affects Versions: 3.2.0
>Reporter: maobaolong
>Assignee: maobaolong
>Priority: Major
> Labels: RBF
> Fix For: 3.2.0
>
> Attachments: HDFS-13226.001.patch, HDFS-13226.002.patch,
> HDFS-13226.003.patch, HDFS-13226.004.patch
>
>
> one of the mount entry source path rule is that the source path must start
> with '\', somebody didn't follow the rule and execute the following command:
> {code:bash}
> $ hdfs dfsrouteradmin -add addnode/ ns1 /addnode/
> {code}
> But, the console show we are successful add this entry.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13226) RBF: We should throw the failure validate and refuse this mount entry
[
https://issues.apache.org/jira/browse/HDFS-13226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16390619#comment-16390619
]
maobaolong commented on HDFS-13226:
---
[~linyiqun] So, what about v003 patch, it fit the HAState#checkOperation style,
and the end user really want to realize what mistake he has made, and we want
to change the return type from boolean to void.
> RBF: We should throw the failure validate and refuse this mount entry
> -
>
> Key: HDFS-13226
> URL: https://issues.apache.org/jira/browse/HDFS-13226
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs
>Affects Versions: 3.2.0
>Reporter: maobaolong
>Assignee: maobaolong
>Priority: Major
> Labels: RBF
> Fix For: 3.2.0
>
> Attachments: HDFS-13226.001.patch, HDFS-13226.002.patch,
> HDFS-13226.003.patch, HDFS-13226.004.patch
>
>
> one of the mount entry source path rule is that the source path must start
> with '\', somebody didn't follow the rule and execute the following command:
> {code:bash}
> $ hdfs dfsrouteradmin -add addnode/ ns1 /addnode/
> {code}
> But, the console show we are successful add this entry.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13226) RBF: We should throw the failure validate and refuse this mount entry
[
https://issues.apache.org/jira/browse/HDFS-13226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16390697#comment-16390697
]
maobaolong commented on HDFS-13226:
---
My next patch will base on v003, let me complete the doc and test follow your
comment. Thank you for your review.
> RBF: We should throw the failure validate and refuse this mount entry
> -
>
> Key: HDFS-13226
> URL: https://issues.apache.org/jira/browse/HDFS-13226
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs
>Affects Versions: 3.2.0
>Reporter: maobaolong
>Assignee: maobaolong
>Priority: Major
> Labels: RBF
> Fix For: 3.2.0
>
> Attachments: HDFS-13226.001.patch, HDFS-13226.002.patch,
> HDFS-13226.003.patch, HDFS-13226.004.patch
>
>
> one of the mount entry source path rule is that the source path must start
> with '\', somebody didn't follow the rule and execute the following command:
> {code:bash}
> $ hdfs dfsrouteradmin -add addnode/ ns1 /addnode/
> {code}
> But, the console show we are successful add this entry.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-13245) RBF: State store DBMS implementation
maobaolong created HDFS-13245: - Summary: RBF: State store DBMS implementation Key: HDFS-13245 URL: https://issues.apache.org/jira/browse/HDFS-13245 Project: Hadoop HDFS Issue Type: Sub-task Components: hdfs Reporter: maobaolong -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13226) RBF: We should throw the failure validate and refuse this mount entry
[
https://issues.apache.org/jira/browse/HDFS-13226?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
maobaolong updated HDFS-13226:
--
Attachment: HDFS-13226.005.patch
> RBF: We should throw the failure validate and refuse this mount entry
> -
>
> Key: HDFS-13226
> URL: https://issues.apache.org/jira/browse/HDFS-13226
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs
>Affects Versions: 3.2.0
>Reporter: maobaolong
>Assignee: maobaolong
>Priority: Major
> Labels: RBF
> Fix For: 3.2.0
>
> Attachments: HDFS-13226.001.patch, HDFS-13226.002.patch,
> HDFS-13226.003.patch, HDFS-13226.004.patch, HDFS-13226.005.patch
>
>
> one of the mount entry source path rule is that the source path must start
> with '\', somebody didn't follow the rule and execute the following command:
> {code:bash}
> $ hdfs dfsrouteradmin -add addnode/ ns1 /addnode/
> {code}
> But, the console show we are successful add this entry.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13226) RBF: We should throw the failure validate and refuse this mount entry
[
https://issues.apache.org/jira/browse/HDFS-13226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16391352#comment-16391352
]
maobaolong commented on HDFS-13226:
---
[~linyiqun] I've upload a new patch just now, not sure i have addressed all
your comments.
* I update the invalidate method java doc, remove the return value's
description.
* Create a new test case in TestMountTable.
* I keep the start with "//" check, and another Jira will complete the check
rule of mount entry.
> RBF: We should throw the failure validate and refuse this mount entry
> -
>
> Key: HDFS-13226
> URL: https://issues.apache.org/jira/browse/HDFS-13226
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs
>Affects Versions: 3.2.0
>Reporter: maobaolong
>Assignee: maobaolong
>Priority: Major
> Labels: RBF
> Fix For: 3.2.0
>
> Attachments: HDFS-13226.001.patch, HDFS-13226.002.patch,
> HDFS-13226.003.patch, HDFS-13226.004.patch, HDFS-13226.005.patch
>
>
> one of the mount entry source path rule is that the source path must start
> with '\', somebody didn't follow the rule and execute the following command:
> {code:bash}
> $ hdfs dfsrouteradmin -add addnode/ ns1 /addnode/
> {code}
> But, the console show we are successful add this entry.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13241) RBF: TestRouterSafemode failed if the port 8888 is in use
[
https://issues.apache.org/jira/browse/HDFS-13241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16391376#comment-16391376
]
maobaolong commented on HDFS-13241:
---
I use a tool start a tcp server and bind the port , then, i run the test
all of rbf, the test are all passed. So i think the others RBF tests is good
now.
{code:bash}
mvn -Dtest=org.apache.hadoop.hdfs.server.federation.** test
...
[INFO] BUILD SUCCESS
[INFO]
[INFO] Total time: 06:00 min
[INFO] Finished at: 2018-03-08T23:08:28+08:00
[INFO] Final Memory: 163M/1531M
[INFO]
{code}
> RBF: TestRouterSafemode failed if the port is in use
> -
>
> Key: HDFS-13241
> URL: https://issues.apache.org/jira/browse/HDFS-13241
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs, test
>Affects Versions: 3.2.0
>Reporter: maobaolong
>Assignee: maobaolong
>Priority: Major
> Attachments: HDFS-13241.001.patch
>
>
> TestRouterSafemode failed if the port is in use.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-13241) RBF: TestRouterSafemode failed if the port 8888 is in use
[
https://issues.apache.org/jira/browse/HDFS-13241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16391376#comment-16391376
]
maobaolong edited comment on HDFS-13241 at 3/8/18 3:13 PM:
---
[~elgoiri]
I use a tool start a tcp server and bind the port , then, i run the test
all of rbf, the test are all passed. So i think the others RBF tests is good
now.
{code}
mvn -Dtest=org.apache.hadoop.hdfs.server.federation.** test
...
[INFO] BUILD SUCCESS
[INFO]
[INFO] Total time: 06:00 min
[INFO] Finished at: 2018-03-08T23:08:28+08:00
[INFO] Final Memory: 163M/1531M
[INFO]
{code}
was (Author: maobaolong):
I use a tool start a tcp server and bind the port , then, i run the test
all of rbf, the test are all passed. So i think the others RBF tests is good
now.
{code:bash}
mvn -Dtest=org.apache.hadoop.hdfs.server.federation.** test
...
[INFO] BUILD SUCCESS
[INFO]
[INFO] Total time: 06:00 min
[INFO] Finished at: 2018-03-08T23:08:28+08:00
[INFO] Final Memory: 163M/1531M
[INFO]
{code}
> RBF: TestRouterSafemode failed if the port is in use
> -
>
> Key: HDFS-13241
> URL: https://issues.apache.org/jira/browse/HDFS-13241
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs, test
>Affects Versions: 3.2.0
>Reporter: maobaolong
>Assignee: maobaolong
>Priority: Major
> Attachments: HDFS-13241.001.patch
>
>
> TestRouterSafemode failed if the port is in use.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13248) RBF: namenode need to choose block location for the client
[ https://issues.apache.org/jira/browse/HDFS-13248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16391383#comment-16391383 ] maobaolong commented on HDFS-13248: --- Maybe we can modify the client logic, let the client get the correct namenode from router, then the client call to namenode. Not sure this is a wise way, because people doesn't want to update client package, further more, some body doesn't use the client. > RBF: namenode need to choose block location for the client > -- > > Key: HDFS-13248 > URL: https://issues.apache.org/jira/browse/HDFS-13248 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Weiwei Wu >Priority: Major > > When execute a put operation via router, the namenode will choose block > location for the router, not for the real client. This will affect the file's > locality. > I think on both namennode and router, we should add a new addBlock method, or > add a parameter for the current addBlock method, to pass the real client > information. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12615) Router-based HDFS federation phase 2
[ https://issues.apache.org/jira/browse/HDFS-12615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16391387#comment-16391387 ] maobaolong commented on HDFS-12615: --- [~elgoiri] I've create a new issue to track the dbms state store. BTW, I have a question, should us config the remote ns or data center's router to client? > Router-based HDFS federation phase 2 > > > Key: HDFS-12615 > URL: https://issues.apache.org/jira/browse/HDFS-12615 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Íñigo Goiri >Assignee: Íñigo Goiri >Priority: Major > Labels: RBF > > This umbrella JIRA tracks set of improvements over the Router-based HDFS > federation (HDFS-10467). -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13241) RBF: TestRouterSafemode failed if the port 8888 is in use
[ https://issues.apache.org/jira/browse/HDFS-13241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16392664#comment-16392664 ] maobaolong commented on HDFS-13241: --- [~elgoiri] After look at the jenkins failed test, i think the router.router is null. > RBF: TestRouterSafemode failed if the port is in use > - > > Key: HDFS-13241 > URL: https://issues.apache.org/jira/browse/HDFS-13241 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs, test >Affects Versions: 3.2.0 >Reporter: maobaolong >Assignee: maobaolong >Priority: Major > Attachments: HDFS-13241.001.patch > > > TestRouterSafemode failed if the port is in use. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-13241) RBF: TestRouterSafemode failed if the port 8888 is in use
[ https://issues.apache.org/jira/browse/HDFS-13241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16392664#comment-16392664 ] maobaolong edited comment on HDFS-13241 at 3/9/18 10:01 AM: [~elgoiri] After look at the jenkins failed test, i think the router.router is null. And on my computer, this test was passed. was (Author: maobaolong): [~elgoiri] After look at the jenkins failed test, i think the router.router is null. > RBF: TestRouterSafemode failed if the port is in use > - > > Key: HDFS-13241 > URL: https://issues.apache.org/jira/browse/HDFS-13241 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs, test >Affects Versions: 3.2.0 >Reporter: maobaolong >Assignee: maobaolong >Priority: Major > Attachments: HDFS-13241.001.patch > > > TestRouterSafemode failed if the port is in use. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13226) RBF: We should throw the failure validate and refuse this mount entry
[
https://issues.apache.org/jira/browse/HDFS-13226?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
maobaolong updated HDFS-13226:
--
Attachment: HDFS-13226.006.patch
> RBF: We should throw the failure validate and refuse this mount entry
> -
>
> Key: HDFS-13226
> URL: https://issues.apache.org/jira/browse/HDFS-13226
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs
>Affects Versions: 3.2.0
>Reporter: maobaolong
>Assignee: maobaolong
>Priority: Major
> Labels: RBF
> Fix For: 3.2.0
>
> Attachments: HDFS-13226.001.patch, HDFS-13226.002.patch,
> HDFS-13226.003.patch, HDFS-13226.004.patch, HDFS-13226.005.patch,
> HDFS-13226.006.patch
>
>
> one of the mount entry source path rule is that the source path must start
> with '\', somebody didn't follow the rule and execute the following command:
> {code:bash}
> $ hdfs dfsrouteradmin -add addnode/ ns1 /addnode/
> {code}
> But, the console show we are successful add this entry.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13226) RBF: We should throw the failure validate and refuse this mount entry
[
https://issues.apache.org/jira/browse/HDFS-13226?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
maobaolong updated HDFS-13226:
--
Attachment: (was: HDFS-13226.006.patch)
> RBF: We should throw the failure validate and refuse this mount entry
> -
>
> Key: HDFS-13226
> URL: https://issues.apache.org/jira/browse/HDFS-13226
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs
>Affects Versions: 3.2.0
>Reporter: maobaolong
>Assignee: maobaolong
>Priority: Major
> Labels: RBF
> Fix For: 3.2.0
>
> Attachments: HDFS-13226.001.patch, HDFS-13226.002.patch,
> HDFS-13226.003.patch, HDFS-13226.004.patch, HDFS-13226.005.patch,
> HDFS-13226.006.patch
>
>
> one of the mount entry source path rule is that the source path must start
> with '\', somebody didn't follow the rule and execute the following command:
> {code:bash}
> $ hdfs dfsrouteradmin -add addnode/ ns1 /addnode/
> {code}
> But, the console show we are successful add this entry.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13226) RBF: We should throw the failure validate and refuse this mount entry
[
https://issues.apache.org/jira/browse/HDFS-13226?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
maobaolong updated HDFS-13226:
--
Attachment: HDFS-13226.006.patch
> RBF: We should throw the failure validate and refuse this mount entry
> -
>
> Key: HDFS-13226
> URL: https://issues.apache.org/jira/browse/HDFS-13226
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs
>Affects Versions: 3.2.0
>Reporter: maobaolong
>Assignee: maobaolong
>Priority: Major
> Labels: RBF
> Fix For: 3.2.0
>
> Attachments: HDFS-13226.001.patch, HDFS-13226.002.patch,
> HDFS-13226.003.patch, HDFS-13226.004.patch, HDFS-13226.005.patch,
> HDFS-13226.006.patch
>
>
> one of the mount entry source path rule is that the source path must start
> with '\', somebody didn't follow the rule and execute the following command:
> {code:bash}
> $ hdfs dfsrouteradmin -add addnode/ ns1 /addnode/
> {code}
> But, the console show we are successful add this entry.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13226) RBF: We should throw the failure validate and refuse this mount entry
[
https://issues.apache.org/jira/browse/HDFS-13226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16392898#comment-16392898
]
maobaolong commented on HDFS-13226:
---
[~linyiqun] [~elgoiri] Thank you for your suggestion for improve my issue. I
think i have addressed your comments. PTAL.
[~linyiqun] Sorry for misunderstand your mind, I will create a new Jira to
change the logic of validate method.
> RBF: We should throw the failure validate and refuse this mount entry
> -
>
> Key: HDFS-13226
> URL: https://issues.apache.org/jira/browse/HDFS-13226
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs
>Affects Versions: 3.2.0
>Reporter: maobaolong
>Assignee: maobaolong
>Priority: Major
> Labels: RBF
> Fix For: 3.2.0
>
> Attachments: HDFS-13226.001.patch, HDFS-13226.002.patch,
> HDFS-13226.003.patch, HDFS-13226.004.patch, HDFS-13226.005.patch,
> HDFS-13226.006.patch
>
>
> one of the mount entry source path rule is that the source path must start
> with '\', somebody didn't follow the rule and execute the following command:
> {code:bash}
> $ hdfs dfsrouteradmin -add addnode/ ns1 /addnode/
> {code}
> But, the console show we are successful add this entry.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-15133) Use rocksdb to store NameNode inode and blockInfo
maobaolong created HDFS-15133: - Summary: Use rocksdb to store NameNode inode and blockInfo Key: HDFS-15133 URL: https://issues.apache.org/jira/browse/HDFS-15133 Project: Hadoop HDFS Issue Type: Improvement Reporter: maobaolong Maybe we don't need checkpoint to a fsimage file, the rocksdb checkpoint can achieve the same request. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15133) Use rocksdb to store NameNode inode and blockInfo
[ https://issues.apache.org/jira/browse/HDFS-15133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] maobaolong updated HDFS-15133: -- Component/s: namenode Affects Version/s: 3.3.0 > Use rocksdb to store NameNode inode and blockInfo > - > > Key: HDFS-15133 > URL: https://issues.apache.org/jira/browse/HDFS-15133 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 3.3.0 >Reporter: maobaolong >Priority: Major > > Maybe we don't need checkpoint to a fsimage file, the rocksdb checkpoint can > achieve the same request. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15133) Use rocksdb to store NameNode inode and blockInfo
[ https://issues.apache.org/jira/browse/HDFS-15133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] maobaolong updated HDFS-15133: -- Description: Maybe we don't need checkpoint to a fsimage file, the rocksdb checkpoint can achieve the same request. This is ozone and alluxio way to manage meta data of master node. was:Maybe we don't need checkpoint to a fsimage file, the rocksdb checkpoint can achieve the same request. > Use rocksdb to store NameNode inode and blockInfo > - > > Key: HDFS-15133 > URL: https://issues.apache.org/jira/browse/HDFS-15133 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 3.3.0 >Reporter: maobaolong >Priority: Major > > Maybe we don't need checkpoint to a fsimage file, the rocksdb checkpoint can > achieve the same request. > This is ozone and alluxio way to manage meta data of master node. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15133) Use rocksdb to store NameNode inode and blockInfo
[ https://issues.apache.org/jira/browse/HDFS-15133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17020727#comment-17020727 ] maobaolong commented on HDFS-15133: --- [~iwasakims] [~hemanthboyina] After review the HDFS-5389 and HDFS-8286, i saw the similar design, but they are all aimed to invent a new approach to achieve the goal. This jira mainly hope to reuse the approach of how to manage racksdb which implemented in HDDS. The RDBStore and TypedTable can be responsible for the kv store manager, so we can starts all work by the moving the code of RDBStore related to hadoop-common, so that ozone and hdfs or yarn and other component can use this wonderful feature without any more effort. > Use rocksdb to store NameNode inode and blockInfo > - > > Key: HDFS-15133 > URL: https://issues.apache.org/jira/browse/HDFS-15133 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 3.3.0 >Reporter: maobaolong >Priority: Major > > Maybe we don't need checkpoint to a fsimage file, the rocksdb checkpoint can > achieve the same request. > This is ozone and alluxio way to manage meta data of master node. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-15137) Move RDBStore logic from apache-ozone into hadoop-commons module of apache-hadoop
maobaolong created HDFS-15137: - Summary: Move RDBStore logic from apache-ozone into hadoop-commons module of apache-hadoop Key: HDFS-15137 URL: https://issues.apache.org/jira/browse/HDFS-15137 Project: Hadoop HDFS Issue Type: Sub-task Reporter: maobaolong -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-15138) Use RDBStore and TypedTable to manage the inode of namenode
maobaolong created HDFS-15138: - Summary: Use RDBStore and TypedTable to manage the inode of namenode Key: HDFS-15138 URL: https://issues.apache.org/jira/browse/HDFS-15138 Project: Hadoop HDFS Issue Type: Sub-task Components: namenode Affects Versions: 3.3.0 Reporter: maobaolong Replace FSDirectory.inodeMap.map from GSet to rocksdb. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-15139) Use RDBStore and TypedTable to manage the blockinfo of namenode
maobaolong created HDFS-15139: - Summary: Use RDBStore and TypedTable to manage the blockinfo of namenode Key: HDFS-15139 URL: https://issues.apache.org/jira/browse/HDFS-15139 Project: Hadoop HDFS Issue Type: Sub-task Components: namenode Affects Versions: 3.3.0 Reporter: maobaolong replace the BlockManager.BlocksMap.blocks from GSet to rocksdb -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15133) Use rocksdb to store NameNode inode and blockInfo
[ https://issues.apache.org/jira/browse/HDFS-15133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17024858#comment-17024858 ] maobaolong commented on HDFS-15133: --- [~brahmareddy] [~ayushtkn], After some work these day, i give some update now. - I copy the package `org.apache.hadoop.hdds.utils.db` from hadoop-ozone project to the package `org.apache.hadoop.hdds.utils.db` of the module hadoop-common of the project hadoop. In fact i think i have done the sub task HDFS-15137 - I'm doing a refactor for INodeMap, i try to abstract the `GSet map` into a concept named `InodeStore`. It have two implementation, one named HeapInodeStore, as the name, it is the origin mode, the other named RocksInodeStore, it work as rocksdb mode. In fact, i'm doing the part of https://issues.apache.org/jira/browse/HDFS-15138. But, i met some problem, - It is hard to measure the size of a rocksdb? - how to implements the method getMapIterator graceful for the RocksInodeStore. - Clear method will be implements to reset the rocksdb for the RocksInodeStore. Future Job - I will commit my work to my branch named rocks-metastore, my repo is https://github.com/maobaolong/hadoop. As now, it contains compile error now, so i will commit several days later. - Feel free to discuss with my here or in the slack hdfs channel - Feel free to take these jira task or help me to correct my mistake - Doing some minor test. > Use rocksdb to store NameNode inode and blockInfo > - > > Key: HDFS-15133 > URL: https://issues.apache.org/jira/browse/HDFS-15133 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 3.3.0 >Reporter: maobaolong >Priority: Major > > Maybe we don't need checkpoint to a fsimage file, the rocksdb checkpoint can > achieve the same request. > This is ozone and alluxio way to manage meta data of master node. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
