[jira] [Assigned] (HDFS-14098) use fsck tools for EC blockId will throw NullPointerException
[
https://issues.apache.org/jira/browse/HDFS-14098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
luoge123 reassigned HDFS-14098:
---
Assignee: Aihua Xu (was: luoge123)
> use fsck tools for EC blockId will throw NullPointerException
> -
>
> Key: HDFS-14098
> URL: https://issues.apache.org/jira/browse/HDFS-14098
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: erasure-coding
>Affects Versions: 3.0.0
>Reporter: luoge123
>Assignee: Aihua Xu
>Priority: Major
>
>
> when EC file have some block missing, use fsck tool for EC blockId whil throw
> NullPointerException
> {code:java}
> hdfs fsck -blockId blk_-9223372036800049376
> fsck information is:
> Block Id: blk_-9223372036800049376
> Block belongs to: /logdata/test.lzo
> No. of Expected Replica: 9
> No. of live Replica: 5
> No. of excess Replica: 0
> No. of stale Replica: 0
> No. of decommissioned Replica: 0
> No. of decommissioning Replica: 0
> No. of corrupted Replica: 0
> null
> {code}
> namenode will throw NullPointerException:
> {code:java}
> 2018-11-26 15:59:35,107 WARN org.apache.hadoop.hdfs.server.namenode.NameNode:
> Fsck on blockId 'blk_-9223372036800049376
> java.lang.NullPointerException
> at
> org.apache.hadoop.hdfs.server.namenode.NamenodeFsck.blockIdCK(NamenodeFsck.java:270)
> at
> org.apache.hadoop.hdfs.server.namenode.NamenodeFsck.fsck(NamenodeFsck.java:313)
> at
> org.apache.hadoop.hdfs.server.namenode.FsckServlet$1.run(FsckServlet.java:67)
> {code}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14098) use fsck tools for EC blockId will throw NullPointerException
[
https://issues.apache.org/jira/browse/HDFS-14098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16782923#comment-16782923
]
luoge123 commented on HDFS-14098:
-
[~aihuaxu], sorry for delay, I assigned it to you.
> use fsck tools for EC blockId will throw NullPointerException
> -
>
> Key: HDFS-14098
> URL: https://issues.apache.org/jira/browse/HDFS-14098
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: erasure-coding
>Affects Versions: 3.0.0
>Reporter: luoge123
>Assignee: luoge123
>Priority: Major
>
>
> when EC file have some block missing, use fsck tool for EC blockId whil throw
> NullPointerException
> {code:java}
> hdfs fsck -blockId blk_-9223372036800049376
> fsck information is:
> Block Id: blk_-9223372036800049376
> Block belongs to: /logdata/test.lzo
> No. of Expected Replica: 9
> No. of live Replica: 5
> No. of excess Replica: 0
> No. of stale Replica: 0
> No. of decommissioned Replica: 0
> No. of decommissioning Replica: 0
> No. of corrupted Replica: 0
> null
> {code}
> namenode will throw NullPointerException:
> {code:java}
> 2018-11-26 15:59:35,107 WARN org.apache.hadoop.hdfs.server.namenode.NameNode:
> Fsck on blockId 'blk_-9223372036800049376
> java.lang.NullPointerException
> at
> org.apache.hadoop.hdfs.server.namenode.NamenodeFsck.blockIdCK(NamenodeFsck.java:270)
> at
> org.apache.hadoop.hdfs.server.namenode.NamenodeFsck.fsck(NamenodeFsck.java:313)
> at
> org.apache.hadoop.hdfs.server.namenode.FsckServlet$1.run(FsckServlet.java:67)
> {code}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-14098) use fsck tools for EC blockId will throw NullPointerException
[
https://issues.apache.org/jira/browse/HDFS-14098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
luoge123 reassigned HDFS-14098:
---
Assignee: luoge123
> use fsck tools for EC blockId will throw NullPointerException
> -
>
> Key: HDFS-14098
> URL: https://issues.apache.org/jira/browse/HDFS-14098
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: erasure-coding
>Affects Versions: 3.0.0
>Reporter: luoge123
>Assignee: luoge123
>Priority: Major
>
>
> when EC file have some block missing, use fsck tool for EC blockId whil throw
> NullPointerException
> {code:java}
> hdfs fsck -blockId blk_-9223372036800049376
> fsck information is:
> Block Id: blk_-9223372036800049376
> Block belongs to: /logdata/test.lzo
> No. of Expected Replica: 9
> No. of live Replica: 5
> No. of excess Replica: 0
> No. of stale Replica: 0
> No. of decommissioned Replica: 0
> No. of decommissioning Replica: 0
> No. of corrupted Replica: 0
> null
> {code}
> namenode will throw NullPointerException:
> {code:java}
> 2018-11-26 15:59:35,107 WARN org.apache.hadoop.hdfs.server.namenode.NameNode:
> Fsck on blockId 'blk_-9223372036800049376
> java.lang.NullPointerException
> at
> org.apache.hadoop.hdfs.server.namenode.NamenodeFsck.blockIdCK(NamenodeFsck.java:270)
> at
> org.apache.hadoop.hdfs.server.namenode.NamenodeFsck.fsck(NamenodeFsck.java:313)
> at
> org.apache.hadoop.hdfs.server.namenode.FsckServlet$1.run(FsckServlet.java:67)
> {code}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-14098) use fsck tools for EC blockId will throw NullPointerException
luoge123 created HDFS-14098:
---
Summary: use fsck tools for EC blockId will throw
NullPointerException
Key: HDFS-14098
URL: https://issues.apache.org/jira/browse/HDFS-14098
Project: Hadoop HDFS
Issue Type: Bug
Components: erasure-coding
Affects Versions: 3.0.0
Reporter: luoge123
when EC file have some block missing, use fsck tool for EC blockId whil throw
NullPointerException
{code:java}
hdfs fsck -blockId blk_-9223372036800049376
fsck information is:
Block Id: blk_-9223372036800049376
Block belongs to: /logdata/test.lzo
No. of Expected Replica: 9
No. of live Replica: 5
No. of excess Replica: 0
No. of stale Replica: 0
No. of decommissioned Replica: 0
No. of decommissioning Replica: 0
No. of corrupted Replica: 0
null
{code}
namenode will throw NullPointerException:
{code:java}
2018-11-26 15:59:35,107 WARN org.apache.hadoop.hdfs.server.namenode.NameNode:
Fsck on blockId 'blk_-9223372036800049376
java.lang.NullPointerException
at
org.apache.hadoop.hdfs.server.namenode.NamenodeFsck.blockIdCK(NamenodeFsck.java:270)
at
org.apache.hadoop.hdfs.server.namenode.NamenodeFsck.fsck(NamenodeFsck.java:313)
at org.apache.hadoop.hdfs.server.namenode.FsckServlet$1.run(FsckServlet.java:67)
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-13543) when datanode have some unmounted disks, disk balancer should skip these disks not throw IllegalArgumentException
luoge123 created HDFS-13543:
---
Summary: when datanode have some unmounted disks, disk balancer
should skip these disks not throw IllegalArgumentException
Key: HDFS-13543
URL: https://issues.apache.org/jira/browse/HDFS-13543
Project: Hadoop HDFS
Issue Type: Bug
Components: diskbalancer
Affects Versions: 3.0.0
Reporter: luoge123
when datanode has an unmounted disk, disk balancer get disk capacity from
report is zero, this will case getVolumeInfoFromStorageReports throw
IllegalArgumentException
{code:java}
java.lang.IllegalArgumentException
at com.google.common.base.Preconditions.checkArgument(Preconditions.java:72)
at
org.apache.hadoop.hdfs.server.diskbalancer.datamodel.DiskBalancerVolume.setUsed(DiskBalancerVolume.java:268)
at
org.apache.hadoop.hdfs.server.diskbalancer.connectors.DBNameNodeConnector.getVolumeInfoFromStorageReports(DBNameNodeConnector.java:148)
at
org.apache.hadoop.hdfs.server.diskbalancer.connectors.DBNameNodeConnector.getNodes(DBNameNodeConnector.java:90)
at
org.apache.hadoop.hdfs.server.diskbalancer.datamodel.DiskBalancerCluster.readClusterInfo(DiskBalancerCluster.java:133)
at
org.apache.hadoop.hdfs.server.diskbalancer.command.Command.readClusterInfo(Command.java:123)
at
org.apache.hadoop.hdfs.server.diskbalancer.command.ReportCommand.execute(ReportCommand.java:74)
at
org.apache.hadoop.hdfs.tools.DiskBalancerCLI.dispatch(DiskBalancerCLI.java:468)
at org.apache.hadoop.hdfs.tools.DiskBalancerCLI.run(DiskBalancerCLI.java:183)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
at org.apache.hadoop.hdfs.tools.DiskBalancerCLI.main(DiskBalancerCLI.java:164)
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12555) HDFS federation should support configure secondary directory
[ https://issues.apache.org/jira/browse/HDFS-12555?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] luoge123 updated HDFS-12555: Status: Patch Available (was: Open) > HDFS federation should support configure secondary directory > - > > Key: HDFS-12555 > URL: https://issues.apache.org/jira/browse/HDFS-12555 > Project: Hadoop HDFS > Issue Type: Improvement > Components: federation > Environment: 2.6.0-cdh5.10.0 >Reporter: luoge123 >Assignee: luoge123 > Fix For: 3.0.0 > > Attachments: HDFS-12555.001.patch, HDFS-12555.002.patch, > HDFS-12555.003.patch, HDFS-12555.004.patch > > > HDFS federation support multiple namenodes horizontally scales the file > system namespace. As the amount of data grows, using a single group of > namenodes to manage a single directory, namenode still achieves performance > bottlenecks. In order to reduce the pressure of namenode, we can split out > the secondary directory, and manager it by a new namenode. This is > transparent for users. > For example, nn1 only manager the /user directory, when nn1 achieve > performance bottlenecks, we can split out /user/hive directory, and ues nn2 > to manager it. > That means core-site.xml should support as follows configuration. > >fs.viewfs.mounttable.nsX.link./user >hdfs://nn1:8020/user > > >fs.viewfs.mounttable.nsX.link./user/hive >hdfs://nn2:8020/user/hive > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12555) HDFS federation should support configure secondary directory
[ https://issues.apache.org/jira/browse/HDFS-12555?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] luoge123 updated HDFS-12555: Attachment: HDFS-12555.004.patch > HDFS federation should support configure secondary directory > - > > Key: HDFS-12555 > URL: https://issues.apache.org/jira/browse/HDFS-12555 > Project: Hadoop HDFS > Issue Type: Improvement > Components: federation > Environment: 2.6.0-cdh5.10.0 >Reporter: luoge123 >Assignee: luoge123 > Fix For: 3.0.0 > > Attachments: HDFS-12555.001.patch, HDFS-12555.002.patch, > HDFS-12555.003.patch, HDFS-12555.004.patch > > > HDFS federation support multiple namenodes horizontally scales the file > system namespace. As the amount of data grows, using a single group of > namenodes to manage a single directory, namenode still achieves performance > bottlenecks. In order to reduce the pressure of namenode, we can split out > the secondary directory, and manager it by a new namenode. This is > transparent for users. > For example, nn1 only manager the /user directory, when nn1 achieve > performance bottlenecks, we can split out /user/hive directory, and ues nn2 > to manager it. > That means core-site.xml should support as follows configuration. > >fs.viewfs.mounttable.nsX.link./user >hdfs://nn1:8020/user > > >fs.viewfs.mounttable.nsX.link./user/hive >hdfs://nn2:8020/user/hive > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12555) HDFS federation should support configure secondary directory
[ https://issues.apache.org/jira/browse/HDFS-12555?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] luoge123 updated HDFS-12555: Status: Open (was: Patch Available) > HDFS federation should support configure secondary directory > - > > Key: HDFS-12555 > URL: https://issues.apache.org/jira/browse/HDFS-12555 > Project: Hadoop HDFS > Issue Type: Improvement > Components: federation > Environment: 2.6.0-cdh5.10.0 >Reporter: luoge123 >Assignee: luoge123 > Fix For: 3.0.0 > > Attachments: HDFS-12555.001.patch, HDFS-12555.002.patch, > HDFS-12555.003.patch > > > HDFS federation support multiple namenodes horizontally scales the file > system namespace. As the amount of data grows, using a single group of > namenodes to manage a single directory, namenode still achieves performance > bottlenecks. In order to reduce the pressure of namenode, we can split out > the secondary directory, and manager it by a new namenode. This is > transparent for users. > For example, nn1 only manager the /user directory, when nn1 achieve > performance bottlenecks, we can split out /user/hive directory, and ues nn2 > to manager it. > That means core-site.xml should support as follows configuration. > >fs.viewfs.mounttable.nsX.link./user >hdfs://nn1:8020/user > > >fs.viewfs.mounttable.nsX.link./user/hive >hdfs://nn2:8020/user/hive > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12555) HDFS federation should support configure secondary directory
[ https://issues.apache.org/jira/browse/HDFS-12555?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] luoge123 updated HDFS-12555: Fix Version/s: (was: 2.6.0) 3.0.0 Target Version/s: 3.0.0 (was: 2.6.0) Status: Patch Available (was: Open) > HDFS federation should support configure secondary directory > - > > Key: HDFS-12555 > URL: https://issues.apache.org/jira/browse/HDFS-12555 > Project: Hadoop HDFS > Issue Type: Improvement > Components: federation > Environment: 2.6.0-cdh5.10.0 >Reporter: luoge123 >Assignee: luoge123 > Fix For: 3.0.0 > > Attachments: HDFS-12555.001.patch, HDFS-12555.002.patch, > HDFS-12555.003.patch > > > HDFS federation support multiple namenodes horizontally scales the file > system namespace. As the amount of data grows, using a single group of > namenodes to manage a single directory, namenode still achieves performance > bottlenecks. In order to reduce the pressure of namenode, we can split out > the secondary directory, and manager it by a new namenode. This is > transparent for users. > For example, nn1 only manager the /user directory, when nn1 achieve > performance bottlenecks, we can split out /user/hive directory, and ues nn2 > to manager it. > That means core-site.xml should support as follows configuration. > >fs.viewfs.mounttable.nsX.link./user >hdfs://nn1:8020/user > > >fs.viewfs.mounttable.nsX.link./user/hive >hdfs://nn2:8020/user/hive > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12555) HDFS federation should support configure secondary directory
[ https://issues.apache.org/jira/browse/HDFS-12555?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] luoge123 updated HDFS-12555: Attachment: HDFS-12555.003.patch > HDFS federation should support configure secondary directory > - > > Key: HDFS-12555 > URL: https://issues.apache.org/jira/browse/HDFS-12555 > Project: Hadoop HDFS > Issue Type: Improvement > Components: federation > Environment: 2.6.0-cdh5.10.0 >Reporter: luoge123 >Assignee: luoge123 > Fix For: 3.0.0 > > Attachments: HDFS-12555.001.patch, HDFS-12555.002.patch, > HDFS-12555.003.patch > > > HDFS federation support multiple namenodes horizontally scales the file > system namespace. As the amount of data grows, using a single group of > namenodes to manage a single directory, namenode still achieves performance > bottlenecks. In order to reduce the pressure of namenode, we can split out > the secondary directory, and manager it by a new namenode. This is > transparent for users. > For example, nn1 only manager the /user directory, when nn1 achieve > performance bottlenecks, we can split out /user/hive directory, and ues nn2 > to manager it. > That means core-site.xml should support as follows configuration. > >fs.viewfs.mounttable.nsX.link./user >hdfs://nn1:8020/user > > >fs.viewfs.mounttable.nsX.link./user/hive >hdfs://nn2:8020/user/hive > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12555) HDFS federation should support configure secondary directory
[ https://issues.apache.org/jira/browse/HDFS-12555?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] luoge123 updated HDFS-12555: Status: Open (was: Patch Available) > HDFS federation should support configure secondary directory > - > > Key: HDFS-12555 > URL: https://issues.apache.org/jira/browse/HDFS-12555 > Project: Hadoop HDFS > Issue Type: Improvement > Components: federation > Environment: 2.6.0-cdh5.10.0 >Reporter: luoge123 >Assignee: luoge123 > Fix For: 2.6.0 > > Attachments: HDFS-12555.001.patch, HDFS-12555.002.patch > > > HDFS federation support multiple namenodes horizontally scales the file > system namespace. As the amount of data grows, using a single group of > namenodes to manage a single directory, namenode still achieves performance > bottlenecks. In order to reduce the pressure of namenode, we can split out > the secondary directory, and manager it by a new namenode. This is > transparent for users. > For example, nn1 only manager the /user directory, when nn1 achieve > performance bottlenecks, we can split out /user/hive directory, and ues nn2 > to manager it. > That means core-site.xml should support as follows configuration. > >fs.viewfs.mounttable.nsX.link./user >hdfs://nn1:8020/user > > >fs.viewfs.mounttable.nsX.link./user/hive >hdfs://nn2:8020/user/hive > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12555) HDFS federation should support configure secondary directory
[ https://issues.apache.org/jira/browse/HDFS-12555?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] luoge123 updated HDFS-12555: Attachment: (was: HDFS-12555.003.patch) > HDFS federation should support configure secondary directory > - > > Key: HDFS-12555 > URL: https://issues.apache.org/jira/browse/HDFS-12555 > Project: Hadoop HDFS > Issue Type: Improvement > Components: federation > Environment: 2.6.0-cdh5.10.0 >Reporter: luoge123 >Assignee: luoge123 > Fix For: 2.6.0 > > Attachments: HDFS-12555.001.patch, HDFS-12555.002.patch > > > HDFS federation support multiple namenodes horizontally scales the file > system namespace. As the amount of data grows, using a single group of > namenodes to manage a single directory, namenode still achieves performance > bottlenecks. In order to reduce the pressure of namenode, we can split out > the secondary directory, and manager it by a new namenode. This is > transparent for users. > For example, nn1 only manager the /user directory, when nn1 achieve > performance bottlenecks, we can split out /user/hive directory, and ues nn2 > to manager it. > That means core-site.xml should support as follows configuration. > >fs.viewfs.mounttable.nsX.link./user >hdfs://nn1:8020/user > > >fs.viewfs.mounttable.nsX.link./user/hive >hdfs://nn2:8020/user/hive > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12555) HDFS federation should support configure secondary directory
[ https://issues.apache.org/jira/browse/HDFS-12555?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] luoge123 updated HDFS-12555: Affects Version/s: (was: 2.6.0) > HDFS federation should support configure secondary directory > - > > Key: HDFS-12555 > URL: https://issues.apache.org/jira/browse/HDFS-12555 > Project: Hadoop HDFS > Issue Type: Improvement > Components: federation > Environment: 2.6.0-cdh5.10.0 >Reporter: luoge123 >Assignee: luoge123 > Fix For: 2.6.0 > > Attachments: HDFS-12555.001.patch, HDFS-12555.002.patch, > HDFS-12555.003.patch > > > HDFS federation support multiple namenodes horizontally scales the file > system namespace. As the amount of data grows, using a single group of > namenodes to manage a single directory, namenode still achieves performance > bottlenecks. In order to reduce the pressure of namenode, we can split out > the secondary directory, and manager it by a new namenode. This is > transparent for users. > For example, nn1 only manager the /user directory, when nn1 achieve > performance bottlenecks, we can split out /user/hive directory, and ues nn2 > to manager it. > That means core-site.xml should support as follows configuration. > >fs.viewfs.mounttable.nsX.link./user >hdfs://nn1:8020/user > > >fs.viewfs.mounttable.nsX.link./user/hive >hdfs://nn2:8020/user/hive > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12555) HDFS federation should support configure secondary directory
[ https://issues.apache.org/jira/browse/HDFS-12555?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] luoge123 updated HDFS-12555: Attachment: HDFS-12555.003.patch > HDFS federation should support configure secondary directory > - > > Key: HDFS-12555 > URL: https://issues.apache.org/jira/browse/HDFS-12555 > Project: Hadoop HDFS > Issue Type: Improvement > Components: federation >Affects Versions: 2.6.0 > Environment: 2.6.0-cdh5.10.0 >Reporter: luoge123 >Assignee: luoge123 > Fix For: 2.6.0 > > Attachments: HDFS-12555.001.patch, HDFS-12555.002.patch, > HDFS-12555.003.patch > > > HDFS federation support multiple namenodes horizontally scales the file > system namespace. As the amount of data grows, using a single group of > namenodes to manage a single directory, namenode still achieves performance > bottlenecks. In order to reduce the pressure of namenode, we can split out > the secondary directory, and manager it by a new namenode. This is > transparent for users. > For example, nn1 only manager the /user directory, when nn1 achieve > performance bottlenecks, we can split out /user/hive directory, and ues nn2 > to manager it. > That means core-site.xml should support as follows configuration. > >fs.viewfs.mounttable.nsX.link./user >hdfs://nn1:8020/user > > >fs.viewfs.mounttable.nsX.link./user/hive >hdfs://nn2:8020/user/hive > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12555) HDFS federation should support configure secondary directory
[ https://issues.apache.org/jira/browse/HDFS-12555?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] luoge123 updated HDFS-12555: Status: Open (was: Patch Available) > HDFS federation should support configure secondary directory > - > > Key: HDFS-12555 > URL: https://issues.apache.org/jira/browse/HDFS-12555 > Project: Hadoop HDFS > Issue Type: Improvement > Components: federation >Affects Versions: 2.6.0 > Environment: 2.6.0-cdh5.10.0 >Reporter: luoge123 >Assignee: luoge123 > Fix For: 2.6.0 > > Attachments: HDFS-12555.001.patch, HDFS-12555.002.patch, > HDFS-12555.003.patch > > > HDFS federation support multiple namenodes horizontally scales the file > system namespace. As the amount of data grows, using a single group of > namenodes to manage a single directory, namenode still achieves performance > bottlenecks. In order to reduce the pressure of namenode, we can split out > the secondary directory, and manager it by a new namenode. This is > transparent for users. > For example, nn1 only manager the /user directory, when nn1 achieve > performance bottlenecks, we can split out /user/hive directory, and ues nn2 > to manager it. > That means core-site.xml should support as follows configuration. > >fs.viewfs.mounttable.nsX.link./user >hdfs://nn1:8020/user > > >fs.viewfs.mounttable.nsX.link./user/hive >hdfs://nn2:8020/user/hive > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12555) HDFS federation should support configure secondary directory
[ https://issues.apache.org/jira/browse/HDFS-12555?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] luoge123 updated HDFS-12555: Status: Patch Available (was: Open) > HDFS federation should support configure secondary directory > - > > Key: HDFS-12555 > URL: https://issues.apache.org/jira/browse/HDFS-12555 > Project: Hadoop HDFS > Issue Type: Improvement > Components: federation >Affects Versions: 2.6.0 > Environment: 2.6.0-cdh5.10.0 >Reporter: luoge123 >Assignee: luoge123 > Fix For: 2.6.0 > > Attachments: HDFS-12555.001.patch, HDFS-12555.002.patch, > HDFS-12555.003.patch > > > HDFS federation support multiple namenodes horizontally scales the file > system namespace. As the amount of data grows, using a single group of > namenodes to manage a single directory, namenode still achieves performance > bottlenecks. In order to reduce the pressure of namenode, we can split out > the secondary directory, and manager it by a new namenode. This is > transparent for users. > For example, nn1 only manager the /user directory, when nn1 achieve > performance bottlenecks, we can split out /user/hive directory, and ues nn2 > to manager it. > That means core-site.xml should support as follows configuration. > >fs.viewfs.mounttable.nsX.link./user >hdfs://nn1:8020/user > > >fs.viewfs.mounttable.nsX.link./user/hive >hdfs://nn2:8020/user/hive > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12555) HDFS federation should support configure secondary directory
[ https://issues.apache.org/jira/browse/HDFS-12555?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] luoge123 updated HDFS-12555: Affects Version/s: 2.6.0 Status: Patch Available (was: Open) > HDFS federation should support configure secondary directory > - > > Key: HDFS-12555 > URL: https://issues.apache.org/jira/browse/HDFS-12555 > Project: Hadoop HDFS > Issue Type: Improvement > Components: federation >Affects Versions: 2.6.0 > Environment: 2.6.0-cdh5.10.0 >Reporter: luoge123 >Assignee: luoge123 > Fix For: 2.6.0 > > Attachments: HDFS-12555.001.patch, HDFS-12555.002.patch > > > HDFS federation support multiple namenodes horizontally scales the file > system namespace. As the amount of data grows, using a single group of > namenodes to manage a single directory, namenode still achieves performance > bottlenecks. In order to reduce the pressure of namenode, we can split out > the secondary directory, and manager it by a new namenode. This is > transparent for users. > For example, nn1 only manager the /user directory, when nn1 achieve > performance bottlenecks, we can split out /user/hive directory, and ues nn2 > to manager it. > That means core-site.xml should support as follows configuration. > >fs.viewfs.mounttable.nsX.link./user >hdfs://nn1:8020/user > > >fs.viewfs.mounttable.nsX.link./user/hive >hdfs://nn2:8020/user/hive > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12555) HDFS federation should support configure secondary directory
[ https://issues.apache.org/jira/browse/HDFS-12555?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] luoge123 updated HDFS-12555: Status: Open (was: Patch Available) > HDFS federation should support configure secondary directory > - > > Key: HDFS-12555 > URL: https://issues.apache.org/jira/browse/HDFS-12555 > Project: Hadoop HDFS > Issue Type: Improvement > Components: federation > Environment: 2.6.0-cdh5.10.0 >Reporter: luoge123 >Assignee: luoge123 > Fix For: 2.6.0 > > Attachments: HDFS-12555.001.patch, HDFS-12555.002.patch > > > HDFS federation support multiple namenodes horizontally scales the file > system namespace. As the amount of data grows, using a single group of > namenodes to manage a single directory, namenode still achieves performance > bottlenecks. In order to reduce the pressure of namenode, we can split out > the secondary directory, and manager it by a new namenode. This is > transparent for users. > For example, nn1 only manager the /user directory, when nn1 achieve > performance bottlenecks, we can split out /user/hive directory, and ues nn2 > to manager it. > That means core-site.xml should support as follows configuration. > >fs.viewfs.mounttable.nsX.link./user >hdfs://nn1:8020/user > > >fs.viewfs.mounttable.nsX.link./user/hive >hdfs://nn2:8020/user/hive > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12555) HDFS federation should support configure secondary directory
[ https://issues.apache.org/jira/browse/HDFS-12555?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] luoge123 updated HDFS-12555: Attachment: (was: HDFS-12555.003.patch) > HDFS federation should support configure secondary directory > - > > Key: HDFS-12555 > URL: https://issues.apache.org/jira/browse/HDFS-12555 > Project: Hadoop HDFS > Issue Type: Improvement > Components: federation > Environment: 2.6.0-cdh5.10.0 >Reporter: luoge123 >Assignee: luoge123 > Fix For: 2.6.0 > > Attachments: HDFS-12555.001.patch, HDFS-12555.002.patch > > > HDFS federation support multiple namenodes horizontally scales the file > system namespace. As the amount of data grows, using a single group of > namenodes to manage a single directory, namenode still achieves performance > bottlenecks. In order to reduce the pressure of namenode, we can split out > the secondary directory, and manager it by a new namenode. This is > transparent for users. > For example, nn1 only manager the /user directory, when nn1 achieve > performance bottlenecks, we can split out /user/hive directory, and ues nn2 > to manager it. > That means core-site.xml should support as follows configuration. > >fs.viewfs.mounttable.nsX.link./user >hdfs://nn1:8020/user > > >fs.viewfs.mounttable.nsX.link./user/hive >hdfs://nn2:8020/user/hive > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12555) HDFS federation should support configure secondary directory
[ https://issues.apache.org/jira/browse/HDFS-12555?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] luoge123 updated HDFS-12555: Attachment: HDFS-12555.003.patch > HDFS federation should support configure secondary directory > - > > Key: HDFS-12555 > URL: https://issues.apache.org/jira/browse/HDFS-12555 > Project: Hadoop HDFS > Issue Type: Improvement > Components: federation > Environment: 2.6.0-cdh5.10.0 >Reporter: luoge123 >Assignee: luoge123 > Fix For: 2.6.0 > > Attachments: HDFS-12555.001.patch, HDFS-12555.002.patch, > HDFS-12555.003.patch > > > HDFS federation support multiple namenodes horizontally scales the file > system namespace. As the amount of data grows, using a single group of > namenodes to manage a single directory, namenode still achieves performance > bottlenecks. In order to reduce the pressure of namenode, we can split out > the secondary directory, and manager it by a new namenode. This is > transparent for users. > For example, nn1 only manager the /user directory, when nn1 achieve > performance bottlenecks, we can split out /user/hive directory, and ues nn2 > to manager it. > That means core-site.xml should support as follows configuration. > >fs.viewfs.mounttable.nsX.link./user >hdfs://nn1:8020/user > > >fs.viewfs.mounttable.nsX.link./user/hive >hdfs://nn2:8020/user/hive > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12555) HDFS federation should support configure secondary directory
[ https://issues.apache.org/jira/browse/HDFS-12555?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] luoge123 updated HDFS-12555: Target Version/s: 2.6.0 > HDFS federation should support configure secondary directory > - > > Key: HDFS-12555 > URL: https://issues.apache.org/jira/browse/HDFS-12555 > Project: Hadoop HDFS > Issue Type: Improvement > Components: federation > Environment: 2.6.0-cdh5.10.0 >Reporter: luoge123 >Assignee: luoge123 > Fix For: 2.6.0 > > Attachments: HDFS-12555.001.patch, HDFS-12555.002.patch > > > HDFS federation support multiple namenodes horizontally scales the file > system namespace. As the amount of data grows, using a single group of > namenodes to manage a single directory, namenode still achieves performance > bottlenecks. In order to reduce the pressure of namenode, we can split out > the secondary directory, and manager it by a new namenode. This is > transparent for users. > For example, nn1 only manager the /user directory, when nn1 achieve > performance bottlenecks, we can split out /user/hive directory, and ues nn2 > to manager it. > That means core-site.xml should support as follows configuration. > >fs.viewfs.mounttable.nsX.link./user >hdfs://nn1:8020/user > > >fs.viewfs.mounttable.nsX.link./user/hive >hdfs://nn2:8020/user/hive > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12555) HDFS federation should support configure secondary directory
[ https://issues.apache.org/jira/browse/HDFS-12555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16293792#comment-16293792 ] luoge123 commented on HDFS-12555: - Thanks [~hanishakoneru] and [~arpitagarwal] > HDFS federation should support configure secondary directory > - > > Key: HDFS-12555 > URL: https://issues.apache.org/jira/browse/HDFS-12555 > Project: Hadoop HDFS > Issue Type: Improvement > Components: federation > Environment: 2.6.0-cdh5.10.0 >Reporter: luoge123 >Assignee: luoge123 > Fix For: 2.6.0 > > Attachments: HDFS-12555.001.patch, HDFS-12555.002.patch > > > HDFS federation support multiple namenodes horizontally scales the file > system namespace. As the amount of data grows, using a single group of > namenodes to manage a single directory, namenode still achieves performance > bottlenecks. In order to reduce the pressure of namenode, we can split out > the secondary directory, and manager it by a new namenode. This is > transparent for users. > For example, nn1 only manager the /user directory, when nn1 achieve > performance bottlenecks, we can split out /user/hive directory, and ues nn2 > to manager it. > That means core-site.xml should support as follows configuration. > >fs.viewfs.mounttable.nsX.link./user >hdfs://nn1:8020/user > > >fs.viewfs.mounttable.nsX.link./user/hive >hdfs://nn2:8020/user/hive > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12555) HDFS federation should support configure secondary directory
[ https://issues.apache.org/jira/browse/HDFS-12555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16292008#comment-16292008 ] luoge123 commented on HDFS-12555: - HI [~hanishakoneru], this patch already available, and works well for our cluster, but I can't assign this issue to myself, so this patch can't test by Hadoop QA > HDFS federation should support configure secondary directory > - > > Key: HDFS-12555 > URL: https://issues.apache.org/jira/browse/HDFS-12555 > Project: Hadoop HDFS > Issue Type: Improvement > Components: federation > Environment: 2.6.0-cdh5.10.0 >Reporter: luoge123 > Fix For: 2.6.0 > > Attachments: HDFS-12555.001.patch, HDFS-12555.002.patch > > > HDFS federation support multiple namenodes horizontally scales the file > system namespace. As the amount of data grows, using a single group of > namenodes to manage a single directory, namenode still achieves performance > bottlenecks. In order to reduce the pressure of namenode, we can split out > the secondary directory, and manager it by a new namenode. This is > transparent for users. > For example, nn1 only manager the /user directory, when nn1 achieve > performance bottlenecks, we can split out /user/hive directory, and ues nn2 > to manager it. > That means core-site.xml should support as follows configuration. > >fs.viewfs.mounttable.nsX.link./user >hdfs://nn1:8020/user > > >fs.viewfs.mounttable.nsX.link./user/hive >hdfs://nn2:8020/user/hive > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12555) HDFS federation should support configure secondary directory
[ https://issues.apache.org/jira/browse/HDFS-12555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16208780#comment-16208780 ] luoge123 commented on HDFS-12555: - Hi, [~bharatviswa] I have no authority to assign this issue to myself, can you help me? > HDFS federation should support configure secondary directory > - > > Key: HDFS-12555 > URL: https://issues.apache.org/jira/browse/HDFS-12555 > Project: Hadoop HDFS > Issue Type: Improvement > Components: federation > Environment: 2.6.0-cdh5.10.0 >Reporter: luoge123 > Fix For: 2.6.0 > > Attachments: HDFS-12555.001.patch, HDFS-12555.002.patch > > > HDFS federation support multiple namenodes horizontally scales the file > system namespace. As the amount of data grows, using a single group of > namenodes to manage a single directory, namenode still achieves performance > bottlenecks. In order to reduce the pressure of namenode, we can split out > the secondary directory, and manager it by a new namenode. This is > transparent for users. > For example, nn1 only manager the /user directory, when nn1 achieve > performance bottlenecks, we can split out /user/hive directory, and ues nn2 > to manager it. > That means core-site.xml should support as follows configuration. > >fs.viewfs.mounttable.nsX.link./user >hdfs://nn1:8020/user > > >fs.viewfs.mounttable.nsX.link./user/hive >hdfs://nn2:8020/user/hive > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12555) HDFS federation should support configure secondary directory
[ https://issues.apache.org/jira/browse/HDFS-12555?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] luoge123 updated HDFS-12555: Attachment: HDFS-12555.002.patch > HDFS federation should support configure secondary directory > - > > Key: HDFS-12555 > URL: https://issues.apache.org/jira/browse/HDFS-12555 > Project: Hadoop HDFS > Issue Type: Improvement > Components: federation > Environment: 2.6.0-cdh5.10.0 >Reporter: luoge123 > Fix For: 2.6.0 > > Attachments: HDFS-12555.001.patch, HDFS-12555.002.patch > > > HDFS federation support multiple namenodes horizontally scales the file > system namespace. As the amount of data grows, using a single group of > namenodes to manage a single directory, namenode still achieves performance > bottlenecks. In order to reduce the pressure of namenode, we can split out > the secondary directory, and manager it by a new namenode. This is > transparent for users. > For example, nn1 only manager the /user directory, when nn1 achieve > performance bottlenecks, we can split out /user/hive directory, and ues nn2 > to manager it. > That means core-site.xml should support as follows configuration. > >fs.viewfs.mounttable.nsX.link./user >hdfs://nn1:8020/user > > >fs.viewfs.mounttable.nsX.link./user/hive >hdfs://nn2:8020/user/hive > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12555) HDFS federation should support configure secondary directory
[ https://issues.apache.org/jira/browse/HDFS-12555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16201505#comment-16201505 ] luoge123 commented on HDFS-12555: - In current federation function, if the /user directory has been configured as follows : fs.viewfs.mounttable.nsX.link./user hdfs://nn1:8020/user then, we can't add the follow /user/hive configuration to core-site.xml any more. fs.viewfs.mounttable.nsX.link./user/hive hdfs://nn2:8020/user/hive My improvement is that the two configuration items all exist in core-site.xml at same time. After support this configuration, we need copy data from hdfs://nn1:8020/user/hive to hdfs://nn2:8020/user/hive, fastcp or distcp tools can help us. And then we can delete hdfs://nn1:8020/user/hive. After we done this, the meta data in /user directory except /user/hive all manager by nn1, and read and write requests are send to nn1. The meta data for /user/hive manager by nn2, read and write requests for /user/hive are send to nn2. I'm not sure this explanation could answer your question or not, if you have any questions, please fell free to contact me. > HDFS federation should support configure secondary directory > - > > Key: HDFS-12555 > URL: https://issues.apache.org/jira/browse/HDFS-12555 > Project: Hadoop HDFS > Issue Type: Improvement > Components: federation > Environment: 2.6.0-cdh5.10.0 >Reporter: luoge123 > Fix For: 2.6.0 > > Attachments: HDFS-12555.001.patch > > > HDFS federation support multiple namenodes horizontally scales the file > system namespace. As the amount of data grows, using a single group of > namenodes to manage a single directory, namenode still achieves performance > bottlenecks. In order to reduce the pressure of namenode, we can split out > the secondary directory, and manager it by a new namenode. This is > transparent for users. > For example, nn1 only manager the /user directory, when nn1 achieve > performance bottlenecks, we can split out /user/hive directory, and ues nn2 > to manager it. > That means core-site.xml should support as follows configuration. > >fs.viewfs.mounttable.nsX.link./user >hdfs://nn1:8020/user > > >fs.viewfs.mounttable.nsX.link./user/hive >hdfs://nn2:8020/user/hive > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12555) HDFS federation should support configure secondary directory
[ https://issues.apache.org/jira/browse/HDFS-12555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16198064#comment-16198064 ] luoge123 commented on HDFS-12555: - Thanks for your kindly advise, i’m working on it > HDFS federation should support configure secondary directory > - > > Key: HDFS-12555 > URL: https://issues.apache.org/jira/browse/HDFS-12555 > Project: Hadoop HDFS > Issue Type: Improvement > Components: federation > Environment: 2.6.0-cdh5.10.0 >Reporter: luoge123 > Fix For: 2.6.0 > > Attachments: HDFS-12555.001.patch > > > HDFS federation support multiple namenodes horizontally scales the file > system namespace. As the amount of data grows, using a single group of > namenodes to manage a single directory, namenode still achieves performance > bottlenecks. In order to reduce the pressure of namenode, we can split out > the secondary directory, and manager it by a new namenode. This is > transparent for users. > For example, nn1 only manager the /user directory, when nn1 achieve > performance bottlenecks, we can split out /user/hive directory, and ues nn2 > to manager it. > That means core-site.xml should support as follows configuration. > >fs.viewfs.mounttable.nsX.link./user >hdfs://nn1:8020/user > > >fs.viewfs.mounttable.nsX.link./user/hive >hdfs://nn2:8020/user/hive > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12555) HDFS federation should support configure secondary directory
[ https://issues.apache.org/jira/browse/HDFS-12555?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] luoge123 updated HDFS-12555: Environment: 2.6.0-cdh5.10.0 Hadoop Flags: Reviewed > HDFS federation should support configure secondary directory > - > > Key: HDFS-12555 > URL: https://issues.apache.org/jira/browse/HDFS-12555 > Project: Hadoop HDFS > Issue Type: Improvement > Components: federation > Environment: 2.6.0-cdh5.10.0 >Reporter: luoge123 > Fix For: 2.6.0 > > Attachments: HDFS-12555.001.patch > > > HDFS federation support multiple namenodes horizontally scales the file > system namespace. As the amount of data grows, using a single group of > namenodes to manage a single directory, namenode still achieves performance > bottlenecks. In order to reduce the pressure of namenode, we can split out > the secondary directory, and manager it by a new namenode. This is > transparent for users. > For example, nn1 only manager the /user directory, when nn1 achieve > performance bottlenecks, we can split out /user/hive directory, and ues nn2 > to manager it. > That means core-site.xml should support as follows configuration. > >fs.viewfs.mounttable.nsX.link./user >hdfs://nn1:8020/user > > >fs.viewfs.mounttable.nsX.link./user/hive >hdfs://nn2:8020/user/hive > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12555) HDFS federation should support configure secondary directory
[ https://issues.apache.org/jira/browse/HDFS-12555?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] luoge123 updated HDFS-12555: Fix Version/s: 2.6.0 > HDFS federation should support configure secondary directory > - > > Key: HDFS-12555 > URL: https://issues.apache.org/jira/browse/HDFS-12555 > Project: Hadoop HDFS > Issue Type: Improvement > Components: federation >Reporter: luoge123 > Fix For: 2.6.0 > > Attachments: HDFS-12555.001.patch > > > HDFS federation support multiple namenodes horizontally scales the file > system namespace. As the amount of data grows, using a single group of > namenodes to manage a single directory, namenode still achieves performance > bottlenecks. In order to reduce the pressure of namenode, we can split out > the secondary directory, and manager it by a new namenode. This is > transparent for users. > For example, nn1 only manager the /user directory, when nn1 achieve > performance bottlenecks, we can split out /user/hive directory, and ues nn2 > to manager it. > That means core-site.xml should support as follows configuration. > >fs.viewfs.mounttable.nsX.link./user >hdfs://nn1:8020/user > > >fs.viewfs.mounttable.nsX.link./user/hive >hdfs://nn2:8020/user/hive > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12555) HDFS federation should support configure secondary directory
[ https://issues.apache.org/jira/browse/HDFS-12555?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] luoge123 updated HDFS-12555: Attachment: HDFS-12555.001.patch > HDFS federation should support configure secondary directory > - > > Key: HDFS-12555 > URL: https://issues.apache.org/jira/browse/HDFS-12555 > Project: Hadoop HDFS > Issue Type: Improvement > Components: federation >Reporter: luoge123 > Attachments: HDFS-12555.001.patch > > > HDFS federation support multiple namenodes horizontally scales the file > system namespace. As the amount of data grows, using a single group of > namenodes to manage a single directory, namenode still achieves performance > bottlenecks. In order to reduce the pressure of namenode, we can split out > the secondary directory, and manager it by a new namenode. This is > transparent for users. > For example, nn1 only manager the /user directory, when nn1 achieve > performance bottlenecks, we can split out /user/hive directory, and ues nn2 > to manager it. > That means core-site.xml should support as follows configuration. > >fs.viewfs.mounttable.nsX.link./user >hdfs://nn1:8020/user > > >fs.viewfs.mounttable.nsX.link./user/hive >hdfs://nn2:8020/user/hive > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-12555) HDFS federation should support configure secondary directory
luoge123 created HDFS-12555: --- Summary: HDFS federation should support configure secondary directory Key: HDFS-12555 URL: https://issues.apache.org/jira/browse/HDFS-12555 Project: Hadoop HDFS Issue Type: Improvement Components: federation Reporter: luoge123 HDFS federation support multiple namenodes horizontally scales the file system namespace. As the amount of data grows, using a single group of namenodes to manage a single directory, namenode still achieves performance bottlenecks. In order to reduce the pressure of namenode, we can split out the secondary directory, and manager it by a new namenode. This is transparent for users. For example, nn1 only manager the /user directory, when nn1 achieve performance bottlenecks, we can split out /user/hive directory, and ues nn2 to manager it. That means core-site.xml should support as follows configuration. fs.viewfs.mounttable.nsX.link./user hdfs://nn1:8020/user fs.viewfs.mounttable.nsX.link./user/hive hdfs://nn2:8020/user/hive -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
