[
https://issues.apache.org/jira/browse/HDFS-10309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15249743#comment-15249743
]
Amit Anand commented on HDFS-10309:
-----------------------------------
After building the patched jar I was able to test it successfully:
{code}
hdfs@bcpc-vm3:~$ hdfs balancer
16/04/20 08:05:54 INFO balancer.Balancer: namenodes = [hdfs://Test-Laptop]
16/04/20 08:05:54 INFO balancer.Balancer: parameters =
Balancer.BalancerParameters [BalancingPolicy.Node, threshold = 10.0, max idle
iteration = 5, #excluded nodes = 0, #included nodes = 0, #source nodes = 0,
#blockpools = 0, run during upgrade = false]
16/04/20 08:05:54 INFO balancer.Balancer: included nodes = []
16/04/20 08:05:54 INFO balancer.Balancer: excluded nodes = []
16/04/20 08:05:54 INFO balancer.Balancer: source nodes = []
Time Stamp Iteration# Bytes Already Moved Bytes Left To Move
Bytes Being Moved
16/04/20 08:05:55 INFO balancer.KeyManager: Block token params received from
NN: update interval=10hrs, 0sec, token lifetime=10hrs, 0sec
16/04/20 08:05:55 INFO block.BlockTokenSecretManager: Setting block keys
16/04/20 08:05:55 INFO balancer.KeyManager: Update block keys every 2hrs,
30mins, 0sec
16/04/20 08:05:55 INFO balancer.Balancer: dfs.balancer.movedWinWidth = 5400000
(default=5400000)
16/04/20 08:05:55 INFO balancer.Balancer: dfs.balancer.moverThreads = 1000
(default=1000)
16/04/20 08:05:55 INFO balancer.Balancer: dfs.balancer.dispatcherThreads = 200
(default=200)
16/04/20 08:05:55 INFO balancer.Balancer:
dfs.datanode.balance.max.concurrent.moves = 5 (default=5)
16/04/20 08:05:55 INFO balancer.Balancer: dfs.balancer.getBlocks.size =
2147483648 (default=2147483648)
16/04/20 08:05:55 INFO balancer.Balancer: dfs.balancer.getBlocks.min-block-size
= 10485760 (default=10485760)
16/04/20 08:05:55 INFO block.BlockTokenSecretManager: Setting block keys
16/04/20 08:05:55 INFO balancer.Balancer: dfs.balancer.max-size-to-move =
10737418240 (default=10737418240)
16/04/20 08:05:55 INFO balancer.Balancer: dfs.blocksize = 134217728
(default=134217728)
16/04/20 08:05:55 INFO net.NetworkTopology: Adding a new node:
/default-rack/192.168.100.14:1004
16/04/20 08:05:55 INFO net.NetworkTopology: Adding a new node:
/default-rack/192.168.100.15:1004
16/04/20 08:05:55 INFO net.NetworkTopology: Adding a new node:
/default-rack/192.168.100.13:1004
16/04/20 08:05:55 INFO balancer.Balancer: 0 over-utilized: []
16/04/20 08:05:55 INFO balancer.Balancer: 0 underutilized: []
The cluster is balanced. Exiting...
Apr 20, 2016 8:05:55 AM 0 0 B 0 B
-1 B
Apr 20, 2016 8:05:56 AM Balancing took 2.179 seconds
{code}
> HDFS Balancer doesn't honor dfs.blocksize value defined with suffix k(kilo),
> m(mega), g(giga)
> ---------------------------------------------------------------------------------------------
>
> Key: HDFS-10309
> URL: https://issues.apache.org/jira/browse/HDFS-10309
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: balancer & mover
> Affects Versions: 2.8.0
> Reporter: Amit Anand
> Assignee: Amit Anand
> Fix For: 2.8.0
>
> Attachments: HDFS-10309.01.patch
>
>
> While running HDFS Balancer I get error given below when {{dfs.blockSize}} is
> defined with suffix {{k(kilo), m(mega), g(giga)}} in {{hdfs-site.xml}}. In my
> deployment {{dfs.blocksize}} is set to {{128m}}.
> {code}
> hdfs@bcpc-vm1:/home/ubuntu$ hdfs balancer
> 16/04/19 08:49:51 INFO balancer.Balancer: namenodes = [hdfs://Test-Laptop]
> 16/04/19 08:49:51 INFO balancer.Balancer: parameters =
> Balancer.BalancerParameters [BalancingPolicy.Node, threshold = 10.0, max idle
> iteration = 5, #excluded nodes = 0, #included nodes = 0, #source
> nodes = 0, #blockpools = 0, run during upgrade = false]
> 16/04/19 08:49:51 INFO balancer.Balancer: included nodes = []
> 16/04/19 08:49:51 INFO balancer.Balancer: excluded nodes = []
> 16/04/19 08:49:51 INFO balancer.Balancer: source nodes = []
> Time Stamp Iteration# Bytes Already Moved Bytes Left To Move
> Bytes Being Moved
> 16/04/19 08:49:52 INFO balancer.KeyManager: Block token params received from
> NN: update interval=10hrs, 0sec, token lifetime=10hrs, 0sec
> 16/04/19 08:49:52 INFO block.BlockTokenSecretManager: Setting block keys
> 16/04/19 08:49:52 INFO balancer.KeyManager: Update block keys every 2hrs,
> 30mins, 0sec
> 16/04/19 08:49:52 INFO balancer.Balancer: dfs.balancer.movedWinWidth =
> 5400000 (default=5400000)
> 16/04/19 08:49:52 INFO balancer.Balancer: dfs.balancer.moverThreads = 1000
> (default=1000)
> 16/04/19 08:49:52 INFO balancer.Balancer: dfs.balancer.dispatcherThreads =
> 200 (default=200)
> 16/04/19 08:49:52 INFO balancer.Balancer:
> dfs.datanode.balance.max.concurrent.moves = 5 (default=5)
> 16/04/19 08:49:52 INFO balancer.Balancer: dfs.balancer.getBlocks.size =
> 2147483648 (default=2147483648)
> 16/04/19 08:49:52 INFO balancer.Balancer:
> dfs.balancer.getBlocks.min-block-size = 10485760 (default=10485760)
> 16/04/19 08:49:52 INFO block.BlockTokenSecretManager: Setting block keys
> 16/04/19 08:49:52 INFO balancer.Balancer: dfs.balancer.max-size-to-move =
> 10737418240 (default=10737418240)
> Apr 19, 2016 8:49:52 AM Balancing took 1.408 seconds
> 16/04/19 08:49:52 ERROR balancer.Balancer: Exiting balancer due an exception
> java.lang.NumberFormatException: For input string: "128m"
> at
> java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
> at java.lang.Long.parseLong(Long.java:589)
> at java.lang.Long.parseLong(Long.java:631)
> at
> org.apache.hadoop.conf.Configuration.getLong(Configuration.java:1311)
> at
> org.apache.hadoop.hdfs.server.balancer.Balancer.getLong(Balancer.java:221)
> at
> org.apache.hadoop.hdfs.server.balancer.Balancer.<init>(Balancer.java:281)
> at
> org.apache.hadoop.hdfs.server.balancer.Balancer.run(Balancer.java:660)
> at
> org.apache.hadoop.hdfs.server.balancer.Balancer$Cli.run(Balancer.java:774)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
> at
> org.apache.hadoop.hdfs.server.balancer.Balancer.main(Balancer.java:903)
> {code}
> However, the workaround for this is to run {{hdfs balancer}} with passing
> numeric value for {{dfs.blocksize}} or change your {{hdfs-site.xml}}.
> {code}
> hdfs balancer -Ddfs.blocksize=134217728
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
