[
https://issues.apache.org/jira/browse/HDFS-13692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yiqun Lin updated HDFS-13692:
-----------------------------
Description:
StorageInfoDefragmenter floods log when compacting StorageInfo TreeSet. In
{{StorageInfoDefragmenter#scanAndCompactStorages}}, it prints for all the
StorageInfo under each DN. If there are 1k nodes in cluster, and each node has
10 data dir configured, it will print 10k lines every compact interval time (10
mins). The log looks large, we could switch log level from INFO to DEBUG in
{{StorageInfoDefragmenter#scanAndCompactStorages}}.
{noformat}
2018-06-19 10:18:48,849 INFO [StorageInfoMonitor]
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: StorageInfo TreeSet
fill ratio DS-329bd988-a558-43a6-b31c-9142548b0179 : 0.8222276264591439
2018-06-19 10:18:48,849 INFO [StorageInfoMonitor]
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: StorageInfo TreeSet
fill ratio DS-b5505847-1389-4a80-b9d8-876172a83897 : 0.933351976137211
2018-06-19 10:18:48,849 INFO [StorageInfoMonitor]
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: StorageInfo TreeSet
fill ratio DS-ca164b2f-0a2c-4b26-8e99-f0ece0909997 : 0.9330040998881849
2018-06-19 10:18:48,849 INFO [StorageInfoMonitor]
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: StorageInfo TreeSet
fill ratio DS-89b912ba-339b-45e3-b981-541b22690ccb : 0.9314626719970249
2018-06-19 10:18:48,849 INFO [StorageInfoMonitor]
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: StorageInfo TreeSet
fill ratio DS-89c0377b-a49c-4288-9304-e104d98de5bd : 0.9309580852251582
2018-06-19 10:18:48,849 INFO [StorageInfoMonitor]
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: StorageInfo TreeSet
fill ratio DS-5ffad4d2-168a-446d-a92e-ef46a82f26f8 : 0.8938870614035088
2018-06-19 10:18:48,849 INFO [StorageInfoMonitor]
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: StorageInfo TreeSet
fill ratio DS-eecbbd34-10f4-4647-8710-0f5963da3aaa : 0.8963103205353998
2018-06-19 10:18:48,849 INFO [StorageInfoMonitor]
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: StorageInfo TreeSet
fill ratio DS-7aafa122-433f-49c8-bf00-11bcdd8ce048 : 0.8950508004926109
2018-06-19 10:18:48,849 INFO [StorageInfoMonitor]
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: StorageInfo TreeSet
fill ratio DS-eb9ba675-9c23-40a1-9241-c314dc0e2867 : 0.8947356866877415
{noformat}
was:
StorageInfoDefragmenter floods log when compacting StorageInfoTreeset. In
{{StorageInfoDefragmenter#scanAndCompactStorages}}, it prints for all the
StorageInfo under each DN. If there are 1k nodes in cluster, and each node has
10 data dir configured, it will print 10k lines every compact interval time (10
mins). The log looks large, we could switch log level from INFO to DEBUG in
{{StorageInfoDefragmenter#scanAndCompactStorages}}.
{noformat}
2018-06-19 10:18:48,849 INFO [StorageInfoMonitor]
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: StorageInfo TreeSet
fill ratio DS-329bd988-a558-43a6-b31c-9142548b0179 : 0.8222276264591439
2018-06-19 10:18:48,849 INFO [StorageInfoMonitor]
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: StorageInfo TreeSet
fill ratio DS-b5505847-1389-4a80-b9d8-876172a83897 : 0.933351976137211
2018-06-19 10:18:48,849 INFO [StorageInfoMonitor]
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: StorageInfo TreeSet
fill ratio DS-ca164b2f-0a2c-4b26-8e99-f0ece0909997 : 0.9330040998881849
2018-06-19 10:18:48,849 INFO [StorageInfoMonitor]
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: StorageInfo TreeSet
fill ratio DS-89b912ba-339b-45e3-b981-541b22690ccb : 0.9314626719970249
2018-06-19 10:18:48,849 INFO [StorageInfoMonitor]
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: StorageInfo TreeSet
fill ratio DS-89c0377b-a49c-4288-9304-e104d98de5bd : 0.9309580852251582
2018-06-19 10:18:48,849 INFO [StorageInfoMonitor]
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: StorageInfo TreeSet
fill ratio DS-5ffad4d2-168a-446d-a92e-ef46a82f26f8 : 0.8938870614035088
2018-06-19 10:18:48,849 INFO [StorageInfoMonitor]
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: StorageInfo TreeSet
fill ratio DS-eecbbd34-10f4-4647-8710-0f5963da3aaa : 0.8963103205353998
2018-06-19 10:18:48,849 INFO [StorageInfoMonitor]
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: StorageInfo TreeSet
fill ratio DS-7aafa122-433f-49c8-bf00-11bcdd8ce048 : 0.8950508004926109
2018-06-19 10:18:48,849 INFO [StorageInfoMonitor]
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: StorageInfo TreeSet
fill ratio DS-eb9ba675-9c23-40a1-9241-c314dc0e2867 : 0.8947356866877415
{noformat}
> StorageInfoDefragmenter floods log when compacting StorageInfo TreeSet
> ----------------------------------------------------------------------
>
> Key: HDFS-13692
> URL: https://issues.apache.org/jira/browse/HDFS-13692
> Project: Hadoop HDFS
> Issue Type: Improvement
> Affects Versions: 3.1.0
> Reporter: Yiqun Lin
> Priority: Minor
>
> StorageInfoDefragmenter floods log when compacting StorageInfo TreeSet. In
> {{StorageInfoDefragmenter#scanAndCompactStorages}}, it prints for all the
> StorageInfo under each DN. If there are 1k nodes in cluster, and each node
> has 10 data dir configured, it will print 10k lines every compact interval
> time (10 mins). The log looks large, we could switch log level from INFO to
> DEBUG in {{StorageInfoDefragmenter#scanAndCompactStorages}}.
> {noformat}
> 2018-06-19 10:18:48,849 INFO [StorageInfoMonitor]
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: StorageInfo
> TreeSet fill ratio DS-329bd988-a558-43a6-b31c-9142548b0179 :
> 0.8222276264591439
> 2018-06-19 10:18:48,849 INFO [StorageInfoMonitor]
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: StorageInfo
> TreeSet fill ratio DS-b5505847-1389-4a80-b9d8-876172a83897 : 0.933351976137211
> 2018-06-19 10:18:48,849 INFO [StorageInfoMonitor]
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: StorageInfo
> TreeSet fill ratio DS-ca164b2f-0a2c-4b26-8e99-f0ece0909997 :
> 0.9330040998881849
> 2018-06-19 10:18:48,849 INFO [StorageInfoMonitor]
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: StorageInfo
> TreeSet fill ratio DS-89b912ba-339b-45e3-b981-541b22690ccb :
> 0.9314626719970249
> 2018-06-19 10:18:48,849 INFO [StorageInfoMonitor]
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: StorageInfo
> TreeSet fill ratio DS-89c0377b-a49c-4288-9304-e104d98de5bd :
> 0.9309580852251582
> 2018-06-19 10:18:48,849 INFO [StorageInfoMonitor]
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: StorageInfo
> TreeSet fill ratio DS-5ffad4d2-168a-446d-a92e-ef46a82f26f8 :
> 0.8938870614035088
> 2018-06-19 10:18:48,849 INFO [StorageInfoMonitor]
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: StorageInfo
> TreeSet fill ratio DS-eecbbd34-10f4-4647-8710-0f5963da3aaa :
> 0.8963103205353998
> 2018-06-19 10:18:48,849 INFO [StorageInfoMonitor]
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: StorageInfo
> TreeSet fill ratio DS-7aafa122-433f-49c8-bf00-11bcdd8ce048 :
> 0.8950508004926109
> 2018-06-19 10:18:48,849 INFO [StorageInfoMonitor]
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: StorageInfo
> TreeSet fill ratio DS-eb9ba675-9c23-40a1-9241-c314dc0e2867 :
> 0.8947356866877415
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
