Li Zhexi created HBASE-30231:
--------------------------------
Summary: LoadIncrementalHFiles failed for MetricsException:
Metrics source RegionServer,Sub=IO already exists
Key: HBASE-30231
URL: https://issues.apache.org/jira/browse/HBASE-30231
Project: HBase
Issue Type: Bug
Components: metrics
Affects Versions: 2.6.1
Reporter: Li Zhexi
Assignee: Li Zhexi
when called
org.apache.hadoop.hbase.tool.LoadIncrementalHFiles#doBulkLoad(Path hfofDir,
final Admin admin, Table table, RegionLocator regionLocator) to load HFiles,
with hadoop jar command.
Showed the Error message:
[LoadIncrementalHFiles-19] INFO org.apache.hadoop.io.compress.CodecPool - Got
brand-new compressor [.snappy]
[LoadIncrementalHFiles-19] INFO org.apache.hadoop.io.compress.CodecPool - Got
brand-new compressor [.snappy]
[LoadIncrementalHFiles-19] INFO org.apache.hadoop.metrics2.impl.MetricsConfig -
Loaded properties from hadoop-metrics2.properties
[LoadIncrementalHFiles-19] INFO
org.apache.hadoop.metrics2.impl.MetricsSystemImpl - Scheduled Metric snapshot
period at 10 second(s).
[LoadIncrementalHFiles-19] INFO
org.apache.hadoop.metrics2.impl.MetricsSystemImpl - HBase metrics system started
[LoadIncrementalHFiles-19] INFO
org.apache.hadoop.hbase.metrics.MetricRegistries - Loaded MetricRegistries
class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl
ERROR tool.LoadIncrementalHFiles: Unexpected execution exception during
splitting
java.util.concurrent.ExecutionException:
org.apache.hadoop.metrics2.MetricsException: Metrics source RegionServer,sub=IO
already exists!
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:192)
at
org.apache.hadoop.hbase.tool.LoadIncrementalHFiles.groupOrSplitPhase(LoadIncrementalHFiles.java:662)
at
org.apache.hadoop.hbase.tool.LoadIncrementalHFiles.performBulkLoad(LoadIncrementalHFiles.java:454)
at
org.apache.hadoop.hbase.tool.LoadIncrementalHFiles.doBulkLoad(LoadIncrementalHFiles.java:371)
at
org.apache.hadoop.hbase.tool.LoadIncrementalHFiles.doBulkLoad(LoadIncrementalHFiles.java:285)
at com.xx.xx.XXLoadFile.main(XXLoadFile.java:113)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:332)
at org.apache.hadoop.util.RunJar.main(RunJar.java:247)
Caused by: org.apache.hadoop.metrics2.MetricsException: Metrics source
RegionServer,sub-IO already exists!
at
org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.newSourceName(DefaultMetricsSystem.java:152)
at
org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.sourceName(DefaultMetricsSystem.java:125)
at
org.apache.hadoop.metrics2.impl.MetricsSystemImpl.register(MetricsSystemImpl.java:229)
at
org.apache.hadoop.hbase.metrics.BaseSourceImpl.<init>(BaseSourceImpl.java:115)
at org.apache.hadoop.hbase.io.MetricsIOSourceImpl.<init>(IOSourceImpl.java:44)
at org.apache.hadoop.hbase.io.MetricsIOSourceImpl.<init>(IOSourceImpl.java:39)
at
org.apache.hadoop.hbase.regionserver.MetricsRegionServerSourceFactoryImpl.createIO(MetricsRegionServerSourceFactoryImpl.java:99)
at org.apache.hadoop.hbase.io.MetricsIO.<init>(MetricsIO.java:36)
at org.apache.hadoop.hbase.io.MetricsIO.getInstance(MetricsIO.java:52)
at org.apache.hadoop.hbase.io.hfile.HFile.updateWriteLatency(HFile.java:204)
at
org.apache.hadoop.hbase.io.hfile.HFileBlock$Writer.finishBlockAndWriteHeaderAndData(HFileBlock.java:1047)
at
org.apache.hadoop.hbase.io.hfile.HFileBlock$Writer.writeHeaderAndData(HFileBlock.java:1032)
at
org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexWriter.writeIndexBlocks(HFileBlockIndex.java:1069)
at
org.apache.hadoop.hbase.io.hfile.HFileWriterImpl.close(HFileWriterImpl.java:646)
at
org.apache.hadoop.hbase.regionserver.StoreFileWriter$SingleStoreFileWriter.close(StoreFileWriter.java:786)
at
org.apache.hadoop.hbase.regionserver.StoreFileWriter$SingleStoreFileWriter.access$1300(StoreFileWriter.java:493)
at
org.apache.hadoop.hbase.regionserver.StoreFileWriter.close(StoreFileWriter.java:293)
at
org.apache.hadoop.hbase.tools.LoadIncrementalHFiles.copyHFileHalf(LoadIncrementalHFiles.java:1209)
at
org.apache.hadoop.hbase.tools.LoadIncrementalHFiles.splitStoreFile(LoadIncrementalHFiles.java:1157)
at
org.apache.hadoop.hbase.tools.LoadIncrementalHFiles.splitStoreFile(LoadIncrementalHFiles.java:702)
at
org.apache.hadoop.hbase.tools.LoadIncrementalHFiles.groupOrSplit(LoadIncrementalHFiles.java:825)
at
org.apache.hadoop.hbase.tools.LoadIncrementalHFiles$2.call(LoadIncrementalHFiles.java:652)
at
org.apache.hadoop.hbase.tools.LoadIncrementalHFiles$2.call(LoadIncrementalHFiles.java:648)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
The probability of this issue occurring is very low, it only happened once, and
it succeeded after a retry. It cannot be reproduced.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
