[
https://issues.apache.org/jira/browse/HBASE-30231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Li Zhexi updated HBASE-30231:
-----------------------------
Description:
when called
org.apache.hadoop.hbase.tool.LoadIncrementalHFiles#doBulkLoad(Path hfofDir,
final Admin admin, Table table, RegionLocator regionLocator) to load HFiles,
with hadoop jar command.
Showed the Error message:
[LoadIncrementalHFiles-19] INFO org.apache.hadoop.io.compress.CodecPool - Got
brand-new compressor [.snappy]
[LoadIncrementalHFiles-19] INFO org.apache.hadoop.io.compress.CodecPool - Got
brand-new compressor [.snappy]
[LoadIncrementalHFiles-19] INFO org.apache.hadoop.metrics2.impl.MetricsConfig -
Loaded properties from hadoop-metrics2.properties
[LoadIncrementalHFiles-19] INFO
org.apache.hadoop.metrics2.impl.MetricsSystemImpl - Scheduled Metric snapshot
period at 10 second(s).
[LoadIncrementalHFiles-19] INFO
org.apache.hadoop.metrics2.impl.MetricsSystemImpl - HBase metrics system started
[LoadIncrementalHFiles-19] INFO
org.apache.hadoop.hbase.metrics.MetricRegistries - Loaded MetricRegistries
class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl
ERROR tool.LoadIncrementalHFiles: Unexpected execution exception during
splitting
java.util.concurrent.ExecutionException:
org.apache.hadoop.metrics2.MetricsException: Metrics source RegionServer,sub=IO
already exists!
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:192)
at
org.apache.hadoop.hbase.tool.LoadIncrementalHFiles.groupOrSplitPhase(LoadIncrementalHFiles.java:662)
at
org.apache.hadoop.hbase.tool.LoadIncrementalHFiles.performBulkLoad(LoadIncrementalHFiles.java:454)
at
org.apache.hadoop.hbase.tool.LoadIncrementalHFiles.doBulkLoad(LoadIncrementalHFiles.java:371)
at
org.apache.hadoop.hbase.tool.LoadIncrementalHFiles.doBulkLoad(LoadIncrementalHFiles.java:285)
at com.xx.xx.XXLoadFile.main(XXLoadFile.java:113)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:332)
at org.apache.hadoop.util.RunJar.main(RunJar.java:247)
Caused by: org.apache.hadoop.metrics2.MetricsException: Metrics source
RegionServer,sub-IO already exists!
at
org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.newSourceName(DefaultMetricsSystem.java:152)
at
org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.sourceName(DefaultMetricsSystem.java:125)
at
org.apache.hadoop.metrics2.impl.MetricsSystemImpl.register(MetricsSystemImpl.java:229)
at
org.apache.hadoop.hbase.metrics.BaseSourceImpl.<init>(BaseSourceImpl.java:115)
at org.apache.hadoop.hbase.io.MetricsIOSourceImpl.<init>(IOSourceImpl.java:44)
at org.apache.hadoop.hbase.io.MetricsIOSourceImpl.<init>(IOSourceImpl.java:39)
at
org.apache.hadoop.hbase.regionserver.MetricsRegionServerSourceFactoryImpl.createIO(MetricsRegionServerSourceFactoryImpl.java:99)
at org.apache.hadoop.hbase.io.MetricsIO.<init>(MetricsIO.java:36)
at org.apache.hadoop.hbase.io.MetricsIO.getInstance(MetricsIO.java:52)
at org.apache.hadoop.hbase.io.hfile.HFile.updateWriteLatency(HFile.java:204)
at
org.apache.hadoop.hbase.io.hfile.HFileBlock$Writer.finishBlockAndWriteHeaderAndData(HFileBlock.java:1047)
at
org.apache.hadoop.hbase.io.hfile.HFileBlock$Writer.writeHeaderAndData(HFileBlock.java:1032)
at
org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexWriter.writeIndexBlocks(HFileBlockIndex.java:1069)
at
org.apache.hadoop.hbase.io.hfile.HFileWriterImpl.close(HFileWriterImpl.java:646)
at
org.apache.hadoop.hbase.regionserver.StoreFileWriter$SingleStoreFileWriter.close(StoreFileWriter.java:786)
at
org.apache.hadoop.hbase.regionserver.StoreFileWriter$SingleStoreFileWriter.access$1300(StoreFileWriter.java:493)
at
org.apache.hadoop.hbase.regionserver.StoreFileWriter.close(StoreFileWriter.java:293)
at
org.apache.hadoop.hbase.tools.LoadIncrementalHFiles.copyHFileHalf(LoadIncrementalHFiles.java:1209)
at
org.apache.hadoop.hbase.tools.LoadIncrementalHFiles.splitStoreFile(LoadIncrementalHFiles.java:1157)
at
org.apache.hadoop.hbase.tools.LoadIncrementalHFiles.splitStoreFile(LoadIncrementalHFiles.java:702)
at
org.apache.hadoop.hbase.tools.LoadIncrementalHFiles.groupOrSplit(LoadIncrementalHFiles.java:825)
at
org.apache.hadoop.hbase.tools.LoadIncrementalHFiles$2.call(LoadIncrementalHFiles.java:652)
at
org.apache.hadoop.hbase.tools.LoadIncrementalHFiles$2.call(LoadIncrementalHFiles.java:648)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
The probability of this issue occurring is very low, it only happened once, and
it succeeded after a retry. It cannot be reproduced.
Seems HFile.updateWriteLatency 100% initilize MetricsIO, does it necessary for
client work？
was:
when called
org.apache.hadoop.hbase.tool.LoadIncrementalHFiles#doBulkLoad(Path hfofDir,
final Admin admin, Table table, RegionLocator regionLocator) to load HFiles,
with hadoop jar command.
Showed the Error message:
[LoadIncrementalHFiles-19] INFO org.apache.hadoop.io.compress.CodecPool - Got
brand-new compressor [.snappy]
[LoadIncrementalHFiles-19] INFO org.apache.hadoop.io.compress.CodecPool - Got
brand-new compressor [.snappy]
[LoadIncrementalHFiles-19] INFO org.apache.hadoop.metrics2.impl.MetricsConfig -
Loaded properties from hadoop-metrics2.properties
[LoadIncrementalHFiles-19] INFO
org.apache.hadoop.metrics2.impl.MetricsSystemImpl - Scheduled Metric snapshot
period at 10 second(s).
[LoadIncrementalHFiles-19] INFO
org.apache.hadoop.metrics2.impl.MetricsSystemImpl - HBase metrics system started
[LoadIncrementalHFiles-19] INFO
org.apache.hadoop.hbase.metrics.MetricRegistries - Loaded MetricRegistries
class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl
ERROR tool.LoadIncrementalHFiles: Unexpected execution exception during
splitting
java.util.concurrent.ExecutionException:
org.apache.hadoop.metrics2.MetricsException: Metrics source RegionServer,sub=IO
already exists!
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:192)
at
org.apache.hadoop.hbase.tool.LoadIncrementalHFiles.groupOrSplitPhase(LoadIncrementalHFiles.java:662)
at
org.apache.hadoop.hbase.tool.LoadIncrementalHFiles.performBulkLoad(LoadIncrementalHFiles.java:454)
at
org.apache.hadoop.hbase.tool.LoadIncrementalHFiles.doBulkLoad(LoadIncrementalHFiles.java:371)
at
org.apache.hadoop.hbase.tool.LoadIncrementalHFiles.doBulkLoad(LoadIncrementalHFiles.java:285)
at com.xx.xx.XXLoadFile.main(XXLoadFile.java:113)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:332)
at org.apache.hadoop.util.RunJar.main(RunJar.java:247)
Caused by: org.apache.hadoop.metrics2.MetricsException: Metrics source
RegionServer,sub-IO already exists!
at
org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.newSourceName(DefaultMetricsSystem.java:152)
at
org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.sourceName(DefaultMetricsSystem.java:125)
at
org.apache.hadoop.metrics2.impl.MetricsSystemImpl.register(MetricsSystemImpl.java:229)
at
org.apache.hadoop.hbase.metrics.BaseSourceImpl.<init>(BaseSourceImpl.java:115)
at org.apache.hadoop.hbase.io.MetricsIOSourceImpl.<init>(IOSourceImpl.java:44)
at org.apache.hadoop.hbase.io.MetricsIOSourceImpl.<init>(IOSourceImpl.java:39)
at
org.apache.hadoop.hbase.regionserver.MetricsRegionServerSourceFactoryImpl.createIO(MetricsRegionServerSourceFactoryImpl.java:99)
at org.apache.hadoop.hbase.io.MetricsIO.<init>(MetricsIO.java:36)
at org.apache.hadoop.hbase.io.MetricsIO.getInstance(MetricsIO.java:52)
at org.apache.hadoop.hbase.io.hfile.HFile.updateWriteLatency(HFile.java:204)
at
org.apache.hadoop.hbase.io.hfile.HFileBlock$Writer.finishBlockAndWriteHeaderAndData(HFileBlock.java:1047)
at
org.apache.hadoop.hbase.io.hfile.HFileBlock$Writer.writeHeaderAndData(HFileBlock.java:1032)
at
org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexWriter.writeIndexBlocks(HFileBlockIndex.java:1069)
at
org.apache.hadoop.hbase.io.hfile.HFileWriterImpl.close(HFileWriterImpl.java:646)
at
org.apache.hadoop.hbase.regionserver.StoreFileWriter$SingleStoreFileWriter.close(StoreFileWriter.java:786)
at
org.apache.hadoop.hbase.regionserver.StoreFileWriter$SingleStoreFileWriter.access$1300(StoreFileWriter.java:493)
at
org.apache.hadoop.hbase.regionserver.StoreFileWriter.close(StoreFileWriter.java:293)
at
org.apache.hadoop.hbase.tools.LoadIncrementalHFiles.copyHFileHalf(LoadIncrementalHFiles.java:1209)
at
org.apache.hadoop.hbase.tools.LoadIncrementalHFiles.splitStoreFile(LoadIncrementalHFiles.java:1157)
at
org.apache.hadoop.hbase.tools.LoadIncrementalHFiles.splitStoreFile(LoadIncrementalHFiles.java:702)
at
org.apache.hadoop.hbase.tools.LoadIncrementalHFiles.groupOrSplit(LoadIncrementalHFiles.java:825)
at
org.apache.hadoop.hbase.tools.LoadIncrementalHFiles$2.call(LoadIncrementalHFiles.java:652)
at
org.apache.hadoop.hbase.tools.LoadIncrementalHFiles$2.call(LoadIncrementalHFiles.java:648)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
The probability of this issue occurring is very low, it only happened once, and
it succeeded after a retry. It cannot be reproduced.
> LoadIncrementalHFiles failed for MetricsException: Metrics source
> RegionServer,Sub=IO already exists
> ----------------------------------------------------------------------------------------------------
>
> Key: HBASE-30231
> URL: https://issues.apache.org/jira/browse/HBASE-30231
> Project: HBase
> Issue Type: Bug
> Components: metrics
> Affects Versions: 2.6.1
> Reporter: Li Zhexi
> Assignee: Li Zhexi
> Priority: Major
>
> when called
> org.apache.hadoop.hbase.tool.LoadIncrementalHFiles#doBulkLoad(Path hfofDir,
> final Admin admin, Table table, RegionLocator regionLocator) to load HFiles,
> with hadoop jar command.
> Showed the Error message:
> [LoadIncrementalHFiles-19] INFO org.apache.hadoop.io.compress.CodecPool - Got
> brand-new compressor [.snappy]
> [LoadIncrementalHFiles-19] INFO org.apache.hadoop.io.compress.CodecPool - Got
> brand-new compressor [.snappy]
> [LoadIncrementalHFiles-19] INFO org.apache.hadoop.metrics2.impl.MetricsConfig
> - Loaded properties from hadoop-metrics2.properties
> [LoadIncrementalHFiles-19] INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl - Scheduled Metric snapshot
> period at 10 second(s).
> [LoadIncrementalHFiles-19] INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl - HBase metrics system
> started
> [LoadIncrementalHFiles-19] INFO
> org.apache.hadoop.hbase.metrics.MetricRegistries - Loaded MetricRegistries
> class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl
> ERROR tool.LoadIncrementalHFiles: Unexpected execution exception during
> splitting
> java.util.concurrent.ExecutionException:
> org.apache.hadoop.metrics2.MetricsException: Metrics source
> RegionServer,sub=IO already exists!
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
> at
> org.apache.hadoop.hbase.tool.LoadIncrementalHFiles.groupOrSplitPhase(LoadIncrementalHFiles.java:662)
> at
> org.apache.hadoop.hbase.tool.LoadIncrementalHFiles.performBulkLoad(LoadIncrementalHFiles.java:454)
> at
> org.apache.hadoop.hbase.tool.LoadIncrementalHFiles.doBulkLoad(LoadIncrementalHFiles.java:371)
> at
> org.apache.hadoop.hbase.tool.LoadIncrementalHFiles.doBulkLoad(LoadIncrementalHFiles.java:285)
> at com.xx.xx.XXLoadFile.main(XXLoadFile.java:113)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:332)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:247)
> Caused by: org.apache.hadoop.metrics2.MetricsException: Metrics source
> RegionServer,sub-IO already exists!
> at
> org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.newSourceName(DefaultMetricsSystem.java:152)
> at
> org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.sourceName(DefaultMetricsSystem.java:125)
> at
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl.register(MetricsSystemImpl.java:229)
> at
> org.apache.hadoop.hbase.metrics.BaseSourceImpl.<init>(BaseSourceImpl.java:115)
> at
> org.apache.hadoop.hbase.io.MetricsIOSourceImpl.<init>(IOSourceImpl.java:44)
> at
> org.apache.hadoop.hbase.io.MetricsIOSourceImpl.<init>(IOSourceImpl.java:39)
> at
> org.apache.hadoop.hbase.regionserver.MetricsRegionServerSourceFactoryImpl.createIO(MetricsRegionServerSourceFactoryImpl.java:99)
> at org.apache.hadoop.hbase.io.MetricsIO.<init>(MetricsIO.java:36)
> at org.apache.hadoop.hbase.io.MetricsIO.getInstance(MetricsIO.java:52)
> at org.apache.hadoop.hbase.io.hfile.HFile.updateWriteLatency(HFile.java:204)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock$Writer.finishBlockAndWriteHeaderAndData(HFileBlock.java:1047)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock$Writer.writeHeaderAndData(HFileBlock.java:1032)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexWriter.writeIndexBlocks(HFileBlockIndex.java:1069)
> at
> org.apache.hadoop.hbase.io.hfile.HFileWriterImpl.close(HFileWriterImpl.java:646)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileWriter$SingleStoreFileWriter.close(StoreFileWriter.java:786)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileWriter$SingleStoreFileWriter.access$1300(StoreFileWriter.java:493)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileWriter.close(StoreFileWriter.java:293)
> at
> org.apache.hadoop.hbase.tools.LoadIncrementalHFiles.copyHFileHalf(LoadIncrementalHFiles.java:1209)
> at
> org.apache.hadoop.hbase.tools.LoadIncrementalHFiles.splitStoreFile(LoadIncrementalHFiles.java:1157)
> at
> org.apache.hadoop.hbase.tools.LoadIncrementalHFiles.splitStoreFile(LoadIncrementalHFiles.java:702)
> at
> org.apache.hadoop.hbase.tools.LoadIncrementalHFiles.groupOrSplit(LoadIncrementalHFiles.java:825)
> at
> org.apache.hadoop.hbase.tools.LoadIncrementalHFiles$2.call(LoadIncrementalHFiles.java:652)
> at
> org.apache.hadoop.hbase.tools.LoadIncrementalHFiles$2.call(LoadIncrementalHFiles.java:648)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:750)
>
> The probability of this issue occurring is very low, it only happened once,
> and it succeeded after a retry. It cannot be reproduced.
> Seems HFile.updateWriteLatency 100% initilize MetricsIO, does it necessary
> for client work？
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
