[
https://issues.apache.org/jira/browse/HBASE-30231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18137333#comment-18137333
]
Andor Molnar commented on HBASE-30231:
--------------------------------------
Claude created fix2 with a different approach.
----------------------------
I'll make IO metrics opt-in: they stay a no-op until the RegionServer turns
them on, so clients and tools never register the source. First, finding where
the RegionServer sets up its metrics.
Nothing else depends on the IO metrics. I'll build this on top of fix 1,
turning the metrics on in the {{HRegionServer}} constructor (which also covers
{{{}HMaster{}}}).
Now tests for the enable switch.
{{testEnable}} and the existing {{testMetrics}} would both register the same
source in one JVM. I'll check whether HBase's tests already tolerate that.
HBase's metrics tests call {{{}HELPER.init(){}}}, which tells Hadoop to allow
repeated registration. I'll do the same, which also lets {{testEnable}} check
that it gets the real source.
Re-running the stand-in compile check with the new tests.
All five new tests pass with the stand-ins, and the enabled flag resets cleanly
between tests. Committing fix 2 on top of fix 1.
I've written the patch for fix 2: HFile IO metrics are now off unless a
RegionServer or Master turns them on, so clients and tools never start the
metrics system. Like fix 1, I couldn't run a real Maven build here, so I
compiled the changed files with stand-ins for the external classes and the new
tests passed that way. Please run {{TestMetricsIO}} and a quick mini-cluster
test before posting.
*What changes*
* *{{MetricsIO}}* now has {{enable()}} and {{{}isEnabled(){}}}. Until
{{enable()}} is called, {{getInstance()}} returns a shared instance that
ignores every update and registers nothing.
* *{{HRegionServer}}* calls {{MetricsIO.enable()}} in its constructor.
{{HMaster}} extends {{{}HRegionServer{}}}, so the master is covered too.
* *Server behaviour is unchanged.* The real metrics source is still created on
first use, as HBASE-27966 requires, so RegionServer and Master report the same
metrics as today.
* {*}Bulk-load clients{*}, MapReduce tasks writing HFiles and tools like
HFilePrettyPrinter no longer start the metrics system or register a
"RegionServer" source.
* *Tests:* two new ones check that metrics are off by default and that
{{enable()}} switches to the real source. The test class now calls
{{{}HELPER.init(){}}}, as other HBase metrics tests do, so registering the same
source twice in one test run is allowed. A test-only {{resetForTesting()}}
clears the static state between tests.
Fix 1 still matters after this: a RegionServer could still hit the original
"already exists" failure, and fix 1 stops that from breaking reads and writes.
> LoadIncrementalHFiles failed for MetricsException: Metrics source
> RegionServer,Sub=IO already exists
> ----------------------------------------------------------------------------------------------------
>
> Key: HBASE-30231
> URL: https://issues.apache.org/jira/browse/HBASE-30231
> Project: HBase
> Issue Type: Bug
> Components: metrics
> Affects Versions: 2.6.1
> Reporter: Li Zhexi
> Assignee: Li Zhexi
> Priority: Major
> Attachments: HBASE-30231-fix2.branch-2.6.patch,
> HBASE-30231.branch-2.6.patch
>
>
> when called
> org.apache.hadoop.hbase.tool.LoadIncrementalHFiles#doBulkLoad(Path hfofDir,
> final Admin admin, Table table, RegionLocator regionLocator) to load HFiles,
> with hadoop jar command.
> Showed the Error message:
> [LoadIncrementalHFiles-19] INFO org.apache.hadoop.io.compress.CodecPool - Got
> brand-new compressor [.snappy]
> [LoadIncrementalHFiles-19] INFO org.apache.hadoop.io.compress.CodecPool - Got
> brand-new compressor [.snappy]
> [LoadIncrementalHFiles-19] INFO org.apache.hadoop.metrics2.impl.MetricsConfig
> - Loaded properties from hadoop-metrics2.properties
> [LoadIncrementalHFiles-19] INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl - Scheduled Metric snapshot
> period at 10 second(s).
> [LoadIncrementalHFiles-19] INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl - HBase metrics system
> started
> [LoadIncrementalHFiles-19] INFO
> org.apache.hadoop.hbase.metrics.MetricRegistries - Loaded MetricRegistries
> class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl
> ERROR tool.LoadIncrementalHFiles: Unexpected execution exception during
> splitting
> java.util.concurrent.ExecutionException:
> org.apache.hadoop.metrics2.MetricsException: Metrics source
> RegionServer,sub=IO already exists!
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
> at
> org.apache.hadoop.hbase.tool.LoadIncrementalHFiles.groupOrSplitPhase(LoadIncrementalHFiles.java:662)
> at
> org.apache.hadoop.hbase.tool.LoadIncrementalHFiles.performBulkLoad(LoadIncrementalHFiles.java:454)
> at
> org.apache.hadoop.hbase.tool.LoadIncrementalHFiles.doBulkLoad(LoadIncrementalHFiles.java:371)
> at
> org.apache.hadoop.hbase.tool.LoadIncrementalHFiles.doBulkLoad(LoadIncrementalHFiles.java:285)
> at com.xx.xx.XXLoadFile.main(XXLoadFile.java:113)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:332)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:247)
> Caused by: org.apache.hadoop.metrics2.MetricsException: Metrics source
> RegionServer,sub-IO already exists!
> at
> org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.newSourceName(DefaultMetricsSystem.java:152)
> at
> org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.sourceName(DefaultMetricsSystem.java:125)
> at
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl.register(MetricsSystemImpl.java:229)
> at
> org.apache.hadoop.hbase.metrics.BaseSourceImpl.<init>(BaseSourceImpl.java:115)
> at
> org.apache.hadoop.hbase.io.MetricsIOSourceImpl.<init>(IOSourceImpl.java:44)
> at
> org.apache.hadoop.hbase.io.MetricsIOSourceImpl.<init>(IOSourceImpl.java:39)
> at
> org.apache.hadoop.hbase.regionserver.MetricsRegionServerSourceFactoryImpl.createIO(MetricsRegionServerSourceFactoryImpl.java:99)
> at org.apache.hadoop.hbase.io.MetricsIO.<init>(MetricsIO.java:36)
> at org.apache.hadoop.hbase.io.MetricsIO.getInstance(MetricsIO.java:52)
> at org.apache.hadoop.hbase.io.hfile.HFile.updateWriteLatency(HFile.java:204)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock$Writer.finishBlockAndWriteHeaderAndData(HFileBlock.java:1047)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock$Writer.writeHeaderAndData(HFileBlock.java:1032)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexWriter.writeIndexBlocks(HFileBlockIndex.java:1069)
> at
> org.apache.hadoop.hbase.io.hfile.HFileWriterImpl.close(HFileWriterImpl.java:646)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileWriter$SingleStoreFileWriter.close(StoreFileWriter.java:786)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileWriter$SingleStoreFileWriter.access$1300(StoreFileWriter.java:493)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileWriter.close(StoreFileWriter.java:293)
> at
> org.apache.hadoop.hbase.tools.LoadIncrementalHFiles.copyHFileHalf(LoadIncrementalHFiles.java:1209)
> at
> org.apache.hadoop.hbase.tools.LoadIncrementalHFiles.splitStoreFile(LoadIncrementalHFiles.java:1157)
> at
> org.apache.hadoop.hbase.tools.LoadIncrementalHFiles.splitStoreFile(LoadIncrementalHFiles.java:702)
> at
> org.apache.hadoop.hbase.tools.LoadIncrementalHFiles.groupOrSplit(LoadIncrementalHFiles.java:825)
> at
> org.apache.hadoop.hbase.tools.LoadIncrementalHFiles$2.call(LoadIncrementalHFiles.java:652)
> at
> org.apache.hadoop.hbase.tools.LoadIncrementalHFiles$2.call(LoadIncrementalHFiles.java:648)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:750)
>
> The probability of this issue occurring is very low, it only happened once,
> and it succeeded after a retry. It cannot be reproduced.
> Seems HFile.updateWriteLatency 100% initilize MetricsIO, does it necessary
> for client work？
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
