chaijunjie created HBASE-28541:
----------------------------------
Summary: RegionServer abort when max sequence id wrong
Key: HBASE-28541
URL: https://issues.apache.org/jira/browse/HBASE-28541
Project: HBase
Issue Type: Bug
Affects Versions: 2.2.3
Reporter: chaijunjie
When we disable a table, some region close failed because max sequence id is
less than the old max sequence id, then RS abort...Then the region close
success on another RS..
we just use bulkload...is it releated?
--------------------------------------------------------------------------------------------
2024-04-20 23:44:20,611 | INFO |
RpcServer.default.FPBQ.Fifo.handler=483,queue=33,port=21302 | Bulk-load file
hdfs://hacluster/hbase/staging/cdr__boss20240420__t20amapi48eb4ce0k0t80jatvnnhpm5rohaj26a5rdsp0s7kifokcdd32ko89u7n/info/4f43f6cf569643da8ecc140668fa726e
is on different filesystem than the destination store. Copying file over to
destination filesystem. |
org.apache.hadoop.hbase.regionserver.HRegionFileSystem.bulkLoadStoreFile(HRegionFileSystem.java:540)
2024-04-20 23:44:20,645 | INFO |
RpcServer.default.FPBQ.Fifo.handler=483,queue=33,port=21302 | Copied
hdfs://hacluster/hbase/staging/cdr__boss20240420__t20amapi48eb4ce0k0t80jatvnnhpm5rohaj26a5rdsp0s7kifokcdd32ko89u7n/info/4f43f6cf569643da8ecc140668fa726e
to temporary path on destination filesystem:
viewfs://ClusterX/hbase/data/default/boss20240420/dc30be37cbd0ddee2f49712c55fd36cd/.tmp/6f7e97f2e3994881a6e04b36759f495e
|
org.apache.hadoop.hbase.regionserver.HRegionFileSystem.bulkLoadStoreFile(HRegionFileSystem.java:544)
2024-04-20 23:44:20,648 | INFO |
RpcServer.default.FPBQ.Fifo.handler=483,queue=33,port=21302 | Loaded HFile
viewfs://ClusterX/hbase/data/default/boss20240420/dc30be37cbd0ddee2f49712c55fd36cd/.tmp/6f7e97f2e3994881a6e04b36759f495e
into dc30be37cbd0ddee2f49712c55fd36cd/info as
viewfs://ClusterX/hbase/data/default/boss20240420/dc30be37cbd0ddee2f49712c55fd36cd/info/3946dc15ea1c49b582a2ed175fe12ec3_SeqId_26_
- updating store file list. |
org.apache.hadoop.hbase.regionserver.HStore.bulkLoadHFile(HStore.java:908)
2024-04-20 23:44:20,664 | INFO |
RpcServer.default.FPBQ.Fifo.handler=483,queue=33,port=21302 | Loaded HFile
viewfs://ClusterX/hbase/data/default/boss20240420/dc30be37cbd0ddee2f49712c55fd36cd/info/3946dc15ea1c49b582a2ed175fe12ec3_SeqId_26_
into dc30be37cbd0ddee2f49712c55fd36cd/info |
org.apache.hadoop.hbase.regionserver.HStore.bulkLoadHFile(HStore.java:942)
2024-04-20 23:44:20,664 | INFO |
RpcServer.default.FPBQ.Fifo.handler=483,queue=33,port=21302 | Successfully
loaded
viewfs://ClusterX/hbase/data/default/boss20240420/dc30be37cbd0ddee2f49712c55fd36cd/.tmp/6f7e97f2e3994881a6e04b36759f495e
into dc30be37cbd0ddee2f49712c55fd36cd/info (new location:
viewfs://ClusterX/hbase/data/default/boss20240420/dc30be37cbd0ddee2f49712c55fd36cd/info/3946dc15ea1c49b582a2ed175fe12ec3_SeqId_26_)
| org.apache.hadoop.hbase.regionserver.HStore.bulkLoadHFile(HStore.java:914)
2024-04-20 23:44:23,592 | INFO |
RpcServer.default.FPBQ.Fifo.handler=483,queue=33,port=21302 | Validating hfile
at
viewfs://ClusterX/tenant/hw_cdr/dsttab/load_20240420232000/boss20240420/info/a97bb2bfcb5244279bcedbe20ec8322b
for inclusion in b3d102e51ca3b78a1078580ed8a002de/info |
org.apache.hadoop.hbase.regionserver.HStore.assertBulkLoadHFileOk(HStore.java:817)
2024-04-20 23:44:23,669 | INFO |
RpcServer.default.FPBQ.Fifo.handler=483,queue=33,port=21302 | Bulk-load file
hdfs://hacluster/hbase/staging/cdr__boss20240420__t20amapi48eb4ce0k0t80jatvnnhpm5rohaj26a5rdsp0s7kifokcdd32ko89u7n/info/a97bb2bfcb5244279bcedbe20ec8322b
is on different filesystem than the destination store. Copying file over to
destination filesystem. |
org.apache.hadoop.hbase.regionserver.HRegionFileSystem.bulkLoadStoreFile(HRegionFileSystem.java:540)
2024-04-20 23:44:23,743 | INFO |
RpcServer.default.FPBQ.Fifo.handler=483,queue=33,port=21302 | Copied
hdfs://hacluster/hbase/staging/cdr__boss20240420__t20amapi48eb4ce0k0t80jatvnnhpm5rohaj26a5rdsp0s7kifokcdd32ko89u7n/info/a97bb2bfcb5244279bcedbe20ec8322b
to temporary path on destination filesystem:
viewfs://ClusterX/hbase/data/default/boss20240420/b3d102e51ca3b78a1078580ed8a002de/.tmp/f40ec16d10ab4474ace7af41d9126aa0
|
org.apache.hadoop.hbase.regionserver.HRegionFileSystem.bulkLoadStoreFile(HRegionFileSystem.java:544)
2024-04-20 23:44:23,747 | INFO |
RpcServer.default.FPBQ.Fifo.handler=483,queue=33,port=21302 | Loaded HFile
viewfs://ClusterX/hbase/data/default/boss20240420/b3d102e51ca3b78a1078580ed8a002de/.tmp/f40ec16d10ab4474ace7af41d9126aa0
into b3d102e51ca3b78a1078580ed8a002de/info as
viewfs://ClusterX/hbase/data/default/boss20240420/b3d102e51ca3b78a1078580ed8a002de/info/7cfcf858a6b34bd1b9d47ccce5dde9e4_SeqId_41_
- updating store file list. |
org.apache.hadoop.hbase.regionserver.HStore.bulkLoadHFile(HStore.java:908)
2024-04-20 23:44:23,760 | INFO |
RpcServer.default.FPBQ.Fifo.handler=483,queue=33,port=21302 | Loaded HFile
viewfs://ClusterX/hbase/data/default/boss20240420/b3d102e51ca3b78a1078580ed8a002de/info/7cfcf858a6b34bd1b9d47ccce5dde9e4_SeqId_41_
into b3d102e51ca3b78a1078580ed8a002de/info |
org.apache.hadoop.hbase.regionserver.HStore.bulkLoadHFile(HStore.java:942)
2024-04-20 23:44:23,760 | INFO |
RpcServer.default.FPBQ.Fifo.handler=483,queue=33,port=21302 | Successfully
loaded
viewfs://ClusterX/hbase/data/default/boss20240420/b3d102e51ca3b78a1078580ed8a002de/.tmp/f40ec16d10ab4474ace7af41d9126aa0
into b3d102e51ca3b78a1078580ed8a002de/info (new location:
viewfs://ClusterX/hbase/data/default/boss20240420/b3d102e51ca3b78a1078580ed8a002de/info/7cfcf858a6b34bd1b9d47ccce5dde9e4_SeqId_41_)
| org.apache.hadoop.hbase.regionserver.HStore.bulkLoadHFile(HStore.java:914)
2024-04-20 23:44:25,443 | INFO |
RpcServer.default.FPBQ.Fifo.handler=483,queue=33,port=21302 | Validating hfile
at
viewfs://ClusterX/tenant/hw_cdr/dsttab/load_20240420232000/boss20240420/info/4dac6fe338ed45028dd4ac859d02c386
for inclusion in eec5518fe33d8c6b5f99f66e20bf782d/info |
org.apache.hadoop.hbase.regionserver.HStore.assertBulkLoadHFileOk(HStore.java:817)
2024-04-20 23:44:25,720 | INFO |
RpcServer.default.FPBQ.Fifo.handler=483,queue=33,port=21302 | Bulk-load file
hdfs://hacluster/hbase/staging/cdr__boss20240420__t20amapi48eb4ce0k0t80jatvnnhpm5rohaj26a5rdsp0s7kifokcdd32ko89u7n/info/4dac6fe338ed45028dd4ac859d02c386
is on different filesystem than the destination store. Copying file over to
destination filesystem. |
org.apache.hadoop.hbase.regionserver.HRegionFileSystem.bulkLoadStoreFile(HRegionFileSystem.java:540)
2024-04-20 23:44:25,798 | INFO |
RpcServer.default.FPBQ.Fifo.handler=483,queue=33,port=21302 | Copied
hdfs://hacluster/hbase/staging/cdr__boss20240420__t20amapi48eb4ce0k0t80jatvnnhpm5rohaj26a5rdsp0s7kifokcdd32ko89u7n/info/4dac6fe338ed45028dd4ac859d02c386
to temporary path on destination filesystem:
viewfs://ClusterX/hbase/data/default/boss20240420/eec5518fe33d8c6b5f99f66e20bf782d/.tmp/09cb80cf4b5d47b78ddf31eafce4f5f6
|
org.apache.hadoop.hbase.regionserver.HRegionFileSystem.bulkLoadStoreFile(HRegionFileSystem.java:544)
2024-04-20 23:44:25,802 | INFO |
RpcServer.default.FPBQ.Fifo.handler=483,queue=33,port=21302 | Loaded HFile
viewfs://ClusterX/hbase/data/default/boss20240420/eec5518fe33d8c6b5f99f66e20bf782d/.tmp/09cb80cf4b5d47b78ddf31eafce4f5f6
into eec5518fe33d8c6b5f99f66e20bf782d/info as
viewfs://ClusterX/hbase/data/default/boss20240420/eec5518fe33d8c6b5f99f66e20bf782d/info/6d674b96ccec40339a4a1f486a65c4a7_SeqId_20_
- updating store file list. |
org.apache.hadoop.hbase.regionserver.HStore.bulkLoadHFile(HStore.java:908)
2024-04-20 23:44:25,823 | INFO |
RpcServer.default.FPBQ.Fifo.handler=483,queue=33,port=21302 | Loaded HFile
viewfs://ClusterX/hbase/data/default/boss20240420/eec5518fe33d8c6b5f99f66e20bf782d/info/6d674b96ccec40339a4a1f486a65c4a7_SeqId_20_
into eec5518fe33d8c6b5f99f66e20bf782d/info |
org.apache.hadoop.hbase.regionserver.HStore.bulkLoadHFile(HStore.java:942)
2024-04-20 23:44:25,823 | INFO |
RpcServer.default.FPBQ.Fifo.handler=483,queue=33,port=21302 | Successfully
loaded
viewfs://ClusterX/hbase/data/default/boss20240420/eec5518fe33d8c6b5f99f66e20bf782d/.tmp/09cb80cf4b5d47b78ddf31eafce4f5f6
into eec5518fe33d8c6b5f99f66e20bf782d/info (new location:
viewfs://ClusterX/hbase/data/default/boss20240420/eec5518fe33d8c6b5f99f66e20bf782d/info/6d674b96ccec40339a4a1f486a65c4a7_SeqId_20_)
| org.apache.hadoop.hbase.regionserver.HStore.bulkLoadHFile(HStore.java:914)
2024-04-20 23:44:28,837 | INFO |
RpcServer.default.FPBQ.Fifo.handler=483,queue=33,port=21302 | Validating hfile
at
viewfs://ClusterX/tenant/hw_cdr/dsttab/load_20240420232000/boss20240420/info/ef44caa4e05440569f76a05d351e4a62
for inclusion in 21d5667a8bcd59088ae865f5d9660283/info |
org.apache.hadoop.hbase.regionserver.HStore.assertBulkLoadHFileOk(HStore.java:817)
2024-04-20 23:44:28,950 | INFO |
RpcServer.default.FPBQ.Fifo.handler=483,queue=33,port=21302 | Bulk-load file
hdfs://hacluster/hbase/staging/cdr__boss20240420__t20amapi48eb4ce0k0t80jatvnnhpm5rohaj26a5rdsp0s7kifokcdd32ko89u7n/info/ef44caa4e05440569f76a05d351e4a62
is on different filesystem than the destination store. Copying file over to
destination filesystem. |
org.apache.hadoop.hbase.regionserver.HRegionFileSystem.bulkLoadStoreFile(HRegionFileSystem.java:540)
2024-04-20 23:44:29,007 | INFO |
RpcServer.default.FPBQ.Fifo.handler=483,queue=33,port=21302 | Copied
hdfs://hacluster/hbase/staging/cdr__boss20240420__t20amapi48eb4ce0k0t80jatvnnhpm5rohaj26a5rdsp0s7kifokcdd32ko89u7n/info/ef44caa4e05440569f76a05d351e4a62
to temporary path on destination filesystem:
viewfs://ClusterX/hbase/data/default/boss20240420/21d5667a8bcd59088ae865f5d9660283/.tmp/1be6287d04084bf48afd8eb0e0e63a6e
|
org.apache.hadoop.hbase.regionserver.HRegionFileSystem.bulkLoadStoreFile(HRegionFileSystem.java:544)
2024-04-20 23:44:29,010 | INFO |
RpcServer.default.FPBQ.Fifo.handler=483,queue=33,port=21302 | Loaded HFile
viewfs://ClusterX/hbase/data/default/boss20240420/21d5667a8bcd59088ae865f5d9660283/.tmp/1be6287d04084bf48afd8eb0e0e63a6e
into 21d5667a8bcd59088ae865f5d9660283/info as
viewfs://ClusterX/hbase/data/default/boss20240420/21d5667a8bcd59088ae865f5d9660283/info/fb872c3f89e543c6b54fcb54898112d4_SeqId_16_
- updating store file list. |
org.apache.hadoop.hbase.regionserver.HStore.bulkLoadHFile(HStore.java:908)
2024-04-20 23:44:29,063 | INFO |
RpcServer.default.FPBQ.Fifo.handler=415,queue=15,port=21302 | Validating hfile
at
viewfs://ClusterX/tenant/hw_cdr/dsttab/load_20240420232000/boss20240420/info/6fdde06815624c5095d39e7b98cc81c7
for inclusion in 0bf7d39f1708c5e50444c810f60d3f9f/info |
org.apache.hadoop.hbase.regionserver.HStore.assertBulkLoadHFileOk(HStore.java:817)
2024-04-20 23:44:29,066 | INFO |
RpcServer.default.FPBQ.Fifo.handler=483,queue=33,port=21302 | Loaded HFile
viewfs://ClusterX/hbase/data/default/boss20240420/21d5667a8bcd59088ae865f5d9660283/info/fb872c3f89e543c6b54fcb54898112d4_SeqId_16_
into 21d5667a8bcd59088ae865f5d9660283/info |
org.apache.hadoop.hbase.regionserver.HStore.bulkLoadHFile(HStore.java:942)
2024-04-20 23:44:29,066 | INFO |
RpcServer.default.FPBQ.Fifo.handler=483,queue=33,port=21302 | Successfully
loaded
viewfs://ClusterX/hbase/data/default/boss20240420/21d5667a8bcd59088ae865f5d9660283/.tmp/1be6287d04084bf48afd8eb0e0e63a6e
into 21d5667a8bcd59088ae865f5d9660283/info (new location:
viewfs://ClusterX/hbase/data/default/boss20240420/21d5667a8bcd59088ae865f5d9660283/info/fb872c3f89e543c6b54fcb54898112d4_SeqId_16_)
| org.apache.hadoop.hbase.regionserver.HStore.bulkLoadHFile(HStore.java:914)
2024-04-20 23:44:29,116 | INFO |
RpcServer.default.FPBQ.Fifo.handler=415,queue=15,port=21302 | Bulk-load file
hdfs://hacluster/hbase/staging/cdr__boss20240420__t20amapi48eb4ce0k0t80jatvnnhpm5rohaj26a5rdsp0s7kifokcdd32ko89u7n/info/6fdde06815624c5095d39e7b98cc81c7
is on different filesystem than the destination store. Copying file over to
destination filesystem. |
org.apache.hadoop.hbase.regionserver.HRegionFileSystem.bulkLoadStoreFile(HRegionFileSystem.java:540)
2024-04-20 23:44:29,182 | INFO |
RpcServer.default.FPBQ.Fifo.handler=415,queue=15,port=21302 | Copied
hdfs://hacluster/hbase/staging/cdr__boss20240420__t20amapi48eb4ce0k0t80jatvnnhpm5rohaj26a5rdsp0s7kifokcdd32ko89u7n/info/6fdde06815624c5095d39e7b98cc81c7
to temporary path on destination filesystem:
viewfs://ClusterX/hbase/data/default/boss20240420/0bf7d39f1708c5e50444c810f60d3f9f/.tmp/0b6350c99fca4b06ba873f49a1ddb835
|
org.apache.hadoop.hbase.regionserver.HRegionFileSystem.bulkLoadStoreFile(HRegionFileSystem.java:544)
2024-04-20 23:44:29,185 | INFO |
RpcServer.default.FPBQ.Fifo.handler=415,queue=15,port=21302 | Loaded HFile
viewfs://ClusterX/hbase/data/default/boss20240420/0bf7d39f1708c5e50444c810f60d3f9f/.tmp/0b6350c99fca4b06ba873f49a1ddb835
into 0bf7d39f1708c5e50444c810f60d3f9f/info as
viewfs://ClusterX/hbase/data/default/boss20240420/0bf7d39f1708c5e50444c810f60d3f9f/info/2ae95a60b63541538939adfef0ab2190_SeqId_34_
- updating store file list. |
org.apache.hadoop.hbase.regionserver.HStore.bulkLoadHFile(HStore.java:908)
2024-04-20 23:44:29,201 | INFO |
RpcServer.default.FPBQ.Fifo.handler=415,queue=15,port=21302 | Loaded HFile
viewfs://ClusterX/hbase/data/default/boss20240420/0bf7d39f1708c5e50444c810f60d3f9f/info/2ae95a60b63541538939adfef0ab2190_SeqId_34_
into 0bf7d39f1708c5e50444c810f60d3f9f/info |
org.apache.hadoop.hbase.regionserver.HStore.bulkLoadHFile(HStore.java:942)
2024-04-20 23:44:29,201 | INFO |
RpcServer.default.FPBQ.Fifo.handler=415,queue=15,port=21302 | Successfully
loaded
viewfs://ClusterX/hbase/data/default/boss20240420/0bf7d39f1708c5e50444c810f60d3f9f/.tmp/0b6350c99fca4b06ba873f49a1ddb835
into 0bf7d39f1708c5e50444c810f60d3f9f/info (new location:
viewfs://ClusterX/hbase/data/default/boss20240420/0bf7d39f1708c5e50444c810f60d3f9f/info/2ae95a60b63541538939adfef0ab2190_SeqId_34_)
| org.apache.hadoop.hbase.regionserver.HStore.bulkLoadHFile(HStore.java:914)
2024-04-20 23:44:31,993 | INFO |
RpcServer.default.FPBQ.Fifo.handler=415,queue=15,port=21302 | Validating hfile
at
viewfs://ClusterX/tenant/hw_cdr/dsttab/load_20240420232000/boss20240420/info/f886ffd9d3d44804bcf164ff14ff24a6
for inclusion in bf26eb5268636a04d990e9a99da516fa/info |
org.apache.hadoop.hbase.regionserver.HStore.assertBulkLoadHFileOk(HStore.java:817)
2024-04-20 23:44:32,071 | INFO |
RpcServer.default.FPBQ.Fifo.handler=415,queue=15,port=21302 | Bulk-load file
hdfs://hacluster/hbase/staging/cdr__boss20240420__t20amapi48eb4ce0k0t80jatvnnhpm5rohaj26a5rdsp0s7kifokcdd32ko89u7n/info/f886ffd9d3d44804bcf164ff14ff24a6
is on different filesystem than the destination store. Copying file over to
destination filesystem. |
org.apache.hadoop.hbase.regionserver.HRegionFileSystem.bulkLoadStoreFile(HRegionFileSystem.java:540)
2024-04-20 23:44:32,179 | INFO |
RpcServer.default.FPBQ.Fifo.handler=415,queue=15,port=21302 | Copied
hdfs://hacluster/hbase/staging/cdr__boss20240420__t20amapi48eb4ce0k0t80jatvnnhpm5rohaj26a5rdsp0s7kifokcdd32ko89u7n/info/f886ffd9d3d44804bcf164ff14ff24a6
to temporary path on destination filesystem:
viewfs://ClusterX/hbase/data/default/boss20240420/bf26eb5268636a04d990e9a99da516fa/.tmp/bb3afca8b1c34f30a44fbc23416e43c7
|
org.apache.hadoop.hbase.regionserver.HRegionFileSystem.bulkLoadStoreFile(HRegionFileSystem.java:544)
2024-04-20 23:44:32,182 | INFO |
RpcServer.default.FPBQ.Fifo.handler=415,queue=15,port=21302 | Loaded HFile
viewfs://ClusterX/hbase/data/default/boss20240420/bf26eb5268636a04d990e9a99da516fa/.tmp/bb3afca8b1c34f30a44fbc23416e43c7
into bf26eb5268636a04d990e9a99da516fa/info as
viewfs://ClusterX/hbase/data/default/boss20240420/bf26eb5268636a04d990e9a99da516fa/info/e7cafc760651400f95b9929bc4dfe09f_SeqId_24_
- updating store file list. |
org.apache.hadoop.hbase.regionserver.HStore.bulkLoadHFile(HStore.java:908)
2024-04-20 23:44:32,194 | INFO |
RpcServer.default.FPBQ.Fifo.handler=415,queue=15,port=21302 | Loaded HFile
viewfs://ClusterX/hbase/data/default/boss20240420/bf26eb5268636a04d990e9a99da516fa/info/e7cafc760651400f95b9929bc4dfe09f_SeqId_24_
into bf26eb5268636a04d990e9a99da516fa/info |
org.apache.hadoop.hbase.regionserver.HStore.bulkLoadHFile(HStore.java:942)
2024-04-20 23:44:32,194 | INFO |
RpcServer.default.FPBQ.Fifo.handler=415,queue=15,port=21302 | Successfully
loaded
viewfs://ClusterX/hbase/data/default/boss20240420/bf26eb5268636a04d990e9a99da516fa/.tmp/bb3afca8b1c34f30a44fbc23416e43c7
into bf26eb5268636a04d990e9a99da516fa/info (new location:
viewfs://ClusterX/hbase/data/default/boss20240420/bf26eb5268636a04d990e9a99da516fa/info/e7cafc760651400f95b9929bc4dfe09f_SeqId_24_)
| org.apache.hadoop.hbase.regionserver.HStore.bulkLoadHFile(HStore.java:914)
2024-04-20 23:44:32,643 | INFO | LruBlockCacheStatsExecutor | totalSize=810.84
MB, freeSize=2.37 GB, max=3.16 GB, blockCount=6432, accesses=2141008613,
hits=42674300, hitRatio=1.99%, , cachingAccesses=42757862,
cachingHits=42611408, cachingHitsRatio=99.66%, evictions=863177, evicted=0,
evictedPerRun=0.0 |
org.apache.hadoop.hbase.io.hfile.LruBlockCache.logStats(LruBlockCache.java:940)
2024-04-20 23:44:32,967 | INFO | BucketCacheStatsExecutor |
failedBlockAdditions=0, totalSize=2.00 GB, freeSize=2.00 GB, usedSize=1.60 MB,
cacheSize=1.55 MB, accesses=2103504114, hits=59423, IOhitsPerSecond=0,
IOTimePerHit=NaN, hitRatio=0.00%, cachingAccesses=290936, cachingHits=946,
cachingHitsRatio=0.33%, evictions=49, evicted=289802,
evictedPerRun=5914.326530612245 |
org.apache.hadoop.hbase.io.hfile.bucket.BucketCache.logStats(BucketCache.java:637)
2024-04-20 23:44:35,020 | INFO |
org.apache.ranger.audit.queue.AuditBatchQueue0 | Audit Status Log:
name=hbaseRegional.async.summary.batch,
finalDestination=hbaseRegional.async.summary.batch.log4j, interval=01:02.099
minutes, events=18, succcessCount=8, totalEvents=10351899,
totalSuccessCount=1046226 |
org.apache.ranger.audit.provider.BaseAuditHandler.logStatus(BaseAuditHandler.java:332)
2024-04-20 23:44:38,469 | INFO | p1tdhdpdn-v-7:21302Replication Statistics #0
| Global stats: WAL Edits Buffer Used=0B, Limit=268435456B
|
org.apache.hadoop.hbase.replication.regionserver.Replication$ReplicationStatisticsTask.printStats(Replication.java:241)
2024-04-20 23:45:35,922 | INFO |
org.apache.ranger.audit.queue.AuditBatchQueue0 | Audit Status Log:
name=hbaseRegional.async.summary.batch,
finalDestination=hbaseRegional.async.summary.batch.log4j, interval=01:00.902
minutes, events=2, succcessCount=2, totalEvents=10351901,
totalSuccessCount=1046228 |
org.apache.ranger.audit.provider.BaseAuditHandler.logStatus(BaseAuditHandler.java:332)
2024-04-20 23:45:52,951 | INFO | HBase-Metrics2-1 | Stopping HBase metrics
system... |
org.apache.hadoop.metrics2.impl.MetricsSystemImpl.stop(MetricsSystemImpl.java:210)
2024-04-20 23:45:52,958 | INFO | HBase-Metrics2-1 | HBase metrics system
stopped. |
org.apache.hadoop.metrics2.impl.MetricsSystemImpl.stop(MetricsSystemImpl.java:216)
2024-04-20 23:45:53,459 | WARN | HBase-Metrics2-1 | Cannot locate
configuration: tried
hadoop-metrics2-hbase.properties,hadoop-metrics2.properties |
org.apache.hadoop.metrics2.impl.MetricsConfig.loadFirst(MetricsConfig.java:133)
2024-04-20 23:45:53,459 | INFO | HBase-Metrics2-1 | Scheduled Metric snapshot
period at 10 second(s). |
org.apache.hadoop.metrics2.impl.MetricsSystemImpl.startTimer(MetricsSystemImpl.java:378)
2024-04-20 23:45:53,460 | INFO | HBase-Metrics2-1 | HBase metrics system
started |
org.apache.hadoop.metrics2.impl.MetricsSystemImpl.start(MetricsSystemImpl.java:191)
2024-04-20 23:47:38,739 | INFO |
RpcServer.default.FPBQ.Fifo.handler=415,queue=15,port=21302 | Chore
[ScheduledChore: Name: QuotaRefresherChore Period: 300000 Unit: MILLISECONDS]
is enabled. |
org.apache.hadoop.hbase.ChoreService.scheduleChore(ChoreService.java:157)
2024-04-20 23:47:41,926 | INFO |
org.apache.ranger.audit.queue.AuditBatchQueue0 | Audit Status Log:
name=hbaseRegional.async.summary.batch.log4j, interval=03:30.005 minutes,
events=10, succcessCount=10, totalEvents=1046228, totalSuccessCount=1046228 |
org.apache.ranger.audit.provider.BaseAuditHandler.logStatus(BaseAuditHandler.java:332)
2024-04-20 23:48:36,094 | INFO |
org.apache.ranger.audit.queue.AuditBatchQueue0 | Audit Status Log:
name=hbaseRegional.async.summary.batch,
finalDestination=hbaseRegional.async.summary.batch.log4j, interval=01:00.169
minutes, events=3787, succcessCount=896, totalEvents=10355688,
totalSuccessCount=1047124 |
org.apache.ranger.audit.provider.BaseAuditHandler.logStatus(BaseAuditHandler.java:332)
2024-04-20 23:49:25,976 | INFO |
RpcServer.default.FPBQ.Fifo.handler=301,queue=1,port=21302 | Validating hfile
at
viewfs://ClusterX/tenant/hw_cdr/dsttab/load_20240420233000/boss20240420/info/6403649b9cff4e2e9a716edfb91eed02
for inclusion in 24b5f9451a7c43c78f510539ac3124b6/info |
org.apache.hadoop.hbase.regionserver.HStore.assertBulkLoadHFileOk(HStore.java:817)
2024-04-20 23:49:26,330 | INFO |
RpcServer.default.FPBQ.Fifo.handler=301,queue=1,port=21302 | Bulk-load file
hdfs://hacluster/hbase/staging/cdr__boss20240420__npj0dvc9jt9shu2lknr23t8d0l9qu9v84rjc1atk9m69ludreffg4plgpjeo5jlb/info/6403649b9cff4e2e9a716edfb91eed02
is on different filesystem than the destination store. Copying file over to
destination filesystem. |
org.apache.hadoop.hbase.regionserver.HRegionFileSystem.bulkLoadStoreFile(HRegionFileSystem.java:540)
2024-04-20 23:49:26,762 | INFO |
RpcServer.default.FPBQ.Fifo.handler=301,queue=1,port=21302 | Copied
hdfs://hacluster/hbase/staging/cdr__boss20240420__npj0dvc9jt9shu2lknr23t8d0l9qu9v84rjc1atk9m69ludreffg4plgpjeo5jlb/info/6403649b9cff4e2e9a716edfb91eed02
to temporary path on destination filesystem:
viewfs://ClusterX/hbase/data/default/boss20240420/24b5f9451a7c43c78f510539ac3124b6/.tmp/a3eb0160995742509ccd235ebfcfa281
|
org.apache.hadoop.hbase.regionserver.HRegionFileSystem.bulkLoadStoreFile(HRegionFileSystem.java:544)
2024-04-20 23:49:26,768 | INFO |
RpcServer.default.FPBQ.Fifo.handler=301,queue=1,port=21302 | Loaded HFile
viewfs://ClusterX/hbase/data/default/boss20240420/24b5f9451a7c43c78f510539ac3124b6/.tmp/a3eb0160995742509ccd235ebfcfa281
into 24b5f9451a7c43c78f510539ac3124b6/info as
viewfs://ClusterX/hbase/data/default/boss20240420/24b5f9451a7c43c78f510539ac3124b6/info/775e33b5b20242f1821c8b52d9b93ddf_SeqId_16_
- updating store file list. |
org.apache.hadoop.hbase.regionserver.HStore.bulkLoadHFile(HStore.java:908)
2024-04-20 23:49:26,782 | INFO |
RpcServer.default.FPBQ.Fifo.handler=301,queue=1,port=21302 | Loaded HFile
viewfs://ClusterX/hbase/data/default/boss20240420/24b5f9451a7c43c78f510539ac3124b6/info/775e33b5b20242f1821c8b52d9b93ddf_SeqId_16_
into 24b5f9451a7c43c78f510539ac3124b6/info |
org.apache.hadoop.hbase.regionserver.HStore.bulkLoadHFile(HStore.java:942)
2024-04-20 23:49:26,782 | INFO |
RpcServer.default.FPBQ.Fifo.handler=301,queue=1,port=21302 | Successfully
loaded
viewfs://ClusterX/hbase/data/default/boss20240420/24b5f9451a7c43c78f510539ac3124b6/.tmp/a3eb0160995742509ccd235ebfcfa281
into 24b5f9451a7c43c78f510539ac3124b6/info (new location:
viewfs://ClusterX/hbase/data/default/boss20240420/24b5f9451a7c43c78f510539ac3124b6/info/775e33b5b20242f1821c8b52d9b93ddf_SeqId_16_)
| org.apache.hadoop.hbase.regionserver.HStore.bulkLoadHFile(HStore.java:914)
2024-04-20 23:49:27,148 | INFO |
RpcServer.default.FPBQ.Fifo.handler=301,queue=1,port=21302 | Validating hfile
at
viewfs://ClusterX/tenant/hw_cdr/dsttab/load_20240420233000/boss20240420/info/474cd88557de4a999c14173dd65c8817
for inclusion in 281284cc6ea724c51c83ce23e5249f7b/info |
org.apache.hadoop.hbase.regionserver.HStore.assertBulkLoadHFileOk(HStore.java:817)
2024-04-20 23:49:27,219 | INFO |
RpcServer.default.FPBQ.Fifo.handler=301,queue=1,port=21302 | Bulk-load file
hdfs://hacluster/hbase/staging/cdr__boss20240420__npj0dvc9jt9shu2lknr23t8d0l9qu9v84rjc1atk9m69ludreffg4plgpjeo5jlb/info/474cd88557de4a999c14173dd65c8817
is on different filesystem than the destination store. Copying file over to
destination filesystem. |
org.apache.hadoop.hbase.regionserver.HRegionFileSystem.bulkLoadStoreFile(HRegionFileSystem.java:540)
2024-04-20 23:49:27,257 | INFO |
RpcServer.default.FPBQ.Fifo.handler=301,queue=1,port=21302 | Copied
hdfs://hacluster/hbase/staging/cdr__boss20240420__npj0dvc9jt9shu2lknr23t8d0l9qu9v84rjc1atk9m69ludreffg4plgpjeo5jlb/info/474cd88557de4a999c14173dd65c8817
to temporary path on destination filesystem:
viewfs://ClusterX/hbase/data/default/boss20240420/281284cc6ea724c51c83ce23e5249f7b/.tmp/7d352cb6f77047e9a39bc35379f70a92
|
org.apache.hadoop.hbase.regionserver.HRegionFileSystem.bulkLoadStoreFile(HRegionFileSystem.java:544)
2024-04-20 23:49:27,260 | INFO |
RpcServer.default.FPBQ.Fifo.handler=301,queue=1,port=21302 | Loaded HFile
viewfs://ClusterX/hbase/data/default/boss20240420/281284cc6ea724c51c83ce23e5249f7b/.tmp/7d352cb6f77047e9a39bc35379f70a92
into 281284cc6ea724c51c83ce23e5249f7b/info as
viewfs://ClusterX/hbase/data/default/boss20240420/281284cc6ea724c51c83ce23e5249f7b/info/8fb058fa1e4c48fb862e559f873a1390_SeqId_32_
- updating store file list. |
org.apache.hadoop.hbase.regionserver.HStore.bulkLoadHFile(HStore.java:908)
2024-04-20 23:49:27,276 | INFO |
RpcServer.default.FPBQ.Fifo.handler=301,queue=1,port=21302 | Loaded HFile
viewfs://ClusterX/hbase/data/default/boss20240420/281284cc6ea724c51c83ce23e5249f7b/info/8fb058fa1e4c48fb862e559f873a1390_SeqId_32_
into 281284cc6ea724c51c83ce23e5249f7b/info |
org.apache.hadoop.hbase.regionserver.HStore.bulkLoadHFile(HStore.java:942)
2024-04-20 23:49:27,276 | INFO |
RpcServer.default.FPBQ.Fifo.handler=301,queue=1,port=21302 | Successfully
loaded
viewfs://ClusterX/hbase/data/default/boss20240420/281284cc6ea724c51c83ce23e5249f7b/.tmp/7d352cb6f77047e9a39bc35379f70a92
into 281284cc6ea724c51c83ce23e5249f7b/info (new location:
viewfs://ClusterX/hbase/data/default/boss20240420/281284cc6ea724c51c83ce23e5249f7b/info/8fb058fa1e4c48fb862e559f873a1390_SeqId_32_)
| org.apache.hadoop.hbase.regionserver.HStore.bulkLoadHFile(HStore.java:914)
2024-04-20 23:49:27,417 | INFO | regionserver/p1tdhdpdn-v-7:21302.Chore.11 |
MemstoreFlusherChore requesting flush of
boss20240405,H3020209~,1712160114717.4e7bdeeb9b222b113307199da2810e18. because
4e7bdeeb9b222b113307199da2810e18/info has an old edit so flush to free WALs
after random delay 239909 ms |
org.apache.hadoop.hbase.regionserver.HRegionServer$PeriodicMemStoreFlusher.chore(HRegionServer.java:2059)
2024-04-20 23:49:30,094 | INFO |
org.apache.ranger.audit.queue.AuditBatchQueue0 | Audit Status Log:
name=hbaseRegional.async.summary.batch.log4j, interval=01:48.168 minutes,
events=896, succcessCount=896, totalEvents=1047124, totalSuccessCount=1047124 |
org.apache.ranger.audit.provider.BaseAuditHandler.logStatus(BaseAuditHandler.java:332)
2024-04-20 23:49:32,643 | INFO | LruBlockCacheStatsExecutor | totalSize=811.85
MB, freeSize=2.37 GB, max=3.16 GB, blockCount=6440, accesses=2141096719,
hits=42676155, hitRatio=1.99%, , cachingAccesses=42759717,
cachingHits=42613255, cachingHitsRatio=99.66%, evictions=863207, evicted=0,
evictedPerRun=0.0 |
org.apache.hadoop.hbase.io.hfile.LruBlockCache.logStats(LruBlockCache.java:940)
2024-04-20 23:49:32,967 | INFO | BucketCacheStatsExecutor |
failedBlockAdditions=0, totalSize=2.00 GB, freeSize=2.00 GB, usedSize=1.60 MB,
cacheSize=1.55 MB, accesses=2103590357, hits=59423, IOhitsPerSecond=0,
IOTimePerHit=NaN, hitRatio=0.00%, cachingAccesses=290936, cachingHits=946,
cachingHitsRatio=0.33%, evictions=49, evicted=289802,
evictedPerRun=5914.326530612245 |
org.apache.hadoop.hbase.io.hfile.bucket.BucketCache.logStats(BucketCache.java:637)
2024-04-20 23:49:38,469 | INFO | p1tdhdpdn-v-7:21302Replication Statistics #0
| Global stats: WAL Edits Buffer Used=0B, Limit=268435456B
|
org.apache.hadoop.hbase.replication.regionserver.Replication$ReplicationStatisticsTask.printStats(Replication.java:241)
2024-04-20 23:49:39,094 | INFO |
org.apache.ranger.audit.queue.AuditBatchQueue0 | Audit Status Log:
name=hbaseRegional.async.summary.batch,
finalDestination=hbaseRegional.async.summary.batch.log4j, interval=01:03.000
minutes, events=4, succcessCount=2, totalEvents=10355692,
totalSuccessCount=1047126 |
org.apache.ranger.audit.provider.BaseAuditHandler.logStatus(BaseAuditHandler.java:332)
2024-04-20 23:50:52,951 | INFO | HBase-Metrics2-1 | Stopping HBase metrics
system... |
org.apache.hadoop.metrics2.impl.MetricsSystemImpl.stop(MetricsSystemImpl.java:210)
2024-04-20 23:50:52,959 | INFO | HBase-Metrics2-1 | HBase metrics system
stopped. |
org.apache.hadoop.metrics2.impl.MetricsSystemImpl.stop(MetricsSystemImpl.java:216)
2024-04-20 23:50:53,461 | WARN | HBase-Metrics2-1 | Cannot locate
configuration: tried
hadoop-metrics2-hbase.properties,hadoop-metrics2.properties |
org.apache.hadoop.metrics2.impl.MetricsConfig.loadFirst(MetricsConfig.java:133)
2024-04-20 23:50:53,476 | INFO | HBase-Metrics2-1 | Scheduled Metric snapshot
period at 10 second(s). |
org.apache.hadoop.metrics2.impl.MetricsSystemImpl.startTimer(MetricsSystemImpl.java:378)
2024-04-20 23:50:53,476 | INFO | HBase-Metrics2-1 | HBase metrics system
started |
org.apache.hadoop.metrics2.impl.MetricsSystemImpl.start(MetricsSystemImpl.java:191)
2024-04-20 23:51:14,537 | INFO |
regionserver/p1tdhdpdn-v-7:21302-longCompactions-9 |
0bf7d39f1708c5e50444c810f60d3f9f#info#compaction#11852 average throughput is
3.49 MB/second, slept 1444 time(s) and total slept time is 3969809 ms. 8 active
operations remaining, total limit is 18.00 MB/second |
org.apache.hadoop.hbase.regionserver.throttle.PressureAwareThroughputController.finish(PressureAwareThroughputController.java:145)
2024-04-20 23:51:15,245 | INFO |
regionserver/p1tdhdpdn-v-7:21302-longCompactions-9 | Completed compaction of 15
(all) file(s) in 0bf7d39f1708c5e50444c810f60d3f9f/info of
0bf7d39f1708c5e50444c810f60d3f9f into 8ae2d25f22a34a01a8b895b0bfa7394e(size=2.2
G), total size for store is 2.2 G. This selection was in queue for 0sec, and
took 1hrs, 9mins, 7sec to execute. |
org.apache.hadoop.hbase.regionserver.HStore.logCompactionEndMessage(HStore.java:1669)
2024-04-20 23:51:15,245 | INFO |
regionserver/p1tdhdpdn-v-7:21302-longCompactions-9 | Completed compaction
Request=regionName=boss20240420,H1030107628~,1713456095893.0bf7d39f1708c5e50444c810f60d3f9f.,
storeName=info, fileCount=15, fileSize=2.3 G (30.8 M, 1.2 G, 1003.2 M, 25.0 M,
7.1 M, 94.0 K, 83.3 K, 27.0 K, 23.4 K, 21.6 K, 14.7 K, 10.9 K, 17.4 K, 10.9 K,
12.8 K), priority=2, time=1713624128170; duration=1hrs, 9mins, 7sec |
org.apache.hadoop.hbase.regionserver.CompactSplit$CompactionRunner.doCompaction(CompactSplit.java:632)
2024-04-20 23:53:27,327 | INFO | MemStoreFlusher.3 | Unable to get remote
Address |
org.apache.ranger.authorization.hbase.RangerAuthorizationCoprocessor.getRemoteAddress(RangerAuthorizationCoprocessor.java:214)
2024-04-20 23:53:27,376 | INFO | MemStoreFlusher.3 | Flushing
4e7bdeeb9b222b113307199da2810e18 1/1 column families, dataSize=1.87 MB
heapSize=2.50 MB |
org.apache.hadoop.hbase.regionserver.HRegion.logFatLineOnFlush(HRegion.java:2870)
2024-04-20 23:53:27,524 | INFO | MemStoreFlusher.3 | Flushed memstore data
size=1.87 MB at sequenceid=146705 (bloomFilter=true),
to=viewfs://ClusterX/hbase/data/default/boss20240405/4e7bdeeb9b222b113307199da2810e18/.tmp/info/98071aebc1e4481aafc41f8e1e8d2243
|
org.apache.hadoop.hbase.regionserver.DefaultStoreFlusher.flushSnapshot(DefaultStoreFlusher.java:85)
2024-04-20 23:53:27,554 | INFO | MemStoreFlusher.3 | Added
viewfs://ClusterX/hbase/data/default/boss20240405/4e7bdeeb9b222b113307199da2810e18/info/98071aebc1e4481aafc41f8e1e8d2243,
entries=2940, sequenceid=146705, filesize=190.2 K |
org.apache.hadoop.hbase.regionserver.HStore.commitFile(HStore.java:1145)
2024-04-20 23:53:27,555 | INFO | MemStoreFlusher.3 | Finished flush of
dataSize ~1.87 MB/1960410, heapSize ~2.50 MB/2618256, currentSize=0 B/0 for
4e7bdeeb9b222b113307199da2810e18 in 179ms, sequenceid=146705, compaction
requested=false |
org.apache.hadoop.hbase.regionserver.HRegion.internalFlushCacheAndCommit(HRegion.java:3079)
2024-04-20 23:53:30,121 | INFO |
org.apache.ranger.audit.queue.AuditBatchQueue0 | Audit Status Log:
name=hbaseRegional.async.summary.batch.log4j, interval=04:00.027 minutes,
events=2, succcessCount=2, totalEvents=1047126, totalSuccessCount=1047126 |
org.apache.ranger.audit.provider.BaseAuditHandler.logStatus(BaseAuditHandler.java:332)
2024-04-20 23:53:39,121 | INFO |
org.apache.ranger.audit.queue.AuditBatchQueue0 | Audit Status Log:
name=hbaseRegional.async.summary.batch,
finalDestination=hbaseRegional.async.summary.batch.log4j, interval=01:00.001
minutes, events=2, succcessCount=1, totalEvents=10355694,
totalSuccessCount=1047127 |
org.apache.ranger.audit.provider.BaseAuditHandler.logStatus(BaseAuditHandler.java:332)
2024-04-20 23:54:32,643 | INFO | LruBlockCacheStatsExecutor | totalSize=812.98
MB, freeSize=2.37 GB, max=3.16 GB, blockCount=6449, accesses=2141184975,
hits=42678224, hitRatio=1.99%, , cachingAccesses=42761786,
cachingHits=42615315, cachingHitsRatio=99.66%, evictions=863237, evicted=0,
evictedPerRun=0.0 |
org.apache.hadoop.hbase.io.hfile.LruBlockCache.logStats(LruBlockCache.java:940)
2024-04-20 23:54:32,967 | INFO | BucketCacheStatsExecutor |
failedBlockAdditions=0, totalSize=2.00 GB, freeSize=2.00 GB, usedSize=1.60 MB,
cacheSize=1.55 MB, accesses=2103676600, hits=59423, IOhitsPerSecond=0,
IOTimePerHit=NaN, hitRatio=0.00%, cachingAccesses=290936, cachingHits=946,
cachingHitsRatio=0.33%, evictions=49, evicted=289802,
evictedPerRun=5914.326530612245 |
org.apache.hadoop.hbase.io.hfile.bucket.BucketCache.logStats(BucketCache.java:637)
2024-04-20 23:54:32,972 | INFO | MobFileCache #0 | MobFileCache Statistics,
access: 0, miss: 0, hit: 0, hit ratio: 0%, evicted files: 0 |
org.apache.hadoop.hbase.mob.MobFileCache.printStatistics(MobFileCache.java:341)
2024-04-20 23:54:38,469 | INFO | p1tdhdpdn-v-7:21302Replication Statistics #0
| Global stats: WAL Edits Buffer Used=0B, Limit=268435456B
|
org.apache.hadoop.hbase.replication.regionserver.Replication$ReplicationStatisticsTask.printStats(Replication.java:241)
2024-04-20 23:55:10,095 | INFO | regionserver/p1tdhdpdn-v-7:21302.logRoller |
Rolled WAL
/hbase/WALs/p1tdhdpdn-v-7,21302,1704995671636/p1tdhdpdn-v-7%2C21302%2C1704995671636.1713624909375
with entries=584, filesize=1.11 MB; new WAL
/hbase/WALs/p1tdhdpdn-v-7,21302,1704995671636/p1tdhdpdn-v-7%2C21302%2C1704995671636.1713628510024
|
org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.logRollAndSetupWalProps(AbstractFSWAL.java:727)
2024-04-20 23:55:10,095 | INFO | Log-Archiver-0 | Archiving
viewfs://ClusterX/hbase/WALs/p1tdhdpdn-v-7,21302,1704995671636/p1tdhdpdn-v-7%2C21302%2C1704995671636.1713621309238
to
viewfs://ClusterX/hbase/oldWALs/p1tdhdpdn-v-7%2C21302%2C1704995671636.1713621309238
|
org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.archiveLogFile(AbstractFSWAL.java:708)
2024-04-20 23:55:52,951 | INFO | HBase-Metrics2-1 | Stopping HBase metrics
system... |
org.apache.hadoop.metrics2.impl.MetricsSystemImpl.stop(MetricsSystemImpl.java:210)
2024-04-20 23:55:52,958 | INFO | HBase-Metrics2-1 | HBase metrics system
stopped. |
org.apache.hadoop.metrics2.impl.MetricsSystemImpl.stop(MetricsSystemImpl.java:216)
2024-04-20 23:55:53,460 | WARN | HBase-Metrics2-1 | Cannot locate
configuration: tried
hadoop-metrics2-hbase.properties,hadoop-metrics2.properties |
org.apache.hadoop.metrics2.impl.MetricsConfig.loadFirst(MetricsConfig.java:133)
2024-04-20 23:55:53,461 | INFO | HBase-Metrics2-1 | Scheduled Metric snapshot
period at 10 second(s). |
org.apache.hadoop.metrics2.impl.MetricsSystemImpl.startTimer(MetricsSystemImpl.java:378)
2024-04-20 23:55:53,461 | INFO | HBase-Metrics2-1 | HBase metrics system
started |
org.apache.hadoop.metrics2.impl.MetricsSystemImpl.start(MetricsSystemImpl.java:191)
2024-04-20 23:59:32,643 | INFO | LruBlockCacheStatsExecutor | totalSize=814.11
MB, freeSize=2.36 GB, max=3.16 GB, blockCount=6458, accesses=2141273489,
hits=42680456, hitRatio=1.99%, , cachingAccesses=42764018,
cachingHits=42617538, cachingHitsRatio=99.66%, evictions=863267, evicted=0,
evictedPerRun=0.0 |
org.apache.hadoop.hbase.io.hfile.LruBlockCache.logStats(LruBlockCache.java:940)
2024-04-20 23:59:32,967 | INFO | BucketCacheStatsExecutor |
failedBlockAdditions=0, totalSize=2.00 GB, freeSize=2.00 GB, usedSize=1.60 MB,
cacheSize=1.55 MB, accesses=2103762905, hits=59423, IOhitsPerSecond=0,
IOTimePerHit=NaN, hitRatio=0.00%, cachingAccesses=290936, cachingHits=946,
cachingHitsRatio=0.33%, evictions=49, evicted=289802,
evictedPerRun=5914.326530612245 |
org.apache.hadoop.hbase.io.hfile.bucket.BucketCache.logStats(BucketCache.java:637)
2024-04-20 23:59:38,469 | INFO | p1tdhdpdn-v-7:21302Replication Statistics #0
| Global stats: WAL Edits Buffer Used=0B, Limit=268435456B
|
org.apache.hadoop.hbase.replication.regionserver.Replication$ReplicationStatisticsTask.printStats(Replication.java:241)
2024-04-21 00:00:15,428 | INFO |
RS_CLOSE_REGION-regionserver/p1tdhdpdn-v-7:21302-16 | Close
8975d95b7a69dc45b0055630545c2e25 |
org.apache.hadoop.hbase.regionserver.handler.UnassignRegionHandler.process(UnassignRegionHandler.java:107)
2024-04-21 00:00:15,428 | INFO |
RS_CLOSE_REGION-regionserver/p1tdhdpdn-v-7:21302-16 | Unable to get remote
Address |
org.apache.ranger.authorization.hbase.RangerAuthorizationCoprocessor.getRemoteAddress(RangerAuthorizationCoprocessor.java:214)
2024-04-21 00:00:15,488 | INFO |
RS_CLOSE_REGION-regionserver/p1tdhdpdn-v-7:21302-16 | Unable to get remote
Address |
org.apache.ranger.authorization.hbase.RangerAuthorizationCoprocessor.getRemoteAddress(RangerAuthorizationCoprocessor.java:214)
2024-04-21 00:00:15,488 | INFO |
RS_CLOSE_REGION-regionserver/p1tdhdpdn-v-7:21302-16 | Waiting for flushes and
compactions to finish for the region
boss20240221,H5CC9923038~,1708358595748.8975d95b7a69dc45b0055630545c2e25. |
org.apache.hadoop.hbase.regionserver.HRegion.waitForFlushesAndCompactions(HRegion.java:1924)
2024-04-21 00:00:15,488 | INFO |
RS_CLOSE_REGION-regionserver/p1tdhdpdn-v-7:21302-16 | Total wait time for
flushes and compaction for the region
boss20240221,H5CC9923038~,1708358595748.8975d95b7a69dc45b0055630545c2e25. is:
0ms |
org.apache.hadoop.hbase.regionserver.HRegion.waitForFlushesAndCompactions(HRegion.java:1960)
2024-04-21 00:00:15,488 | INFO |
RS_CLOSE_REGION-regionserver/p1tdhdpdn-v-7:21302-16 | Closing region
boss20240221,H5CC9923038~,1708358595748.8975d95b7a69dc45b0055630545c2e25. |
org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1696)
2024-04-21 00:00:15,839 | INFO |
RS_CLOSE_REGION-regionserver/p1tdhdpdn-v-7:21302-2 | Close
0d541b4f0bf68727cfd0f2dfffc4dde5 |
org.apache.hadoop.hbase.regionserver.handler.UnassignRegionHandler.process(UnassignRegionHandler.java:107)
2024-04-21 00:00:15,840 | INFO |
RS_CLOSE_REGION-regionserver/p1tdhdpdn-v-7:21302-2 | Unable to get remote
Address |
org.apache.ranger.authorization.hbase.RangerAuthorizationCoprocessor.getRemoteAddress(RangerAuthorizationCoprocessor.java:214)
2024-04-21 00:00:15,840 | INFO |
RS_CLOSE_REGION-regionserver/p1tdhdpdn-v-7:21302-2 | Unable to get remote
Address |
org.apache.ranger.authorization.hbase.RangerAuthorizationCoprocessor.getRemoteAddress(RangerAuthorizationCoprocessor.java:214)
2024-04-21 00:00:15,840 | INFO |
RS_CLOSE_REGION-regionserver/p1tdhdpdn-v-7:21302-2 | Waiting for flushes and
compactions to finish for the region
boss20240221,H5CC9915320~,1708358595748.0d541b4f0bf68727cfd0f2dfffc4dde5. |
org.apache.hadoop.hbase.regionserver.HRegion.waitForFlushesAndCompactions(HRegion.java:1924)
2024-04-21 00:00:15,840 | INFO |
RS_CLOSE_REGION-regionserver/p1tdhdpdn-v-7:21302-2 | Total wait time for
flushes and compaction for the region
boss20240221,H5CC9915320~,1708358595748.0d541b4f0bf68727cfd0f2dfffc4dde5. is:
0ms |
org.apache.hadoop.hbase.regionserver.HRegion.waitForFlushesAndCompactions(HRegion.java:1960)
2024-04-21 00:00:15,840 | INFO |
RS_CLOSE_REGION-regionserver/p1tdhdpdn-v-7:21302-2 | Closing region
boss20240221,H5CC9915320~,1708358595748.0d541b4f0bf68727cfd0f2dfffc4dde5. |
org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1696)
2024-04-21 00:00:16,349 | INFO |
RS_CLOSE_REGION-regionserver/p1tdhdpdn-v-7:21302-42 | Close
b8083f48bb2d591ca5aff2931d6e1ff7 |
org.apache.hadoop.hbase.regionserver.handler.UnassignRegionHandler.process(UnassignRegionHandler.java:107)
2024-04-21 00:00:16,349 | INFO |
RS_CLOSE_REGION-regionserver/p1tdhdpdn-v-7:21302-42 | Unable to get remote
Address |
org.apache.ranger.authorization.hbase.RangerAuthorizationCoprocessor.getRemoteAddress(RangerAuthorizationCoprocessor.java:214)
2024-04-21 00:00:16,349 | INFO |
RS_CLOSE_REGION-regionserver/p1tdhdpdn-v-7:21302-42 | Unable to get remote
Address |
org.apache.ranger.authorization.hbase.RangerAuthorizationCoprocessor.getRemoteAddress(RangerAuthorizationCoprocessor.java:214)
2024-04-21 00:00:16,349 | INFO |
RS_CLOSE_REGION-regionserver/p1tdhdpdn-v-7:21302-42 | Waiting for flushes and
compactions to finish for the region
boss20240221,H5CC9907525~,1708358595748.b8083f48bb2d591ca5aff2931d6e1ff7. |
org.apache.hadoop.hbase.regionserver.HRegion.waitForFlushesAndCompactions(HRegion.java:1924)
2024-04-21 00:00:16,349 | INFO |
RS_CLOSE_REGION-regionserver/p1tdhdpdn-v-7:21302-42 | Total wait time for
flushes and compaction for the region
boss20240221,H5CC9907525~,1708358595748.b8083f48bb2d591ca5aff2931d6e1ff7. is:
0ms |
org.apache.hadoop.hbase.regionserver.HRegion.waitForFlushesAndCompactions(HRegion.java:1960)
2024-04-21 00:00:16,349 | INFO |
RS_CLOSE_REGION-regionserver/p1tdhdpdn-v-7:21302-42 | Closing region
boss20240221,H5CC9907525~,1708358595748.b8083f48bb2d591ca5aff2931d6e1ff7. |
org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1696)
2024-04-21 00:00:16,631 | INFO |
RS_CLOSE_REGION-regionserver/p1tdhdpdn-v-7:21302-16 | Closed
boss20240221,H5CC9923038~,1708358595748.8975d95b7a69dc45b0055630545c2e25. |
org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1913)
2024-04-21 00:00:16,820 | INFO |
RS_CLOSE_REGION-regionserver/p1tdhdpdn-v-7:21302-47 | Close
00c03d83783ca21c7480b26b0f7b099e |
org.apache.hadoop.hbase.regionserver.handler.UnassignRegionHandler.process(UnassignRegionHandler.java:107)
2024-04-21 00:00:16,820 | INFO |
RS_CLOSE_REGION-regionserver/p1tdhdpdn-v-7:21302-47 | Unable to get remote
Address |
org.apache.ranger.authorization.hbase.RangerAuthorizationCoprocessor.getRemoteAddress(RangerAuthorizationCoprocessor.java:214)
2024-04-21 00:00:16,820 | INFO |
RS_CLOSE_REGION-regionserver/p1tdhdpdn-v-7:21302-47 | Unable to get remote
Address |
org.apache.ranger.authorization.hbase.RangerAuthorizationCoprocessor.getRemoteAddress(RangerAuthorizationCoprocessor.java:214)
2024-04-21 00:00:16,820 | INFO |
RS_CLOSE_REGION-regionserver/p1tdhdpdn-v-7:21302-47 | Waiting for flushes and
compactions to finish for the region
boss20240221,H5309912782~,1708358595748.00c03d83783ca21c7480b26b0f7b099e. |
org.apache.hadoop.hbase.regionserver.HRegion.waitForFlushesAndCompactions(HRegion.java:1924)
2024-04-21 00:00:16,820 | INFO |
RS_CLOSE_REGION-regionserver/p1tdhdpdn-v-7:21302-47 | Total wait time for
flushes and compaction for the region
boss20240221,H5309912782~,1708358595748.00c03d83783ca21c7480b26b0f7b099e. is:
0ms |
org.apache.hadoop.hbase.regionserver.HRegion.waitForFlushesAndCompactions(HRegion.java:1960)
2024-04-21 00:00:16,820 | INFO |
RS_CLOSE_REGION-regionserver/p1tdhdpdn-v-7:21302-47 | Closing region
boss20240221,H5309912782~,1708358595748.00c03d83783ca21c7480b26b0f7b099e. |
org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1696)
2024-04-21 00:00:17,018 | INFO |
RS_CLOSE_REGION-regionserver/p1tdhdpdn-v-7:21302-0 | Close
2b67a4ae42d57326ff9cf11a222b33ab |
org.apache.hadoop.hbase.regionserver.handler.UnassignRegionHandler.process(UnassignRegionHandler.java:107)
2024-04-21 00:00:17,018 | INFO |
RS_CLOSE_REGION-regionserver/p1tdhdpdn-v-7:21302-0 | Unable to get remote
Address |
org.apache.ranger.authorization.hbase.RangerAuthorizationCoprocessor.getRemoteAddress(RangerAuthorizationCoprocessor.java:214)
2024-04-21 00:00:17,018 | INFO |
RS_CLOSE_REGION-regionserver/p1tdhdpdn-v-7:21302-0 | Unable to get remote
Address |
org.apache.ranger.authorization.hbase.RangerAuthorizationCoprocessor.getRemoteAddress(RangerAuthorizationCoprocessor.java:214)
2024-04-21 00:00:17,026 | INFO |
RS_CLOSE_REGION-regionserver/p1tdhdpdn-v-7:21302-0 | Waiting for flushes and
compactions to finish for the region
boss20240221,H52R9907~,1708515296800.2b67a4ae42d57326ff9cf11a222b33ab. |
org.apache.hadoop.hbase.regionserver.HRegion.waitForFlushesAndCompactions(HRegion.java:1924)
2024-04-21 00:00:17,027 | INFO |
RS_CLOSE_REGION-regionserver/p1tdhdpdn-v-7:21302-0 | Total wait time for
flushes and compaction for the region
boss20240221,H52R9907~,1708515296800.2b67a4ae42d57326ff9cf11a222b33ab. is: 0ms
|
org.apache.hadoop.hbase.regionserver.HRegion.waitForFlushesAndCompactions(HRegion.java:1960)
2024-04-21 00:00:17,027 | INFO |
RS_CLOSE_REGION-regionserver/p1tdhdpdn-v-7:21302-0 | Closing region
boss20240221,H52R9907~,1708515296800.2b67a4ae42d57326ff9cf11a222b33ab. |
org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1696)
2024-04-21 00:00:17,189 | INFO |
RS_CLOSE_REGION-regionserver/p1tdhdpdn-v-7:21302-16 | Closed
8975d95b7a69dc45b0055630545c2e25 |
org.apache.hadoop.hbase.regionserver.handler.UnassignRegionHandler.process(UnassignRegionHandler.java:147)
2024-04-21 00:00:17,447 | INFO |
RS_CLOSE_REGION-regionserver/p1tdhdpdn-v-7:21302-2 | Closed
boss20240221,H5CC9915320~,1708358595748.0d541b4f0bf68727cfd0f2dfffc4dde5. |
org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1913)
2024-04-21 00:00:17,603 | WARN |
RS_CLOSE_REGION-regionserver/p1tdhdpdn-v-7:21302-0 | Fatal error occurred while
closing region 2b67a4ae42d57326ff9cf11a222b33ab, aborting... |
org.apache.hadoop.hbase.regionserver.handler.UnassignRegionHandler.handleException(UnassignRegionHandler.java:152)
java.io.IOException: The new max sequence id 328 is less than the old max
sequence id 330
at
org.apache.hadoop.hbase.wal.WALSplitUtil.writeRegionSequenceIdFile(WALSplitUtil.java:461)
at
org.apache.hadoop.hbase.regionserver.HRegion.writeRegionCloseMarker(HRegion.java:1265)
at org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1893)
at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1625)
at
org.apache.hadoop.hbase.regionserver.handler.UnassignRegionHandler.process(UnassignRegionHandler.java:116)
at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:106)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2024-04-21 00:00:17,610 | ERROR |
RS_CLOSE_REGION-regionserver/p1tdhdpdn-v-7:21302-0 | ***** ABORTING region
server p1tdhdpdn-v-7,21302,1704995671636: Failed to close region
2b67a4ae42d57326ff9cf11a222b33ab and can not recover ***** |
org.slf4j.helpers.MarkerIgnoringBase.error(MarkerIgnoringBase.java:159)
java.io.IOException: The new max sequence id 328 is less than the old max
sequence id 330
at
org.apache.hadoop.hbase.wal.WALSplitUtil.writeRegionSequenceIdFile(WALSplitUtil.java:461)
at
org.apache.hadoop.hbase.regionserver.HRegion.writeRegionCloseMarker(HRegion.java:1265)
at org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1893)
at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1625)
at
org.apache.hadoop.hbase.regionserver.handler.UnassignRegionHandler.process(UnassignRegionHandler.java:116)
at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:106)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
