apurtell commented on PR #4338:
URL: https://github.com/apache/hbase/pull/4338#issuecomment-1098561420
Pushed an update that brings all of the above discussion together.
- Compactions might be concurrent against a given store and the Compactor is
shared among them. Add a comment to the class javadoc to not put mutable state
into shared class fields. Make all Compactor class fields final, with the
exception of `keepSeqIdPeriod`, which is set by unit tests.
- Pass `writer` and `progress` instances through compaction as method
parmeters of `performCompaction` to improve MT-safety of these code paths.
- Scope compaction progress to each compaction to improve MT-safety overall
and the accuracy of compaction progress reporting.
All compaction unit tests pass.
[INFO] -------------------------------------------------------
[INFO] T E S T S
[INFO] -------------------------------------------------------
[INFO] Running
org.apache.hadoop.hbase.util.compaction.TestMajorCompactionTTLRequest
[INFO] Running
org.apache.hadoop.hbase.util.compaction.TestMajorCompactionRequest
[INFO] Running
org.apache.hadoop.hbase.quotas.policies.TestNoWritesCompactionsViolationPolicyEnforcement
[INFO] Running org.apache.hadoop.hbase.regionserver.TestMinorCompaction
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
1.427 s - in
org.apache.hadoop.hbase.quotas.policies.TestNoWritesCompactionsViolationPolicyEnforcement
[INFO] Running
org.apache.hadoop.hbase.regionserver.TestDateTieredCompactionPolicyOverflow
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
2.942 s - in
org.apache.hadoop.hbase.regionserver.TestDateTieredCompactionPolicyOverflow
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
4.878 s - in
org.apache.hadoop.hbase.util.compaction.TestMajorCompactionTTLRequest
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
4.918 s - in org.apache.hadoop.hbase.util.compaction.TestMajorCompactionRequest
[INFO] Running org.apache.hadoop.hbase.regionserver.TestCompactionState
[INFO] Running
org.apache.hadoop.hbase.regionserver.TestCompactionArchiveConcurrentClose
[INFO] Running
org.apache.hadoop.hbase.regionserver.throttle.TestCompactionWithThroughputController
[INFO] Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
8.935 s - in org.apache.hadoop.hbase.regionserver.TestMinorCompaction
[INFO] Running
org.apache.hadoop.hbase.regionserver.TestCompactionWithCoprocessor
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
5.828 s - in
org.apache.hadoop.hbase.regionserver.TestCompactionArchiveConcurrentClose
[INFO] Running
org.apache.hadoop.hbase.regionserver.TestCompactionWithByteBuff
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
34.662 s - in org.apache.hadoop.hbase.regionserver.TestCompactionWithByteBuff
[INFO] Running
org.apache.hadoop.hbase.regionserver.TestCompactionFileNotFound
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
18.873 s - in org.apache.hadoop.hbase.regionserver.TestCompactionFileNotFound
[INFO] Running
org.apache.hadoop.hbase.regionserver.TestCompactionArchiveIOException
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
11.024 s - in
org.apache.hadoop.hbase.regionserver.TestCompactionArchiveIOException
[INFO] Running
org.apache.hadoop.hbase.regionserver.TestCompactionLifeCycleTracker
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
84.897 s - in
org.apache.hadoop.hbase.regionserver.throttle.TestCompactionWithThroughputController
[INFO] Running org.apache.hadoop.hbase.regionserver.TestMajorCompaction
[WARNING] Tests run: 3, Failures: 0, Errors: 0, Skipped: 1, Time
elapsed: 16.319 s - in
org.apache.hadoop.hbase.regionserver.TestCompactionLifeCycleTracker
[INFO] Running
org.apache.hadoop.hbase.regionserver.TestDateTieredCompactionPolicy
[INFO] Tests run: 15, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
5.984 s - in org.apache.hadoop.hbase.regionserver.TestDateTieredCompactionPolicy
[INFO] Running
org.apache.hadoop.hbase.regionserver.TestCompactionInDeadRegionServer
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
26.748 s - in
org.apache.hadoop.hbase.regionserver.TestCompactionInDeadRegionServer
[INFO] Running
org.apache.hadoop.hbase.regionserver.querymatcher.TestCompactionScanQueryMatcher
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
0.896 s - in
org.apache.hadoop.hbase.regionserver.querymatcher.TestCompactionScanQueryMatcher
[INFO] Running org.apache.hadoop.hbase.regionserver.TestCompaction
[INFO] Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
199.59 s - in org.apache.hadoop.hbase.regionserver.TestCompactionWithCoprocessor
[INFO] Running
org.apache.hadoop.hbase.regionserver.TestDateTieredCompactionPolicyHeterogeneousStorage
[INFO] Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
4.228 s - in
org.apache.hadoop.hbase.regionserver.TestDateTieredCompactionPolicyHeterogeneousStorage
[INFO] Running
org.apache.hadoop.hbase.regionserver.compactions.TestFIFOCompactionPolicy
[INFO] Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
219.608 s - in org.apache.hadoop.hbase.regionserver.TestCompactionState
[INFO] Running
org.apache.hadoop.hbase.regionserver.compactions.TestStripeCompactionPolicy
[INFO] Tests run: 36, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
11.35 s - in
org.apache.hadoop.hbase.regionserver.compactions.TestStripeCompactionPolicy
[INFO] Running
org.apache.hadoop.hbase.regionserver.TestCompactionAfterBulkLoad
[INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
24.276 s - in
org.apache.hadoop.hbase.regionserver.compactions.TestFIFOCompactionPolicy
[INFO] Running
org.apache.hadoop.hbase.master.region.TestMasterRegionCompaction
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
8.307 s - in org.apache.hadoop.hbase.regionserver.TestCompactionAfterBulkLoad
[INFO] Running
org.apache.hadoop.hbase.rsgroup.TestRSGroupMajorCompactionTTL
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
17.496 s - in org.apache.hadoop.hbase.master.region.TestMasterRegionCompaction
[INFO] Running org.apache.hadoop.hbase.mob.TestMobStoreCompaction
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
5.738 s - in org.apache.hadoop.hbase.mob.TestMobStoreCompaction
[INFO] Running org.apache.hadoop.hbase.mob.TestMobCompactionWithDefaults
[INFO] Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
208.078 s - in org.apache.hadoop.hbase.regionserver.TestCompaction
[INFO] Running
org.apache.hadoop.hbase.mob.TestMobCompactionRegularRegionBatchMode
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
115.164 s - in org.apache.hadoop.hbase.rsgroup.TestRSGroupMajorCompactionTTL
[INFO] Running
org.apache.hadoop.hbase.mob.TestMobCompactionOptRegionBatchMode
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
187.897 s - in org.apache.hadoop.hbase.mob.TestMobCompactionWithDefaults
[INFO] Running org.apache.hadoop.hbase.mob.TestMobCompactionOptMode
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
179.166 s - in org.apache.hadoop.hbase.mob.TestMobCompactionOptMode
[INFO] Running
org.apache.hadoop.hbase.client.TestAsyncTableGetMultiThreadedWithBasicCompaction
[WARNING] Tests run: 1, Failures: 0, Errors: 0, Skipped: 1, Time
elapsed: 0.022 s - in
org.apache.hadoop.hbase.client.TestAsyncTableGetMultiThreadedWithBasicCompaction
[INFO] Running
org.apache.hadoop.hbase.client.TestMobRestoreSnapshotFromClientGetCompactionState
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
290.012 s - in org.apache.hadoop.hbase.mob.TestMobCompactionOptRegionBatchMode
[INFO] Running
org.apache.hadoop.hbase.client.TestAsyncTableGetMultiThreadedWithEagerCompaction
[WARNING] Tests run: 1, Failures: 0, Errors: 0, Skipped: 1, Time
elapsed: 0.026 s - in
org.apache.hadoop.hbase.client.TestAsyncTableGetMultiThreadedWithEagerCompaction
[INFO] Running
org.apache.hadoop.hbase.client.TestRestoreSnapshotFromClientGetCompactionState
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
25.904 s - in
org.apache.hadoop.hbase.client.TestMobRestoreSnapshotFromClientGetCompactionState
[INFO] Running
org.apache.hadoop.hbase.TestAcidGuaranteesWithNoInMemCompaction
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
321.549 s - in
org.apache.hadoop.hbase.mob.TestMobCompactionRegularRegionBatchMode
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
31.118 s - in
org.apache.hadoop.hbase.client.TestRestoreSnapshotFromClientGetCompactionState
[INFO] Tests run: 27, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
688.559 s - in org.apache.hadoop.hbase.regionserver.TestMajorCompaction
[INFO] Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
201.193 s - in org.apache.hadoop.hbase.TestAcidGuaranteesWithNoInMemCompaction
[INFO] Results:
[WARNING] Tests run: 181, Failures: 0, Errors: 0, Skipped: 3
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
