EdColeman opened a new issue #1598:
URL: https://github.com/apache/accumulo/issues/1598
Recent Jenkins build failed - it looks like there was resource contention.
The current test spins up multiple compaction (4) and the monitors the metrics
for the FATE transactions to start. It looks like the FATE transactions are
queued, but never run. The 4 transactions were used during development to
drive interesting behaviour, but are not strictly necessary for a functional
check and the number of simultaneous running transactions can be reduced.
`Compactions launched:
2020-04-27T13:16:46,388` [util.SlowOps] DEBUG: Compaction running for
MasterMetricsIT_compactionMetrics0_0
2020-04-27T13:16:46,388 [util.SlowOps] DEBUG: Compaction running for
MasterMetricsIT_compactionMetrics0_1
2020-04-27T13:16:46,388 [util.SlowOps] DEBUG: Compaction running for
MasterMetricsIT_compactionMetrics0_2
2020-04-27T13:16:49,393 [util.SlowOps] DEBUG: Compaction running for
MasterMetricsIT_compactionMetrics0_3
2020-04-27T13:16:49,393 [functional.MasterMetricsIT] INFO : Received metrics
LineUpdate{lastUpdate=26017212078, line='1587993403681 master.fate:
Context=master, ProcessName=Master, Hostname=jenkins.revelc.net,
currentFateOps=0, totalFateOps=2, totalZkConnErrors=0, FateTxState_NEW=0,
FateTxState_IN_PROGRESS=0, FateTxState_FAILED_IN_PROGRESS=0,
FateTxState_FAILED=0, FateTxState_SUCCESSFUL=0, FateTxState_UNKNOWN=0'}
When the test timed out, the FATEs still had not appeared:
2020-04-27T13:20:04,412 [functional.MasterMetricsIT] INFO : Received metrics
update LineUpdate{lastUpdate=221069792248, line='1587993598681 master.fate:
Context=master, ProcessName=Master, Hostname=jenkins.revelc.net,
currentFateOps=0, totalFateOps=2, totalZkConnErrors=0, FateTxState_NEW=0,
FateTxState_IN_PROGRESS=0, FateTxState_FAILED_IN_PROGRESS=0,
FateTxState_FAILED=0, FateTxState_SUCCESSFUL=0, FateTxState_UNKNOWN=0'}
2020-04-27T13:20:11,758 [miniclusterImpl.MiniAccumuloClusterImpl] WARN :
Unexpectedly had 4 task(s) remaining in threadpool for execution when being
stopped
`
The timestamps in the metrics show the metrics are updating. There was some
FATE activity that occurred - however, the running compactions never appeared -
likely due to resource contention.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
