[
https://issues.apache.org/jira/browse/ACCUMULO-2166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13867105#comment-13867105
]
Keith Turner commented on ACCUMULO-2166:
----------------------------------------
How much memory are you giving the tserver jvm? Seeing the log sort messages
makes me wonder if tserver.sort.buffer.size needs to be adjusted.
> tserver recovering tablet on startup fails with "Garbage collection may be
> interfering with lock keep-alive"
> ------------------------------------------------------------------------------------------------------------
>
> Key: ACCUMULO-2166
> URL: https://issues.apache.org/jira/browse/ACCUMULO-2166
> Project: Accumulo
> Issue Type: Bug
> Reporter: Ben Popp
>
> on a locally built 1.6.0-SNAPSHOT based on the SHA
> 417902e218c566333b6ea5ac492186ae305e5e16 (Dec 31st) alongside an apache
> hadoop 1.2.1 install.
> I had a single node deployment on my laptop, which had probably been
> interrupted by my laptop sleep during some ingest.
> When i restarted the tserver, I got a "FATAL: Garbage collection may be
> interfering with lock keep-alive. Halting." log message after some recovery
> attempts and the tserver shut down.
> I restarted and had similar results twice. [~vines] and I tried to attach a
> debugger to the tserver on startup the next time, and recovery completed
> without any failures! I have restarted the tserver since then with no
> issues.
> The only notable thing special about my workspace (since it affects the
> startup paths) is that i'm using some custom Authorizer and Authenticator
> plugins in Accumulo.
> Log excerpts:
> {noformat}
> ==============
> FIRST TIME
> ==============
> 2014-01-09 12:42:58,241 [server.Accumulo] INFO : tserver starting
> ...snip...
> 2014-01-09 12:44:47,820 [tserver.Tablet] INFO : Starting Write-Ahead Log
> recovery for d<;7006d
> 2014-01-09 12:44:47,821 [tserver.TabletServer] INFO : Looking for
> hdfs://localhost:9000/accumulo160b/recovery/2dceae9f-98c2-4f0c-9c66-27f4ab52113a/finished
> 2014-01-09 12:44:47,822 [tserver.TabletServer] INFO : Looking for
> hdfs://localhost:9000/accumulo160b/recovery/61b9709a-c945-41ae-946c-fc6425f4c96a/finished
> 2014-01-09 12:44:47,823 [log.SortedLogRecovery] INFO : Looking at mutations
> from
> hdfs://localhost:9000/accumulo160b/recovery/2dceae9f-98c2-4f0c-9c66-27f4ab52113a
> for d<;7006d
> 2014-01-09 12:44:47,900 [log.SortedLogRecovery] INFO : Looking at mutations
> from
> hdfs://localhost:9000/accumulo160b/recovery/61b9709a-c945-41ae-946c-fc6425f4c96a
> for d<;7006d
> 2014-01-09 12:44:47,990 [log.SortedLogRecovery] INFO : Scanning for mutations
> starting at sequence number 2 for tid 110
> 2014-01-09 12:45:32,510 [util.Halt] FATAL: Garbage collection may be
> interfering with lock keep-alive. Halting.
> ==============
> SECOND TIME
> ==============
> 2014-01-09 13:55:17,890 [server.Accumulo] INFO : tserver starting
> ...snip...
> 2014-01-09 13:55:33,687 [log.LogSorter] INFO : Copying
> hdfs://localhost:9000/accumulo160b/wal/localhost+9997/7e36fa46-95df-4b58-a1e4-6a29f43843a8
> to
> hdfs://localhost:9000/accumulo160b/recovery/7e36fa46-95df-4b58-a1e4-6a29f43843a8
> 2014-01-09 13:55:33,688 [log.LogSorter] INFO : Copying
> hdfs://localhost:9000/accumulo160b/wal/localhost+9997/e3428689-9011-4908-8990-9bd079978983
> to
> hdfs://localhost:9000/accumulo160b/recovery/e3428689-9011-4908-8990-9bd079978983
> 2014-01-09 13:56:31,628 [util.Halt] FATAL: Garbage collection may be
> interfering with lock keep-alive. Halting.
> ==============
> THIRD TIME
> ==============
> 2014-01-09 14:04:16,315 [server.Accumulo] INFO : tserver starting
> ...snip...
> 2014-01-09 14:04:32,078 [log.LogSorter] INFO : Copying
> hdfs://localhost:9000/accumulo160b/wal/localhost+9997/e3428689-9011-4908-8990-9bd079978983
> to
> hdfs://localhost:9000/accumulo160b/recovery/e3428689-9011-4908-8990-9bd079978983
> 2014-01-09 14:04:32,079 [log.LogSorter] INFO : Copying
> hdfs://localhost:9000/accumulo160b/wal/localhost+9997/a7467054-ec80-4734-974d-7c55c201b332
> to
> hdfs://localhost:9000/accumulo160b/recovery/a7467054-ec80-4734-974d-7c55c201b332
> 2014-01-09 14:04:32,370 [util.NativeCodeLoader] WARN : Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> 2014-01-09 14:04:32,811 [log.LogSorter] INFO : Finished log sort
> a7467054-ec80-4734-974d-7c55c201b332 11543822 bytes 1 parts in 732ms
> 2014-01-09 14:04:32,817 [log.LogSorter] INFO : Copying
> hdfs://localhost:9000/accumulo160b/wal/localhost+9997/7e36fa46-95df-4b58-a1e4-6a29f43843a8
> to
> hdfs://localhost:9000/accumulo160b/recovery/7e36fa46-95df-4b58-a1e4-6a29f43843a8
> 2014-01-09 14:05:27,482 [util.Halt] FATAL: Garbage collection may be
> interfering with lock keep-alive. Halting.
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
