[
https://issues.apache.org/jira/browse/ACCUMULO-962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13554168#comment-13554168
]
Eric Newton commented on ACCUMULO-962:
--------------------------------------
Thanks for the logs, Josh.
These same logs contain load failures: the fetch of the !METADATA table
information returns nothing.
For example:
{noformat}
2013-01-13 00:34:01,299 [tabletserver.TabletServer] INFO :
n004/10.10.32.104:9997: got assignment from master:
6;55b22d0e56041914;55a1cac083126f18
2013-01-13 00:34:01,307 [tabletserver.TabletServer] INFO : Reporting tablet
6;55b22d0e56041914;55a1cac083126f18 assignment failure: unable to verify Tablet
Information
2013-01-13 00:34:01,307 [tabletserver.TabletServer] WARN : Failed to find
metadata entry for 6;55b22d0e56041914;55a1cac083126f18 found 6;56d916872b020ccc
{noformat}
It seems as though sometimes a METADATA table scan skips information, or
there's some other kind of conflict going on when this data is read.
It looks like tablet 6;56d916872b020ccc was loaded at roughly the same time as
6;55b22d0e56041914.
> Failed to do close consistency check for tablet
> -----------------------------------------------
>
> Key: ACCUMULO-962
> URL: https://issues.apache.org/jira/browse/ACCUMULO-962
> Project: Accumulo
> Issue Type: Bug
> Components: tserver
> Affects Versions: 1.5.0
> Environment: Small cluster: CentOS 5.7, Cloudera Hadoop 3 update 5
> Reporter: Josh Elser
> Assignee: Eric Newton
> Priority: Minor
>
> After updating Accumulo trunk and deploying to a small cluster, I tend to
> find the following table problem is shown on the monitor. It seems to be a
> false positive, as I can ignore it and everything goes according to plan.
> Some context from the TServer which logged the exception:
> {noformat}
> 2013-01-13 00:39:36,692 [tabletserver.MinorCompactor] DEBUG: Begin minor
> compaction /accumulo/tables/7/t-001gnfo/F001gqrl.rf_tmp
> 7;3de353f7ced916e4;3dd2f1a9fbe76ce8
> 2013-01-13 00:39:36,713 [tabletserver.TabletServer] DEBUG: Got unloadTablet
> message from user: !SYSTEM
> 2013-01-13 00:39:36,714 [tabletserver.TabletServer] DEBUG: UpSess null 9,549
> in 0.153s, at=[0 1 0.01 138] ft=0.103s(pt=0.002s lt=0.051s ct=0.050s)
> 2013-01-13 00:39:36,714 [tabletserver.TabletServer] DEBUG: Failures: 7, first
> extent 7;0f6c8b439581063c;0f5c28f5c28f5c4 successful commits: 0
> 2013-01-13 00:39:36,714 [tabletserver.TabletServer] DEBUG: Got unloadTablet
> message from user: !SYSTEM
> 2013-01-13 00:39:36,714 [tabletserver.TabletServer] DEBUG: Got unloadTablet
> message from user: !SYSTEM
> 2013-01-13 00:39:36,732 [tabletserver.TabletServer] DEBUG: Got unloadTablet
> message from user: !SYSTEM
> 2013-01-13 00:39:36,782 [tabletserver.TabletServer] DEBUG: UpSess null 4,703
> in 0.101s, at=[0 0 0.00 138] ft=0.070s(pt=0.002s lt=0.033s ct=0.035s)
> 2013-01-13 00:39:36,782 [tabletserver.TabletServer] DEBUG: Failures: 7, first
> extent 7;0f6c8b439581063c;0f5c28f5c28f5c4 successful commits: 0
> 2013-01-13 00:39:36,826 [tabletserver.TabletServer] DEBUG: UpSess null 9,513
> in 0.127s, at=[0 0 0.00 138] ft=0.088s(pt=0.003s lt=0.052s ct=0.033s)
> 2013-01-13 00:39:36,826 [tabletserver.TabletServer] DEBUG: Failures: 7, first
> extent 7;0f6c8b439581063c;0f5c28f5c28f5c4 successful commits: 0
> 2013-01-13 00:39:36,863 [tabletserver.TabletServer] DEBUG: UpSess null 4,688
> in 0.071s, at=[0 0 0.00 136] ft=0.048s(pt=0.003s lt=0.022s ct=0.023s)
> 2013-01-13 00:39:36,863 [tabletserver.TabletServer] DEBUG: Failures: 5, first
> extent 7;0f8d4fdf3b645a34;0f7ced916872b038 successful commits: 0
> 2013-01-13 00:39:36,868 [tabletserver.TabletServer] DEBUG: Got unloadTablet
> message from user: !SYSTEM
> 2013-01-13 00:39:36,868 [tabletserver.TabletServer] DEBUG: Got unloadTablet
> message from user: !SYSTEM
> 2013-01-13 00:39:36,868 [tabletserver.TabletServer] DEBUG: Got unloadTablet
> message from user: !SYSTEM
> 2013-01-13 00:39:36,879 [tabletserver.TabletServer] DEBUG: Got unloadTablet
> message from user: !SYSTEM
> 2013-01-13 00:39:36,886 [tabletserver.Compactor] DEBUG: Compaction
> 7;3de353f7ced916e4;3dd2f1a9fbe76ce8 25,236 read | 25,236 written | 141,775
> entries/sec | 0.178 secs
> 2013-01-13 00:39:36,888 [tabletserver.Tablet] DEBUG: Logs for memory
> compacted: 7;3de353f7ced916e4;3dd2f1a9fbe76ce8
> 10.10.32.121+9997/ea6c3aab-02ac-4599-b50e-b00df41887ed
> 2013-01-13 00:39:36,888 [tabletserver.Tablet] DEBUG: Logs for memory
> compacted: 7;3de353f7ced916e4;3dd2f1a9fbe76ce8
> 10.10.32.121+9997/d1aaf814-6209-46f3-8997-36541ea0dcc2
> 2013-01-13 00:39:36,888 [tabletserver.Tablet] DEBUG: Logs for current memory:
> 7;3de353f7ced916e4;3dd2f1a9fbe76ce8
> 10.10.32.121+9997/ea6c3aab-02ac-4599-b50e-b00df41887ed
> 2013-01-13 00:39:36,895 [log.TabletServerLogger] DEBUG: wrote MinC finish
> 11312: writeTime:3ms
> 2013-01-13 00:39:36,895 [tabletserver.Tablet] TABLET_HIST:
> 7;3de353f7ced916e4;3dd2f1a9fbe76ce8 MinC [memory] -> /t-001gnfo/F001gqrl.rf
> 2013-01-13 00:39:36,895 [tabletserver.Tablet] DEBUG: MinC finish lock 0.00
> secs 7;3de353f7ced916e4;3dd2f1a9fbe76ce8
> 2013-01-13 00:39:36,895 [tabletserver.NativeMap] DEBUG: Deallocating native
> map 0x00002aaab8f40540
> 2013-01-13 00:39:36,903 [tabletserver.Tablet] DEBUG:
> completeClose(saveState=true completeClose=true)
> 7;3de353f7ced916e4;3dd2f1a9fbe76ce8
> 2013-01-13 00:39:36,904 [tabletserver.NativeMap] DEBUG: Allocated native map
> 0x00002aaabc85e210
> 2013-01-13 00:39:36,904 [tabletserver.MinorCompactor] DEBUG: Begin minor
> compaction /accumulo/tables/7/t-001gnfo/F001gqrm.rf_tmp
> 7;3de353f7ced916e4;3dd2f1a9fbe76ce8
> 2013-01-13 00:39:36,919 [tabletserver.LargestFirstMemoryManager] DEBUG:
> BEFORE compactionThreshold = 0.851 maxObserved = 731,712,351
> 2013-01-13 00:39:36,919 [tabletserver.LargestFirstMemoryManager] DEBUG: AFTER
> compactionThreshold = 0.851
> 2013-01-13 00:39:36,935 [tabletserver.Compactor] DEBUG: Compaction
> 7;3de353f7ced916e4;3dd2f1a9fbe76ce8 138 read | 138 written | 69,000
> entries/sec | 0.002 secs
> 2013-01-13 00:39:36,938 [tabletserver.Tablet] DEBUG: Logs for memory
> compacted: 7;3de353f7ced916e4;3dd2f1a9fbe76ce8
> 10.10.32.121+9997/ea6c3aab-02ac-4599-b50e-b00df41887ed
> 2013-01-13 00:39:36,948 [log.TabletServerLogger] DEBUG: wrote MinC finish
> 11314: writeTime:0ms
> 2013-01-13 00:39:36,948 [tabletserver.Tablet] TABLET_HIST:
> 7;3de353f7ced916e4;3dd2f1a9fbe76ce8 MinC [memory] -> /t-001gnfo/F001gqrm.rf
> 2013-01-13 00:39:36,949 [tabletserver.Tablet] DEBUG: MinC finish lock 0.00
> secs 7;3de353f7ced916e4;3dd2f1a9fbe76ce8
> 2013-01-13 00:39:36,949 [tabletserver.NativeMap] DEBUG: Deallocating native
> map 0x00002aaabd00fd10
> 2013-01-13 00:39:36,979 [tabletserver.Tablet] ERROR: Failed to do close
> consistency check for tablet 7;3de353f7ced916e4;3dd2f1a9fbe76ce8
> java.lang.RuntimeException: Unexpected row 7;3df3b645a1cac0e expected
> 7;3de353f7ced916e4
> at
> org.apache.accumulo.server.util.MetadataTable.getFileAndLogEntries(MetadataTable.java:838)
> at
> org.apache.accumulo.server.tabletserver.Tablet.closeConsistencyCheck(Tablet.java:2761)
> at
> org.apache.accumulo.server.tabletserver.Tablet.completeClose(Tablet.java:2714)
> at
> org.apache.accumulo.server.tabletserver.Tablet.close(Tablet.java:2592)
> at
> org.apache.accumulo.server.tabletserver.TabletServer$UnloadTabletHandler.run(TabletServer.java:2350)
> at
> org.apache.accumulo.core.util.LoggingRunnable.run(LoggingRunnable.java:34)
> at
> org.apache.accumulo.cloudtrace.instrument.TraceRunnable.run(TraceRunnable.java:47)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at
> org.apache.accumulo.cloudtrace.instrument.TraceRunnable.run(TraceRunnable.java:47)
> at
> org.apache.accumulo.core.util.LoggingRunnable.run(LoggingRunnable.java:34)
> at java.lang.Thread.run(Thread.java:662)
> {noformat}
> I can make the entire collection of logs available too if necessary.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
