[
https://issues.apache.org/jira/browse/ACCUMULO-3787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Newton resolved ACCUMULO-3787.
-----------------------------------
Resolution: Fixed
> Multiple locations for Metadata tablet
> --------------------------------------
>
> Key: ACCUMULO-3787
> URL: https://issues.apache.org/jira/browse/ACCUMULO-3787
> Project: Accumulo
> Issue Type: Bug
> Components: master, tserver
> Reporter: Josh Elser
> Assignee: Josh Elser
> Priority: Blocker
> Labels: 1.7.0_QA
> Fix For: 1.8.0
>
>
> Running continuous ingest with agitation
> {noformat}
> 2015-05-08 23:59:44,381 [state.ZooTabletStateStore] DEBUG: Returning root
> tablet state:
> +r<<@(null,c6402.ambari.apache.org:9997[14d34e748060089],c6402.ambari.apache.org:9997[14d34e748060089])
> 2015-05-08 23:59:44,381 [master.Master] DEBUG: Root Table location State:
> +r<<@(null,c6402.ambari.apache.org:9997[14d34e748060089],c6402.ambari.apache.org:9997[14d34e748060089])
> 2015-05-08 23:59:44,381 [master.Master] DEBUG: [Root Table]: scan time 0.00
> seconds
> 2015-05-08 23:59:44,381 [master.Master] DEBUG: [Root Table] sleeping for
> 60.00 seconds
> 2015-05-08 23:59:44,412 [master.Master] ERROR: Error processing table state
> for store Normal Tablets
> java.lang.RuntimeException: java.lang.IllegalStateException: Tablet has
> multiple locations : !0<
> at
> org.apache.accumulo.server.master.state.MetaDataTableScanner.<init>(MetaDataTableScanner.java:72)
> at
> org.apache.accumulo.server.master.state.MetaDataStateStore.iterator(MetaDataStateStore.java:70)
> at
> org.apache.accumulo.master.TabletGroupWatcher.run(TabletGroupWatcher.java:173)
> Caused by: java.lang.IllegalStateException: Tablet has multiple locations :
> !0<
> at
> org.apache.accumulo.core.metadata.MetadataLocationObtainer.getMetadataLocationEntries(MetadataLocationObtainer.java:227)
> at
> org.apache.accumulo.core.metadata.MetadataLocationObtainer.lookupTablet(MetadataLocationObtainer.java:114)
> at
> org.apache.accumulo.core.client.impl.TabletLocatorImpl.lookupTabletLocation(TabletLocatorImpl.java:463)
> at
> org.apache.accumulo.core.client.impl.TabletLocatorImpl.lookupTabletLocationAndCheckLock(TabletLocatorImpl.java:634)
> at
> org.apache.accumulo.core.client.impl.TabletLocatorImpl._locateTablet(TabletLocatorImpl.java:625)
> at
> org.apache.accumulo.core.client.impl.TabletLocatorImpl.binRanges(TabletLocatorImpl.java:280)
> at
> org.apache.accumulo.core.client.impl.TabletLocatorImpl.binRanges(TabletLocatorImpl.java:355)
> at
> org.apache.accumulo.core.client.impl.TimeoutTabletLocator.binRanges(TimeoutTabletLocator.java:100)
> at
> org.apache.accumulo.core.client.impl.TabletServerBatchReaderIterator.binRanges(TabletServerBatchReaderIterator.java:233)
> at
> org.apache.accumulo.core.client.impl.TabletServerBatchReaderIterator.lookup(TabletServerBatchReaderIterator.java:220)
> at
> org.apache.accumulo.core.client.impl.TabletServerBatchReaderIterator.<init>(TabletServerBatchReaderIterator.java:154)
> at
> org.apache.accumulo.core.client.impl.TabletServerBatchReader.iterator(TabletServerBatchReader.java:115)
> at
> org.apache.accumulo.server.master.state.MetaDataTableScanner.<init>(MetaDataTableScanner.java:66)
> ... 2 more
> 2015-05-08 23:59:44,412 [master.Master] DEBUG: Metadata Tablets location
> State:
> !0;~<@(null,c6402.ambari.apache.org:9997[14d34e748060080],c6402.ambari.apache.org:9997[14d34e748060080])
> 2015-05-08 23:59:44,439 [master.Master] ERROR: Error processing table state
> for store Metadata Tablets
> java.lang.RuntimeException:
> org.apache.accumulo.server.master.state.TabletLocationState$BadLocationStateException:
> found two locations for the same extent !0<:
> c6401.ambari.apache.org:9997[14d34e748060065] and
> c6402.ambari.apache.org:9997[14d34e748060080]
> at
> org.apache.accumulo.server.master.state.MetaDataTableScanner.fetch(MetaDataTableScanner.java:191)
> at
> org.apache.accumulo.server.master.state.MetaDataTableScanner.next(MetaDataTableScanner.java:130)
> at
> org.apache.accumulo.server.master.state.MetaDataTableScanner.next(MetaDataTableScanner.java:49)
> at
> org.apache.accumulo.master.TabletGroupWatcher.run(TabletGroupWatcher.java:175)
> Caused by:
> org.apache.accumulo.server.master.state.TabletLocationState$BadLocationStateException:
> found two locations for the same extent !0<:
> c6401.ambari.apache.org:9997[14d34e748060065] and
> c6402.ambari.apache.org:9997[14d34e748060080]
> at
> org.apache.accumulo.server.master.state.MetaDataTableScanner.createTabletLocationState(MetaDataTableScanner.java:160)
> at
> org.apache.accumulo.server.master.state.MetaDataTableScanner.fetch(MetaDataTableScanner.java:187)
> ... 3 more
> 2015-05-08 23:59:44,439 [master.Master] DEBUG: Attempting repair on !0<
> 2015-05-08 23:59:44,450 [master.Master] WARN : Found a tablet hosted on
> multiple servers, attempting to repair
> 2015-05-08 23:59:44,451 [master.Master] ERROR: Metadata table is inconsistent
> at !0< and all assigned/future tservers are still online.
> 2015-05-08 23:59:44,577 [master.Master] DEBUG: Metadata Tablets location
> State:
> !0;~<@(null,c6402.ambari.apache.org:9997[14d34e748060080],c6402.ambari.apache.org:9997[14d34e748060080])
> 2015-05-08 23:59:44,590 [master.Master] ERROR: Error processing table state
> for store Metadata Tablets
> java.lang.RuntimeException:
> org.apache.accumulo.server.master.state.TabletLocationState$BadLocationStateException:
> found two locations for the same extent !0<:
> c6401.ambari.apache.org:9997[14d34e748060065] and
> c6402.ambari.apache.org:9997[14d34e748060080]
> at
> org.apache.accumulo.server.master.state.MetaDataTableScanner.fetch(MetaDataTableScanner.java:191)
> at
> org.apache.accumulo.server.master.state.MetaDataTableScanner.next(MetaDataTableScanner.java:130)
> at
> org.apache.accumulo.server.master.state.MetaDataTableScanner.next(MetaDataTableScanner.java:49)
> at
> org.apache.accumulo.master.TabletGroupWatcher.run(TabletGroupWatcher.java:175)
> Caused by:
> org.apache.accumulo.server.master.state.TabletLocationState$BadLocationStateException:
> found two locations for the same extent !0<:
> c6401.ambari.apache.org:9997[14d34e748060065] and
> c6402.ambari.apache.org:9997[14d34e748060080]
> at
> org.apache.accumulo.server.master.state.MetaDataTableScanner.createTabletLocationState(MetaDataTableScanner.java:160)
> at
> org.apache.accumulo.server.master.state.MetaDataTableScanner.fetch(MetaDataTableScanner.java:187)
> ... 3 more
> {noformat}
> Agitation had stopped all of Accumulo. When I brought it back up, root came
> back online but metadata never did due to the above issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
