Jim I suspect your installation used files in /tmp that have been cleaned up. Look at your hdfs-site.xml and ensure the dfs.name.dir and dfs.data.dir are not in /tmp. Then look at your zoo.cfg directory and ensure the dataDir is defined and not /tmp.
Mike On Tue, May 23, 2017 at 2:02 AM o haya <oh...@yahoo.com> wrote: > Hi, > > I noticed that in the monitor_nodejs.whatever.com.debug.log file I am > seeing these messages: > > 2017-05-23 01:56:51,035 [impl.ThriftScanner] DEBUG: Failed to locate > tablet for table : !0 row : ~err_ > 2017-05-23 01:56:57,173 [monitor.Monitor] INFO : Failed to obtain problem > reports > java.lang.RuntimeException: > org.apache.accumulo.core.client.impl.ThriftScanner$ScanTimedOutException > at > org.apache.accumulo.core.client.impl.ScannerIterator.hasNext(ScannerIterator.java:160) > at > org.apache.accumulo.server.problems.ProblemReports$3.hasNext(ProblemReports.java:252) > at > org.apache.accumulo.server.problems.ProblemReports.summarize(ProblemReports.java:310) > at org.apache.accumulo.monitor.Monitor.fetchData(Monitor.java:374) > at org.apache.accumulo.monitor.Monitor$2.run(Monitor.java:519) > at > org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35) > at java.lang.Thread.run(Thread.java:745) > Caused by: > org.apache.accumulo.core.client.impl.ThriftScanner$ScanTimedOutException > at > org.apache.accumulo.core.client.impl.ThriftScanner.scan(ThriftScanner.java:243) > at > org.apache.accumulo.core.client.impl.ScannerIterator$Reader.run(ScannerIterator.java:79) > at > org.apache.accumulo.core.client.impl.ScannerIterator.hasNext(ScannerIterator.java:150) > ... 6 more > 2017-05-23 01:56:57,512 [impl.ThriftScanner] DEBUG: Failed to locate > tablet for table : !0 row : ~err_ > 2017-05-23 01:57:03,604 [monitor.Monitor] INFO : Failed to obtain problem > reports > java.lang.RuntimeException: > org.apache.accumulo.core.client.impl.ThriftScanner$ScanTimedOutException > at > org.apache.accumulo.core.client.impl.ScannerIterator.hasNext(ScannerIterator.java:160) > at > org.apache.accumulo.server.problems.ProblemReports$3.hasNext(ProblemReports.java:252) > at > org.apache.accumulo.server.problems.ProblemReports.summarize(ProblemReports.java:310) > at org.apache.accumulo.monitor.Monitor.fetchData(Monitor.java:374) > at org.apache.accumulo.monitor.Monitor$2.run(Monitor.java:519) > at > org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35) > at java.lang.Thread.run(Thread.java:745) > Caused by: > org.apache.accumulo.core.client.impl.ThriftScanner$ScanTimedOutException > at > org.apache.accumulo.core.client.impl.ThriftScanner.scan(ThriftScanner.java:243) > at > org.apache.accumulo.core.client.impl.ScannerIterator$Reader.run(ScannerIterator.java:79) > at > org.apache.accumulo.core.client.impl.ScannerIterator.hasNext(ScannerIterator.java:150) > ... 6 more > 2017-05-23 01:57:03,944 [impl.ThriftScanner] DEBUG: Failed to locate > tablet for table : !0 row : ~err_ > > > > > -------------------------------------------- > On Tue, 5/23/17, o haya <oh...@yahoo.com> wrote: > > Subject: scan of small table stopped working (hangs) > To: user@accumulo.apache.org > Date: Tuesday, May 23, 2017, 1:56 AM > > Hi, > > I have a small Accumulo test instance, > that I installed following this page: > > > https://www.digitalocean.com/community/tutorials/how-to-install-the-big-data-friendly-apache-accumulo-nosql-database-on-ubuntu-14-04 > > I've been doing testing and was able to > successfully run the SimpleIngestClient from the minicluster > project and create a table, testTable. > > After that, I could use the Accumulo > shell and scan the table, after logging in as a user, "demo" > and running "scan -t testTable". > > However, tonight, after I brought up > the machine and started Hadoop > (/apps/hadoop/sbin/start-dfs.sh), then zookeeper > (/apps/zookeeper/bin/zkServer.sh start) and finally, > Accumulo (/apps/accumulo/bin/start-all.sh). > > And then I started the Accumulo shell > and logged in as "demo" and did "scan -t testTable" and it > just hangs, with no output. > > I found another thread that suggested > enabling "debug on" and here is the output: > > root@accumulo1> > debug on > 2017-05-23 01:44:37,524 > [admin.TableOperations] TRACE: tid=13 Fetching list of > tables... > 2017-05-23 01:44:37,526 > [admin.TableOperations] TRACE: tid=13 Fetched 5 table names > in 0.001 secs > 2017-05-23 01:44:37,526 > [rpc.ThriftUtil] TRACE: Opening normal transport > 2017-05-23 01:44:37,532 > [impl.ThriftTransportPool] TRACE: Creating new connection to > connection to localhost:9997 > 2017-05-23 01:44:37,534 > [impl.ThriftTransportPool] TRACE: Returned connection > localhost:9997 (120000) ioCount: 170 > 2017-05-23 01:44:37,534 > [admin.TableOperations] TRACE: tid=13 Fetching list of > namespaces... > 2017-05-23 01:44:37,534 > [admin.TableOperations] TRACE: tid=13 Fetched 2 namespaces > in 0.000 secs > root@accumulo1> > scan -t testTable > 2017-05-23 01:44:51,547 > [admin.TableOperations] TRACE: tid=13 Checking if table > testTable exists... > 2017-05-23 01:44:51,547 > [admin.TableOperations] TRACE: tid=13 Checked existance of > true in 0.000 secs > 2017-05-23 01:44:51,548 > [rpc.ThriftUtil] TRACE: Opening normal transport > 2017-05-23 01:44:51,549 > [impl.ThriftTransportPool] TRACE: Creating new connection to > connection to localhost:9997 > 2017-05-23 01:44:51,565 > [impl.ThriftTransportPool] TRACE: Returned connection > localhost:9997 (120000) ioCount: 6808 > 2017-05-23 01:44:51,565 > [admin.TableOperations] TRACE: tid=13 Checking if table > testTable exists... > 2017-05-23 01:44:51,566 > [admin.TableOperations] TRACE: tid=13 Checked existance of > true in 0.000 secs > 2017-05-23 01:44:51,566 > [impl.ThriftTransportPool] TRACE: Using existing connection > to localhost:9997 > 2017-05-23 01:44:51,582 > [impl.ThriftTransportPool] TRACE: Returned connection > localhost:9997 (120000) ioCount: 6808 > 2017-05-23 01:44:51,583 > [impl.ThriftTransportPool] TRACE: Using existing connection > to localhost:9997 > 2017-05-23 01:44:51,594 > [impl.ThriftTransportPool] TRACE: Returned connection > localhost:9997 (120000) ioCount: 6808 > 2017-05-23 01:44:51,594 > [admin.TableOperations] TRACE: tid=13 Checking if table > testTable exists... > 2017-05-23 01:44:51,594 > [admin.TableOperations] TRACE: tid=13 Checked existance of > true in 0.000 secs > 2017-05-23 01:44:51,594 > [impl.ThriftTransportPool] TRACE: Using existing connection > to localhost:9997 > 2017-05-23 01:44:51,608 > [impl.ThriftTransportPool] TRACE: Returned connection > localhost:9997 (120000) ioCount: 6808 > 2017-05-23 01:44:51,609 > [impl.ThriftTransportPool] TRACE: Using existing connection > to localhost:9997 > 2017-05-23 01:44:51,614 > [impl.ThriftTransportPool] TRACE: Returned connection > localhost:9997 (120000) ioCount: 130 > 2017-05-23 01:44:51,620 [shell.Shell] > DEBUG: Found no scan iterators to set > 2017-05-23 01:44:51,644 > [impl.TabletLocatorImpl] TRACE: tid=13 Locating tablet > table=7 row= skipRow=false retry=false > 2017-05-23 01:44:51,644 > [impl.TabletLocatorImpl] TRACE: tid=13 Locating tablet > table=!0 row=7; skipRow=false retry=false > 2017-05-23 01:44:51,644 > [impl.RootTabletLocator] TRACE: tid=13 Looking up root > tablet location in zookeeper. > 2017-05-23 01:44:51,658 > [impl.RootTabletLocator] TRACE: tid=13 Found root tablet at > null in 0.014 secs > 2017-05-23 01:44:51,659 > [impl.TabletLocatorImpl] TRACE: tid=13 Located tablet null > at null in 0.014 secs > 2017-05-23 01:44:51,659 > [impl.TabletLocatorImpl] TRACE: tid=13 Located tablet null > at null in 0.015 secs > 2017-05-23 01:44:51,659 > [impl.ThriftScanner] DEBUG: Failed to locate tablet for > table : 7 row : > 2017-05-23 01:44:51,759 > [impl.TabletLocatorImpl] TRACE: tid=13 Locating tablet > table=7 row= skipRow=false retry=false > 2017-05-23 01:44:51,759 > [impl.TabletLocatorImpl] TRACE: tid=13 Locating tablet > table=!0 row=7; skipRow=false retry=false > 2017-05-23 01:44:51,759 > [impl.RootTabletLocator] TRACE: tid=13 Looking up root > tablet location in zookeeper. > 2017-05-23 01:44:51,759 > [impl.RootTabletLocator] TRACE: tid=13 Found root tablet at > null in 0.000 secs > 2017-05-23 01:44:51,759 > [impl.TabletLocatorImpl] TRACE: tid=13 Located tablet null > at null in 0.000 secs > 2017-05-23 01:44:51,760 > [impl.TabletLocatorImpl] TRACE: tid=13 Located tablet null > at null in 0.000 secs > 2017-05-23 01:44:51,760 > [impl.ThriftScanner] TRACE: Failed to locate tablet for > table : 7 row : > 2017-05-23 01:44:51,952 > [impl.TabletLocatorImpl] TRACE: tid=13 Locating tablet > table=7 row= skipRow=false retry=false > 2017-05-23 01:44:51,952 > [impl.TabletLocatorImpl] TRACE: tid=13 Locating tablet > table=!0 row=7; skipRow=false retry=false > 2017-05-23 01:44:51,952 > [impl.RootTabletLocator] TRACE: tid=13 Looking up root > tablet location in zookeeper. > 2017-05-23 01:44:51,952 > [impl.RootTabletLocator] TRACE: tid=13 Found root tablet at > null in 0.000 secs > 2017-05-23 01:44:51,952 > [impl.TabletLocatorImpl] TRACE: tid=13 Located tablet null > at null in 0.000 secs > 2017-05-23 01:44:51,953 > [impl.TabletLocatorImpl] TRACE: tid=13 Located tablet null > at null in 0.000 secs > 2017-05-23 01:44:51,953 > [impl.ThriftScanner] TRACE: Failed to locate tablet for > table : 7 row : > 2017-05-23 01:44:52,302 > [impl.TabletLocatorImpl] TRACE: tid=13 Locating tablet > table=7 row= skipRow=false retry=false > 2017-05-23 01:44:52,302 > [impl.TabletLocatorImpl] TRACE: tid=13 Locating tablet > table=!0 row=7; skipRow=false retry=false > 2017-05-23 01:44:52,302 > [impl.RootTabletLocator] TRACE: tid=13 Looking up root > tablet location in zookeeper. > 2017-05-23 01:44:52,303 > [impl.RootTabletLocator] TRACE: tid=13 Found root tablet at > null in 0.000 secs > 2017-05-23 01:44:52,303 > [impl.TabletLocatorImpl] TRACE: tid=13 Located tablet null > at null in 0.000 secs > 2017-05-23 01:44:52,303 > [impl.TabletLocatorImpl] TRACE: tid=13 Located tablet null > at null in 0.001 secs > 2017-05-23 01:44:52,303 > [impl.ThriftScanner] TRACE: Failed to locate tablet for > table : 7 row : > 2017-05-23 01:44:52,960 > [impl.TabletLocatorImpl] TRACE: tid=13 Locating tablet > table=7 row= skipRow=false retry=false > 2017-05-23 01:44:52,960 > [impl.TabletLocatorImpl] TRACE: tid=13 Locating tablet > table=!0 row=7; skipRow=false retry=false > 2017-05-23 01:44:52,960 > [impl.RootTabletLocator] TRACE: tid=13 Looking up root > tablet location in zookeeper. > 2017-05-23 01:44:52,960 > [impl.RootTabletLocator] TRACE: tid=13 Found root tablet at > null in 0.000 secs > 2017-05-23 01:44:52,960 > [impl.TabletLocatorImpl] TRACE: tid=13 Located tablet null > at null in 0.000 secs > 2017-05-23 01:44:52,960 > [impl.TabletLocatorImpl] TRACE: tid=13 Located tablet null > at null in 0.000 secs > 2017-05-23 01:44:52,960 > [impl.ThriftScanner] TRACE: Failed to locate tablet for > table : 7 row : > 2017-05-23 01:44:54,282 > [impl.TabletLocatorImpl] TRACE: tid=13 Locating tablet > table=7 row= skipRow=false retry=false > 2017-05-23 01:44:54,282 > [impl.TabletLocatorImpl] TRACE: tid=13 Locating tablet > table=!0 row=7; skipRow=false retry=false > 2017-05-23 01:44:54,282 > [impl.RootTabletLocator] TRACE: tid=13 Looking up root > tablet location in zookeeper. > 2017-05-23 01:44:54,282 > [impl.RootTabletLocator] TRACE: tid=13 Found root tablet at > null in 0.000 secs > 2017-05-23 01:44:54,282 > [impl.TabletLocatorImpl] TRACE: tid=13 Located tablet null > at null in 0.000 secs > 2017-05-23 01:44:54,282 > [impl.TabletLocatorImpl] TRACE: tid=13 Located tablet null > at null in 0.000 secs > 2017-05-23 01:44:54,282 > [impl.ThriftScanner] TRACE: Failed to locate tablet for > table : 7 row : > 2017-05-23 01:44:56,898 > [impl.TabletLocatorImpl] TRACE: tid=13 Locating tablet > table=7 row= skipRow=false retry=false > 2017-05-23 01:44:56,898 > [impl.TabletLocatorImpl] TRACE: tid=13 Locating tablet > table=!0 row=7; skipRow=false retry=false > 2017-05-23 01:44:56,898 > [impl.RootTabletLocator] TRACE: tid=13 Looking up root > tablet location in zookeeper. > 2017-05-23 01:44:56,898 > [impl.RootTabletLocator] TRACE: tid=13 Found root tablet at > null in 0.000 secs > 2017-05-23 01:44:56,898 > [impl.TabletLocatorImpl] TRACE: tid=13 Located tablet null > at null in 0.000 secs > 2017-05-23 01:44:56,898 > [impl.TabletLocatorImpl] TRACE: tid=13 Located tablet null > at null in 0.000 secs > 2017-05-23 01:44:56,898 > [impl.ThriftScanner] TRACE: Failed to locate tablet for > table : 7 row : > 2017-05-23 01:45:02,204 > [impl.TabletLocatorImpl] TRACE: tid=13 Locating tablet > table=7 row= skipRow=false retry=false > 2017-05-23 01:45:02,204 > [impl.TabletLocatorImpl] TRACE: tid=13 Locating tablet > table=!0 row=7; skipRow=false retry=false > 2017-05-23 01:45:02,204 > [impl.RootTabletLocator] TRACE: tid=13 Looking up root > tablet location in zookeeper. > 2017-05-23 01:45:02,204 > [impl.RootTabletLocator] TRACE: tid=13 Found root tablet at > null in 0.000 secs > 2017-05-23 01:45:02,204 > [impl.TabletLocatorImpl] TRACE: tid=13 Located tablet null > at null in 0.000 secs > 2017-05-23 01:45:02,205 > [impl.TabletLocatorImpl] TRACE: tid=13 Located tablet null > at null in 0.000 secs > 2017-05-23 01:45:02,205 > [impl.ThriftScanner] TRACE: Failed to locate tablet for > table : 7 row : > 2017-05-23 01:45:07,560 > [impl.TabletLocatorImpl] TRACE: tid=13 Locating tablet > table=7 row= skipRow=false retry=false > 2017-05-23 01:45:07,560 > [impl.TabletLocatorImpl] TRACE: tid=13 Locating tablet > table=!0 row=7; skipRow=false retry=false > 2017-05-23 01:45:07,560 > [impl.RootTabletLocator] TRACE: tid=13 Looking up root > tablet location in zookeeper. > 2017-05-23 01:45:07,560 > [impl.RootTabletLocator] TRACE: tid=13 Found root tablet at > null in 0.000 secs > 2017-05-23 01:45:07,561 > [impl.TabletLocatorImpl] TRACE: tid=13 Located tablet null > at null in 0.000 secs > 2017-05-23 01:45:07,561 > [impl.TabletLocatorImpl] TRACE: tid=13 Located tablet null > at null in 0.000 secs > 2017-05-23 01:45:07,561 > [impl.ThriftScanner] TRACE: Failed to locate tablet for > table : 7 row : > 2017-05-23 01:45:13,002 > [impl.TabletLocatorImpl] TRACE: tid=13 Locating tablet > table=7 row= skipRow=false retry=false > 2017-05-23 01:45:13,002 > [impl.TabletLocatorImpl] TRACE: tid=13 Locating tablet > table=!0 row=7; skipRow=false retry=false > 2017-05-23 01:45:13,002 > [impl.RootTabletLocator] TRACE: tid=13 Looking up root > tablet location in zookeeper. > 2017-05-23 01:45:13,002 > [impl.RootTabletLocator] TRACE: tid=13 Found root tablet at > null in 0.000 secs > 2017-05-23 01:45:13,002 > [impl.TabletLocatorImpl] TRACE: tid=13 Located tablet null > at null in 0.000 secs > 2017-05-23 01:45:13,002 > [impl.TabletLocatorImpl] TRACE: tid=13 Located tablet null > at null in 0.000 secs > 2017-05-23 01:45:13,002 > [impl.ThriftScanner] TRACE: Failed to locate tablet for > table : 7 row : > 2017-05-23 01:45:17,530 > [impl.TabletLocatorImpl] TRACE: tid=13 Locating tablet > table=7 row= skipRow=false retry=false > 2017-05-23 01:45:17,530 > [impl.TabletLocatorImpl] TRACE: tid=13 Locating tablet > table=!0 row=7; skipRow=false retry=false > 2017-05-23 01:45:17,530 > [impl.RootTabletLocator] TRACE: tid=13 Looking up root > tablet location in zookeeper. > 2017-05-23 01:45:17,530 > [impl.RootTabletLocator] TRACE: tid=13 Found root tablet at > null in 0.000 secs > 2017-05-23 01:45:17,530 > [impl.TabletLocatorImpl] TRACE: tid=13 Located tablet null > at null in 0.000 secs > 2017-05-23 01:45:17,530 > [impl.TabletLocatorImpl] TRACE: tid=13 Located tablet null > at null in 0.000 secs > 2017-05-23 01:45:17,530 > [impl.ThriftScanner] TRACE: Failed to locate tablet for > table : 7 row : > 2017-05-23 01:45:22,802 > [impl.TabletLocatorImpl] TRACE: tid=13 Locating tablet > table=7 row= skipRow=false retry=false > 2017-05-23 01:45:22,803 > [impl.TabletLocatorImpl] TRACE: tid=13 Locating tablet > table=!0 row=7; skipRow=false retry=false > 2017-05-23 01:45:22,803 > [impl.RootTabletLocator] TRACE: tid=13 Looking up root > tablet location in zookeeper. > 2017-05-23 01:45:22,803 > [impl.RootTabletLocator] TRACE: tid=13 Found root tablet at > null in 0.000 secs > 2017-05-23 01:45:22,803 > [impl.TabletLocatorImpl] TRACE: tid=13 Located tablet null > at null in 0.000 secs > 2017-05-23 01:45:22,803 > [impl.TabletLocatorImpl] TRACE: tid=13 Located tablet null > at null in 0.000 secs > 2017-05-23 01:45:22,803 > [impl.ThriftScanner] TRACE: Failed to locate tablet for > table : 7 row : > 2017-05-23 01:45:28,247 > [impl.TabletLocatorImpl] TRACE: tid=13 Locating tablet > table=7 row= skipRow=false retry=false > 2017-05-23 01:45:28,247 > [impl.TabletLocatorImpl] TRACE: tid=13 Locating tablet > table=!0 row=7; skipRow=false retry=false > 2017-05-23 01:45:28,247 > [impl.RootTabletLocator] TRACE: tid=13 Looking up root > tablet location in zookeeper. > 2017-05-23 01:45:28,248 > [impl.RootTabletLocator] TRACE: tid=13 Found root tablet at > null in 0.000 secs > 2017-05-23 01:45:28,248 > [impl.TabletLocatorImpl] TRACE: tid=13 Located tablet null > at null in 0.000 secs > 2017-05-23 01:45:28,248 > [impl.TabletLocatorImpl] TRACE: tid=13 Located tablet null > at null in 0.000 secs > 2017-05-23 01:45:28,248 > [impl.ThriftScanner] TRACE: Failed to locate tablet for > table : 7 row : > 2017-05-23 01:45:30,781 [ipc.Client] > DEBUG: stopping client from cache: org.apache.hadoop.ipc.Client@a95842c > 2017-05-23 01:45:30,782 [ipc.Client] > DEBUG: removing client from cache: org.apache.hadoop.ipc.Client@a95842c > 2017-05-23 01:45:30,782 [ipc.Client] > DEBUG: stopping actual client because no more references > remain: org.apache.hadoop.ipc.Client@a95842c > 2017-05-23 01:45:30,782 [ipc.Client] > DEBUG: Stopping client > > > Can anyone tell from the above why the > scan command is no longer working? > > Thanks, > Jim > >
