Did you re-initialize Accumulo after nuking the ZK entries?
Yamini Joshi wrote:
I restarted my zookeeper to get rid of all the accumulo entries in the
zookeeper. Now I am getting this error in my monitor debug log:
2016-10-07 10:07:46,548 [impl.ThriftScanner] DEBUG: Failed to locate
tablet for table : !0 row : ~err_\00
2016-10-07 10:07:49,781 [monitor.Monitor] INFO : Failed to obtain
problem reports
java.lang.RuntimeException:
org.apache.accumulo.core.client.impl.ThriftScanner$ScanTimedOutException
at
org.apache.accumulo.core.client.impl.ScannerIterator.hasNext(ScannerIterator.java:161)
at
org.apache.accumulo.server.problems.ProblemReports$3.hasNext(ProblemReports.java:252)
at
org.apache.accumulo.server.problems.ProblemReports.summarize(ProblemReports.java:310)
at org.apache.accumulo.monitor.Monitor.fetchData(Monitor.java:371)
at org.apache.accumulo.monitor.Monitor$2.run(Monitor.java:511)
at
org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
at java.lang.Thread.run(Thread.java:745)
Caused by:
org.apache.accumulo.core.client.impl.ThriftScanner$ScanTimedOutException
at
org.apache.accumulo.core.client.impl.ThriftScanner.scan(ThriftScanner.java:230)
at
org.apache.accumulo.core.client.impl.ScannerIterator$Reader.run(ScannerIterator.java:80)
at
org.apache.accumulo.core.client.impl.ScannerIterator.hasNext(ScannerIterator.java:151)
... 6 more
Best regards,
Yamini Joshi
On Fri, Oct 7, 2016 at 10:08 AM, Sean Busbey <[email protected]
<mailto:[email protected]>> wrote:
tracers used to be under the instance id in 1.6. I can confirm that in
1.7.2 it is definitely at /tracers. I just ran into this issue last
week.
@Yamini where do you get the Node does not exist error? Could you
copy/paste the whole stacktrace? When I followed the same workaround
Josh mentioned, restarting the Trace servers was sufficient to have
the znode recreated.
On Fri, Oct 7, 2016 at 10:01 AM, Josh Elser <[email protected]
<mailto:[email protected]>> wrote:
> I had pulled up 1.7.2 and I thought /tracers was at the root of
ZooKeeper
> (not within the /accumulo/$instance.id <http://instance.id>
znode). Maybe I looked at the wrong
> code though?
>
> Michael Wall wrote:
>>
>> Maybe this is already known, so apologies in advance. But don't you
>> need to do
>>
>> $ $ZOOKEEPER_HOME/bin/zkCli.sh -server m4:2181,m5:2181,m6:2181
>>>
>>> rmr accumulo/WHATEVER-INSTANCE-ID/tracers
>>
>>
>> instead of rmr /tracers
>>
>> If you don't know the instance id, I you can do the following in
zookeeper
>>
>> get /accumulo/instances/INSTANCE_NAME
>>
>>
>>
>> On Thu, Oct 6, 2016 at 6:26 PM, Josh Elser <[email protected]
<mailto:[email protected]>
>> <mailto:[email protected] <mailto:[email protected]>>> wrote:
>>
>> Uh, I am confused now. /tracers/ is the default parent znode
that
>> the Accumulo Tracer process registers itself in. We can also see
>> this is what was happening with your snippet.
>>
>> I don't understand how that would not exist. Have you properly
>> configured ZooKeeper to act as a quorom? (e.g. if you zkCli.sh
>> -server to your ZK servers individually, do you see a consistent
>> view of ZK?)
>>
>> You can try to remove the directory in HDFS you configured
Accumulo
>> to use, and re `accumulo init` to get back to an initial
state. I'm
>> not sure how you got into this situation to begin with.
>>
>> Yamini Joshi wrote:
>>
>> 1.7.2
>>
>>
>> Best regards,
>> Yamini Joshi
>>
>> On Thu, Oct 6, 2016 at 4:17 PM, Josh Elser
<[email protected] <mailto:[email protected]>
>> <mailto:[email protected] <mailto:[email protected]>>
>> <mailto:[email protected] <mailto:[email protected]>
<mailto:[email protected] <mailto:[email protected]>>>>
>> wrote:
>>
>> Hrm, maybe I am looking at a newer version of
Accumulo than
>> what
>> you're using. What version are you on?
>>
>> Yamini Joshi wrote:
>>
>> Thank you for replying Josh.
>> I am getting this error:
>> Node does not exist: /tracers
>>
>> I did ls and saw accumulo there. Should I
delete that?
>> When I try to delete accumulo, it gives me a
>> Authentication is
>> not valid
>> error.
>>
>> Best regards,
>> Yamini Joshi
>>
>> On Thu, Oct 6, 2016 at 2:57 PM, Josh Elser
>> <[email protected] <mailto:[email protected]>
<mailto:[email protected] <mailto:[email protected]>>
>> <mailto:[email protected] <mailto:[email protected]>
<mailto:[email protected] <mailto:[email protected]>>>
>> <mailto:[email protected] <mailto:[email protected]>
<mailto:[email protected] <mailto:[email protected]>>
>> <mailto:[email protected] <mailto:[email protected]>
<mailto:[email protected] <mailto:[email protected]>>>>>
>> wrote:
>>
>> Hi Yamini,
>>
>> The /tracers node in ZooKeeper was likely
>> initialized
>> previously
>> using an Accumulo configuration that has a
>> different value for
>> instance.secret. You can delete this znode
to move
>> forward
>> if you
>> have no other Accumulo instance you intend
to run
>> on this
>> ZK instance.
>>
>> $ $ZOOKEEPER_HOME/bin/zkCli.sh -server
>> m4:2181,m5:2181,m6:2181
>> > rmr /tracers
>>
>> You can also set a different value for
>> trace.zookeeper.path in
>> accumulo-site.xml. After doing this, you
can try
>> running a
>> start-all.sh again to start the tracer.
>>
>> You also mentioned that the Monitor did
not start
>> but did not
>> include an exception for that service. The
error you
>> included only
>> shows that the Accumulo
Master/TabletServer did not
>> correctly start
>> and assign the system tables
(accumulo.root and
>> accumulo.metadata).
>>
>>
>> Yamini Joshi wrote:
>>
>> Hello all
>>
>> I am trying to configure a multinode
cluster
>> with
>> accumulo. I am
>> using 3
>> machines with the config as follows:
>>
>> M4: Hadoop Master/Acc Slave
>> M5: Accumulo Master/Monitor/Gc
>> M6: Slave
>>
>> I have configured zookeeper ensemble
on the
>> cluster
>> with all the 3
>> nodes. The Hadoop part seems to work
fine, I
>> can see
>> the console
>> and run
>> MR jobs easily. I can also see the
zookeeper
>> cluster
>> with M2 as the
>> leader. My problem is with accumulo, after
>> accumulo init:
>> bin/accumulo init
>> which seems to run without any
error, after
>> I isuue
>> the command
>> bin/start-all.sh
>> the Monitor process can't start. I am
>> getting this
>> error in
>> my tracer log:
>> [start.Main] ERROR: Thread 'tracer' died.
>>
>> org.apache.zookeeper.KeeperException$NoAuthException:
>> KeeperErrorCode =
>> NoAuth for /tracers/trace-
>> at
>>
>>
>>
>>
org.apache.zookeeper.KeeperException.create(KeeperException.java:113)
>> at
>>
>>
>>
>> org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
>> at
>>
org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
>> at
>>
>>
>>
>>
org.apache.accumulo.fate.zookeeper.ZooUtil.putEphemeralSequential(ZooUtil.java:464)
>> at
>>
>>
>>
>>
org.apache.accumulo.fate.zookeeper.ZooReaderWriter.putEphemeralSequential(ZooReaderWriter.java:99)
>> at
>>
>>
>>
>>
org.apache.accumulo.tracer.TraceServer.registerInZooKeeper(TraceServer.java:304)
>> at
>>
>>
>>
>> org.apache.accumulo.tracer.TraceServer.<init>(TraceServer.java:241)
>> at
>>
>>
>>
org.apache.accumulo.tracer.TraceServer.main(TraceServer.java:346)
>> at
>>
>>
>>
>>
org.apache.accumulo.tracer.TracerExecutable.execute(TracerExecutable.java:33)
>> at
>> org.apache.accumulo.start.Main$1.run(Main.java:120)
>> at
java.lang.Thread.run(Thread.java:745)
>>
>> And this in my monitor debug log:
>> Failed to locate tablet for table : !0
row :
>> ~err_\100
>>
>> I can see that the accumulo dir has been
>> created in the
>> HDFS.
>> Can anyone
>> please guide me set this up? I am
attaching my
>> config
>> files here
>> (Rest
>> all the same generated as a result of
>> bin_config file).
>>
>> Best regards,
>> Yamini Joshi
>>
>>
>>
>>
>
--
busbey
