All,
I resolved the issue. Per Eric's suggestion I changed my swappiness
setting down to 10, and also disabled all IPV6 stuff on my VM. Once I
did this and changed the IP address of my monitor to 127.0.0.1 my
monitor started working.
Thanks guys. :-)
Regards,
Pete
On Wed, Sep 11, 2013 at 2:12 PM, John Vines <[email protected]
<mailto:[email protected]>> wrote:
Furthermore, tservers being down shouldn't prevent the monitor
page from coming up at all, so check for any other relevant
messages in of the monitor's logs.
On Wed, Sep 11, 2013 at 2:08 PM, Pete Carlson <[email protected]
<mailto:[email protected]>> wrote:
How do I determine if the rest of Accumulo is up and running
since I can't use the monitor?
Inside the monitor logs I see messages that it connected to
Hadoop and Zookeeper, but then later on I see this message
stating there are no tablet servers:
2013-09-11 11:39:29,199 [server.Accumulo] WARN : System
swappiness setting is greater than ten (60) which can cause
time-sensitive operations to be delayed. Accumulo is time
sensitive because it needs to maintain distributed lock agreement.
2013-09-11 11:39:32,506 [impl.ServerClient] WARN :
tracer:localhost.localdomain There are no tablet servers:
check that zookeeper and accumulo are running.
And when I log in to the accumulo shell I get the warning:
There are no tablet servers: check that zookeeper and accumulo
are running.
Could the system swappiness setting of 60 be causing my issue?
Regards,
Pete
On Wed, Sep 11, 2013 at 1:13 PM, John Vines <[email protected]
<mailto:[email protected]>> wrote:
Is the rest of Accumulo up and running? And is that the
only type of error in the Monitor logs?
On Wed, Sep 11, 2013 at 1:04 PM, Pete Carlson
<[email protected] <mailto:[email protected]>> wrote:
Ok, I deleted /accumulo out of hdfs, initialized
accumulo, and re-started Accumulo. I still can't see
the monitor IN my VM browser. I still get the same
Monitor info message in the logs.
Pete
On Wed, Sep 11, 2013 at 11:55 AM, John Vines
<[email protected] <mailto:[email protected]>> wrote:
You need to delete the /accumulo directory out of hdfs
On Wed, Sep 11, 2013 at 11:54 AM, Pete Carlson
<[email protected] <mailto:[email protected]>>
wrote:
I removed the monitor certificates as
suggested, and I no longer get any messages in
the monitor_localhost.err file.
When I look in the
monitor_localhost.localdomain.log file and saw
the following INFO/Exception message:
2013-09-11 11:42:50,012 [monitor.Monitor] INFO
: Failed to obtain problem reports
java.lang.RuntimeException:
org.apache.accumulo.core.client.impl.ThriftScanner$ScanTimedOutException
at
org.apache.accumulo.core.client.impl.ScannerIterator.hasNext(ScannerIterator.java:174)
at
org.apache.accumulo.server.problems.ProblemReports$3.hasNext(ProblemReports.java:241)
at
org.apache.accumulo.server.problems.ProblemReports.summarize(ProblemReports.java:299)
at
org.apache.accumulo.server.monitor.Monitor.fetchData(Monitor.java:392)
at
org.apache.accumulo.server.monitor.Monitor$2.run(Monitor.java:504)
at
org.apache.accumulo.core.util.LoggingRunnable.run(LoggingRunnable.java:34)
at java.lang.Thread.run(Thread.java:724)
Caused by:
org.apache.accumulo.core.client.impl.ThriftScanner$ScanTimedOutException
at
org.apache.accumulo.core.client.impl.ThriftScanner.scan(ThriftScanner.java:212)
at
org.apache.accumulo.core.client.impl.ScannerIterator$Reader.run(ScannerIterator.java:82)
at
org.apache.accumulo.core.client.impl.ScannerIterator.hasNext(ScannerIterator.java:164)
... 6 more
I was thinking this was related to the fact I
didn't yet have Thrift 0.9.0 installed (due to
dependency issues). I now have Thrift 0.9.0
installed. I saved off my config files, blew
away the accumulo directory and extracted it
again from the tar.gz and copied over my conf
files.
However, when I run the
accumulo-1.5.0/bin/accumulo init command again
it gives me the following error:
[util.Initialize] FATAL: It appears this
location was previously initialized, exiting ...
I can still start Accumulo like before with
the bin/start-all.sh command but the Accumulo
monitor still doesn't work IN my VM.
Any other ideas to try?
Pete
On Wed, Sep 11, 2013 at 10:57 AM, John Vines
<[email protected] <mailto:[email protected]>>
wrote:
Either try rerunning the monitor
certificates or remove the portion from
your accumulo-site that defines use of the
certificate. I suggest the latter simply
to remove variables at hand.
On Wed, Sep 11, 2013 at 10:54 AM, Pete
Carlson <[email protected]
<mailto:[email protected]>> wrote:
John,
What is the monitor process called so
I can verify its running? I did look
at the thread monitor logs, and under
monitor_localhost.err it gives a
FileNotFoundException because it can't
find the keystore.jks under
accumulo-1.5.0/conf/
I would have thought when I ran
./generate_monitor_certificate.sh it
would have created this.
I was trying to run the monitor in a
browser IN the VM. Thats how I found
this problem.
Regards,
Pete
On Wed, Sep 11, 2013 at 10:41 AM, John
Vines <[email protected]
<mailto:[email protected]>> wrote:
Again, please check to make sure
the monitor process is running and
it's not reporting any errors in
it's logs.
If you can verify that (and please
verify that first), try accessing
the monitor in a browser IN the VM.
On Wed, Sep 11, 2013 at 10:36 AM,
Pete Carlson <[email protected]
<mailto:[email protected]>> wrote:
All,
Just to clarify I have a
Centos VM running on my
Windows 7 box, and I have
Accumulo, Hadoop, and
Zookeeper running on it
standalone. I was trying to
open the Accumulo monitor
within that VM and it failed.
However I can open the Hadoop
Namenode and Jobtracker web
interfaces in my browser just
fine.
Brendan, I was able to connect
to my VM using netcat on port
50095.
However, when I try to telnet
to that port on my Win7 with
telnet using my VM's ip
address and port i.e.,
telnet <IP address of vm> 50095
I get a message: ....Could not
open connection to the host,
on port 50095: Connect failed.
Pete
On Wed, Sep 11, 2013 at 9:15
AM, Brendan Heussler
<[email protected]
<mailto:[email protected]>>
wrote:
Can you try to connect to
your VM using netcat on
port 50095?
On VM:
nc -l 50095
On Client:
nc <ip-address of VM> 50095
Then type "test" or
something on your Client
machine. The message
should be sent to your VM.
This will help to test the
connectivity of the two
machines over port 50095
Brendan
On Tue, Sep 10, 2013 at
5:22 PM, John Vines
<[email protected]
<mailto:[email protected]>>
wrote:
Are you certain the
monitor is running and
was able to bind to
the port? Please check
the monitor logs for
any errors.
On Tue, Sep 10, 2013
at 5:18 PM, Pete
Carlson
<[email protected]
<mailto:[email protected]>>
wrote:
John,
I don't see port
50095 in use in
the VM. When I
become root and
type lsof -i -P |
grep :50095 I
don't get anything
back.
Any suggestions on
how to get the
port running? I
tried editing the
iptables with the
following command:
iptables -I INPUT
-p tcp --dport
50095 -j ACCEPT
and then I typed
service iptables save
to save my iptable
changes, but when
I restarted my VM
I didn't see any
difference.
Regards,
Pete
Looking for a
unique woodcraft?
Check out my
Facebook page:
www.facebook.com/oqwcrafts
<http://www.facebook.com/oqwcrafts>
Also on Etsy:
https://www.etsy.com/shop/OlneyQualityWoodCrft
On Tue, Sep 10,
2013 at 4:56 PM,
John Vines
<[email protected]
<mailto:[email protected]>>
wrote:
You don't see
50095 in use
locally on the
VM or in your
box? You
should only
see it in use
in the VM. If
not, make sure
it's running.
If it is,
check the
monitor logs
and see what's
up.
On Tue, Sep
10, 2013 at
4:51 PM, Pete
Carlson
<[email protected]
<mailto:[email protected]>>
wrote:
Hi John,
No, it
looks like
port 50095
wasn't
open on my
VM. I
tried
modifying
the IP
tables to
make it
opened,
and
restarted
my VM but
when I run
the lsof
-i command
I don't
see port
50095 listed.
Pete
Regards,
Pete
Looking
for a
unique
woodcraft?
Check out
my
Facebook
page:
www.facebook.com/oqwcrafts
<http://www.facebook.com/oqwcrafts>
Also on
Etsy:
https://www.etsy.com/shop/OlneyQualityWoodCrft
On Tue,
Sep 10,
2013 at
3:56 PM,
John Vines
<[email protected]
<mailto:[email protected]>>
wrote:
Do you
have
the
port
open
to
your VM?
On
Tue,
Sep
10,
2013
at
3:54
PM,
Pete
Carlson
<[email protected]
<mailto:[email protected]>>
wrote:
Hi,
I
just
set up
a
free
standing
Accumulo
1.5 instance
on
my
linux
VM, and
when
I
run
bin/start-all.sh
it
starts
up
accumulo,
and also
connects
to
my
free
standing
Zookeeper
and HDFS
instances
.
However,
when
I
try to
pull
up
the Accumulo
master
in
my
FireFox
browser
(Firefox
ESR 10.0.12)
i.e.,
http://accumulomaster:50095/status for
health
and
monitoring,
I
get an
unable
to
connect
error.
I
also
tried
running
./generate_monitor_certificate.sh
and copied
those
properties
in
my
accumulo-site.xml
but I
still
get the
same
result.
Is
there
something
else
I
need
to
configure?
Regards,
Pete
