Jay,
It would be great to see a ticket to keep us honest on making the init
scripts better. A "user's perspective" would be good. If you'd prefer
not to make one, let me know and I'll write one up.
Christopher,
In regards to your comment about the init scripts being rough, are they
better in 1.6? Is this something that we should try to work on
improving? I wasn't sure if you had things coming down the pipe to
address this already.
On 8/27/13 9:02 AM, Shipper, Jay [USA] wrote:
We figured out the issue: The JVM didn't have enough memory to start the
process.
*_________________*
*Jay Shipper*
Booz | Allen | Hamilton
[email protected] <mailto:[email protected]>
From: <Shipper>, Merck User <[email protected]
<mailto:[email protected]>>
Reply-To: "[email protected] <mailto:[email protected]>"
<[email protected] <mailto:[email protected]>>
Date: Monday, August 26, 2013 9:06 PM
To: "[email protected] <mailto:[email protected]>"
<[email protected] <mailto:[email protected]>>
Subject: Re: [External] Re: Accumulo (1.5.0) Master Will Not Start
There's more than one? I'm checking accumulo's logs directory (in our
installation's case, /usr/lib/accumulo/logs, which links to where we
store our logs).
*_________________*
*Jay Shipper*
Booz | Allen | Hamilton
[email protected] <mailto:[email protected]>
From: John Vines <[email protected] <mailto:[email protected]>>
Reply-To: "[email protected] <mailto:[email protected]>"
<[email protected] <mailto:[email protected]>>,
"[email protected] <mailto:[email protected]>" <[email protected]
<mailto:[email protected]>>
Date: Monday, August 26, 2013 8:59 PM
To: "[email protected] <mailto:[email protected]>"
<[email protected] <mailto:[email protected]>>
Subject: Re: [External] Re: Accumulo (1.5.0) Master Will Not Start
Which log directory are you checking?
On Mon, Aug 26, 2013 at 8:18 PM, Shipper, Jay [USA] <[email protected]
<mailto:[email protected]>> wrote:
It's not making any .out/.err files in the log directory, nor is it
updating the accumulo master log file.
Here's the result of the command you suggested, following the point
where the script connects to HDFS (I have substituted <host ip> for
the actual IP):
---
+ disown
+ SUDOPID=22238
+ sleep 5
+ sudo -u accumulo /usr/lib/accumulo/bin/accumulo master --address
<host ip>
+ '[' -z 22238 ']'
++ ps --ppid 22238 -o pid=
+ PID=
+ '[' -z '' ']'
+ echo Failed
Failed
+ return 1
+ RETVAL=1
+ exit 1
---
*_________________*
*Jay Shipper*
Booz | Allen | Hamilton
[email protected] <mailto:[email protected]>
From: John Vines <[email protected] <mailto:[email protected]>>
Reply-To: "[email protected]
<mailto:[email protected]>" <[email protected]
<mailto:[email protected]>>, "[email protected]
<mailto:[email protected]>" <[email protected] <mailto:[email protected]>>
Date: Monday, August 26, 2013 6:52 PM
To: "[email protected] <mailto:[email protected]>"
<[email protected] <mailto:[email protected]>>
Subject: Re: [External] Re: Accumulo (1.5.0) Master Will Not Start
Including the .out and .err files? Try running it as bash -x
/etc/init.d/accumulo-master start and see if you spot anything there.
On Mon, Aug 26, 2013 at 6:48 PM, Shipper, Jay [USA]
<[email protected] <mailto:[email protected]>> wrote:
As I mentioned, nothing is being logged in the master's logs
when I attempt to start the service.
*_________________*
*Jay Shipper*
Booz | Allen | Hamilton
304 National Business Parkway
Room 5048
Annapolis Junction, MD 20701
Phone: 301-419-5437
[email protected] <mailto:[email protected]>
From: John Vines <[email protected] <mailto:[email protected]>>
Reply-To: "[email protected]
<mailto:[email protected]>" <[email protected]
<mailto:[email protected]>>, "[email protected]
<mailto:[email protected]>" <[email protected]
<mailto:[email protected]>>
Date: Monday, August 26, 2013 4:21 PM
To: "[email protected] <mailto:[email protected]>"
<[email protected] <mailto:[email protected]>>
Subject: [External] Re: Accumulo (1.5.0) Master Will Not Start
Please check the various master logs and see if anything is
being reported there.
On Mon, Aug 26, 2013 at 4:10 PM, Shipper, Jay [USA]
<[email protected] <mailto:[email protected]>> wrote:
Here is what is being logged to the console when I try to
start the accumulo (1.5.0) master service:
---
$ sudo service accumulo-master start
Not running
Starting...
2013-08-26 19:46:41,674 [vfs.AccumuloVFSClassLoader] WARN :
ignoring classpath entry
file:///usr/lib/accumulo/lib/ext/[^.].*.jar
2013-08-26 19:46:43,342 [server.Accumulo] INFO : Attempting
to talk to zookeeper
2013-08-26 19:46:43,505 [server.Accumulo] INFO : Zookeeper
connected and initialized, attemping to talk to HDFS
2013-08-26 19:46:43,658 [server.Accumulo] INFO : Connected
to HDFS
Failed
---
What could cause this behavior? HDFS and Zookeeper both
appear fine, the accumulo monitor shows the zookeeper
servers, and all the other accumulo services are still up on
their respective nodes. Nothing is getting logged in our
accumulo logs directory.
If you need any other information to make a diagnosis, just
ask for what you need. We have a working cluster with
pretty much the same setup (EC2, CentOS, Hadoop 2, etc.),
which is why we're very puzzled.
Thanks,
*_________________*
*Jay Shipper*
Booz | Allen | Hamilton
[email protected] <mailto:[email protected]>
