Processed: Re: Bug#766977: elasticsearch not starting

[Debian Bug Tracking System]

Processing control commands:

> retitle -1 elasticsearch: Please provide a systemd service file
Bug #766977 [elasticsearch] elasticsearch not starting
Changed Bug title to 'elasticsearch: Please provide a systemd service file' 
from 'elasticsearch not starting'

-- 
766977: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=766977
Debian Bug Tracking System
Contact ow...@bugs.debian.org with problems

__
This is the maintainer address of Debian's Java team
<http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-java-maintainers>. 
Please use
debian-j...@lists.debian.org for discussions and questions.

Bug#766977: elasticsearch not starting

[John Paul Adrian Glaubitz]

Control: retitle -1 elasticsearch: Please provide a systemd service file

> However the fact remains, systemd is confusingly reporting ES as
> being started when it is not. IMHO, that's an issue.

Not an issue here. Without START_DAEMON=true, systemd correctly reports
the unit as run successfully and detects the daemon to have exited:

root@vs84:~> systemctl status elasticsearch.service
● elasticsearch.service - LSB: Starts elasticsearch
   Loaded: loaded (/etc/init.d/elasticsearch)
   Active: active (exited) since Tue 2015-10-06 10:17:28 CEST; 1 weeks 1
days ago
root@vs84:~>

Enabling elasticsearch with START_DAEMON=true and systemd will also
detect the process itself as running:

root@vs84:~> systemctl status elasticsearch.service
● elasticsearch.service - LSB: Starts elasticsearch
   Loaded: loaded (/etc/init.d/elasticsearch)
   Active: active (running) since Wed 2015-10-14 10:51:31 CEST; 3s ago
  Process: 17109 ExecStop=/etc/init.d/elasticsearch stop (code=exited,
status=0/SUCCESS)
  Process: 17149 ExecStart=/etc/init.d/elasticsearch start (code=exited,
status=0/SUCCESS)
   CGroup: /system.slice/elasticsearch.service
   └─17203 /usr/lib/jvm/java-7-openjdk-amd64/bin/java -Xms256m
-Xmx1g -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC
-XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSIn...

Oct 14 10:51:31 vs84 elasticsearch[17149]: Starting Elasticsearch Server:.
root@vs84:~>

systemd is behaving absolutely correctly and as expected. Without
any systemd support from the elasticsearch service, systemd has
to resort to using the LSB compatibility layer and has to evaluate
the status codes of the SysV scripts provided by elasticsearch.

If these scripts return with SUCCESS on "start", systemd has to
assume the service was run successfully albeit it detects that
the process associated with it has exited (which is what you
can see above). There are cases where a unit is run and exits
shortly after (so-called "oneshot" unit in systemd), so it's safe
to assume for systemd to assume that the unit was run successfully
as long as the exit code was zero.

In order to properly fix this problem, the elasticsearch maintainers
should add systemd support to their package by providing a service
file. This is a good idea anyway since systemd is the default init
system since Debian Jessie and SystemV scripts is something we would
like to get rid of in Debian.

Adrian

-- 
 .''`.  John Paul Adrian Glaubitz
: :' :  Debian Developer - glaub...@debian.org
`. `'   Freie Universitaet Berlin - glaub...@physik.fu-berlin.de
  `-GPG: 62FF 8A75 84E0 2956 9546  0006 7426 3B37 F5B5 F913

__
This is the maintainer address of Debian's Java team
. 
Please use
debian-j...@lists.debian.org for discussions and questions.

Bug#766977: elasticsearch not starting

[Rémi Cardona]

On Thu, 16 Jul 2015 11:38:05 -0700 "Nathan D. Smith" 
 wrote:

Any progress on this?

I have the same issues as described above in Jessie. systemctl is
incorrect about the startup state, and it does not run.


I had to edit /etc/default/elasticsearch and uncomment 
START_DAEMON=true. systemctl stop + start and it runs.


However the fact remains, systemd is confusingly reporting ES as being 
started when it is not. IMHO, that's an issue.


AFAICT, most daemon packages in Debian actually start said daemon by 
default. I think the "default" file should just ship with START_DAEMON=true.


Cheers,

Rémi

__
This is the maintainer address of Debian's Java team
. 
Please use
debian-j...@lists.debian.org for discussions and questions.

Bug#766977: elasticsearch not starting

[Hilko Bengen]

* Philipp Hug:

 I suspect you did also run into the following problem, which is related:
 https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=768178

Uh-oh, this is bad. I strongly suspect that elasticsearch is not the
only package that has this issue.

systemd's behavior is interesting, I suspect that there's room for
improvement for the init script. I'll definitely need to look more
closely into this.

Cheers,
-Hilko

__
This is the maintainer address of Debian's Java team
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-java-maintainers. 
Please use
debian-j...@lists.debian.org for discussions and questions.

Bug#766977: elasticsearch not starting

[Philipp Hug]

Hi Jamie,

Ist it possible that you just missed to set START_DAEMON=true
in /etc/default/elasticsearch

But I guess it would make sense to have this set to true by default.

Philipp
__
This is the maintainer address of Debian's Java team
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-java-maintainers. 
Please use
debian-j...@lists.debian.org for discussions and questions.

Bug#766977: elasticsearch not starting

[Jamie Norrish]

On Mon, 2014-11-17 at 23:06 +, Philipp Hug wrote:

 Ist it possible that you just missed to set START_DAEMON=true
 in /etc/default/elasticsearch

I had been trying it with both, and it seemed to make no difference. But
having just tested the whole process again, elasticsearch is starting
and running more or less consistently (I suspect the less part is just
a problem with me rather than the program).

So it's likely that, with the path issue fixed, this bug can be closed.

 But I guess it would make sense to have this set to true by default.

I don't mind either way, provided it's documented.

Jamie

__
This is the maintainer address of Debian's Java team
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-java-maintainers. 
Please use
debian-j...@lists.debian.org for discussions and questions.

Bug#766977: elasticsearch not starting

[Philipp Hug]

Hi Jamie,

I suspect you did also run into the following problem, which is related:
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=768178

So after having elasticsearch installed with START_DAEMON not set, systemd
still thinks that elasticsearch is running because the init script returned
0.
Running service elasticsearch start will have no effect because of this.

Philipp

On Tue Nov 18 2014 at 12:48:48 AM Jamie Norrish ja...@artefact.org.nz
wrote:

 On Mon, 2014-11-17 at 23:06 +, Philipp Hug wrote:

  Ist it possible that you just missed to set START_DAEMON=true
  in /etc/default/elasticsearch

 I had been trying it with both, and it seemed to make no difference. But
 having just tested the whole process again, elasticsearch is starting
 and running more or less consistently (I suspect the less part is just
 a problem with me rather than the program).

 So it's likely that, with the path issue fixed, this bug can be closed.

  But I guess it would make sense to have this set to true by default.

 I don't mind either way, provided it's documented.

 Jamie


__
This is the maintainer address of Debian's Java team
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-java-maintainers. 
Please use
debian-j...@lists.debian.org for discussions and questions.

Bug#766977: elasticsearch not starting

[Jamie Norrish]

On Tue, 2014-11-18 at 00:04 +, Philipp Hug wrote:

 I suspect you did also run into the following problem, which is related:
 https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=768178
 
 So after having elasticsearch installed with START_DAEMON not set, systemd
 still thinks that elasticsearch is running because the init script returned
 0.
 Running service elasticsearch start will have no effect because of this.

Ah, interesting, thanks. That would certainly explain some of the
messages and odd behaviour that I've been seeing while testing this -
including it not appearing that setting START_DAEMON=true made any
difference, when starting it without explicitly stopping it first).
Still my own stupid fault, though!

Thanks so much for the help!

Jamie

__
This is the maintainer address of Debian's Java team
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-java-maintainers. 
Please use
debian-j...@lists.debian.org for discussions and questions.

Bug#766977: elasticsearch not starting

[Jamie Norrish]

On Thu, 2014-11-06 at 00:50 +0100, Hilko Bengen wrote:

 I ended up with systemd at some point, too. And I still have no idea how
 to debug daemon startup problems. :-(

Is there someone among the systemd maintainers who might be willing to
lend a hand here?

Jamie

__
This is the maintainer address of Debian's Java team
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-java-maintainers. 
Please use
debian-j...@lists.debian.org for discussions and questions.

Bug#766977: elasticsearch not starting

[Jamie Norrish]

On Tue, 2014-11-04 at 22:34 +0100, Hilko Bengen wrote:

   $ sudo -u elasticsearch bash -x 
  /usr/share/elasticsearch/bin/elasticsearch. 
 Could you please try if upgrading to version 1.0.3+dfsg-5 (which I just
 uploaded to unstable) fixes this problem?

The above command now runs elasticsearch properly.

It doesn't appear that /etc/init.d/elasticsearch works, however. It
appears to die immediately on start; a second invocation also states
that it is started (rather than finding the PID and stating that it is
already running). This is with unchanged configuration files; is there
something that needs to be modified (and if so, could this be documented
in the output of the init script, please)?

Jamie

__
This is the maintainer address of Debian's Java team
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-java-maintainers. 
Please use
debian-j...@lists.debian.org for discussions and questions.

Bug#766977: elasticsearch not starting

[Hilko Bengen]

* Jamie Norrish:

   $ sudo -u elasticsearch bash -x 
  /usr/share/elasticsearch/bin/elasticsearch. 
 Could you please try if upgrading to version 1.0.3+dfsg-5 (which I just
 uploaded to unstable) fixes this problem?

 The above command now runs elasticsearch properly.

So far so good...

 It doesn't appear that /etc/init.d/elasticsearch works, however.

What init system do you have? SysV? systemd?

What output does

# invoke-rc.d elasticsearch status

produce?

Cheers,
-Hilko

__
This is the maintainer address of Debian's Java team
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-java-maintainers. 
Please use
debian-j...@lists.debian.org for discussions and questions.

Bug#766977: elasticsearch not starting

[Jamie Norrish]

On Wed, 2014-11-05 at 23:57 +0100, Hilko Bengen wrote:

  It doesn't appear that /etc/init.d/elasticsearch works, however.
 What init system do you have? SysV? systemd?

systemd. Sorry for not specifying that before.

 # invoke-rc.d elasticsearch status

● elasticsearch.service - LSB: Starts elasticsearch
   Loaded: loaded (/etc/init.d/elasticsearch)
   Active: active (exited) since Thu 2014-11-06 07:44:19 NZDT; 4h 15min
ago
  Process: 8487 ExecStop=/etc/init.d/elasticsearch stop (code=exited,
status=0/SUCCESS)
  Process: 8521 ExecStart=/etc/init.d/elasticsearch start (code=exited,
status=0/SUCCESS)

Jamie

__
This is the maintainer address of Debian's Java team
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-java-maintainers. 
Please use
debian-j...@lists.debian.org for discussions and questions.

Bug#766977: elasticsearch not starting

[Hilko Bengen]

* Jamie Norrish:

  It doesn't appear that /etc/init.d/elasticsearch works, however.
 What init system do you have? SysV? systemd?
 systemd. Sorry for not specifying that before.

I ended up with systemd at some point, too. And I still have no idea how
to debug daemon startup problems. :-(

 # invoke-rc.d elasticsearch status
 ● elasticsearch.service - LSB: Starts elasticsearch
Loaded: loaded (/etc/init.d/elasticsearch)
Active: active (exited) since Thu 2014-11-06 07:44:19 NZDT; 4h 15min
 ago
   Process: 8487 ExecStop=/etc/init.d/elasticsearch stop (code=exited,
 status=0/SUCCESS)
   Process: 8521 ExecStart=/etc/init.d/elasticsearch start (code=exited,
 status=0/SUCCESS)

Here it looks like this:

,
| ● elasticsearch.service - LSB: Starts elasticsearch
|Loaded: loaded (/etc/init.d/elasticsearch)
|Active: active (running) since Tue 2014-11-04 16:54:11 CET; 1 day 7h ago
|CGroup: /system.slice/elasticsearch.service
|└─31692 /usr/lib/jvm/java-7-openjdk-amd64//bin/java -Xms256m 
-Xmx1...
| 
| Nov 04 16:54:11 ataraxia elasticsearch[31671]: Starting Elasticsearch Server:.
| Nov 04 16:54:11 ataraxia systemd[1]: Started LSB: Starts elasticsearch.
`

Cheers,
-Hilko

__
This is the maintainer address of Debian's Java team
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-java-maintainers. 
Please use
debian-j...@lists.debian.org for discussions and questions.

Bug#766977: elasticsearch not starting

[Hilko Bengen]

* Jamie Norrish:

 I am also experiencing this problem, with the IOException trying to
 write to /usr/share/elasticsearch/data/elasticsearch/nodes/49 when
 running:

  $ sudo -u elasticsearch bash -x /usr/share/elasticsearch/bin/elasticsearch. 

Could you please try if upgrading to version 1.0.3+dfsg-5 (which I just
uploaded to unstable) fixes this problem?

Thank you.
-Hilko

__
This is the maintainer address of Debian's Java team
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-java-maintainers. 
Please use
debian-j...@lists.debian.org for discussions and questions.

Bug#766977: elasticsearch not starting

[Jamie Norrish]

I am also experiencing this problem, with the IOException trying to
write to /usr/share/elasticsearch/data/elasticsearch/nodes/49 when
running:

 $ sudo -u elasticsearch bash -x /usr/share/elasticsearch/bin/elasticsearch. 

My /etc/elasticsearch/elasticsearch.yml does not have path.data set (it
is unchanged from what is shipped with the package; indeed, I haven't
touched anything in the package, and did not previously have upstream's
package installed either). I can see that /etc/init.d/elasticsearch sets
the DATA_DIR to /var/lib/elasticsearch, but nothing is written there.

Jamie

__
This is the maintainer address of Debian's Java team
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-java-maintainers. 
Please use
debian-j...@lists.debian.org for discussions and questions.

Bug#766977: elasticsearch not starting

[Gillu]

 We never shipped 0.90.7 within Debian -- am I assuming correctly that
 you used the .deb files provided by upstream?

Yes, this is very likely. I must have forgot this, apologies for not mentioning
it before. This must be it.

 What output do you get when you run something like the following?

I've attached the output. Looks like the lock location is owned by root and is
thus not writable. Am I correct to assume that /usr/share/elasticsearch/data
should be owned by elasticsearch?

 What java packages do you have installed?

ii  ca-certificates-java20140324   all
ii  libatk-wrapper-java 0.30.5-1   all
ii  libdom4j-java   1.6.1+dfsg.3-2 all
ii  libhyperic-sigar-java   1.6.4+dfsg-2   amd64
ii  libisorelax-java2004-8 all
ii  libjaxen-java   1.1.6-1all
ii  libjdom1-java   1.1.3-1all
ii  libjna-java 4.1.0-1all
ii  libjts-java 1.11-1 all
ii  liblog4j1.2-java1.2.17-5   all
ii  liblucene4-java 4.6.1+dfsg-1   all
ii  libmsv-java 2009.1+dfsg1-4 all
ii  librelaxng-datatype-java1.0+ds1-3  all
ii  libspatial4j-java   0.3-1  all
ii  libxerces2-java 2.11.0-7   all
ii  libxml-commons-external-java1.4.01-2   all
ii  libxml-commons-resolver1.1-java 1.2-7  all
ii  libxom-java 1.2.10-1   all
ii  libxpp2-java2.1.10-7   all
ii  libxpp3-java1.1.4c-2   all
ii  tzdata-java 2014h-2all

Cheers,
Kristen


elasticsearch.out
Description: Binary data
__
This is the maintainer address of Debian's Java team
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-java-maintainers. 
Please use
debian-j...@lists.debian.org for discussions and questions.

Bug#766977: elasticsearch not starting

[Hilko Bengen]

* Gillu:

 We never shipped 0.90.7 within Debian -- am I assuming correctly that
 you used the .deb files provided by upstream?

 Yes, this is very likely. I must have forgot this, apologies for not
 mentioning it before. This must be it.

No worries. :-)

 What output do you get when you run something like the following?
 I've attached the output. Looks like the lock location is owned by
 root and is thus not writable. Am I correct to assume that
 /usr/share/elasticsearch/data should be owned by elasticsearch?

Not quite. Debian's package adheres to the Filesystem Hierarchy
Standard: Data that may be written to does not belong into /usr/.

,
| {1.0.3}: Initialization Failed ...
| - ElasticsearchIllegalStateException[Failed to obtain node lock, is the 
following location writable?: [/usr/share/elasticsearch/data/elasticsearch]]
|   IOException[failed to obtain lock on 
/usr/share/elasticsearch/data/elasticsearch/nodes/49]
|   IOException[Cannot create directory: 
/usr/share/elasticsearch/data/elasticsearch/nodes/49]
`

We don't actually ship a /usr/share/elasticsearch/data directory -- the
data is put into /var/lib/elasticsearch instead. Please check if you
have path.data set in your elasticsearch.yml.

Cheers,
-Hilko

__
This is the maintainer address of Debian's Java team
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-java-maintainers. 
Please use
debian-j...@lists.debian.org for discussions and questions.

Bug#766977: elasticsearch not starting

[K. Gilden]

Package: elasticsearch
Version: 1.0.3+dfsg-4
Severity: important

Dear Maintainer,

I upgraded my system on 10/18 and from that point onwards elasticsearch (0.90.7
- 1.0.3+dfsg-3) doesn't start when invoking service elasticsearch start.
Running /usr/share/elasticsearch/bin/elasticsearch starts up the server just
fine.

The last full system upgrade before that was run on 12/10.

Best regards,
K. Gilden



-- System Information:
Debian Release: jessie/sid
  APT prefers testing
  APT policy: (500, 'testing')
Architecture: amd64 (x86_64)

Kernel: Linux 3.16-3-amd64 (SMP w/2 CPU cores)
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash

Versions of packages elasticsearch depends on:
ii  adduser3.113+nmu3
ii  default-jre2:1.7-52
ii  libhyperic-sigar-java  1.6.4+dfsg-2
ii  libjna-java4.1.0-1
ii  libjts-java1.11-1
ii  liblog4j1.2-java   1.2.17-5
ii  liblucene4-java4.6.1+dfsg-1
ii  libspatial4j-java  0.3-1

elasticsearch recommends no packages.

elasticsearch suggests no packages.

-- no debconf information

__
This is the maintainer address of Debian's Java team
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-java-maintainers. 
Please use
debian-j...@lists.debian.org for discussions and questions.

Bug#766977: elasticsearch not starting

[Hilko Bengen]

* K. Gilden:

 I upgraded my system on 10/18 and from that point onwards elasticsearch 
 (0.90.7
 - 1.0.3+dfsg-3) doesn't start when invoking service elasticsearch start.
 Running /usr/share/elasticsearch/bin/elasticsearch starts up the server just
 fine.

We never shipped 0.90.7 within Debian -- am I assuming correctly that
you used the .deb files provided by upstream?

What output do you get when you run something like the following?

$ sudo -u elasticsearch bash -x /usr/share/elasticsearch/bin/elasticsearch

What java packages do you have installed? (dpkg -l '*-java')

Cheers,
-Hilko

__
This is the maintainer address of Debian's Java team
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-java-maintainers. 
Please use
debian-j...@lists.debian.org for discussions and questions.