Bug#770706: [PKG-Openstack-devel] Bug#770706: Bug#770706: Bug#770706: keystone.service does not start, /var/run/keystone not created

2014-12-05 Thread Gaudenz Steinlin 
Thomas Goirand z...@debian.org writes:

 On 12/04/2014 05:42 PM, Gaudenz Steinlin wrote:
 with the same global section in 
 /usr/share/keystone/keystone-systemd.conf. One can then use 
 /etc/openstack.conf to switch back to syslog or stdout or stderr or 
 whatever she wants globally.
 
 That looks like a good solution to me to get rid of the /etc/default
 files without loosing any functionality. This would avoid duplicating
 configuration settings in configuration files and /etc/default files.
 Looks like the best solution to me, but ovviously post jessie.
 
 Gaudenz

 I not sure this is a good solution. The only correct way is to configure
 it on the command line, with the init system, as we need a per-daemon
 configuration to keep previous functionality. I'm not even sure that
 with a non-existent /etc/default/openstack.conf, the daemons would all
 continue to run without complaining.

I'm not sure if I understand your concerns. As far as I understood what
Mikael found out it's possible to start the daemons with multiple
configuration files. With this it's possible to have a configuration
file with general OpenStack settings, one with project specific settings
and one with service specific settings.

To give you an example for nova-compute we would have these 3
configuration files (paths and names may change, this is only an
example):
- /etc/openstack/openstack.conf - OpenStack generic settings
- /etc/nova/nova.conf   - Settings for all nova services
- /etc/nova/nova-compute.conf   - Settings only for nova-compute

The init system (every init system, not just systemd) would then start
the nova-compute service with this command:
nova-compute --config /etc/openstack/openstack.conf --config 
/etc/nova/nova.conf --config /etc/nova/nova-compute.conf

I currently don't see anything that can be done today with /etc/default
files that can't be done with this scheme. Am I missing something?

Sure this would need some changes to existing configuration and maybe
even a migration in the postinst script. But as this is all not targeted
at jessie we have plenty of time to get this right. (Which does not mean
I want to wait with the implementation until the next freeze.)


 I very much prefer the current state of things with (shell)
 configuration files in /etc/default, and I don't see any valid reason
 why we would stop using that.

The very compelling reason I see to get rid of the /etc/default files is
that we can have the same flexibility without them and that they
duplicate configuration settings from the config files (eg. logging). To
me having two ways to configure the same thing seems like a significant
disadvantage. That's why I'd prefer to get rid of /etc/default/xxx.

Gaudenz


signature.asc
Description: PGP signature

Bug#770706: [PKG-Openstack-devel] Bug#770706: Bug#770706: Bug#770706: keystone.service does not start, /var/run/keystone not created

2014-12-04 Thread Thomas Goirand 
FYI, I have pushed the patch from Mikael Cluseau to the
debian/experimental branch of openstack-pkg-tools (and therefore, all
the Juno packages are currently rebuilding automatically in my Jenkins).
Then later we can test that... though the build is for Wheezy, so I'm
not sure if that's so relevant. Is systemd working well in Wheezy?

Cheers,

Thomas Goirand (zigo)


-- 
To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org

Bug#770706: [PKG-Openstack-devel] Bug#770706: Bug#770706: Bug#770706: keystone.service does not start, /var/run/keystone not created

2014-12-04 Thread Gaudenz Steinlin 

Hi

Mikaël Cluseau mclus...@isi.nc writes:

 Hi Gaudenz,

 I will focus on the so called minimal set for now. If you don't mind, 
 we will open another bug to improve systemd's integration after this one 
 is fixed.

 On 12/03/2014 04:58 AM, Gaudenz Steinlin wrote:
 Did you test the patches with the minimal set of changes. I think we 
 need strong arguments that they are working if we want to propose them 
 to the Release Team for jessie. I currently don't have a good setup to 
 test such patches. 

 I did with keystone but developper tests are never to be trusted ;-) Its 
 quite easy to test BTW, as root:

 apt-get build-dep openstack-pkg-tools
 git clone -b debian/unstable 
 https://github.com/MikaelCluseau/debian-openstack-pkg-tools.git
 cd debian-openstack-pkg-tools
 debuild -us -uc -b
 dpkg -i ../openstack-pkg-tools_19_all.deb

 Then you can build any other OpenStack package from source. For 
 instance, keystone:

 apt-get source keystone
 cd keystone-*/
 debuild -us -uc -b
 dpkg -i ../keystone_*.deb

OK I tested keystone on unstable now and this works as it should. It's
definitely an improvement over the current unit file which does not work
at all. I did not go down the route to do a full openstack installation
but I also installed nova and the service units there work too. I did
not go all the way to have a working setup, but the problems I have seem
to be unrelated to the systemd unit files.

One thing I noticed is that all services are disabled by default even
though they were configured through debconf and should be ready to use.
Is this intentional as an off by default behavior or is this a bug. How
is this handled with sysvinit?

Gaudenz


--
To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org

Bug#770706: [PKG-Openstack-devel] Bug#770706: Bug#770706: Bug#770706: keystone.service does not start, /var/run/keystone not created

2014-12-03 Thread Thomas Goirand 
On 12/03/2014 04:32 AM, Mikaël Cluseau wrote:
 As I said at the beginning, yes. Here the complete patch for the
 immediate review:

I forwarded this to the release team:
https://bugs.debian.org/771929

Let's see what the release team says.

Cheers,

Thomas


--
To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org

Bug#770706: [PKG-Openstack-devel] Bug#770706: Bug#770706: Bug#770706: keystone.service does not start, /var/run/keystone not created

2014-12-02 Thread Mikaël Cluseau 

Hi Gaudenz,

I will focus on the so called minimal set for now. If you don't mind, 
we will open another bug to improve systemd's integration after this one 
is fixed.


On 12/03/2014 04:58 AM, Gaudenz Steinlin wrote:
Did you test the patches with the minimal set of changes. I think we 
need strong arguments that they are working if we want to propose them 
to the Release Team for jessie. I currently don't have a good setup to 
test such patches. 


I did with keystone but developper tests are never to be trusted ;-) Its 
quite easy to test BTW, as root:


apt-get build-dep openstack-pkg-tools
git clone -b debian/unstable 
https://github.com/MikaelCluseau/debian-openstack-pkg-tools.git

cd debian-openstack-pkg-tools
debuild -us -uc -b
dpkg -i ../openstack-pkg-tools_19_all.deb

Then you can build any other OpenStack package from source. For 
instance, keystone:


apt-get source keystone
cd keystone-*/
debuild -us -uc -b
dpkg -i ../keystone_*.deb

There's a long test phase during the build, maybe Thomas can suggest a 
trick to avoid it :-)



What do you mean by dangerous choices?


Thomas considered the choice of not creating /var/lock/* dangerous.


If we really need them they have
to be created by systemd and sysv init scripts as /var/lock is a symlink
into /run which is typically a tmpfs.


This is included in my last patch proposal.

Do you have a complete patch (including the recent discussion) which 
can be applied to openstack-pkg-tools and used to rebuild test 
packages? I think this needs thorough testing if it's to be proposed 
for jessie.


As I said at the beginning, yes. Here the complete patch for the 
immediate review:


± git log -1 --stat -p
commit 8b3217905de88f2037f0bbc6eac46dca776c8f48
Author: Mikaël Cluseau mclus...@isi.nc
Date:   Sun Nov 30 18:45:25 2014 +1100

Fix for bug #770706
---
 init-template/init-script-template   | 23 +--
 init-template/pkgos-gen-systemd-unit | 10 +-
 2 files changed, 18 insertions(+), 15 deletions(-)

diff --git a/init-template/init-script-template 
b/init-template/init-script-template

index 0326b5d..fd20957 100644
--- a/init-template/init-script-template
+++ b/init-template/init-script-template
@@ -36,11 +36,13 @@ fi
 # Exit if the package is not installed
 [ -x $DAEMON ] || exit 0

-# Create /var/lock/X, /var/run/X, /var/lib/X and /var/log/X
-for i in lock run log lib ; do
-mkdir -p /var/$i/${PROJECT_NAME}
-chown ${SYSTEM_USER} /var/$i/${PROJECT_NAME}
-done
+# If ran as root, create /var/lock/X, /var/run/X, /var/lib/X and 
/var/log/X as needed

+if [ x$USER = xroot ] ; then
+for i in lock run log lib ; do
+mkdir -p /var/$i/${PROJECT_NAME}
+chown ${SYSTEM_USER} /var/$i/${PROJECT_NAME}
+done
+fi

 # This defines init_is_upstart which we use later on (+ more...)
 . /lib/lsb/init-functions
@@ -65,6 +67,10 @@ do_stop() {
 return $RETVAL
 }

+do_systemd_start() {
+exec $DAEMON $DAEMON_ARGS
+}
+
 case $1 in
 start)
 init_is_upstart  /dev/null 21  exit 1
@@ -88,11 +94,8 @@ status)
 status_of_proc $DAEMON $NAME  exit 0 || exit $?
 ;;
 systemd-start)
-do_start
+do_systemd_start
 ;;
-systemd-stop)
-do_stop
-;;
 restart|force-reload)
 init_is_upstart  /dev/null 21  exit 1
 log_daemon_msg Restarting $DESC $NAME
@@ -110,7 +113,7 @@ restart|force-reload)
 esac
 ;;
 *)
-echo Usage: $SCRIPTNAME 
{start|stop|status|restart|force-reload|systemd-start|systemd-stop} 2
+echo Usage: $SCRIPTNAME 
{start|stop|status|restart|force-reload|systemd-start} 2

 exit 3
 ;;
 esac
diff --git a/init-template/pkgos-gen-systemd-unit 
b/init-template/pkgos-gen-systemd-unit

index b97e2a9..4c41ef0 100755
--- a/init-template/pkgos-gen-systemd-unit
+++ b/init-template/pkgos-gen-systemd-unit
@@ -12,7 +12,7 @@ fi
 if [ -z ${SYSTEM_USER} ] ; then
 SYSTEM_USER=${PROJECT_NAME}
 fi
-if [ -z ${SYSTEM_USER} ] ; then
+if [ -z ${SYSTEM_GROUP} ] ; then
 SYSTEM_GROUP=${PROJECT_NAME}
 fi

@@ -33,12 +33,12 @@ $AFTER
 [Service]
 User=${SYSTEM_USER}
 Group=${SYSTEM_GROUP}
+WorkingDirectory=/var/lib/${PROJECT_NAME}
+PermissionsStartOnly=true
+ExecStartPre=/bin/mkdir -p /var/lock/${PROJECT_NAME} 
/var/log/${PROJECT_NAME} /var/lib/${PROJECT_NAME}
+ExecStartPre=/bin/chown ${SYSTEM_USER}:${SYSTEM_GROUP} 
/var/lock/${PROJECT_NAME} /var/log/${PROJECT_NAME} /var/lib/${PROJECT_NAME}

 ExecStart=${SCRIPTNAME} systemd-start
-ExecStop=${SCRIPTNAME} systemd-stop
-RuntimeDirectory=${PROJECT_NAME}
-PIDFile=/var/run/${PROJECT_NAME}/${NAME}.pid
 Restart=on-failure
-Type=forking

 [Install]
 WantedBy=multi-user.target


--
To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org

Bug#770706: [PKG-Openstack-devel] Bug#770706: Bug#770706: Bug#770706: keystone.service does not start, /var/run/keystone not created

2014-11-30 Thread Gaudenz Steinlin 
Hi

Thanks for working on this. This looks like a step in the right
direction. I however still have some problems with the proposed
approach.

Mikaël Cluseau mclus...@isi.nc writes:

 Hi all,

 I tried to minimize the changes to get close to a unit file in the 
 systemd spirit, while still using on the init-script to keep same 
 features as before. The patch below reflect the following actions:


I still don't like the idea of using the sysv init script here. I'd
rather simplify the init script to the point where this is no longer
needed and we can just have the following in our unit file:
EnvironmentFile=/etc/default/openstack /etc/default/${PROJECT_NAME}

And then use the proper variables in the ExecStart setting.


 1. don't auto-create the /var/*/${PROJECT_NAME} folder if not root, as 
 it will fail anyway.

Most of these should not be created by the init script anyway. Creating
/var/lib/X and /var/log/X there seems like a bug to me. These should be
part of the package. That's how it's done in all other packages I know.

 2. add a do_systemd_start function to implement the systemd-start 
 argument that will exec' the daemon with the previously computed args.

That's an improvement over the current state. But how do you ensure that
the daemon is started in the foreground?

 3. since then systemd will manage the daemon, the systemd-stop argument 
 can be removed.
 4. since the script doesn't create the necessary folders anymore, create 
 what's required using the systemd unit file.

 The point 4 is where I did the most invasive choices; I assumed that
 1. /var/run/${PROJECT_NAME} is not needed since PID file is not created 
 anymore (process managed by systemd);
 2. /var/lib/${PROJECT_NAME} doesn't have to be created as it is created 
 when the package is installed;

The please remove it from the init script alltogether. It's also not
needed when under sysv then.

 3. /var/lock/${PROJECT_NAME}, AFAIK, is not needed when using systemd;

What was it used for? Seems strange that there is a difference between
sysv and systemd here.

 4. /var/log/${PROJECT_NAME} is the only one that is both required and 
 can be considered volatile.

No this should definitely not be considered volatile. It's ok to delete
the logfiles inside, but no one can expect the daemon to still work if
he just removes this directory completely. For all other packages I know
if this is part of the package and not created on the fly.

I would really prefer to have a full systemd unit file that does not
depend on the sysv script at all. It's fine and probably a good idea to
use the files in /etc/default/.

Gaudenz


 If any of these choices is wrong, it's easy to add the folder in the 
 ExecStartPre lines of the systemd unit file.

 If the systemd-stop argument should still be supported, I think it 
 should be noop or print a depreciation warning.

 As a consequence of these changes, the unit file doesn't need its 
 RuntimeDirectory and PIDFile directives. Since the daemon is exec'd, the 
 Type falls back to the default (simple) so it is removed too.

 I don't how to do less than that, so here is my minimal patch proposal:

 diff --git a/init-template/init-script-template 
 b/init-template/init-script-template
 index 0326b5d..fd20957 100644
 --- a/init-template/init-script-template
 +++ b/init-template/init-script-template
 @@ -36,11 +36,13 @@ fi
   # Exit if the package is not installed
   [ -x $DAEMON ] || exit 0

 -# Create /var/lock/X, /var/run/X, /var/lib/X and /var/log/X
 -for i in lock run log lib ; do
 -mkdir -p /var/$i/${PROJECT_NAME}
 -chown ${SYSTEM_USER} /var/$i/${PROJECT_NAME}
 -done
 +# If ran as root, create /var/lock/X, /var/run/X, /var/lib/X and 
 /var/log/X as needed
 +if [ x$USER = xroot ] ; then
 +for i in lock run log lib ; do
 +mkdir -p /var/$i/${PROJECT_NAME}
 +chown ${SYSTEM_USER} /var/$i/${PROJECT_NAME}
 +done
 +fi

   # This defines init_is_upstart which we use later on (+ more...)
   . /lib/lsb/init-functions
 @@ -65,6 +67,10 @@ do_stop() {
   return $RETVAL
   }

 +do_systemd_start() {
 +exec $DAEMON $DAEMON_ARGS
 +}
 +
   case $1 in
   start)
   init_is_upstart  /dev/null 21  exit 1
 @@ -88,11 +94,8 @@ status)
   status_of_proc $DAEMON $NAME  exit 0 || exit $?
   ;;
   systemd-start)
 -do_start
 +do_systemd_start
   ;;
 -systemd-stop)
 -do_stop
 -;;
   restart|force-reload)
   init_is_upstart  /dev/null 21  exit 1
   log_daemon_msg Restarting $DESC $NAME
 @@ -110,7 +113,7 @@ restart|force-reload)
   esac
   ;;
   *)
 -echo Usage: $SCRIPTNAME 
 {start|stop|status|restart|force-reload|systemd-start|systemd-stop} 2
 +echo Usage: $SCRIPTNAME 
 {start|stop|status|restart|force-reload|systemd-start} 2
   exit 3
   ;;
   esac
 diff --git a/init-template/pkgos-gen-systemd-unit 
 b/init-template/pkgos-gen-systemd-unit
 index b97e2a9..09cf3e5 100755
 --- a/init-template/pkgos-gen-systemd-unit
 +++ b/init-template/pkgos-gen-systemd-unit

Bug#770706: [PKG-Openstack-devel] Bug#770706: Bug#770706: Bug#770706: keystone.service does not start, /var/run/keystone not created

2014-11-27 Thread Mikaël Cluseau 

On 11/27/2014 06:08 PM, Gaudenz Steinlin wrote:

I think we can agree that *if* we implement the systemd fully *then*
pid-files become useless.

As you say it would be the best way, I would like to (try to) spend a
significant time to implement the systemd way this week-end. Just... I
don't want to it just for the trash, so I'd like to know if you'd agree
with this and if the release team will (probably) accept it, or not.
I agree with the pidfile part. I can't comment on the release team. If
that's important to you best ask them yourself. But even if this does
not end up in jessie we need this for the future. So work in this is
definitely not lost. Maybe you could also implement some of the security
features that systemd provides[1] in the unit files. But that's probably
more than the Release Team is willing to accept.

Gaudenz

[1] http://0pointer.net/public/systemd-nluug-2014.pdf


Thank you for the suggestions. I will try to implements things in 2 
parts, I think it's better that to believe I can fool the release team 
with a big commit :-) After many years, it's the first time I try to 
contribute to Debian, so I'll rely heavily on comments. Let's try!



--
To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org

Bug#770706: [PKG-Openstack-devel] Bug#770706: Bug#770706: Bug#770706: keystone.service does not start, /var/run/keystone not created

2014-11-26 Thread Stig Sandbeck Mathisen 
Gaudenz Steinlin gaud...@debian.org writes:

 I'll try to find some time to have a look, but can't promise anything.

Adding the following files fixed this issue for me:

head /etc/tmpfiles.d/*
== /etc/tmpfiles.d/ceilometer.conf ==
d /run/ceilometer 0755 ceilometer ceilometer - -

== /etc/tmpfiles.d/glance.conf ==
d /run/glance 0755 glance glance - -

== /etc/tmpfiles.d/keystone.conf ==
d /run/keystone 0755 keystone keystone - -

Packages will normally install these files into /usr/lib/tmpfiles.d/
instead.

If the packages:

* contain the file debian/packagename.tmpfile with content described
  above.

* is built using debhelper with dh_systemd (I see this is true for the
  keystone package)

...this will create the run directories when the packages are installed,
 as well as on boot, and then only if systemd is running as pid 1.

-- 
Stig Sandbeck Mathisen


-- 
To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org

Bug#770706: [PKG-Openstack-devel] Bug#770706: Bug#770706: Bug#770706: keystone.service does not start, /var/run/keystone not created

2014-11-26 Thread Mikaël Cluseau 

The quick-fix of tmpfiles.d is what I did first and will (probably) work.

But, if I assemble this

On 11/27/2014 11:10 AM, Thomas Goirand wrote:

Do we agree that with systemd the pid-file logic becomes useless

Unfortunately, no, I don't agree. Not with the (poor) way we've designed
things in openstack-pkg-tools, which still uses start-stop-daemon.


and this

On 11/27/2014 11:14 AM, Thomas Goirand wrote:

Best would be if we could just get rid of the sysv-rc wrapper, and just
go the systemd way fully.


I think we can agree that *if* we implement the systemd fully *then* 
pid-files become useless.


As you say it would be the best way, I would like to (try to) spend a 
significant time to implement the systemd way this week-end. Just... I 
don't want to it just for the trash, so I'd like to know if you'd agree 
with this and if the release team will (probably) accept it, or not.



--
To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org

Bug#770706: [PKG-Openstack-devel] Bug#770706: Bug#770706: Bug#770706: keystone.service does not start, /var/run/keystone not created

2014-11-26 Thread Gaudenz Steinlin 
Mikaël Cluseau mclus...@isi.nc writes:

 The quick-fix of tmpfiles.d is what I did first and will (probably) work.

 But, if I assemble this

 On 11/27/2014 11:10 AM, Thomas Goirand wrote:
 Do we agree that with systemd the pid-file logic becomes useless
 Unfortunately, no, I don't agree. Not with the (poor) way we've designed
 things in openstack-pkg-tools, which still uses start-stop-daemon.

 and this

 On 11/27/2014 11:14 AM, Thomas Goirand wrote:
 Best would be if we could just get rid of the sysv-rc wrapper, and just
 go the systemd way fully.

 I think we can agree that *if* we implement the systemd fully *then* 
 pid-files become useless.

 As you say it would be the best way, I would like to (try to) spend a 
 significant time to implement the systemd way this week-end. Just... I 
 don't want to it just for the trash, so I'd like to know if you'd agree 
 with this and if the release team will (probably) accept it, or not.

I agree with the pidfile part. I can't comment on the release team. If
that's important to you best ask them yourself. But even if this does
not end up in jessie we need this for the future. So work in this is
definitely not lost. Maybe you could also implement some of the security
features that systemd provides[1] in the unit files. But that's probably
more than the Release Team is willing to accept.

Gaudenz

[1] http://0pointer.net/public/systemd-nluug-2014.pdf


--
To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org

Bug#770706: [PKG-Openstack-devel] Bug#770706: Bug#770706: Bug#770706: keystone.service does not start, /var/run/keystone not created

2014-11-24 Thread Gaudenz Steinlin 
Thomas Goirand z...@debian.org writes:

 On 11/24/2014 01:52 AM, Gaudenz Steinlin wrote:
 Why do you set PIDFile at all? IMO pidfiles are unnecessary when the
 daemons managed by systemd run in the foreground and thus there is no
 need for a pidfile at all.

 The point is, the way things are designed currently, we don't use
 foreground at all right now, we continue to use start-stop-daemon and
 execute the init.d startup script, even when using systemd. This is a
 bad design that would need improvement, and I would welcome a better
 implementation within openstack-pkg-tools. I would very much like to see
 some patches to use foreground (the way it's done in Fedora?), but it
 should support all properties and implementation of what's done in the
 init scripts.

I see. Wouldn't it be better to completely rely on the sysvinit
compatibility of systemd then and not ship any unit files at all? What's
the advantage of having systemd units that only wrap around sysv init
scripts?


 On 11/24/2014 04:54 AM, Mikaël Cluseau wrote: Hi,
 isn't it a duplicate of
 https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=767711 ?

 Yes it is. And that's a general issue with all OpenStack packages in
 Sid/Jessie right now. We need to fix all of this before the 5th of
 December deadline.

I'll try to find some time to have a look, but can't promise anything.

Gaudenz


--
To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org

Bug#770706: [PKG-Openstack-devel] Bug#770706: Bug#770706: Bug#770706: keystone.service does not start, /var/run/keystone not created

2014-11-23 Thread Mikaël Cluseau 

On 24/11/2014 10:20, Thomas Goirand wrote:
Yes it is. And that's a general issue with all OpenStack packages in 
Sid/Jessie right now. We need to fix all of this before the 5th of 
December deadline.


I will try to help. I didn't knew if the intention was to keep a 
simple compatibility for systems without systemd.


I'm GMT+11. Do you prefer to track on this bug or on #767711?


--
To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org