Re: [openstack-dev] [tc][pbr][packaging][all] Definition of Data Files (config) in setup.cfg

[Andrew Laski]

On Tue, Jun 14, 2016, at 05:36 AM, Julien Danjou wrote:
> On Sun, Jun 12 2016, Monty Taylor wrote:
> 
> > I will call hogwash on that. That we install files such as that which
> > are actually code has been a deployer nightmare since day one, and the
> > solution is not to make pip smarter, it's to stop installing such files.
> >
> > NOW - if it's quite literally a file that just exists better as a file
> > and not direct python code - that's fine - just put it into the package
> > itself and then load it either by relative paths from the package or via
> > pkg_resources. For instance, one could put paste.conf into
> > /usr/local/lib/python2.7/site-packages/keystone/paste.ini instead of
> > into etc.
> >
> > Which is my way of saying:
> >
> > - If it's a file that one expects the user to edit, having pip/setup.py
> > install it is a bad idea
> > - If it's a file that one does NOT expect the user to edit, put it into
> > the package.
> 
> I agree with Monty here. The best design we could have is to have
> software that work by default with 0 configuration file.
> Unfortunately, this is not the case for any OpenStack server, and no
> design path has been drawn to follow yet. I'd be glad to see some change
> in this area.
> 
> We're almost at this point in Telemetry projects, we just have to
> carry/ship policy.json (no way to override easily AFAIK) and Paste
> config (no better way to tweak middleware config so far).
> (and the oslo-config-generator configuration file, but that should be
> embeddable easily).

For policy there is now https://review.openstack.org/#/c/309152/ which
allows for registering rules in code. It is currently being incorporated
into Nova https://review.openstack.org/#/c/290155/.

> 
> We'll try to push further in that direction in the future. Let's hope
> everyone will follow. :)
> 
> -- 
> Julien Danjou
> ;; Free Software hacker
> ;; https://julien.danjou.info
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe:
> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> Email had 1 attachment:
> + signature.asc
>   1k (application/pgp-signature)

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tc][pbr][packaging][all] Definition of Data Files (config) in setup.cfg

[Thomas Goirand]

On 06/12/2016 10:45 PM, Monty Taylor wrote:
> On 06/11/2016 10:13 PM, Robert Collins wrote:
>> On 12 June 2016 at 11:31, Thomas Goirand  wrote:
>>> On 06/11/2016 05:02 AM, Robert Collins wrote:
>> ...
>>> Then keystone.conf ends up installed in /usr/etc/keystone when setup.py
>>> install is used. With this configuration, it would be installed
>>> correctly within a venv.
>>
>> Except that noone uses 'setup.py install' - *except* for rpm and dpkg
>> packagers. Everyone else uses pip, and pip won't pass new options.
>> Worse, pip creates wheels and then installs the wheels, and until PEP
>> 491 [1] is finalised and implemented, wheels won't change their
>> behaviour.
>>
>>> If instead we have:
>>> [files]
>>> data_files =
>>> /etc/keystone/keystone.conf = etc/keystone.conf.sample
>>>
>>> [ notice the leading / for the destination... ]
>>>
>>> Then keystone.conf gets installed on the root of the filesystem instead
>>> of properly be installed within the venv. Though it's installed properly
>>> for distributions in /etc/keystone.
>>>
>>> To address this problem, Julien Danjou attempted to add a --sysconfdir
>>> option in PBR. Though it was rejected by Robert, who wrote in the code
>>> review that it should be addressed within setuptools or disutils.
>>
>> Not should. Must be. A local change to pbr won't solve the problem,
>> except for the very narrow case of package creators (who already have
>> excellent tools - its a single one-line file to fix it on Debian, and
>> I presume roughly the same on RPM distributions). It certainly won't
>> solve it for virtual envs.
>>
>>> If I got the above facts wrong, feel free to let everyone know. Now, my
>>> reply to Robert...
>>> === End of context ===
>>>
>>> Robert,
>>>
>>> I do not agree that we *must* wait until politics are discussed and a
>>
>> I'm going to quote what I said back in February - 4 months ago: "Ok,
>> so here's where I got to:
>>
>> - wheel doesn't support or understand /etc per se
>> - setuptools and distutils disagree on the meaning of data files paths
>> - wheel takes the distutils value today
>> - pep 491, which seems moribund, is an attempt to explicitly support
>> sysconf dir's like /etc, and thats what is needed in general.
>>
>> I think the way forward to get this available as a sensible,
>> supportable feature is to discuss it on the distutils-sig list, and
>> that community can help figure out a design, including the backwards
>> compat implications."
>>
>> I don't -2 lightly, and I did considerable research into this patch
>> before leaving the -2 in place.
>>
>> With regards to politics and consensus in the upstream packaging
>> community - well, it would move faster if folk who are distro folk and
>> thus want to see the mess sorted were collaborating there: its been 4
>> months since Julien's original patch and no discussion has taken
>> place, nor has anyone volunteered in that community to do the work.
>> Its not super hard, and I'd be happy to guide anyone interested in
>> reviving PEP 491 (or drafting an alternative). What I'm not interested
>> in doing is adding an ineffective hack to pbr which we'll have to
>> maintain for an extended period, which isn't actually in pbr's domain
>> (pbr exists to prevent duplication amongst OpenStack projects and
>> provide solid and sane policy, not to replace wheel and setuptools).
>> I'm sorry if that sounds like a hard line, but we had an ongoing
>> series of bugs in pbr - the gift that keeps on giving - from the
>> previous attempts to work around defects upstream of pbr rather than
>> fixing those defects directly, which is a much better strategy : less
>> cost, less technical debt, benefits the entire community, gets better
>> outcomes.
> 
> I'm going to have to second what Robert says here.
> 
> Also - I would like to add the following.
> 
> Having pip or setup.py install install a config file is at best a
> strange thing to want, as it does not have the mechanisms to deal with
> templating in values, or dealing with conflicts with locally changed
> file values on upgrades.
> 
> Consider the case where a person does "pip install keystone" to install
> keystone.
> 
> If that command installs keystone.conf into /etc/keystone - there is
> _NO_ way that that config file will have any useful values. The user
> will then edit the file to add the values into the file that need to be
> added. Then, the user will, at a later point in time do "pip install -U
> keystone" to upgrade the keystone installation.
> 
> What does pip do? Does it overwrite /etc/keystone/keystone.conf? What
> about the user's values? Does it NOT overwrite
> /etc/keystone/keystone.conf? Why not?
> 
> This is, btw, why dpkg and rpm and friends exist. It's their job.
> 
> Now, you might argue, there are files, such as the paste config files
> which are configs that the software needs to install to be useful.
> 
> I will call hogwash on that. That we install files such as that which
> are actually code has 

Re: [openstack-dev] [tc][pbr][packaging][all] Definition of Data Files (config) in setup.cfg

[Julien Danjou]

On Sun, Jun 12 2016, Monty Taylor wrote:

> I will call hogwash on that. That we install files such as that which
> are actually code has been a deployer nightmare since day one, and the
> solution is not to make pip smarter, it's to stop installing such files.
>
> NOW - if it's quite literally a file that just exists better as a file
> and not direct python code - that's fine - just put it into the package
> itself and then load it either by relative paths from the package or via
> pkg_resources. For instance, one could put paste.conf into
> /usr/local/lib/python2.7/site-packages/keystone/paste.ini instead of
> into etc.
>
> Which is my way of saying:
>
> - If it's a file that one expects the user to edit, having pip/setup.py
> install it is a bad idea
> - If it's a file that one does NOT expect the user to edit, put it into
> the package.

I agree with Monty here. The best design we could have is to have
software that work by default with 0 configuration file.
Unfortunately, this is not the case for any OpenStack server, and no
design path has been drawn to follow yet. I'd be glad to see some change
in this area.

We're almost at this point in Telemetry projects, we just have to
carry/ship policy.json (no way to override easily AFAIK) and Paste
config (no better way to tweak middleware config so far).
(and the oslo-config-generator configuration file, but that should be
embeddable easily).

We'll try to push further in that direction in the future. Let's hope
everyone will follow. :)

-- 
Julien Danjou
;; Free Software hacker
;; https://julien.danjou.info


signature.asc
Description: PGP signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tc][pbr][packaging][all] Definition of Data Files (config) in setup.cfg

[Monty Taylor]

On 06/11/2016 10:13 PM, Robert Collins wrote:
> On 12 June 2016 at 11:31, Thomas Goirand  wrote:
>> On 06/11/2016 05:02 AM, Robert Collins wrote:
> ...
>> Then keystone.conf ends up installed in /usr/etc/keystone when setup.py
>> install is used. With this configuration, it would be installed
>> correctly within a venv.
> 
> Except that noone uses 'setup.py install' - *except* for rpm and dpkg
> packagers. Everyone else uses pip, and pip won't pass new options.
> Worse, pip creates wheels and then installs the wheels, and until PEP
> 491 [1] is finalised and implemented, wheels won't change their
> behaviour.
> 
>> If instead we have:
>> [files]
>> data_files =
>> /etc/keystone/keystone.conf = etc/keystone.conf.sample
>>
>> [ notice the leading / for the destination... ]
>>
>> Then keystone.conf gets installed on the root of the filesystem instead
>> of properly be installed within the venv. Though it's installed properly
>> for distributions in /etc/keystone.
>>
>> To address this problem, Julien Danjou attempted to add a --sysconfdir
>> option in PBR. Though it was rejected by Robert, who wrote in the code
>> review that it should be addressed within setuptools or disutils.
> 
> Not should. Must be. A local change to pbr won't solve the problem,
> except for the very narrow case of package creators (who already have
> excellent tools - its a single one-line file to fix it on Debian, and
> I presume roughly the same on RPM distributions). It certainly won't
> solve it for virtual envs.
> 
>> If I got the above facts wrong, feel free to let everyone know. Now, my
>> reply to Robert...
>> === End of context ===
>>
>> Robert,
>>
>> I do not agree that we *must* wait until politics are discussed and a
> 
> I'm going to quote what I said back in February - 4 months ago: "Ok,
> so here's where I got to:
> 
> - wheel doesn't support or understand /etc per se
> - setuptools and distutils disagree on the meaning of data files paths
> - wheel takes the distutils value today
> - pep 491, which seems moribund, is an attempt to explicitly support
> sysconf dir's like /etc, and thats what is needed in general.
> 
> I think the way forward to get this available as a sensible,
> supportable feature is to discuss it on the distutils-sig list, and
> that community can help figure out a design, including the backwards
> compat implications."
> 
> I don't -2 lightly, and I did considerable research into this patch
> before leaving the -2 in place.
> 
> With regards to politics and consensus in the upstream packaging
> community - well, it would move faster if folk who are distro folk and
> thus want to see the mess sorted were collaborating there: its been 4
> months since Julien's original patch and no discussion has taken
> place, nor has anyone volunteered in that community to do the work.
> Its not super hard, and I'd be happy to guide anyone interested in
> reviving PEP 491 (or drafting an alternative). What I'm not interested
> in doing is adding an ineffective hack to pbr which we'll have to
> maintain for an extended period, which isn't actually in pbr's domain
> (pbr exists to prevent duplication amongst OpenStack projects and
> provide solid and sane policy, not to replace wheel and setuptools).
> I'm sorry if that sounds like a hard line, but we had an ongoing
> series of bugs in pbr - the gift that keeps on giving - from the
> previous attempts to work around defects upstream of pbr rather than
> fixing those defects directly, which is a much better strategy : less
> cost, less technical debt, benefits the entire community, gets better
> outcomes.

I'm going to have to second what Robert says here.

Also - I would like to add the following.

Having pip or setup.py install install a config file is at best a
strange thing to want, as it does not have the mechanisms to deal with
templating in values, or dealing with conflicts with locally changed
file values on upgrades.

Consider the case where a person does "pip install keystone" to install
keystone.

If that command installs keystone.conf into /etc/keystone - there is
_NO_ way that that config file will have any useful values. The user
will then edit the file to add the values into the file that need to be
added. Then, the user will, at a later point in time do "pip install -U
keystone" to upgrade the keystone installation.

What does pip do? Does it overwrite /etc/keystone/keystone.conf? What
about the user's values? Does it NOT overwrite
/etc/keystone/keystone.conf? Why not?

This is, btw, why dpkg and rpm and friends exist. It's their job.

Now, you might argue, there are files, such as the paste config files
which are configs that the software needs to install to be useful.

I will call hogwash on that. That we install files such as that which
are actually code has been a deployer nightmare since day one, and the
solution is not to make pip smarter, it's to stop installing such files.

NOW - if it's quite literally a file that just 

Re: [openstack-dev] [tc][pbr][packaging][all] Definition of Data Files (config) in setup.cfg

[Robert Collins]

On 12 June 2016 at 11:31, Thomas Goirand  wrote:
> On 06/11/2016 05:02 AM, Robert Collins wrote:
...
> Then keystone.conf ends up installed in /usr/etc/keystone when setup.py
> install is used. With this configuration, it would be installed
> correctly within a venv.

Except that noone uses 'setup.py install' - *except* for rpm and dpkg
packagers. Everyone else uses pip, and pip won't pass new options.
Worse, pip creates wheels and then installs the wheels, and until PEP
491 [1] is finalised and implemented, wheels won't change their
behaviour.

> If instead we have:
> [files]
> data_files =
> /etc/keystone/keystone.conf = etc/keystone.conf.sample
>
> [ notice the leading / for the destination... ]
>
> Then keystone.conf gets installed on the root of the filesystem instead
> of properly be installed within the venv. Though it's installed properly
> for distributions in /etc/keystone.
>
> To address this problem, Julien Danjou attempted to add a --sysconfdir
> option in PBR. Though it was rejected by Robert, who wrote in the code
> review that it should be addressed within setuptools or disutils.

Not should. Must be. A local change to pbr won't solve the problem,
except for the very narrow case of package creators (who already have
excellent tools - its a single one-line file to fix it on Debian, and
I presume roughly the same on RPM distributions). It certainly won't
solve it for virtual envs.

> If I got the above facts wrong, feel free to let everyone know. Now, my
> reply to Robert...
> === End of context ===
>
> Robert,
>
> I do not agree that we *must* wait until politics are discussed and a

I'm going to quote what I said back in February - 4 months ago: "Ok,
so here's where I got to:

- wheel doesn't support or understand /etc per se
- setuptools and distutils disagree on the meaning of data files paths
- wheel takes the distutils value today
- pep 491, which seems moribund, is an attempt to explicitly support
sysconf dir's like /etc, and thats what is needed in general.

I think the way forward to get this available as a sensible,
supportable feature is to discuss it on the distutils-sig list, and
that community can help figure out a design, including the backwards
compat implications."

I don't -2 lightly, and I did considerable research into this patch
before leaving the -2 in place.

With regards to politics and consensus in the upstream packaging
community - well, it would move faster if folk who are distro folk and
thus want to see the mess sorted were collaborating there: its been 4
months since Julien's original patch and no discussion has taken
place, nor has anyone volunteered in that community to do the work.
Its not super hard, and I'd be happy to guide anyone interested in
reviving PEP 491 (or drafting an alternative). What I'm not interested
in doing is adding an ineffective hack to pbr which we'll have to
maintain for an extended period, which isn't actually in pbr's domain
(pbr exists to prevent duplication amongst OpenStack projects and
provide solid and sane policy, not to replace wheel and setuptools).
I'm sorry if that sounds like a hard line, but we had an ongoing
series of bugs in pbr - the gift that keeps on giving - from the
previous attempts to work around defects upstream of pbr rather than
fixing those defects directly, which is a much better strategy : less
cost, less technical debt, benefits the entire community, gets better
outcomes.

-Rob

[1]: https://www.python.org/dev/peps/pep-0491/

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tc][pbr][packaging][all] Definition of Data Files (config) in setup.cfg

[Thomas Goirand]

On 06/11/2016 05:02 AM, Robert Collins wrote:
> The underlying problem is as you say that distutils and setuptools and
> wheel are all different, and until there is a PEP addressing this
> *and* implemented in both wheel and setuptools, we won't get
> consistent behaviour. I'm pretty sure Daniel Hoth is interested in
> making this happen - there have been threads about it on distutils-sig
> inside the last year, but we (me/sachi/steven) hadn't gotten onto it
> as a thing to help with. I suggest chatting to Daniel as a first point
> of call to see where his effort is at and what help is needed.
> 
> Note that we *need* the wheel and setuptools behaviour to be
> consistent, or we will have differences between distro package
> building, devstack, and ansible uses.. amongst others.
> 
> -Rob

=== A bit of context for everyone to understand ===
Just so that everyone understands and gets the context. If in a
setup.cfg managed by PBR, we have something like this:

[files]
data_files =
etc/keystone/keystone.conf = etc/keystone.conf.sample

Then keystone.conf ends up installed in /usr/etc/keystone when setup.py
install is used. With this configuration, it would be installed
correctly within a venv.

If instead we have:
[files]
data_files =
/etc/keystone/keystone.conf = etc/keystone.conf.sample

[ notice the leading / for the destination... ]

Then keystone.conf gets installed on the root of the filesystem instead
of properly be installed within the venv. Though it's installed properly
for distributions in /etc/keystone.

To address this problem, Julien Danjou attempted to add a --sysconfdir
option in PBR. Though it was rejected by Robert, who wrote in the code
review that it should be addressed within setuptools or disutils.

If I got the above facts wrong, feel free to let everyone know. Now, my
reply to Robert...
=== End of context ===

Robert,

I do not agree that we *must* wait until politics are discussed and a
consensus appears within the upstream python distutils / setuptools
folks. The issue we're trying to address here is annoying everyone, both
upstream using venv, and downstream package maintainers. The
--sysconfdir option that we've tried to add to PBR would be a very
pragmatic answer to the issue that would effectively be the end to this
non-sense.

I do agree though that it'd be nicer if things were to be addressed
within setuptools directly. But this can be done later. In the mean
while, each and every downstream distributions (which means at least 5
distros currently) have to fix the non-sense of config files installed
within /usr/etc.

I also would like to point out that it isn't a good idea for upstream
authors to double-guess what downstream package maintainers are
expecting. For example in Debian, "foo" service to ships its
configuration files within /usr/share/foo/foo.conf, and copies it in the
postinst to /etc/foo/foo.conf, so that it can be owned by the "foo"
system user. So if PBR+setuptools are automatically installing stuff for
me in /etc/foo, I would also need to work this around. So a new option
--no-config-file-install" would be useful for me. I'd happily add it to
PBR, though I'm afraid that you, Robert, will veto it, and that my patch
wont ever be merged. I'm busy with 380+ packages for OpenStack, and
don't want to waste too much time, so your view here would be useful
before I invest some time on this patch.

Thoughts anyone?

Cheers,

Thomas Goirand (zigo)


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tc][pbr][packaging][all] Definition of Data Files (config) in setup.cfg

[Robert Collins]

The underlying problem is as you say that distutils and setuptools and
wheel are all different, and until there is a PEP addressing this
*and* implemented in both wheel and setuptools, we won't get
consistent behaviour. I'm pretty sure Daniel Hoth is interested in
making this happen - there have been threads about it on distutils-sig
inside the last year, but we (me/sachi/steven) hadn't gotten onto it
as a thing to help with. I suggest chatting to Daniel as a first point
of call to see where his effort is at and what help is needed.

Note that we *need* the wheel and setuptools behaviour to be
consistent, or we will have differences between distro package
building, devstack, and ansible uses.. amongst others.

-Rob

On 11 June 2016 at 13:50, Morgan Fainberg  wrote:
> There has been a bit of back[1] and forth[2][3][4][5] between at least one
> packaging group and a few folks who are trying to define data files (config)
> in the setup.cfg to aid/ease installation within virtual environments.
>
> From what I can tell, there has been an issue with setuptools that makes
> this a particularly sticky situation and difficult to address in PBR via an
> option like --sysconfdir due to a disagreement between setuptools and
> disttools disagreeing upon the meaning of "data-files".  [6]
>
> Before this turns into a nightmare of add-patches/revert/add/revert and
> making things highly inconsistent within OpenStack, I'd like to get feedback
> from the packaging teams on where this impacts them. I know that a number of
> folks carry effectively this patch internally to make working with VENVs
> easier.
>
> We should absolutely address this in setuptools/distutils/etc but a clear
> direction forward so that the projects under OpenStack remain consistent for
> users, developers, and packagers would be good at this point.
>
> I know that keystone has had a lot of work done to ensure we can work in
> most "general" environments, but installing the data-files within a VENV is
> much more simple than a bunch of special casing to "find" the config files.
>
> In short, I think OpenStack needs to define (even if it's a short period of
> time) what we consider "data-files", we can always revisit this when/if we
> have a clear path forward via PBR, setuptools, disttools, etc.
>
> [1] https://review.openstack.org/#/c/322086/
> [2] https://review.openstack.org/#/c/326152/
> [3] http://git.openstack.org/cgit/openstack/neutron/tree/setup.cfg#n24
> [4] http://git.openstack.org/cgit/openstack/gnocchi/tree/setup.cfg#n87
> [5] http://git.openstack.org/cgit/openstack/aodh/tree/setup.cfg#n30
> [6] https://review.openstack.org/#/c/274077/
>
> Cheers,
> --Morgan
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [tc][pbr][packaging][all] Definition of Data Files (config) in setup.cfg

[Morgan Fainberg]

There has been a bit of back[1] and forth[2][3][4][5] between at least one
packaging group and a few folks who are trying to define data files
(config) in the setup.cfg to aid/ease installation within virtual
environments.

>From what I can tell, there has been an issue with setuptools that makes
this a particularly sticky situation and difficult to address in PBR via an
option like --sysconfdir due to a disagreement between setuptools and
disttools disagreeing upon the meaning of "data-files".  [6]

Before this turns into a nightmare of add-patches/revert/add/revert and
making things highly inconsistent within OpenStack, I'd like to get
feedback from the packaging teams on where this impacts them. I know that a
number of folks carry effectively this patch internally to make working
with VENVs easier.

We should absolutely address this in setuptools/distutils/etc but a clear
direction forward so that the projects under OpenStack remain consistent
for users, developers, and packagers would be good at this point.

I know that keystone has had a lot of work done to ensure we can work in
most "general" environments, but installing the data-files within a VENV is
much more simple than a bunch of special casing to "find" the config files.

In short, I think OpenStack needs to define (even if it's a short period of
time) what we consider "data-files", we can always revisit this when/if we
have a clear path forward via PBR, setuptools, disttools, etc.

[1] https://review.openstack.org/#/c/322086/
[2] https://review.openstack.org/#/c/326152/
[3] http://git.openstack.org/cgit/openstack/neutron/tree/setup.cfg#n24
[4] http://git.openstack.org/cgit/openstack/gnocchi/tree/setup.cfg#n87
[5] http://git.openstack.org/cgit/openstack/aodh/tree/setup.cfg#n30
[6] https://review.openstack.org/#/c/274077/

Cheers,
--Morgan
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev