Re: /usr/local is loved by Debian Python people?
On Mon, Feb 02, 2009 at 11:46:57PM -0500, Yaroslav Halchenko wrote: what was the actual use-case they solved? As I understand it, the use case is analogous to the addition of ‘/usr/local/bin’ to the ‘PATH’ environment variable in the default shell profile. ok -- that would be my next target! ;) I think you misunderstand the purpose of /usr/local. It is not a place for users to put stuff only they need. It can't be, because /usr/local is not writeable by ordinary user, but just by the system administrator. /usr/local is the place for the system administrator to manually install programs, libraries, and other stuff not tracked by Debian's packaging system. As such it is assumed that the system administrator does know what he's doing and Debian won't second-guess him and let those things in /usr/local take precedence. - Sebastian -- To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Re: /usr/local is loved by Debian Python people?
Thank you guys for reference to FHS. I did read it long ago and now glanced over again to confirm my knowledge. Actually, since FHS does not say anything about ownership of the files in FH, so Debian exploits that in its own way: $ ls -ld /usr /usr/local 0 drwxr-xr-x 15 root root 440 2008-02-17 14:56 /usr/ 0 drwxrwsr-x 10 root staff 272 2006-11-03 15:49 /usr/local/ and that is due to Debian policy (9.1.2). Unfortunately I can't find clear description of the 'staff' group destiny nowhere in Debian documentation (ie policy or dev-ref), but clearly there is now difference between pure Administrator role (ie root) and possibly a larger group of users in 'staff' system group. What I know though, that in real-world situations, users who need to test their software by installing under '/local' are added to 'staff' group by the Administrator (sure thing, most of the time it is the same 'real' person ;)). That makes it easy to 'locally' customize the system, and I take it as an advantage. That way Python developers install needed eggs and custom versions of python. And that is what actually then drives them and other users of the system into the madness -- that now /usr/local-installed things interfere with functioning of components installed properly by system administrator under /usr/(non-local). And that there is no transparent way to adjust such setup. If in the case of shells, presence of /usr/local paths in the PATH is easy to configure due to present configuration files (as I mentioned before), also in the case of libraries, it is easily configurable via modification of /etc/ld.so.conf.d/libc.conf, in Python's case, there is NO way to have it configured in the desired way -- ie not to have /usr/local components loaded automagically. And that is the problem. On Mon, 02 Feb 2009, Guy Hulbert wrote: I think you will find the answer here: http://www.pathname.com/fhs/ On Tue, 03 Feb 2009, Sebastian Rittau wrote: I think you misunderstand the purpose of /usr/local. It is not a place for users to put stuff only they need. It can't be, because /usr/local is not writeable by ordinary user, but just by the system administrator. /usr/local is the place for the system administrator to manually install programs, libraries, and other stuff not tracked by Debian's packaging system. As such it is assumed that the system administrator does know what he's doing and Debian won't second-guess him and let those things in /usr/local take precedence. -- Yaroslav Halchenko Research Assistant, Psychology Department, Rutgers-Newark Student Ph.D. @ CS Dept. NJIT Office: (973) 353-1412 | FWD: 82823 | Fax: (973) 353-1171 101 Warren Str, Smith Hall, Rm 4-105, Newark NJ 07102 WWW: http://www.linkedin.com/in/yarik -- To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Re: /usr/local is loved by Debian Python people?
On Tue, 2009-03-02 at 09:19 -0500, Yaroslav Halchenko wrote: Having proper configuration file under /etc sounds much more viable and correct solution. You should do it under /usr/share/APP/ the /etc/APP/ directory is for your .ini files (see FHS). I have: /usr/share/APP/perl/ /usr/share/APP/sbin/ Scripts in sbin will have unshift @INC,'/usr/share/APP/perl' or use lib 'usr/share/APP/perl' It is a PITA for development but ... -- --gh -- To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Re: /usr/local is loved by Debian Python people?
On Tue, 2009-03-02 at 09:06 -0500, Yaroslav Halchenko wrote: in Python's case, there is NO way to have it configured in the desired way -- ie not to have /usr/local components loaded automagically. And that is the problem. You can do: #!python import sys and edit sys.path I am doing the equivalent thing in perl. -- --gh -- To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Re: /usr/local is loved by Debian Python people?
1. remove /usr/local from site.py This denies the ability of the system administrator to put local overrides for Python library files in place as simply and consistently as is done for other libraries and executables. indeed, precedence is smth I didn't think through in my suggested approach. not sure what I was smoking that late evening actually: *$ PYTHONPATH=/buga python -c import sys; print sys.path ['', '/buga', '/usr/lib/python24.zip', ... so precedence capability is there but that is irrelevant since that method of custom pythonX.X-local is indeed ugly. Having proper configuration file under /etc sounds much more viable and correct solution. -- Yaroslav Halchenko Research Assistant, Psychology Department, Rutgers-Newark Student Ph.D. @ CS Dept. NJIT Office: (973) 353-1412 | FWD: 82823 | Fax: (973) 353-1171 101 Warren Str, Smith Hall, Rm 4-105, Newark NJ 07102 WWW: http://www.linkedin.com/in/yarik -- To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Re: /usr/local is loved by Debian Python people?
Le lundi 02 février 2009 à 22:35 -0500, Yaroslav Halchenko a écrit : why in a hackish site.py those /usr/local paths are added? Because we want to follow the FHS, which is all but hackish - but Python, in many ways, doesn’t make it easy. The real question is more: why aren’t these paths correctly used? Said otherwise, bug #338572. -- .''`. : :' : We are debian.org. Lower your prices, surrender your code. `. `' We will add your hardware and software distinctiveness to `-our own. Resistance is futile. signature.asc Description: Ceci est une partie de message numériquement signée
Re: /usr/local is loved by Debian Python people?
On Tue, 03 Feb 2009, Guy Hulbert wrote: You should do it under /usr/share/APP/ the /etc/APP/ directory is for your .ini files (see FHS). I have: /usr/share/APP/perl/ /usr/share/APP/sbin/ Scripts in sbin will have unshift @INC,'/usr/share/APP/perl' or use lib 'usr/share/APP/perl' It is a PITA for development but ... hm... sorry, but I don't see the actual point... configuration (.ini) needs to be under /etc, and that is what I am aiming at... What I want is to have just 1 tiny section in /etc/python/debian_config something like [PATH] add-local = true or just path-add-local = true in the [DEFAULT] instead of a monstrous hackish /etc/python2.X/site.py which was later on actually removed the honor to be a config file and replaced in 2.5.2-1 with somewhat more concise sitecustomize.py, but I am not even sure if it is appropriate place to add a 1-liner to customize sys.path. And imho it is not a config file per se -- it is once again just a snippet of code, although nice and concise. Also such decision (to include or not-include local dirs) is not python-version specific, so imho /etc/python/debian_config sounds like a perfect location to specify such behavior -- Yaroslav Halchenko Research Assistant, Psychology Department, Rutgers-Newark Student Ph.D. @ CS Dept. NJIT Office: (973) 353-1412 | FWD: 82823 | Fax: (973) 353-1171 101 Warren Str, Smith Hall, Rm 4-105, Newark NJ 07102 WWW: http://www.linkedin.com/in/yarik -- To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Re: /usr/local is loved by Debian Python people?
thank you Josselin, I've seen that one while I checked if there is any open issue and I agree that FHS is all by hackish -- but unfortunately Debian has diverged from it in terms of ownership, like we figured out elsewhere in this thread: msg-id: 20090203143602.gt25...@washoe.rutgers.edu so, it opens up new horizons on how to fix appropriately to account for such divergence in a sane way On Tue, 03 Feb 2009, Josselin Mouette wrote: Le lundi 02 février 2009 à 22:35 -0500, Yaroslav Halchenko a écrit : why in a hackish site.py those /usr/local paths are added? Because we want to follow the FHS, which is all but hackish - but Python, in many ways, doesn’t make it easy. The real question is more: why aren’t these paths correctly used? Said otherwise, bug #338572. -- Yaroslav Halchenko Research Assistant, Psychology Department, Rutgers-Newark Student Ph.D. @ CS Dept. NJIT Office: (973) 353-1412 | FWD: 82823 | Fax: (973) 353-1171 101 Warren Str, Smith Hall, Rm 4-105, Newark NJ 07102 WWW: http://www.linkedin.com/in/yarik -- To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Re: /usr/local is loved by Debian Python people?
ok -- here is my tentative cruel patch as a proof of concept. It should
not alter current behavior, but would allow to control for it.
See it attached
configuration (.ini) needs to be under /etc, and that is what I am
aiming at...
--
Yaroslav Halchenko
Research Assistant, Psychology Department, Rutgers-Newark
Student Ph.D. @ CS Dept. NJIT
Office: (973) 353-1412 | FWD: 82823 | Fax: (973) 353-1171
101 Warren Str, Smith Hall, Rm 4-105, Newark NJ 07102
WWW: http://www.linkedin.com/in/yarik
--- /etc/python/debian_config.orig 2009-02-03 10:26:30.0 -0500
+++ /etc/python/debian_config 2009-02-03 10:40:03.0 -0500
@@ -1,3 +1,7 @@
[DEFAULT]
# how to byte-compile (comma separated: standard, optimize)
byte-compile = standard
+
+# either to add /usr/local/lib/pythonX.X/site-packages
+# to default sys.path.
+path-add-local = true
--- /usr/lib/python2.5/site.py.orig2 2009-02-03 10:25:45.0 -0500
+++ /usr/lib/python2.5/site.py 2009-02-03 10:39:14.0 -0500
@@ -171,7 +171,17 @@
prefixes = [sys.prefix]
if sys.exec_prefix != sys.prefix:
prefixes.append(sys.exec_prefix)
-prefixes.insert(0, '/usr/local')
+try:
+cfg_pref = 'path-add-local'
+path_add_local = \
+bool({'0':0, 'false':0, '1':1, 'true':1}[
+ [l.replace(cfg_pref, '').strip(' =\t\n').lower()
+for l in open('/etc/python/debian_config').readlines()
+if l.startswith(cfg_pref)][-1]])
+except:
+path_add_local = False
+if path_add_local:
+prefixes.insert(0, '/usr/local')
for prefix in prefixes:
if prefix:
if sys.platform in ('os2emx', 'riscos'):
signature.asc
Description: Digital signature
Re: /usr/local is loved by Debian Python people?
On Tue, 2009-03-02 at 09:06 -0500, Yaroslav Halchenko wrote: Unfortunately I can't find clear description of the 'staff' group destiny nowhere in Debian documentation Fortunately, I just spent 20-30 minutes going through this on Sunday. http://www.debian.org/doc/manuals/securing-debian-howto/ch12.en.html Scroll down to: 12.1.12 Operating system users and groups I was reporting something to secur...@debian.org ... they acknowledged my initial inquiry but have not responded on the issue I pointed out (very minor) but you are looking in exactly the same place. -- --gh -- To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Re: /usr/local is loved by Debian Python people?
On Tue, 2009-03-02 at 10:00 -0500, Yaroslav Halchenko wrote: It is a PITA for development but ... hm... sorry, but I don't see the actual point... It's actually quite easy for someone in the 'staff' group to get root privileges ... I told secur...@debian.org on Sunday exactly how and exactly how to fix it but no-one got back to me about whether they care. It also requires some social engineering but nothing that would be suspicious. Anyhow, I'm writing stuff to do sysadmin so for me that's the point. Because /usr/local comes first in the default perl and python paths the 'staff' group is automatically trusted ... and I'm not so trusting. -- --gh -- To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Re: /usr/local is loved by Debian Python people?
ho ho -- thank you Guy! so, here it is: ,-, | staff: Allows users to add local modifications to the system (/usr/local, | | /home) without needing root privileges. Compare with group adm, which is more | | related to monitoring/security. | `-' Hence, Debian, by design (and by policy), allows users to modify the system... So 'staff' group is much broader than notion of 'Administrator', hence, it might be unsafe to add a user to staff group without ability to prevent default behavior of the system to use the content of /usr/local. NB: actually on a freshly installed lenny system: $ ls -ld /home 4 drwxr-xr-x 10 root root 4096 2009-01-03 16:23 /home/ and I don't see actual need for 'staff' to modify /home, since staff group is not authorized to add users. On Tue, 03 Feb 2009, Guy Hulbert wrote: Fortunately, I just spent 20-30 minutes going through this on Sunday. http://www.debian.org/doc/manuals/securing-debian-howto/ch12.en.html Scroll down to: 12.1.12 Operating system users and groups I was reporting something to secur...@debian.org ... they acknowledged my initial inquiry but have not responded on the issue I pointed out (very minor) but you are looking in exactly the same place. -- Yaroslav Halchenko Research Assistant, Psychology Department, Rutgers-Newark Student Ph.D. @ CS Dept. NJIT Office: (973) 353-1412 | FWD: 82823 | Fax: (973) 353-1171 101 Warren Str, Smith Hall, Rm 4-105, Newark NJ 07102 WWW: http://www.linkedin.com/in/yarik -- To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Re: /usr/local is loved by Debian Python people?
On Tue, Feb 03, 2009 at 09:18:31AM -0500, Guy Hulbert wrote: On Tue, 2009-03-02 at 09:06 -0500, Yaroslav Halchenko wrote: Unfortunately I can't find clear description of the 'staff' group destiny nowhere in Debian documentation Fortunately, I just spent 20-30 minutes going through this on Sunday. http://www.debian.org/doc/manuals/securing-debian-howto/ch12.en.html Scroll down to: 12.1.12 Operating system users and groups You can also find this in /usr/share/doc/base-paswd/users-and-groups.txt.gz Regards Floris -- Debian GNU/Linux -- The Power of Freedom www.debian.org | www.gnu.org | www.kernel.org -- To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Re: /usr/local is loved by Debian Python people?
On Tue, 2009-03-02 at 20:22 +, Floris Bruynooghe wrote: Fortunately, I just spent 20-30 minutes going through this on Sunday. http://www.debian.org/doc/manuals/securing-debian-howto/ch12.en.html Scroll down to: 12.1.12 Operating system users and groups You can also find this in /usr/share/doc/base-paswd/users-and-groups.txt.gz Ah yes. I was using google though. -- --gh -- To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
/usr/local is loved by Debian Python people?
Hi to all Debian Python-ers, I know that it is written policy (not as strong as DFSG, nor Debian policy but still a policy) I know that it has been such way for a while by now and not too many people raised concern to cause any action I know that not too many of regular users are hurt but I can't really comprehend WHY Debian's installation of Python decided to diverge from a common behavior on other distributions: why in a hackish site.py those /usr/local paths are added? what was the actual use-case they solved? isn't it more natural for people installing smth under /usr/local to adjust their PYTHONPATH env variable and be happy without interfering with other users of the system they share, who do not want to use what is under /usr/local? why should I in my script to take care about infiltration of /usr/local away from sys.path to prevent such interference mentioned above? or may be there is a magic 'PYTHON_NO_LOCAL=1' environment variable which would allow me to avoid such a pain on per-user basis? why it was hardcoded in the distributed non-configuration site.py, which I can't even configure to prevent such behavior without doing tricks to prevent its automagic 'fix' on every upgrade? I would suggest my idiosyncratic solution, which imho would remove some magic and make things transparent and consistent: 1. remove /usr/local from site.py 2. for convenience of users who like to run smth with /usr/local in mind, but hate to tune PYTHONPATH provide pythonX.X-local wrapper which adds /usr/local/... paths to PYTHONPATH prior to call to pythonX.X provide alternatives for pythonX.X to choose between the two (native pythonX.X and pythonX.X-local), with pure pythonX.X being default. is there anyone else who feels similar way? -- Yaroslav Halchenko Research Assistant, Psychology Department, Rutgers-Newark Student Ph.D. @ CS Dept. NJIT Office: (973) 353-1412 | FWD: 82823 | Fax: (973) 353-1171 101 Warren Str, Smith Hall, Rm 4-105, Newark NJ 07102 WWW: http://www.linkedin.com/in/yarik signature.asc Description: Digital signature
Re: /usr/local is loved by Debian Python people?
Yaroslav Halchenko deb...@onerussian.com writes: WHY Debian's installation of Python decided to diverge from a common behavior on other distributions: why in a hackish site.py those /usr/local paths are added? I don't know what “hackish” means here. I'll assume you are asking only “why are the ‘/usr/local/foo’ directories in the Python import search path?”. what was the actual use-case they solved? As I understand it, the use case is analogous to the addition of ‘/usr/local/bin’ to the ‘PATH’ environment variable in the default shell profile. isn't it more natural for people installing smth under /usr/local What is ‘smth’? I can't find a package by that name. to adjust their PYTHONPATH env variable and be happy without interfering with other users of the system they share, who do not want to use what is under /usr/local? Again, the idea of ‘/usr/local’ is that it allows the site administrator to install files, libraries, binaries, etc. that will override the OS-installed versions. Adjusting the search path so that those files are found first is necessary to allow that precedence. why should I in my script to take care about infiltration of /usr/local away from sys.path to prevent such interference mentioned above? You would need to ask the system administrator who installed files in that location. Debian, according to Policy, never installs any program or library in ‘/usr/local’; that's the point. why it was hardcoded in the distributed non-configuration site.py, which I can't even configure to prevent such behavior without doing tricks to prevent its automagic 'fix' on every upgrade? Can't the user who wants to avoid what the system administrator has deliberately put in place just modify their own ‘PYTHONPATH’ environment variable? Again, analogous to the situation with shell executable search paths. I would suggest my idiosyncratic solution, which imho would remove some magic and make things transparent and consistent: 1. remove /usr/local from site.py This denies the ability of the system administrator to put local overrides for Python library files in place as simply and consistently as is done for other libraries and executables. 2. for convenience of users who like to run smth with /usr/local in mind, but hate to tune PYTHONPATH What is ‘smth’, and how are its users different in their needs for a consisten search path precedence? Why are such users to be treated differently from those who override the ‘PATH’ variable? is there anyone else who feels similar way? Perhaps, but the case has yet to be made why Python should be treated differently in this regard from so many other parts of the system. -- \ “I must say that I find television very educational. The minute | `\ somebody turns it on, I go to the library and read a book.” | _o__)—Groucho Marx | Ben Finney -- To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Re: /usr/local is loved by Debian Python people?
Hi Ben, Thank you for your rapid reply! WHY Debian's installation of Python decided to diverge from a common behavior on other distributions: why in a hackish site.py those /usr/local paths are added? I don't know what “hackish” means here. I meant to say hackish fashion I guess ;) as for meanings: dict hackish: ..skips.. Said of something that is or involves a hack. dict hack: ..skips.. 6. (Computers) A quick and inelegant, though functional solution to a programming problem. I'll assume you are asking only “why are the ‘/usr/local/foo’ directories in the Python import search path?”. rright ;) what was the actual use-case they solved? As I understand it, the use case is analogous to the addition of ‘/usr/local/bin’ to the ‘PATH’ environment variable in the default shell profile. ok -- that would be my next target! ;) isn't it more natural for people installing smth under /usr/local What is ‘smth’? I can't find a package by that name. sorry -- me typing fast smth == something to adjust their PYTHONPATH env variable and be happy without interfering with other users of the system they share, who do not want to use what is under /usr/local? Again, the idea of ‘/usr/local’ is that it allows the site administrator to install files, libraries, binaries, etc. that will override the OS-installed versions. Adjusting the search path so that those files are found first is necessary to allow that precedence. so, freshly installed parts under /usr/local should precede system-wide ones... why then keep system-wide installed parts at all? ok ok -- it might be desired to keep packages in a sane state (ie not removing them forcing depends, if there are packages which depend on them). I can only then pride those who would need to figure out anything in the bug reports (bless python which gives backtraces with full paths in them) ...sorry for non-constructive blurb here... just to want to accent on the fact, that package versioning now in such bug-reports becomes irrelevant if smth is preceded them in /usr/local. why should I in my script to take care about infiltration of /usr/local away from sys.path to prevent such interference mentioned above? You would need to ask the system administrator who installed files in that location. Debian, according to Policy, never installs any program or library in ‘/usr/local’; that's the point. I know that Debian never does such evil thing... that is why I usually love to see my /usr/local as empty as possible ;) and I love Debian ;) But what if I haven't made friends with my sysadmin? Am I doomed to 'infiltration' approach now? why it was hardcoded in the distributed non-configuration site.py, which I can't even configure to prevent such behavior without doing tricks to prevent its automagic 'fix' on every upgrade? Can't the user who wants to avoid what the system administrator has deliberately put in place just modify their own ‘PYTHONPATH’ environment variable? Again, analogous to the situation with shell executable search paths. nope -- there is no PYTHONPATH env variable predefining those paths (unlike PATH). site.py populates sys.path at runtime. 1. remove /usr/local from site.py This denies the ability of the system administrator to put local overrides for Python library files in place as simply and consistently as is done for other libraries and executables. indeed, precedence is smth I didn't think through in my suggested approach. I guess either making sys.path configurable in some config file under /etc/, or controllable with PYTHON_PRECEDE_LOCAL=1 environment variable handled in site.py, would resolve this inconsistency. There could be better solutions as well 2. for convenience of users who like to run smth with /usr/local in mind, but hate to tune PYTHONPATH What is ‘smth’, and how are its users different in their needs for a consisten search path precedence? it is the same smth as above ;) sorry again so far it is inconsistent with other distributions, as far as I see it Why are such users to be treated differently from those who override the ‘PATH’ variable? is there anyone else who feels similar way? Perhaps, but the case has yet to be made why Python should be treated differently in this regard from so many other parts of the system. does bash hard-code PATH inside its code? nope, it is in a configuration file /etc/profile, and further can be tuned up in ~/.profile to infiltrate ones for a session if desired. or better directly from a configuration file, which /etc/profile is. once again -- site.py is not a configuration file, it is not among other configuration files (ie under /etc), there is no way to control the situation now, besides begging sysadmin to remove his crap, or populating the code with sys.path filtering list comprehensions, hence making it incompatible with those systems where /local installation has to be used for a good reason. -- Yaroslav Halchenko
Re: /usr/local is loved by Debian Python people?
On Mon, 2009-02-02 at 22:35 -0500, Yaroslav Halchenko wrote: WHY Debian's installation of Python decided to diverge from a common behavior on other distributions: Yaroslav. I think you will find the answer here: http://www.pathname.com/fhs/ I have a slightly different problem with perl. why in a hackish site.py those /usr/local paths are added? what was the actual use-case they solved? -- --gh -- To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
