Re: [Catalog-sig] Package comments
René Dudfield wrote: On Wed, Nov 4, 2009 at 9:55 AM, M.-A. Lemburg m...@egenix.com wrote: Regarding the usefulness of such a feature, take the PIL package as example: http://pypi.python.org/pypi/PIL/1.1.6 Package rating (3 votes): 4.667 * 4 points: 1 vote * 5 points: 2 votes Ratings range from 0 to 5 (best). Package Comments: * Hugely stable, been around forever, and works. Unfortunate distribution naming causes problems with setuptools. (chrisw, 2009-10-05, points) * super cool lib - #1 pick! just sad to see the packaging difficulties. (jensens, 2009-10-05, points) Those comments are not really all that useful for a user, since they put too much emphasis on a non-package related issue. This is like saying: Great bag, but doesn't come in pink, so only 4 points.. An educated user will notice, a casual user will just see the negative vibes and not bother with PIL, since it causes problems - now *that* is sad. hello, I do see how they could be perceived as not being good comments. However I see them differently, as being useful... Those comments are both positive, and friendly... but are also offering constructive critisism. They give an overall good impression of PIL(that it deserves). They both mention a common problem people have with PIL/Image/python imaging. I've had this problem myself, and I've helped people with this problem over the years. It's great feedback for the authors of PIL in my opinion, and is a win for the comments on pypi. If the authors decide to fix the problems mentioned, then it's a win for users of PIL too. People with commercial packages on there would be right to not like comments in some respects. Since commercial organisations often like to control their PR as much as possible. So in this way, the comments are not such a good thing for PIL. However also, commercial organisations pay a lot for feedback, QA and market research... so in a way it is also good for commercial packages too. The python community likes openness I would say, and comments go towards more openness. Comments move the communication on pypi from entirely author based, towards letting users speak as well. Should openness be valued more than valuing an authors wishes? There are enough ways available already to contact the author, provide constructive criticism, feedback, etc. Users can do so both in public and private ways. The problem with making PyPI a social platform is that you are removing the neutral view on things from the index - something I think encourages people to upload their stuff to PyPI in the first place. By not having comments on PyPI you don't limit the openness we have in the Python community in any way. Users can still search for user comments (and will readily find them), if they care to do so, or just ask on c.l.p for first-hand experience with a particular package or recommendation on which to choose. Packages can already remove themselves from pypi if they don't like it. However, as someone said before, being able to disable certain features would still let them use the features they want. Making ratings and comments optional would certainly help. Only leaving the option to remove the complete package in order to opt-out of these features is not a very Pythonic way to go about these things (and indeed does limit the openness). -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Nov 04 2009) Python/Zope Consulting and Support ...http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ ::: Try our new mxODBC.Connect Python Database Interface for free ! eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig
Re: [Catalog-sig] Package comments
Laura Creighton wrote: In a message of Wed, 04 Nov 2009 20:13:45 +0100, M.-A. Lemburg writes: snip That said, I don't think it's a good idea to try to reinvent a wheel that has already been invented many times over. Just look at the successful systems running on e.g. Amazon and eBay. We don't really need to go through all the pain they've gone through to build successful systems again. -- Marc-Andre Lemburg eGenix.com I don't think that the Amazon system is successful. I think that you are confusing two things -- 'people like to rate things' with 'the rating is meaningful'. I think that the Amazon rating system satifices people's desire to rate things, but is a detriment to actually finding a technical book that will help you do what you want to do. I wouldn't generalize that. I've often found information in those reviews that was otherwise hard to find. My overall experience with the Amazon system is more on the positive side. However, perhaps Amazon is the wrong comparison: their review system focuses on longer comments, whereas the eBay system has a focus on very terse comments. Most users will probably only look at two things: the overall rating figure and the negative comments. The PyPI approach appears to follow the eBay example, or at least the comments I found so far (not that many) go in this direction. Only fairly detailed reviews can help with that, and only to the extent that they exclude certain works, leaving you to review the remainder on your own. Why should software be any different? Also, remember, amazon makes money no matter whose books are purchased, But we would rather help people find the best packages for their purposes, no? I think that a rating system harms this goal, and therefore we should not do it. I tend to agree. Discussions on newsgroups, user reports in blogs, etc. are far more informative than an eBay style rating system. Perhaps requiring at least 50 words for the comment would help ;-) -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Nov 04 2009) Python/Zope Consulting and Support ...http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ ::: Try our new mxODBC.Connect Python Database Interface for free ! eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig
Re: [Catalog-sig] Poll started
Martin v. Löwis wrote: I just started a poll on the PyPI rating system, at http://pypi.python.org/pypi I'll announce it more widely tomorrow. I think the poll is missing an important option: [ ] Allow package owners to disallow comments and/or ratings. But more importantly: how does that voting system work ? I don't see any buttons or checkboxes on the wiki page to which the public poll link redirects. I also tried logging in to the wiki. No change... Ah, after logging in to PyPI, I now get to see some choices. Suggested reqording: If you want to vote, you need to login to PyPI (on the right). Regards, -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Nov 12 2009) Python/Zope Consulting and Support ...http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ ::: Try our new mxODBC.Connect Python Database Interface for free ! eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig
Re: [Catalog-sig] Poll started
Martin v. Löwis wrote: M.-A. Lemburg wrote: Martin v. Löwis wrote: I'll announce it more widely tomorrow. I think the poll is missing an important option: [ ] Allow package owners to disallow comments and/or ratings. :-( Why didn't you add this to the wiki page in the past days? Sorry about that: Last time I looked on that page the voting section wasn't there yet. I actually don't understand the option: what is and/or? Package owners should get two checkboxes: [ ] Allow comments [ ] Allow ratings That's about as open as it can get. The only question left then is what to use as default. I think it's too late now, now that the poll has started. It's just a poll to get a feeling for user sentiments, not a vote of any kind. Even as poll it will be difficult to interpret the responses to guide the development. These are the current figures: Allow ratings and comments on all packages (status quo) 76 Allow package owners to disallow comments (ratings unmodified). 67 Allow comments, but only send them to package owners (ratings unmodified). 18 Disallow comments (ratings unmodified). 15 Disallow ratings and comments (status three months ago).68 Now let's say you leave the current implementation in place. This would mean that you satisfy the user feelings of 76 users, but also means that you don't respect the answers of 168 users - even though the poll makes it look like keeping the status quo is getting a slight majority. Also, I didn't hear anybody say that they wanted to opt out of ratings - people opposed to ratings always wanted to turn ratings off entirely, as they felt it was a useless way of communicating, or not appropriate for PyPI, etc. I think the best way to handle all this is by giving the package authors the choice of whether to accept ratings or comments as two separate check-boxes. If you then use the status of three months ago as default (ie. rating and comments are turned off by default), you'd avoid all the controversy these features have caused. It's also a good idea to provide a check-box to disable sending out emails to the authors for comments. And if you want to make the users who answered Allow comments, but only send them to package owners happy, also add an option to keep comments private. That said, I still believe that PyPI should stay the neutral place for uploading packages. We already have enough controversy trying to sort out the whole package installation story. IMHO, it's not really useful to open up another can of worms right now. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Nov 13 2009) Python/Zope Consulting and Support ...http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ ::: Try our new mxODBC.Connect Python Database Interface for free ! eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig
Re: [Catalog-sig] Poll Problem
Chris Withers wrote: Martin v. Löwis wrote: But yes, if you have a way, please reduce my votes to one ;-) Done! Let me know if whether I missed any accounts, and revote if I indeed found them all. Can you please document here what you've done to ensure there are no duplicate votes from other people who really are trying to stuff the ballot? I think there's a misunderstanding here: the poll is not an official vote on the future of the PyPI comment and rating system, it's just a market research method to get at least some rough idea of what users (with PyPI accounts) want. Nothing more. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Nov 14 2009) Python/Zope Consulting and Support ...http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ ::: Try our new mxODBC.Connect Python Database Interface for free ! eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig
Re: [Catalog-sig] Poll Problem
P.J. Eby wrote: At 08:15 PM 11/14/2009 +0100, M.-A. Lemburg wrote: Chris Withers wrote: Martin v. Löwis wrote: But yes, if you have a way, please reduce my votes to one ;-) Done! Let me know if whether I missed any accounts, and revote if I indeed found them all. Can you please document here what you've done to ensure there are no duplicate votes from other people who really are trying to stuff the ballot? I think there's a misunderstanding here: the poll is not an official vote on the future of the PyPI comment and rating system, it's just a market research method to get at least some rough idea of what users (with PyPI accounts) want. Nothing more. Then why does the poll page itself say that The option that receives most votes will be implemented? Good question. I think that's a documentation bug :-) The wiki page uses a much better description (http://wiki.python.org/moin/PyPIComments): There is a public poll running to determine what users think of the rating and commenting features. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Nov 14 2009) Python/Zope Consulting and Support ...http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ ::: Try our new mxODBC.Connect Python Database Interface for free ! eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig
Re: [Catalog-sig] OpenID login to PyPI
Georg Brandl wrote: Is the plan to eventually disable non-OpenID authentication? I hope not. Same here. This whole OpenID thing appears to introduce more problems than it solves. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Nov 16 2009) Python/Zope Consulting and Support ...http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ ::: Try our new mxODBC.Connect Python Database Interface for free ! eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig
Re: [Catalog-sig] OpenID login to PyPI
Martin v. Löwis wrote: Well, python-openid is written in Python, so it should be possible to add whatever special functionality you need. For this, I would like to see contributions. I found myself that it is absolutely necessary to understand the actual message flow, so I couldn't have used the package before implementing a client myself - perhaps I might now be able to understand how to use it. For mod_auth_openid, I still cannot grasp how to use it. With mod_auth_openid, you'd use a separate login page for the OpenID login, so there wouldn't be a mixup between the two variants. That would be unfortunate UI clutter, IMO. Since you were looking for simplification of the used code, it may be worthwhile looking at these two options to outsource the complexity into a 3rd party tool. Both come with their own session database to handle the authentication process sessions. Please go ahead and do so. Note that I'm only suggesting to look at these things in order to simplify the implementation. At least for python-openid, my feeling would be that it is actually more complicated to use the library than to write my own. For mod_auth_openid, a shallow glance also looks like using it would be quite some challenge. Like I said in my email: I'm not a fan of OpenID and don't understand why so much complexity has to be added just to avoid letting a browser fill in the username/password automatically. My suggestions were only targeting your statement that you want to keep the code simple. If you don't think that's possible using python-openid or Paul's suggestion to use mod_auth_openid, that's fine. Regarding contributing to the PyPI implementation: I'm not aware of how this could be done. To me, PyPI appears to be a closed application that is solely under your control. After some searching I did find what appears to be the repository of the currently used code base: https://svn.python.org/packages/trunk/pypi/ However, it is not clear how this is deployed on the pypi.python.org server. Is there a staging installation to be used for testing ? -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Nov 18 2009) Python/Zope Consulting and Support ...http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ ::: Try our new mxODBC.Connect Python Database Interface for free ! eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig
Re: [Catalog-sig] PyPI Development
M.-A. Lemburg wrote: Martin v. Löwis wrote: However, it is not clear how this is deployed on the pypi.python.org server. Is there a staging installation to be used for testing ? See http://wiki.python.org/moin/CheeseShopDev Thanks for the link... I should have searched for CheeseShop instead of PyPI. I'd like to work on adding support for registering more package download information, including the possibility to register download links (as opposed to uploading files to PyPI itself) and more information on the intended target platform and Python version of a particular download resource. I'm getting a permission denied when trying to do a checkout using this URL (taken from the wiki page): svn+ssh://svn.python.org/data/repos/packages/trunk/pypi Has the URL changed or do I have to use some other setup on the client side than for Python SVN access ? Thanks, -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Nov 19 2009) Python/Zope Consulting and Support ...http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ ::: Try our new mxODBC.Connect Python Database Interface for free ! eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig
Re: [Catalog-sig] PyPI Development
Martin v. Löwis wrote: I'm getting a permission denied when trying to do a checkout using this URL (taken from the wiki page): svn+ssh://svn.python.org/data/repos/packages/trunk/pypi Has the URL changed or do I have to use some other setup on the client side than for Python SVN access ? Just go with the read-only checkout for the moment. This specific URL only works for Richard Jones; I personally use the https checkout myself to commit, as well. In any case, it's separate from the python-dev setup. Ok, thanks, got that working... there's just a minor nit: orig/SVN-PyPI make checkout Error validating server certificate for 'https://svn.python.org:443': - The certificate is not issued by a trusted authority. Use the fingerprint to validate the certificate manually! Certificate information: - Hostname: svn.python.org - Valid: from Apr 10 00:00:00 2007 GMT until Apr 9 23:59:59 2010 GMT - Issuer: http://www.usertrust.com, The USERTRUST Network, Salt Lake City, UT, US - Fingerprint: fc:67:c1:59:f5:57:71:29:f5:13:50:bc:c9:16:f1:74:da:1f:12:98 (R)eject or accept (t)emporarily? t Too bad, Subversion doesn't offer the option to permanently accept that certificate. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Nov 20 2009) Python/Zope Consulting and Support ...http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ ::: Try our new mxODBC.Connect Python Database Interface for free ! eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig
Re: [Catalog-sig] PyPI Development
Chris Withers wrote: M.-A. Lemburg wrote: Ok, thanks, got that working... there's just a minor nit: orig/SVN-PyPI make checkout Error validating server certificate for 'https://svn.python.org:443': - The certificate is not issued by a trusted authority. Use the fingerprint to validate the certificate manually! Certificate information: - Hostname: svn.python.org - Valid: from Apr 10 00:00:00 2007 GMT until Apr 9 23:59:59 2010 GMT - Issuer: http://www.usertrust.com, The USERTRUST Network, Salt Lake City, UT, US - Fingerprint: fc:67:c1:59:f5:57:71:29:f5:13:50:bc:c9:16:f1:74:da:1f:12:98 (R)eject or accept (t)emporarily? t It usually does, but I do know you can also add to the trusted authorities, although I can't remember how to do that off hand :-S I'm going to try something along these lines: http://subversion.tigris.org/ds/viewMessage.do?dsForumId=1065dsMessageId=2405077 The message comes up with every 'svn update' which is a bit annoying. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Nov 20 2009) Python/Zope Consulting and Support ...http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ ::: Try our new mxODBC.Connect Python Database Interface for free ! eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig
Re: [Catalog-sig] PyPI Development
M.-A. Lemburg wrote: Chris Withers wrote: M.-A. Lemburg wrote: Ok, thanks, got that working... there's just a minor nit: orig/SVN-PyPI make checkout Error validating server certificate for 'https://svn.python.org:443': - The certificate is not issued by a trusted authority. Use the fingerprint to validate the certificate manually! Certificate information: - Hostname: svn.python.org - Valid: from Apr 10 00:00:00 2007 GMT until Apr 9 23:59:59 2010 GMT - Issuer: http://www.usertrust.com, The USERTRUST Network, Salt Lake City, UT, US - Fingerprint: fc:67:c1:59:f5:57:71:29:f5:13:50:bc:c9:16:f1:74:da:1f:12:98 (R)eject or accept (t)emporarily? t It usually does, but I do know you can also add to the trusted authorities, although I can't remember how to do that off hand :-S I'm going to try something along these lines: http://subversion.tigris.org/ds/viewMessage.do?dsForumId=1065dsMessageId=2405077 The message comes up with every 'svn update' which is a bit annoying. This worked for me: ~/.subversion/servers: [groups] svn.python.org = svn.python.org [svn.python.org] ssl-authority-files = /home/lemburg/orig/certs/svn.python.org.pem Use this command to get the PEM file: openssl s_client -showcerts -host svn.python.org -port 443 The PEM file should look something like this: -BEGIN CERTIFICATE- MIIEdDCCA1ygAwIBAgIQRL4Mi1AAJLQR0zYq/mUK/TANBgkqhkiG9w0BAQUFADCB lzELMAkGA1UEBhMCVVMxCzAJBgNVBAgTAlVUMRcwFQYDVQQHEw5TYWx0IExha2Ug Q2l0eTEeMBwGA1UEChMVVGhlIFVTRVJUUlVTVCBOZXR3b3JrMSEwHwYDVQQLExho dHRwOi8vd3d3LnVzZXJ0cnVzdC5jb20xHzAdBgNVBAMTFlVUTi1VU0VSRmlyc3Qt SGFyZHdhcmUwHhcNOTkwNzA5MTgxMDQyWhcNMTkwNzA5MTgxOTIyWjCBlzELMAkG A1UEBhMCVVMxCzAJBgNVBAgTAlVUMRcwFQYDVQQHEw5TYWx0IExha2UgQ2l0eTEe MBwGA1UEChMVVGhlIFVTRVJUUlVTVCBOZXR3b3JrMSEwHwYDVQQLExhodHRwOi8v d3d3LnVzZXJ0cnVzdC5jb20xHzAdBgNVBAMTFlVUTi1VU0VSRmlyc3QtSGFyZHdh cmUwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCx98M4P7Sof885glFn 0G2f0v9Y8+efK+wNiVSZuTiZFvfgIXlIwrthdBKWHTxqctU8EGc6Oe0rE81m65UJ M6Rsl7HoxuzBdXmcRl6Nq9Bq/bkqVRcQVLMZ8Jr28bFdtqdt++BxF2uiiPsA3/4a MXcMmgF6sTLjKwEHOG7DpV4jvEWbe1DByTCP2+UretNb+zNAHqDVmBe8i4fDidNd oI6yqqr2jmmIBsX6iSHzCJ1pLgkzmykNRg+MzEk0sGlRvfkGzWitZky8PqxhvQqI DsjfPe58BEydCl5rkdbux+0ojatNh4lz0G6k0B4WixThdkQDf2Os5M1JnMWS9Ksy oUhbAgMBAAGjgbkwgbYwCwYDVR0PBAQDAgHGMA8GA1UdEwEB/wQFMAMBAf8wHQYD VR0OBBYEFKFyXyYbKJhDlV0HN9WFlp1L0sNFMEQGA1UdHwQ9MDswOaA3oDWGM2h0 dHA6Ly9jcmwudXNlcnRydXN0LmNvbS9VVE4tVVNFUkZpcnN0LUhhcmR3YXJlLmNy bDAxBgNVHSUEKjAoBggrBgEFBQcDAQYIKwYBBQUHAwUGCCsGAQUFBwMGBggrBgEF BQcDBzANBgkqhkiG9w0BAQUFAAOCAQEARxkP3nTGmZev/K0oXnWO6y1n7k57K9cM //bey1WiCuFMVGWTYGufEpytXoMs61quwOQt9ABjHbjAbPLPSbtNk28Gpgoiskli CE7/yMgUsogWXecB5BKV5UU0s4tpvc+0hY91UZ59Ojg6FEgSxvunOxqNDYJAB+gE CJChicsZUN/KHAG8HQQZexB2lzvukJDKxA4fFm517zP4029bHpbj4HR3dHuKom4t 3XbWOTCC8KucUvIqx69JXn7HaOWCgchqJ/kniCrVWFCVH/A7HFe7fRQ5YiuayZSS KqMiDP+JJn1fIytH1xUdqWqeUQ0qUZ6B+dQ7XnASfxAynB67nfhmqA== -END CERTIFICATE- -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Nov 20 2009) Python/Zope Consulting and Support ...http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ ::: Try our new mxODBC.Connect Python Database Interface for free ! eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig
Re: [Catalog-sig] Extending the package meta-data with more detailed download information
Martin v. Löwis wrote: Yes, you could have distutils generate most of those fields, except for the URL and comment. What's the use case for these new data? Is this for files also hosted at PyPI, or for files hosted elsewhere? Who would use that information, and what for? As mentioned in the original email, the use case is to have PyPI list download information for all distribution files of a package together with a set of details that allow automatic selection of a file for installation by a package manager tool. PyPI currently only provides limited details for each uploaded file and doesn't support referencing files hosted on other servers at all. I'd like to extend PyPI to also provide support for listing files on other servers as well as making it possible for the package manager tools to automatically select the right installation file for the intended target platform. At eGenix we often find that users download and install the wrong distribution files for the target Python installation - even though we do try to make this selection process as simple as possible on the download pages. To enhance the user experience we were thinking about providing a tool to automate the whole process. However, if PIP can provide this functionality based on the extended distribution file information, we'd rather point our users to PIP. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Nov 23 2009) Python/Zope Consulting and Support ...http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ ::: Try our new mxODBC.Connect Python Database Interface for free ! eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig
Re: [Catalog-sig] Poll results
Martin v. Löwis wrote: The poll is now closed, with these results Allow ratings and comments on all packages (status quo) 237 Allow package owners to disallow comments (ratings unmodified). 139 Allow comments, but only send them to package owners (ratings unmodified). 33 Disallow comments (ratings unmodified). 24 Disallow ratings and comments (status three months ago). 88 My interpretation is that the result is inconclusive. A clear majority does want a comment facility (in some form); another (not so clear) majority wants to see something changed. So it might be best to implement what appears as middle-ground, i.e. allow package authors to opt out of comments. Opinions? Same as before the poll: 1. Add comment opt-in/opt-out for authors 2. Add rating opt-in/opt-out for authors 3. Disallow anonymous ratings, i.e. show the rating given by a user and some more as a result of all the discussion around the poll: 4. Allow comment without rating 5. Allow rating without comment With those features, package authors can implement most of the poll's options, except for the only send comments to authors which I find a bit strange, since the package pages already list an email address for this purpose. That said, I still believe that this whole comment and rating system on PyPI is not very helpful. The needed developer and admin time to maintain such a system is better invested elsewhere. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Dec 02 2009) Python/Zope Consulting and Support ...http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ ::: Try our new mxODBC.Connect Python Database Interface for free ! eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig
Re: [Catalog-sig] New fields in the Metadata for PyPI
Tarek Ziadé wrote: Hello As suggested here, then discussed at Distutils-SIG, I would like to propose the addition of two more fields for the upcoming Metadata 1.2 (PEP 345) that could be used at PyPI on projects pages. Repository-Browse-URL A string containing the URL for the package's browsable repository. It can be any url of the repository (trunk, branch, tags) as long as it can be browsed from a web browser. Example: http://svn.python.org/view/python/trunk Bug-Tracker-URL A string containing the URL for the package's bug tracker. Example: http://bugs.python.org/ Other fields were proposed, but were controversial. You can follow the latest discussions at distutils-sig about them, If these fields are added, PyPI could display those links and offer a standard way for end users to interact with the projects maintainers, besides the comment/rating feature. While more structured meta-data is generally better than less, I wonder why we have to add URLs for all these things. The home page of a project will usually provide the URLs in some form already and if there is no home page, the long description can be used. A valid argument for the duplication would be to provide the user with faster and more standardized access to those resources. OTOH, they don't really mean anything for computerized consumption. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Dec 02 2009) Python/Zope Consulting and Support ...http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ ::: Try our new mxODBC.Connect Python Database Interface for free ! eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig
Re: [Catalog-sig] Extending the package meta-data with more detailed download information
Martin v. Löwis wrote: I'd like to extend PyPI to also provide support for listing files on other servers as well as making it possible for the package manager tools to automatically select the right installation file for the intended target platform. In such a scenario, how do people get the files onto the server? Will they typically know the URLs of the files upfront? Yes. ISTM that it would be better to be able to add new such URLs one by one, e.g. through a web form. I don't think that's a feasible approach, e.g. eGenix typically creates around 50 distribution files for every single release of a product. In most cases, only the version number changes, so this can easily be done using sed or in a text editor. I agree, though, that an RPC interface would also be possible to automate such a registration process, e.g. via a new register_distribution command. To enhance the user experience we were thinking about providing a tool to automate the whole process. However, if PIP can provide this functionality based on the extended distribution file information, we'd rather point our users to PIP. If the use case is automated download, I think the list of properties should be restricted to the set of properties relevant for that use case. The properties I mentioned are all necessary to get reliable automatic downloads working. The other properties (e.g. comments) are already available in the database. In general, I don't see a problem with having a few more properties of a distribution file. They don't all have to be displayed in the PyPI GUI (which is typically used by users rather than automated scripts), but can well serve package managers which need more detail to make reliable decisions. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Dec 03 2009) Python/Zope Consulting and Support ...http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ ::: Try our new mxODBC.Connect Python Database Interface for free ! eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig
Re: [Catalog-sig] Extending the package meta-data with more detailed download information
Martin v. Löwis wrote: ISTM that it would be better to be able to add new such URLs one by one, e.g. through a web form. I don't think that's a feasible approach, e.g. eGenix typically creates around 50 distribution files for every single release of a product. In most cases, only the version number changes, so this can easily be done using sed or in a text editor. I agree, though, that an RPC interface would also be possible to automate such a registration process, e.g. via a new register_distribution command. Notice that there really isn't much difference between a form and an RPC interface. The register and upload commands exclusively use forms. Technically, that's true. However, a web form is normally meant for web users to work with and in this case, you can hardly expect a user to enter all those details by hand. It would also be error-prone. That's why I used the term RPC. I was actually thinking of using the XML-RPC interface to PyPI, but the REST interface would do just as well. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Dec 03 2009) Python/Zope Consulting and Support ...http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ ::: Try our new mxODBC.Connect Python Database Interface for free ! eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig
Re: [Catalog-sig] Troubled by changes to PyPI usage agreement
Ben Finney wrote: Howdy all, The new wording is one that I can't agree to: = […] + liContent is restricted to Python packages and related information only./li + liAny content uploaded to PyPI is provided on a non-confidential basis./li + liThe PSF is free to use or disseminate any content that I upload on an + unrestricted basis for any purpose. In particular, the PSF and all other + users of the web site are granted an irrevocable, worldwide, royalty-free, + nonexclusive license to reproduce, distribute, transmit, display, perform, + and publish the content, including in digital form./li + liI represent and warrant that I have complied with all government + regulations […] = The content that I submit to PyPI is licensed under specific license terms. That certainly does *not* allow the PSF to “use or disseminate any content that I upload on an unrestricted basis for any purpose”, etc.; it allows only those acts permitted by the license terms granted in the work. AFAIK, the part 3 you are referring to is meant to allow the PSF to publish the content you upload on PyPI to users of PyPI. The same text can be found on the general legal page: http://www.python.org/about/legal/ so, in theory, you don't even have to reconfirm the PyPI terms, since you're automatically bound by them via the general legal page terms. The wording of the clause is indeed a bit broad, in particular, granting a redistribution right not only to the PSF, but also to any web site user. Note however, that you don't grant a right to sub-license. Since you typically upload package archives to PyPI, content only refers to those archives, not what's in the archives. The files in the archive are still covered by their respective licenses and there's nothing the PSF or the web site users can do to change those licenses, since they did not get the right to sub-license. OTOH, you lose control over the dissemination of your package archives, since the terms are irrevocable and that could result in some legal problems in case you get e.g. a DMCA-like take-down notice for one of your packages. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Dec 04 2009) Python/Zope Consulting and Support ...http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ ::: Try our new mxODBC.Connect Python Database Interface for free ! eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig
Re: [Catalog-sig] [PSF-Board] Troubled by changes to PyPI usage agreement
VanL wrote: Doug Hellmann wrote: We have to grant the PSF the rights to distribute the files if we're uploading them to be hosted on PyPI. Does the new wording imply that we're licensing the use of that code under those terms, or just granting distribution rights the file containing the code? It feels like the latter, but I don't know how the word perform is interpreted in this context. Doug is essentially right here. By this agreement, the PSF gets the particular rights to: - reproduce: We can copy it (in memory, or in preparation to send a copy) - distribute: We can cause other people to receive it. - transmit: We can send it out on a signal. - display: We can show the content to other people. - perform: A term of art for showing some sorts of works (think audio-visual or theater works) - publish: We can offer and provide copies to other people. ... including in digital form: And do all that stuff using computer representations of whatever you upload. You will find that all of these are very closely related, if not synonyms. Basically, we can receive your work, copy it, and provide it to other people in a variety of ways. This does not give the PSF the right to relicense your work, nor to create derivative works -- just to pass it on to anybody who happens to wander by the PyPI web page. Right, but why does the clause also give this permission to all other users of the web site and why does the license need to be irrevocable ? -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Dec 04 2009) Python/Zope Consulting and Support ...http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ ::: Try our new mxODBC.Connect Python Database Interface for free ! eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig
Re: [Catalog-sig] [PSF-Board] Troubled by changes to PyPI usage agreement
Noah Kantrowitz wrote: VanL wrote: Doug Hellmann wrote: We have to grant the PSF the rights to distribute the files if we're uploading them to be hosted on PyPI. Does the new wording imply that we're licensing the use of that code under those terms, or just granting distribution rights the file containing the code? It feels like the latter, but I don't know how the word perform is interpreted in this context. Doug is essentially right here. By this agreement, the PSF gets the particular rights to: - reproduce: We can copy it (in memory, or in preparation to send a copy) - distribute: We can cause other people to receive it. - transmit: We can send it out on a signal. - display: We can show the content to other people. - perform: A term of art for showing some sorts of works (think audio-visual or theater works) - publish: We can offer and provide copies to other people. ... including in digital form: And do all that stuff using computer representations of whatever you upload. You will find that all of these are very closely related, if not synonyms. Basically, we can receive your work, copy it, and provide it to other people in a variety of ways. This does not give the PSF the right to relicense your work, nor to create derivative works -- just to pass it on to anybody who happens to wander by the PyPI web page. Right, but why does the clause also give this permission to all other users of the web site and why does the license need to be irrevocable ? PyPI has user-run mirrors, some internal and some public. Those are likely only a handful of users who'd need the added permissions and it doesn't explain the need for an irrevocable license. If you replace all other users of the web site with users granted permission by the PSF to use the PyPI data, the mirror requirement would be dealt with in a way that doesn't require giving redistribution rights to the general public. The irrevocable appears to be unnecessary, since developers can already revoke the permission by simply deleting the uploaded files. Note that the two paragraphs were added after I asked the board on their views of having crypto code on PyPI. The conclusion was that pypi.python.org would only be seen as platform for distribution, without the PSF actually redistributing the uploaded code and the uploader would be the one to determine whether it's ok to upload the code or not. That's a convenient understanding for the PSF, since it doesn't have to control the uploaded code. However, the current wording makes it look a lot like the PSF is in fact regarding itself as a redistributor of the PyPI hosted code, so the PSF would have to follow export regulations of the Netherlands (where the servers are hosted) w/r to redistribution and reexport of crypto code. This again, is not really convenient for the PSF, since export rules are complicated. IMHO, it would be better to clearly state that PyPI is only providing a hosting service for the uploaded files, with the uploading user controlling the content and only imposing some limits of what can be uploaded rather than creating a licensing relationship between the uploader and the PSF, ie. the PSF provides the web space, the user the content - thereby avoiding all these issues. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Dec 07 2009) Python/Zope Consulting and Support ...http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ ::: Try our new mxODBC.Connect Python Database Interface for free ! eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig
Re: [Catalog-sig] [PSF-Board] Troubled by changes to PyPI usage agreement
VanL wrote: M.-A. Lemburg wrote: Those are likely only a handful of users who'd need the added permissions and it doesn't explain the need for an irrevocable license. The irrevocability is there to protect the PSF. It is so that no one can claim later that they got mad at the PSF and revoked the PSF's ability to redistribute something that they previously uploaded. If you replace all other users of the web site with users granted permission by the PSF to use the PyPI data, the mirror requirement would be dealt with in a way that doesn't require giving redistribution rights to the general public. This also makes it easier for people to pass along PyPI packages to their friends. As I have explained before, this doesn't give anybody the right to relicense the content. What is provided to the PSF (and those who get the package from the PSF) is the right to pass on to others exactly what was received. The irrevocable appears to be unnecessary, since developers can already revoke the permission by simply deleting the uploaded files. You are thinking like an engineer, not like a lawyer. It doesn't have to make sense, it just is. I guess that's the main problem here: developers do tend to think like engineers, not like lawyers, yet they still have to read and accept the new terms. Perhaps a FAQ entry is needed to get the trouble resolved, or a slightly altered text that doesn't cause the raising eyebrows to begin with. FWIW, Google uses similar wording in their TOC, but they also include this passage, which makes developers feel a lot better: http://www.google.com/accounts/TOS?loc=US: 11. Content licence from you ... This licence is for the sole purpose of enabling Google to display, distribute and promote the Services. ... ie. Google is *not* free to use or disseminate such content on an unrestricted basis for any purpose, but only for the sole purpose of providing the service in question. That's a fair deal. They also don't extend the license to all users of the web-site, since that's not necessary: the content will come with a license that defines how to use, copy and distribute it. And these licenses, as example, may include a restriction on reexporting code to an embargo country which the PyPI license doesn't, but is needed in order to protect the package author from knowingly permitting reexport of such code. What I'm trying to say is: Less is more :-) The PSF should just try get the absolute minimum of what's needed to provide the service and protect itself against issues with e.g. export regulations, take-down notices, etc. Nothing more. Note that the two paragraphs were added after I asked the board on their views of having crypto code on PyPI. The conclusion was that pypi.python.org would only be seen as platform for distribution, without the PSF actually redistributing the uploaded code and the uploader would be the one to determine whether it's ok to upload the code or not. That's a convenient understanding for the PSF, since it doesn't have to control the uploaded code. Not quite right. From the point of view of the United States, export takes place when US-sourced code is uploaded to the server in the Netherlands. This is done by the person uploading, so that is the person that we require to have previously complied with any export restrictions. You are incorrect about your assertion that the PSF does not redistribute the code. It does. However, the current wording makes it look a lot like the PSF is in fact regarding itself as a redistributor of the PyPI hosted code, so the PSF would have to follow export regulations of the Netherlands (where the servers are hosted) w/r to redistribution and reexport of crypto code. This again, is not really convenient for the PSF, since export rules are complicated. See above. I have rendered no opinion on Netherlands export laws, as I am not qualified to do so. The question asked of me was with regard to possible PSF complications relative to PyPI and crypto code. As the PSF is a United States corporation, the advice was rendered relative to US law. Understood. Thanks for clarifying this. Doesn't the PSF have to follow the Netherlands' law for things that it publishes on the python.org servers ? IMHO, it would be better to clearly state that PyPI is only providing a hosting service for the uploaded files, with the uploading user controlling the content and only imposing some limits of what can be uploaded rather than creating a licensing relationship between the uploader and the PSF, ie. the PSF provides the web space, the user the content - thereby avoiding all these issues. This is incorrect on several counts. The PSF is not a licensor under the PyPI text, and therefore the text does not create a licensing relationship between the PSF and anyone else. Besides, your proposed solution would not solve the problem. Perhaps there's a misunderstanding here: the text
Re: [Catalog-sig] [PSF-Board] Troubled by changes to PyPI usage agreement
Steve Holden, Chairman, PSF wrote: Adding a Google-like clause might make us seem less Draconian. Here's a proposal for a less controversial text based on the Google terms: PyPI is a service provided by the PSF. In order to be able to distribute the content you upload to PyPI to web site users, the PSF asks you to agree to and affirmatively acknowledge the following: 1. Content is restricted to Python packages and related information only. 2. Any content uploaded to PyPI is provided on a non-confidential basis. 3. The PSF is granted an irrevocable, worldwide, royalty-free, nonexclusive license to reproduce, distribute, transmit, display, perform, and publish the content, including in digital form. This licence is for the sole purpose of enabling the PSF to display, distribute and promote the content on PyPI. 4. I represent and warrant that I have complied with all government regulations concerning the transfer or export of any content I upload to the PyPI servers in The Netherlands. In particular, if I am subject to United States law, I represent and warrant that I have obtained the proper governmental authorization for the export of the content I upload. I further affirm that any content I provide is not intended for use by a government end-user as defined in part 772 of the United States Export Administration Regulations. The general terms on the python.org legal page would have to be changed in the same way. Thanks, -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Dec 09 2009) Python/Zope Consulting and Support ...http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ ::: Try our new mxODBC.Connect Python Database Interface for free ! eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig
Re: [Catalog-sig] [PSF-Board] Troubled by changes to PyPI usage agreement
Terry Reedy wrote: M.-A. Lemburg wrote: Steve Holden, Chairman, PSF wrote: Adding a Google-like clause might make us seem less Draconian. Here's a proposal for a less controversial text based on the Google terms: I like the third part better. Thanks. PyPI is a service provided by the PSF. In order to be able to distribute the content you upload to PyPI to web site users, the PSF asks you to agree to and affirmatively acknowledge the following: 1. Content is restricted to Python packages and related information only. 2. Any content uploaded to PyPI is provided on a non-confidential basis. 3. The PSF is granted an irrevocable, worldwide, royalty-free, nonexclusive license to reproduce, distribute, transmit, display, perform, and publish the content, including in digital form. This licence is for the sole purpose of enabling the PSF to display, distribute and promote the content on PyPI. 4. I represent and warrant that I have complied with all government regulations concerning the transfer or export of any content I upload to the PyPI servers in The Netherlands. In particular, if I am subject to United States law, I represent and warrant that I have obtained the proper governmental authorization for the export of the content I upload. I further affirm that any content I provide is not intended for use by a government end-user as defined in part 772 of the United States Export Administration Regulations. The fourth section might scare people off without further explanation somewhere, as it could be taken to imply that people have to get a US gov permit to upload, which almost no one has done. If this is only about crypto software, it should say so. I do not understand the last sentence at all as open-source licenses do not usually exclude specific users. I cannot affirm something that is complete gobble talk to me. The clause has three parts: a) I represent and warrant that I have complied with all government regulations concerning the transfer or export of any content I upload to the PyPI servers in The Netherlands. This part is written in a general way and is needed to cover export regulations which may be imposed by the country of the uploader when uploading (exporting) applications to a server in the The Netherlands. For many countries these export regulations are variants of the things laid out in the Wassenaar Arrangement which covers crypto code, but also other software technologies that may be considered dual-use: http://www.wassenaar.org/ in particular: http://www.wassenaar.org/controllists/2009/WA-LIST%20%2809%29%201/WA-LIST%20%2809%29%201.pdf Most software will fall under the GENERAL SOFTWARE NOTE (with some special rules for crypto software), but countries may still implement additional rules such as the ones currently imposed by the US (you have to send them an email with the link to the download location - see http://www.bis.doc.gov/encryption/pubavailencsourcecodenofify.html). Since the exact regulations depend on the country from where the code is uploaded, the clause can't be more specific. I added the location of the servers to the original clause to make the export nature of the upload more specific. b) In particular, if I am subject to United States law, I represent and warrant that I have obtained the proper governmental authorization for the export of the content I upload. This part only applies to US uploaders. Note that the US regulations have a subtle detail: they apply to all US-origin content. E.g. if you export some dual-use system software written in the US from Germany to Cuba, the US can put you on their embargo list. c) I further affirm that any content I provide is not intended for use by a government end-user as defined in part 772 of the United States Export Administration Regulations. This part applies to all uploaders. The restriction appears to be a super-set of the embargo restrictions for various individuals - most of those are government end-users. I find that clause too board as well, since it prevents government users in general to use PyPI packages. Furthermore, the embargo lists also includes companies and, of course, whole countries, which this clause does not cover. See e.g. EU: http://ec.europa.eu/external_relations/cfsp/sanctions/docs/measures_en.pdf US: http://www.bis.doc.gov/news/2009/2009-fpr.pdf (note how e.g. Cuba is on the US list, but not on the EU list) I'm not sure why the clause is needed. Perhaps Van could clarify this. IMHO, part a) already covers everything that is needed w/r to export restrictions. All this with the usual IANAL disclaimer. I've read a lot on these things when we started shipping a pyOpenSSL distribution. Some of the things I found are listed above. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Dec 11 2009) Python/Zope Consulting and Support ...http://www.egenix.com/ mxODBC.Zope.Database.Adapter
Re: [Catalog-sig] [Distutils] packaging terminology confusion
Tarek Ziadé wrote: On Thu, Jan 7, 2010 at 10:29 PM, Brad Allen bradallen...@gmail.com wrote: On Thu, Jan 7, 2010 at 2:40 PM, P.J. Eby p...@telecommunity.com wrote: As for projects: fine with me; PyPI would then be the Python Project Index. +1 If this gets general agreement, there are probably some places where the word 'package' should be replaced with the word 'project', right? For example, should PEP-345 be retitled as Metadata for Python Software Projects 1.2 ? +1, we should reflect this terminology in all new PEPs and in Distutils doc as well I don't think we need to change anything - most Python software components come as Python packages nowadays, so the terminology 'package' we've used all these years is correct. OTOH, sets of components like Zope are indeed projects. However, the number of PyPI packages which could reasonably be called a project is relatively small and even those use Python packages to separate their namespaces. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jan 07 2010) Python/Zope Consulting and Support ...http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ ::: Try our new mxODBC.Connect Python Database Interface for free ! eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig
Re: [Catalog-sig] [Distutils] packaging terminology confusion
Brad Allen wrote: On Thu, Jan 7, 2010 at 3:51 PM, M.-A. Lemburg m...@egenix.com wrote: I don't think we need to change anything - most Python software components come as Python packages nowadays, so the terminology 'package' we've used all these years is correct. Do you mean only 'package' in the sense of an __init__.py container, or in the sense of a setup.py container? Both usages have been in use for years, but only the __init__.py package was formally recognized. We have used the term 'package' for both a Python software component as well as the Python module directories on sys.path with a __init__.py marker. I usually refer to the latter as 'Python package' and the former as 'package'. What you download is a 'distribution file' which could be an exe, a tarball, an egg, etc. There are many forms such a package distribution can take and just as many ways to install them. Once installed a 'package' usually creates a single 'Python package' (at top-level or some lower level). As a result, after installation there is little difference between a Python software component called 'package' and the module directory called 'Python package'. qed :-) -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jan 07 2010) Python/Zope Consulting and Support ...http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ ::: Try our new mxODBC.Connect Python Database Interface for free ! eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig
Re: [Catalog-sig] [PSF-Board] Troubled by changes to PyPI usage agreement
Hi Steve, has there been any progress on this ? M.-A. Lemburg wrote: Steve Holden, Chairman, PSF wrote: Adding a Google-like clause might make us seem less Draconian. Here's a proposal for a less controversial text based on the Google terms: PyPI is a service provided by the PSF. In order to be able to distribute the content you upload to PyPI to web site users, the PSF asks you to agree to and affirmatively acknowledge the following: 1. Content is restricted to Python packages and related information only. 2. Any content uploaded to PyPI is provided on a non-confidential basis. 3. The PSF is granted an irrevocable, worldwide, royalty-free, nonexclusive license to reproduce, distribute, transmit, display, perform, and publish the content, including in digital form. This licence is for the sole purpose of enabling the PSF to display, distribute and promote the content on PyPI. 4. I represent and warrant that I have complied with all government regulations concerning the transfer or export of any content I upload to the PyPI servers in The Netherlands. In particular, if I am subject to United States law, I represent and warrant that I have obtained the proper governmental authorization for the export of the content I upload. I further affirm that any content I provide is not intended for use by a government end-user as defined in part 772 of the United States Export Administration Regulations. The general terms on the python.org legal page would have to be changed in the same way. I've attached a message explaining some of the reasons for part 4. Thanks, -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jan 18 2010) Python/Zope Consulting and Support ...http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ ::: Try our new mxODBC.Connect Python Database Interface for free ! eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ ---BeginMessage--- Terry Reedy wrote: M.-A. Lemburg wrote: Steve Holden, Chairman, PSF wrote: Adding a Google-like clause might make us seem less Draconian. Here's a proposal for a less controversial text based on the Google terms: I like the third part better. Thanks. PyPI is a service provided by the PSF. In order to be able to distribute the content you upload to PyPI to web site users, the PSF asks you to agree to and affirmatively acknowledge the following: 1. Content is restricted to Python packages and related information only. 2. Any content uploaded to PyPI is provided on a non-confidential basis. 3. The PSF is granted an irrevocable, worldwide, royalty-free, nonexclusive license to reproduce, distribute, transmit, display, perform, and publish the content, including in digital form. This licence is for the sole purpose of enabling the PSF to display, distribute and promote the content on PyPI. 4. I represent and warrant that I have complied with all government regulations concerning the transfer or export of any content I upload to the PyPI servers in The Netherlands. In particular, if I am subject to United States law, I represent and warrant that I have obtained the proper governmental authorization for the export of the content I upload. I further affirm that any content I provide is not intended for use by a government end-user as defined in part 772 of the United States Export Administration Regulations. The fourth section might scare people off without further explanation somewhere, as it could be taken to imply that people have to get a US gov permit to upload, which almost no one has done. If this is only about crypto software, it should say so. I do not understand the last sentence at all as open-source licenses do not usually exclude specific users. I cannot affirm something that is complete gobble talk to me. The clause has three parts: a) I represent and warrant that I have complied with all government regulations concerning the transfer or export of any content I upload to the PyPI servers in The Netherlands. This part is written in a general way and is needed to cover export regulations which may be imposed by the country of the uploader when uploading (exporting) applications to a server in the The Netherlands. For many countries these export regulations are variants of the things laid out in the Wassenaar Arrangement which covers crypto code, but also other software technologies that may be considered dual-use: http://www.wassenaar.org/ in particular: http://www.wassenaar.org/controllists/2009/WA-LIST%20%2809
Re: [Catalog-sig] [PSF-Board] Troubled by changes to PyPI usage agreement
Steve Holden wrote: None that I am aware of, but Martin is the one who's been making changes most recently. I don't think there's been any input from Van on this yet, but I've been busy and may have forgotten or missed something. Thanks. As far as I can tell, the text on the PyPI registration page hasn't changed since we last discussed the problem. Since there is currently discussion going on about setting up PyPI mirrors that are not maintained by the PSF, I think that we need to do something about the licensing terms soonish, esp. since most package authors who have uploaded things to PyPI will not be aware of any changes to the terms and conditions of using PyPI. Any such change will have to be made more visible on the site and the Python announcement list than is currently done (you only see the text if you click on register on PyPI - which, of course, already registered users will unlikely do on a regular basis :-). Regards, -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jan 20 2010) Python/Zope Consulting and Support ...http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ ::: Try our new mxODBC.Connect Python Database Interface for free ! eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ regards Steve M.-A. Lemburg wrote: Hi Steve, has there been any progress on this ? M.-A. Lemburg wrote: Steve Holden, Chairman, PSF wrote: Adding a Google-like clause might make us seem less Draconian. Here's a proposal for a less controversial text based on the Google terms: PyPI is a service provided by the PSF. In order to be able to distribute the content you upload to PyPI to web site users, the PSF asks you to agree to and affirmatively acknowledge the following: 1. Content is restricted to Python packages and related information only. 2. Any content uploaded to PyPI is provided on a non-confidential basis. 3. The PSF is granted an irrevocable, worldwide, royalty-free, nonexclusive license to reproduce, distribute, transmit, display, perform, and publish the content, including in digital form. This licence is for the sole purpose of enabling the PSF to display, distribute and promote the content on PyPI. 4. I represent and warrant that I have complied with all government regulations concerning the transfer or export of any content I upload to the PyPI servers in The Netherlands. In particular, if I am subject to United States law, I represent and warrant that I have obtained the proper governmental authorization for the export of the content I upload. I further affirm that any content I provide is not intended for use by a government end-user as defined in part 772 of the United States Export Administration Regulations. The general terms on the python.org legal page would have to be changed in the same way. I've attached a message explaining some of the reasons for part 4. Thanks, Subject: Re: [Catalog-sig] [PSF-Board] Troubled by changes to PyPI usage agreement From: M.-A. Lemburg m...@egenix.com Date: Fri, 11 Dec 2009 01:45:10 +0100 To: Terry Reedy tjre...@udel.edu To: Terry Reedy tjre...@udel.edu CC: catalog-sig@python.org, VanL v...@python.org Terry Reedy wrote: M.-A. Lemburg wrote: Steve Holden, Chairman, PSF wrote: Adding a Google-like clause might make us seem less Draconian. Here's a proposal for a less controversial text based on the Google terms: I like the third part better. Thanks. PyPI is a service provided by the PSF. In order to be able to distribute the content you upload to PyPI to web site users, the PSF asks you to agree to and affirmatively acknowledge the following: 1. Content is restricted to Python packages and related information only. 2. Any content uploaded to PyPI is provided on a non-confidential basis. 3. The PSF is granted an irrevocable, worldwide, royalty-free, nonexclusive license to reproduce, distribute, transmit, display, perform, and publish the content, including in digital form. This licence is for the sole purpose of enabling the PSF to display, distribute and promote the content on PyPI. 4. I represent and warrant that I have complied with all government regulations concerning the transfer or export of any content I upload to the PyPI servers in The Netherlands. In particular, if I am subject to United States law, I represent and warrant that I have obtained the proper governmental authorization for the export of the content I
Re: [Catalog-sig] [PSF-Board] Troubled by changes to PyPI usage agreement
Tarek Ziadé wrote: On Wed, Jan 20, 2010 at 8:06 PM, M.-A. Lemburg m...@egenix.com wrote: Steve Holden wrote: Agreed. Until this issue is resolved we can't allow (public) third-party mirrors. Given the recent adverse reactions to PyPi changes we should be careful not to cause any further offense. Perhaps the PSF should start creating a list of official public mirrors. We would then need to add a mention of this to the PyPI usage license I posted. Setting up an official mirror would then require approval of the PSF and allow the PSF to monitor and maintain a good quality service. Package manager tools could then use the list of official mirrors to find a nearby mirror for downloading the package information. That's basically what I've proposed in PEP 381 : 1/ the mirror maintainer proposes its mirror in the Catalog mailing-list 2/ if it's accepted, it's added in the list of IPs in the pypi DNS 3/ client tools like PIP develop a geloc feature to use the nearest mirror, by getting the list of IPs http://www.python.org/dev/peps/pep-0381/#how-a-client-can-use-pypi-and-its-mirrors I could add in the PEP the fact that the mirror has to be accepted by the PSF Tarek Ziadé wrote: On Wed, Jan 20, 2010 at 10:34 PM, Tarek Ziadé ziade.ta...@gmail.com wrote: [..] http://www.python.org/dev/peps/pep-0381/#how-a-client-can-use-pypi-and-its-mirrors I could add in the PEP the fact that the mirror has to be accepted by the PSF See also : http://www.python.org/dev/peps/pep-0381/#mirror-listing-and-registering Any such addition will have to go by the PSF since the PSF runs the official PyPI index. Note that this is only required for PyPI mirrors that are made public, ie. redistribute the PyPI data without the PSF being directly involved. Background: the change recently introduced on the PyPI registration page does cover such redistributions by any user of PyPI, but that change is controversial and, what's more important, does not cover data uploaded to PyPI by users who registered before the change. I'd like to get the wording on the terms changed to make the PSF the only partner in the agreement and let it control the redistribution terms - much like we do in the Python contribution forms. Just to clarify: a private mirror would basically just be standard use of the PyPI data without the redistribution part, so that use is fine in all cases. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jan 20 2010) Python/Zope Consulting and Support ...http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ ::: Try our new mxODBC.Connect Python Database Interface for free ! eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig
Re: [Catalog-sig] [PSF-Board] Troubled by changes to PyPI usage agreement
Martin v. Löwis wrote: Steve Holden wrote: Agreed. Until this issue is resolved we can't allow (public) third-party mirrors. Given the recent adverse reactions to PyPi changes we should be careful not to cause any further offense. I quite disagree on that statement; I see the issue of mirrors as completely unrelated, and I also don't think the current issue is causing any offense. Any public mirror would trigger a redistribution of PyPI uploaded data without the PSF or the uploading PyPI user owning the material having a say about the redistribution terms. This may sound harmless at first, but if we get more VyperLogix folks setting up PyPI mirrors, I'm sure that a lot of developers will not be happy about the PSF being careless about who is allowed to redistribute the PyPI data. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jan 20 2010) Python/Zope Consulting and Support ...http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ ::: Try our new mxODBC.Connect Python Database Interface for free ! eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig
Re: [Catalog-sig] [PSF-Board] Troubled by changes to PyPI usage agreement
Martin v. Löwis wrote: Sure, the PEP can be used as basis for the decision process, but someone still has to make the decision to add a mirror or not and these people should be appointed to by the PSF - much like we have an infrastructure committee to see after the python.org site. The situation is a lot like with the Python trademarks: The good guys always come and ask for permission. The bad guys don't. With PEP 381, the bad guys won't get any attention. You can already run as many public mirrors of PyPI as you want, but nobody will notice. To become an official mirror, you have to ask for permission already; see the PEP. In order to go after them we need a clear set of rules for setting up and running a PyPI mirror and disallowing setups that don't follow these rules. See the PEP. Like I said: the PEP can be used to document the technical requirements of being accepted as official mirror, but it doesn't cover any of the legal requirements the PSF will need to put in place in order to prevent unofficial mirrors which use the content to e.g. increase their page rank, add advertisement and other drive-by revenue, misrepresent authorship, add malware to popular packages, etc. Such requirements are not within the scope of a PEP. The decision process itself is also not within the scope of the PEP. It only outlines the technical details of official mirrors. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jan 21 2010) Python/Zope Consulting and Support ...http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ ::: Try our new mxODBC.Connect Python Database Interface for free ! eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig
Re: [Catalog-sig] [PSF-Board] Troubled by changes to PyPI usage agreement
Steve Holden wrote: Tarek Ziadé wrote: 2010/1/20 Martin v. Löwis mar...@v.loewis.de: Of course, there's also a human dimension : we suppose that the people running the mirror are people we can trust because they can technically do malicious things in the mirror since we don't really have any real protection (*yet*). That's not true: users of mirrors can verify that the mirrors are authentic. Neither can malicious operators modify the contents of their mirrors without clients noticing, nor can careless mirror operators threaten the integrity of a mirror even assuming somebody breaks into the mirror. But users can't verify that the archive they download using tools like easy_install are the real ones. If I am a bad guy and I run a mirror, I can change a setup.py file in an archive and make it do malicious things on the computer, and let easy_install execute it for me. The only verification done is the md5 hash on the file, which can be changed on the mirror (nothing prevents the mirror to compute its own MD5 fragments in the download URLs) Regards Tarek I have in the past suggested that we consider hosting services at diverse places. I'd have thought this was a prima facie case for distributed hosting facilities. If we have that, we have no need for mirrors, but instead for systems management. I know of at least three reputable US hosting companies who I am pretty sure would help, and a major academic hosting organization too. Also maybe Snakebite might be another location? That would be my preference as well. Services like Amazon CloudFront or Akamai could easily scale up to millions of users. Since the PyPI data is already available as set of static files (at least from looking at the simple/ index), pushing this through such content delivery systems should easily be possible. Such a setup would avoid all the legalese issues, since the PSF would run the infrastructure and also make monitoring the service a lot easier. Moreover, all the complicated stuff like on-demand content distribution is handled by those services automatically. This is an infrastructure committee task really. Brett? Martin? Brett: maybe your closing report to the board could summarize the overall hosting situation? -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jan 21 2010) Python/Zope Consulting and Support ...http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ ::: Try our new mxODBC.Connect Python Database Interface for free ! eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig
Re: [Catalog-sig] [PSF-Board] Troubled by changes to PyPI usage agreement
Tarek Ziadé wrote: On Thu, Jan 21, 2010 at 5:29 PM, M.-A. Lemburg m...@egenix.com wrote: [..] Sure, we could do all those things, but such a process will cause a lot of admin overhead on part of the PSF. Which process ? the non-web mirroring requires no effort/work from the PSF. The only effort that is required is technical, and 70% done at this point I'd say. No, it's not only technical. The administration overhead comes into play when looking at the legal side of things and that does not only require doing initial checks of whether the mirrors are adhering to a set of technical standards, but also whether they adhere to legal ones. We'd avoid all that with e.g. a cloud setup run by the PSF and get all the monitoring, statistics, etc. for free. Using a content delivery system we'd avoid such administration work. The PSF would have to sign agreements with 10-20 mirror providers, it wouldn't have to setup a monitoring system, keep checking the mirror web content, etc. What is a content delivery system here ? do you mean by that that the PSF would run the mirror by itself ? if so, how this is going to work technically ? how would it be different ? See e.g. http://aws.amazon.com/cloudfront/ for such a system. Using Amazon would even also the PSF to run the web front end based on the same system and data. The main advantage is that they take care of all the sys admin stuff and provide a unified platform to work with. Let me state it differently : what if each mirror maintainer is a PSF member ? does that addresse the legal/admin issues ? Perhaps the legal ones (can't say, we'd have to ask Van), but certainly not the admin issues: Each mirror maintainer would have to invest time in setting up such a server, so instead of simplifying things, we'd make them more work intense... and then you have to manage software updates on those servers, fight network problems, handle differences between server platforms and OSes, implement fail-over, etc. etc. - basically all the usual operations stuff needed to maintain a distributed cluster. With a service like Amazon or Akamai you just have one platform and/or API to worry about. All the sys admin overhead is done by others and edge distribution comes right with the service. Moreover, there would also be mirrors in parts of the world that are currently not well covered by Pythonistas and thus less likely to get a local mirror server setup. This is just a matter of having a server IP in that part of the world. You'd also have to have the data in that part of the world if you want to benefit from a local mirror. That's what edge distribution is all about. And in reality, as long as the main areas US, Europe, Austrialia, etc are served, this fits our needs. But some people will probably have to go through several nodes to reach a mirror, but we can't have a server per major city. So in any case, we are improving the situation, not making it worse. That's not what I'm saying. I hope I'm making myself clear. If not please let me know. What I'm trying to say is that it is more effective to look at existing solutions to these standard problems and work from there instead of reinventing yet another distribution network mechanism from ground up. This will save you a lot of work and it will also simplify the legalese and administration from the PSF side of things. How to arrange all this is really a PSF question more than anything else. Also note that using a static file layout would make the whole synchronization mechanism a lot easier - not only for content delivery networks, but also for dedicated volunteer run mirrors. There are lots of mirror scripts out there that work with rsync or FTP, so no need to reinvent the wheel. Those scripts already exist and are in usage in the tools that are mirroring pypi. They are not rsync but http calls, but that's about it. Ok, so that wheel has already been reinvented :-) AFAICTL, all the data on PyPI is static and can be rendered as files in a directory dump. A simple cronjob could take care of this every few minutes or so and extract the data to a local directory which is then made accessible to mirrors. People are already doing rsync-like mirrors. But that's quite an incomplete mirror. Why is that ? Why can't the PyPI data be extracted to the file system to make it more accessible to standard tools ? The whole point of the work I've been doing with Martin (partially reflected in PEP 381) is to be able to have the download statistics for each archive, no matter which mirror was used to download the file. That's quite a valuable information. Indeed and Amazon will provide those to you without having to do any extra work: http://aws.amazon.com/about-aws/whats-new/2009/05/07/amazon-cloudfront-adds-access-logging-capability/ http://developer.amazonwebservices.com/connect/entry.jspa?externalID=2440 -- Marc-Andre Lemburg eGenix.com Professional Python
Re: [Catalog-sig] [PSF-Board] Troubled by changes to PyPI usage agreement
Tarek Ziadé wrote: On Thu, Jan 21, 2010 at 8:08 PM, M.-A. Lemburg m...@egenix.com wrote: [..] Those scripts already exist and are in usage in the tools that are mirroring pypi. They are not rsync but http calls, but that's about it. Ok, so that wheel has already been reinvented :-) That's historical. The HTTP interface existed already and is used by PIP and al. People built mirroring scripts on the top of it, and used the changelog file to get the updates. IOW, this works *today* with the existing software. Anyways, I guess I'll wait for the PSF decision to continue on that PEP. Last, if the PSF-maintained cloud-based system is preferred over what we've worked on, it would be great if it's still built on the top of an open protocol that is not tied to a cloud system, because people will be able to use it for their own PyPI systems. For instance, plone.org has a PyPI interface nowadays, so people can upload distributions over there using Distutils. And it could have mirrors as well. And accurate stats... :) Sure :-) Sorry for spoiling the fun a bit, but like you said: this is a PSF decision to make and has more strings attached to it than just technical ones. Cheers, -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jan 22 2010) Python/Zope Consulting and Support ...http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ ::: Try our new mxODBC.Connect Python Database Interface for free ! eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig
Re: [Catalog-sig] [PSF-Board] Troubled by changes to PyPI usage agreement
Martin v. Löwis wrote: Like I said: the PEP can be used to document the technical requirements of being accepted as official mirror, but it doesn't cover any of the legal requirements the PSF will need to put in place in order to prevent unofficial mirrors Ok - as we are discussing official only mirrors, why are you now changing the subject to unofficial mirrors? Please see my original email: the good guys will always try to play nice, the bad guys won't. In order to make it clear that PyPI data may only be mirrored for redistribution with PSF authorization, we need to add proper notices to PyPI and also prevent such mirroring technically (if possible). -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jan 22 2010) Python/Zope Consulting and Support ...http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ ::: Try our new mxODBC.Connect Python Database Interface for free ! eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig
Re: [Catalog-sig] [PSF-Board] Troubled by changes to PyPI usage agreement
Martin v. Löwis wrote: What about restricting the mirrors to the non web part in that case ? I think MAL is talking about a completely different setup: the unofficial mirror. The unofficial mirror doesn't follow any protocol; it's just a mirror of PyPI using the standard API to fetch all data available. People have been operating unofficial mirrors of various qualities for several years. Now, MAL is then concerned about malicious users of unofficial servers. They might try to get their mirror high-ranking in Google, and then redirect regular PyPI users to them. There isn't anything that the PEP can do about that. Right. There has only been one such malicious mirror in the past. The PSF legal council was fairly helpful in dealing with that case. Indeed, but only on trademark terms, not based on rules that cover permission to mirror the data in the first place. All that said, I think we can get a much simpler setup working using an existing cloud distribution system. This will solve the legal issues, the technical ones, simplify monitoring, remove the need to have sys admins looking after the mirror servers, provide statistics, etc. etc. And it's cost effective too, since the time spent on all of the above values a lot higher than the few dollars it costs to run such a setup on Akamai or Amazon. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jan 22 2010) Python/Zope Consulting and Support ...http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ ::: Try our new mxODBC.Connect Python Database Interface for free ! eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig
Re: [Catalog-sig] [PSF-Board] Troubled by changes to PyPI usage agreement
Martin v. Löwis wrote: OK. That's not what I understood since he proposed a cloud system run by the PSF for the PyPI mirrors. Which implied (to me) those were official mirrors. For some reason (which I don't understand) MAL is opposed to the notion of mirrors. If the complaints about a mirroring system are somehow fundamental, there isn't anything we can do about it, so I propose we just proceed, anyway. The way the PEP process is structured, it would then be up to the BDFL (not the PSF board) to ultimately decide on the entire subject. The PEP only covers the technical issues. I'm not arguing against anything technical here. When it comes to the implementation of the PEP on python.org, things look different though. The PSF is running the python.org site and PyPI. It is also the legal partner when signing up for PyPI usage and needs to looks after that uploaded data. As a result, the PSF has to decide what to do about mirrors. For plone.org's PyPI installation, the Plone Foundation would have to decide, for other PyPI installation the resp. entities running those installations will have to decide. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jan 22 2010) Python/Zope Consulting and Support ...http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ ::: Try our new mxODBC.Connect Python Database Interface for free ! eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig
Re: [Catalog-sig] [PSF-Board] Troubled by changes to PyPI usage agreement
Martin v. Löwis wrote: In order to make it clear that PyPI data may only be mirrored for redistribution with PSF authorization, we need to add proper notices to PyPI and also prevent such mirroring technically (if possible). However, I don't think this is factually the case: *anybody* can indeed mirror the data in any way they like. This is how it is, and how it should be. Sure, downloading things from PyPI is its main intent and that's also what the proposed PyPI terms allow the PSF to do. However, there's a huge difference between just downloading content and redistributing it. The latter is what we're discussing here. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jan 23 2010) Python/Zope Consulting and Support ...http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ ::: Try our new mxODBC.Connect Python Database Interface for free ! eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig
Re: [Catalog-sig] [PSF-Board] Troubled by changes to PyPI usage agreement
Martin v. Löwis wrote: The second version does seem much more user-friendly, somehow, and should calm fears about potential abuse of content by the Foundation. Are we going to go with that? Not without legal advise. I would hope that users will also get permission to make copies of the content uploaded to PyPI, without having to study the license conditions of each and every packet that they may want to copy (and they may actually have to make a copy first to find out what the licensing conditions are). So ISTM that MAL's version is *less* user-friendly (at least, to the users of PyPI who actually want to use the packages that get uploaded). If you read the text, you'll notice that nothing will change for PyPI users. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jan 23 2010) Python/Zope Consulting and Support ...http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ ::: Try our new mxODBC.Connect Python Database Interface for free ! eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig
Re: [Catalog-sig] [PSF-Board] Troubled by changes to PyPI usage agreement
Martin v. Löwis wrote: Terry Reedy wrote: On 1/23/2010 5:03 AM, Martin v. Löwis wrote: Indeed, that's what we are discussing here. And, I can only repeat myself: anybody can download the data and redistribute it in any way they like. This is how it is, and how it should be. Many sites have the restriction that people can download for own use but *not* redistribute, at least not publicly. Can you give examples of such sites (preferably examples where the content is not originally owned by the site distributing it)? Google Code - the example I used as basis for the updated terms - is a popular OSS site. Most commercial sites don't allow redistribution of the applications you download - at least not without explicit permission. Many freeware applications also restrict the way you may distribute the apps, e.g. they often explicitly disallow putting them on CDs in order to prevent magazines from cashing in on the drive-by-revenue. For software containing crypto code, managing the distribution channels is required by government law in most countries (see http://rechten.uvt.nl/koops/cryptolaw/cls-sum.htm). Software using GPL-like terms also put special requirements on distribution channels, in fact, a major part of the GPL mechanism is built around public distribution of software (which is one of the issues some people have with it and the reason why the AGPL was created). The current terms on PyPI have the potential of overriding all such restrictions. And people should *not* be able to 'redistribute in any way they like', for instance with claim of authorship or copyright or malware attached. Either case would involve modification. Copying and creating derivative works are fairly different, so when one is granted people wouldn't assume to have the other as well. I think we now all know that you're not in favor of changing anything reagrding the new terms on PyPI. That's fine. Please do acknowledge, though, that there are a number of people who do not feel comfortable with those new terms. Whether or not to change them, is up to the PSF board. I've posted a proposal which tries to find a balance between what the PSF has to ask from the uploading developer and what the PyPI user can reasonably expect. It's now up to the board to decide. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jan 25 2010) Python/Zope Consulting and Support ...http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ ::: Try our new mxODBC.Connect Python Database Interface for free ! eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig
Re: [Catalog-sig] The Softpedia spam
Tarek Ziadé wrote: Hello, The Softpedia website sends an email to everyone that register or uploads something at PyPI. This is clearly a spam and their website don't care about our projects. I am not sure if they use the PubSubHubbub thing, but I was wondering how we could prevent these unsolicited mails. If they use PubSubHubbub, maybe we could set up a black list of subscribers people can manage at their level, if they reconstruct the emails by reading the RSS feed, maybe we should not publish this info (even with the @ transformed into at ). Unfortunately, that's what you get when providing APIs to extract all the data from PyPI. Not even the terms on the PyPI service can be used to prevent that (something I'll try to change now that I'm on the PSF board again). We should really disallow redistribution of the PyPI meta data and uploads without prior written consent from the PSF. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, May 06 2010) Python/Zope Consulting and Support ...http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ ::: Try our new mxODBC.Connect Python Database Interface for free ! eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig
Re: [Catalog-sig] The Softpedia spam
Tarek Ziadé wrote: On Thu, May 6, 2010 at 4:50 PM, M.-A. Lemburg m...@egenix.com wrote: Tarek Ziadé wrote: Hello, The Softpedia website sends an email to everyone that register or uploads something at PyPI. This is clearly a spam and their website don't care about our projects. I am not sure if they use the PubSubHubbub thing, but I was wondering how we could prevent these unsolicited mails. If they use PubSubHubbub, maybe we could set up a black list of subscribers people can manage at their level, if they reconstruct the emails by reading the RSS feed, maybe we should not publish this info (even with the @ transformed into at ). Unfortunately, that's what you get when providing APIs to extract all the data from PyPI. Not even the terms on the PyPI service can be used to prevent that (something I'll try to change now that I'm on the PSF board again). We should really disallow redistribution of the PyPI meta data and uploads without prior written consent from the PSF. Well the problem is not about the distribution of the metadata because for OSS projects, you'll always have your email somewhere in the tarball. I am not sure what you want to do at PSF level, but I wouldn't want the PSF to restrict the usage of my own project info if I upload them at PyPI. PyPI is just *one* recipient for projects and don't own people data. Sorry, perhaps I wasn't clear: when uploading things to PyPI you accept the PyPI terms. These terms currently allow anyone to take the data from PyPI and publically redistribute it without any restrictions. I think it's better to only allow the PSF to redistribute data that it got from the PyPI package authors. Redistribution in the form that Softpedia uses to attract visitors and make revenue on the ads they have on their site is not something the PSF would normally tolerate. However, with the current terms, there's nothing the PSF can do about it. As package author, you are, of course, free to upload your packages wherever you want, the PyPI terms only apply to the data that you passed on to the PSF for display. The problem is about the usage of the APIs PyPI provides : Softpedia has set up a automatic process that gets triggered every time something is uploaded. So It's all about spam, as usual. If we can control how the APIs are used, we will defeat this bot. What I propose is: - set up authentication for the XML-RPC APIs, in order to control this. If a user starts to use XML-RPC calls in his bots, it's easy to shut it down. - set up a restricted list of subscribers for the PubSubHubbub protocol (I am not sure if this protocol supports authentication, but I guess we can set something up) - avoid displaying any email or derived emails on anonymous page I'm not sure how that would work. Package manager tools would then all have to use this authentication mechanism. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, May 06 2010) Python/Zope Consulting and Support ...http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ ::: Try our new mxODBC.Connect Python Database Interface for free ! eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig
Re: [Catalog-sig] The Softpedia spam
Tarek Ziadé wrote: On Thu, May 6, 2010 at 5:18 PM, M.-A. Lemburg m...@egenix.com wrote: [..] Sorry, perhaps I wasn't clear: when uploading things to PyPI you accept the PyPI terms. These terms currently allow anyone to take the data from PyPI and publically redistribute it without any restrictions. I think it's better to only allow the PSF to redistribute data that it got from the PyPI package authors. I am not sure what it means that the PSF redistributes data. Is this http://www.python.org/about/legal or another text ? That text needs some care as well, yes. I was referring to this text on PyPI: http://pypi.python.org/pypi?%3Aaction=register_form By registering to upload content to PyPI, I agree and affirmatively acknowledge the following: 1. Content is restricted to Python packages and related information only. 2. Any content uploaded to PyPI is provided on a non-confidential basis. 3. The PSF is free to use or disseminate any content that I upload on an unrestricted basis for any purpose. In particular, the PSF and all other users of the web site are granted an irrevocable, worldwide, royalty-free, nonexclusive license to reproduce, distribute, transmit, display, perform, and publish the content, including in digital form. 4. I represent and warrant that I have complied with all government regulations concerning the transfer or export of any content I upload to PyPI. In particular, if I am subject to United States law, I represent and warrant that I have obtained the proper governmental authorization for the export of the content I upload. I further affirm that any content I provide is not intended for use by a government end-user as defined in part 772 of the United States Export Administration Regulations. A list of prohibited usage (combined with authentication) should be enough to prevent the problem as far as I understand. For instance, here's SourceForge's one http://sourceforge.net/apps/trac/sitelegal/wiki/Terms_of_Use#a2.YOURUSEOFSOURCEFORGE.NET Extract: ...using any information obtained from SourceForge.net in order to contact, advertise to, solicit, or sell to any user without such user's prior explicit consent (including non-commercial contacts like chain letters); Right, we'd need something along those lines. [..] What I propose is: - set up authentication for the XML-RPC APIs, in order to control this. If a user starts to use XML-RPC calls in his bots, it's easy to shut it down. - set up a restricted list of subscribers for the PubSubHubbub protocol (I am not sure if this protocol supports authentication, but I guess we can set something up) - avoid displaying any email or derived emails on anonymous page I'm not sure how that would work. Package manager tools would then all have to use this authentication mechanism. Yes but they would need to use an account therefore have an identity when they run their scripts. Hmm, wouldn't that require all pip users to have PyPI account ? For instance, PyPI can have API calls quota per user, and a white list of users that are allowed to have an unlimited number of API calls. (managed manually) IOW, allow stuff like cheesecake ratings or whatever, to subscribe, and be able to block Softpedia. It's a limited protection but should be enough: I don't think the Softpedia staff will work on defeating this by registering hundreds of zombies at PyPI. But I understand that it also needs the legal part, I'll work on the legal stuff and leave the technical side to you :-) -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, May 06 2010) Python/Zope Consulting and Support ...http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ ::: Try our new mxODBC.Connect Python Database Interface for free ! eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig
Re: [Catalog-sig] The Softpedia spam
Noah Kantrowitz wrote: I think most FOSS authors are aware that putting their email in a package is effectively putting it in the clear on the internet. I think we have come beyond the days of noah (at) coderanger [dot] net and all those silly tricks that were popular not too long ago. If an author is excessively concerned about spam, they shouldn't put their email in author_email. Is that field mandatory now or something? Softpedia is a little annoying with the emails, but I've found them useful personally (along with versiontracker) when looking for OS X software before. Freshmeat is a similar index of FOSS projects, and I've definitely used that before. Is there some reason we are objecting to including PyPI data in other software catalogs? If it makes it a tiny bit easier to find Python software, I'm all for it. No, but the PSF should be asked for permission before using the data on some other site. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, May 07 2010) Python/Zope Consulting and Support ...http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ 2010-04-23: Released mxODBC.Zope.DA 2.0.1http://zope.egenix.com/ ::: Try our new mxODBC.Connect Python Database Interface for free ! eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig
Re: [Catalog-sig] The Softpedia spam
Noah Kantrowitz wrote: On May 7, 2010, at 12:47 AM, M.-A. Lemburg wrote: Noah Kantrowitz wrote: I think most FOSS authors are aware that putting their email in a package is effectively putting it in the clear on the internet. I think we have come beyond the days of noah (at) coderanger [dot] net and all those silly tricks that were popular not too long ago. If an author is excessively concerned about spam, they shouldn't put their email in author_email. Is that field mandatory now or something? Softpedia is a little annoying with the emails, but I've found them useful personally (along with versiontracker) when looking for OS X software before. Freshmeat is a similar index of FOSS projects, and I've definitely used that before. Is there some reason we are objecting to including PyPI data in other software catalogs? If it makes it a tiny bit easier to find Python software, I'm all for it. No, but the PSF should be asked for permission before using the data on some other site. Permission is probably not a good thing to inject, too much risk of being picky on who can use the data. If it is available to anyone, it should be available to all. I would agree that as a professional courtesy it would be nice if people would let us know if they are mining PyPI, but you are dipping into dangerous territory if you put a gate in front of it. Why do you think so ? The PSF would most certainly apply the same openness it is applying for its own trademarks. I believe that package authors uploading things to PyPI should be able to trust that the PSF (being behind PyPI) uses this data with the appropriate care. The same is true if you upload data to Freshmeat, Sourceforge and other such sites. Why should PyPI be different ? -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, May 07 2010) Python/Zope Consulting and Support ...http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ 2010-04-23: Released mxODBC.Zope.DA 2.0.1http://zope.egenix.com/ ::: Try our new mxODBC.Connect Python Database Interface for free ! eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig
Re: [Catalog-sig] The Softpedia spam
Noah Kantrowitz wrote: On May 7, 2010, at 12:57 AM, M.-A. Lemburg wrote: Noah Kantrowitz wrote: On May 7, 2010, at 12:47 AM, M.-A. Lemburg wrote: Noah Kantrowitz wrote: I think most FOSS authors are aware that putting their email in a package is effectively putting it in the clear on the internet. I think we have come beyond the days of noah (at) coderanger [dot] net and all those silly tricks that were popular not too long ago. If an author is excessively concerned about spam, they shouldn't put their email in author_email. Is that field mandatory now or something? Softpedia is a little annoying with the emails, but I've found them useful personally (along with versiontracker) when looking for OS X software before. Freshmeat is a similar index of FOSS projects, and I've definitely used that before. Is there some reason we are objecting to including PyPI data in other software catalogs? If it makes it a tiny bit easier to find Python software, I'm all for it. No, but the PSF should be asked for permission before using the data on some other site. Permission is probably not a good thing to inject, too much risk of being picky on who can use the data. If it is available to anyone, it should be available to all. I would agree that as a professional courtesy it would be nice if people would let us know if they are mining PyPI, but you are dipping into dangerous territory if you put a gate in front of it. Why do you think so ? The PSF would most certainly apply the same openness it is applying for its own trademarks. I believe that package authors uploading things to PyPI should be able to trust that the PSF (being behind PyPI) uses this data with the appropriate care. The same is true if you upload data to Freshmeat, Sourceforge and other such sites. Why should PyPI be different ? I just don't think the PSF or this SIG should be in the business of saying who can access PyPI (which is what this boils down to at a philosophical level). That said, I also have a lot of faith in the judgement of the PSF and if they felt they could take on this (large) responsibility I wouldn't fight it that hard. I would fight harder to say that this shouldn't be the job of the SIG though. This would be the PSF's task, since the relationship is between the package author and the PSF, not this SIG, although the PSF could approach the SIG for help, e.g. in order to define where to draw the line. Please also note that the PSF would not be in the business of saying who can access PyPI, only in the business of saying who is allowed to publicly redistribute that data and under which conditions. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, May 07 2010) Python/Zope Consulting and Support ...http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ 2010-04-23: Released mxODBC.Zope.DA 2.0.1http://zope.egenix.com/ ::: Try our new mxODBC.Connect Python Database Interface for free ! eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig
Re: [Catalog-sig] PyPI down again...
Chris Withers wrote: ...would be good to know what brought it down before and what has brought it down again. It works for me. As an interim solution, what do I need to do to get access to the box running PyPI so I can get in and investigate/restart Apache? Since PyPI is a rather essential Python resource, is there some monitoring in place to automatically notify the webmasters ? Something like e.g. a Zenoss instance checking whether PyPI is pingable. If not, we'd need to address this in the PSF infrastructure committee. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jun 11 2010) Python/Zope Consulting and Support ...http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ 2010-07-19: EuroPython 2010, Birmingham, UK37 days to go ::: Try our new mxODBC.Connect Python Database Interface for free ! eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig
Re: [Catalog-sig] PyPI down again...
Tarek Ziadé wrote: On Fri, Jun 11, 2010 at 11:06 PM, M.-A. Lemburg m...@egenix.com wrote: Martin v. Löwis wrote: For a smaller project, start putting mirror support into setuptools or distribute; this would make short (several hours) outages less severe for the class of users that want permanent availability for downloading. It's unlikely that the mirrors would break when the master goes down; they just stop mirroring. A better and cleaner strategy is to put the static PyPI information up on Amazon Cloudscape and have DNS take care of providing local mirrors (edge servers) to setuptools et al. Such a setup won't require any complicated mirror logic in any of the existing client tools. By moving the PyPI installation to Amazon AWS, we could also get the RPC access distributed to more than just one server. As I said before, the PSF infrastructure committee needs to get on of the job of getting this implemented (including funding this development). If someone wants to volunteer helping with the setup, please contact the PSF at p...@python.org. What about continuing the work that was started last year ? (and not finished due to a lack of time) There's a PEP we have started about a mirroring infrastructure: http://www.python.org/dev/peps/pep-0381/ Some of its parts are already implemented in PyPI, and what we need now is to work on the client side (pip, distribute, etc) and bootstrap one or two mirrors using the protocol. We've had some private discussions about this, so I'm just going to summarize... The idea here is not to override the mirror PEP ideas, but to use the existing PyPI installation and put the content needed for the most widely distributed package tool (currently setuptools and zc.buildout) on a content delivery network (CDN) in order to have it highly available on a managed edge network. Amazon Cloudfront is such a CDN and has Python interfaces, hence the idea to use Cloudfront. I asked for volunteers, because I didn't know enough about Amazon Cloudfront to write up a proposal and don't have the cycles available to implement such a setup myself. In the meantime, I've done some research and now know enough to write a proposal for the PSF board to consider. If the board thinks it's a good idea, we'll need to pursue finding volunteers to implement it. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jun 13 2010) Python/Zope Consulting and Support ...http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ 2010-07-19: EuroPython 2010, Birmingham, UK35 days to go ::: Try our new mxODBC.Connect Python Database Interface for free ! eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig
Re: [Catalog-sig] PyPI down again...
Martin v. Löwis wrote: I think it overlaps a bit the PEP goal, which is to set up a network of mirrors, and have them listed in the PyPI DNS so clients can switch from one mirror to another.(and even do geoloc!) JFTR, this already exists. a.mirrors.pypi.python.org and b.mirrors.pypi.python.org are already there and could be used by clients. Well maybe this is the best path to follow right now, as it will be done faster, without having to interact with much people to set it up, so it's a quick win. My main worry (besides the client integration) is statistics: I do want to get download statistics. So anybody implementing it would have to find a way of fetching the download numbers from Amazon. Download statistics are readily available from Amazon Cloudfront, so no worries: you'll get statistics for all edge server downloads. http://docs.amazonwebservices.com/AmazonCloudFront/latest/DeveloperGuide/index.html?AccessLogs.html But it will probably kill the mirroring protocol idea from the PEP in the process, which I think is superior in the long term since it provides a standardized ground for the community to set up mirrors independently from pypi.python.org. I also remain skeptical that this cloud idea is useful at all. Amazon Cloudfront is a *beta* service. So they aren't sure themselves whether it works correctly - and there have been reports about two-day outages of EC2, for bitbucket.org. There also have been complaints about the available bandwidth. So I'm not sure whether replacing a single point of failure with a different one is actually improving anything. Amazon Cloudfront uses S3 as basis for the service, S3 has been around for years and has a very stable uptime: http://www.readwriteweb.com/archives/amazon_s3_exceeds__percent_uptime.php Cloudfront itself has been around since Nov 2008. You can check their current online status using this panel: http://status.aws.amazon.com/ Apart from the gained availability and outsourced management, we'd also get faster downloads in most parts of the world, due to the local caching Cloudfront is applying (and this can be used to further increase the availability, since we can control the expiry time of those local copies). So in summary we are replacing a single point of failure with N points of failure (with N being the number of edge caching servers they use). Regaring the bitbucket problem you mentioned: EC2 is their virtual server service, which we don't use. The bitbucket problems originated from a) a DDoS attack on their virtual servers running on EC2 and b) a problem with the Amazon EBS, which is their virtualized SAN, and was related to the way the DDoS was done (EBS and the DDoS attack both used UDP): http://blog.bitbucket.org/2009/10/04/on-our-extended-downtime-amazon-and-whats-coming/ And to re-iterate, the problem wasn’t really Amazon EC2 or EBS, it was isolated to our case, due to the nature of the attack. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jun 14 2010) Python/Zope Consulting and Support ...http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ 2010-07-19: EuroPython 2010, Birmingham, UK34 days to go ::: Try our new mxODBC.Connect Python Database Interface for free ! eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig
Re: [Catalog-sig] PyPI down again...
Mathieu Leduc-Hamel wrote: Agreed, that's why I think it would be useful to simply put all meta data into a SQLite database file and ship that as static file as well. Local clients could then download the database file (probably only a few MB) and work on it locally. I don't think it would be easy to do that right now since the database store more informations than only the metadata of the all packages, you don't wanna give all the informations about users accounts by example... PyPI uses PostgreSQL as database backend, so the SQLite database file would be a (partial) copy of that database. Of course, it would have to only contain meta-data that is also visible via the web GUI. And, I don't understand how it can be perform with always updated informations like ones on the pypi website, your database is always updated, than it's not possible to have a completely updated one... True, but I think only very few users are really after real-time data from PyPI. Those can use the true RPC interfaces. For the others, a static copy created and updated every 10-20 minutes or so, is likely good enough. Anyway, it's an idea based on the 80/20 rule :-) Search might not by the big deal if the data and a good cache interface is implemented, the number of parallel connexion is something else... This would also make searches in PyPI a lot faster... not only because searches could be done locally, but also because the server wouldn't have to handle the load of those searches from hundreds of clients. But, definitively, yes, I'll consider that. Thanks ! -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jun 14 2010) Python/Zope Consulting and Support ...http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ 2010-07-19: EuroPython 2010, Birmingham, UK34 days to go ::: Try our new mxODBC.Connect Python Database Interface for free ! eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig
[Catalog-sig] Proposal: Move PyPI static data to the cloud for better availability
As mentioned, I've been working on a proposal text for the cloud idea. Here's a first draft. Please have a look and let me know whether I've missed any important facts. Thanks. I intend to post the proposal to the PSF board (of which I'm a member, in case you shouldn't know) and to have it vote on the proposal in one of the next board meetings. PSF-Proposal: 100 Title: Move PyPI static data to the cloud for better availability Version: Draft 1 Last-Modified: 2010-06-15 Author: m...@lemburg.com (Marc-André Lemburg) Discussions-To: catalog-sig@python.org Status: Draft Type: Informational Created: 2010-06-14 Post-History: Proposal: Move PyPI static data to the cloud for better availability Motivation -- PyPI has in recent months seen several outages with the index not being unavailable to both users using the web GUI interface as well as package administration tools such as easy_install from setuptools. As more and more Python applications rely on tools such as easy_install for direct installation, or zc.buildout to manage the complete software configuration cycle, the PyPI infrastructure receives more and more attention from the Python community. In order to maintain its credibility as software repository, to support the many different projects relying on the PyPI infrastructure and the many users who rely on the simplified installation process enabled by PyPI, the PSF needs to take action and move the essential parts of PyPI to a more robust infrastructur that provides: * scalability * 24/7 system administration management * geo-localized fast and reliable access Current Situation - PyPI is currently run from a single server hosted in The Netherlands (ximinez.python.org). This server is run by a very small team of sys admin. PyPI itself has in recent months been mostly maintained by one developer: Martin von Loewis. Projects are underway to enhance PyPI in various ways, including a proposal to add external mirroring (PEP 381), but these are all far from being finalized or implemented. Usage - PyPI provides four different mechanisms for accessing the stored information: * a web GUI that is meant for use by humans * an RPC interface which is mostly used for uploading new content * a semi-static /simple package listing, used by setuptools * a static area /packages for package download files and documentation, used by both the web GUI and setuptools The /simple package listing is dump of all packages in PyPI using a simple HTML page with links to sub-pages for each package. These sub-pages provide links to download files and external references. External tools like easy_install only use the /simple package listing together with the hosted package download files. While the /simple package listing is currently dynamically created from the database in real-time, this is not really needed for normal operation. A static copy created every 10-20 minutes would provide the same level of service in much the same way. Moving static data to a CDN --- Under the proposal the static information stored in PyPI (meta-information as well as package download files and documentation) is moved to a content delivery network (CDN). For this purpose, the /simple package listing is replaced with a static copy that is recreated every 10-20 minutes using a cronjob on the PyPI server. At the same intervals, another script will scan the package and documentation files under /packages for updates and upload any changes to the CDN for neartime availability. By using a CDN the PSF will enable and provide: * high availability of the static PyPI content * offload management to the CDN * enable geo-localized downloads, i.e. the files are hosted on a nearby server * faster downloads * more reliability and scalability * move away from a single point of failure setup Note that the proposal does not cover distribution of the dynamic parts of PyPI. As a result uploads to PyPI may still fail if the PyPI server goes down. However, these dynamic parts are currently not being used by the existing package installation tools. Choice of CDN: Amazon Cloudfront To keep the costs low for the PSF, Amazon Cloudfront appears to be the bext choice for CDN. Cloudfront is supported by a set of Python libraries (e.g. Amazon S3 lib and boto), upload scripts are readily available and can easily be customized. http://www.saltycrane.com/blog/2008/12/card-store-project-4-notes-using-amazons-cloudfront/ Other CDNs, such as Akamai, are either more expensive or require custom integration. Availability of Python-based tools is not always given, in fact, accessing such information is difficult for most of the proporietary CDNs. Cloudfront: quality of service -- Amazon Cloudfront uses S3 as basis for the service, S3 has been
Re: [Catalog-sig] PyPI down again...
Mathieu Leduc-Hamel wrote: To continue the discussion about a rewrite or a cleanup of the Pypi codebase, I'm from Montreal-Python usergroup and I'm say that yes at the first the current codebase of pypi seem to be very unclear and difficult to maintain. But it's not an impossible mission and we are currently in the process of: - Adding functional test. The test coverage is now around 40% percent. - When we'll reach a more complete coverage, we want to replace the psycopg api by SQLAlchemy - Replace many manual manipulation of the metadata by a more robust and straightforward way of dealing with (distutils2 might be the option there) At first I was thinking about rewriting everything using the chishop project (an implementation of PyPi using django). But having the control of the code source and not dependent of any framework is maybe a better idea. More than, despite the frequent outage, pypi is working today, then just a modernization of code base seem to be best idea. By the wat, after a code review of tarek, a very useful thing might be to find a better way to deal and implement contributions coming from community. Right now Tarek is responsible of making the link between our effert and the work of Martin but we don't have any official public mirror of the source code and any roadmap. You should be able to get access to the Python sandbox repository and add your project there: http://svn.python.org/projects/sandbox/trunk/ If that's not an option, I'd suggest you have a look at one of the other public repo sites such as launchpad. Note that working on PyPI needs a somewhat different development approach since any changes will be run on a live system. In my experience the best way to do this is by gradually changing things (rather than introduce big structural changes such as using SA instead of a native adapter) and keeping a close eye on the log files for any problems. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jun 15 2010) Python/Zope Consulting and Support ...http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ 2010-07-19: EuroPython 2010, Birmingham, UK33 days to go ::: Try our new mxODBC.Connect Python Database Interface for free ! eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig
Re: [Catalog-sig] Proposal: Move PyPI static data to the cloud for better availability
Steven D'Aprano wrote: On Tue, 15 Jun 2010 09:49:03 pm M.-A. Lemburg wrote: As mentioned, I've been working on a proposal text for the cloud idea. Here's a first draft. Please have a look and let me know whether I've missed any important facts. Thanks. I think the most important missed fact is, just how unreliable is PyPI currently? Does anyone know? I know there's a number of people complaining that it's down all the time, or even occasionally, but I think that we need to know the magnitude of the problem that needs solving. What's the average length of time between outages? What's the average length of the outage? Just saying that there's been several outages in recent months is awfully hand-wavy. I'm sorry, but I can't provide any numbers since there doesn't appear to be any monitoring in place to pull those numbers from. What I can say is that from reading the various mailing lists, PyPI is down often enough to let people start discussions about it and that's the point I want to address: In order to maintain its credibility as software repository, to support the many different projects relying on the PyPI infrastructure and the many users who rely on the simplified installation process enabled by PyPI, the PSF needs to take action and move the essential parts of PyPI to a more robust infrastructur that provides: * scalability * 24/7 system administration management * geo-localized fast and reliable access Setting up some Zenoss or Nagios monitoring system to take care of monitoring the PyPI server (and our other servers) would be a separate project. [...] Amazon Cloudfront uses S3 as basis for the service, S3 has been around for years and has a very stable uptime: http://www.readwriteweb.com/archives/amazon_s3_exceeds__percent_u ptime.php Is there anyone here who has personal experience with Cloudfront and is willing to vouch for it? Or argue against it? We can only go so far based on Amazon's marketing material. I don't have personal experience with Cloudfront, but have advised companies to use Amazon EC2 and S3 as disaster recovery and backup solution. So far, none of them has ever complained. While doing research for the proposal, I've read a lot of posts about people using Amazon S3 and Cloudfront. The overall feedback is very positive. If things still don't work out for us, we can always go back to the single server setup. The proposal doesn't bind us to Cloudfront or the CDN setup in any way. One thing that does worry me: So in summary we are replacing a single point of failure with N points of failure (with N being the number of edge caching servers they use). I don't think this means what you seem to think it means. If you replace a single point of failure with N points of failure, your overall reliability goes down, not up, since there are now more things to go wrong. Assuming that they're independent points of failure, that means your total number of failures will increase by a factor of N. For example, if a single edge server in (say) Australia goes down, Amazon might not count it as an outage for the purpose of calculating their 99.99% reliability since the system as a whole is still up, but conceivably Australian users might see an outage (or at least a slow-down). With N servers, I'd expect N times the number of individual outages, with Amazon presumably only counting it as system down if all N servers go down at the same time. It's poor wording, I agree. Thanks for pointing this out. The math is correct, though, I believe... Let's say all servers have a probability of being unavailable of P(Server down) = q (with q in [0,1]). Let's further assume that all servers are independent of each other. The probability of none of the servers being available then is P(System down) = q^N = q Cloudfront uses a DNS round-robin system with a TTL of 60 seconds, and returns more than just one cache server per edge node, e.g. in Germany I get 8 cache servers: dig d1ylr6sba64qi3.cloudfront.net ;; ANSWER SECTION: d1ylr6sba64qi3.cloudfront.net. 57 INCNAME d1ylr6sba64qi3.ams1.cloudfront.net. d1ylr6sba64qi3.ams1.cloudfront.net. 57 IN A 216.137.59.184 d1ylr6sba64qi3.ams1.cloudfront.net. 57 IN A 216.137.59.250 d1ylr6sba64qi3.ams1.cloudfront.net. 57 IN A 216.137.59.84 d1ylr6sba64qi3.ams1.cloudfront.net. 57 IN A 216.137.59.106 d1ylr6sba64qi3.ams1.cloudfront.net. 57 IN A 216.137.59.15 d1ylr6sba64qi3.ams1.cloudfront.net. 57 IN A 216.137.59.102 d1ylr6sba64qi3.ams1.cloudfront.net. 57 IN A 216.137.59.40 d1ylr6sba64qi3.ams1.cloudfront.net. 57 IN A 216.137.59.118 ;; AUTHORITY SECTION: ams1.cloudfront.net.141251 IN NS ns-ams1-01.cloudfront.net. ams1.cloudfront.net.141251 IN NS ns-ams1-02.cloudfront.net. The probability of all 8 server being down is P(Edge node down) = q^8 = q Assuming that Amazon's system monitoring is fast enough to detect the edge node down
Re: [Catalog-sig] Proposal: Move PyPI static data to the cloud for better availability
Alexis Métaireau wrote: Hello, Firstly, as Tarek said in another thread, I'm afraid this kill the PEP381 about making a mirroring infrastructure. Having a infrastructure hosted on a cloud platform may be confortable, and probably needed to have a 24/7 running system, but we need to take care of letting possible the creation of new public mirrors, outside from the Amazon (or whatever) cloud infrastructure. The proposal doesn't prevent that. However, please note that setting up public mirrors not under PSF control has its own set of (legal) problems, which the PSF hosted cloud setup avoids. On Tue, Jun 15, 2010 at 1:49 PM, M.-A. Lemburg m...@egenix.com wrote: PyPI is currently run from a single server hosted in The Netherlands (ximinez.python.org). This server is run by a very small team of sys admin. As Martin von Löwis said, this already exists. a.mirrors.pypi.python.org and b.mirrors.pypi.python.org are already there and could be used by clients. Maybe Martin can you explain us (apologies if this is already done somewhere) how things are working from now ? Is this possible to rely on the existing work rather than using a cloud system ? What's the in place infrastructure ? In order to use those two servers, you'd still need to implement the redirection changes or client side tool changes and, what's more important, you'd need to administer and monitor those servers 24/7 to achieve similar uptime. The latter is what the proposal is all about: we're outsourcing the administration and monitoring to a service provider. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jun 15 2010) Python/Zope Consulting and Support ...http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ 2010-07-19: EuroPython 2010, Birmingham, UK33 days to go ::: Try our new mxODBC.Connect Python Database Interface for free ! eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig
Re: [Catalog-sig] Proposal: Move PyPI static data to the cloud for better availability
Michael Crute wrote: On Tue, Jun 15, 2010 at 7:49 AM, M.-A. Lemburg m...@egenix.com wrote: As mentioned, I've been working on a proposal text for the cloud idea. Here's a first draft. Please have a look and let me know whether I've missed any important facts. Thanks. What about a set of volunteer mirrors of PyPi similar to the way CPAN and Linux distributions handle this problem. pypi.python.org? That approach eliminates any cost for the PSF and might ultimately result in better reliability. With the volunteer mirror system you would still statically generate the files and just make them available for rsync then setup a page to allow mirrors to register (see CPAN). If you take this approach I would be happy to donate a mirror to the pool. Thanks for the offer. Setting up such a network based on PSF partner organizations (to avoid the legal problems) would work indeed, but it would both take longer to setup and require more work on the administration side. I still think that the cloud proposal is more cost effective and faster to setup. If it doesn't work out, we can always go back to such a network of servers that we administer on our own. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jun 15 2010) Python/Zope Consulting and Support ...http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ 2010-07-19: EuroPython 2010, Birmingham, UK33 days to go ::: Try our new mxODBC.Connect Python Database Interface for free ! eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig
Re: [Catalog-sig] Proposal: Move PyPI static data to the cloud for better availability
Tarek Ziadé wrote: On Tue, Jun 15, 2010 at 6:02 PM, M.-A. Lemburg m...@egenix.com wrote: Alexis Métaireau wrote: Hello, Firstly, as Tarek said in another thread, I'm afraid this kill the PEP381 about making a mirroring infrastructure. Having a infrastructure hosted on a cloud platform may be confortable, and probably needed to have a 24/7 running system, but we need to take care of letting possible the creation of new public mirrors, outside from the Amazon (or whatever) cloud infrastructure. The proposal doesn't prevent that. However, please note that setting up public mirrors not under PSF control has its own set of (legal) problems, which the PSF hosted cloud setup avoids. Mirrors already exists out there, so unless you ban them (which would be a really bad idea) setting up a cloud will not fix any legal issue if you think there's a legal issue. In any case, you can't prevent people from creating mirrors even if you would say its illegal. Moreover, having mirrors provided by the community is way better than relying on one single entity (the PSF) for this. (if we think decentralized) So I think it would be better to focus on PEP 381, and make those existing mirrors comply with it. And maybe work on the legal issues you've mentioned That can all happen in parallel. On Tue, Jun 15, 2010 at 1:49 PM, M.-A. Lemburg m...@egenix.com wrote: PyPI is currently run from a single server hosted in The Netherlands (ximinez.python.org). This server is run by a very small team of sys admin. As Martin von Löwis said, this already exists. a.mirrors.pypi.python.org and b.mirrors.pypi.python.org are already there and could be used by clients. Maybe Martin can you explain us (apologies if this is already done somewhere) how things are working from now ? Is this possible to rely on the existing work rather than using a cloud system ? What's the in place infrastructure ? In order to use those two servers, you'd still need to implement the redirection changes or client side tool changes and, what's more important, you'd need to administer and monitor those servers 24/7 to achieve similar uptime. Not at all because the registered mirrors would be in the DNS round robin, and the clients would just have to switch to another mirror if a mirror is down. (that's explained in PEP 381) Someone would still have to provide the system administration for those servers and also make sure that the servers do actually provide up-to-date snapshots. DNS round-robin will help with finding the servers, not with the other aspects. Something the PEP should focus a bit more on is the freshness guarantee of the mirror data. It currently puts this important detail into the hands of the client software, so every package tool will have to find it's own way of determining whether to use a mirror or not. Another important feature missing from the PEP is data consistency. Since a client tool would only communicate with one mirror, it will ultimately have to trust the information on that server, including the MD5 sums. This makes it rather easy to manipulate data on the servers (not by the admins, but by hackers manipulating those servers). Having digitally signed packages, like you do on many Linux repository servers, would solve this issue, but also require a complete verification infrastructure on the client side. You don't need any of this with the cloud caching approach. Such a decentralized system is far more reliable than any centralized system, and won't cost anything to the PSF. We'll see :-) The latter is what the proposal is all about: we're outsourcing the administration and monitoring to a service provider. Having a better PyPI server is of course a good idea, don't get me wrong. But it doesn't really solve anything at this point. Obviously I have a different opinion, otherwise I wouldn't have written the proposal :-) -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jun 15 2010) Python/Zope Consulting and Support ...http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ 2010-07-19: EuroPython 2010, Birmingham, UK33 days to go ::: Try our new mxODBC.Connect Python Database Interface for free ! eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig
Re: [Catalog-sig] Proposal: Move PyPI static data to the cloud for better availability
Jesus Cea wrote: On 15/06/10 13:49, M.-A. Lemburg wrote: Server side: upload cronjobs Since the /simple index tree is currently being created dynamically, we'd need to create static copies of it at regular intervals in order to upload the content to the S3 bucket. This can easily be done using tools such as wget or curl. Both the static copy of the /simple tree and the static files uploaded to /packages then need to be uploaded or updated in the S3 bucket by a cronjob running every 10-20 minutes. I don't comment about the convenience to migrate or not. But having to wait 20 minutes to deploy my just released package to my datacenter is a bit inconvenient to me :-). Would be nice to change PYPI code just to dump simple each time the database changes. Perusing the RSS, the load should be low and actually less demanding to CPU and database server (if you only update simple with the changes, not rebuilding everything each time). I'll leave that for a version 2.0 of the cloud idea :-) My main interest now is getting something done with only requiring minimal changes to the PyPI software. Note that with community servers that only mirror once a day, you'd have to wait up to a whole day for your package updates to become visible worldwide. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jun 15 2010) Python/Zope Consulting and Support ...http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ 2010-07-19: EuroPython 2010, Birmingham, UK33 days to go ::: Try our new mxODBC.Connect Python Database Interface for free ! eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig
Re: [Catalog-sig] Proposal: Move PyPI static data to the cloud for better availability
Tarek Ziadé wrote: On Tue, Jun 15, 2010 at 7:34 PM, M.-A. Lemburg m...@egenix.com wrote: [..] So I think it would be better to focus on PEP 381, and make those existing mirrors comply with it. And maybe work on the legal issues you've mentioned That can all happen in parallel. I really doubt it. You have come with a cloud proposal and want it to be funded by the PSF. Your proposal is basically a proprietary mirroring system, and it competes with the mirroring protocol we wanted to build, based on the existing mirrors the community has. I'm not trying to compete with your mirror PEP, just trying to solve a problem. So far I don't see any advantage in a cloud-based mirror managed by the PSF, compared to a round of community mirrors. We can have it up and running in a few days and it doesn't require any changes to existing client tools, that's the main argument. The proposal solves a problem we have now and doesn't get in the way of PEP 381. Instead it buys it more time to get finalized, implemented and deployed on the client side. If you need funding for PEP 381, please write a proposal. This would then also need to address the problem of added administration overhead (screening mirror server providers, getting them registered or removed, monitored and verified for correct operation, etc.). -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jun 15 2010) Python/Zope Consulting and Support ...http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ 2010-07-19: EuroPython 2010, Birmingham, UK33 days to go ::: Try our new mxODBC.Connect Python Database Interface for free ! eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig
Re: [Catalog-sig] Proposal: Move PyPI static data to the cloud for better availability
Martin v. Löwis wrote: I read pep 381 long time ago and I don't remember how/when a mirror would update, but I do remember it doesn't mandate digital signatures (signed by pypi central node, verified by setuptoolsfriends). That is a big gap, in my opinion. The PEP doesn't explain the digital signing that is going on in mirroring. See http://mail.python.org/pipermail/catalog-sig/2009-March/002018.html This is fully implemented (except that client would need to verify the signatures, and except key rollover hasn't happened yet). That's good to know, but I think some parts of this will have to be discussed some more: /serverkey Public DSA key of the server, in the PEM format as generated by openssl dsa -pubout (i.e. RFC 3280 SubjectPublicKeyInfo, with the algorithm 1.3.14.3.2.12). This URL must *not* be mirrored, and clients must fetch the official serverkey from PyPI directly. The serverkey * How will clients be sure that they are getting the correct key ? * What would a client do if the PyPI server is down ? * How would clients protect their local cached copy of the server key against manipulation ? * Without access to OpenSSL and M2Crypto, how would clients apply the check ? Also, please consider that access to crypto code is restricted in some parts of the world. Users in those countries would have to be able to turn off verification. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jun 15 2010) Python/Zope Consulting and Support ...http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ 2010-07-19: EuroPython 2010, Birmingham, UK33 days to go ::: Try our new mxODBC.Connect Python Database Interface for free ! eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig
Re: [Catalog-sig] Proposal: Move PyPI static data to the cloud for better availability
Martin v. Löwis wrote: PyPI itself has in recent months been mostly maintained by one developer: Martin von Loewis. Projects are underway to enhance PyPI in various ways, including a proposal to add external mirroring (PEP 381), but these are all far from being finalized or implemented. That's not at all accurate: PEP 381 is almost completely implemented in the mirroring tools. Which parts of PEP 381 are implemented ? Client-side support is missing, but isn't strictly necessary as users could manually point their setuptools installation to a mirror. That's not a good argument. Users like setuptools because they can run: easy_install stuff and let it do whatever it needs to do. It's important not to require changes on the client side. While the /simple package listing is currently dynamically created from the database in real-time, this is not really needed for normal operation. A static copy created every 10-20 minutes would provide the same level of service in much the same way. For normal operation (i.e. on the master copy), this would be really insufficient. Users expect, in automated build processes, that the packages they upload are available for *immediate* download. Power users and developers will probably want that, but those can hook up to the PyPI server directly if they have such a need. For the majority, waiting 10-20 minutes should be fine. Note that the push idea is part of the plan, but won't happen in the initial rollout. Under the proposal the static information stored in PyPI (meta-information as well as package download files and documentation) is moved to a content delivery network (CDN). There is a good chance that, before that proposal is implemented, the PEP 381 implementation is completed. Including getting all client side package tools updated and deployed to the existing users ? At the same intervals, another script will scan the package and documentation files under /packages for updates and upload any changes to the CDN for neartime availability. Not sure why you wouldn't push every change immediately to the CDN, though. The proposal wants to do without changing PyPI code where possible. This is planned for a later release. If this can be had without any major changes, we can also add it to phase one. Cloudfront itself has been around since Nov 2008. Please add that Amazon considers Cloudfront as a beta service. I don't think that makes a difference. The beta term is a web 2.0 marketing term, nothing more. But I'll add it anyway. The pypi.python.org domain would then have to be setup to map to multiple IP addresses via DNS round-robin, one entry for each redirection server, e.g. pypi.python.org. IN A 123.123.123.1 pypi.python.org. IN A 123.123.123.2 pypi.python.org. IN A 123.123.123.3 pypi.python.org. IN A 123.123.123.4 I don't think this works if one of the servers fails (or, worse, produces a hanging connection). What piece of software would implement the fallback to the next machine? AFAIK, the package tools don't currently implement any kind of fail- over. While this would be good to have and provide a better user experience, it's not required. The user would just need to restart the command and then get a new server IP address to try - just like you do in a web browser if a page doesn't load. That's still a lot better than not being able to download anything at all. The alternative would be a proxy setup, which then again introduces a single point of failure (unless you setup a HA cluster). The mirror PEP shares this problem with the cloud proposal. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jun 15 2010) Python/Zope Consulting and Support ...http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ 2010-07-19: EuroPython 2010, Birmingham, UK33 days to go ::: Try our new mxODBC.Connect Python Database Interface for free ! eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig
Re: [Catalog-sig] Proposal: Move PyPI static data to the cloud for better availability
Tarek Ziadé wrote: On Tue, Jun 15, 2010 at 10:14 PM, M.-A. Lemburg m...@egenix.com wrote: I'm not trying to compete with your mirror PEP, just trying to solve a problem. We are trying to solve the same problem, aren't we ? Sure, but the intent is not to compete with the PEP. Even with the cloud proposal implemented, we can still have a mirror setup like the one you propose. That is : avoiding any downtime when PyPI is used by setuptools and derived tools. So if you solve this problem by implementing a cloud system backed by a PSF funding, and managed by the PSF, and if you claim that there will be no more downtime, then PEP 381 will be useless. No, not at all. The PSF would not be the only user of the PEP and the client tools. If all client tools implement the things you suggested in the PEP, we'd have a lot more possibilities. I am just arguing that I don't think it's the best solution, compared to what was started e.g. a community network of mirrors. I've heard you, but still disagree. I think we'll just have to leave it at that. So far I don't see any advantage in a cloud-based mirror managed by the PSF, compared to a round of community mirrors. We can have it up and running in a few days and it doesn't require any changes to existing client tools, that's the main argument. The global uptime of PyPI in this last year was probably around 99.9%, so I don't think we are in such a rush to set up something in any case. The problem occured in the past, and was fixed in a matter of hours. every. time. It's just that everytime it happens it makes us all want to improve the system. So why don't we implement the best solution ? Maybe we could use a wiki page and work on a synthetic overview of the pros and cons. Again: I don't want to compete against the PEP. I'm looking for a solution that's easy to implement and doesn't get in the way. That's all. Nothing more. If you can come up with a solution that's ready in a month or two, I'll happily wait. The proposal solves a problem we have now and doesn't get in the way of PEP 381. Instead it buys it more time to get finalized, implemented and deployed on the client side. If you need funding for PEP 381, please write a proposal. I won't. I think we should decide here, all together, what is the best technical solution to set up mirrors (e.g. cloud vs community) Then, ask for its funding from the PSF. This would then also need to address the problem of added administration overhead (screening mirror server providers, getting them registered or removed, monitored and verified for correct operation, etc.). This overhead is minimum compared to an in-house administration of a full mirroring system based on a cloud imho. YMMV, but my experience with these systems is that they cause a lot less overhead than anything you administer yourself. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jun 15 2010) Python/Zope Consulting and Support ...http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ 2010-07-19: EuroPython 2010, Birmingham, UK33 days to go ::: Try our new mxODBC.Connect Python Database Interface for free ! eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig
Re: [Catalog-sig] [OT] Nagios / Shinken
Antoine Pitrou wrote: M.-A. Lemburg mal at egenix.com writes: Setting up some Zenoss or Nagios monitoring system to take care of monitoring the PyPI server (and our other servers) would be a separate project. Just for the record, I would mention that someone started a rewrite of the Nagios software in Python: http://www.shinken-monitoring.org/ According to the author, the Python rewrite is also much faster than the original C software: http://www.shinken-monitoring.org/features/huge-performances/ Probably a good showcase of the using a dynamic language allows you to focus on a better architecture argument :-) Zenoss is written in Python and uses Zope for the web GUI. It has a large community around it and provides all the enterprise features you'd need from such a system. http://www.zenoss.com/ and Zenoss can use Nagios plugins as well. I'd see a chance for such a new tool, though: Zenoss can be very complicated to setup, esp. if you're not using SNMP on all your machines. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jun 16 2010) Python/Zope Consulting and Support ...http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ 2010-07-19: EuroPython 2010, Birmingham, UK32 days to go ::: Try our new mxODBC.Connect Python Database Interface for free ! eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig
Re: [Catalog-sig] Proposal: Move PyPI static data to the cloud for better availability
Antoine Pitrou wrote: Tarek Ziadé ziade.tarek at gmail.com writes: And we happen to have this network already: lots of people will host a PyPI mirror as soon as it's easy to set one imho. You must be careful that the mirrors are properly managed and administered, though. Having stale/dysfunctioning mirrors is worse than having no mirrors at all. It is likely that some people will setup a mirror and then forget to take care about it. Like our buildbots really. Right, it's that administration overhead I was referring to. Perhaps we should just let the users decide: a) they use the default PyPI access (which we then enhance by caching the content in the cloud) b) they setup their easy_install or zc.buildout to pull data from a mirror network by enabling a configuration option Since implementing option b) will require updating existing package tools on the client side anyway, the extra configuration shouldn't be a problem. Option a) requires no changes whatsoever on the client side. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jun 16 2010) Python/Zope Consulting and Support ...http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ 2010-07-19: EuroPython 2010, Birmingham, UK32 days to go ::: Try our new mxODBC.Connect Python Database Interface for free ! eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig
Re: [Catalog-sig] [Proposal] Registered packages must provide the source code distribution on PyPI
Andreas Jung wrote: Hi there, I propose a policy change for packages registered with PyPI: - packages registered on PyPI have at least one release I'm not sure what you mean with release. Every package on PyPI is a release, since it comes with a version number. - one release of registered package on PyPI _must_ contain a valid source code distribution (sdist) -100 You'd outrule commercial packages that don't come with a source distribution. PyPI is for everyone, not only for open source packages. Furthermore, not all package authors want to upload their packages to PyPI. And lastly, uploading packages to PyPI (still) has a serious problem: setuptools doesn't know the distinction between UCS2 and UCS4, so uploading eggs for Unix platforms doesn't work out in practice. setuptools also doesn't know that e.g. a Mac OS X fat release may still contain the right binaries for a non-fat build of Python. There are other issues as well, e.g. eGenix produces around 50 release files for every package release amounting to around 150 MB in some cases. It's currently just not feasable to use PyPI for that. - packages registered on PyPI without releases or without source code release are subject to be removed after N days after the day of registration Same as above. Why? Any package registered on PyPI is possibly crucial to any kind of development and deployment. Packages hosted on external servers (referenced through a download_url) are subject to come and go - packages once released should be available at any time from a well-known location (PyPI). Dependencies on the availability of external downloads servers other than PyPI are hardly acceptable for real-world development and deployments. I think it's for the package users to decide whether they trust a package author to maintain his or her package. That's not something PyPI can change. As an example: the Plone CMS buildouts depend on python-openid. This package is registered with PyPI http://pypi.python.org/pypi/python-openid but references to http://openidenabled.com/files/python-openid/packages/python-openid-2.2.4.tar.gz For whatever reason the download URL is no longer working. In fact: openidenabled.com now points to http://www.janrain.com. That's a problem with that particular package, so you should contact the package author. Just because one URL goes away doesn't mean that *all* PyPI package authors who host their software elsewhere are in poor standing. Other reasons for disappearing package in the past: - network or server outages of external servers - users changed their organization and the organization removed content of their former employees I'd say you open a support request for PyPI and then let a sys admin add a note to the package or remove the broken download URL. PyPI is a valuable and crucial resource for Python development. It must be kept up-to-date and consistent. I don't care about the arguments that were made in the past against stronger rules (openness etc.). If that's so, but why should we then care about your arguments ? There are a lot of Python programmers around that are not Python geeks as most of us are and they just become pissed of when packages come and go or are not in the place where one would expect them. That's the nature of the Internet. Besides, would you really want to use a package that's not being maintained anymore ? Even if you do have a source or binary distribution for a package on PyPI, would you really continue to use it if you don't know the author and it hadn't had any release for 3 years ? You can't just blindly rely on things that were uploaded to PyPI and the proposed policy change won't make a difference in that respect. PyPI is a community resource - but community does not mean anarchy where everyone should be able to upload its package crap without looking left and right and having the community and its needs in mind. I think that's asked a bit too much of the package authors. PyPI is just a resource to announce and catalog Python packages, nothing more. PyPI must become a stable package index. Everything registered with PyPI must be available at any time (mirrors, distributing PyPI in the cloud...). I agree that everything uploaded to PyPI should be available anytime, but not that everything registered with PyPI also has to be uploaded to PyPI. Making PyPI more reliable will likely increase the number of package authors who trust PyPI to host their packages. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jun 17 2010) Python/Zope Consulting and Support ...http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ 2010-07-19: EuroPython 2010, Birmingham, UK31
Re: [Catalog-sig] [Proposal] Registered packages must provide the source code distribution on PyPI
Martin v. Löwis wrote: For such packages: send out an email to the package maintainer informing him about the problem and instructing him to fix the problem within N days. After N days: recheck the package state and unregister the package if necessary. Or perhaps a less rude approach: introduce status field for each package (ACTIVE/INACTIVE) and set the state to INACTIVE when the package does not comply with this policy. Inactive packages won't be listed on PyPI and won't be searchable on PyPI. Inactive status should be visible to the author (in logged-in state) with some warning Package is inactive..please upload your sdist). Ok. If nobody opposes to this right now, it's fine with me as well. However, I won't be able to work on this for several months to come. IMO, it's a waste of energy: if a package is useless, just don't use it, and be done. There are many packages on PyPI that are useless to me despite having a source release. Agreed. PyPI can't replace the due-diligence that every package user has to apply before making a choice to invest time into using it. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jun 17 2010) Python/Zope Consulting and Support ...http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ 2010-07-19: EuroPython 2010, Birmingham, UK31 days to go ::: Try our new mxODBC.Connect Python Database Interface for free ! eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig
Re: [Catalog-sig] [Proposal] Registered packages must provide the source code distribution on PyPI
Andreas Jung wrote: M.-A. Lemburg wrote: Andreas Jung wrote: Hi there, I propose a policy change for packages registered with PyPI: - packages registered on PyPI have at least one release I'm not sure what you mean with release. Every package on PyPI is a release, since it comes with a version number. This is a package without a release: http://pypi.python.org/pypi/python-openid It has a name and a version number, so it's a release. It may be an unavailable release, just like say, Windows 98, is not available anymore - and that didn't have a source release file to download either :-) And I can see that you've added a comment to the package that the download URL is not working - that's good, since it will warn users to double-check. - one release of registered package on PyPI _must_ contain a valid source code distribution (sdist) -100 You'd outrule commercial packages that don't come with a source distribution. PyPI is for everyone, not only for open source packages. Commercial package are a special case - I agree. The majority of all PyPI are non-commercial. In addition you could also upload binary release in addition to your own download server. See my other comments: we might want to do that in the future, but at the moment, uploading 50 release files with around 150MB every time we do a release is not within range. Furthermore, not all package authors want to upload their packages to PyPI. And this is _exactly_ the problem. If you are a package author and want to make your packages available to the public through PyPI, you should be obligated for publishing the related distribution files on PyPI: for the sake of availability and in order for being independent of your own infrastructure. Otherwise I have the (arrogant) opinion: go away - if you are a package author and want to use PyPI: ensure that your software is available to everyone at any time. What about those package authors who host their package elsewhere for various reasons and *do* make sure that their infrastructure is available - even if PyPI is down ? I have the feeling that you had a problem with that one package you mentioned and the proposal was just a reaction to the associated anger with that. It's not fair to start policing all packages on PyPI just because of that one incident you had. PyPI is not a kindergarten - PyPI is an important resource for professional Python development. CPAN is better organized and more reliable for more than ten years than PyPI ever was. To be fair, CPAN has been around a lot longer than PyPI. Regarding reliability of PyPI: as you've probably seen, I'm taking that seriously and want to enhance the reliability of PyPI. Regarding PyPI being used as resource for professional development: the zc.buildout approach has taken that idea a bit far, IMHO. PyPI wasn't designed to be used by automated download and installation tools that install hundreds of packages as opposed to the few packages that users request manually via easy_install. It's good to see, that PyPI can still cope with that approach and pushing the data to the cloud and/or mirror servers will enhance that performance even more. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jun 17 2010) Python/Zope Consulting and Support ...http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ 2010-07-19: EuroPython 2010, Birmingham, UK31 days to go ::: Try our new mxODBC.Connect Python Database Interface for free ! eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig
Re: [Catalog-sig] [Proposal] Registered packages must provide the source code distribution on PyPI
Andreas Jung wrote: M.-A. Lemburg wrote: I guess it's better to tell the package authors about your use of their packages and offer them help in hosting their packages on more reliable infrastructures. If that doesn't solve your problem, it's likely better to either setup your own index to override the PyPI one (should be easy to do in zc.buildout and AFAIK at least Plone is already doing that), or you stop using the package and look for alternatives. Sorry - I disagree completely. As developer I am into developing software and not into building private infrastructure to get around the deficiencies of PyPI Well, we're trying to change those ... and the ignorance of some package maintainers caring about the needs of the developers using their packages. ... can't help with this, though. Package authors typically have a wide range of motivations to write and share software for others to use. They don't necessarily share your views or see a need to fulfill your particular requirements. If you do have a business requirement to rely on their packages, I'd suggest you'd ask those package authors for a support contract. That would likely help them adapt to your needs ;-) Back to your proposal: In your particular case, I don't see how the proposal would have helped you - under the proposal, the package would have been removed from the PyPI index, so either way, there would have been no working automatic access to the package download links. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jun 17 2010) Python/Zope Consulting and Support ...http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ 2010-07-19: EuroPython 2010, Birmingham, UK31 days to go ::: Try our new mxODBC.Connect Python Database Interface for free ! eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig
Re: [Catalog-sig] [Proposal] Registered packages must provide the source code distribution on PyPI
Kai Diefenbach wrote: Hi, On 2010-06-17 11:51:13 +0200, M.-A. Lemburg said: Back to your proposal: In your particular case, I don't see how the proposal would have helped you - under the proposal, the package would have been removed from the PyPI index, so either way, there would have been no working automatic access to the package download links. Why? Crap without source code distribution will never be published so no one can ever build a dependency on that. AJ: packages once released should be available at any time from a well-known location (PyPI) Problem solved. Please have a look at the package in question. The only problem with it is that the download URL registered on PyPI no longer works. It redirects to the download page where you can find the source distribution. Not much or a problem for a user searching for the archives. Only a problem for setuptools and zc.buildout that don't ship with enough AI to figure out :-) To get back to your argument: Crap *with* source code distribution would still get published, so people would still build dependencies on it. How does this solve the problem ? Note that Andreas wasn't talking about crappy software, he was only complaining about the fact that automatic downloads via setuptools sometimes don't work for some packages on PyPI. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jun 17 2010) Python/Zope Consulting and Support ...http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ 2010-07-19: EuroPython 2010, Birmingham, UK31 days to go ::: Try our new mxODBC.Connect Python Database Interface for free ! eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig
Re: [Catalog-sig] [Proposal] Registered packages must provide the source code distribution on PyPI
Patrick Gerken wrote: On Thu, Jun 17, 2010 at 12:47, M.-A. Lemburg m...@egenix.com wrote: Kai Diefenbach wrote: Hi, On 2010-06-17 11:51:13 +0200, M.-A. Lemburg said: Back to your proposal: In your particular case, I don't see how the proposal would have helped you - under the proposal, the package would have been removed from the PyPI index, so either way, there would have been no working automatic access to the package download links. Why? Crap without source code distribution will never be published so no one can ever build a dependency on that. AJ: packages once released should be available at any time from a well-known location (PyPI) Problem solved. Please have a look at the package in question. The only problem with it is that the download URL registered on PyPI no longer works. It redirects to the download page where you can find the source distribution. And thats exactly what Andreas' argument is targeting. Not much or a problem for a user searching for the archives. Only a problem for setuptools and zc.buildout that don't ship with enough AI to figure out :-) To get back to your argument: Crap *with* source code distribution would still get published, so people would still build dependencies on it. How does this solve the problem ? Not putting the source release on pypi is just one indicator of crappy software. I agree that this is not a crap indicator for commercial software. There is a big number of users using tools that download tools in an automated fashion from pypi, and it is a reasonable request that source once being published to be available forever. If I understand it correctly, you are against this proposal, that would have protected users of setuptools/distribute/zc.buildouts from problems due to python-openid, because it would disallow the publication of information about commercial packages on pypi? What I'm saying is that it's better to contact the package authors whose entries cause problems than to force some policy on all PyPI package entries which carelessly puts packages that are not hosted on PyPI into the same category as crappy software. I see a point in that, but what is more important, having a catalog to browse or having a reliable repository of software to download? As a plone user who uses zc.buildout I very much prefer reliable downloads. Its not fun to search for the reason a supposedly repeatable buildout suddenly fails because a company decided to rename itself. It is well possible to delete package listings on PyPI. Wouldn't you rather be informed about this by way of an error report in zc.buildout than by finding that the package name has changed a few years later ? How about only listing packages with provided source code on the simple interface? afaik buildout always uses that, so a package python-openid is visible in the end-user view, but not installable via buildout. That way nobody would ever have had created a dependency on it in the first place. If such external links are a problem for zc.buildout, why don't you add an option to zc.buildout that prevents using such packages ? This is well possible by checking the /simple index entry for links to package download files: http://pypi.python.org/simple/python-openid/ vs. http://pypi.python.org/simple/zc.buildout/ BTW: what are all those bug links doing on the zc.buildout index page ? They look a lot like a good possibility for injecting trojans. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jun 17 2010) Python/Zope Consulting and Support ...http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ 2010-07-19: EuroPython 2010, Birmingham, UK31 days to go ::: Try our new mxODBC.Connect Python Database Interface for free ! eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig
Re: [Catalog-sig] [Proposal] Registered packages must provide the source code distribution on PyPI
Andreas Jung wrote: Tres Seaver wrote: Note however that Andreas' proposal was to require that 'sdists' be uploaded. I personally won't use binary-only packages, but it has historically been true that PyPI was intended to support them, as well as to support registration of packages hosted offsite. Andreas' proposal doesn't address either of those cases. A more precise requirement would be: - upload the sdist if your package is open-source - upload the official distribution package if you are package is commercial Basically...upload everything that you would also keep on your own server as official distribution. We cannot force authors to do this. There may be other reasons why they can't upload such things to PyPI, e.g. crypto, trademark and copyright laws, or even corporate rules if the author is maintaining the package as part of his or her job. What we can do, is make it more attractive to upload distribution files to PyPI and also to make the whole find the right file to download and install story easy enough for automatic tools to not just give up. For that to work, we'd need to rethink the infrastructure a bit more, though: If more package authors start shipping egg files for the various Unix platforms as both UCS2 and UCS4 and for 3 or 4 different Python versions and keep those files around for several releases, we'll run into problems with having to mirror all those download files. We've been doing this for several years now and it's probably an extreme example, but just as reference: we have almost 6GB of Python archives up on our servers and that's just for ~10 packages. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jun 17 2010) Python/Zope Consulting and Support ...http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ 2010-07-19: EuroPython 2010, Birmingham, UK31 days to go ::: Try our new mxODBC.Connect Python Database Interface for free ! eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig
Re: [Catalog-sig] [Proposal] Registered packages must provide the source code distribution on PyPI
Benji York wrote: On Thu, Jun 17, 2010 at 7:40 AM, M.-A. Lemburg m...@egenix.com wrote: http://pypi.python.org/simple/zc.buildout/ BTW: what are all those bug links doing on the zc.buildout index page ? PyPI scrapes all the links from the long description; for many projects that includes a change log with links to fixed bugs. Isn't that dangerous ? AFAIK, setuptools would start opening all those URLs and might find download files which are not necessarily under full control of the author, e.g. anyone could add a comment to a bug report or wiki page with a link to an egg file on some rogue server. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jun 17 2010) Python/Zope Consulting and Support ...http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ 2010-07-19: EuroPython 2010, Birmingham, UK31 days to go ::: Try our new mxODBC.Connect Python Database Interface for free ! eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig
Re: [Catalog-sig] [Proposal] Registered packages must provide the source code distribution on PyPI
Andreas Jung wrote: M.-A. Lemburg wrote: Andreas Jung wrote: Tres Seaver wrote: Note however that Andreas' proposal was to require that 'sdists' be uploaded. I personally won't use binary-only packages, but it has historically been true that PyPI was intended to support them, as well as to support registration of packages hosted offsite. Andreas' proposal doesn't address either of those cases. A more precise requirement would be: - upload the sdist if your package is open-source - upload the official distribution package if you are package is commercial Basically...upload everything that you would also keep on your own server as official distribution. We cannot force authors to do this. There may be other reasons why they can't upload such things to PyPI, e.g. crypto, trademark and copyright laws, or even corporate rules if the author is maintaining the package as part of his or her job. You are once again talking about edge cases. In general the majority of all externally hosted packages are not affected by such issues and should be hosted on PyPI. Well, there's certainly some reason why the authors chose not to host on PyPI. I can only list a few. If more package authors start shipping egg files for the various Unix platforms as both UCS2 and UCS4 and for 3 or 4 different Python versions and keep those files around for several releases, we'll run into problems with having to mirror all those download files. There is in general zero need for uploading eggs for various Python versions if the module is Python only. I have seen packages with upload for Python 2.2, 2.3, 2.4, 2.5, 2.6, 2.7, 3.0, 3.1 for Python-only packages. This is really nonsense...a single sdist is usally good enough...I bring it to the point: a bunch of Python developer have no idea about package hygiene and use PyPI as package toilet. If you ship Python-only packages with precompiled .pyc/.pyo files, you do need to upload one version per Python version. The marshal format and pyc magic often changes between releases. Some developers probably don't know that if they switch off the pyc compilation step, they'd get a single .egg file for all Python versions they support. In that case, we'd need to educate them, not call them names. If you want more people to upload and host their packages on PyPI, you have to: * make PyPI itself more robust and stable (we're working on that) * improve the tools to make both uploads and downloads easier (perhaps you could help with this) * convince people that their code is in good hands on PyPI (we'd need to get the PyPI terms straightened to help with this part) Suggesting that they can never remove a release from PyPI or are not allowed to rename a package is not going to attract more developers to PyPI. Calling them names, suggesting that their software is crap or that they use PyPI as dump, isn't going to attract anyone either. Anyway, I think I've said everything I wanted to say about this. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jun 17 2010) Python/Zope Consulting and Support ...http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ 2010-07-19: EuroPython 2010, Birmingham, UK31 days to go ::: Try our new mxODBC.Connect Python Database Interface for free ! eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig
Re: [Catalog-sig] PyPI template improvements
Simon de Vlieger wrote: On 17 jun 2010, at 22:44, Martin v. Löwis wrote: In web app land, supported browsers usually means the ones the designer targets: e.g., including IE= 7 in the list means that the designer doesn't have to include workarounds for stupid glitches in earlier IEs (or even test the design against those versions). For CSS, this means that the site's appearance will be sometimes wonky when running with an older-than-supported browser version. Features which depend on Javascript may not work at all, or only in degraded mode. I have a really hard time answering that question then: there was no web designer involved in creating PyPI (*). The browser that the *authors* of the service target are really the ones I mentioned: all of them. There is one browser that gets special attention, and flaws relating to it get fixed faster than for any other browser: setuptools. Regards, Martin (*) of course, it uses the layout of python.org, which did have a web designer; for this design, I don't know the answer. Martin, a question from me. Does setuptools browse the main pypi pages or does it use the simple version? setuptools used to parse the main web pages of PyPI. This was then changed and the /simple index invented. All recent versions of setuptools default to using the /simple index. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jun 18 2010) Python/Zope Consulting and Support ...http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ 2010-07-19: EuroPython 2010, Birmingham, UK30 days to go ::: Try our new mxODBC.Connect Python Database Interface for free ! eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig
Re: [Catalog-sig] Extra links on the PyPI /simple index package pages
P.J. Eby wrote: At 11:10 AM 6/18/2010 +0200, M.-A. Lemburg wrote: Martin v. Löwis wrote: Am 17.06.2010 15:16, schrieb M.-A. Lemburg: Benji York wrote: On Thu, Jun 17, 2010 at 7:40 AM, M.-A. Lemburgm...@egenix.com wrote: http://pypi.python.org/simple/zc.buildout/ BTW: what are all those bug links doing on the zc.buildout index page ? PyPI scrapes all the links from the long description; for many projects that includes a change log with links to fixed bugs. Isn't that dangerous ? AFAIK, setuptools would start opening all those URLs and might find download files which are not necessarily under full control of the author, e.g. anyone could add a comment to a bug report or wiki page with a link to an egg file on some rogue server. I think you misunderstand. Links originate *only* from the long description. The package owner has full control over that. I was referring to the linked assets that the package owner may not have full control over, e.g. in the above case, you have links pointing to launchpad and one to file://. Such links (except the file:// one) can be useful in the package description, e.g. to point to a bug tracking system, documentation or other resources, but they are not really needed to point setuptools to download locations. This is a misunderstanding of what setuptools does. Setuptools only retrieves URLs that are *specifically designated* as a home page or download link (using the rel field of the A tag on the PyPI /simple page), or which are a recognizable download URL supplied by way of the long_description. So, the risk you are describing does not actually exist. If you think the package owner is opening up a security threat by including the links in the first place - yes, that's indeed a risk. Is this feature still needed for setuptools ? Yes. We have download URLs and homepage URLs which should be enough for setuptools to search and find the links to package download files. No. This would only be the case if the project's author had some other form of hosting. For example, if you had a subversion repository for your development trunk, but didn't have any place to host an HTML page to link to it, the long_description would be the only way (AFAIK at present) for you to securely provide a link to that repository for setuptools (or humans) to use. The author could setup the home page or download URL to point to that repository (SVN makes the repos available as HTML pages as well). See also: http://peak.telecommunity.com/DevCenter/setuptools#making-your-package-available-for-easyinstall and: http://peak.telecommunity.com/DevCenter/PackageIndexAPI for more information on how the link parsing and retrieval works. It is a common misconception that setuptools spiders pages for links; the truth is, it only reads the home and download URLs provided via the PyPI metadata, and those only if they're not obviously links to a package tarball (or zip, egg, etc.). All other links must visibly point to something downloadable, or else they're ignored. So in summary, the /simple index page doesn't need to include any URLs from the long_description that do not have a rel attribute set, or end with one of the fixed set of archive extensions or with #egg= -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jun 21 2010) Python/Zope Consulting and Support ...http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ 2010-07-19: EuroPython 2010, Birmingham, UK27 days to go ::: Try our new mxODBC.Connect Python Database Interface for free ! eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig
[Catalog-sig] Proposal: Move PyPI static data to the cloud for better availability (version 2)
After the discussions, we've had on the catalog sig, I have updated the proposal to include comments and clarifications regarding the setup and it's relationship to the mirror PEP (see the end of the proposal). While I don't think that the proposal has an influence on whether or when PEP 381 gets rolled out or not, I will delay the PSF board vote on the proposal until the August board meeting. Perhaps that will even encourage developers to put more time on PEP 381. Regarding the costs of the cloud idea, I think this would actually be a good way of getting more donations for the PSF - possibly even with a net win. It's one of the few visible and tangible things the PSF has to offer to the community. Overall, I think this is a net win for everybody: users, developers and the PSF. PSF-Proposal: 100 Title: Move PyPI static data to the cloud for better availability Version: 2 Last-Modified: 2010-06-29 Author: m...@lemburg.com (Marc-André Lemburg) Discussions-To: catalog-sig@python.org Status: Draft Type: Informational Created: 2010-06-14 Post-History: Proposal: Move PyPI static data to the cloud for better availability Motivation -- PyPI has in recent months seen several outages with the index not being unavailable to both users using the web GUI interface as well as package administration tools such as easy_install from setuptools. As more and more Python applications rely on tools such as easy_install for direct installation, or zc.buildout to manage the complete software configuration cycle, the PyPI infrastructure receives more and more attention from the Python community. While we don't have hard numbers available (there doesn't appear to be any monitoring in place), the number of discussions about PyPI outtages in the mailing lists has increased to a point where we cannot simply ignore those complaints anymore. In order to maintain its credibility as software repository, to support the many different projects relying on the PyPI infrastructure and the many users who rely on the simplified installation process enabled by PyPI, the PSF needs to take action and move the essential parts of PyPI to a more robust infrastructur that provides: * scalability * 24/7 outsourced system administration management * redundant storage * geo-localized fast and reliable access Current Situation - PyPI is currently run from a single server hosted in The Netherlands (ximinez.python.org). This server is run by a very small team of sys admin. PyPI itself has in recent months been mostly maintained by one developer: Martin von Loewis. Projects are underway to enhance PyPI in various ways, including a proposal to add external mirroring (PEP 381), but these are still a long way from being finalized and implemented in the existing client tools. According to Martin, the server side features of PEP 381, including a few undocumented extensions to provide package signatures, are already implemented. However, without client tools to make use of them, this is not going to change the current situation for existing PyPI users. Furthermore those client tools enhancements would first have to get adopted by PyPI users by either replacing their client tools with updated versions or switching to new client tools, which is likely going to take months to years. Existing client tool users won't see an immediate improvement. Usage - PyPI provides four different mechanisms for accessing the stored information: * a web GUI that is meant for use by humans * an RPC interface which is mostly used for uploading new content * a semi-static /simple package listing, used by setuptools * a static area /packages for package download files and documentation, used by both the web GUI and setuptools The /simple package listing is dump of all packages in PyPI using a simple HTML page with links to sub-pages for each package. These sub-pages provide links to download files and external references. External tools like easy_install only use the /simple package listing together with the hosted package download files. While the /simple package listing is currently dynamically created from the database in real-time, this is not really needed for normal operation. A static copy created every 10-20 minutes would provide the same level of service in much the same way. Moving static data to a CDN --- Under the proposal the static information stored in PyPI (meta-information as well as package download files and documentation) is moved to a content delivery network (CDN). For this purpose, the /simple package listing is replaced with a static copy that is recreated every 10-20 minutes using a cronjob on the PyPI server. At the same intervals, another script will scan the package and documentation files under /packages for updates and upload any changes to the CDN for neartime availability. By using a
Re: [Catalog-sig] Recent PyPI changes
Alexis Metaireau wrote: On Tue, 2010-07-27 at 09:52 +0100, Chris Withers wrote: there is now a way to request release information in JSON, see http://tinyurl.com/38lefsp That's indeed cool, and useful, but we can't rely on this while crawling, too bad this JSON is not replicated on the mirrors. It could help a lot, since there is currently no way to request the metadatas statically in others way that downloading the distribution archives and extracting them. (we also could use xmlrpc, but that's not static). What's the process I have to follow in order to get this mirrored ? Does that sounds good for you ? IOW, whats needed to have this as a requirements for mirrors? Easiest would be to dump the complete release information (PKG-INFO) to a text file using the name format version_pkg_info in the simple/ index. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jul 27 2010) Python/Zope Consulting and Support ...http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ ::: Try our new mxODBC.Connect Python Database Interface for free ! eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig
Re: [Catalog-sig] Mirror list detection/construction - PEP 381
Martin v. Löwis wrote: Thoughts? I've been thinking that *.pypi.python.org should always yield A records, not CNAMEs. +1 It may be that this becomes difficult with Google appengine, though. There's no rule without exception :-) -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jul 27 2010) Python/Zope Consulting and Support ...http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ ::: Try our new mxODBC.Connect Python Database Interface for free ! eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig
Re: [Catalog-sig] PyPI reverse download
Martin v. Löwis wrote: Am 27.07.2010 22:46, schrieb M.-A. Lemburg: Martin v. Löwis wrote: I'll be implementing a feature for PyPI where you can POST to a certain action (revdownload), and then PyPI will POST the file requested to an URL that was passed; this is need to make blobs work on AppEngine. Any objections? Could you provide more detail on how this would work and why this is needed for AppEngine ? Ok, here is the long story. First, I tried to use the approach of pypione, using blobs for distributions. That won't work because blobs are limited to 1MB. Then I tried using lists of blobs instead. That won't work because the HTTP response size in urlfetch is limited to 10GB. Then I tried using Range: headers to mirror large files in pieces. That won't work because I then wouldn't be able to serve the files to setuptools, unless that would also start to use Range: headers. The only way to serve files larger than 10GB is the blobstore. However, apps can neither read from nor write to the blobstore. The only way to read from it is to serve the file, and the only way to write to it is through a POST from the outside. BTW, Google has kindly granted the app access to the blobstore (which is a for-fees feature only), and also kindly increased the store quota (which is 1GB in the free service, when a PyPI mirror needs about 15GB). Thanks for the details. One aspect I still don't understand is why you'd want to upload the whole PyPI mirror image in one go. Wouldn't it be better to just upload the distribution files separately ? (I don't think any of those is more than a few 10MB in size) Another aspect I don't (yet) understand is why these uploads would have to be initiated from outside the main PyPI server. I suspect that you want to use this feature to sync an AppEngine mirror with the data on the main server. For that, you'd only need to be able to upload data from that one server to the AppEngine blobstore. This should be possible without any external request to a PyPI RPC interface, simply via a script run via a cronjob or perhaps triggered by a new distribution file upload. The Amazon Cloudfront mirror would essentially work in the same way. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jul 27 2010) Python/Zope Consulting and Support ...http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ ::: Try our new mxODBC.Connect Python Database Interface for free ! eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig
Re: [Catalog-sig] PEP 345 Update
Martin v. Löwis wrote: 1/ The obsolete field could be used to say this is the new version of X, like the name of the project has changed. So the new version obsoletes the old one. Using this field, I think that the installers should remove the old releases (by prompting the user). -1. The installer should only take such action for new versions of the same package, not for packages that declare themselves replacements for others. In general, I think the two fields obsoletes and conflicts should only be used in an informational way. I'm not even sure whether it's a good idea to add them at all, due to the possibly negative effect of having package owners abuse these fields to push their particular variant of providing a solution to an application space. In case of obsoletes, it _should_ also be possible to install both of them simultaneously. Maybe some other other distribution depends on the original one, and can't work with the new one. Agreed. The same is true for conflicting packages: even if packages A and D conflict, one application may use the package combination A,B,C while another may be using D,E,F - both without any conflicts. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Aug 23 2010) Python/Zope Consulting and Support ...http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ ::: Try our new mxODBC.Connect Python Database Interface for free ! eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig
Re: [Catalog-sig] PEP 345 Update
Tarek Ziadé wrote: On Mon, Aug 23, 2010 at 1:24 PM, M.-A. Lemburg m...@egenix.com wrote: .. In case of obsoletes, it _should_ also be possible to install both of them simultaneously. Maybe some other other distribution depends on the original one, and can't work with the new one. Agreed. The same is true for conflicting packages: even if packages A and D conflict, one application may use the package combination A,B,C while another may be using D,E,F - both without any conflicts. But we are in the same scope. This can be applied for Conflicts because they are supposed to be incompatible. It's exactly the same incompatibility than having Foo depending on Bar 1.0 and Baz depending on Bar 0.9. Since you can't have two versions of Bar running in the same Python packages. So while Obsoletes allows you to have two times the same package. Conflicts indicates that its strictly impossible. Please see my example: It is well possible that you have two sets of packages installed which are mutually exclusive, but don't interfere which each other when used by different applications. If an installer were to bar you from installing packages from set 1 if you already have packages from set 2 installed, you wouldn't be able to use those two applications at the same time, even though they don't use a mixed combination of packages from set 1 and 2 (which would conflict). Whether there is a conflict depends on the applications using the packages. The packages themselves are not in conflict. Think of having both Zope and Django installed: one could regard this as conflict, since both provide web server functionality, but in reality they can both coexist in site-packages without any problem. All the other cases (packages using the same module/package name or version dependency conflicts) can easily be figured out by the installer without such an extra conflicts field. Could you perhaps point me to a list of things you had in mind with the conflicts field ? Perhaps I'm just missing the main idea behind this field. I do see a problem with such a general purpose conflicts field, regardless of what the use case is. If the installer prevents packages from being installed by means of defining such a conflicts field, this can be used to externally make package sets incompatible, e.g. a package of framework 1 could effectively prevent the installation of a competing framework 2, even though they both can live happily side-by-side on sys.path. And there's nothing the authors of framework 2 could do about this. In the same line of thought, I believe it would be better to change the obsoletes field to obsoleted-by, i.e. the direction changes and gives the author of the obsoleted package full control over what he thinks is a better version of his package, rather than have some other author make that choice for him. That way you avoid the social implications of one package trying to compete against another by means of the obsoletes field. In summary: As PyPI grows and we are seeing the first conflicts between package authors trying to push their particular idea of implementing a certain task, we should take such social implications of standards into account and, if possible, design the standards in a way that doesn't encourage such behavior. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Aug 24 2010) Python/Zope Consulting and Support ...http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ ::: Try our new mxODBC.Connect Python Database Interface for free ! eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig
Re: [Catalog-sig] PEP 345 Update
Alexis Métaireau wrote: Le 08/24/2010 05:12 PM, M.-A. Lemburg a écrit : I do see a problem with such a general purpose conflicts field, regardless of what the use case is. If the installer prevents packages from being installed by means of defining such a conflicts field, this can be used to externally make package sets incompatible, e.g. a package of framework 1 could effectively prevent the installation of a competing framework 2, even though they both can live happily side-by-side on sys.path. And there's nothing the authors of framework 2 could do about this. of course, we can force the installation in installers with --force, in such a case, but that's not why the conflict field is. Eg. If both can leave together, they're not conflicting :) I guess that's where part of the misunderstanding originates: the type of conflict can be many things and may well only show up in certain applications using the packages, but not necessarily all of them. At the very least, there should be an extra field Conflict-Description which describes the type of conflict to the user and then lets him or her decide whether the conflict is a problem or not for the intended use. I have a feeling that the introduction of this field originated from some application space problem that you're trying to fix at the package level. Also note that a the use cases you and others have mentioned can easily be solved using meta-packages and dependencies on these or by the installer checking whether there are module name or file conflicts (i.e. two packages wanting to install the same module or create the same file). -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Aug 25 2010) Python/Zope Consulting and Support ...http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ ::: Try our new mxODBC.Connect Python Database Interface for free ! eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig
Re: [Catalog-sig] [Python-Dev] egg_info in PyPI
Martin v. Löwis wrote: So you are fine with publishing slightly incorrect metadata at PyPI ? I am not. I really have no intuition for in how many cases the data will be incorrect. However, if users find that the data is incorrect for specific package, they ought to complain to the maintainer. I don't understand the rational behind providing flawed data at PyPI. I am -1 on that. -1 sounds much better than vetoing it :-) Taking the votes together, I currently arrive at Tarek -1 Michael -1 (?) Jannis -1 (?) Jim +1 Thomas +1 Any other opinions? -1 as well. The formats and file-names are just a complete mess and not all packages on PyPI are available as eggs or compatible to setuptools. I think the information from PEP 345 which is already made available via the XML-RPC .release_data() interface is sufficient for most cases. In those cases where it is not, we should extend the meta data. http://www.python.org/dev/peps/pep-0345/ http://wiki.python.org/moin/PyPiXmlRpc It may be worthwhile providing the same sort of information as static PKG-INFO file alongside the /simple index entries as well (e.g. via a link on the package page), to make it possible to extract the meta information without having to use the RPC mechanism and making use of the mirror infrastructure. http://www.python.org/dev/peps/pep-0314/ http://www.python.org/dev/peps/pep-0390/ PS: Could you put your DZUG talk up online somewhere. It includes some very valuable information on the various available APIs that is difficult to find elswhere. Thanks, -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Sep 18 2010) Python/Zope Consulting and Support ...http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ ::: Try our new mxODBC.Connect Python Database Interface for free ! eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig
Re: [Catalog-sig] egg_info in PyPI
Martin v. Löwis wrote: IMHO, most of this has been resolved by means of the XML-RPC interface via the .release_data() method: http://wiki.python.org/moin/PyPiXmlRpc If there's something missing, we should add it there. Very clearly, most of egg-info is missing there. However, I sense that people are very opposed to adding it there. I merely asked whether it was ok to expose the egg-info data at all, and was met with strong opposition - nobody actually asked what specific API to expose them I had in mind. Well, most depends on what you view as really important meta-data. Here's an example EGG-INFO dir: dependency_links.txt This file lists extra links to search for dependencies. This could be added to the meta-data. native_libs.txt This is not really needed, since I think we all agree that keeping eggs zipped up when installing them is not a good idea. not-zip-safe Ditto. PKG-INFO This is basically what we already have in .release_data(). SOURCES.txt I don't know the purpose of this file. It's a manifest of source files. The setuptools docs say it's not used anywhere. top_level.txt This file just lists the top-level package names made available by the package. We could add a meta-data field for this information or use the Provides: field for it. namespace_packages.txt This file lists the namespace packages used by the package. We could add a meta-data field for this or just use the Requires: field for it. Additionally, it would make sense to have this data around as static file for the PyPI mirrors to pick up. That's how I would have implemented it. Alas, this very idea was just shot down. I think most of the response was targeted at the EGG-INFO meta-data format, not the information itself. If you look at the above mess of upper- and lower-case filenames, files which only serve as marker and others having actual content, different content types, etc. I think it's clear that using this particular format is not a good idea. We can do better :-) -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Sep 20 2010) Python/Zope Consulting and Support ...http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ ::: Try our new mxODBC.Connect Python Database Interface for free ! eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig
Re: [Catalog-sig] egg_info in PyPI
Tarek Ziadé wrote:
On Mon, Sep 20, 2010 at 9:26 PM, Martin v. Löwis mar...@v.loewis.de wrote:
IMHO, most of this has been resolved by means of the XML-RPC
interface via the .release_data() method:
http://wiki.python.org/moin/PyPiXmlRpc
If there's something missing, we should add it there.
Very clearly, most of egg-info is missing there. However, I
sense that people are very opposed to adding it there.
I merely asked whether it was ok to expose the egg-info
data at all, and was met with strong opposition - nobody
actually asked what specific API to expose them I had in
mind.
If the use case is to provide a way to build a dependency graph just
by querying PyPI, that's exactly what we have in mind in pep 345.
I think I lost you there... .release_data() does provide the
PEP 345 meta-data, at least according to the documentation.
Whether PyPI actually has the data depends on how the release
was registered with PyPI, of course.
BTW: Something in the RPC interface or the distutils
register command is not really up to speed yet, since the
data appears to be a bit mangled at the moment (key and value
are reversed for a few entries):
pprint.pprint(client.release_data('egenix-mxodbc', '3.1.0'))
{'3.1.0': 'cheesecake_documentation_id',
'Windows,Linux,FreeBSD,Solaris,Mac OS X,AIX': 'version',
'_pypi_hidden': Boolean False at 933488,
'_pypi_ordering': 15,
'classifiers': ['Development Status :: 5 - Production/Stable',
'Development Status :: 6 - Mature',
'Environment :: Console',
'Environment :: No Input/Output (Daemon)',
'Intended Audience :: Developers',
'License :: Other/Proprietary License',
'Natural Language :: English',
'Operating System :: MacOS',
'Operating System :: Microsoft :: Windows',
'Operating System :: OS Independent',
'Operating System :: Other OS',
'Operating System :: POSIX',
'Operating System :: Unix',
'Programming Language :: C',
'Programming Language :: Python',
'Programming Language :: Python :: 2',
'Programming Language :: Python :: 2.3',
'Programming Language :: Python :: 2.4',
'Programming Language :: Python :: 2.5',
'Programming Language :: Python :: 2.6',
'Programming Language :: Python :: 2.7',
'Topic :: Communications',
'Topic :: Database',
'Topic :: Database :: Database Engines/Servers',
'Topic :: Database :: Front-Ends',
'Topic :: Documentation',
'Topic :: Internet',
'Topic :: Office/Business',
'Topic :: Software Development',
'Topic :: Software Development :: Libraries',
'Topic :: Software Development :: Libraries :: Application
Frameworks',
'Topic :: Software Development :: Libraries :: Python Modules',
'Topic :: Utilities'],
'description': 'The mxODBC Database Interface for Python is a commercial
add-on to our\nopen-source
eGenix mx Extension Series, providing robust and proven database\naccess for
Python on all major
platforms, such as Windows, Linux, Mac OS X, \nFreeBSD, Solaris, using a single
API.\n\nmxODBC works
with Python 2.3 - 2.7 and supports 32-bit as well as\n64-bit
platforms.\n\nPlease contact
sa...@egenix.com for evaluation licenses or visit the\neGenix.com online-shop
to purchase licenses
at http://www.egenix.com/.\n\nThis software is brought to you by eGenix.com and
distributed
under\nthe eGenix.com Commercial License 1.3.0.',
'eGenix.com GmbH': 'author_email',
'home_page': 'http://www.egenix.com/products/python/mxODBC/',
'http://pypi.python.org/pypi/egenix-mxodbc': 'author',
'http://www.egenix.com/products/python/mxODBC/': 'platform',
'i...@egenix.com': 'download_url',
'keywords': 'package_url',
'license': 'eGenix.com Commercial License 1.3.0; Copyright (c) 1997-2000,
Marc-Andre Lemburg, All
Rights Reserved; Copyright (c) 2000-2010, eGenix.com Software GmbH, All Rights
Reserved',
'maintainer': 'requires_python',
'maintainer_email': 'cheesecake_code_kwalitee_id',
'name': 'egenix-mxodbc',
'release_url': 'http://pypi.python.org/pypi/egenix-mxodbc/3.1.0',
'stable_version': 'cheesecake_installability_id',
'summary': 'eGenix mxODBC - ODBC Database Interface for Python'}
--
Marc-Andre Lemburg
eGenix.com
Professional Python Services directly from the Source (#1, Sep 20 2010)
Python/Zope Consulting and Support ...http://www.egenix.com/
mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/
mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/
::: Try our new mxODBC.Connect Python
Re: [Catalog-sig] Updating a package
wander.lairson wrote: 2011/1/4 Andreas Jung li...@zopyx.com: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Huh? http://pypi.python.org/pypi/pyusb/0.4.2 lists an email and http://sourceforge.net/users/wander_lairson has a See me a message link. In addition: this project has a mailing list. - -aj I think my last email was a bit confused (or I could not undertand yours). When I try to submit pyusb 1.0 to PyPi, I got an error, because the maintainer in the PyPi is not me, it feels like the maintainer is an user called reid, I didn't find an email to contact him, this is the source of trouble. If I am doing some mistake, please let me know, I am a newbie in PyPi... You will have to file a support request to get the PyPI owner of the package changed. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jan 04 2011) Python/Zope Consulting and Support ...http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ ::: Try our new mxODBC.Connect Python Database Interface for free ! eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig
Re: [Catalog-sig] API search by python version (or classifier)
Tarek Ziadé wrote: On Wed, Jan 26, 2011 at 3:36 AM, Brian Jones bkjo...@gmail.com wrote: ... http://mail.python.org/pipermail/catalog-sig/2010-September/003301.html I'm not positive we're on the same page. You seem to be talking about adding some specific new data in a specific way to the index, and *then* making it available. I'm talking about making the *current* data available to programs the same way it's available via a web interface. It seems your ideas were shot down because it's coming, if I read it right. I myself am kind of confused about that, because my understanding is also that it's coming in the form of a tool that can only be used for Python 3 packages :/ No, the tool will be usable by python2 packages as well. I don't want to do that thread again but basically, egg_info are platform-specific, whereas the new metadata version (PEP 345) will have them right. PyPI already supports PEP 345. Does that mean that the XML-RPC interface .release_data(package_name, version) (see http://wiki.python.org/moin/PyPiXmlRpc for details) already returns the full set of PEP 345 meta data ? I don't really see any problem with adding those new fields to the .search() API, but perhaps I'm missing something. The discussion Martin mentioned was about egg-infos files or directories. It doesn't seem to relate to searching for meta data, since - as you say - egg-info is meta data for egg files that is individual build files, not releases. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jan 26 2011) Python/Zope Consulting and Support ...http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ ::: Try our new mxODBC.Connect Python Database Interface for free ! eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig
Re: [Catalog-sig] API search by python version (or classifier)
Tarek Ziadé wrote: Oh... so what about this: - PyPI publishes setuptools metadata, but in Metadata 1.2 format. The conversion is pretty simple, it's just translating the requires.txt into PEP 345 fields. - We add a marker so we know that those metadata are from setuptools That way, PyPI has an unified interface to get the dependencies, and if there's an issue in project X because the static metadata fails on platform Y, we ask the project to provide real PEP 345 metadata. +1 -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jan 27 2011) Python/Zope Consulting and Support ...http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ ::: Try our new mxODBC.Connect Python Database Interface for free ! eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig
Re: [Catalog-sig] Total number of packages in PyPI
Baiju M wrote: On Sun, Jan 30, 2011 at 11:48 PM, Martin v. Löwis mar...@v.loewis.de wrote: But many of the packages are not really existing in PyPI: http://paste.pocoo.org/raw/329292/ Is there any auto-generated list of these packages somewhere ? I don't know what a not really existing package is; For those package, there is no page with user visible information in pypi index but there is an empty simple index. For example, this page give 404: http://pypi.python.org/pypi/slapos.slapmonitor/ Whereas, this page gives an empty page: http://pypi.python.org/simple/slapos.slapmonitor/ Sounds like a bug in PyPI. Perhaps those are deleted packages that, for some reason, still get displayed on the simple/ index ?! E.g. http://pypi.python.org/simple/Webware/: Links for Webware 0.8 home_page 0.8 download_url 0.8.1 home_page 0.8.1 download_url http://pypi.python.org/pypi/Webware/: Not Found () but there is a package: http://pypi.python.org/pypi/Webware for Python/ -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jan 30 2011) Python/Zope Consulting and Support ...http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ ::: Try our new mxODBC.Connect Python Database Interface for free ! eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig
Re: [Catalog-sig] [Distutils] pypi/packages/docs.python.org
Martin v. Löwis wrote: As for a big link: if you think your page should have one, you are free to make it yourself already. Sure but, 1/ I have never asked for the Downloads ↓ link either, but the UI did add it, and it's really more ergonomic. 2/ I have never asked for Latest Version: 0.6.14 on this page : hit ttp://pypi.python.org/pypi/distribute/0.6.9 [...] And I think a link to packages.python.org/distribute is at the same level I can sympathize with that view. Cc'ing catalog-sig here: if anybody would *not* want to have a Documentation link automatically generated that points to packages.python.org/project, please speak up. That would only make sense if there's something uploaded to the PyPI docs dir. If PyPI can detect that, +1. Otherwise, I think it's better not adding such an automatic link, since no link is better than one going nowhere. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Mar 06 2011) Python/Zope Consulting and Support ...http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ ::: Try our new mxODBC.Connect Python Database Interface for free ! eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig
Re: [Catalog-sig] A simple printer of nested lists
Chris Withers wrote: Hi, Any chance we could put a block in place on packages that mention this phrase? Has anyone had any joy tracking down the author of the book that resulted in this avalanche of junk? At the very least they could publish an errata explaining their horrific mistake... I'm obviously missing some context here, but there are indeed quite a few copies of seemingly the same module up PyPI: http://pypi.python.org/pypi?:action=searchterm=simple+printer+of+nested+lists Is it possible that the book used the module to explain creating distutils packages and uploading them to PyPI ? -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Mar 24 2011) Python/Zope Consulting and Support ...http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ ::: Try our new mxODBC.Connect Python Database Interface for free ! eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig
Re: [Catalog-sig] A simple printer of nested lists
Chris Withers wrote: On 24/03/2011 08:58, Ronald Oussoren wrote: On 24 Mar, 2011, at 7:46, Chris Withers wrote: Hi, Any chance we could put a block in place on packages that mention this phrase? Has anyone had any joy tracking down the author of the book that resulted in this avalanche of junk? At the very least they could publish an errata explaining their horrific mistake... It might be this onehttp://my.safaribooksonline.com/book/programming/python/9781449397524/sharing-your-code-modules-of-functions/update_pypi_with_your_new_code#X2ludGVybmFsX0ZsYXNoUmVhZGVyP3htbGlkPTk3ODE0NDkzOTc1MjQvNjA= (I'm not a safari subscriber and cannot see the entire page because of that, but this page did turn up in a google search for the phrase plus the word book. Yup, I believe it's the one, and the publication date ties in with the start of the problem. Does anyone have contact details for Paul Barry? Google does... :-) http://www.oreillynet.com/pub/au/3677 -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Mar 24 2011) Python/Zope Consulting and Support ...http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ ::: Try our new mxODBC.Connect Python Database Interface for free ! eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig
Re: [Catalog-sig] Some minor issues with page content
Daniel Greenfeld wrote: Hey guys, I get a 404 when I go here which is off the package upload page: http://www.python.org/doc/dist/package-upload.html Also, when I go to the tutorial I see instructions pointing at easy_install. I thought the move was towards Pip. See http://wiki.python.org/moin/CheeseShopTutorial Feel free to add a new section on pip. AFAIK, pip only supports source-code packages (please correct if this is no longer the case), so easy_install is still the way to go for binary packages. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Apr 18 2011) Python/Zope Consulting and Support ...http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ ::: Try our new mxODBC.Connect Python Database Interface for free ! eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig
Re: [Catalog-sig] How about a dedicated web service mirror?
Daniel Greenfeld wrote: Here is something that we all might find useful. When Python Packages launches it will be hitting the XMLRPC server over 10,000 times a day - at least once per listed package. Django Packages already hits PyPI about 3000 times a day. We'll be doing this in a distributed task queu but the fact remains that the server is going to be hit a decent amount. Is that really necessary ? PyPI has a mechanism to subscribe to recent changes only, which should be enough for the needs of the sites you mention. Because of the relative ease of setting up the Packaginator tool these sites are building other people will also be using this same API. Or maybe they'll build their own system and do even more. So maybe dedicated mirrors that truncate the web UI and just provide the web service API? -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Apr 18 2011) Python/Zope Consulting and Support ...http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ ::: Try our new mxODBC.Connect Python Database Interface for free ! eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig
Re: [Catalog-sig] PyPI mirror key rollover
Martin v. Löwis wrote: Am 28.04.2011 10:26, schrieb M.-A. Lemburg: Martin v. Löwis wrote: I came up with a key rollover scheme for the server key on PyPI. [...] The key rollover will be logged in the PyPI journal, using an empty package name and an empty release. TOOLS USING THE JOURNAL MAY NEED TO BE FIXED TO ACCOMMODATE EMPTY PACKAGE NAMES. Earlier today, such a journal entry was already added; I took it out again when I noticed that some tools actually do need to be fixed. I can't comment on the other parts of the proposal, but the above suggestions doesn't sound like a good solution: an empty package name in the update stream looks more like a server or client decoding bug than a trigger to do a key update. Oops, I forgot a critical detail: the action string in the journal entry would be keyrotate. Ah, ok. Makes more sense then :-) Wouldn't it be better to use a descriptive package name such as pypi-serverkey-update together with a package version which identifies the new serverkey version as trigger ? That would not be good - tools would (rightly) assume that there is a package with that name, and try to mirror it. Well, you could create a package under that name which then contains a module with all known server keys. Might be useful to have for other purposes as well. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Apr 28 2011) Python/Zope Consulting and Support ...http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ 2011-06-20: EuroPython 2011, Florence, Italy 53 days to go ::: Try our new mxODBC.Connect Python Database Interface for free ! eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig
Re: [Catalog-sig] PyPI's external packages
Tres Seaver wrote: On 05/13/2011 01:12 PM, exar...@twistedmatrix.com wrote: On 03:21 pm, mer...@netwok.org wrote: Le 12/05/2011 21:04, exar...@twistedmatrix.com a ýcrit : On 03:57 pm, ziade.ta...@gmail.com wrote: [...] I'll definitely do something in distutils2 but maybe someone has a better idea ? Make it easier to upload packages to PyPI. For example, [...] make upload work even if the package files exist on the filesystem somewhere already. You 19re referring to the fact that 1Cupload 1D won 19t push files already present in dist/, right? More or less, yes. I don 19t know whether this was a design choice of Martin or merely done for code simplicity, but currently the command will only push files that are products of a command run from the same command line, e.g. 1Csdist upload 1D. If you run 1Csdist and then 1Csdist upload 1D, is the sdist recreated even though there have been no changes to the files? Yes, the sdist is recreated. If no, having to run 1Csdist upload 1D is not inefficient, and makes you be explicit about the files you want to push, which is IMO good. Do you agree? If yes, I think it 19s a bug. I'm not concerned about the efficiency, though. I'm concerned about the quality of the build. I want to generate a release, *test it*, and then upload it. Being forced to generate it and upload it at the same time gets in the way of this. ISTM that if you run 'python setup.py sdist' a second time and get a different result (without changing any included files), then you have a built-in quality problem *in your setup.py.* Just to clarify: creating an sdist really only means copying over the files from the MANIFEST into a temporary dir and then running tar or zip on the temporary directory. You can tell distutils to keep the temporary dir around by using the --keep-temp option on sdist. This will reduce the operation to just running the archiver on the temp dir again. The same can be done for bdist_* commands by having distutils skip the build process. Since I don't see how such trivial operations can affect the quality of the code, the discussion sounds a bit artificial :-) Note that if you want to do full testing of a release, you'll have to download the package from PyPI and then run the tests on the downloaded archive to be sure that things work - everything else is just preparation for this final run :-) -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, May 16 2011) Python/Zope Consulting and Support ...http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ 2011-06-20: EuroPython 2011, Florence, Italy 35 days to go ::: Try our new mxODBC.Connect Python Database Interface for free ! eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig
Re: [Catalog-sig] PyPI's external packages
John J Lee wrote: On Mon, 16 May 2011, M.-A. Lemburg wrote: [...] Just to clarify: creating an sdist really only means copying over the files from the MANIFEST into a temporary dir and then running tar or zip on the temporary directory. You can tell distutils to keep the temporary dir around by using the --keep-temp option on sdist. This will reduce the operation to just running the archiver on the temp dir again. The same can be done for bdist_* commands by having distutils skip the build process. Since I don't see how such trivial operations can affect the quality of the code, the discussion sounds a bit artificial :-) Perhaps a file gets left behind the second time somehow that wasn't present the first time, or vice versa (and MANIFEST includes that file). If the presence or absence of that file breaks stuff, you end up uploading a broken file. Doing your release in an automated way makes this less likely, but anybody who has maintained a complicated buildbot setup knows that this kind of thing does happen. Note that if you want to do full testing of a release, you'll have to download the package from PyPI and then run the tests on the downloaded archive to be sure that things work - everything else is just preparation for this final run :-) The upload step is interesting one from the PoV of testing because performing it implies this is the release, come and get it. There are other such steps (tagging, sending announcements), but this is one of them. So it's nice to be as confident as possible at this point that you didn't screw it up. Right, but the only way to find out is by downloading what you uploaded and testing that package :-) There are lots of things that can go wrong. In my experience, most of them are user errors, e.g. you forgot to update the MANIFEST file, bump the version, remove debug settings, etc. The chance of two consecutive runs of sdist creating different archives is rather small, compared to those sources of error. That said, it's easy to get the upload command to use an already created distribution file for the upload: just add a new distutils command which sets .distribution.dist_files to what list of files you want to upload. This could also be added as an option to the distutils upload command, so that you can run: python setup.py upload --dist-files=dist/my-release-1.2.3.* -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, May 16 2011) Python/Zope Consulting and Support ...http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ 2011-06-20: EuroPython 2011, Florence, Italy 35 days to go ::: Try our new mxODBC.Connect Python Database Interface for free ! eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig
Re: [Catalog-sig] PyPI's external packages
M.-A. Lemburg wrote: The chance of two consecutive runs of sdist creating different archives is rather small, compared to those sources of error. That said, it's easy to get the upload command to use an already created distribution file for the upload: just add a new distutils command which sets .distribution.dist_files to what list of files you want to upload. This could also be added as an option to the distutils upload command, so that you can run: python setup.py upload --dist-files=dist/my-release-1.2.3.* Here's a trick which will do the same without having to write any new code: First run: python setup.py sdist --keep-temp Then run your tests on the created archive locally. Second run: python setup.py sdist --dry-run upload The second run won't build a new archive, it'll just upload the already created archive to PyPI. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, May 16 2011) Python/Zope Consulting and Support ...http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ 2011-06-20: EuroPython 2011, Florence, Italy 35 days to go ::: Try our new mxODBC.Connect Python Database Interface for free ! eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig
Re: [Catalog-sig] PyPI's external packages
John J Lee wrote: On Mon, 16 May 2011, M.-A. Lemburg wrote: That said, it's easy to get the upload command to use an already created distribution file for the upload: just add a new distutils command which sets .distribution.dist_files to what list of files you want to upload. [...] Just add a new distutils command? ISTR distutils isn't very open to that? IWBNI if distutils (or some replacement for it -- distutils2?) supported this out of the box. Actually, extending distutils is a lot easier than many people tend to believe. Adding a new command or extending an existing one is done by subclassing from an existing command class and then registering this new class with distutils using the cmdclass keyword argument to setup(): http://docs.python.org/distutils/extending.html?highlight=cmdclass Adding an option to distutils' upload command class in Python 3.3 would work as well, of course, but takes a lot longer to become usable for package authors. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, May 16 2011) Python/Zope Consulting and Support ...http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ 2011-06-20: EuroPython 2011, Florence, Italy 35 days to go ::: Try our new mxODBC.Connect Python Database Interface for free ! eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig
Re: [Catalog-sig] PyPI looks down
Carl Meyer wrote: On 05/21/2011 01:58 PM, M.-A. Lemburg wrote: Carl Meyer wrote: On 05/21/2011 03:00 AM, Chris Withers wrote: It install packages with C extensions yet? Sure, I do it every day. You just need a compiler. ... and all the external dependencies such a package may have. pip really needs to support a binary format to be more user friendly in this respect in order to completely replace easy_install and make use of all the eggs on PyPI. I would like to see a resolution to http://bugs.python.org/issue8654 first; at the moment, I consider the use of binary packages on PyPI broken (particularly for Python 3, which pip now supports), because the finding mechanism can easily download an egg that doesn't work on your build of Python. eggs mostly work for pure-Python packages and packages for Windows. The situation for other platforms is less than ideal, mostly because setuptools mainly relies on the egg file name alone, which includes a non-canoncical platform name and doesn't respect UCS2/UCS4 builds. There are essentially two ways to fix this: 1. standardize the egg filenames in a way that lets pip recognize compatible egg files 2. add a mapping of egg filenames to a platform information dictionary which pip can download to select the right egg file Option 1: PEP 3149 - ABI version tagged .so files could help with this option, since it addresses the various Python build options. We'd then only have to define a list or function which allows checking the platform string embedded in the filename against the platform running pip. Such a function would be useful for other things as well, especially on platforms that often change the platform name like e.g. FreeBSD or Mac OS X and Solaris which add universal builds to make things even more interesting. Such a function would also be useful for other applications, e.g. ones that try to determine plugin compatibility or want to switch implementations based on the platform, but not necessarily depending on the platforms OS version or 32/64-bit variant. The good news is that it's easy to come up with egg filenames that easy_install does not recognize, so both egg filename schemes could live side by side for a while. Option 2: This would allow more freedom and is easier to extend in the future, but also requires more work, both on the PyPI side, the distutils side and the client tools side. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, May 26 2011) Python/Zope Consulting and Support ...http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ 2011-05-23: Released eGenix mx Base 3.2.0 http://python.egenix.com/ 2011-05-25: Released mxODBC 3.1.1 http://python.egenix.com/ 2011-06-20: EuroPython 2011, Florence, Italy 25 days to go ::: Try our new mxODBC.Connect Python Database Interface for free ! eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig
Re: [Catalog-sig] Is this spam?
Chris Withers wrote: This package: http://pypi.python.org/pypi/PDFTron%20PDFNet%20SDK%20for%20Python/5.7 ...feels a lot like spam. The mention of Python if you follow through to their page is pretty minimal. Not sure it belongs on PyPI. What do others think? Have you tried downloading their SDK ? http://www.pdftron.com/pdfnet/downloads.html It comes with a Python wrapper for their library, so it's legitimate. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jun 03 2011) Python/Zope Consulting and Support ...http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ 2011-05-23: Released eGenix mx Base 3.2.0 http://python.egenix.com/ 2011-05-25: Released mxODBC 3.1.1 http://python.egenix.com/ 2011-06-20: EuroPython 2011, Florence, Italy 17 days to go ::: Try our new mxODBC.Connect Python Database Interface for free ! eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig
Re: [Catalog-sig] Add link to secure connection to the PyPI front page
Martin v. Löwis wrote: Which makes me wonder, why is it that PyPI doesn't use a universally accepted SSL cert instead of the CAcert one? Note: I'm a CAcert assurer myself but would prefer using a cert by one of the commercial CAs for the sake of the users. Any opinions? Primarily because of lack of volunteer time. Buying a certificate is a big effort, issuing a cacert one is simple. And before anybody says no, it's not difficult, or no, it shouldn't be difficult, please consider volunteering for the next ten years to manage the PSF server certificates (as one of the key problems that makes it difficult is that responsibilities change so often with volunteers). Perhaps we could get Pat, the PSF secretary and administrator to deal with the paperwork that's needed to get a certificate. Installing it is not really such a major task, once you have the paperwork done. Should we take this to the PSF board for discussion ? -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jun 04 2011) Python/Zope Consulting and Support ...http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ 2011-05-23: Released eGenix mx Base 3.2.0 http://python.egenix.com/ 2011-05-25: Released mxODBC 3.1.1 http://python.egenix.com/ 2011-06-20: EuroPython 2011, Florence, Italy 16 days to go ::: Try our new mxODBC.Connect Python Database Interface for free ! eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig
