Re: [Distutils] Python people want CPAN and how the latter came about
sstein...@gmail.com wrote: Hmm ... I think you meant bikeshedding (energetic discussion of meaningless details) -- woodshedding *is* an actual slang term, but one that originated among musicians, meaning to practice one's skills alone (i.e., to go off to a woodshed where no one can hear you). Sometimes a woodshed in the country is the best place to write music. There's a special character that you get in the music that isn't possible living in the city. Country air.. David ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Python people want CPAN and how the latter came about
On Sat, Dec 26, 2009 at 05:03, David Cournapeau courn...@gmail.com wrote: I guess you meant upload. The reasons I see for making some things, in particular upload mandatory are as follows: - file upload makes pure rsync-based mirroring. In the case of CRAN, you mirror with one rsync command. No need for fancy scheme, new packages, etc... But you can already do that with the files that are uploaded. You claim that to mirror PyPI, you have to *also* include files that are *not* hosted on PyPI. That makes no sense. Sorry. - lack of tarballs/installers means that when you use an installer, for example easy_install, it has to find it in another way. Which is already does, do that's not a problem. This way is based on scraping I agree that it would be better if the package uploaders included a download URL in the metadata. This for some reason seems unusual. the website may be dead, not available anymore, etc... In that case, installation fails. This is true. I agree it would be better of people uploaded their packages to PyPI. But this is a difference in attitude, and it's a difference between forcing people to upload, and helping people. It really is what it comes down to. I tried to avoid saying it in such plain terms, but the message seems not to reach through. I think we should figure out *why* people are not uploading and then help them fix the reason this is not so. You say we should tell them to upload or bugger off. That will mean that people buggers off. That is still NOT an improvement, and it will NOT become an improvement because you repeat it more times. Requiring uploads will make PyPI *worse* not better and means it will have *less* packages, and when you can't even find a package via a download URL or scraping, well, then you can't find it automatically at all. I did not know it was even allowed to register non open source code in Pypi. That's the first time I see the argument given (and the first time a real argument is given). Is this a major requirement ? Since I have given exactly that argument: That licenses and similar prevents people from uploading some code, that is proof that you don't read my answers. You should. -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Python people want CPAN and how the latter came about
Lennart Regebro wrote: On Sat, Dec 26, 2009 at 05:03, David Cournapeau courn...@gmail.com wrote: The reasons I see for making some things, in particular upload mandatory are as follows: - file upload makes pure rsync-based mirroring. In the case of CRAN, you mirror with one rsync command. No need for fancy scheme, new packages, etc... But you can already do that with the files that are uploaded. You claim that to mirror PyPI, you have to *also* include files that are *not* hosted on PyPI. That makes no sense. Sorry. When some people say I want to mirror PyPI, I think they mean I want to have a private copy of all of the files I need to install a given set of packages I need. This is often a requirement because the computer(s) where an installation is occurring are behind firewalls without general Internet access or are production servers where administrative rules require that all installation repositories be locked down. (I'm not saying that David is using mirroring in this sense, but I know some people, including me, do use it that way.) Actually mirroring PyPI (modulo Martin's issues with accounts, etc.) can be achieved with rsync or other ways of just copying the files on PyPI. Producing a private PyPI with all of the packages you need for installation is less straight forward when all of the files are not uploaded to PyPI. I usually do it by trial and error, but I've not spent any time looking into automated solutions. Eric. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Python people want CPAN and how the latter came about
Hi Lennart, Lennart Regebro regebro at gmail.com writes: On Wed, Dec 23, 2009 at 10:32, S. Mueller wrote: I have to say that I vastly prefer not to have any authorization and allow anyone to release anything in any namespace. But then I am getting fanatically anarchical in these issues. You can not organize freedom. But you have to. This is clearly a case of citation rape. ;) No, I really mean it whan I say you can't. And you never *have* to do the impossible, and trying just leads to problems. I realize this is a matter of attitude, but if the sentence I want CPAN means I want restrictions and controls and people preventing others from uploading stuff, then they are misguided. Sorry, but I'm not being philosophical when I say you have to authorize access to things. Apparently the Python repository does, too. Or otherwise I'll upload a few popular packages with high version numbers that contain viruses for New Year. I'm not talking about restricting freedom. Now, on CPAN, I *can* upload anything even if not authorized to do so. It just won't be part of the official indexes if I upload a new version of the database interface DBI. What you're saying here means you virtually throw away the ability to do anything useful with namespace meta data. *shrugs* Namespaces has no metadata in Python. They are just namespaces, no more, no less. The names of your *packages* is protected on PyPI. But several people can use the same *namespace*. Ie, nobody can upload a Twisted package, except those who have the permission. But people can upload a zope.my.own.package, even though the namespace zope is already used by other packages. Same for CPAN. You automatically register an exact namespace by uploading a file that contains it. But you don't get it recursively. Please recall my explanation of how in Perl a namespace == package name == class name. If I upload Steffens::Module, somebody else can upload Steffens::Module::Plugin. Just Steffens::Module is restricted from the point on of the initial upload. That we do out meta data stuff on package/namespace/class names as opposed to distribution names has the huge benefit of interoperability between distributions. Think about it like this: If you install any module from CPAN (and only the valid ones end up in the index), you can use all of them in the same application. If module A and B could both implement Config::Parser, then you couldn't use both of them at the same time. This would be true for Python too. But Python doesn't try to pretend that all the packages that exist are some sort of standard library, and therefore don't try to put them all into one sort of hierarchical organized namespace. And to be honest I don't see the point of doing that. We're not pretending anything. We're not forcing anything except that you don't override somebody elses work. We advise on proper choice of namespaces. But in the end, we never force anybody to adhere to our preferences. By our, I mean an arbitrary bunch of experienced contributors who offer advice for new contributors. Most of those wouldn't even have the power to impose any restrictions. Those who do, use it only extremely sparely. For example, when somebody has passed away or simply asks for help in passing maintainership to someone else. Still. We allow for a lot of creative freedom. We just don't want a random newbie upload a shiny package DB which implements his idea of a database interface when it's really the name of the package that implements the Perl debugger. He can still upload. The file will be accessible in his CPAN directory. Users can install it from the CPAN shell as install NEWBIEUSERID/DB-1.00.tar.gz, but not as install DB or $ cpan DB. I see. But IMO Perl then there starts out with trying to organize freedom from the start, and that then leads to the above problem that newbies can come up and mess up this so called organized freedom, which means you need to restrict it even more by having people control and restrict the namespaces, etc, etc. You end up having to have more organisation to fix the problems your organisation caused. This is, without trying to be rude or anything, the fate of all bureaucracies. You're wrong. The Perl/PAUSE/CPAN system works exceptionally well. But it does so because we regulate a lot less than you think. Let me recap the major points once more: a) Anybody can upload anything (theoretically, even you wedding pictures) a1) If it's a virus and identified, it'll be deleted. a2) Anything else goes into CPAN. b) PAUSE extracts archives and checks whether it looks like a Perl module distribution. b1) If not, it's simply ignored (but still mirrored) b2) If so, it's scanned for Perl classes. (== namespaces == packages) c) The classes found are added to the official index. c1) This is the case if the classes have never been uploaded before. c2) Also, if *you* uploaded them before. c3) Or if the guy who first
Re: [Distutils] Python people want CPAN and how the latter came about
On Sat, Dec 26, 2009 at 11:17, Steffen Mueller smuel...@cpan.org wrote: This is clearly a case of citation rape. ;) Possibly, but I tried to extract the essence of the misunderstanding. Maybe I was boiling too hard. :) Sorry, but I'm not being philosophical when I say you have to authorize access to things. But the discussion was not things. The discussion was specifically *namespaces*. Same for CPAN. You automatically register an exact namespace by uploading a file that contains it. But you don't get it recursively. Please recall my explanation of how in Perl a namespace == package name == class name. It isn't in Python. That we do out meta data stuff on package/namespace/class names as opposed to distribution names has the huge benefit of interoperability between distributions. What problems would that be? We're not pretending anything. We're not forcing anything except that you don't override somebody elses work. We advise on proper choice of namespaces. But in the end, we never force anybody to adhere to our preferences. By our, I mean an arbitrary bunch of experienced contributors who offer advice for new contributors. Most of those wouldn't even have the power to impose any restrictions. Those who do, use it only extremely sparely. For example, when somebody has passed away or simply asks for help in passing maintainership to someone else. Then there simply is no difference between PyPI and CPAN in this regard. You're wrong. The Perl/PAUSE/CPAN system works exceptionally well. But it does so because we regulate a lot less than you think. The question then becomes why it is claimed that PyPI needs to regulate *more* when it's pretty obvious once the confusion has cleared that the only regulation on both systems is that you can't upload a new version of somebody else's package. It's just that you call packages namespaces and we don't. (Or well, on CPAN it canbe uploaded, but it won't be visible, in PyPI it can't be uploaded). This is a safeguard against insanity and it's the thing that means that you can trust cpan PAR to always install the Perl Archive Toolkit that was released by Autrijus Tang, Roderich Schupp, or myself (we share co-maintenance). It's never going to be some random junk. And that you auto-register a namespace on upload is the guarantee. And obviously on PyPI, it's first come first serve as well. But nobody would call a db package db if one already exists. Why would they do that? What's the point? Why would I make a completely new package called Twisted for example? There already is one. It's just a mindset that is completely incomprehensible to me. Then you clearly do not understand what it is like to be a) malicious b) new, young, inexperienced c) stupid. So? What does that have to do with anything? Why would PyPI need more regulation/organisation to handle that? Especially if CPAN doesn't? As far as I can see, this is another case of: Python needs CPAN! We have it already. :-) -- Lennart Regebro: http://regebro.wordpress.com/ Python 3 Porting: http://python-incompatibility.googlecode.com/ +33 661 58 14 64 ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Python people want CPAN and how the latter came about
On Sat, Dec 26, 2009 at 11:13, Eric Smith e...@trueblade.com wrote: When some people say I want to mirror PyPI, I think they mean I want to have a private copy of all of the files I need to install a given set of packages I need. This is often a requirement because the computer(s) where an installation is occurring are behind firewalls without general Internet access or are production servers where administrative rules require that all installation repositories be locked down. Right, setting up your *own package index* is not *mirroring*. You do not need to mirror all of PyPI, and you do not need to mirror the PyPI metadata do do this. And the argument of file uploads has been used for mirroring as well, namely for third-party services. And that argument still does not hold water. Setting up package indexes has benefits and there are as mentioned several different ways, one being a mirrored package index, ie caching the other setting up special indexes for different versions of the software etc. Yes, it's correct, if people would just add download URLs or upload the packages on PyPI, downloading all the files to make your own package index would be somewhat easier. Is this really a major difference to CPAN? If people don't upload a package to CPAN, what do you do? Does this get magically resolved somehow? I don't see how that would work. * If you only use packages on PyPI, it's easy. * If you only use packages on CPAN, it's easy. * If you use packages not on PyPI, it's more complicated. * If you use packages not on CPAN, it's more complicated. Seems to be the same to me. And it ends with the same conclusion: We must figure WHY people don't upload to PyPI and FIX THEIR PROBLEM. Saying Upload or bugger off does NOT solve this problem. -- Lennart Regebro: http://regebro.wordpress.com/ Python 3 Porting: http://python-incompatibility.googlecode.com/ +33 661 58 14 64 ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Python people want CPAN and how the latter came about
When some people say I want to mirror PyPI, I think they mean I want to have a private copy of all of the files I need to install a given set of packages I need. This is often a requirement because the computer(s) where an installation is occurring are behind firewalls without general Internet access or are production servers where administrative rules require that all installation repositories be locked down. For this specific need, automated solutions do exist, allowing either explicit specification of the packages that you want to have available, or transparently caching all the ones that you happened to access. Regards, Martin ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Python people want CPAN and how the latter came about
Martin v. Löwis a écrit : It is unreliable [bugs.launchpad.net/pypi-mirror/+bug/386143] and lacks the pre-extracted metadata. I wouldn't call it a mirror tool, for it is not an exact copy of PyPI data[1]. *** [1] In computing, a mirror is an exact copy of a data set. On the Internet, a mirror site is an exact copy of another Internet site. ... In this strict sense, PyPI will never have a mirror, nor does any of the other project hosting services have a mirror in the strict sense. In particular, I'm not willing (and not allowed to) give mirrors access to PyPI's user database: account names and email addresses. Mirrors are just mirrors, We can strip down the pypi database to bare-mirroring minimum. Mirrors could just be read-only. Thus by deleting or changing all personal accounts and other critical informations to something random. We do not need complete pypi database on mirrors. It's largely enough to get the distributions from and to install things. It doesn't require also much developments to get it ready. Just the script that make the current database informationless and download safe and limited access to the filesystem where are stored the distributions for replication. We can imagine some xmlrpc mecanisms to synchronise mirrors databases without implicating to change the mirror database after some time. I didn't look to existing pypimirroring tools, i think some implemention (z3c.pypimirror at least) must do something in that way and could be a good starter. Regards, Martin -- Cordialement, KiOrKY GPG Key FingerPrint: 0x1A1194B7681112AF signature.asc Description: OpenPGP digital signature ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Python people want CPAN and how the latter came about
Sorry, but I'm not being philosophical when I say you have to authorize access to things. Apparently the Python repository does, too. Or otherwise I'll upload a few popular packages with high version numbers that contain viruses for New Year. Maybe it's a terminology matter. I have to authorize: who is I? In PyPI, no person ever authorizes access. Yet, you still cannot upload newer versions of popular packages. Package names are registered on a first-come first-served basis. You could register a popular package if you were the first one to do so. However, all the popular packages are already registered, and then only the person originally performing the registration may upload newer versions (strictly speaking, that person can also delegate that permission to others, but that is besides the point that the archive operators will never need to authorize anything). Now, on CPAN, I *can* upload anything even if not authorized to do so. It just won't be part of the official indexes if I upload a new version of the database interface DBI. In PyPI, you can upload the popular package under a different name. It will be indexed and all, but people still may not use it because of the different name. That we do out meta data stuff on package/namespace/class names as opposed to distribution names has the huge benefit of interoperability between distributions. I think you misunderstood the Python term distribution here (or I misunderstand the point you make). A distribution is a tar file (or such); it's what library authors distribute. There can't be interoperability between them, at least not in the way I understand interoperability. Think about it like this: If you install any module from CPAN (and only the valid ones end up in the index), you can use all of them in the same application. If module A and B could both implement Config::Parser, then you couldn't use both of them at the same time. This would be true for Python too. But Python doesn't try to pretend that all the packages that exist are some sort of standard library, and therefore don't try to put them all into one sort of hierarchical organized namespace. And to be honest I don't see the point of doing that. We're not pretending anything. We're not forcing anything except that you don't override somebody elses work. I think the point here was that we see the advantage that CPAN imposes with the namespace registrations primarily as theoretical. Yes, it does prevent two people putting code into Config::Parser, and yes, in theory, it may be that they do the same in Python with PyPI. In practice, that is never a problem - there are so many names to chose from, and if you do happen to conflict with somebody else's naming choice, AND there is indeed interest in using both packages simultaneously, your users will ask you to rename your package. However, that happens so rarely if at all that it hasn't been a real concern. And obviously on PyPI, it's first come first serve as well. But nobody would call a db package db if one already exists. Why would they do that? What's the point? Why would I make a completely new package called Twisted for example? There already is one. It's just a mindset that is completely incomprehensible to me. Then you clearly do not understand what it is like to be a) malicious b) new, young, inexperienced c) stupid. If you have malicious users, and unsuspecting users download and run their code, no namespacing mechanism can stop them, neither in CPAN nor in PyPI. Malicious users would be able to bypass any checks that are performed, and experienced users, code review etc. needs to discover that. New or stupid users may accidentally create colliding names. However, in Python, packages aren't called db, but, say, psycopg2, Twisted, django, or zope. Chances are fairly low that new users accidentally come up with such a name. Regards, Martin ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Python people want CPAN and how the latter came about
In a message of Sat, 26 Dec 2009 13:03:32 +0900, David Cournapeau writes: I guess you meant upload. The reasons I see for making some things, in particular upload mandatory are as follows: - file upload makes pure rsync-based mirroring. In the case of CRAN, you mirror with one rsync command. No need for fancy scheme, new packages, etc... - lack of tarballs/installers means that when you use an installer, for example easy_install, it has to find it in another way. This way is based on scraping: the website may be dead, not available anymore, etc... In that case, installation fails. Different websites may also have different limitations (think about corporate networks). This, of course, is one reason why some people want to do exactly this. Right now I don't know any way to say 'under no circumstances, ever, let easy_install near my code because it will do very bad things to it'. - more generally, if there is an information missing on Pypi, be it in the form of metadata, data, etc..., it means it will have to be inferred back. Making upload easier seems a very weak argument to justify this. Ah, this isn't the argument about laziness and easy of use. This is the argument about control. Many commercial organisations have a policy that they, and only they host their own code. Â And they will not be willing to change this policy even if you convice them that it is in their interest to Open Source some of it, or perhaps the python bindings to some of it. Â How do we want to interoperate with these people and their code? I did not know it was even allowed to register non open source code in Pypi. That's the first time I see the argument given (and the first time a real argument is given). Is this a major requirement ? It's not about non-open source code. It's about code that is open sourced, but still copyright the company that made it. The company may have strict rules about where it hosts its code, for reasons of guarding its reputation, protection from lawsuits, and the like. This doesn't stop somebody else from taking their code and then uploading it to pypi, of course, but that's not what the corporate mandate says. And, with pypi adding more features, and starting to branch into customer support, and bug tracking, and a rating system and the whole social networking bit ... there is more and more reasons for companies to decide that the burden of being part of the 'pypi whole experience' doesn't justify making the upload. I liked things a whole lot better when pypi was about being a package index, and _only_ about being a package index, and where those people who had ideas about improving the user experience were free to go out there and write their own programs to do the same, but where none of these has any sort of 'official recognition' and where, of course, others who didn't want that sort of experience were free to ignore the whole thing. Laura David ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Python people want CPAN and how the latter came about
On Sat, Dec 26, 2009 at 13:15, Laura Creighton l...@openend.se wrote: This, of course, is one reason why some people want to do exactly this. Right now I don't know any way to say 'under no circumstances, ever, let easy_install near my code because it will do very bad things to it'. Well Not registering on PyPI? :-) Another way is to not include a setup.py. easy_install will then not try to run setup.py install. -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Python people want CPAN and how the latter came about
On Dec 26, 2009, at 7:15 AM, Laura Creighton wrote: Right now I don't know any way to say 'under no circumstances, ever, let easy_install near my code because it will do very bad things to it'. Uh...I think you just did. I liked things a whole lot better when pypi was about being a package index, and _only_ about being a package index, and where those people who had ideas about improving the user experience were free to go out there and write their own programs to do the same, but where none of these has any sort of 'official recognition' and where, of course, others who didn't want that sort of experience were free to ignore the whole thing. I think that, in the whole CPAN-ification of PyPI discussion, an absurd amount feature creep has come into the discussion. I think the ratings discussion was the tiny crystal that started the whole gigantic snowball. At the bottom of everything CPAN's repository is just a glorified, rsync-able FTP site with a bunch of stuff in directories. Everything on top of that is window dressing. The PyPI discussions seem to be tending toward mixing the window dressing with the framing, to use a building analogy, and what that will result in is a weak frame and ugly windows. A building that slowly (or quickly) falls down under its own weight, and looks bad doing it. I think that splitting package storage and pointers to off-repository storage (for those who don't upload to PyPI) metadata about the stored packages tools for creating stored packages tools for retrieving stored packages tools for installing packages would go a long way towards unobfuscating this whole discussion. Yes, I'm sure someone will disagree with some fine-point of that division but isn't that what woodshedding is all about? S ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Python people want CPAN and how the latter came about
Hi Martin, Martin v. Löwis martin at v.loewis.de writes: Maybe it's a terminology matter. I have to authorize: who is I? In PyPI, no person ever authorizes access. Yet, you still cannot upload newer versions of popular packages. Package names are registered on a first-come first-served basis. You could register a popular package if you were the first one to do so. However, all the popular packages are already registered, and then only the person originally performing the registration may upload newer versions (strictly speaking, that person can also delegate that permission to others, but that is besides the point that the archive operators will never need to authorize anything). You're right. It's terminology. This is exactly how PAUSE works. I had a longer reply to Lennart's post written up when firefox bombed out. I should have subscribed to the list instead :/ Now, on CPAN, I *can* upload anything even if not authorized to do so. It just won't be part of the official indexes if I upload a new version of the database interface DBI. In PyPI, you can upload the popular package under a different name. It will be indexed and all, but people still may not use it because of the different name. Hmm. If you replace different name here with different package name(s), then it's the same for CPAN. Simply renaming the distribution, however, doesn't work there. That we do out meta data stuff on package/namespace/class names as opposed to distribution names has the huge benefit of interoperability between distributions. I think you misunderstood the Python term distribution here (or I misunderstand the point you make). A distribution is a tar file (or such); it's what library authors distribute. There can't be interoperability between them, at least not in the way I understand interoperability. Maybe I wasn't clear. But in the end, I think the misunderstanding comes down to a difference between Perl and Python: Perl mixes class names and namespaces (= class hierarchies instead of namespaces as a language feature) whereas Python has them separate. By distribution, I also meant tarballs. Interoperability in the sense of using library A, B, and C all in the same project (be it library D or an application). If you do that, you need to make sure the fully resolved class names (including their namespace in the Python case) is unique between those libraries. Otherwise there'll be a clash. I think the point here was that we see the advantage that CPAN imposes with the namespace registrations primarily as theoretical. Yes, it does prevent two people putting code into Config::Parser, and yes, in theory, it may be that they do the same in Python with PyPI. In practice, that is never a problem - there are so many names to chose from, and if you do happen to conflict with somebody else's naming choice, AND there is indeed interest in using both packages simultaneously, your users will ask you to rename your package. However, that happens so rarely if at all that it hasn't been a real concern. Nit: Renaming packages is really only possible if they have no users. Also, all authors think they wrote the ONE TRUE config parser. So they must have Config::Parser. I humbly think on this point, IF Python namespaces don't do the disambiguation for you anyway, PyPI gets it wrong. On the other hand, if you think in monolithic, large libraries as opposed to small, highly specialized and reusable components that make for the bulk of CPAN, this may not be immediately obvious. You say namespace registration. I'm not sure what you're refering to, but 99% of CPAN is just the regular first-come auto-registration. The manual registration that is still possible is a mere relic. If you have malicious users, and unsuspecting users download and run their code, no namespacing mechanism can stop them, neither in CPAN nor in PyPI. Malicious users would be able to bypass any checks that are performed, and experienced users, code review etc. needs to discover that. That's true. But lowering the barrier by making it easy to upload a new package that has the same name as a popular one but a higher version number is silly. But that is academic. Neither CPAN or PyPI make this mistake. New or stupid users may accidentally create colliding names. However, in Python, packages aren't called db, but, say, psycopg2, Twisted, django, or zope. Chances are fairly low that new users accidentally come up with such a name. DB was a pathological example. It's the class name of the perl-internal debugger but lends itself well to different interpretation. Best regards, Steffen ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Python people want CPAN and how the latter came about
The PyPI discussions seem to be tending toward mixing the window dressing with the framing, to use a building analogy, and what that will result in is a weak frame and ugly windows. A building that slowly (or quickly) falls down under its own weight, and looks bad doing it. I think that splitting package storage and pointers to off-repository storage (for those who don't upload to PyPI) metadata about the stored packages tools for creating stored packages tools for retrieving stored packages tools for installing packages would go a long way towards unobfuscating this whole discussion Is not what sourceforge already provide? Susebuild server could provide support to complete the loop: package storage and pointers to off-repository storage (for those who don't upload to PyPI) metadata about the stored packages 1 sorce/project management/metadata (sourceforge) tools for creating stored packages tools for retrieving stored packages 2 continuous build binary (susebuild) 3 package repositpry download (with programmatic api) on susebuild tools for installing packages yum/yast/apt etc tools are already there for linux o and supported on opensuse build server Regards, Antonio If you can forgive my self promotion I've used this approach in a project of mine (pyvm.sf.net): is not completely automated, but it fits the bill point by point: moreover it has support for unitesting too. Yes, I'm sure someone will disagree with some fine-point of that division but isn't that what woodshedding is all about? ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Python people want CPAN and how the latter came about
sstein...@gmail.com wrote: I think that splitting package storage and pointers to off-repository storage (for those who don't upload to PyPI) metadata about the stored packages tools for creating stored packages tools for retrieving stored packages tools for installing packages would go a long way towards unobfuscating this whole discussion. Yes, I'm sure someone will disagree with some fine-point of that division but isn't that what woodshedding is all about? OT (diction): Hmm ... I think you meant bikeshedding (energetic discussion of meaningless details) -- woodshedding *is* an actual slang term, but one that originated among musicians, meaning to practice one's skills alone (i.e., to go off to a woodshed where no one can hear you). Steve ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Python people want CPAN and how the latter came about
On Dec 26, 2009, at 4:17 PM, Stephen Waterbury wrote: sstein...@gmail.com wrote: I think that splittingpackage storage and pointers to off-repository storage (for those who don't upload to PyPI) metadata about the stored packages tools for creating stored packages tools for retrieving stored packages tools for installing packages would go a long way towards unobfuscating this whole discussion. Yes, I'm sure someone will disagree with some fine-point of that division but isn't that what woodshedding is all about? OT (diction): Hmm ... I think you meant bikeshedding (energetic discussion of meaningless details) -- woodshedding *is* an actual slang term, but one that originated among musicians, meaning to practice one's skills alone (i.e., to go off to a woodshed where no one can hear you). I know that, I'm a musician, too..it was kind of a play on the fact that simple bikeshedding seems to turn into woodshedding i.e. it goes beyond the normal bikeshedding and becomes beating a dead bikeshed... I couldn't come up with a conjoined word fast enough so I just let it go as I wrote it. S ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Python people want CPAN and how the latter came about
Greetings Lennart, On 12/24/2009 10:27 PM, Lennart Regebro wrote: On Fri, Dec 25, 2009 at 05:39, Sridhar Ratnakumar sridh...@activestate.com wrote: Is it because of this benefit to package authors that we are withholding the implementation of a simple archive that would: 1) simplify the tools to no rely on adhoc web scrapping There are better ways to do that. May I ask, what would they be? 2) reduce the downtime for users by rsync/ftp mirroring This is true, but the idea to upload them by robots is preferable in my opinion. Again it's a difference between trying to force other people to behave to your expectations vs trying to make it easier for others to behave to your expectations. 3) have package sources mirrored so project owners do not have to worry about downtime of their servers. That's *their* problem. If they don't want to upload, then they don't want to upload. As the original proposal is to retain the existing behavior for already registered/uploaded package releases (such as Twisted) so existing systems will continue to work, but implement the suggested upload rules only for new requests (creation/register)- so as to gradually improve the quality of PyPI like that of other packaging systems - by encouraging authors to generate a reasonably good sdist (setup.py + PKG-INFO) and uploading them .. and consequently enabling the move towards a static archive that can easily be mirrored, I fail to see just what good is achieved by retaining the status quo. If I want to use a web service, I obviously have to adhere to their rules and policies. Nobody is forcing me to do so. I assume in good faith that package authors will be happy to adapt to the new system .. for the benefit of everyone. I will be happy to be proven otherwise. (Speculations are useless; how about we actually ask the package authors themselves?) 4) enable proliferation of third-party tools like CPAN? That won't help. Why not? Do you conceive of any reason apart from CPAN-like archives that would help in proliferation of mirror sites and third-party sites? I ask because I personally went through significant hurdles to setup a daily PyPI mirror-like area. I just don't see how someone merely interested in writing a third-party service, or setup a mirror of PyPI would be *most likely inclined* to face similar hurdles before giving up. Because I went through these hurdles, I was able to appreciate CPAN's design while reading about it [cpan.org/misc/ZCAN.html]. Nope, it matters not whether the metadata can be retrived via a simple HTTP GET or XmlRpc. OK. Then you have two proposals: 1. Require uploading, which is a bad idea and 2. Making it easier to mirror the metadata, which seems reasonable, assuming it's currently hard. :) Here's one idea (example only): $ tar zxf foo-0.1.tar.gz $ cp foo-0.1/PKG-INFO foo-0.1.tar.gz.PKG-INFO Metadata is definitely needed. Otherwise, I'd have to extract the tarball of each and every release of a pacticular package, in order to even find their version number (it is unreliable to parse the filename to get version number). Yes, but it's not particularly unreliable to compare the filename to see if it had been handled before. You don't even need to parse the version number for most services that work on the tarballs. It is indeed unreliable to rely on filenames to get package versions (unless that sdist is generated by the `setup.py sdist` command). As I've mentioned elsewhere, some packages have weird filenames (eg: latest.zip, foo.py); some others have '.dev' suffix in the filenames while setup.py:version (hence PKG-INFO) will not have the '.dev' prefix. And several other issues that I cannot recall right now. I am not speculating as I've actually experimented with the PyPI index, mirroring it .. handling the metadata in packages, and building it. As for the sdists, the following tools would need it: testing service, quality ratings, thirdparty package managers (enstaller, PyPM) .. and not to mention the various mirror sites. Yes, but since thay have the source package, and will have to unpack it and build the packages anyway, they also have the metadata. It is not that simple. PyPM backend, for instance, is not monolithic as in doing only a sequential build of packages. It first loads the dependency graph (for which metadata - PKG-INFO/requires.txt - is required) from our internal mirror over the network. It is expensive to go extract each and every tarball .. from each build machine. After loading the dependency graph, and then comparing it with existing repository .. every day, new builds happen. Certain packages even lack metadata (eg: no PKG-INFO in Twisted's sdist) in their source distributions .. which is another issue altogether. Further, I can imagine search.cpan.org (which is not hosted by cpan.org folks) using only the metadata without touching the source distributions. -srid
Re: [Distutils] Python people want CPAN and how the latter came about
On Fri, Dec 25, 2009 at 08:22, David Cournapeau courn...@gmail.com wrote: Is there any hard data to back up the idea that making some things mandatory when registering/uploading to Pypi is detrimental to Pypi's goal, or is it just opinion ? It's just how humans work. Hard data comes from 200 years of economic and social sciences. It is very difficult for me to understand the rationale for this. This is specific to Pypi AFAIK Not at all. (compared to CRAN, hackage, etc...), and the advantages of making it mandatory are obvious and hard facts (easy to mirror It does not make it more easy to mirror. When you mirror PyPI's file storage, you obviously do not mirror what is not there. easy to do basic consistency checks, easy to interoperate), I don't see how that helps either. whereas the downsides are far from obvious. The downside is *extremely* obvious: You make it harder to register packages on PyPI. That means less packages will be registered. As it stands today, the requirement that you would have to upload the packages to PyPI as well as register them means, for example, that neither Twisted nor PIL would not be listed on PyPI. Two very popular and well respected packages. That *is* an obvious downside. -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Python people want CPAN and how the latter came about
On Fri, Dec 25, 2009 at 5:52 PM, Lennart Regebro rege...@gmail.com wrote: It is very difficult for me to understand the rationale for this. This is specific to Pypi AFAIK Not at all. Which other system similar to Pypi do you know which works like Pypi in that regard ? Neither CRAN, hackage, opensuse build service, rpm/deb repositories work like this to cite the ones I am somehow familiar with. It seems that CPAN does not either if I understand from the discussion. David ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Python people want CPAN and how the latter came about
On Fri, Dec 25, 2009 at 09:00, Sridhar Ratnakumar sridh...@activestate.com wrote: Greetings Lennart, On 12/24/2009 10:27 PM, Lennart Regebro wrote: On Fri, Dec 25, 2009 at 05:39, Sridhar Ratnakumar sridh...@activestate.com wrote: Is it because of this benefit to package authors that we are withholding the implementation of a simple archive that would: 1) simplify the tools to no rely on adhoc web scrapping There are better ways to do that. May I ask, what would they be? Have links in the metadata to the file locations. That means you don't have to scrape the websites to find the links, the links would be in the metadata for the packages, or accessible in some other easy way. Scraping would no longer be needed, without requiring uploads to PyPI. That's *their* problem. If they don't want to upload, then they don't want to upload. As the original proposal is to retain the existing behavior for already registered/uploaded package releases (such as Twisted) so existing systems will continue to work, but implement the suggested upload rules only for new requests (creation/register)- so as to gradually improve the quality of PyPI like that of other packaging systems - by encouraging authors to generate a reasonably good sdist (setup.py + PKG-INFO) and uploading them No, that's not encouraging, that's requiring and forcing. That is NOT the same thing. Again: If you tell people you have to upload to register, the effect of that is to NOT register. It will not make anybody upload, it will make them NOT register. It has already been explained in this discussion why the Twisted folks doesn't upload to PyPI: Because it for various reasons doesn't work for them. Your solution to get more packages to PyPI is to tell people to upload or bugger off. My solution is to fix the reason they don't upload. It really is that easy: If you tell people to upload or bugger off, they will bugger off. If I want to use a web service, I obviously have to adhere to their rules and policies. Nobody is forcing me to do so. Exactly. Nobody is forcing anybody to use PyPI. By making it HARDER to use and have MORE requirements LESS people will use it. Is it really that hard to understand? I assume in good faith that package authors will be happy to adapt to the new system You are wrong. .. for the benefit of everyone. No, its not for the benefit of everyone. It's for the benefit of adherence to random rules with no purpose. If we want more packages on PyPI, we should fix the reasons that not everyone uploads their packages. And yes, I *am* going to repeat this in different ways and wordings until your ears fall off. ;-) Why not? Do you conceive of any reason apart from CPAN-like archives that would help in proliferation of mirror sites and third-party sites? The point is that we *have* a CPAN like archive. because I personally went through significant hurdles to setup a daily PyPI mirror-like area. I just don't see how someone merely interested in writing a third-party service, or setup a mirror of PyPI would be *most likely inclined* to face similar hurdles before giving up. You are so focused and stuck on that before you can do anything else you have to mirror PyPI completely, only using rsync. I don't see what that would have to be so. Rsync is not the be all and end all of mirroring and most third party services do not need to mirror. They need to get data, and that's possible quite easily. Yes, but it's not particularly unreliable to compare the filename to see if it had been handled before. You don't even need to parse the version number for most services that work on the tarballs. It is indeed unreliable to rely on filenames to get package versions Yes, but it's not particularly unreliable to compare the filename to see if it had been handled before. You don't even need to parse the version number for most services that work on the tarballs. I am not speculating as I've actually experimented with the PyPI index, mirroring it .. handling the metadata in packages, and building it. Yes, but again, most third party service would not mirror it. A mirror would. But a mirror is only one type of third-party service out a many. Yes, but since thay have the source package, and will have to unpack it and build the packages anyway, they also have the metadata. It is not that simple. PyPM backend, for instance, is not monolithic as in doing only a sequential build of packages. It first loads the dependency graph (for which metadata - PKG-INFO/requires.txt - is required) from our internal mirror over the network. It is expensive to go extract each and every tarball .. from each build machine. After loading the dependency graph, and then comparing it with existing repository .. every day, new builds happen. You mean you build every package that also *depends* on a package that has changed? Yeah, that does require the metadata. But as I said, an easy way to mirror the metadata
Re: [Distutils] Python people want CPAN and how the latter came about
On Fri, Dec 25, 2009 at 10:06, David Cournapeau courn...@gmail.com wrote: Which other system similar to Pypi do you know which works like Pypi in that regard ? The whole world. Let's take this again: If you make it more complicated and have more requirements that has to be fulfilled before you can register a package on PyPI, LESS PACKAGES WILL BE REGISTERED. Again, requiring that you also upload files will not mean that everyone uploads their packages, but it will mean less packages get registered. Is this unclear? -- Lennart Regebro: http://regebro.wordpress.com/ Python 3 Porting: http://python-incompatibility.googlecode.com/ +33 661 58 14 64 ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Python people want CPAN and how the latter came about
On Fri, Dec 25, 2009 at 6:12 PM, Lennart Regebro rege...@gmail.com wrote: Let's take this again: If you make it more complicated and have more requirements that has to be fulfilled before you can register a package on PyPI, LESS PACKAGES WILL BE REGISTERED. It is not obvious at all (that the number would be significantly less). Other systems do work even though they are much more restricting. David ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Python people want CPAN and how the latter came about
On Fri, Dec 25, 2009 at 10:33, David Cournapeau courn...@gmail.com wrote: On Fri, Dec 25, 2009 at 6:12 PM, Lennart Regebro rege...@gmail.com wrote: Let's take this again: If you make it more complicated and have more requirements that has to be fulfilled before you can register a package on PyPI, LESS PACKAGES WILL BE REGISTERED. It is not obvious at all (that the number would be significantly less). Other systems do work even though they are much more restricting. Of course they work. That's not the point. And I'm sorry, it is obvious that it will be fewer packages registered the more work it is to register packages and the more restrictions there are on registrations. And as the benefits you want with it can be easily reached in other ways, this is the wrong path to go down. Maybe it's only obvious of you have studied some sort of social sciences or tried hard to understand management or similar things so you have gotten some basic insight into how people work as groups, but I'm not sure at all that's necessary. I really think it's completely dead obvious that the higher the requirements you have on people the more they will fail to fulfill your requirements. People are lazy. You can't demand that they aren't, you have to work with the laziness. We can't just tell package authors that they have to upload packages, you have to ask them why they do not upload them already, and then fix the problems that prevent them from uploading. -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Python people want CPAN and how the latter came about
On Fri, Dec 25, 2009 at 6:55 PM, Lennart Regebro rege...@gmail.com wrote: And I'm sorry, it is obvious that it will be fewer packages registered the more work it is to register packages and the more restrictions there are on registrations. The question that matters is how significant this effect is, not that it happens. Optimizing the number of packages independently of any other criteria does not make much sense. I think many people within the group of disatisfied Pypi users would be happy to have less packages for a better overall experience. Pypi claims to have ~ 10 000 packages; I quickly counted that hackageDB has ~ 2000 packages, CRAN claims a similar number, and I think haskell community is much smaller than python (R's certainly is). And as the benefits you want with it can be easily reached in other ways, this is the wrong path to go down. If it were, nobody would make the argument about making things more consistent for Pypi. The goal is to make Pypi better, and easy mirroring as well as reliable experience is part of that. CRAN for example has tens if not hundred of mirrors, it works very well on all supported platforms, for people who are not necessarily programmers, and they undoubtly have much less resources than python. That's a much more convincing data point that any handwaving about so called social issues and what not, although you could argue that scientists are a particular community (but that's the one I care in the first place when I do python). David ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Python people want CPAN and how the latter came about
In a message of Fri, 25 Dec 2009 19:43:24 +0900, David Cournapeau writes: I think many people within the group of disatisfied Pypi users would be happy to have less packages for a better overall experience. Aside from the fact that the word you want is _fewer_ not _less_ when you are talking about a countables, I suspect this statement is true. But I don't see any relationship between 'forcing people to download their packages' and 'giving users a better experience'. And I see one major use case where forcing people to download their packages simply will not happen. Many commercial organisations have a policy that they, and only they host their own code. And they will not be willing to change this policy even if you convice them that it is in their interest to Open Source some of it, or perhaps the python bindings to some of it. How do we want to interoperate with these people and their code? Laura ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Python people want CPAN and how the latter came about
Tres Seaver a écrit : kiorky wrote: I would say that having a package author *not* upload the distributions is their right, but I would likely avoid using such a package, That depend, people can not upload their packages because previous bad experience, for false generated sdist for example. just on that basis. Note that I build per-project mirrors of the pacakges I use anyway, in part not to depend on either PyPI You depend on them anyway in first place anyway, at the first installation, even in dev or pre-production modes. And having problems at those stages have maybe less drawbacks but you are nevertheless blocked. Having a single archive which supports mirrors officially would just be safer than a single archived not officially mirrored with thirdparty satellite mirrors which can be randomly down. And having Personal/Corporate PyPi/eggs mirrors are beyond the scope, here, i think. It's just an additional and mandatory security policy to deploy projects nowodays. or other download sources for supporting apps in production: I just prefer to use only freely-distributable software. As, i think, mostly of us including me. And 99,9% softwares registered on Pypi. So, comes my idea that we would have just to get the source distributions where they are no matter how they would have been generated and mirror them as-is on Pypi which could be the only thing to mirror (and i don't say here that mirroring pypi is synomym of easy, Lennart) to get a bit safer. In a nowodays projet, i get often errors with thirdparty mirrors. It may be just bad chance, but i got problems. Tres. -- Cordialement, KiOrKY GPG Key FingerPrint: 0x1A1194B7681112AF signature.asc Description: OpenPGP digital signature ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Python people want CPAN and how the latter came about
On Fri, Dec 25, 2009 at 13:27, kiorky kio...@cryptelium.net wrote: So, comes my idea that we would have just to get the source distributions where they are no matter how they would have been generated and mirror them as-is on Pypi which could be the only thing to mirror (and i don't say here that mirroring pypi is synomym of easy, Lennart) to get a bit safer. For clarification: I think this is a good idea. I notice the complete lack of requiring uploads, or for that matter requiring anything out of the package authors. -- Lennart Regebro: http://regebro.wordpress.com/ Python 3 Porting: http://python-incompatibility.googlecode.com/ +33 661 58 14 64 ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Python people want CPAN and how the latter came about
On 12/25/2009 1:09 AM, Lennart Regebro wrote: Why not? Do you conceive of any reason apart from CPAN-like archives that would help in proliferation of mirror sites and third-party sites? The point is that we *have* a CPAN like archive. I am amazed at the amount of denial in this discussion, and I'll pass. I have nothing further to discuss. -srid ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Python people want CPAN and how the latter came about
On Fri, Dec 25, 2009 at 19:30, Sridhar Ratnakumar sridh...@activestate.com wrote: I am amazed at the amount of denial in this discussion, and I'll pass. Explanations are not denial. Several concrete points where PyPI can be improved has come forward. Requiring upload is not one of those and will not make it better, even if it is possible it makes it more CPAN-like. Neither is the requirement that metadata must be available in the same file structure as the distribution files something that will magically make PyPI into CPAN. The idea that to be as good as CPAN is has to be exactly like CPAN and work exactly like CPAN is misguided. -- Lennart Regebro: http://regebro.wordpress.com/ Python 3 Porting: http://python-incompatibility.googlecode.com/ +33 661 58 14 64 ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Python people want CPAN and how the latter came about
But it's only butter, in fact i am just happy with sdist upload. Although SSH is quite a heavy development on PyPI side, it means we would have to implement an SSH server. (like Zope did I think for their development server, using Paramiko IIRC) cvs.zope.org / svn.zope.org (same machine) run a stock sshd: they use the command= prefix on users' pubkeys to limit what that key can be used to do (only SVN / CVS operations for any non-admin users). That works well because both cvs and subversion have hard-coded support for a remote server application, along with a proprietary protocol. Adding that kind of protocol to an application that is primarily based on http is not straight-forward (it can be done, of course). Regards, Martin ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Python people want CPAN and how the latter came about
Ben Finney wrote: That isn't a good argument. By the same logic, PyPI should not reject *any* upload, to avoid “forcing” uploaders to do extra work. PyPI's rejection of certain uploads is primarily to prevent spam from being uploaded. Am I wrong, then, in thinking that PyPI will reject an upload with malformed metadata? I don't want to elaborate the specifics in something that gets archived. If you want to know what precise checks are performed, read the source code. In principle, yes, you are wrong: PyPI may accept uploads even if the metadata are malformed. To my understanding, this discussion is about arguing whether an upload that is missing the package should be rejected by PyPI. Ah. In PyPI, there are two kinds of write interactions: registration, and upload. The latter is what brings distribution files on PyPI. This is the one I'm talking about. The registration is *only* about metadata (i.e. no files at all), and it does check the consistency of the metadata. Regards, Martin ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Python people want CPAN and how the latter came about
Martin v. Löwis a écrit : Although SSH is quite a heavy development on PyPI side, it means we would have to implement an SSH server. (like Zope did I think for their development server, using Paramiko IIRC) cvs.zope.org / svn.zope.org (same machine) run a stock sshd: they use the command= prefix on users' pubkeys to limit what that key can be used to do (only SVN / CVS operations for any non-admin users). That works well because both cvs and subversion have hard-coded support for a remote server application, along with a proprietary protocol. Adding that kind of protocol to an application that is primarily based on http is not straight-forward (it can be done, of course). Additionnal to limit via the command= prefix, making ssh wrapper scripts to allow a subset of commands or using simple things like rssh is really simple to do to just allow controlled access. We are not obliged to make the application aware of the underlying ssh infra. For example, we can upload our packages somewhere on 'the host' using plain scp and we can have other mecanisms to load them in the pypi database. Regards, Martin -- Cordialement, KiOrKY GPG Key FingerPrint: 0x1A1194B7681112AF signature.asc Description: OpenPGP digital signature ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Python people want CPAN and how the latter came about
It is unreliable [bugs.launchpad.net/pypi-mirror/+bug/386143] and lacks the pre-extracted metadata. I wouldn't call it a mirror tool, for it is not an exact copy of PyPI data[1]. *** [1] In computing, a mirror is an exact copy of a data set. On the Internet, a mirror site is an exact copy of another Internet site. ... In this strict sense, PyPI will never have a mirror, nor does any of the other project hosting services have a mirror in the strict sense. In particular, I'm not willing (and not allowed to) give mirrors access to PyPI's user database: account names and email addresses. Regards, Martin ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Python people want CPAN and how the latter came about
On Fri, Dec 25, 2009 at 8:22 PM, Laura Creighton l...@openend.se wrote: In a message of Fri, 25 Dec 2009 19:43:24 +0900, David Cournapeau writes: I think many people within the group of disatisfied Pypi users would be happy to have less packages for a better overall experience. Aside from the fact that the word you want is _fewer_ not _less_ when you are talking about a countables, I suspect this statement is true. Sorry about that, I don't always double check my English. But I don't see any relationship between 'forcing people to download their packages' and 'giving users a better experience'. I guess you meant upload. The reasons I see for making some things, in particular upload mandatory are as follows: - file upload makes pure rsync-based mirroring. In the case of CRAN, you mirror with one rsync command. No need for fancy scheme, new packages, etc... - lack of tarballs/installers means that when you use an installer, for example easy_install, it has to find it in another way. This way is based on scraping: the website may be dead, not available anymore, etc... In that case, installation fails. Different websites may also have different limitations (think about corporate networks). - more generally, if there is an information missing on Pypi, be it in the form of metadata, data, etc..., it means it will have to be inferred back. Making upload easier seems a very weak argument to justify this. Many commercial organisations have a policy that they, and only they host their own code. And they will not be willing to change this policy even if you convice them that it is in their interest to Open Source some of it, or perhaps the python bindings to some of it. How do we want to interoperate with these people and their code? I did not know it was even allowed to register non open source code in Pypi. That's the first time I see the argument given (and the first time a real argument is given). Is this a major requirement ? David ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Python people want CPAN and how the latter came about
1/ Missing packages (eg: Twisted is not there); which is why easy_install/pip had to resolve to scrapping project webpages for guessing download links. In CPAN, almost all module authors upload their sources via PAUSE. How do you propose to change that? I think it should be the choice of the package authors whether they upload their software to the central repository, or to their own home page. 2/ No metadata: When only source tarballs are stored [pypi.python.org/packages/source/P/Pylons/], what is the reliable way to a) get the source for latest version, Extract a version number from each file name, and sort the versions, then use the largest (which is 0.9.7 at the moment). b) get the source for a particular version? Put the version number into the file name, and access the resulting file. The former is more of a community issue. Often Python package authors are not using `sdist upload` (whereas this seems to be the convention in the Perl world). My guess is that this is enforced by the tools. If they don't upload to PAUSE, CPAN.pm won't be able to download it. Now, you are free to build a tool that enforces the same restriction. I would doubt that people would use it, since it couldn't install many packages. What this means is that PyPI has to serve the purpose of being a central package repository (like CPAN) by a) disallowing mere listings (without sources) and requiring sources to be stored in the server, b) storing the metadata along with the sources (so anyone processing it wouldn't have to extract the source and rely on a PKG-INFO file - which may or may not exist). If you want to retrieve the metadata for a specific version without XML-RPC, you can access http://pypi.python.org/pypi?:action=doapname=Pylonsversion=0.9.7 Regards, Martin ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Python people want CPAN and how the latter came about
3/ Indices such as http://www.cpan.org/modules/01modules.mtime.html (or TXT files) to get a) recently released packages, b) list of release versions for a particular package, and so on (which would obviate the XmlRpc interface) That is available as http://pypi.python.org/pypi?%3Aaction=rss Regards, Martin ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Python people want CPAN and how the latter came about
The tiny 10MB upload is a blocker i think also. For example, for 'minitage.paste.extras', it's not hosted on pypi because of that. The limit is 20MB now. If you need a larger limit let me know. Unfortunately, I cannot find out what the size of minitage.paste.extras is, as their download URL (http://distfiles.minitage.org/public/externals/minitage/) seems broken. Regards, Martin ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Python people want CPAN and how the latter came about
What's with the interest in having packages hosted on PyPI? Because it would then be more like CPAN. Some people claim that one key of CPAN's success is that it has all files stored in the archive, and that it then allows mirroring with rsync and ftp. They claim that without that, PyPI can't be simple, and without that, the requirement Python needs CPAN can't be met. I'm not specifically opposed to this, but I don't see any reason it would benefit anyone either. It'd be awesome if someone could explain this. Perhaps if the answer were included somewhere on PyPI or in the distutils documentation, more people would see the light and upload their packages. For all use cases except of mirroring, I don't see the need to upload, either. For mirroring, it would be easier to write mirroring tools if all packages uploaded their code (assuming you want to end up with a complete off-line mirror). Regards, Martin ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Python people want CPAN and how the latter came about
A separate issue with setup.py upload, though, is that it really wants one of two undesirable things: * the upload is done at the same time as the release package is generated * the release package is generated twice The former requires that proper credentials are available to whoever is creating the release package. Historically for Twisted, this isn't how things have been set up. We could probably deal with it, but it would be nice if it were not a requirement. It actually isn't. Upload will upload all files that are listed on distribution.dist_files, which is a three-tuple of (command/filetype, pyversion, filename). So if you somehow (e.g. with a new command, or hard-coded) put stuff into dist_files, you could arrange for python setup.py pickup_files upload to upload the pre-built files; thereby you can upload files as source that had not been generated by sdist. Regards, Martin ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Python people want CPAN and how the latter came about
Martin v. Löwis mar...@v.loewis.de writes: I think it should be the choice of the package authors whether they upload their software to the central repository, or to their own home page. Why do you think that should continue? Some of the costs of that inconsistency have already been described in this thread. What are the benefits to PyPI users of this inconsistency, and are we sure that the benefits outweigh the costs? -- \ “I got contacts, but I only need them when I read, so I got | `\ flip-ups.” —Steven Wright | _o__) | Ben Finney ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Python people want CPAN and how the latter came about
Finally somebody had few doubts about CPAN...please have a look ti a just-posted article on slashdot. That mess is CPAN was my original reason to discard perl in first (and switching to python): no two installed perl are ever the same. No way to reliably reproduce the same environment and no auditing possible: and this is a requirement in a professional environment. I was (and I'm still) happy the way distutils works: it might be enhanched but it's right in the scope (creating an installer). The way the software is indexed, managed, have the deps tracked and retrieved should not be in that scope: it is a job for something else (a tool). If I had to push for two ideas I need for my work (yes it is done in python): - integration with the system installer and updating mechanisms in place - don't assume internet available Regards, Antonio ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Python people want CPAN and how the latter came about
Martin v. Löwis a écrit : The tiny 10MB upload is a blocker i think also. For example, for 'minitage.paste.extras', it's not hosted on pypi because of that. The limit is 20MB now. If you need a larger limit let me know. Unfortunately, I cannot find out what the size of minitage.paste.extras is, as their download URL (http://distfiles.minitage.org/public/externals/minitage/) seems broken. Uhm, the server was down a little time earlier. It is fine now, normally. The size of this dist is 12mb. The last i tried to upload it was maybe 8 monthes ago if you changed the limit in the meantime. Regards, Martin -- Cordialement, KiOrKY GPG Key FingerPrint: 0x1A1194B7681112AF signature.asc Description: OpenPGP digital signature ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Python people want CPAN and how the latter came about
Martin v. Löwis a écrit : The tiny 10MB upload is a blocker i think also. For example, for 'minitage.paste.extras', it's not hosted on pypi because of that. The limit is 20MB now. If you need a larger limit let me know. Cool! I tested with success the upload. Size of the dist is 12MB. However, i don't like also that much the sdist upload command for big files. It may be an idea to setup/add some ssh+key procedure to upload packages using ssh. Then we could use rsync or such program that can resume upload on errors without having to reupload the whole. Nevertheless, i don't know what's worse, because i don't think there are that much big files to upload. In the packages i maintain, there is only minitage.paste.extras concerned for example. Maybe some started would be to find packages on pypi without dists and verify on their relative download page the bigest size we can find out there. Unfortunately, I cannot find out what the size of minitage.paste.extras is, as their download URL (http://distfiles.minitage.org/public/externals/minitage/) seems broken. Server was down a little earlier, /me was sleeping. It's fine now. -- Cordialement, KiOrKY GPG Key FingerPrint: 0x1A1194B7681112AF signature.asc Description: OpenPGP digital signature ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Python people want CPAN and how the latter came about
2009/12/24 exar...@twistedmatrix.com: [..] Release packages for Twisted are constructed using some extra file- finding logic that sdist doesn't provide. Additionally, for years distutils was seen as a blind alley, so we didn't bother to try to create a distutils-friendly substitute. Partially because it seems that distutils has turned a corner over the last year, we are actually (slowly, with difficulty) working towards a more distutils-integrated solution (we're going to try to override sdist with a command based on our existing custom file-finding code). This may result in something we can use with setup.py upload (but this isn't currently the primary goal, it would just be a happy side effect). That's quite a good news. Let me know if I can help somehow in that process. In any case I am interested in the problems you've had with sdist in order to improve it. Wether this happens in Distutils or in Distribute if it's too late for 2.7 Tarek ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Python people want CPAN and how the latter came about
Ben Finney a écrit : Martin v. Löwis mar...@v.loewis.de writes: I think it should be the choice of the package authors whether they upload their software to the central repository, or to their own home page. Why do you think that should continue? Some of the costs of that inconsistency have already been described in this thread. What are the Totally agree, all must be on pypi. benefits to PyPI users of this inconsistency, and are we sure that the benefits outweigh the costs? -- Cordialement, KiOrKY GPG Key FingerPrint: 0x1A1194B7681112AF signature.asc Description: OpenPGP digital signature ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Python people want CPAN and how the latter came about
On Thu, Dec 24, 2009 at 11:06 AM, kiorky kio...@cryptelium.net wrote: Martin v. Löwis a écrit : The tiny 10MB upload is a blocker i think also. For example, for 'minitage.paste.extras', it's not hosted on pypi because of that. The limit is 20MB now. If you need a larger limit let me know. Cool! I tested with success the upload. Size of the dist is 12MB. However, i don't like also that much the sdist upload command for big files. It may be an idea to setup/add some ssh+key procedure to upload packages using ssh. Then we could use rsync or such program that can resume upload on errors without having to reupload the whole. I don't think it worth the pain, with the speed of nowadays connections. But I could add a curl upload progress bar in the upload command. If you think this is useful let me know. That said, some people have expressed the desire to be able to interact with PyPI through SSH so they could drop the basic authentication and use their keys when registering/uploading. But that was not related to the size of files. Tarek -- Tarek Ziadé | http://ziade.org ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Python people want CPAN and how the latter came about
Tarek Ziadé a écrit : On Thu, Dec 24, 2009 at 11:06 AM, kiorky kio...@cryptelium.net wrote: Martin v. Löwis a écrit : I don't think it worth the pain, with the speed of nowadays connections. But I could add a curl upload progress bar in the upload command. If you think this is useful let me know. No. progress is useless but indeed beautiful and a nice enlightement. If it's a 2 minutes dev, it can be nice, on the other hand i think we have much problems to solve :p. The only feature i thought useful is partial upload (to resume on errors). We, in France, have for the whole good connections but not everyone. I have a quite poor connection here for example :). That said, some people have expressed the desire to be able to interact with PyPI through SSH so they could drop the basic authentication and use their keys when registering/uploading. But that was not related to the size of files. Both i think, it's easier to use the transfer programs that we are used to use to transfer things. And ssh is damn good for that, easy to setup, secure and so on. But it's only butter, in fact i am just happy with sdist upload. Tarek -- Cordialement, KiOrKY GPG Key FingerPrint: 0x1A1194B7681112AF signature.asc Description: OpenPGP digital signature ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Python people want CPAN and how the latter came about
I think it should be the choice of the package authors whether they upload their software to the central repository, or to their own home page. Why do you think that should continue? Some of the costs of that inconsistency have already been described in this thread. What are the benefits to PyPI users of this inconsistency, and are we sure that the benefits outweigh the costs? The benefits are not to the package users, clearly. Instead, they are to the package authors, which don't have to change their release processes (as also described in this thread). Regards, Martin ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Python people want CPAN and how the latter came about
On Thu, Dec 24, 2009 at 11:20 AM, kiorky kio...@cryptelium.net wrote: [..] That said, some people have expressed the desire to be able to interact with PyPI through SSH so they could drop the basic authentication and use their keys when registering/uploading. But that was not related to the size of files. Both i think, it's easier to use the transfer programs that we are used to use to transfer things. And ssh is damn good for that, easy to setup, secure and so on. But it's only butter, in fact i am just happy with sdist upload. Although SSH is quite a heavy development on PyPI side, it means we would have to implement an SSH server. (like Zope did I think for their development server, using Paramiko IIRC) ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Python people want CPAN and how the latter came about
On Thu, Dec 24, 2009 at 10:25, Ben Finney ben+pyt...@benfinney.id.au wrote: Martin v. Löwis mar...@v.loewis.de writes: I think it should be the choice of the package authors whether they upload their software to the central repository, or to their own home page. Why do you think that should continue? Because otherwise you can't register packages in PyPI without uploading them, and that means that those who do not want to upload them also will not register them. Forcing people to do what you think they should do will not make them make more or better work. It will just make them do *less* work. It's better to figure out why some people don't upload the packages, and if they have a reasonable reason, fix it. -- Lennart Regebro: http://regebro.wordpress.com/ I thought I could organize freedom. How Scandinavian of me. --Björk +33 661 58 14 64 ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Python people want CPAN and how the latter came about
python setup.py pickup_files upload to upload the pre-built files; thereby you can upload files as source that had not been generated by sdist. That's why I've proposed to add a --dist-file option to the upload command, The tricky thing may be to find out what kind of file that is, so that option would somehow need two parameters. Regards, Martin ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Python people want CPAN and how the latter came about
On Thu, Dec 24, 2009 at 02:59:28AM -, exar...@twistedmatrix.com wrote: There have been a few responses to Glyph's mention that setup.py upload doesn't work for Twisted. I'm much more curious to hear replies to his other point - nobody cares anyway. What's with the interest in having packages hosted on PyPI? I'm not specifically opposed to this, but I don't see any reason it would benefit anyone either. It'd be awesome if someone could explain this. Perhaps if the answer were included somewhere on PyPI or in the distutils documentation, more people would see the light and upload their packages. Jean-Paul Some reasons to have PyPI host packages have already been mentioned in this thread: it makes mirroring easier, and it makes it easier for individuals to build new services (web sites primarily) that present new interfaces to the Python package collection. Mirroring for its own sake is some use, but being able to grab the entire Python package repository easily from a single source is valuable for the second goal, that of furnishing the foundation (shoulders of giants and all that) for those with vision (and round tuits) to take the next step. If I wanted to host a site that (e.g.) indexed Python modules from PyPI by module (not package) name, and extracted and provided the documentation in HTML format, from what I've been reading I'd have to build a scraper or XMLRPC tool to walk PyPI, and then for each package, download it from another site (that may not have the uptime or scalability of PyPI), a nontrivial burden on aspiring visionaries that just want to build an addition and then go have a beer and discuss further improvements. For CPAN, as others have said already, a recursive FTP or rsync would do the job leaving people free to spend their time innovating. (As a point of practical interest, what _would_ be the most efficient way to download the entire set of Python modules listed on PyPI? A search comes up with z3c.pypimirror, http://pypi.python.org/pypi/z3c.pypimirror; is this the standard tool?) It's about affordability, in the sense Donald Norman uses it in The Design of Everyday Things. Does PyPI afford extension? Should it? Having one way to do it is Pythonic; so is having several people rush off and build new things sufficiently chaotic as to be unpythonic? Arguably no, because they would be prototypes - brainstorming - and the community can synthesize the best of each (hey, if TMTOWTDI Perl folk can do it...) and reintegrate them as necessary and proper. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Python people want CPAN and how the latter came about
Although SSH is quite a heavy development on PyPI side, it means we would have to implement an SSH server. (like Zope did I think for their development server, using Paramiko IIRC) Not necessarily: it might be possible to use sshd. Regards, Martin ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Python people want CPAN and how the latter came about
2009/12/24 Martin v. Löwis mar...@v.loewis.de: python setup.py pickup_files upload to upload the pre-built files; thereby you can upload files as source that had not been generated by sdist. That's why I've proposed to add a --dist-file option to the upload command, The tricky thing may be to find out what kind of file that is, so that option would somehow need two parameters. Yes. I was considering something in these lines: $ python setup.py upload --dist-file=sdist:/path/to/archive.tgz where the files are suffixed by their format. But that's just a first thought. Regards, Martin -- Tarek Ziadé | http://ziade.org ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Python people want CPAN and how the latter came about
Some reasons to have PyPI host packages have already been mentioned in this thread: it makes mirroring easier, and it makes it easier for individuals to build new services (web sites primarily) that present new interfaces to the Python package collection. Mirroring for its own sake is some use, but being able to grab the entire Python package repository easily from a single source is valuable for the second goal, that of furnishing the foundation (shoulders of giants and all that) for those with vision (and round tuits) to take the next step. That is fairly easily possible today, even without everybody uploading all files. It isn't easy *per se*, but needs a lot of code. However, this code has already been written, and using it is fairly easy. If I wanted to host a site that (e.g.) indexed Python modules from PyPI by module (not package) name, and extracted and provided the documentation in HTML format, from what I've been reading I'd have to build a scraper or XMLRPC tool to walk PyPI, and then for each package, download it from another site (that may not have the uptime or scalability of PyPI), a nontrivial burden on aspiring visionaries that just want to build an addition and then go have a beer and discuss further improvements. Not at all. You would just use one of the ten or so packages that already do precisely that, and use it. (As a point of practical interest, what _would_ be the most efficient way to download the entire set of Python modules listed on PyPI? A search comes up with z3c.pypimirror, http://pypi.python.org/pypi/z3c.pypimirror; is this the standard tool?) There are a number of other mirroring tools, such as EggBasket and collective.eggproxy. For mirroring the whole index, pypimirror is probably the best starting point. Regards, Martin ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Python people want CPAN and how the latter came about
On Thu, Dec 24, 2009 at 12:00 PM, Martin v. Löwis mar...@v.loewis.de wrote: [..] There are a number of other mirroring tools, such as EggBasket and collective.eggproxy. For mirroring the whole index, pypimirror is probably the best starting point. collective.eggproxy is particular though: it's a proxy-cache on PyPI (on any other PyPI-like server), that can be used by developers to collect in a local cache the archives they have downloaded at least once, so they don't call PyPI anymore. The tree structure is downloaded but is filled only on requests. IOW it gets filled when users uses it as their PyPI index. In tools like buildout for instance, and they end up with a local subset of the archive they *really* use and need. We did that in my previous company because we didn't want to set up and maintain a 7 giga mirror, while we just used probably less than 5% of PyPI archives. The caveat is that when a new version is up, it is downloaded only when someone claims for it, unlike mirrors that gets updated in crons. But it's not really a big deal because my developers team back then was updating their buildouts more often that any mirror cron out there I am pretty sure ;) Tarek ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Python people want CPAN and how the latter came about
Martin v. Löwis a écrit : Although SSH is quite a heavy development on PyPI side, it means we would have to implement an SSH server. (like Zope did I think for their development server, using Paramiko IIRC) Not necessarily: it might be possible to use sshd. Thas was my underlying though too. Regards, Martin ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig -- Cordialement, KiOrKY GPG Key FingerPrint: 0x1A1194B7681112AF signature.asc Description: OpenPGP digital signature ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Python people want CPAN and how the latter came about
Lennart Regebro a écrit : On Thu, Dec 24, 2009 at 10:25, Ben Finney ben+pyt...@benfinney.id.au wrote: Martin v. Löwis mar...@v.loewis.de writes: Because otherwise you can't register packages in PyPI without uploading them, and that means that those who do not want to upload them also will not register them. Forcing people to do what you think they should do will not make them Why forcing them ? How about a cronjob or something like that which find packages without distribution and get from there all distributions possible on their relative homepages and mirror them on Pypi ? It's not perfect, but it s an idea. One of the drawback is that Pypi can have outdated things. It would be also a big help in the meantime packagers update their release procedure. make more or better work. It will just make them do *less* work. It's better to figure out why some people don't upload the packages, and if they have a reasonable reason, fix it. -- Cordialement, KiOrKY GPG Key FingerPrint: 0x1A1194B7681112AF signature.asc Description: OpenPGP digital signature ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Python people want CPAN and how the latter came about
Martin v. Löwis mar...@v.loewis.de writes: The benefits are not to the package users, clearly. That should immediately make us suspicious, then. If PyPI is for anything, surely it is for the package recipients *more than* for package developers. Instead, they are to the package authors, which don't have to change their release processes (as also described in this thread). The needs of package developers are important, of course; but, I argue, always in the service of making packages available to recipients. Lennart Regebro rege...@gmail.com writes: Forcing people to do what you think they should do will not make them make more or better work. It will just make them do *less* work. That isn't a good argument. By the same logic, PyPI should not reject *any* upload, to avoid “forcing” uploaders to do extra work. No, PyPI needs to reject uploads on some criteria; those criteria will necessarily involve work (whether manual or automated) on the part of uploaders. This discussion is about what those criteria should be. -- \“Why was I with her? She reminds me of you. In fact, she | `\reminds me more of you than you do!” —Groucho Marx | _o__) | Ben Finney ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Python people want CPAN and how the latter came about
Ben Finney wrote: Martin v. Löwis mar...@v.loewis.de writes: The benefits are not to the package users, clearly. That should immediately make us suspicious, then. If PyPI is for anything, surely it is for the package recipients *more than* for package developers. I thought so when adding voting and comments to PyPI. Boy was I wrong. That isn't a good argument. By the same logic, PyPI should not reject *any* upload, to avoid “forcing” uploaders to do extra work. PyPI's rejection of certain uploads is primarily to prevent spam from being uploaded. It has nothing to do with setting quality standards of some kind. Regards, Martin ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Python people want CPAN and how the latter came about
On Dec 24, 2009, at 2:02 AM, Lennart Regebro wrote: On Thu, Dec 24, 2009 at 06:06, sstein...@gmail.com sstein...@gmail.com wrote: Right now, installing e.g. Twisted, requires finding the website, figuring out which exact file to download, then figuring out exactly how to get it installed. Nope. # super-duper-python-thing-just-like-cpan-only-better -i Twisted Should do it $ /opt/python26/bin/virtualenv twist [snip] Ok, so easy_install works for Twisted. Yay. Installed /tmp/twist/lib/python2.6/site-packages/zope.interface-3.5.3-py2.6-linux-i686.egg Finished processing dependencies for Twisted Done! At least that's what I get from all this... We have *that*. Two versions of it even, as pip would likely work as well. Ok, there's easy_install, the direct download route, and pip. I think that's exactly *not*: # super-duper-python-thing-just-like-cpan-only-better -i Twisted And, I picked Twisted 'cause it was already in the discussion. There are at least three ways to install it and I'm really concerned that what you just seemed to suggest was that the much-maligned-and-soon-to-be-deprecated `easy_install` is the `super-duper-python-thing-just-like-cpan-only-better` of today? S ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Python people want CPAN and how the latter came about
On Thu, Dec 24, 2009 at 11:46, David Robins pyt...@davidrobins.net wrote: Some reasons to have PyPI host packages have already been mentioned in this thread: it makes mirroring easier Uhm, mirroring PyPI, no. Because those packages aren't there. Setting up your own package server, yes. It's not exactly the same thing. Plone has set up their own mirroring of all the packages needed to set up Plone, to avoid having problems with downed servers etc. That's not a PyPI mirror, it's something else. PyPI is not a vague set of every package that exists in CPAN seems to be in some peoples view of the world. It's an index of packages. You can also host them there. Some people choose not to. That's up to them, and should continue to be up to them. Making it easier to actually find the package would probably be a good thing, such as including the URL's for all the files in the metadata for a package, or similar. Forcing everybody to upload to PyPI is not the correct road to travel. and it makes it easier for individuals to build new services (web sites primarily) that present new interfaces to the Python package collection. Why? They can simply ignore packages that aren't uploaded. That would increase the incentive to upload. No need to force peoples hand. Freedom baby, yeah! -- Lennart Regebro: http://regebro.wordpress.com/ Python 3 Porting: http://python-incompatibility.googlecode.com/ +33 661 58 14 64 ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Python people want CPAN and how the latter came about
On Thu, Dec 24, 2009 at 2:47 PM, Lennart Regebro rege...@gmail.com wrote: On Thu, Dec 24, 2009 at 11:46, David Robins pyt...@davidrobins.net wrote: Some reasons to have PyPI host packages have already been mentioned in this thread: it makes mirroring easier Uhm, mirroring PyPI, no. Because those packages aren't there. Setting up your own package server, yes. It's not exactly the same thing. Plone has set up their own mirroring of all the packages needed to set up Plone, to avoid having problems with downed servers etc. That's not a PyPI mirror, it's something else. It's their own independant package repository also because they want to have a known good set of packages like Turbogears or Zope has. And that's why at some point, package installers will probably need to merge serveral indexes that are not mirrors. This is what I've described in PEP 381 http://www.python.org/dev/peps/pep-0381 Notice that I've started this PEP right after I've set up plone.org's PyPI system at http://plone.org/products because in my mind, every framework should have its own package repository w.r.t the central PyPI. Tarek ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Python people want CPAN and how the latter came about
On Thu, Dec 24, 2009 at 12:28, kiorky kio...@cryptelium.net wrote: Why forcing them ? How about a cronjob or something like that which find packages without distribution and get from there all distributions possible on their relative homepages and mirror them on Pypi ? Somebody (including a bot) uploading a package is quite different from *requiring* uploads, IMO. But uploading to PyPI is for 99.999% of packages so easy that if it's not done there may be a reason why they don't want to. We may choose to ignore that, but we can only do it for licenses we know allow it, which again means that there may be packages listed on PyPI that are not hosted on PyPI. I personally don't understand why third-party services would need this. -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Python people want CPAN and how the latter came about
On Thu, Dec 24, 2009 at 14:44, sstein...@gmail.com sstein...@gmail.com wrote: Ok, so easy_install works for Twisted. Yay. So, your requirement is fulfilled. I think that's exactly *not*: # super-duper-python-thing-just-like-cpan-only-better -i Twisted No, it's # super-duper-python-thing-just-like-cpan-only-better -i Twisted and # the-other-super-duper-python-thing-just-like-cpan-only-even-better -i Twisted And, I picked Twisted 'cause it was already in the discussion. Yes, and this works pretty much for any other package as well. There are some that are tricky because they require C-libraries that are hard to install etc, but most packages can be installed like this. Complete applications, like Zope2, Turbogears etcs, can often not, but even the last version of Zope 2 can be easy_installed. There are at least three ways to install it and I'm really concerned that what you just seemed to suggest was that the much-maligned-and-soon-to-be-deprecated `easy_install` is the `super-duper-python-thing-just-like-cpan-only-better` of today? That's a strange concern. -- Lennart Regebro: http://regebro.wordpress.com/ Python 3 Porting: http://python-incompatibility.googlecode.com/ +33 661 58 14 64 ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Python people want CPAN and how the latter came about
On Thu, Dec 24, 2009 at 13:17, Ben Finney ben+pyt...@benfinney.id.au wrote: Forcing people to do what you think they should do will not make them make more or better work. It will just make them do *less* work. That isn't a good argument. It's a *fantastic* argument. By the same logic, PyPI should not reject *any* upload, to avoid “forcing” uploaders to do extra work. U yes? Which of course is how it works? The comment confused me, I don't understand what you are trying to say. No, PyPI needs to reject uploads on some criteria; those criteria will necessarily involve work (whether manual or automated) on the part of uploaders. This discussion is about what those criteria should be. No, it does not need to do that at all, and no, the discussion isn't about that in any way. The only discussion related to that has been about requiring the upload of the source. That's it. Nobody has to my knowledge suggested any other requirements. -- Lennart Regebro: http://regebro.wordpress.com/ Python 3 Porting: http://python-incompatibility.googlecode.com/ +33 661 58 14 64 ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Python people want CPAN and how the latter came about
On Dec 24, 2009, at 9:06 AM, Lennart Regebro wrote: I'm really concerned that what you just seemed to suggest was that the much-maligned-and-soon-to-be-deprecated `easy_install` is the `super-duper-python-thing-just-like-cpan-only-better` of today? That's a strange concern. I don't think it's strange to be concerned that one of the SDPTJLCOB options, and the one that you used as your example, is going to go away. If pip is going to be the SDPTJLCOB of the future, with an facade that substitutes for easy_install in the same way that Distribute does for setuptools then great, that's progress! S ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Python people want CPAN and how the latter came about
On Thu, Dec 24, 2009 at 15:46, sstein...@gmail.com sstein...@gmail.com wrote: I don't think it's strange to be concerned that one of the SDPTJLCOB options, and the one that you used as your example, is going to go away. Why? Pip does the same thing. They are commands. And it's not likely to go away anytime soon, what probably happens is that people will use pip more and more until pretty much noone uses easy_install. It completely and utterly fails me to see why this would be anything to worry about. If pip is going to be the SDPTJLCOB of the future, with an facade that substitutes for easy_install in the same way that Distribute does for setuptools then great, that's progress! It'll have an easy_install facade if someone needs it badly enough to write one. But I can't imagine why they would. -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Python people want CPAN and how the latter came about
Hi Lennart, thanks for the reply. Lennart Regebro regebro at gmail.com writes: On Tue, Dec 22, 2009 at 10:14, Steffen Mueller smueller at cpan.org wrote: Let me add two nits here: It's Perl, not PERL. The name of the language is *not* an acronym. Some people are really picky about that. In a language where you can spell any functionality in millions of ways, you are not allowed to spell the language however you want!? Ok, Pörl. I mean Perl. IMO, the There is more than one way to do it mantra gets old very quick. It's generally appreciated as something along the lines of There is more than one way to do it, but most of them are bad. You'd be surprised how many tools are converging into best practices in Perl. Cf. modules like Moose or DBIx::Class, etc. There's still at least one new, sucky configuration parser uploaded to CPAN every day, though :( Anyway. I appreciate the humour ;) OK, so that would be docstrings and stuff then. It's true, if we had something like that it would work as an incentive for making more such documentation. I agree. That helps tremendously. You could get the same page via http://search.cpan.org/perldoc?PAR::Repository::Client. This is an example of how CPAN works on namespace and distribution level. After a new file is uploaded, its contained namespaces (packages/class names) are added to the index if they're not already there. How does it handle if two modules implement the same namespace? There are two ways to get a namespace (non-recursively!). Either you file a form with the PAUSE admins or you're simply the first to upload under the name. If someone does something outrageously stupid wrt. namespaces, he feels the wraith of the community. If that's not enough, the admins can act. But this is very, very, extremely rare. The index always contains a reference to the distribution that contains the newest *authorized* version of any namespace. Who authorizes it? The first come first serve principle. Or if there is a disagreement/ clash between authors, the PAUSE admins. I have to say that I vastly prefer not to have any authorization and allow anyone to release anything in any namespace. But then I am getting fanatically anarchical in these issues. You can not organize freedom. But you have to. What you're saying here means you virtually throw away the ability to do anything useful with namespace meta data. Think about it like this: If you install any module from CPAN (and only the valid ones end up in the index), you can use all of them in the same application. If module A and B could both implement Config::Parser, then you couldn't use both of them at the same time. Now, this may be a case where my lack of Python makes me look silly. In Perl, when I refer to namespaces, I'm actually thinking of hierarchical class structures, not C++ style namespaces which may in turn contain arbitrary classes. Effectively, though, the C++ namespaces + classes end up about the same as Perl's namespaces. Still. We allow for a lot of creative freedom. We just don't want a random newbie upload a shiny package DB which implements his idea of a database interface when it's really the name of the package that implements the Perl debugger. He can still upload. The file will be accessible in his CPAN directory. Users can install it from the CPAN shell as install NEWBIEUSERID/DB-1.00.tar.gz, but not as install DB or $ cpan DB. This is a safeguard against insanity and it's the thing that means that you can trust cpan PAR to always install the Perl Archive Toolkit that was released by Autrijus Tang, Roderich Schupp, or myself (we share co-maintenance). It's never going to be some random junk. And that you auto-register a namespace on upload is the guarantee. Best regards, Steffen PS: Let me comment on another post of yours quickly. No. In the Perl community, the name CPAN doesn't yield confusion. It's just a way to refer to the whole ecosystem including but not limited to: * the FTP/HTTP archive * PAUSE, the Perl Authors Upload Server and the indexes it generates * The various web frontends * The toolchain, i.e. the set of tools that can create new distributions as well as build and install them. * The huge set of tests and modules that do the testing. * The CPAN.pm or CPANPLUS modules that provide the user interface (shell, etc) and do the dependency resolution, downloading, local unpacking, controlling of the build steps. * The cpan and cpanp CLIs to CPAN.pm and CPANPLUS respectively. Let me also refer to Ash Berlin's recent posts on a similar subject. PPS: All, could you please not turn this thread into a flame war? Civility and mutual respect go a long way towards getting shit the fuck done. Pun intended. :P ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Python people want CPAN and how the latter came about
On Dec 24, 2009, at 10:16 AM, Lennart Regebro wrote: On Thu, Dec 24, 2009 at 15:46, sstein...@gmail.com sstein...@gmail.com wrote: I don't think it's strange to be concerned that one of the SDPTJLCOB options, and the one that you used as your example, is going to go away. Why? Pip does the same thing. They are commands. And it's not likely to go away anytime soon, what probably happens is that people will use pip more and more until pretty much noone uses easy_install. It completely and utterly fails me to see why this would be anything to worry about. I guess what I mean is that I'd like to make sure that while moving to pip, that easy_install, as a command name, not as an implementation, gets brought along in the same way that Distribute has brought along setuptools compatibility. IOW in such a way that the improvements in the new code are available without breaking any existing configurations/scripts/workflows etc. If pip is going to be the SDPTJLCOB of the future, with an facade that substitutes for easy_install in the same way that Distribute does for setuptools then great, that's progress! It'll have an easy_install facade if someone needs it badly enough to write one. But I can't imagine why they would. I imagine it would be useful to have a facade because easy_install is familiar and ubiquitous and shouldn't be left rotting in place as pip takes over the world and becomes the SDPTJLCOB any more than Distribute left the rest of setuptools exposed to cause problems once Distribute was installed. S ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Python people want CPAN and how the latter came about
Lennart Regebro a écrit : On Thu, Dec 24, 2009 at 12:28, kiorky kio...@cryptelium.net wrote: Why forcing them ? How about a cronjob or something like that which find packages without distribution and get from there all distributions possible on their relative homepages and mirror them on Pypi ? Somebody (including a bot) uploading a package is quite different from *requiring* uploads, IMO. But uploading to PyPI is for 99.999% of packages so easy that if it's not done there may be a reason why they don't want to. Unless they don't know how to do. We may choose to ignore that, but we can only do it for licenses we know allow it, 99% of pypi packages. which again means that there may be packages listed on PyPI that are not hosted on PyPI. I personally don't understand why third-party services would need this. That's quite out of scope which what i wanted to spot but to answear: The only problem is with licenses which constrain redistribution. We can have logic around trove classifiers to filter them out. If the authors miss the trove, that's their fault. What i would like to see is just to have most of the distributions on Pypi because by experience, pypi get less offline that thirdparty mirrors, and i can set mirrors of pypi easyly. -- Cordialement, KiOrKY GPG Key FingerPrint: 0x1A1194B7681112AF signature.asc Description: OpenPGP digital signature ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Python people want CPAN and how the latter came about
Martin v. Löwis mar...@v.loewis.de writes: Ben Finney wrote: That isn't a good argument. By the same logic, PyPI should not reject *any* upload, to avoid “forcing” uploaders to do extra work. PyPI's rejection of certain uploads is primarily to prevent spam from being uploaded. Am I wrong, then, in thinking that PyPI will reject an upload with malformed metadata? To my understanding, this discussion is about arguing whether an upload that is missing the package should be rejected by PyPI. -- \“Absurdity, n. A statement or belief manifestly inconsistent | `\with one's own opinion.” —Ambrose Bierce, _The Devil's | _o__)Dictionary_, 1906 | Ben Finney ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Python people want CPAN and how the latter came about
On Thu, Dec 24, 2009 at 17:48, sstein...@gmail.com sstein...@gmail.com wrote: I guess what I mean is that I'd like to make sure that while moving to pip, that easy_install, as a command name, not as an implementation, gets brought along in the same way that Distribute has brought along setuptools compatibility. IOW in such a way that the improvements in the new code are available without breaking any existing configurations/scripts/workflows etc. easy_install is a command, basically a wrapper around setuptools install functionality. I doubt many scripts would use it in any more complex way than calling it, in which case moving to pip is a matter of replacing the command. I imagine it would be useful to have a facade because easy_install is familiar and ubiquitous and shouldn't be left rotting in place as pip takes over the world and becomes the SDPTJLCOB any more than Distribute left the rest of setuptools exposed to cause problems once Distribute was installed. I don't even understand that sentence, let alone what you are trying to say. Distribute is a setuptools fork. Pip is not an easy_install fork. I don't really see the parallell. -- Lennart Regebro: http://regebro.wordpress.com/ Python 3 Porting: http://python-incompatibility.googlecode.com/ +33 661 58 14 64 ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Python people want CPAN and how the latter came about
On Thu, Dec 24, 2009 at 22:20, kiorky kio...@cryptelium.net wrote: But uploading to PyPI is for 99.999% of packages so easy that if it's not done there may be a reason why they don't want to. Unless they don't know how to do. It's well documented and very easy. What i would like to see is just to have most of the distributions on Pypi Me too. I just said that I don't think we should require it. Help people to make it easier in various ways I'm all for. -- Lennart Regebro: http://regebro.wordpress.com/ Python 3 Porting: http://python-incompatibility.googlecode.com/ +33 661 58 14 64 ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Python people want CPAN and how the latter came about
On Wed, Dec 23, 2009 at 10:32, smuel...@cpan.org smuel...@cpan.org wrote: I have to say that I vastly prefer not to have any authorization and allow anyone to release anything in any namespace. But then I am getting fanatically anarchical in these issues. You can not organize freedom. But you have to. No, I really mean it whan I say you can't. And you never *have* to do the impossible, and trying just leads to problems. I realize this is a matter of attitude, but if the sentence I want CPAN means I want restrictions and controls and people preventing others from uploading stuff, then they are misguided. What you're saying here means you virtually throw away the ability to do anything useful with namespace meta data. *shrugs* Namespaces has no metadata in Python. They are just namespaces, no more, no less. The names of your *packages* is protected on PyPI. But several people can use the same *namespace*. Ie, nobody can upload a Twisted package, except those who have the permission. But people can upload a zope.my.own.package, even though the namespace zope is already used by other packages. Think about it like this: If you install any module from CPAN (and only the valid ones end up in the index), you can use all of them in the same application. If module A and B could both implement Config::Parser, then you couldn't use both of them at the same time. This would be true for Python too. But Python doesn't try to pretend that all the packages that exist are some sort of standard library, and therefore don't try to put them all into one sort of hierarchical organized namespace. And to be honest I don't see the point of doing that. Still. We allow for a lot of creative freedom. We just don't want a random newbie upload a shiny package DB which implements his idea of a database interface when it's really the name of the package that implements the Perl debugger. He can still upload. The file will be accessible in his CPAN directory. Users can install it from the CPAN shell as install NEWBIEUSERID/DB-1.00.tar.gz, but not as install DB or $ cpan DB. I see. But IMO Perl then there starts out with trying to organize freedom from the start, and that then leads to the above problem that newbies can come up and mess up this so called organized freedom, which means you need to restrict it even more by having people control and restrict the namespaces, etc, etc. You end up having to have more organisation to fix the problems your organisation caused. This is, without trying to be rude or anything, the fate of all bureaucracies. This is a safeguard against insanity and it's the thing that means that you can trust cpan PAR to always install the Perl Archive Toolkit that was released by Autrijus Tang, Roderich Schupp, or myself (we share co-maintenance). It's never going to be some random junk. And that you auto-register a namespace on upload is the guarantee. And obviously on PyPI, it's first come first serve as well. But nobody would call a db package db if one already exists. Why would they do that? What's the point? Why would I make a completely new package called Twisted for example? There already is one. It's just a mindset that is completely incomprehensible to me. I expect what I would call creative freedom, you would call total anarchy. PS: Let me comment on another post of yours quickly. No. In the Perl community, the name CPAN doesn't yield confusion. It's just a way to refer to the whole ecosystem OK, that's not how it sounded in your first post, thanks for clarifying. -- Lennart Regebro: http://regebro.wordpress.com/ Python 3 Porting: http://python-incompatibility.googlecode.com/ +33 661 58 14 64 ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Python people want CPAN and how the latter came about
On Dec 24, 2009, at 5:50 PM, Lennart Regebro wrote: easy_install is a command, basically a wrapper around setuptools install functionality. I doubt many scripts would use it in any more complex way than calling it, in which case moving to pip is a matter of replacing the command. Yes, I'm saying that it would be smart to implement a replacement easy_install with pip doing the work in the background instead of setuptools so that no changes are necessary. I imagine it would be useful to have a facade because easy_install is familiar and ubiquitous and shouldn't be left rotting in place as pip takes over the world and becomes the SDPTJLCOB any more than Distribute left the rest of setuptools exposed to cause problems once Distribute was installed. I don't even understand that sentence, let alone what you are trying to say. Distribute is a setuptools fork. Distribute is a setuptools fork and is intended to replace it in the Python ecosystem. In order to accomplish this, it transparently replaces setuptools. Pip is not an easy_install fork. I don't really see the parallell. The parallel is that as Distribute transparently replaces setuptools, the new easy_install, with pip doing the behind-the-scenes work, should transparently replace the existing easy_install. # easy_install Twisted should still work just as it does now, it'll just do a better job of it with the new machinery running in the background. S ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Python people want CPAN and how the latter came about
On 12/24/2009 12:33 AM, Martin v. Löwis wrote:
1/ Missing packages (eg: Twisted is not there); which is why
easy_install/pip had to resolve to scrapping project webpages for
guessing download links. In CPAN, almost all module authors upload their
sources via PAUSE.
How do you propose to change that?
Bt requiring authors to upload sdists + metadata now onwards.
'sdist upload' would upload the sdist to /packages/source and also have
PyPI generate the metadata from the uploaded sdist. Eg:
/packages/source/f/foo-0.1.tar.gz
/packages/source/f/foo-0.1.tar.gz.PKG-INFO
/packages/source/f/foo-0.1.tar.gz.requires.txt (optional)
If the author prefers to use the web browser to upload, then their sdist
must contain setup.py and PKG-INFO (w/ at least 'name' and 'version').
I would leave the existing setup as it is .. so easy_install/pip would
continue to install packages like Twisted, ClientCookie that, at the
moment, do not have their sdists uploaded in PyPI.
[Martin]
I think it should be the choice of the package authors whether they
upload their software to the central repository, or to their own home
page.
[Ben]
Why do you think that should continue? Some of the costs of that
inconsistency have already been described in this thread. What are the
benefits to PyPI users of this inconsistency, and are we sure that the
benefits outweigh the costs?
[Martin]
The benefits are not to the package users, clearly.Instead, they are
to the package authors, which don't have to change their release
processes (as also described in this thread).
Is it because of this benefit to package authors that we are withholding
the implementation of a simple archive that would: 1) simplify the tools
to no rely on adhoc web scrapping, 2) reduce the downtime for users by
rsync/ftp mirroring, 3) have package sources mirrored so project owners
do not have to worry about downtime of their servers. 4) enable
proliferation of third-party tools like CPAN?
2/ No metadata: When only source tarballs are stored
[pypi.python.org/packages/source/P/Pylons/], what is the reliable way to
a) get the source for latest version,
Extract a version number from each file name, and sort the versions,
then use the largest (which is 0.9.7 at the moment).
b) get the source for a particular version?
Put the version number into the file name, and access the resulting
file.
This assumes that source tarballs are named in a particular format, such
as: ${name}-${version}.tar.gz .. which need not always be the case (I've
come across packages whose source distribution is simply named
latest). This is why we rely on PKG-INFO to retrieve the version.
The reason for asking the two questions above, as pointed out to Lennart
in other email, is this:
Perhaps if I were to rephrase the question, it would be clear this
time: When only source tarballs are stored
[pypi.python.org/packages/source/P/Pylons/], what is the reliable way to
a) get the source for the latest version (when the /P/Pylons contains
multiple versions -- in other words, how do I find the later version in
first place?), b) get the source for a particular version (**without**
having to construct the filename, or do a adhoc matching with filenames
to guess that Pylons-1.2.3.tar.gz corresponds to version 1.2.3)? If the
answer is to do a HTTP GET first, then please see the next response.
[emphasis added]
My next response was:
As the CPAN .meta example was given in the context of having a simple
directory structure that can be mirrored using existing tools like
rsync, what I was pointing out is the lack of such an implementation,
not the functionality itself (which, as you have shown, is currently
supported by doing a HTTP GET that would return a XML content -- not
something that is rsync-friendly).
To explain: it is all about making the PyPI data (sdist + metadata)
mirror-friendly / rsync-friendly.
The former is more of a community issue. Often Python package authors
are not using `sdist upload` (whereas this seems to be the convention in
the Perl world).
My guess is that this is enforced by the tools. If they don't upload
to PAUSE, CPAN.pm won't be able to download it.
Now, you are free to build a tool that enforces the same restriction.
I would doubt that people would use it, since it couldn't install
many packages.
My original intention is to have a simple archive that can be mirroed
using rsync.
What this means is that PyPI has to serve the purpose of being a central
package repository (like CPAN) by a) disallowing mere listings (without
sources) and requiring sources to be stored in the server, b) storing
the metadata along with the sources (so anyone processing it wouldn't
have to extract the source and rely on a PKG-INFO file - which may or
may not exist).
If you want to retrieve the metadata for a specific version without
XML-RPC, you can access
http://pypi.python.org/pypi?:action=doapname=Pylonsversion=0.9.7
As
Re: [Distutils] Python people want CPAN and how the latter came about
On 12/23/2009 10:42 PM, Lennart Regebro wrote: On Wed, Dec 23, 2009 at 23:28, Sridhar Ratnakumar sridh...@activestate.com wrote: I suggested PyPI to disallow mere project listings (without sources) and require sources to be stored in the server. One way to achieve this is requiring package authors to use the `sdist upload` toolchain Which only means the packages who now is not uploaded wouldn't even be listed on PyPI, which is not an improvement. We can do this only for the new projects/uploads. Existing data can be left as it is for backwards compatibility. Here's my updated proposal: [Sridhar's proposal] How do you propose to change that? By requiring authors to upload sdists + metadata now onwards. 'sdist upload' would upload the sdist to /packages/source and also have PyPI generate the metadata from the uploaded sdist. Eg: /packages/source/f/foo-0.1.tar.gz /packages/source/f/foo-0.1.tar.gz.PKG-INFO /packages/source/f/foo-0.1.tar.gz.requires.txt (optional) If the author prefers to use the web browser to upload, then their sdist must contain setup.py and PKG-INFO (w/ at least 'name' and 'version'). I would leave the existing setup as it is .. so easy_install/pip would continue to install packages like Twisted, ClientCookie that, at the moment, do not have their sdists uploaded in PyPI. ... While the specific case mentioned above (metadata for a specific or the latest version of a package) uses HTTP GET and XML, generally speaking .. to get a) the list of recently releases, b) list of all versions of a package, one has to use the XmlRpc API methods `changelog` and `package_releases` respectively. Well, maybe pure http versions of those would help, Nope, it matters not whether the metadata can be retrived via a simple HTTP GET or XmlRpc. but on the other hand, if you automate it, why not use xml-rpc? Because my intention is to have a simple mirror archive (files, directories) that can be mirrored using tools like rsync. As often as the mirror sites would update their content (i.e., one or more times a day). I meant that most of the third-party apps would only need the metadata, or? I might be wrong, I haven't written any yet. :-) The automated documentation that was discussed would only need the source packages. Metadata is definitely needed. Otherwise, I'd have to extract the tarball of each and every release of a pacticular package, in order to even find their version number (it is unreliable to parse the filename to get version number). As for the sdists, the following tools would need it: testing service, quality ratings, thirdparty package managers (enstaller, PyPM) .. and not to mention the various mirror sites. -srid ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Python people want CPAN and how the latter came about
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Ben Finney wrote: Martin v. Löwis mar...@v.loewis.de writes: Ben Finney wrote: That isn't a good argument. By the same logic, PyPI should not reject *any* upload, to avoid “forcing” uploaders to do extra work. PyPI's rejection of certain uploads is primarily to prevent spam from being uploaded. Am I wrong, then, in thinking that PyPI will reject an upload with malformed metadata? To my understanding, this discussion is about arguing whether an upload that is missing the package should be rejected by PyPI. In the language of distutils, recording the metadata about a release is registration; registration and upload happen in separate transactions for relatively good reasone: - - There is not a one-to-one relationsihp between metadata sets (registrations) and distributions (e.g., source dist + windows MSI). - - PiPI only has to parse the PKG-iNFO file, and doesn't need to unpack a distribution to look for it. - - Package authors can choose to keep the actual distribution files elsewhere, e.g., to allow for payment, etc.: one might debate the desirability of such a use case for the community at large, but it is certainly part of the historical use of the cheeseshop. Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAks0RYUACgkQ+gerLs4ltQ52GQCdFhOUpq9c4hcN7vHiFGaOfqm0 ufUAoMc5FtMKyd5GZI2WySsoUgcgbzj9 =dib1 -END PGP SIGNATURE- ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Python people want CPAN and how the latter came about
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 kiorky wrote: Lennart Regebro a écrit : On Thu, Dec 24, 2009 at 12:28, kiorky kio...@cryptelium.net wrote: Why forcing them ? How about a cronjob or something like that which find packages without distribution and get from there all distributions possible on their relative homepages and mirror them on Pypi ? Somebody (including a bot) uploading a package is quite different from *requiring* uploads, IMO. But uploading to PyPI is for 99.999% of packages so easy that if it's not done there may be a reason why they don't want to. Unless they don't know how to do. We may choose to ignore that, but we can only do it for licenses we know allow it, 99% of pypi packages. which again means that there may be packages listed on PyPI that are not hosted on PyPI. I personally don't understand why third-party services would need this. That's quite out of scope which what i wanted to spot but to answear: The only problem is with licenses which constrain redistribution. We can have logic around trove classifiers to filter them out. If the authors miss the trove, that's their fault. What i would like to see is just to have most of the distributions on Pypi because by experience, pypi get less offline that thirdparty mirrors, and i can set mirrors of pypi easyly. I would say that having a package author *not* upload the distributions is their right, but I would likely avoid using such a package, just on that basis. Note that I build per-project mirrors of the pacakges I use anyway, in part not to depend on either PyPI or other download sources for supporting apps in production: I just prefer to use only freely-distributable software. Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAks0RnIACgkQ+gerLs4ltQ61FACgnE0jgLx5jaHTQzMofH+VyT5I YBMAn079ghQ2lLcOwUraISeew81Pe9jt =gUew -END PGP SIGNATURE- ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Python people want CPAN and how the latter came about
On 12/24/2009 3:00 AM, Martin v. Löwis wrote: Some reasons to have PyPI host packages have already been mentioned in this thread: it makes mirroring easier, and it makes it easier for individuals to build new services (web sites primarily) that present new interfaces to the Python package collection. Mirroring for its own sake is some use, but being able to grab the entire Python package repository easily from a single source is valuable for the second goal, that of furnishing the foundation (shoulders of giants and all that) for those with vision (and round tuits) to take the next step. That is fairly easily possible today, even without everybody uploading all files. It isn't easy*per se*, but needs a lot of code. However, this code has already been written, and using it is fairly easy. If I wanted to host a site that (e.g.) indexed Python modules from PyPI by module (not package) name, and extracted and provided the documentation in HTML format, from what I've been reading I'd have to build a scraper or XMLRPC tool to walk PyPI, and then for each package, download it from another site (that may not have the uptime or scalability of PyPI), a nontrivial burden on aspiring visionaries that just want to build an addition and then go have a beer and discuss further improvements. Not at all. You would just use one of the ten or so packages that already do precisely that, and use it. (As a point of practical interest, what_would_ be the most efficient way to download the entire set of Python modules listed on PyPI? A search comes up with z3c.pypimirror, http://pypi.python.org/pypi/z3c.pypimirror; is this the standard tool?) There are a number of other mirroring tools, such as EggBasket and collective.eggproxy. For mirroring the whole index, pypimirror is probably the best starting point. For starters, this is how z3c.pypimirror works: 1) Initial fetch: Traverse http://pypi.python.org/simple/AOPython and for each package's index.html, for each link in it, if the link is a) an actual sdist tarball, download it, b) an external link (project homepage), go scrap the homepage to find any download links (.tar.gz, .zip) and download it. 2) [run this every day] Update for last 24hrs: Use XmlRpc `changelog` method that returns recently released packages in last 24 hrs; and redo the above operation for these updated packages. It is unreliable [bugs.launchpad.net/pypi-mirror/+bug/386143] and lacks the pre-extracted metadata. I wouldn't call it a mirror tool, for it is not an exact copy of PyPI data[1]. I doubt that proliferation of mirror sites / thirdparty tools can happen with anything but simple rsync/ftp based archives. -srid *** [1] In computing, a mirror is an exact copy of a data set. On the Internet, a mirror site is an exact copy of another Internet site. ... ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Python people want CPAN and how the latter came about
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Tarek Ziadé wrote: On Thu, Dec 24, 2009 at 11:20 AM, kiorky kio...@cryptelium.net wrote: [..] That said, some people have expressed the desire to be able to interact with PyPI through SSH so they could drop the basic authentication and use their keys when registering/uploading. But that was not related to the size of files. Both i think, it's easier to use the transfer programs that we are used to use to transfer things. And ssh is damn good for that, easy to setup, secure and so on. But it's only butter, in fact i am just happy with sdist upload. Although SSH is quite a heavy development on PyPI side, it means we would have to implement an SSH server. (like Zope did I think for their development server, using Paramiko IIRC) cvs.zope.org / svn.zope.org (same machine) run a stock sshd: they use the command= prefix on users' pubkeys to limit what that key can be used to do (only SVN / CVS operations for any non-admin users). Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAks0SdcACgkQ+gerLs4ltQ795gCg2FDY353KLgJTXGgqz0CHQ1rE 2/UAoI7kUnEGxF2omBqh58JKSsFKEGVr =yRkp -END PGP SIGNATURE- ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Python people want CPAN and how the latter came about
On Fri, Dec 25, 2009 at 05:39, Sridhar Ratnakumar sridh...@activestate.com wrote: Is it because of this benefit to package authors that we are withholding the implementation of a simple archive that would: 1) simplify the tools to no rely on adhoc web scrapping There are better ways to do that. 2) reduce the downtime for users by rsync/ftp mirroring This is true, but the idea to upload them by robots is preferable in my opinion. Again it's a difference between trying to force other people to behave to your expectations vs trying to make it easier for others to behave to your expectations. 3) have package sources mirrored so project owners do not have to worry about downtime of their servers. That's *their* problem. If they don't want to upload, then they don't want to upload. 4) enable proliferation of third-party tools like CPAN? That won't help. On Fri, Dec 25, 2009 at 05:48, Sridhar Ratnakumar sridh...@activestate.com wrote: On 12/23/2009 10:42 PM, Lennart Regebro wrote: On Wed, Dec 23, 2009 at 23:28, Sridhar Ratnakumar sridh...@activestate.com wrote: I suggested PyPI to disallow mere project listings (without sources) and require sources to be stored in the server. One way to achieve this is requiring package authors to use the `sdist upload` toolchain Which only means the packages who now is not uploaded wouldn't even be listed on PyPI, which is not an improvement. We can do this only for the new projects/uploads. Existing data can be left as it is for backwards compatibility. Which only means the packages who in the future will not be uploaded will not even be listed on PyPI, which is not an improvement. Nope, it matters not whether the metadata can be retrived via a simple HTTP GET or XmlRpc. OK. Then you have two proposals: 1. Require uploading, which is a bad idea and 2. Making it easier to mirror the metadata, which seems reasonable, assuming it's currently hard. :) Metadata is definitely needed. Otherwise, I'd have to extract the tarball of each and every release of a pacticular package, in order to even find their version number (it is unreliable to parse the filename to get version number). Yes, but it's not particularly unreliable to compare the filename to see if it had been handled before. You don't even need to parse the version number for most services that work on the tarballs. As for the sdists, the following tools would need it: testing service, quality ratings, thirdparty package managers (enstaller, PyPM) .. and not to mention the various mirror sites. Yes, but since thay have the source package, and will have to unpack it and build the packages anyway, they also have the metadata. -- Lennart Regebro: http://regebro.wordpress.com/ Python 3 Porting: http://python-incompatibility.googlecode.com/ +33 661 58 14 64 ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Python people want CPAN and how the latter came about
On Fri, Dec 25, 2009 at 3:27 PM, Lennart Regebro rege...@gmail.com wrote: This is true, but the idea to upload them by robots is preferable in my opinion. Again it's a difference between trying to force other people to behave to your expectations vs trying to make it easier for others to behave to your expectations. Is there any hard data to back up the idea that making some things mandatory when registering/uploading to Pypi is detrimental to Pypi's goal, or is it just opinion ? It is very difficult for me to understand the rationale for this. This is specific to Pypi AFAIK (compared to CRAN, hackage, etc...), and the advantages of making it mandatory are obvious and hard facts (easy to mirror, easy to do basic consistency checks, easy to interoperate), whereas the downsides are far from obvious. cheers, David ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Python people want CPAN and how the latter came about
On Wed, Dec 23, 2009 at 02:56, David Lyon david.l...@preisshare.net wrote: On Wed, 23 Dec 2009 01:01:11 +0100, Lennart Regebro rege...@gmail.com wrote: OK, so in the Perl community there is apparently a lot of confusion on what CPAN is. CPAN is plain and simple. There is no confusion, because there is just one 'brand-name' for the whole kit and caboodle. Packaging in the perl world just goes under the name CPAN. That's not what Steffen Mueller said. However, a Perl user (speaking for myself) would typically know nothing about conversion of different package formats, because as far as I am aware the user isn't exposed to that level of complexity. There is no .tar.gz, .zip, .bz2, .exe, .msi or .egg concept of packages in perl. And having to pick one.. that may or may not be right for your configuration. So your usecase, that a Windows user refuses to install anything else than .msi files is solved in CPAN by the user not installing using CPAN. Well, that's handy. And the Python situation is worse how, exactly? Call a perl user a pampered pooch by all means. But all they know is that if they need a module.. then they use CPAN. Right, and in Python neither easy_installer nor pip is included by default. That's obviously a big plus for Perl. After that your argumentation becomes self-contradictory and confused and reverts in to the usual distutils rant, which still isn't constructive. I'd also like an installer to be included in Python core, but there are very good reasons for only including things that are stable and almost unanimously accepted. And although pip looks like it's going to get unanimously accepted it hasn't been yet. And obviously a very vocal minority that constantly does nothing but waste their and others time by complaining on distutils but not doing anything to fix it or replace it isn't helping either. On Wed, Dec 23, 2009 at 07:37, David Lyon david.l...@preisshare.net wrote: Those words are entirely yours. Not those who are able to explain how and why, with or without the rant. I do hope you are not including yourself amongst those who are able to explain how and why. One very important thing to remember is just who started this request to look into 'cpan' style packaging in the first place. If I'm not wrong, it came right from the top. So go hassle whoever that might be. But even then, that's just a response to calls by plain ordinary users in the field. That is the most ridicolous excuse I've heard in a long time. People are saying to Guido that Python needs CPAN and wonders what that means. You come with you usual anti distutils rant, I ask *you* what the connection is, and you tell med to go hassle Guido. Pathetic. Honestly. With CPAN, they would never label discussion or feedback as 'rant'. No, but they would label rants as rants, and what you do is ranting. There is a bit of constructive feedback from you but there sure is no discussion. I refer you to reread Steffens original post. It is very helpful. I completely agree. Maybe you should read it and try to figure out WHY it was helpful, when your answers aren't? No one person is able to do this. That being, redo distutils. You are not the only one who says it needs to be rewritten from scratch. If you are saying that the few people that exists that agree with you about this *also* is not enough, then you are saying that the community does not agree with you that it needs to be rewritten. And if python management aren't united on the need for it, doing a distutils replacement becomes an even weaker proposition. So you are saying that you think distutils must be replaced, and that the only persons who understand why it needs to be replaced are not willing or able to do or even start the work. PEP-345 has been open since 2005. In CPAN it just would have been a beer fight and have been over in a weekend. Maybe they have less people who complain, and use up their and other peoples energies by ranting and complaining instead of coding. In any case, I take your answer as a definite statement from your side that you will not help neither fix nor replace distutils. And then I honestly see no reason why I should listen to you anymore. -- Lennart Regebro: http://regebro.wordpress.com/ Python 3 Porting: http://python-incompatibility.googlecode.com/ +33 661 58 14 64 ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Python people want CPAN and how the latter came about
2009/12/23 David Cournapeau courn...@gmail.com: which is why I am trying to get some agreement on at least some low level mechanisms which can be shared between tools. The exact thing you already complain last time this was discussed. And then I also asked a question, which I have asked many times: Will a replacement of distutils follow the distribution/install PEPs that are being discussed and designed at the moment? But it's good to see a start on the effort you claim is necessary. Thank you. -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Python people want CPAN and how the latter came about
On Wed, Dec 23, 2009 at 5:12 AM, David Cournapeau courn...@gmail.com wrote: On Tue, Dec 22, 2009 at 10:35 PM, Tarek Ziadé ziade.ta...@gmail.com wrote: When you say which could be solved relatively easily I suggest that you take the time to add concise and precise proposals in bugs.python.org so I can work on them. Technically, it is easy. Only have two mechanisms for data files: one for installed data files, and one for extra source files (as done in automake for example): - Extra files only need to be listed (and included in sdist) - Install data files need a target directory. One of the problem with distutils here is that you can only hardcode paths - in my own packaging solution, I use $path variables so that any path defined internally can be reused ($bindir, etc...); something similar could be defined in distutils. distutils uses install schemes with variables too ($base, etc), that are expanded at installation time. and differs depending on the options you pass to the install command. (look at the command/install module) There's only one target directory for all files describe in data_file though, but we could work on extending this if you make a feature request in the bug tracker, ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Python people want CPAN and how the latter came about
On Wed, Dec 23, 2009 at 7:28 PM, Tarek Ziadé ziade.ta...@gmail.com wrote: On Wed, Dec 23, 2009 at 5:12 AM, David Cournapeau courn...@gmail.com wrote: On Tue, Dec 22, 2009 at 10:35 PM, Tarek Ziadé ziade.ta...@gmail.com wrote: When you say which could be solved relatively easily I suggest that you take the time to add concise and precise proposals in bugs.python.org so I can work on them. Technically, it is easy. Only have two mechanisms for data files: one for installed data files, and one for extra source files (as done in automake for example): - Extra files only need to be listed (and included in sdist) - Install data files need a target directory. One of the problem with distutils here is that you can only hardcode paths - in my own packaging solution, I use $path variables so that any path defined internally can be reused ($bindir, etc...); something similar could be defined in distutils. distutils uses install schemes with variables too ($base, etc), that are expanded at installation time. and differs depending on the options you pass to the install command. (look at the command/install module) To be clear: I am talking about the POV of the setup.py writer here. AFAIK, those $path variables are not available in this case: when using data_files, you only have the choice between using absolute files or relative to package path. That's why you had to advise one poster to move his files into a package in one recent email, and that the only solution to another poster was to create a new command (to access those $path vars). The notion of data vs package data does not make much sense IMHO. All current methods should be deprecated, to the profit of the only difference that matters: installed vs non-installed data files. cheers, David ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Python people want CPAN and how the latter came about
On Wed, Dec 23, 2009 at 1:14 PM, David Cournapeau courn...@gmail.com wrote: On Wed, Dec 23, 2009 at 7:28 PM, Tarek Ziadé ziade.ta...@gmail.com wrote: On Wed, Dec 23, 2009 at 5:12 AM, David Cournapeau courn...@gmail.com wrote: On Tue, Dec 22, 2009 at 10:35 PM, Tarek Ziadé ziade.ta...@gmail.com wrote: When you say which could be solved relatively easily I suggest that you take the time to add concise and precise proposals in bugs.python.org so I can work on them. Technically, it is easy. Only have two mechanisms for data files: one for installed data files, and one for extra source files (as done in automake for example): - Extra files only need to be listed (and included in sdist) - Install data files need a target directory. One of the problem with distutils here is that you can only hardcode paths - in my own packaging solution, I use $path variables so that any path defined internally can be reused ($bindir, etc...); something similar could be defined in distutils. distutils uses install schemes with variables too ($base, etc), that are expanded at installation time. and differs depending on the options you pass to the install command. (look at the command/install module) To be clear: I am talking about the POV of the setup.py writer here. AFAIK, those $path variables are not available in this case: when using data_files, you only have the choice between using absolute files or relative to package path. That's why you had to advise one poster to move his files into a package in one recent email, and that the only solution to another poster was to create a new command (to access those $path vars). We are discussing these options as a matter of fact, in PEP 376. Maybe you missed the mails related to that. See: http://wiki.python.org/moin/Distutils/DiscussionOverview feel free to comment and help, ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Python people want CPAN and how the latter came about
On 12/22/2009 10:15 AM, Lennart Regebro wrote: Another point that I really like about the service is that the distribution pages provide links to many other related services that are run by other volunteers. Take for examplehttp://search.cpan.org/dist/PAR-Repository-Client/ There is a link to cpanforum, specifically the relevant discussion forum for the module at hand. It shows a link to the bug/request tracker with the number of open bugs, the one next to it will display a nice hierarchical (recursive) list of dependencies. Right, those third-party services doesn't exist for PyPI. The reason why PyPI does not have such third-party services - I think - is that it lacks the CPAN like simple directory structure that can be easily mirrored using ftp/rsync, to wit: [Steffen Mueller] My thesis is that the huge success of the CPAN has been facilitated by two factors[2]. The first is simplicity. When Jarkko Hietaniemi originally came up with it, the CPAN was (and mostly still is) just an FTP archive with a by-author directory structure that is mirrored many times. http://pypi.python.org/simple does not qualify here, for there is no reliable way one can get, say, the source tarball for package 'Foo' and version '0.3'. For an example, see: http://pypi.python.org/simple/Pylons/ The only way to get, say, version 0.9.7's source code is to scrap that page and get the link 0.9.7_home_page (assuming Pylons-0.9.7.tar.gz is not already available -- which is entire possible for packages in PyPI: missing source tarballs) and further scrap it for download links. Not very friendly for third-party sites to simply update PyPI packages on daily basis (where rsync works very well). Simplicity is nowhere to be found. One solution I can think of is this: make PyPI only do the job of PAUSE as it does for CPAN; and implement a CPAN like simple directory structure to store packages; make PyPI use that as the package data store; deprecate the XmlRpc interface and rely on simple index files - such as http://www.cpan.org/modules/01modules.mtime.html - instead (consequently rethink PEP 381). This is partially implemented for PyPM at ActiveState and I'd be willing to contribute my time towards doing this for PyPI (if at all there is an interest). -srid ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Python people want CPAN and how the latter came about
One solution I can think of is this: make PyPI only do the job of PAUSE as it does for CPAN; and implement a CPAN like simple directory structure to store packages; make PyPI use that as the package data store I don't know what PAUSE is, but I think there is what you want at http://pypi.python.org/packages/ deprecate the XmlRpc interface and rely on simple index files - such as http://www.cpan.org/modules/01modules.mtime.html - instead (consequently rethink PEP 381). This is partially implemented for PyPM at ActiveState and I'd be willing to contribute my time towards doing this for PyPI (if at all there is an interest). I don't think I understand what it is that you want, so I don't know whether I'm interested. Regards, Martin ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Python people want CPAN and how the latter came about
On 12/23/2009 12:18 PM, Martin v. Löwis wrote: One solution I can think of is this: make PyPI only do the job of PAUSE as it does for CPAN; and implement a CPAN like simple directory structure to store packages; make PyPI use that as the package data store I don't know what PAUSE is, but I think there is what you want at PAUSE is the web management interface that allows one to upload/manage packages in CPAN. http://pause.perl.org/ -- similar to PyPI (minus the storage part). http://pypi.python.org/packages/ deprecate the XmlRpc interface and rely on simple index files - such ashttp://www.cpan.org/modules/01modules.mtime.html - instead (consequently rethink PEP 381). This is partially implemented for PyPM at ActiveState and I'd be willing to contribute my time towards doing this for PyPI (if at all there is an interest). I don't think I understand what it is that you want, so I don't know whether I'm interested. What /packages/source/ lacks is: 1/ Missing packages (eg: Twisted is not there); which is why easy_install/pip had to resolve to scrapping project webpages for guessing download links. In CPAN, almost all module authors upload their sources via PAUSE. 2/ No metadata: When only source tarballs are stored [pypi.python.org/packages/source/P/Pylons/], what is the reliable way to a) get the source for latest version, b) get the source for a particular version? In CPAN [cpan.org/modules/by-module/AppConfig/ABW/], each tarball has a .meta file describing the module metadata (similar to PKG-INFO). I don't want XmlRpc, but just files/directories (note simplicity in Steffen's post). The former is more of a community issue. Often Python package authors are not using `sdist upload` (whereas this seems to be the convention in the Perl world). The later is what is most relevant to PyPM (or any thirdparty service/tool). 1 and 2 combined makes it possible for anyone willing to write a third-party PyPI functionality to simply rsync the entire PyPI store and begin implementing the desired features like *.cpan.org (eg: test.pypi.org, quality.pypi.org, etc..) What this means is that PyPI has to serve the purpose of being a central package repository (like CPAN) by a) disallowing mere listings (without sources) and requiring sources to be stored in the server, b) storing the metadata along with the sources (so anyone processing it wouldn't have to extract the source and rely on a PKG-INFO file - which may or may not exist). Tools would consequently use the new /packages/sources store (with full metadata and all registered package sources) without having to resolve to webpage scrapping hacks. -srid PS: Our internal mirror is similar to /packages/sources except it (a) also contains external packages (eg: Twisted), (b) and pre-extracts PKG-INFO and requires.txt out of the source tarballs. Everyday, it uses PyPI's XmlRpc interface to re-download (using easy_install scrapping logic) the recently releases packages ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Python people want CPAN and how the latter came about
On 12/23/2009 1:19 PM, Sridhar Ratnakumar wrote: What /packages/source/ lacks is: 1/ Missing packages (eg: Twisted is not there); which is why easy_install/pip had to resolve to scrapping project webpages for guessing download links. In CPAN, almost all module authors upload their sources via PAUSE. 2/ No metadata: When only source tarballs are stored [pypi.python.org/packages/source/P/Pylons/], what is the reliable way to a) get the source for latest version, b) get the source for a particular version? In CPAN [cpan.org/modules/by-module/AppConfig/ABW/], each tarball has a .meta file describing the module metadata (similar to PKG-INFO). I don't want XmlRpc, but just files/directories (note simplicity in Steffen's post). 3/ Indices such as http://www.cpan.org/modules/01modules.mtime.html (or TXT files) to get a) recently released packages, b) list of release versions for a particular package, and so on (which would obviate the XmlRpc interface) -srid ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Python people want CPAN and how the latter came about
On Wed, Dec 23, 2009 at 20:24, Sridhar Ratnakumar sridh...@activestate.com wrote: The reason why PyPI does not have such third-party services - I think - is that it lacks the CPAN like simple directory structure that can be easily mirrored using ftp/rsync, to wit: Nah, you can do that via /packages/, there is also an API to get the metadata for a package. I think in general it's not an API problem. I think it's partly a problem that nobody has thunk the thought. I think the idea of a site with automatically generated documentation for *every* package is interesting. But I don't have time to work on that right now. Talk to me again in six months, then I might have time for another free-time project. :) 1/ Missing packages (eg: Twisted is not there) The Twisted guys do not upload their packages to PyPI. I think that's a mistake, but it's hardly PyPI's fault. There is no law saying you have to use CPAN either. 2/ No metadata: When only source tarballs are stored [pypi.python.org/packages/source/P/Pylons/], what is the reliable way to a) get the source for latest version Download it from the above location. b) get the source for a particular Download it from the above location. version? In CPAN [cpan.org/modules/by-module/AppConfig/ABW/], each tarball has a .meta file describing the module metadata (similar to PKG-INFO). http://pypi.python.org/pypi?:action=doapname=Twisted%20Mailversion=9.0.0 This is not a problem about missing API or functionality, but that you don't know about it. In the last case that link exists at the bottom of every package page. And you see how it works. don't want XmlRpc, but just files/directories (note simplicity in Steffen's post). It's not XML-RPC because the metadata file is in XML-format. But yes, you can't duplicate both the files and the metadata in one go, you have to do it separately. But that then begs the question: How often do you need to do both? -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Python people want CPAN and how the latter came about
On Wed, Dec 23, 2009 at 10:08 PM, Tarek Ziadé ziade.ta...@gmail.com wrote: On Wed, Dec 23, 2009 at 1:14 PM, David Cournapeau courn...@gmail.com wrote: On Wed, Dec 23, 2009 at 7:28 PM, Tarek Ziadé ziade.ta...@gmail.com wrote: On Wed, Dec 23, 2009 at 5:12 AM, David Cournapeau courn...@gmail.com wrote: On Tue, Dec 22, 2009 at 10:35 PM, Tarek Ziadé ziade.ta...@gmail.com wrote: When you say which could be solved relatively easily I suggest that you take the time to add concise and precise proposals in bugs.python.org so I can work on them. Technically, it is easy. Only have two mechanisms for data files: one for installed data files, and one for extra source files (as done in automake for example): - Extra files only need to be listed (and included in sdist) - Install data files need a target directory. One of the problem with distutils here is that you can only hardcode paths - in my own packaging solution, I use $path variables so that any path defined internally can be reused ($bindir, etc...); something similar could be defined in distutils. distutils uses install schemes with variables too ($base, etc), that are expanded at installation time. and differs depending on the options you pass to the install command. (look at the command/install module) To be clear: I am talking about the POV of the setup.py writer here. AFAIK, those $path variables are not available in this case: when using data_files, you only have the choice between using absolute files or relative to package path. That's why you had to advise one poster to move his files into a package in one recent email, and that the only solution to another poster was to create a new command (to access those $path vars). We are discussing these options as a matter of fact, in PEP 376. I don't see data files mentioned in PEP 376, nor how the PEP is related to this discussion. David ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Python people want CPAN and how the latter came about
On Wed, Dec 23, 2009 at 11:12 PM, David Cournapeau courn...@gmail.com wrote: [..] We are discussing these options as a matter of fact, in PEP 376. I don't see data files mentioned in PEP 376, nor how the PEP is related to this discussion. The PEP contains what has been already discussed. As said earlier, take a look at http://wiki.python.org/moin/Distutils/DiscussionOverview for what is being currently discussed. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Python people want CPAN and how the latter came about
On Dec 23, 2009, at 4:33 PM, Lennart Regebro wrote: 1/ Missing packages (eg: Twisted is not there) The Twisted guys do not upload their packages to PyPI. I think that's a mistake, but it's hardly PyPI's fault. There is no law saying you have to use CPAN either. For what it's worth, we don't upload because it's a big pain, and nobody cares anyway. It's a big pain because there are two ways to upload, and neither one works for us. We can't use 'setup.py upload' because we don't use 'sdist' to produce our tarball releases (although a discussion of why 'sdist' is insufficient is a topic for another post). The other way to upload, manually interacting with a form in a web browser, is annoying and as far as I know it is hostile to automation. Nobody cares because you can 'easy_install twisted' already, and you can find Twisted on the PyPI web page. It's not clear to us what benefits uploading to PyPI would have beyond that. If someone would like to give us a good reason to upload or explain how uploading might be made easy, maybe we'd start doing it :).___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Python people want CPAN and how the latter came about
Lennart Regebro a écrit : On Wed, Dec 23, 2009 at 20:24, Sridhar Ratnakumar sridh...@activestate.com wrote: The Twisted guys do not upload their packages to PyPI. I think that's The tiny 10MB upload is a blocker i think also. For example, for 'minitage.paste.extras', it's not hosted on pypi because of that. -- Cordialement, KiOrKY GPG Key FingerPrint: 0x1A1194B7681112AF signature.asc Description: OpenPGP digital signature ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Python people want CPAN and how the latter came about
On 12/23/2009 1:33 PM, Lennart Regebro wrote: On Wed, Dec 23, 2009 at 20:24, Sridhar Ratnakumar sridh...@activestate.com wrote: The reason why PyPI does not have such third-party services - I think - is that it lacks the CPAN like simple directory structure that can be easily mirrored using ftp/rsync, to wit: Nah, you can do that via /packages/, there is also an API to get the metadata for a package. I think in general it's not an API problem. It is indeed technically possible to do that with the PyPI XmlRpc API alone; but what I was referring to is enabling the mindset: a simple *self-contained* (i.e., without having to use an API to get metadata) directory structure that can simply be mirrored by using existing tools like rsync could *enable* developers interested in providing extending packaging functionality such as testing, quality measurements, documents, search, etc... to easily create such sites and maintain it. At least, this is what - I understand - happened in the Perl community. I think it's partly a problem that nobody has thunk the thought. I think the idea of a site with automatically generated documentation for *every* package is interesting. But I don't have time to work on that right now. Talk to me again in six months, then I might have time for another free-time project. :) 1/ Missing packages (eg: Twisted is not there) The Twisted guys do not upload their packages to PyPI. I think that's a mistake, but it's hardly PyPI's fault. There is no law saying you have to use CPAN either. Yes, as I said it is more of a community issue (than a PyPI one). What I also did mention was that because of this community issue, tools like easy_install/pip had to resolve to scrapping project webpages for guessing download links in an adhoc fashion. Also, further below the mail, I suggested PyPI to disallow mere project listings (without sources) and require sources to be stored in the server. One way to achieve this is requiring package authors to use the `sdist upload` toolchain which automatically creates a source tarball including metadata (in case one forgets to include it). 2/ No metadata: When only source tarballs are stored [pypi.python.org/packages/source/P/Pylons/], what is the reliable way to a) get the source for latest version Download it from the above location. b) get the source for a particular Download it from the above location. Perhaps if I were to rephrase the question, it would be clear this time: When only source tarballs are stored [pypi.python.org/packages/source/P/Pylons/], what is the reliable way to a) get the source for the latest version (when the /P/Pylons contains multiple versions -- in other words, how do I find the later version in first place?), b) get the source for a particular version (without having to construct the filename, or do a adhoc matching with filenames to guess that Pylons-1.2.3.tar.gz corresponds to version 1.2.3)? If the answer is to do a HTTP GET first, then please see the next response. version? In CPAN [cpan.org/modules/by-module/AppConfig/ABW/], each tarball has a .meta file describing the module metadata (similar to PKG-INFO). http://pypi.python.org/pypi?:action=doapname=Twisted%20Mailversion=9.0.0 This is not a problem about missing API or functionality, but that you don't know about it. In the last case that link exists at the bottom of every package page. And you see how it works. As the CPAN .meta example was given in the context of having a simple directory structure that can be mirrored using existing tools like rsync, what I was pointing out is the lack of such an implementation, not the functionality itself (which, as you have shown, is currently supported by doing a HTTP GET that would return a XML content -- not something that is rsync-friendly). don't want XmlRpc, but just files/directories (note simplicity in Steffen's post). It's not XML-RPC because the metadata file is in XML-format. While the specific case mentioned above (metadata for a specific or the latest version of a package) uses HTTP GET and XML, generally speaking .. to get a) the list of recently releases, b) list of all versions of a package, one has to use the XmlRpc API methods `changelog` and `package_releases` respectively. But yes, you can't duplicate both the files and the metadata in one go, you have to do it separately. But that then begs the question: How often do you need to do both? As often as the mirror sites would update their content (i.e., one or more times a day). As often as the (future) third-party sites update their PyPI content (source + metadata). One such user is the PyPM backend itself which at the moment uses the XmlRpc to pull data from PyPI on a daily basis. -srid ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Python people want CPAN and how the latter came about
On Wed, Dec 23, 2009 at 11:20 PM, Glyph Lefkowitz gl...@twistedmatrix.com wrote: On Dec 23, 2009, at 4:33 PM, Lennart Regebro wrote: 1/ Missing packages (eg: Twisted is not there) The Twisted guys do not upload their packages to PyPI. I think that's a mistake, but it's hardly PyPI's fault. There is no law saying you have to use CPAN either. For what it's worth, we don't upload because it's a big pain, and nobody cares anyway. It's a big pain because there are two ways to upload, and neither one works for us. We can't use 'setup.py upload' because we don't use 'sdist' to produce our tarball releases (although a discussion of why 'sdist' is insufficient is a topic for another post). The other way to upload, manually interacting with a form in a web browser, is annoying and as far as I know it is hostile to automation. Note that it wouldn't take long to override the upload command so it works independantly from any other *dist command. I could even add a --dist-file option to it so you can point an existing archive to push at PyPI, so running sdist or another *dist command wouldn't be mandatory anymore ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
