Re: [Python-Dev] Looking for VCS usage scenarios
Brett Cannon wrote: I have yet to have met anyone who thinks git is great while having used another DVCS as extensively (and I mean I have never found someone who has used two DVCSs extensively). It's entirely possible that there's only room for one VCS at a time in the average human brain. I have enough trouble keeping the intricacies of hg in my head from one session to the next. I wouldn't relish the idea of having to use another similar-but-not-quite-the-same tool at the same time. -- Greg ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Optionally using GMP to implement long if available
Martin v. Löwis wrote: On Windows, the GMP binaries would be incorporated into pythonxy.dll. This would force anybody providing a copy of pythonxy.dll to also provide the sources of GMP. I thought the suggestion was to provide a way of optionally compiling Python to use GMP. The standard Python distribution for Windows wouldn't be compiled that way, so it wouldn't have any GMP-related code in its dll. -- Greg ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Optionally using GMP to implement long if available
On Tue, Nov 4, 2008 at 07:32, Stephen J. Turnbull <[EMAIL PROTECTED]>wrote: > "Martin v. Löwis" writes: > > Stephen J. Turnbull wrote: > > > Thomas Wouters writes: > > > > > > > Neither of those (shipping sources or dynamically linking to > > > > GMP) would solve the LGPL issue. People who distribute that > > > > build of Python would still be held by the LGPL -- such as > > > > shipping any sources that they embed that Python into. > > > > > > No, that's exactly what the "L" in "LGPL" means. You only have to > > > ship the sources to any object module (library or program) actually > > > containing GMP code, including any modifications you make to them. So > > > if GMP is kept entirely in a separate library and you don't borrow any > > > code from it, the rest of your code is not affected by the LGPL. > > > > Sure. However, when you ship pythonxy.dll to your users, and it links > > with gmp.dll, you *will* have to ship gmp.dll to your users [and] > > then you have to include the source (of GMP, not Python). > > Right. There is definitely an administrative cost here, one that > Python itself doesn't impose. This may not be a big deal for some > downstream projects, but that is for each project to decide. I agree, > it is quite reasonable for Python to prefer to avoid the issue by > allowing downstream to add GMP or not as it likes, and simplify its > own maintenance burden by only providing the simple portable good- > enough-for-government-work implementation. > > Thomas's wording strongly suggested to me that he thought that the > presence of gmp.dll would force a project to include not only GMP > source, but Python source *and* even my Killer-App's source. That's > not true, and that's the only issue I was trying to address. > Ah, but not true according to who? If you read the LGPL, you'll note that it specifically says that 'linking in' makes the LGPL library part of the work (or rather, makes the whole a derived work.) Considering that, on Windows, the linking is done at compiletime and only the loading is done at runtime, it's unclear to me that this is not the case here. I don't know if RMS and the FSF think of it this way, but I stopped trying to guess their thoughts and actions a long time ago. It is certainly the case that such a combination is enough to scare off corporate lawyers who aren't sure either (most of them, I bet) and would advise against using that build of Python because of the LGPL components. It's not without cost to Python, even discounting the administrativia. -- Thomas Wouters <[EMAIL PROTECTED]> Hi! I'm a .signature virus! copy me into your .signature file to help me spread! ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] file open in python interpreter
adrian golding wrote: > hi all, > > (benjamin and nick thank you!) i have another question to ask something > about permissions for the python interpreter. in my earlier post, i was > saying i want to measure the python script before it is parsed. what > happens is that i will write the measurement of that script file to > another file (call is 'measurereq') i have, it gives a Permission denied > error. > > i have modified other programs to do the same measurement (eg, modified > Bash to measure .sh files) and it works fine. So i am suspecting it has > something to do with the python files. ie, i'm thinking the Permission > denied error is not because of i'm unable to write on my 'measurereq' > file, but its because i'm unable to measure (read) the python file. > > any clues? how can i go around this? You're getting out of python-dev territory now - I'm happy to help with some tips on interpreter internals, but I'm not going to attempt to debug-from-afar a modified version of the interpreter (particularly when the problem relates to file permission issues which are highly OS and file system dependent). That said, the usual debugging tactic of divide and conquer (e.g. adding trace calls to figure out exactly which step is failing, or just calculating the hash without attempting to write it to disk) will probably be your friend. Failing that, you might want to try asking for further assistance on comp.lang.python (aka [EMAIL PROTECTED]). Cheers, Nick. -- Nick Coghlan | [EMAIL PROTECTED] | Brisbane, Australia --- ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Optionally using GMP to implement long if available
2008/11/4 Greg Ewing <[EMAIL PROTECTED]>: > Martin v. Löwis wrote: >> >> you *will* have to ship gmp.dll to your users, as well ... So then > >> you have to include the source (of GMP > > Are you sure? I thought the source-provision requirements > of the *GPL licences only apply when you distribute a > *modified* version of something. Here you're just shipping > an unmodified GMP library. In fact you only have to provide a way to get the sources of the library. You don't have to ship the library with the source. As for the LGPL on Windows, wxWidgets does this for a long time now and a program does not have to be open source or LGPL or whatever. Matthieu -- Information System Engineer, Ph.D. Website: http://matthieu-brucher.developpez.com/ Blogs: http://matt.eifelle.com and http://blog.developpez.com/?blog=92 LinkedIn: http://www.linkedin.com/in/matthieubrucher ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Looking for VCS usage scenarios
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Nov 4, 2008, at 12:21 AM, Stephen J. Turnbull wrote: (2) New repo formats are added frequently, and taking advantage of new features often requires upgrading your repo format. So-called lightweight checkouts can be especially annoying as they involve leaving the history on the server, making distributed work problematic. It should be noted that this is a client-side choice. I can choose to have a heavyweight (i.e. full or partial) local revision history and work in a distributed way, while you choose to do a lightweight checkout and use the repository much as you currently use Subversion. The server side or master repo doesn't care. I like that Bazaar has this option, though I personally don't use it, nor would I teach/introduce it early on, unless I was specifically trying to guide existing Subversion users into the world of dvcs. OTOH, I know people who like this style of working very much. (3) Bazaar is very good at supporting the kind of refactoring that involves lots of file/directory renames and/or splitting/combining. My understanding is that Bazaar tracks changes to directories whereas Mercurial (and git?) do not. According to the Mercurial FAQs I've read, they claim this only affects empty directories. (4) Bazaar is claimed to have especially good merging support. And multiple ones at that. The default almost always DTRT, but what's nice is that if you do see an occasional text conflict, you can usually re-merge with a different algorithm to clear things up. - -Barry -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (Darwin) iQCVAwUBSRBXVnEjvBPtnXfVAQJh3AP+IMnvHQSvuTwQwYCx1zyposZcbF7JBIFc fkbnLBZ1Yn4JDiwPCDthzP2C0h8TAnshjeEPmr+On3sy+DizSxAT73jjhoCY3KoN lqZEaJQ4pGpUg6PNeRjHYxmYQ8vjNhM3gVBXRYjMlEuFG+S5vxhxqcVKvNEjKdl3 vVErzP69D1o= =zcux -END PGP SIGNATURE- ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Looking for VCS usage scenarios
> Trying to expand our buildbot infrastructure to accept patches to test > out or some patch queue manager might be nice, but I want to be > realistic with what we have now. That's why I am not worrying about > this email feature; until I know that we will actually use it and have > the manpower to implement it I consider it a perk that might never be > used. It's all great to dream about the future and what could be, but > that is far beyond just trying to get us to switch to a DVCS and is > just icing on the cake. It's not an email feature. It's about bundling revisions in a way that may be cheaply sent or attached in any way wanted. It is useful in practice. Attachments to bugs, mails, and so on are not novelty nor optimization. Check out how many patches you have attached to the bug tracker. Now consider how better would it be if these patches carried history information which allowed them to be merged considering context, and carried logs, authorship information, and incremental history. -- Gustavo Niemeyer http://niemeyer.net ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] hg branch gone?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Nov 4, 2008, at 9:01 AM, Thomas Wouters wrote: Nope, I have no idea how to edit those pages properly, sorry. I'm sure somebarryone does. I do. I've been meaning to update those pages to mention loggerhead too. I'll try to do that today. - -Barry -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (Darwin) iQCVAwUBSRBZ6XEjvBPtnXfVAQKMHQQApaUnvSUFeYu0hI9wFbLm8peQ1miwlrkI KFcZq26nFtQb/rSualxYR1ln82LIlGSLpR9PkYHzpbN+/FYGGXk+IrZ2OsUHalvd 1gsklmeHBc/CdJijFsfUw9XEQUZWgzobeGJz6RH5m4+JY8bmi1AdaNZ2nUnPd56j 50Gq6R0yqUs= =NwuV -END PGP SIGNATURE- ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] hg branch gone?
On Tue, Nov 4, 2008 at 14:30, Dennis Benzinger <[EMAIL PROTECTED]>wrote: > Hi Thomas! > > Am 03.11.2008 17:54, Thomas Wouters schrieb: > > [...] > > FWIW, I put one up this weekend, and it seems to be intact and OK. > > (bzr+ssh://[EMAIL PROTECTED]/python/2.6/ or > > http://code.python.org/python/2.6/ ) > > [...] > > Can you update http://www.python.org/dev/bazaar/ ? > For example "Branch details" needs to be updated because the trunk > branch is no longer Python 2.6. > Nope, I have no idea how to edit those pages properly, sorry. I'm sure somebarryone does. -- Thomas Wouters <[EMAIL PROTECTED]> Hi! I'm a .signature virus! copy me into your .signature file to help me spread! ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Looking for VCS usage scenarios
2008/11/4 Gustavo Niemeyer <[EMAIL PROTECTED]>: > How large? Which repositories? Which operations? Which version of Bazaar? As large as the Python repository. The Python repository (:-)). Local clone of the repo, when not using a shared repository (I know, "don't do that" - but it is nevertheless a specific performance complaint which I have, relating to the way I want to work with a DVCS). About 1.6. I can't recall precisely which version I tried, but it was fairly recent. > I'm sorry for jumping up on this, but you've mentioned one specific > issue, which is being looked at, and then have generalized here. > Without naming the issues, it's really hard to discuss anything. It > is true that Bazaar performance was much worse in the past, and it is > true that it's being improved, and will continue to improve. In case > Bazaar is a good deal for Python nowadays, it should only get better > over time. There is a serious danger that this discussion will degenerate into DVCS advocacy flamewars. Can I suggest that for now, we concentrate on describing what operations are needed, in abstract terms. I applaud Brett's initial stance that we need to look at what we need to do based on workflow right now - it avoids focussing too much on "how my DVCS has neat feature X which is self-evidently wonderful" arguments (note - I am not saying that Gustavo is doing that, just that things could go that way). Once we have the core workflow covered, *then* would be a sensible time to look very cautiously at features of a generic DVCS which *might* be useful. But let's be conservative here - after all, there is effort in simply replicating the python-checkins email hooks, and expecting people to have the time to add new customisations as well, could be a significant issue. Hmm, having said all this, one huge benefit of any DVCS is the ability for individuals to work on changes offline with full version control (the well-known downside being that people don't publish changes as soon as they should). Two thoughts come from that: 1. Availability of an easy-to-use location for random users to publish branches would be an important factor. 2. The option to easily set up a bidirectional gateway to other DVCSs might be beneficial to some users. The nice thing about both of these is that they *don't* have to be provided by python.org, as long as they are available. Paul. PS I still haven't found time to read Brett's PEP. Sorry. I will before giving into the temptation to add further to this thread. I promise! ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Optionally using GMP to implement long if available
Le Monday 03 November 2008 18:56:37 Paul Miller, vous avez écrit : > Rather than that, what about patching Python's long implementation > to use GMP if it's available, and the default implementation if not? Yes, I like this suggestion of two flavors. Python with GMP and Python without GMP (builtin integer library). It's "easy" to change the long type implementation since only few modules use PyLong internals: - mashal: import/export (.pyc) => core function might be moved to longobject.c - math: use some special features of the PyLong type => macro can be written for that or simplify a #ifdef. I first wrote a patch in January 2008 to test Python3 + GMP. After your email I updated my patch and fix it to pass most regression tests. Depending of the patch version, the overhead (of using GMP instead a builtin library) is between 14% and 3% (Python is slower with GMP). I used pystone which is not a math application. More details at: http://bugs.python.org/issue1814 -- Victor Stinner aka haypo http://www.haypocalc.com/blog/ ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Looking for VCS usage scenarios
Ralf Schmitt schrieb: > On Mon, Nov 3, 2008 at 1:05 AM, Brett Cannon <[EMAIL PROTECTED]> wrote: >> I have started the DVCS PEP which can be seen at >> http://docs.google.com/Doc?id=dg7fctr4_40dvjkdg64 . Not much is there >> beyond the rationale, usage scenarios I plan to use, and what other >> sections I plan to write. >> > > I think you really should not exclude any dvcs based on it's > implementation language. > I.e. requiring it being written in python for the sake of "eating your > own dogfood" is just a very weak argument. git is certainly missing > from your list. And by then, why not include darcs, GNU arch and monotone? Seriously, it's going to be hard enough to choose between hg and bzr (at least I hope so, because it would show that both are excellent tools). And since actually the implementation language makes sense, see the other replies, we don't need to make it even harder. cheers, Georg ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Looking for VCS usage scenarios
On Tue, Nov 4, 2008 at 6:45 PM, Georg Brandl <[EMAIL PROTECTED]> wrote: > Ralf Schmitt schrieb: >> I think you really should not exclude any dvcs based on it's >> implementation language. >> I.e. requiring it being written in python for the sake of "eating your >> own dogfood" is just a very weak argument. git is certainly missing >> from your list. > > And by then, why not include darcs, GNU arch and monotone? > please include all of them and choose the best one. that was my point. Regards, - Ralf ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Looking for VCS usage scenarios
Gustavo Niemeyer writes: > Both arguments strike me as odd. I'm an odd fellow, what can I say? > Having the *option* to leave your history on the server shouldn't > be a problem, right? Only if you later try to use it. > The same goes for (1): having more ways to use the tool isn't > exactly a problem. It isn't exactly a good thing, either: TOOWDTI. However, my intent was simply to respond to Brett, who mentioned that one consideration for a VCS for Python is simplicity/ease of use for occasional contributors, and proliferation of commands and styles is not conducive to that. Bazaar suffers from this problem although in a different way and lesser degree compared to git. > How large? Which repositories? Which operations? Which version > of Bazaar? I'm not going to go there; this has been discussed exhaustively on the [EMAIL PROTECTED] list, as well as on lists of projects considering adoption of bazaar. You should look in their archives for the results, where you will also find the discussion of the importance of the results and plans to address problems by the developers. Here again I'm simply responding to Brett who is under the impression that all dVCSes can scale. At present, bzr is one that cannot; the user must accept far worse performance, across all operations, than is available with the competing dVCSes, and the perception gets worse with scale. > In case Bazaar is a good deal for Python nowadays, it should only > get better over time. This is true. Performance is not everything to everyone. Most Bazaar users don't care at all; they say things like "who cares about a few seconds in bzr log when it gets the merge right almost every time". But we don't yet know that most Python core developers will feel the same way, and it simply is not appropriate at this point in time to dismiss performance concerns with "oh, I'm sure they all can scale." ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Looking for VCS usage scenarios
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Nov 4, 2008, at 1:00 PM, Stephen J. Turnbull wrote: This is true. Performance is not everything to everyone. Most Bazaar users don't care at all; they say things like "who cares about a few seconds in bzr log when it gets the merge right almost every time". But we don't yet know that most Python core developers will feel the same way, and it simply is not appropriate at this point in time to dismiss performance concerns with "oh, I'm sure they all can scale." We're Python programmers. We're used to people telling us our tool is too slow. We just say it does the job superbly and it's usually fast enough. :) - -Barry -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (Darwin) iQCVAwUBSRCQv3EjvBPtnXfVAQK43AQAhexjo3SVeWV9/3PYlR+PQPvZH72XY6XG FuuWN6cfm0XGCAoyRTOv2DfnwYhHw1btYqCeP5yjxOuU1WogOfY1AvpSzWPYDi9R Y1HQArXl9detWSAi5nzL2R8Z57jDnVpLxHscBMGhNxkzyqjQV+Dx2x2Zj3fqw8ac NbOanHZewCk= =4ES6 -END PGP SIGNATURE- ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Looking for VCS usage scenarios
On Tue, Nov 4, 2008 at 12:13 PM, Barry Warsaw <[EMAIL PROTECTED]> wrote: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA1 > > On Nov 4, 2008, at 1:00 PM, Stephen J. Turnbull wrote: > > This is true. Performance is not everything to everyone. Most Bazaar >> users don't care at all; they say things like "who cares about a few >> seconds in bzr log when it gets the merge right almost every time". >> But we don't yet know that most Python core developers will feel the >> same way, and it simply is not appropriate at this point in time to >> dismiss performance concerns with "oh, I'm sure they all can scale." >> > > We're Python programmers. We're used to people telling us our tool is too > slow. We just say it does the job superbly and it's usually fast enough. :) > > - -Barry > I don't agree with completely dismissing performance just because it's Python. Yes, Python is fast enough most of the time, but when it's not we put a lot of effort into making it faster. That's why we have a good collection of modules with C extensions to speed up computationally intensive applications and half a dozen ways of writing said C extensions. I see no excuse to let the fact that it's Python make it acceptable to have an application with otherwise unacceptable performance. -- Cosmin Stejerean http://www.offbytwo.com ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Looking for VCS usage scenarios
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Nov 4, 2008, at 1:21 PM, Cosmin Stejerean wrote: I don't agree with completely dismissing performance just because it's Python. Yes, Python is fast enough most of the time, but when it's not we put a lot of effort into making it faster. That's why we have a good collection of modules with C extensions to speed up computationally intensive applications and half a dozen ways of writing said C extensions. I see no excuse to let the fact that it's Python make it acceptable to have an application with otherwise unacceptable performance. I guess the humor of that got lost in translation. You had to be there. - -Barry -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (Darwin) iQCVAwUBSRCVN3EjvBPtnXfVAQIdOAP/VxEc9ptQdrrJuthZM+wEeYy2eHMTno59 eFGurHysKDl5AUcYPt0dCeP8GAiuvZmU2Vp8/07HmMkgJPPtWbmxHZYTLZ2tEm/A HmXBcsz1icc0dteu0SyeBUwDOz4puv9I5oF6NM879QEt4ykQz0hzM3dVyW6JsfN7 qIlGI8wGEgI= =yUNI -END PGP SIGNATURE- ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Optionally using GMP to implement long if available
Hey, Mark -- let's establish some background here first. It's a fact that the product of two N-bit integers can require 2N bits, and also a fact that lots of HW supports NxN->2N bit integer multiplication directly. However, it's unfortunately also a fact that standard C has no corresponding concept: "*" in C always means NxN->N multiplication (single-width product, with possibility of overflow). I don't know whether C99 improved this situation -- I know it was proposed to add some "double-width integer product" /functions/, but I don't know whether that was adopted. I do know that "*" remained "single-width". [Tim Peters] >> Note that while 32x32->64 multiply is supported by x86 HW, that >> doesn't mean there's a uniform way to get at this HW capability across >> C compilers. In contrast, (at least) efficient HW 15x15->30 multiply >> is universally spelled in C via "i*j" :-) ['Mark Dickinson] > If we're talking about standards and portability, doesn't "i*j" fail > on those (nonexistent?) platforms where the 'int' type is only 16-bits? > Shouldn't this be something like "(long)i * j" instead? Sorry, I should have made type declarations explicit. In Python, the basic long building block is "digit", which is typedef'ed to C unsigned short. C89 guarantees this holds at least 16 bits. Whenever two digits are multiplied, the code intends to cast (at least) one of them to "twodigits" first (if you ever see a spot where this doesn't happen, it's a bug). "twodigits" is typedef'ed to C long. C89 guarantees that a long holds at least 32 bits. So C guarantees that we're doing (at least) 32x32->32 multiplication whenever you see code like digit i, j; twodigits k; k = (twodigits)i * j; In particular, the (at least) 32x32->32 C89 guarantees for that is at least 15x15->30, and the latter is all that longobject.c intends to rely on. Along with the cast to twodigits, this is achieved across all conforming C compilers simply by using infix "*". The code from 1990 still works fine, on everything from cell phones to archaic Cray boxes. > And for 32x32 -> 64, can't this simply be replaced by "(uint64_t)i * j", > where uint64_t is as in C99? I'd hope that most compilers would > manage to turn this into the appropriate 32x32-bit hardware multiply. 1. That's C99, not C89, and therefore less portable. 2. On platforms that support it, this is at least 64x64->64 multiplication, potentially much more expensive than the 32x32->64 (or 31x31->62?) flavor you /intend/ to move to. 3. There's no way to know exactly what compilers will do with this short of staring at generated code. FYI, in a previous life working in speech recognition, under Microsoft's Visual Studio 6 the only way we found to get at the Pentium's 32x32->64 HW ability efficiently was via using inline assembler. For example, using various MSVC spellings of "64-bit int" instead for the inputs usually generated external calls to a long-winded C library "piece by piece" multiplication routine (which, at the time, emulated 64x64->128 multiplication, then threw away the high 64 bits). Again, the fundamental problem here is the disconnect between what some HW is capable of and what C allows to express (at least through C89). That's why it's impossible here to write portable code in C that's also efficient. Even what Python does now is vulnerable on the efficiency count: on some weird platforms, "long" is 64 bits, and so multiplying a pair of twodigits incurs the expense of (usually non-native) 64x64->64 multiplication. > I agree that very-long-integer optimizations probably don't really belong in > Python, Depends in part on whether Python can attract as many obsessed maintainers and porters for such gonzo algorithms as GMP attracts ;-) > but this patch should also provide significant benefits for short > and medium-sized integers. I guess I need to go away and do some > benchmarking... If you can /get at/ HW 32x32->64 int multiply, of course that would be faster. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] hg branch gone?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Nov 4, 2008, at 8:30 AM, Dennis Benzinger wrote: Am 03.11.2008 17:54, Thomas Wouters schrieb: [...] FWIW, I put one up this weekend, and it seems to be intact and OK. (bzr+ssh://[EMAIL PROTECTED]/python/2.6/ or http://code.python.org/python/2.6/ ) [...] Can you update http://www.python.org/dev/bazaar/ ? For example "Branch details" needs to be updated because the trunk branch is no longer Python 2.6. Done. - -Barry -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (Darwin) iQCVAwUBSRCZkXEjvBPtnXfVAQLyswQAnGDLokaK9sHtNGcL92TJfTokGecUKPiH ozgvS86T2Ydo6R9Qb71uJf+q/0iTnSyZ6LsGemKe93KrEwjmsksz9nSFrMkSwEH9 5m69nN2gaOLx7yJq3SHMBup2iJepFa++sK9IBph5eivrRsA3We+olKuOxNkDokKe O1PDssFbmlo= =hWPQ -END PGP SIGNATURE- ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Optionally using GMP to implement long if available
On 2008-11-04 18:38, Victor Stinner wrote: > Le Monday 03 November 2008 18:56:37 Paul Miller, vous avez écrit : >> Rather than that, what about patching Python's long implementation >> to use GMP if it's available, and the default implementation if not? > > Yes, I like this suggestion of two flavors. Python with GMP and Python > without > GMP (builtin integer library). It's "easy" to change the long type > implementation since only few modules use PyLong internals: > - mashal: import/export (.pyc) >=> core function might be moved to longobject.c > - math: use some special features of the PyLong type >=> macro can be written for that > or simplify a #ifdef. Ahem, what about the gazillion extension modules out there ? You cannot simply change the long implementation and expect everything to just work. Besides, like Tim already said: adding lots and lots of #ifdefs is just going to make it harder to maintain the implementation without gaining anything much. I don't really see much of a need for this anyway. If people want to use GMP they can download gmpy or PyGMP or our experimental mxNumber extension: http://www.egenix.com/products/python/mxExperimental/mxNumber/ -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Nov 04 2008) >>> Python/Zope Consulting and Support ...http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ Try mxODBC.Zope.DA for Windows,Linux,Solaris,MacOSX for free ! eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Optionally using GMP to implement long if available
About 31, 32, 63 or 64 bits: I guess that you want to avoid integer overflow.
Intel has an "overflow" flag, changed for all instructions. For other CPUs,
you can use emulate this flag. Example with the type int that I used in my
GMP patch:
Add:
int a, b, sum;
sum = a + b;
exact = ((a < 0) ^ (b < 0)) || ((a < 0) == (sum < 0));
Substract:
int a, b, diff;
diff = a + b;
exact = ((a < 0) == (b < 0)) || ((a < 0) == (diff < 0));
Multiply:
int a, b, product;
if (a == 0 || b == 0) {
product = 0; /* exact */
} else if (a != INT_MIN || (b == 1)) {
product = a * b;
exact = (product / b) == a);
} else {
/* INT_MIN * -1 = -INT_MIN: overflow */
}
Divide:
int a, b, q;
if (a != INT_MIN || b != -1) {
q = a / b; /* exact */
} else {
/* INT_MIN / -1 = -INT_MIN: overflow */
}
Checking overflow may costs more than using a smaller base. Only a benchmark
can answer ;-)
--
Victor Stinner aka haypo
http://www.haypocalc.com/blog/
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Optionally using GMP to implement long if available
[Martin v. Löwis] > Perhaps Tim Peters should also comment here - but if you can come up > with a patch that does that in a portable way, I would be in favor. > The challenge, ISTM, is to make it still compile on all systems > (regardless of how inefficient it might be on minor platforms). I've just uploaded a patch to the issue tracker that allows Python to use 30-bit digits on platforms having a 64-bit integer type: http://bugs.python.org/issue4258 [Tim Peters] > Note that while 32x32->64 multiply is supported by x86 HW, that > doesn't mean there's a uniform way to get at this HW capability across > C compilers. In contrast, (at least) efficient HW 15x15->30 multiply > is universally spelled in C via "i*j" :-) If we're talking about standards and portability, doesn't "i*j" fail on those (nonexistent?) platforms where the 'int' type is only 16-bits? Shouldn't this be something like "(long)i * j" instead? And for 32x32 -> 64, can't this simply be replaced by "(uint64_t)i * j", where uint64_t is as in C99? I'd hope that most compilers would manage to turn this into the appropriate 32x32-bit hardware multiply. I agree that very-long-integer optimizations probably don't really belong in Python, but this patch should also provide significant benefits for short and medium-sized integers. I guess I need to go away and do some benchmarking... Mark ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Using Cython for standard library?
Gerhard Häring wrote:
> Cython is still being worked on (intensively, it seems). Bundling it
> with Python means deciding on a particular version probably for an
> entire major release lifecycle (use Cython x.y.{newest} for Python 2.7,
> for example).
Yes, that is definitely an issue. Cython is heavily being worked on, a lot of
features were implemented this summer. This means that even slightly different
Cython versions can generate heavily different code.
The project has made inclusion into Python's stdlib a goal right from the
beginning. However, it is not currently clear when we will reach a suitable
level of Python language compliance to make a language/feature freeze. Note
that this only means a state where we decide to keep new features out. What's
in there now will not go away, so existing Cython code is pretty much future
proof (it even runs on Py3 already :).
>> I would love to see the option to write the lower levels in something
>> other than C,
Well, Cython is definitely an option for that.
> Absolutely. That's why I tried to reimplement pysqlite in something
> easier to maintain than handwritten Python C API. There's a ctypes-based
> version in its Mercurial repository that's good enough to be used from
> PyPy now. And a started Cython-based one.
ctypes makes sense for projects that do not require a high-speed interface,
i.e. if you do major things behind the interface and only call into it from
time to time, choosing ctypes will keep your code more portable without
requiring a C compiler. However, if speed matters then it's hard to beat
Cython even with hand-written C code.
>> but obviously any choice would have to be a good one.
>> [...] So, the question I see isn't if Cython should be
>> allowed for standard library modules, but if the landscape of such
>> solutions is at a point that any of them is ready to be committed to.
Well, we had reports that existing, non-trivial C modules could be rewritten
in Cython within a few days. But even if you are only talking about new
modules that get added to the stdlib, all you really need to become
independent of a specific Cython version is a good test suite for your code to
make sure it keeps working as expected when you update your compiler.
Stefan
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Using Cython for standard library?
-> >> but obviously any choice would have to be a good one. -> >> [...] So, the question I see isn't if Cython should be -> >> allowed for standard library modules, but if the landscape of such -> >> solutions is at a point that any of them is ready to be committed to. -> -> Well, we had reports that existing, non-trivial C modules could be rewritten -> in Cython within a few days. But even if you are only talking about new -> modules that get added to the stdlib, all you really need to become -> independent of a specific Cython version is a good test suite for your code to -> make sure it keeps working as expected when you update your compiler. Hey, wait, isn't that also a requirement for Py3k? :) --titus -- C. Titus Brown, [EMAIL PROTECTED] ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] XXX do we need a new policy?
On Tue, Nov 4, 2008 at 07:04, "Martin v. Löwis" <[EMAIL PROTECTED]> wrote: > > I see that Guido is not keen on the idea, and I'm not sure my > > observations help sway things one way or the other. OTOH, it would be > > nice if at least we always add our own identifier (initials, nick, email > > address) and a date to the XXX so we at least know who was talking about > > what. > > I find it fairly easy to use "svn annotate" to learn about the source > of an XXX comment. In many cases, the XXX comment is fairly obvious, > anyway, so knowing who added it, and when, doesn't provide much useful > information. > Ah, but that's not true in Subversion: you just see who committed the XXX comment, not who wrote it :-) I've been using the XXX(twouters) approach for a while and it's pretty convenient because who wrote the comment changes how you interpret the comment. See for instance r42313 and r42717 of Python/ceval.c: if I'd known the original comment was added by Jeremy, I would have interpreted the question correctly the first time around. (As it was, Jeremy came up to me at PyCon, I think :) -- Thomas Wouters <[EMAIL PROTECTED]> Hi! I'm a .signature virus! copy me into your .signature file to help me spread! ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Using Cython for standard library?
[EMAIL PROTECTED] wrote: > >>> - Option 2: only distribute generated source files > >>> -- developers still need to have Cython installed > >>> -- you have to trust Cython; who will really review the generated > code? > >> > >> Who reviews the machine code from gcc? > > Gerhard> That's comparing apples and eggs :-P But it may be that I'm a > Gerhard> little paranoid here. > > Agreed. When Cython is as widely used as gcc and has as comprehensive a set > of test cases (who knows? it may well already be comprehensive enough) The test suite is pretty huge already, works on Py2.3 through Py3.0 (without regenerating the C source code, BTW), and it tests C compilation as well as C++ compilation of the generated source code. > and supports all the compilers which Python supports I didn't hear any complains so far. Portable C code is definitely a project goal. > then we can probably skip the output code review step. I think the main advantage for stdlib modules is actually the maintenance cost. Having a single, easy-to-read code base for extension modules that compiles without modification in Py2.6/7 and Py3.0/3.1 (and 2.3, 2.4 and 2.5), makes life a lot easier for both maintainers and backporters. Stefan ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] file open in python interpreter
hi all, (benjamin and nick thank you!) i have another question to ask something about permissions for the python interpreter. in my earlier post, i was saying i want to measure the python script before it is parsed. what happens is that i will write the measurement of that script file to another file (call is 'measurereq') i have, it gives a Permission denied error. i have modified other programs to do the same measurement (eg, modified Bash to measure .sh files) and it works fine. So i am suspecting it has something to do with the python files. ie, i'm thinking the Permission denied error is not because of i'm unable to write on my 'measurereq' file, but its because i'm unable to measure (read) the python file. any clues? how can i go around this? thanks - adrian On Mon, Nov 3, 2008 at 9:57 PM, Nick Coghlan <[EMAIL PROTECTED]> wrote: > Benjamin Peterson wrote: > > On Mon, Nov 3, 2008 at 7:25 AM, Benjamin Peterson > > <[EMAIL PROTECTED]> wrote: > >> On Mon, Nov 3, 2008 at 1:04 AM, adrian golding <[EMAIL PROTECTED]> > wrote: > >>> hi all, i am trying to find out where is the part of the code in the > python > >>> interpreter that opens up the .py file and parses it. in particular, i > am > >>> trying to find the file open command in that file. I greped and i just > want > >>> to make sure this is it: /Python-2.6/Parser/pgenmain.c > >>> i am intending to take a hash measurement of the .py file just before i > open > >>> it to run the script. is the above file the right place to call for > the > >>> measurement before the file open function? > >> You want Parser/tokenizer.c. > > > > Sorry, that's not correct. opening of modules happens in > > Python/import.c. There's also a case in Modules/main.c. > > And some indirect ones from runpy.py (via pkgutils) if you use the -m > switch, or are executing a zipfile or directory. > > But for the specific case of an exact filename being provided on the > command line, then main.c is the one the original poster will want to > look at (line 567 to be exact). > > Cheers, > Nick. > > -- > Nick Coghlan | [EMAIL PROTECTED] | Brisbane, Australia > --- > ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Optionally using GMP to implement long if available
>> OTOH, it should be no big deal to drop a zip archive of the GMP >> sources which correspond to the code bound into the DLL. Martin> How would end users then extract the sources from the DLL? How Martin> would they learn that they even have them in the first place? I think you missed an implied comma in my statement. I meant: 1. distribute pythonxy.dll which binds to GMP. 2. distribute a gmpmn.zip which contains the source for the version used in #1. Skip ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Optionally using GMP to implement long if available
Martin v. Löwis wrote: you *will* have to ship gmp.dll to your users, as well ... So then > you have to include the source (of GMP Are you sure? I thought the source-provision requirements of the *GPL licences only apply when you distribute a *modified* version of something. Here you're just shipping an unmodified GMP library. -- Greg ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Optionally using GMP to implement long if available
Thomas Wouters writes: > Ah, but not true according to who? [...] > It is certainly the case that such a combination is enough to scare > off corporate lawyers who aren't sure either (most of them, I bet) > and would advise against using that build of Python because of the > LGPL components. I don't think this is a matter of "according to whom", except where you have a timid client and a FUD-ful lawyer, which is a plausible case, I admit. Copyright law is very literal about this kind of thing, and does not distinguish between bytes that represent a symbolic name for a function, and bytes that represent the address of the function. If the LGPL licenses symbolic references as long as the function object code is not copied into your executable, then numerical addresses are also OK as long as the function object code is not copied into the executable you distribute. I agree the language used in the LGPL is unfortunately imprecise. But the FSF's intent and understanding is quite clear about the issues you raise. See their License FAQ and David Turner's article about Java vs. the LGPL: http://www.fsf.org/licensing/licenses/gpl-faq.html#Prelinking http://www.fsf.org/licensing/licenses/gpl-faq.html#LGPLv3ContributorVersion http://www.fsf.org/licensing/licenses/lgpl-java.html I see nothing anywhere to indicate that the FSF thinks that the LGPL should work differently in a practical sense on Windows than it does on GNU/Linux, and while Turner's article refers to language environments rather than OSes, it's pretty clear that the assertion that "the LGPL just works as intended and expected" applies to OS environments too. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Optionally using GMP to implement long if available
[EMAIL PROTECTED] wrote: > >> OTOH, it should be no big deal to drop a zip archive of the GMP > >> sources which correspond to the code bound into the DLL. > > Martin> How would end users then extract the sources from the DLL? How > Martin> would they learn that they even have them in the first place? > > I think you missed an implied comma in my statement. I meant: > > 1. distribute pythonxy.dll which binds to GMP. > 2. distribute a gmpmn.zip which contains the source for the version used >in #1. Ah, I see. That *is* a big deal to many Python users, which want to distribute pythonxy.dll as part of their application, in a single file (e.g. py2exe), and then don't want to worry about shipping another zip file along with their single-file executable (plus including a third file explaining why you need to do this). Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Fwd: Removal of GIL through refcounting removal.
On Nov 4, 2008, at 12:11 AM, Josiah Carlson wrote: There is no shortage of algorithms (such as matrix multiplication) that are parallelizable but not particularly good candidates for an IPC-based multiprocessing paradigm. Ahh, but those algorithms aren't going to be written in Python; they are going to be written in C, and are going to manipulate memory directly. With such things, you can use standard Python threads, call into your C runtime, and release the GIL. Alternatively, you can use the mmap module to store your data, shared across multiple processes, using the same direct-memory access as you would with multiple threads and GIL release. That's the thing, right now you can't even use python to sketch one of those algorithms and for example compare performance with a non parallel version. Also with four cores on common intel processor you might be happy with your algo being 4x faster while being quite easy to develop (not needing to use c). I personally think this will be too much work on -- Leonardo Santagada santagada at gmail.com ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Using Cython for standard library?
> The project has made inclusion into Python's stdlib a goal right from the > beginning. Ah, that changes my view of it significantly. If the authors want to contribute it to Python some day, I'm looking forward to that (assuming that they then close their official branch, and make the version inside Python the maintained one). That is also independent of whether standard library modules get written in Cython. I would expect that some may (in particular, if they focus on wrapping an external library), whereas others might stay what they are (in particular, when they are in the real core of the interpreter). > ctypes makes sense for projects that do not require a high-speed interface, > i.e. if you do major things behind the interface and only call into it from > time to time, choosing ctypes will keep your code more portable without > requiring a C compiler. However, if speed matters then it's hard to beat > Cython even with hand-written C code. I would personally prefer a Cython integration over a ctypes one, for the standard library (and supported inclusion of ctypes into Python regardless). Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Optionally using GMP to implement long if available
On Tue, Nov 4, 2008 at 6:33 PM, Tim Peters <[EMAIL PROTECTED]> wrote: > Whenever > two digits are multiplied, the code intends to cast (at least) one of > them to "twodigits" first (if you ever see a spot where this doesn't > happen, it's a bug). "twodigits" is typedef'ed to C long. C89 > guarantees that a long holds at least 32 bits. There are indeed one or two spots that seem to be missing a cast, for example the line "rem -= hi*n" in inplace_divrem1. I've fixed all those I found, in the issue 4258 patch. >> And for 32x32 -> 64, can't this simply be replaced by "(uint64_t)i * j", > 1. That's C99, not C89, and therefore less portable. Understood; my thought was to use uint32_t and uint64_t for digit and twodigits when available (with longs being stored in base 2**30), falling back to the current ushort/ulong/base 2**15 otherwise. On Unix, autoconf makes this easy by providing macros like 'AC_TYPE_INT32_T', which makes sure that int32_t is defined to be an integer type of exact width 32, when available. > 2. On platforms that support it, this is at least 64x64->64 multiplication, > potentially much more expensive than the 32x32->64 (or 31x31->62?) > flavor you /intend/ to move to. > > 3. There's no way to know exactly what compilers will do with this short of > staring at generated code. Yes---I've done the staring for gcc, so I now have precisely *one* data point, which is that various flavours of gcc on x86/x86_64 *are* clever enough to turn (uint64_t)my_uint32 * my_other_uint32 into the appropriate HW instruction. Unfortunately I don't have easy access to other compilers or platforms right now. :-( Am still working on the benchmarking, but I'm definitely seeing speedup on gcc/x86---between 10% and 100% depending on the operations. > FYI, in a previous life working in speech recognition, under > Microsoft's Visual Studio 6 the only way we found to get at the > Pentium's 32x32->64 HW ability efficiently was via using inline > assembler. Urk. We definitely don't want to go there. Though I guess this is how packages like gmp and GP/Pari manage. > C89). That's why it's impossible here to write portable code in C > that's also efficient. Even what Python does now is vulnerable on the But maybe it's possible to write portable code (by providing fallbacks) that turns out to be efficient on the majority of mainstream systems? The extent of the ifdef'ery in the patch is really rather small: one (compound) #ifdef in longintrepr.h for defining digit, twodigits, stwodigits etc, and a couple more for the places where digits are read and written in marshal.c. >> I agree that very-long-integer optimizations probably don't really belong in >> Python, > > Depends in part on whether Python can attract as many obsessed > maintainers and porters for such gonzo algorithms as GMP attracts ;-) Well, you can count me in. :) Mark ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] XXX do we need a new policy?
Guido van Rossum wrote: > On Mon, Nov 3, 2008 at 3:39 PM, Benjamin Peterson > <[EMAIL PROTECTED]> wrote: >> Grepping through Python's sources tells me that we have over 2,000 >> "XXX" comments. The thing that irks me about them is that the have a >> very slow rate of being resolved, since they usually act more as >> "notes to self" rather than easily attainable tasks. >> >> So, I propose that we adopt a policy similar to Twisted's: All "XXX" >> comments must have an issue in the tracker and an accompanying link >> with the source code. That way we'll have a forum for discussing the >> changes and deciding whether they are reasonable enough to >> implemented. > > That seems excessively draconian. Even at Google we don't have a rule > like that, and we're sure big on process and interlinked tools. If you > want to, you could add a different magic keyword, but personally, I > don't see XXX comments necessarily as "to be resolved" -- merely as > flags for someone perusing the code looking to change it or digging > for the cause of some problem to pay special attention. > That's the way I generally use them (e.g. there's one in runpy.py pointing out why _run_module_as_main isn't part of the module's public API). Ben's right that any XXX comments that actually turn out to be real problems should be placed in the issue tracker, but those that are just "here be dragons!" warnings or "could we do this better?" musings aren't really a problem. Even the "is this portable?" questions are OK until we find a platform where the relevant code doesn't work. Cheers, Nick. -- Nick Coghlan | [EMAIL PROTECTED] | Brisbane, Australia --- ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Using Cython for standard library?
>>> Is using Cython for anything in Modules/ really an option? In my limited
>>> experiments with it, I did like it.
>>>
>>> But using it for Python standard library stuff doesn't look quite right to
>>> me:
>>>
>>> - Option 1: distribute Cython with Python and integrate into build process
>>> -- Ouch!
>>
>> Can you be a bit more descriptive?
>
> Cython is still being worked on (intensively, it seems). Bundling it
> with Python means deciding on a particular version probably for
> an entire major release lifecycle (use Cython x.y.{newest} for
> Python 2.7, for example).
If we're talking about Cython for core libraries...
For the subset of python that also works in Cython, I've seen a 10-30%
performance speedups on (admittedly trivial) test code. Conceivably,
*everything* could be Cython-compiled if you cared about performance
more than transparency.
Brendan
anyall.org
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Looking for VCS usage scenarios
Ralf Schmitt wrote: > On Tue, Nov 4, 2008 at 6:45 PM, Georg Brandl <[EMAIL PROTECTED]> wrote: >> Ralf Schmitt schrieb: >>> I think you really should not exclude any dvcs based on it's >>> implementation language. >>> I.e. requiring it being written in python for the sake of "eating your >>> own dogfood" is just a very weak argument. git is certainly missing >>> from your list. >> And by then, why not include darcs, GNU arch and monotone? >> > > please include all of them and choose the best one. that was my point. The PEP can at least mention darcs, arch, monotone, and then drop them on the basis of not having a compelling reason to prefer them to the 3 more popular choices. I don't see any need to go into the "other choices" further than that - our needs aren't wildly divergent from that of other open source projects, and while popularity is a poor final criteria, it's a decent filter to narrow the scope of the initial investigation. For git, without a killer feature that is "oh my god, wow!" when compared to the features offered by Mercurial and Bazaar, the implementation language is enough to push it to the back of the list for me. Sure, that's a pretty lousy criteria in the absolute sense, but given that it is obvious that there are two perfectly acceptable options written in the language we ourselves are implementing, then it should take something pretty impressive to get us to look at a third choice. That would leave the PEP to focus on the Mercurial/Bazaar comparison (as Brett originally intended) and I'm sure even that limited comparison is going to give the infrastructure committee (and any other willing assistants from python-dev) plenty of work to do). The main thing I would suggest is that some kind of rationale like the one above be included into the PEP itself to try to forestall the folks going "but what about darcs/arch/monotone/git/whatever". Also, something that should be done for ANY candidate VCS: translate the current Python developer FAQ to give the appropriate answers for the candidate VCS. Cheers, Nick. -- Nick Coghlan | [EMAIL PROTECTED] | Brisbane, Australia --- ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Looking for VCS usage scenarios
Hello Stephen, > I haven't used Bazaar beyond "bzr pull" of Mailman once a week or so, > so I don't dislike it. Things I have observed or have seen discussed > on the bazaar mailing list that you might want to consider: > (1) The UI is as baroque as git's, once you consider all the plugins >and GUIs that are available. Lots of different workspace styles >(ordinary branching, stacked branching, looms -- similar to >quilts?, lightweight checkouts, ...) are supported with a >corresponding increase in subcommand count and/or options. > (2) New repo formats are added frequently, and taking advantage of new >features often requires upgrading your repo format. So-called >lightweight checkouts can be especially annoying as they involve >leaving the history on the server, making distributed work >problematic. Both arguments strike me as odd. Having the *option* to leave your history on the server shouldn't be a problem, right? The same goes for (1): having more ways to use the tool isn't exactly a problem. > My understanding is that to deal fully with these problems, the Bazaar > developers plan to change the repo file format. Some progress has > been made, (small) quantitative improvements have been made, but AFAIK > bzr still has bad worst-case performance for some common operations on > moderately large repos (way smaller than the Linux kernel). How large? Which repositories? Which operations? Which version of Bazaar? I'm sorry for jumping up on this, but you've mentioned one specific issue, which is being looked at, and then have generalized here. Without naming the issues, it's really hard to discuss anything. It is true that Bazaar performance was much worse in the past, and it is true that it's being improved, and will continue to improve. In case Bazaar is a good deal for Python nowadays, it should only get better over time. -- Gustavo Niemeyer http://niemeyer.net ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] XXX do we need a new policy?
On 02:47 am, [EMAIL PROTECTED] wrote: I disagree. They should be removed when the issue they refer to is removed. No sooner, no later. Simply removing every XXX comment older than a year would not be helpful. The code base is so large that over 2000 XXX doesn't faze me particular. There are over 4000 files in the Python 2.6 source code tarball! The right thing to do with XXX comments is to read them when you're in their vicinity, and to act when the urge becomes too strong to deal with any one in particular. Dealing with them en masse is just asking for a migraine. I _strongly_ agree with what Guido is saying here. I already mentioned this indirectly in my other post to this thread but I feel like it's worth emphasizing, especially given that I think that the Twisted core team (myself included) is widely perceived, for good or ill, as being process-crazed maniacs. We have a sort of meta-policy of "never try to boil the ocean". We started Twisted with a freewheeling, anything-goes commit policy, where if you had an account on the CVS server (my home desktop computer) you could do whatever you wanted: no code review, no compatibility, no testing. There's still a lot of code left over from those days, and a lot of it works fine. Every policy we have applies to _changes_ to the code; nobody has ever proposed a policy modification where we first fix all 1000 instances of X and then institute a policy on future X. Quite the opposite: the whole point of most of our development process is to avoid needing to do broad, sweeping changes. So, while I don't really care much one way or the other about Python's treatment of future XXX, I definitely think it would be a terrible idea to try to deal with all XXXes at once, or even start an ongoing process of gradually farming through the old XXXes; that would almost certainly be a waste of everyone's time. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Using Cython for standard library?
Stefan Behnel wrote: I think the main advantage for stdlib modules is actually the maintenance cost. Having a single, easy-to-read code base for extension modules that compiles without modification in Py2.6/7 and Py3.0/3.1 (and 2.3, 2.4 and 2.5), makes life a lot easier for both maintainers and backporters. Do you have any plans to support/use 3.0 type annotations so that one could develop function-oriented code in 3.0 and then compile efficient C (for whatever CPython version) without adding Python-incompatible cdefs? ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] hg branch gone?
Hi Thomas! Am 03.11.2008 17:54, Thomas Wouters schrieb: > [...] > FWIW, I put one up this weekend, and it seems to be intact and OK. > (bzr+ssh://[EMAIL PROTECTED]/python/2.6/ or > http://code.python.org/python/2.6/ ) > [...] Can you update http://www.python.org/dev/bazaar/ ? For example "Branch details" needs to be updated because the trunk branch is no longer Python 2.6. Thanks, Dennis Benzinger ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] XXX do we need a new policy?
On 3 Nov, 11:44 pm, [EMAIL PROTECTED] wrote: On Mon, Nov 3, 2008 at 3:39 PM, Benjamin Peterson <[EMAIL PROTECTED]> wrote: Grepping through Python's sources tells me that we have over 2,000 "XXX" comments. So, I propose that we adopt a policy similar to Twisted's: All "XXX" comments must have an issue in the tracker and an accompanying link with the source code. That seems excessively draconian. (...) personally, I don't see XXX comments necessarily as "to be resolved" -- merely as flags for someone perusing the code looking to change it or digging for the cause of some problem to pay special attention. For what it's worth, in my experience this is one of the least draconian areas of Twisted policy ;-). I'll describe our reasoning for the policy and the effect that it's had. If a developer working on a feature is inclined to type # somebody, do something to something else somebody.doSomething(somethingElse) then the intent is typically to say "this is a confusing snippet of code, it bears explanation beyond what is clearly readable in the code itself". Documenting the "why" and "what" happens in the docstring, so we only have comments in areas of code where the "how" or the "who" (i.e. "only JP and Glyph understand this") is difficult to understand. In Twisted, this is almost always some wicked busted operating system API, like how you get undocumented errnos out of send(), recv() or select() on some platform. However, if they're inclined to type # TODO: somebody should really do more stuff here somebody.doSomething(somethingElse) most commonly, what this means is "before I merge this feature, I really want to fix this potential issue so we don't need to deal with it in the future". The policy applies to "XXX" because in my personal experience it's always used as a synonym for "TODO". My subjective impression is that in 75% of the cases where a comment like this is caught in a code review, the original author of the branch had actually forgotten they had added a comment like this and will eagerly go and fix it, to avoid dealing with the issues that it might raise in a release in the future. I can't recall a case where an author (myself included) was not happy to be alerted to a comment like this. In a significant minority of cases, although the author has still usually forgotten (if they had remembered they would have filed a ticket preemptively, after all), the task in question is actually too big or too esoteric to bother with in the same branch, so it gets deferred. Filing a ticket is the natural extension of this, so that we don't lose track of the task in question. The consequence of this policy has been higher quality bugfixes (because contributors don't forget what they're doing), and increased visibility into problems in our codebase(s) (since we use this on Divmod projects too) which directly translates into less duplicate grumbling about the same issue over and over again in pointless comments. Also, I hear a lot of kvetching about Twisted policy, especially review latency (I hear you guys are familiar with that problem as well), but as I recall nobody has ever complained that this was too much work. When we instituted the policy, as with most of our policies, there is a grandfather clause: we didn't go back and trawl through all of the existing XXX and TODO comments, we just made it so you couldn't add _new_ XXX and TODO comments without filing and linking to a ticket. I haven't seen "XXX" used as a way to say "pay special attention" because our use of comments is extremely spare. If there's a comment at all that means the code is sufficiently wacky that it warrants special attention. Also, we don't write much code in C, so the preponderance of "sufficiently wacky" code is small ;). Looking at the Python codebase, though, (specifically the 2.6 release), there are definitely "XXX" comments which do not appear to be actionable tasks by their authors. Guido's certainly right about that: not all XXX comments in Python are "to be resolved". But, a subjective assessment of the first couple of pages of such comments indicates that a majority of them are. A lot of them are cryptic, too. One of the benefits of review scrutiny of comments is a chance for the reviewer to pipe up and say "This comment is incomprehensible. Can you explain it (on a ticket) in a way that someone else might fix it, rather than a note to yourself?" (with the implication being "a note to yourself that nobody, including you six months later, is ever going to understand"). I think that a Twisted reviewer encountering a comment like /* XXX From here until type is allocated, "return NULL" leaks bases! */ would probably say something like "XXX should probably be something more informative, like 'WARNING:' or 'CAREFUL:'". Clearly if XXX is being used in this way there's no point in filing a ticket, and the comment here is perfec
Re: [Python-Dev] Using Cython for standard library?
C. Titus Brown wrote: Cython is a non-backwards-compatible fork of Pyrex, forked for the usual reasons [0]. As I see it, there are two main reasons for the fork: (1) I prefer to develop slowly and carefully, whereas the Cython people like to rush ahead and try out wild ideas. (2) There's a difference in philosophy between Pyrex and Cython. The goal of Pyrex is to provide an easy way to write efficient extension modules, and exact compatibility with Python is secondary. The Cython developers want to create a true superset of Python, so compatibility is more important to them. Considering that Pyrex favours efficiency over compatibility with Python where there's a conflict, it would perhaps be more appropriate for standard library use, if such a thing were to be considered at all -- which doesn't seem very likely in the near future. -- Greg ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] file open in python interpreter
> any clues? how can i go around this? What Nick said. Run it in a debugger, set a breakpoint on the failing system call, and triple check that the arguments you are passing are really the ones that you want to pass. Are you sure the current directory is what you think it is? Are you sure that the relative path doesn't, by accident, refer to the root directory? Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Fwd: Removal of GIL through refcounting removal.
On Tue, Nov 4, 2008 at 5:31 AM, Leonardo Santagada <[EMAIL PROTECTED]> wrote: > > On Nov 4, 2008, at 12:11 AM, Josiah Carlson wrote: > >> There is no shortage of algorithms (such as matrix multiplication) that >> are parallelizable but not particularly good candidates for an IPC-based >> multiprocessing paradigm. >> >> Ahh, but those algorithms aren't going to be written in Python; they are >> going to be written in C, and are going to manipulate memory directly. With >> such things, you can use standard Python threads, call into your C runtime, >> and release the GIL. Alternatively, you can use the mmap module to store >> your data, shared across multiple processes, using the same direct-memory >> access as you would with multiple threads and GIL release. > > > That's the thing, right now you can't even use python to sketch one of those > algorithms and for example compare performance with a non parallel version. Why not? The data needs to come from *somewhere*, it doesn't appear from the aether into your RAM. And even if it *does* appear from nowhere, you can always use synthetic data to compare single process vs. multiprocess - in pure Python. > Also with four cores on common intel processor you might be happy with your > algo being 4x faster while being quite easy to develop (not needing to use > c). Indeed, but it's not uncommon that using psyco you can get 2-4x (if not occasionally 10-100x), with similar (though generally superior) performance gains from using Pyrex/Cython. And really, after implementing your base algorithm in Python, translating it to C/C++ is generally pretty straightforward for anything not involving complex data structures or string parsing. > I personally think this will be too much work on And that's up to every programmer to decide for themselves. I've done a few of the things I've mentioned (shared mmaps, C code controlled by Python, etc.), and have been very pleased with the speedups I attained. With better tools (Python, multiprocessing, Pyrex/Cython, ...) now available, I can't help but say, "back in my day, we just wrote it in C and were *happy* that we didn't have to use Fortran". ;) - Josiah ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Looking for VCS usage scenarios
> Also, something that should be done for ANY candidate VCS: translate the > current Python developer FAQ to give the appropriate answers for the > candidate VCS. What I would like to see for at least the favored system: provide a demo installation that is complete in the sense that immediate switchover would be possible. I know this is probably far down the road, but I like to bring it up at this point so that proponents are aware that doing that might be required. Of course, proponents of competing systems are encouraged to do the same thing. FWIW, I don't consider the current bazaar installation sufficient here. It does give a useful insight for those of us unfamiliar with that kind of system, and certainly allows those who want to develop with bzr already to do so - in the dvcs manner. However, for a complete "demo" installation, I'd like to see a repository that contains all revisions, not just selected ones, has post-commit email notifications (or explains why it can't provide these), etc. Regards, Martin P.S. After we move to the new machine, there should be plenty of disk space for this kind of experiment. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Looking for VCS usage scenarios
On Tue, Nov 4, 2008 at 13:28, Nick Coghlan <[EMAIL PROTECTED]> wrote: > Ralf Schmitt wrote: >> On Tue, Nov 4, 2008 at 6:45 PM, Georg Brandl <[EMAIL PROTECTED]> wrote: >>> Ralf Schmitt schrieb: I think you really should not exclude any dvcs based on it's implementation language. I.e. requiring it being written in python for the sake of "eating your own dogfood" is just a very weak argument. git is certainly missing from your list. >>> And by then, why not include darcs, GNU arch and monotone? >>> >> >> please include all of them and choose the best one. that was my point. > > The PEP can at least mention darcs, arch, monotone, and then drop them > on the basis of not having a compelling reason to prefer them to the 3 > more popular choices. I don't see any need to go into the "other > choices" further than that - our needs aren't wildly divergent from that > of other open source projects, and while popularity is a poor final > criteria, it's a decent filter to narrow the scope of the initial > investigation. > Tossed in two sentences. > For git, without a killer feature that is "oh my god, wow!" when > compared to the features offered by Mercurial and Bazaar, the > implementation language is enough to push it to the back of the list for > me. Sure, that's a pretty lousy criteria in the absolute sense, but > given that it is obvious that there are two perfectly acceptable options > written in the language we ourselves are implementing, then it should > take something pretty impressive to get us to look at a third choice. > Which is exactly how I feel. > That would leave the PEP to focus on the Mercurial/Bazaar comparison (as > Brett originally intended) and I'm sure even that limited comparison is > going to give the infrastructure committee (and any other willing > assistants from python-dev) plenty of work to do). > I must be insane. -Brett ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] DVCS PEP update
OK, I give: git is in the running. But do realize it will take a lot for it to beat out bzr or hg. I have emailed some people who have shown allegiance to a specific DVCS to seeif they are willing to fill in the usage scenarios for me along with converting the dev FAQ so that we all have a clear way to learn how to use the things. If any of the people I emailed turn down being the leader fo ra specific DVCS I will email here looking for volunteers. -Brett ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Looking for VCS usage scenarios
Cosmin Stejerean wrote: Yes, Python is fast enough most of the time, but when it's not we put a lot of effort into making it faster. That's why we have a good collection of modules with C extensions to speed up computationally intensive applications So the Pythonic solution is, of course, to make one of the existing VCSes written in C available as an extension module. :-) -- Greg ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Looking for VCS usage scenarios
Cosmin Stejerean writes: > On Tue, Nov 4, 2008 at 12:13 PM, Barry Warsaw <[EMAIL PROTECTED]> wrote: > > We're Python programmers. We're used to people telling us our > > tool is too slow. We just say it does the job superbly and it's > > usually fast enough. :) > I don't agree with completely dismissing performance just because it's > Python. Oh, it's just an in-joke. He's doing a "my snake is longer than your snake" because I sniped at Gustavo with a "TOOWTDI". > I see no excuse to let the fact that it's Python make it acceptable > to have an application with otherwise unacceptable performance. Barry and many others obviously find the performance of non-git VCSes acceptable. On the other hand, Brett has conceded that git's performance has enough fans that it has to be on the table (even though he admits to a strong bias against it, that's the major concession). Performance is *always* a consideration. IMO, now that it has been acknowledged that performance varies considerably among VCSes, it's more important at this point to figure out what Python development's *other* requirements are. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
