Re: [Python-Dev] Status on PEP-431 Timezones
On Tue, Jul 28, 2015 at 10:25 AM, Łukasz Rekucki lreku...@gmail.com wrote: Maybe instead of trying to decide who is wrong and which approach is broken, Python just needs a more clear separation between timezone aware objects and naive ones? Well, the separation is pretty clear already. Tim wants to have naive timezone aware objects, ie datetime objects that have a time zone but ignores the time zone, except when converting to other time zones. I have yet to see a use case for that. //Lennart ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status on PEP-431 Timezones
On Tue, Jul 28, 2015 at 8:11 AM, Tim Peters tim.pet...@gmail.com wrote: [Tim] timedelta objects only store days, seconds, and microseconds, [Lennart Regebro rege...@gmail.com] Except that they don't actually store days. They store 24 hour periods, Not really. A timedelta is truly an integer number of microseconds, and that's all. That's what I said. Timedeltas, internally assume that 1 day is 24 hours. Or 8640 microseconds. That's the assumption internally in the timedelta object. The problem with that being that in the real world that's not true. 24 hours is 24 hours at any time in _any_ time zone, ignoring leap seconds. timedeltas are durations, not points in time. time zones make no sense applied to durations. My point exactly. And should not then adding 8640 microseconds to a datetime actually result in a datetime that happens 8640 microseconds later? ie, it assumes that one day is always 24 hours. That's really not what it's doing That is really exactly what the timedelta is doing, as you yourself, just a few lines above say. used in explanations. What somedatetime+timedelta really does is simpler than that: it adds the number of microseconds represented by the timedelta to somedatetime, No it doesn't. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status on PEP-431 Timezones
On Tue, Jul 28, 2015 at 9:00 AM, Tim Peters tim.pet...@gmail.com wrote: [Lennart Regebro] If you don't have to deal with DST, then you don't have to have tzinfo's in your date objects. There are no tzinfos on date objects. I assume Ronald is talking about datetime objects. Of course, I meant datetime objects. In everything else, I stand by my original claim. If you want naive datetime obejcts, you should use naive datetime objects. My opinion is and remains that intentionally breaking datetime arithmetic to make non-naive objects behave in a naive way was a mistake. //Lennart ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status on PEP-431 Timezones
On Tue, Jul 28, 2015 at 3:17 PM, Mark Lawrence breamore...@yahoo.co.uk wrote: Tim asked for my definition of two weeks so I've given it. With respect to that in reality this is true, for me, with my application, making my statement above correct. For my application we could go from GMT to BST and back on successive days throughout the year and it wouldn't make any difference. Right. You want a timezone naive datetime. Your usecase is covered, no problemo. //Lennart ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status on PEP-431 Timezones
On Tue, Jul 28, 2015 at 1:55 PM, Mark Lawrence breamore...@yahoo.co.uk wrote: One week == 7 days == 7 * 24 hours Two weeks = 2 * (one week) Right, and that of course is not true in actual reality. I know you are not interested in DST, but with a timezone that has DST, two times a year, the above statement is wrong. As I've said elsewhere I've no interest in DST, at least right here, right now, let alone leap seconds. When I run my cashflow forecast the balance in my bank account one year from today isn't going to be influenced by UK clocks falling back to GMT at the end of October and on to BST at the end of next March. And then you should not use timezoned datetimes, but use naive ones. If you don't care about the timezone, then don't use it. Problem solved. It should be noted here that Python is one of the few languages that actually lets you do that. It's not very common to support time zone naive datetimes. Correct. What I would like to know is how many people are in my position, how many people are in the situation of needing every possible combination of dates, times, daylight saving, local time zone rules and anything else you can think of under the sun, and how many are on the scale somewhere in between these two extremes. There are a few positions. 1. Not caring. datetime supports that as of today. This is probably the most common case. That certainly is the case for me most of the time I need to do something with datetimes. It's usually measuring a few seconds of time or calculating dates. 2. Caring about time zones including DST's. IMO, this is the most common case once you care about time zones. You have several time zones, and you want conversion between them to work, and if you say one hour, you mean one hour. Datetime as of today does not support this, and Tim has declared that it never will, at least not before Python 4 (which amounts to much the same thing). 3. The position of Tim and Guido, which is I want my time zone aware datetimes to ignore the time zone, except when converting to other time zones. I have yet to see a use case for that, and hence I am still not convinced that this position is useful, I think it is only based on misunderstanding. 4. ? Are there more positions, something I have missed? //Lennart ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status on PEP-431 Timezones
On Tue, Jul 28, 2015 at 1:55 PM, Mark Lawrence breamore...@yahoo.co.uk wrote: Correct. What I would like to know is how many people are in my position, how many people are in the situation of needing every possible combination of dates, times, daylight saving, local time zone rules and anything else you can think of under the sun, and how many are on the scale somewhere in between these two extremes. I should also point out that this is not about supporting everything under the sun in any form at all. It's about whether the arithmetic on a *time zone aware* date time should use that time zone information, or if the time zone aware datetime should ignore the attached time zone. //Lennart ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status on PEP-431 Timezones
On Tue, Jul 28, 2015 at 10:26 PM, Tim Peters tim.pet...@gmail.com wrote: I have yet to see a use case for that. Of course you have. When you address them, you usually dismiss them as calendar operations (IIRC). Those are not usecases for this broken behaviour. I agree there is a usecase for where you want to add one day to an 8am datetime, and get 8am the next day. Calling them date operations or calendar operations is not dismissing them. I got into this whole mess because I implemented calendars. That use case is the main usecase for those operations. But that usecase is easily handled in several ways. Already today in how datetime works, you have two solutions: The first is to use a time zone naive datetime. This is what most people who want to ignore time zones should use. The other is to separate the datetime into date and time objects and add a day to the date object. But most importantly, there are better ways to solve this that datetime today doesn't support at all: 1. You can use something like dateutil.rrule for repeating events. (works today, but requires a third-party module). 2. timedelta could not incorrectly pretend that one day is always 24 hours, but actually handle days separately, and always increase the days when a day is given. (This would however mean we no longer can support fractional days). 3. There could be a datetelta() object that makes operations on dates, leaving hours, minuts, seconds and microseconds alone, or something like Lubridates Perod and Delta objects, where a Period() essentially operates on wall time, and a Duration() operates on real time. So that's not the usecase I'm asking for. I am specifically asking for a usecase where I want an object that is timezone aware, but ignores the timezone for everything other than conversion to other timezones. Because that's what datetime implements. That's what I want a usecase for. I want the same time next day is not such a usecase. And I don't want that use case for you to convince me that we shouldn't change datetime. You say it breaks too much. OK, if you say so. I don't know. I want to know if that use case actually exists, because I don't think it does. But it doesn't matter whether you _call_ them calendar operations, or anything else. What they're called doesn't change any of the high-order bits: they are use cases, they already work, they have worked that way for a dozen years (an eternity in computer time), they were always intended to work that way, and the docs have always said they work that way. They only work like that because people have adapted to how datetime does things. If datetime had done this properly from the start, it would have worked even better. I do think you're missing my fundamental objection: no matter what intended and documented thing datetime (or any other module) has done all along, and regardless of whether I loved it or hated it, I'd be just as annoying about insisting we cannot intentionally break existing code I stopped arguying for changing datetime two days ago. I've also mentioned that several times. using that thing in non-trivial ways without a justification so compelling that I can't recall a case of it ever happening. Well, I've seen several of those on Stackoverflow. //Lennart ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status on PEP-431 Timezones
On Tue, Jul 28, 2015 at 7:27 AM, Ethan Furman et...@stoneleaf.us wrote: Lennart, are you saying you would leave naive objects alone, and fix the tz-aware objects only? Naive objects are not broken, so they can't be fixed. Which I guess means yes. :-) //Lennart ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status on PEP-431 Timezones
On Mon, Jul 27, 2015 at 4:27 PM, Steve Dower steve.do...@microsoft.com wrote: Am I the only one feeling like this entire thread should be moved to python-ideas at this point? Well, there isn't any idea to discuss. :-) It's just an explanation of the problem space. Perhaps it should be moved somewhere else though. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status on PEP-431 Timezones
On Mon, Jul 27, 2015 at 5:12 PM, Paul Moore p.f.mo...@gmail.com wrote: Does thinking of the problem in terms of timedeltas not containing enough information to make a_time + a_timedelta a well-defined operation if a_time uses a non-fixed-offset timezone, make it any easier to find a way forward? Well, I think it is a well-defined operation, but that datetime currently does it wrongly, to be honest. Adding 3600 seconds to a datetime should move that datetime 3600 seconds forward at all time. I just do not see a usecase for doing anything else, to be honest. But if somebody has one, I'm all ears. The problem here is that the issue of get the next day has been mixed into timedeltas, when it in my opinion is an entirely different issue that should be kept separate from timedeltas. It is possible to implement something so that you can both have realtimedeltas and walltimedeltas where adding one hour might give you two hours (or an error) but as per above I can't think of a usecase. //Lennart ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status on PEP-431 Timezones
On Mon, Jul 27, 2015 at 4:13 PM, Steven D'Aprano st...@pearwood.info wrote: To me, Paul's example is a datetime operation: you start with a datetime (7am today), perform arithmetic on it by adding a period of time (one day), and get a datetime as the result (7am tomorrow). Well, OK, let's propose these wordings: It looks like a date operation, ie, add one to the date, but in reality it's a time operation, ie add 86400 seconds to the time. These things sound similar but are very different. I called it a calendar operation, because these operation include such things as add one year, where you expect to get the 27th of July 2016, but you will get the 26th if you use a timedelta, because 2016 is a leap year. So we need to separate date (or calendar) operations from time operations. The same thing goes with months, add 30 days, and you'll sometimes get the same result as if you add one month and sometimes not. timedelta adds time, not days, month or years. Except when you cross a DST border where it suddenly, surprisingly and intentionally may add more or less time than you told it to. //Lennart ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status on PEP-431 Timezones
On Mon, Jul 27, 2015 at 3:59 PM, R. David Murray rdmur...@bitdance.com wrote: I'm not sure if the opinion of a relatively inexperienced timezone user (whose head hurts when thinking about these things) is relevant, but in case it is: My brief experience with pytz is that it gets this all wrong. (Wrong is in quotes because it isn't wrong, it just doesn't match my use cases). If I add a timedelta to a pytz datetime that crosses the DST boundary, IIRC I get something that still claims it is in the previous timezone (which it therefore seemed to me was really a UTC offset), and I have to call 'normalize' to get it to be in the correct timezone (UTC offset). Right. I don't remember what that does to the time, and I have no intuition about it (I just want it to do the naive arithmetic!) But what is it that you expect? That you add one hour to it, and the datetime moves forward one hour in actual time? That's doable, but during certain circumstance this may mean that you go from 1AM to 1AM, or from 1AM to 3AM. Or do you expect that adding one hour will increase the hour count with one, ie that the wall time increases with one hour? This may actually leave you with a datetime that does not exist, so that is not something you can consistently do. Dates and times are tricky, and especially with DST it is simply possible to make an implementation that is intuitive in all cases to those who don't know about the problems. What we should aim for is an implementation that is easy to understand and hard to make errors with. This makes no sense to me, since I thought a tzinfo object was supposed to represent the timezone including the DST transitions. I presume this comes from the fact that datetime does naive arithmetic and pytz is trying to paste non-naive arithmetic on top? Exactly. So, would it be possible to take the timezone database support from pytz, and continue to implement naive-single-zone arithmetic the way Tim proposes, and have it automatically produce the correct UTC offset and UTC-offset-label afterward, without a normalize call? That depends on your definition of correct. //Lennart ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status on PEP-431 Timezones
On Mon, Jul 27, 2015 at 4:45 PM, Nick Coghlan ncogh...@gmail.com wrote: On 28 July 2015 at 00:27, Steve Dower steve.do...@microsoft.com wrote: Am I the only one feeling like this entire thread should be moved to python-ideas at this point? Since this is an area where the discussion of implementation details and the discussion of the developer experience can easily end up at cross purposes, I'm wondering if there may be value in actually splitting those two discussions into different venues by creating a datetime-sig, and specifically inviting the pytz and dateutil developers to participate in the SIG as well. +1 for that. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status on PEP-431 Timezones
On Mon, Jul 27, 2015 at 6:15 PM, Nikolaus Rath nikol...@rath.org wrote: On Jul 27 2015, Lennart Regebro rege...@gmail.com wrote: That you add one hour to it, and the datetime moves forward one hour in actual time? That's doable, but during certain circumstance this may mean that you go from 1AM to 1AM, or from 1AM to 3AM. Or do you expect that adding one hour will increase the hour count with one, ie that the wall time increases with one hour? This may actually leave you with a datetime that does not exist, so that is not something you can consistently do. Apologies for asking yet another dumb question about this, but I have the impression that a lot of other people are struggling with the basics here too. Can you tell us which of the two operations datetime currently implements? It's intended that the first one is implemented, meaning that datetime.now() + timedelta(hours=24) can result in a datetime somewhere between 23 and 25 hours into the future. Or well, any amount, in theory, I guess some changes are more than an hour, but that's very unusual. And when people talk about explicitly converting to UTC and back, does that mean that if you're (again, with the current implementation) converting to UTC, *then* add the one hour, and then convert back, you get the other operation (that you don't get when you directly add 1 day)? Yes, exactly. //Lennart ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status on PEP-431 Timezones
On Mon, Jul 27, 2015 at 6:42 PM, Alexander Belopolsky alexander.belopol...@gmail.com wrote: I think this describes what was originally your *second*, not *first* option. Yes, you are absolutely correct, I didn't read my own description of the options carefully enough. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status on PEP-431 Timezones
On Mon, Jul 27, 2015 at 9:47 PM, Ethan Furman et...@stoneleaf.us wrote: On 07/27/2015 07:46 AM, Lennart Regebro wrote: Well, OK, let's propose these wordings: It looks like a date operation, ie, add one to the date, but in reality it's a time operation, ie add 86400 seconds to the time. These things sound similar but are very different. I have to disagree. If I have my alarm at 7am (localtime ;) so I can be at work at 8am I don't care exactly how many seconds have passed, that alarm better go off at 7am local time. Right. And then adding 86400 seconds to it is not the right thing to do. //Lennart ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status on PEP-431 Timezones
On Tue, Jul 28, 2015 at 12:03 AM, Tim Peters tim.pet...@gmail.com wrote: timedelta objects only store days, seconds, and microseconds, Except that they don't actually store days. They store 24 hour periods, which, because of timezones changing, is not the same thing. This is also clearly intended, for example timedelta allows floats, and will convert the fraction into seconds. And as you have repeated many times now, the datetime module's arithmetic is naive ie, it assumes that one day is always 24 hours. The problem with that assumption is that it isn't true. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status on PEP-431 Timezones
On Tue, Jul 28, 2015 at 12:11 AM, Ronald Oussoren ronaldousso...@mac.com wrote: I totally agree with that, having worked on applications that had to deal with time a lot and including some where the end of a day was at 4am the following day. That app never had to deal with DST because not only are the transitions at night, the are also during the weekend. If you don't have to deal with DST, then you don't have to have tzinfo's in your date objects. You can have just truly naive objects without DST information, and this will work just fine. I think most people's expectations are that datetime's that *are* time zone aware, should actually deal correctly with those time zones. It might be nice to have time zone aware datetime objects with the right(TM) semantics, but those can and should not replace the naive objects we know and love. Yes, they most certainly should. I will try to shut up now, but let me be clear on that the time zone support as it stands now is intentionally broken. Not naive, *broken*. All the usecases people have here for supporting naive objects would work just as well if they actually used naive objects, ie datetimes with no timezone info. If you explicitly do NOT want the datetimeobject to care about timezones, then you should not add a timezone to the object. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status on PEP-431 Timezones
On Tue, Jul 28, 2015 at 3:22 AM, Mark Lawrence breamore...@yahoo.co.uk wrote: To me a day is precisely 24 hours, no more, no less. OK. In my mission critical code, which I use to predict my cashflow, I use code such as. timedelta(days=14) Is somebody now going to tell me that this isn't actually two weeks? Yes, I'm telling you that, now. The two claims One day is always precisely 24 hours and 14 days is two weeks are not both true. You have to choose one. //Lennart ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status on PEP-431 Timezones
On Mon, Jul 27, 2015 at 12:15 AM, Paul Moore p.f.mo...@gmail.com wrote: I think the current naive semantics are useful and should not be discarded lightly. At an absolute minimum, there should be a clear, documented way to get the current semantics under any changed implementation. As an example, consider an alarm clock. I want it to go off at 7am each morning. I'd feel completely justified in writing tomorrows_alarm = todays_alarm + timedelta(days=1). That's a calendar operation made with a timedelta. The days attribute here is indeed confusing as it doesn't mean 1 day, it means 24 hours. //Lennart ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status on PEP-431 Timezones
On Mon, Jul 27, 2015 at 4:04 AM, Tim Peters tim.pet...@gmail.com wrote: Realistically, default arithmetic behavior can't change in Python 3 (let alone Python 2). Then we can't implement timezones in a reasonable way with the current API, but have to have something like pytz's normalize() function or similar. I'm sorry I've wasted everyones time with this PEP. //Lennart ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status on PEP-431 Timezones
On Mon, Jul 27, 2015 at 9:09 AM, Tim Peters tim.pet...@gmail.com wrote: But for a dozen years it's sufficed to do what Paul did. No, it never did suffice. It's just that people have been doing various kinds of workarounds to compensate for these design flaws. I guess they need to continue to do that for the time being. //Lennart ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status on PEP-431 Timezones
On Mon, Jul 27, 2015 at 9:09 AM, Tim Peters tim.pet...@gmail.com wrote: It's an instance of single-timezone datetime arithmetic, of the datetime + timedelta form. No, because in that case it would always move 24 hours, and it doesn't. It sometimes moves 23 hours or 25 hours, when crossing a DST border. That is a calendar operation, in the disguise of a datetime arithmetic. But this discussion is now moot, if we can't change this, then we can't change this, and PEP431 is dead. The only reasonable way out if this mess is a completely new module for dates and times that doesn't make these kind of fundamental mistakes. I sincerely doubt I have the time to implement that this decade. //Lennart ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status on PEP-431 Timezones
On Mon, Jul 27, 2015 at 10:47 AM, Paul Moore p.f.mo...@gmail.com wrote: I'm confused by your position. If it's 7am on the clock behind me, right now, then how (under the model proposed by the PEP) do I find the datetime value where it will next be 7am on the clock? PEP-431 does not propose to implement calendar operations, and hence does not address that question. //Lennart ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status on PEP-431 Timezones
On Mon, Jul 27, 2015 at 11:05 AM, Paul Moore p.f.mo...@gmail.com wrote: Am I right to think that because you say implement calendar operations this is not, as far as you are aware, something that already exists in the stdlib (outside of datetime)? I'm certainly not aware of an alternative way of doing it. Right, I think you need to use relativedelta (or rrule) from dateutil, unless you want to do it yourself, which of course in most cases is quite easy. //Lennart ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status on PEP-431 Timezones
On Sun, Jul 26, 2015 at 8:05 AM, Tim Peters tim.pet...@gmail.com wrote:
The Python docs also are quite clear about that all arithmetic within
a single timezone is naive. That was intentional. The _intended_
way to do aware arithmetic was always to convert to UTC, do the
arithmetic, then convert back.
We can't explicitly implement incorrect timezone aware arithmetic and
then expect people to not use it. We can make the arithmetic correct,
and we can raise an error when doing tz-aware arithmetic in a
non-fixed timezone. But having an implementation we know is incorrect
and telling people don't do that doesn't seem like a good solution
here.
Why do we even have timezone aware datetimes if we don't intend them
for usage? There could just be naive datetimes, and timezones, and let
strftime take a timezone that is used when formatting. And we could
make date-time creation into a function that parses data including a
timezone, and returns the UTC time of that data.
But then again, if we do that, we could just as well have that
timezone as an attribute on the datetime object, and let strftime use
so it doesn't have to be passed in. And we could let the __init__ of
the datetime take a timezone and do that initial conversion to UTC.
Python's datetime never intended to support that directly.
I think it should. It's expected that it supports it, and there is no
real reason not to support it. The timezone handling becomes
complicated if you base yourself on localtime, and simple if you base
yourself on UTC.
As you agree, we recommend to people to use UTC at all times, and only
use timezones for input and output. Well, what I'm now proposing is to
take that recommendation to heart, and change datetime's
implementation so it does exactly that.
I saw the previous mention of pure vs practical, and that is often
a concern. Here it clearly is not. This is a choice between impure,
complicated and impractical, and pure, simple and practical.
Is it the case that pytz also fails in the cases your attempts fail?
No, that is not the case. And if you wonder why I just don't do it
like pytz does it, it's because that leads to infinite recursion, much
as discussions on this mailing list does. ;-) And this is because we
need to normalize the datetime after arithmatic, but normalizing is
arithmetics.
Batteries included has some attractions all on its own. On top of
that, adding is_dst-like flags to appropriate methods may have major
attractions.
Ah, but it already happens that way
No, in fact it does not. Pytz makes that happen only through a
separate explicit normalize() call (and some deep cleverness to keep
track of which timezone offset it is located in). dateutil.tz can't
guarantee these things to be true, because it doesn't keep track of
ambiguous times. So no, it does not already happen that way.
from dateutil.zoneinfo import gettz
from datetime import *
dt = datetime(2015, 11, 1, 0, 30, tzinfo=est)
dt2 = dt + timedelta(hours=1)
utc = gettz('Etc/UTC')
dtutc = dt.astimezone(utc)
dt2utc = dt2.astimezone(utc)
(dt2utc-dtutc).total_seconds()
7200.0
You add one hour, and you get a datetime that happens two hours later.
So no, it does not already happen that way.
In pytz the datetime will be adjusted after you do the normalize call.
I apologize if I've come off as unduly critical - I truly have been
_only_ trying to find out what the problem is. That helps! Thank
you. Note that I've had nothing to do with datetime (except to use
it) for about a decade. I have no idea what you, or anyone else, has
said about it for years years until this very thread caught my
attention this week. Heck, for all I know, Guido _demanded_ that
datetime arithmetic be changed - although I doubt it ;-)
It's not a question of changing datetime arithmetic per se. The PEP
does indeed mean it has to be changed, but only to support ambiguous
and non-existent times.
It's helpful to me to understand, which I hadn't done before, that
this was never intended to work. That helps me argue for changing
datetimes internal implementation, once I get time to do that. (I'm
currently moving, renovating a new house, trying fix up a garden that
has been neglected for years, and insanely, write my own code editor,
all at the same time, so it won't be anytime soon).
There's more than one decision affecting this In cases where a single
local time corresponds to more than one UTC time (typically at the end
of DST, when a local hour repeats), datetime never did give any clear
way to do the intended conversion from that local time _to_ UTC.
But resolving such ambiguities has nothing to do with how arithmetic
works: it's utterly unsolvable by any means short of supplying new
info (which UTC value is intended? AKA is_dst).
The changing arithmetic discussion is a red herring.
Now my wife insist I help her pack, so this is the end of this
discussion for me. If i continue it will be only as a part of
discussing how we change
Re: [Python-Dev] Status on PEP-431 Timezones
On Sun, Jul 26, 2015 at 2:56 AM, Tim Peters tim.pet...@gmail.com wrote: However, the _body_ of the PEP said nothing whatsoever about altering arithmetic. The body of the PEP sounds like it's mainly just proposing to fold the pytz package into the core. Perhaps doing _just_ that much would get this project unstuck? Hope springs eternal :-) The pytz package has an API and a usage that is different from the datetime() module. One of the things you need to do is that after each time you do arithmetic, you have to normalize the result. This is done because the original API design did not realize the difficulties and complexities of timezone handling and therefore left out things like ambiguous times. The PEP attemps to improved the datetime modules API so that it can handle the ambiguous times. It also says that the implementation will be based on pytz, because it was my assumption that this would be easy, since pytz already handles ambiguous times. During my attempt of implementing it I realized it wasn't easy at all, and it wasn't as easy as folding pytz into the core. Yes, the PEP gives that impression, because that was the assumption when I wrote the draft. Just folding pytz into the core without modifying the API defeats the whole purpose of the PEP, since installing pytz is a trivial task. Like what? I'm still looking for a concrete example of what the problem is (or even a problem). A problem is that you have a datetime, and add a timedelata to it, and it should then result in a datetime that is actually that timedelta later. And if you subtract the same timedelta from the result, it should return a datetime that is equal to the original datetime. This sounds ridiculously simple, and is ridiculously difficult to make happen in all cases that we want to support (Riyahd time zone and leap seconds not included). That IS the specific, concrete problem, and if you don't believe me, there is nothing I can do to convince you. Perhaps I am a complete moron and simply incompetent to do this, and in that case I'm sure you could implement this over a day, and then please do so, but for the love of the founders of computing I'm not going to spend more time repeating it on this mailing list, because then we would do better in having you implement this instead of reading emails. Me repeating this a waste of time for everyone involved, and I will now stop. discussing why Python's datetime is dfferent I was not involved in the discussion then, and even if I had been, that's still before I knew anything about the topic. I don't know what the arguments were, and I don't think it's constructive to try to figure out exactly why that decision was made. That is all to similar to assigning blame, which only makes people feel bad. Those who get blamed feel bad, and those who blame feel like dicks and onlookers get annoyed. Let us look forward instead. I am operating both without any need to defend that decision, as I was not involved in it, and I am operating with 20/20 hindsight as I am one of the few people having tried to implement a timezone implementation that supports ambiguous datetimes based on that decision. And then it is perfectly clear and obvious that the decision was a mistake and that we should rectify it. The only question for me is how and when. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status on PEP-431 Timezones
On Sat, Jul 25, 2015 at 7:12 AM, Tim Peters tim.pet...@gmail.com wrote: [Lennart Regebro rege...@gmail.com] And I would want to remind everyone again that this is not a question of the problem being impossible. It's just really complex to get right in all cases, and that always having the UTC timestamp around gets rid of most of that complexity. Could you please be explicit about what the problem is? Everyone here is guessing at what you think the problem is. The problem is that it is exceedingly complicated to get all the calculations back and forth between local time and UTC to be correct at all times and for all cases. It really doesn't get more specific than that. I don't remember which exact problem it was that made me decide that this was not the correct solution and that we should use UTC internally, but I don't think that matters, because I'm also sure that it was not the last case, as I was far from near the end in adding testcases. Once again I'm sure it's not impossible to somehow come up with an implementation and an API that can do this based on local time, but once again I am of the opinion that it is the wrong thing to do. We should switch to using UTC internally, because that will make everything so much simpler. I am in no way against other people implementing this PEP, but I think you will end up with very complex code that will be hard to maintain. There really is a reason every other date time implementation I know of uses UTC internally, and there really is a reason why everyone always recommends storing date times in UTC with the time zone or offset separately. //Lennart ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status on PEP-431 Timezones
And I would want to remind everyone again that this is not a question of the problem being impossible. It's just really complex to get right in all cases, and that always having the UTC timestamp around gets rid of most of that complexity. On Sat, Jul 25, 2015 at 3:39 AM, Tim Peters tim.pet...@gmail.com wrote: [ISAAC J SCHWABACHER ischwabac...@wisc.edu] ... I disagree with the view Tim had of time zones when he wrote that comment (and that code). It sounds like he views US/Eastern and US/Central as time zones (which they are), but thinks of the various America/Indiana zones as switching back and forth between them, rather than being time zones in their own right You can think of them anyway you like. The point of the code was to provide a simple efficient way to convert from UTC to local time in all time zones in known actual use at the time; the point of the comment was to explain the limitations of the code. Although, as Allexander noted, the stated assumptions are stronger than needed. I think the right perspective is that a time zone *is* the function that its `fromutc()` method implements, Fine by me ;-) although of course we need additional information in order to actually compute (rather than merely mathematically define) its inverse. Daylight Saving Time is a red herring, Overstated. DST is in fact the _only_ real complication in 99.99% of time zones (perhaps even 99.9913% ;-) ). As the docs say, if you have some crazy-ass time zone in mind, fine, that's why fromutc() was exposed (so your; crazy-ass tzinfo class can override it). and assumptions 2 and 4 Nitpick: 4 is a consequence of 2, not an independent assumption. in that exposition are just wrong from this point of view. As above, there is no particular POV in this code: just a specific fromutc() implementation, comments that explain its limitations, and an invitation in the docs to override it if it's not enough for your case. In the worst case, Asia/Riyadh's two years of solar time completely shatter these assumptions. Sure. But, honestly, who cares? Riyadh Solar Time was so off-the-wall that even the Saudis gave up on it 25 years ago (after a miserable 3-year experiment with it). Practicality beats purity. [eliding a more-general view of what time zones really are] I'm not eliding it because I disagree with it, but because time zones are political constructions. The math we make up may or may not be good enough to deal with all future political abominations; for example: ... This assumption would be violated if, for example, some jurisdiction decided to fall back two hours by falling back one hour and then immediately falling back a second hour. I recommend the overthrow of any such jurisdiction and its (annexation by the Netherlands)[3]. That's not objectively any more bizarre than Riyadh Solar Time. Although, if I've lived longer than you, I may be more wary about the creative stupidity of political schemes ;-) ... (Lennart, I think this third assumption is the important part of your no changes within 48 hours of each other assumption, The 48 hours bit came from Alexander. I'm personally unclear on what Lennart's problems are. ... All of these computations can be accomplished by searches of ordered lists and applications of $fromlocal_i$. Do you have real-world use cases in mind beyond supporting long-abandoned Riyadh Solar time? ... With this perspective, arithmetic becomes translate to UTC, operate, translate back, which is as it should be. There _was_ a POV in the datetime design about that: no, that's not how it should be. Blame Guido ;-) If I add, say, 24 hours to noon today, I want to get noon tomorrow, and couldn't care less whether DST started or stopped (or any other political adjustment was made) in between. For that reason, it was wholly intentional that datetime + timedelta treats datetime as naive. If that's not what someone wants, fine, but then they don't want Python's datetime arithmetic BTW, there's no implication that they're wrong for wanting something different; what would be wrong is insisting that datetime's POV is wrong. Both views are valid and useful, depending on the needs of the application. One had to picked as the built-in behavior, and naive won. ... But IIUC what Lennart is complaining about I don't, and I wish he would be more explicit about what the problem(s) is(are). is the fact that the DST flag isn't part of and can't be embedded into a local time, so it's impossible to fold the second parameter to $fromlocal$ into $t$. Without that, a local time isn't rich enough to designate a single point in time and the whole edifice breaks. You can blame Guido for that too ;-) , but in this case I disagree(d) with him: Guido was overly (IMO) annoyed by that the only apparent purpose for a struct tm's tm_ isdst flag was to disambiguate local times in a relative handful of
Re: [Python-Dev] Status on PEP-431 Timezones
On Thu, Jul 23, 2015 at 5:07 PM, Guido van Rossum gu...@python.org wrote: Sorry for reviving this thread, but I was cornered at EuroPython with a question about the status of this PEP. It seems the proposal ran out of steam and has now missed the Python 3.5 train. What happened? Is the problem unsolvable? Or could we get this into 3.6??? It turns out it's very complex to solve this when internally storing the time as the local time. Basically you have to normalize the time (ie check if daylight savings have changed) when doing arithmetic, but normalize is doing arithmetic, and you get infinite recursion. I've tried various ways to solve this but ran out of steam/brainpower. I think we should look into storing as UTC internally instead. It's a big change (and also needs handling pickles in a backwards compatible way) but I think that's the way forward. //Lennart ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status on PEP-431 Timezones
On Wed, Apr 15, 2015 at 11:10 AM, Chris Angelico ros...@gmail.com wrote: Bikeshed: Would arithmetic be based on UTC time or Unix time? It'd be more logical to describe it as adding six hours means adding six hours to the UTC time, but it'd look extremely odd when there's a leap second. It would ignore leap seconds. If you want to call that unix time or not is a matter of opinion. Hm. I guess the internal representation *could* be EPOCH + offset, and local times could be calculated properties, which could be cached (or possibly calculated at creation). //Lennart ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status on PEP-431 Timezones
OK, so I just had a realization. Because we want some internal flag to tell if the datetime is in DST or not, the datetime pickle format will change. And the datetime pickle format changing is the biggest reason I had against changing the internal representation to UTC. So because of this, perhaps we actually *should* change the internal representation to UTC, because that makes the issues I'm fighting with now so much simpler. (I'm currently trying to get arithmetic to do the right thing in all cases, which is crazy complicated). We can add support to unpickle previous datetimes, but we won't be able to add forwards compatibility, meaning that pickles saved in Python 3.5 will not be unpicklable in Python 3.4. Please discuss. //Lennart ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status on PEP-431 Timezones
On Wed, Apr 15, 2015 at 11:43 AM, Lennart Regebro rege...@gmail.com wrote: On Wed, Apr 15, 2015 at 11:10 AM, Chris Angelico ros...@gmail.com wrote: Bikeshed: Would arithmetic be based on UTC time or Unix time? It'd be more logical to describe it as adding six hours means adding six hours to the UTC time, but it'd look extremely odd when there's a leap second. It would ignore leap seconds. If you want to call that unix time or not is a matter of opinion. Hm. I guess the internal representation *could* be EPOCH + offset, and local times could be calculated properties, which could be cached (or possibly calculated at creation). In any case there wold probably need to be a PEP on that, and that means PEP 431 wouldn't make it into 3.5, unless somebody smarter than me want to take a shot at it. //Lennart ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status on PEP-431 Timezones
On Wed, Apr 15, 2015 at 12:40 PM, Isaac Schwabacher ischwabac...@wisc.edu wrote: I am on the fence about EPOCH + offset as an internal representation. On the one hand, it is conceptually cleaner than the horrible byte-packing that exists today, but on the other hand, it has implications for future implementations of leap-second-awareness. For instance, offset measures the difference between the local time and UTC. But is it honest-to-goodness leap-second aware UTC, or is it really Unix time? This could be solved by giving each tzinfo a pointer to the UTC from which it is offset, but that sounds like a can of worms we don't want to open. But if don't and we store EPOCH + offset, we can't add leap second awareness to UTC without corrupting all persisted aware datetimes. That's true, with separate values like there is now we can easily allow 23:59:60 as a timestamp during leap seconds. I'm not entirely sure it makes a big difference though, I don't think we ever wants to deal with leap seconds by default. I don't think we ever want the standard arithmetic deal with leap seconds anyway. datetime(2012, 6, 30, 23, 30) + timedelta(seconds=3600) returning datetime(2012, 7, 1, 0, 29, 59) I guess leap second implementations should rather have special functions for arithmethics that deal with this. //Lennart ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status on PEP-431 Timezones
On Wed, Apr 15, 2015 at 3:23 PM, Stuart Bishop stu...@stuartbishop.net wrote: Huh. I didn't think you would need to change any arithmetic Not really, the problem is in keeping the date normalized after each call, and doing so the right way. Arithmetic remains 'add the timedelta to the naive datetime, and then punt it to the tzinfo to make any necessary adjustments' and I thought this would not need to be changed at all. Just punting it to tzinfo to make adjustments, ie effectively just doing what normalize() does creates infinite recursion as there is more arithmetic in there, so it's not quite that simple. I don't think this can be avoided entirely. Any ideas I can come up with that might help are worse than requiring devs to convert their datetimes to strings in the rare case they need their 3.5 pickles read with 3.4. Pickle forward compatibility isn't really expected anyway... ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status on PEP-431 Timezones
Yeah, I just realized this. As long as you use timedelta, the
difference is of course not one day, but 24 hours. That solves the
problem, but it is surprising in other ways.
In US/Eastern datetime.datetime(2002, 10, 27, 1, 0) -
datetime.timedelta(1) needs to become datetime.datetime(2002, 10, 26,
2, 0)
(Note the hour change)
I was thinking in calendrial arithmetic, which the datetime module
doesn't need to care about.
On Wed, Apr 15, 2015 at 12:59 AM, Stuart Bishop stu...@stuartbishop.net wrote:
On 14 April 2015 at 21:04, Lennart Regebro rege...@gmail.com wrote:
OK, so I realized another thing today, and that is that arithmetic
doesn't necessarily round trip.
For example, 2002-10-27 01:00 US/Eastern comes both in DST and STD.
But 2002-10-27 01:00 US/Eastern STD minus two days is 2002-10-25 01:00
US/Eastern DST
However, 2002-10-25 01:00 US/Eastern DST plus two days is 2002-10-27
01:00 US/Eastern, but it is ambiguous if you want DST or not DST.
And you can't pass in a is_dst flag to __add__, so the arithmatic must
just pick one, and the sensible one is to keep to the same DST.
import pytz
from datetime import datetime, timedelta
tz = pytz.timezone('US/Eastern')
dt = tz.localize(datetime(2002, 10, 27, 1, 0), is_dst=False)
dt2 = tz.normalize(dt - timedelta(days=2) + timedelta(days=2))
dt == dt2
True
tz.normalize(dt - timedelta(days=2))
datetime.datetime(2002, 10, 25, 2, 0, tzinfo=DstTzInfo 'US/Eastern'
EDT-1 day, 20:00:00 DST)
tz.normalize(tz.normalize(dt - timedelta(days=2)) + timedelta(days=2))
datetime.datetime(2002, 10, 27, 1, 0, tzinfo=DstTzInfo 'US/Eastern'
EST-1 day, 19:00:00 STD)
2002-10-27 01:00 US/Eastern is_dst=0 is after the DST transition
(EST). Subtracting 48 hours from it crosses the DST boundary and
should give you 2002-10-27 02:00 US/Eastern is_dst=1, prior to the DST
transition (EDT). Adding 48 hours again goes past 2002-10-27 01:00
EDT, crosses the DST boundary, and gives you back 2002-10-27 01:00
EST.
--
Stuart Bishop stu...@stuartbishop.net
http://www.stuartbishop.net/
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status on PEP-431 Timezones
OK, so I realized another thing today, and that is that arithmetic
doesn't necessarily round trip.
For example, 2002-10-27 01:00 US/Eastern comes both in DST and STD.
But 2002-10-27 01:00 US/Eastern STD minus two days is 2002-10-25 01:00
US/Eastern DST
However, 2002-10-25 01:00 US/Eastern DST plus two days is 2002-10-27
01:00 US/Eastern, but it is ambiguous if you want DST or not DST.
And you can't pass in a is_dst flag to __add__, so the arithmatic must
just pick one, and the sensible one is to keep to the same DST.
That means that:
tz = get_timezone('US/Eastern')
dt = datetime(2002, 10, 27, 1, 0, tz=tz, is_dst=False)
dt2 = dt - 420 + 420
assert dt == dt2
Will fail, which will be unexpected for most people.
I think there is no way around this, but I thought I should flag for
it. This is a good reason to do all your date time arithmetic in UTC.
//Lennart
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status on PEP-431 Timezones
On Tue, Apr 14, 2015 at 4:52 PM, Alexander Belopolsky alexander.belopol...@gmail.com wrote: For those who were not at the conference, can someone summarize the post-PyCon status of this PEP? Is Barry still the BDFL-Delegate? Is there an updated draft? Should this discussion move to python-ideas? There is no status change except that it will need updates, but I'm waiting with that until I know what updates are needed, and that means an implementation is needed first. //Lennart ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status on PEP-431 Timezones
The Open Space was good, and the conclusion was that solution #2 indeed seems to be the right one. We also concluded that likely the datetime() constructor itself needs to grow an is_dst flag. There was no further insight into whether having an offset or a is_dst flag as an attribute, I think that will clear up during the implementation. //Lennart ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status on PEP-431 Timezones
I've booked an open space for 3pm in 512CG. Be there or be to naive! ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status on PEP-431 Timezones
There is plenty of time to book an open space now. I would prefer sometime Saturday. Any wishes? ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status on PEP-431 Timezones
On Wed, Apr 8, 2015 at 9:57 PM, Isaac Schwabacher ischwabac...@wisc.edu wrote: On 15-04-08, Lennart Regebro wrote: === Stdlib option 2: A datetime _is_dst flag === By having a flag on the datetime instance that says this is in DST or not the timezone implementation can be kept simpler. Storing the offset itself instead of a flag makes things conceptually cleaner. The concept would be pretty much the same, but yes, you would be able to save lookups in the dst() call, so that's a benefit. I was planning on storing the dst() offset at least lazily, but I'm not sure that means we can get rid of the flag. As an added bonus, you get a representation that's still meaningful when time changes happen for political rather than DST reasons. That's a good point. Although you would still have to use the is_dst flag to choose in that case, even if neither or both is actually DST. Unless we come up with another name for that flag, which I don't think is important. //Lennart ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status on PEP-431 Timezones
On Thu, Apr 9, 2015 at 1:31 AM, Alexander Belopolsky alexander.belopol...@gmail.com wrote: Storing isdst in the datetime object would allow utcoffset(dt) to distinguish between 1:30AM before clock change and 1:30AM after. Where do you propose to store the offset? If you store it in dt, why would you need the tzinfo? Because otherwise you don't know what tzinfo the datetime uses, and you need to know that if you add or subtract a timedelta, or compare with another datetime. Regardless, whatever the proposal to add timezones to stdlib will end up being, it must include the ability to implement an equivalent of UNIX date command shown above. That's a strftime() issue, I think that issue is already solved. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status on PEP-431 Timezones
On Wed, Apr 8, 2015 at 7:37 PM, Carl Meyer c...@oddbird.net wrote: Hi Lennart, On 04/08/2015 09:18 AM, Lennart Regebro wrote: I wrote PEP-431 two years ago, and never got around to implement it. This year I got some renewed motivation after Berker Peksağ made an effort of implementing it. I'm planning to work more on this during the PyCon sprints, and also have a BoF session or similar during the conference. Anyone interested in a session on this, mail me and we'll set up a time and place! I'm interested in the topic, and would probably attend a BoF at PyCon. Cool! So is adopting pytz's expanded API into the stdlib really a big problem? Maybe, maybe not. But that API is also needlessly complicated, precisely because it's working around the limitations of datetime.tzinfo. In the PEP I remove those limitations but keep the simpler API. With a solution based on how pytz does it, I don't think that's possible. Is this really adequate? pytz's implementation handles far more than is DST or not, it also correctly handles historical timezone changes. How would those be handled under this proposal? Those would still be handled. The flag is only to flag if it's DST or not in a timestamp that is otherwise ambiguous. //Lennart ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Status on PEP-431 Timezones
Hi! I wrote PEP-431 two years ago, and never got around to implement it. This year I got some renewed motivation after Berker Peksağ made an effort of implementing it. I'm planning to work more on this during the PyCon sprints, and also have a BoF session or similar during the conference. Anyone interested in a session on this, mail me and we'll set up a time and place! //Lennart -- If anyone is interested in the details of the problem, this is it. The big problem is the ambiguous times, like 02:30 a time when you move the clock back one hour, as there are two different 02:30's that day. I wrote down my experiences with looking into and trying to implement several different solutions. And the problem there is actually how to tell the datetime if it is before or after the changeover. == How others have solved it == === dateutil.tz: Ignore the problem === dateutil.tz simply ignores the problems with ambiguous datetimes, keeping them ambiguous. === pytz: One timezone instance per changeover === Pytz implements ambiguous datetimes by having one class per timezone. Each change in the UTC offset changes, either because of a DST changeover, or because the timezone changes, is represented as one instance of the class. All instances are held in a list which is a class attribute of the timezone class. You flag in which DST changeover you are by uising different instances as the datetimes tzinfo. Since the timezone this way knows if it is DST or not, the datetime as a whole knows if it's DST or not. Benefits: - Only known possible implementation without modifying stdlib, which of course was a requirement, as pytz is a third-party library. - DST offset can be quickly returned, as it does not need to be calculated. Drawbacks: - A complex and highly magical implementation of timezones that is hard to understand. - Required new normalize()/localize() functions on the timezone, and hence the API is not stdlib's API. - Hundreds of instances per timezone means slightly more memory usage. == Options for PEP 431 == === Stdlib option 0: Ignore it === I don't think this is an option, really. Listed for completness. === Stdlib option 1: One timezone instance per changeover === Option 1 is to do it like pytz, have one timezone instance per changeover. However, this is likely not possible to do without fundamentally changing the datetime API, or making it very hard to use. For example, when creating a datetime instance and passing in a tzinfo today this tzinfo is just attached to the datetime. But when having multiple instances of tzinfos this means you have to select the correct one to pass in. pytz solves this with the .localize() method, which let's the timezone class choose which instance to pass in. We can't pass in the timezone class into datetime(), because that would require datetime.__new__ to create new datetimes as a part of the timezone arithmetic. These in turn, would create new datetimes in __new__ as a part of the timezone arithmetic, which in turn, yeah, you get it... I haven't been able to solve that issue without either changing the API/usage, or getting infinite recursions. Benefits: - Proven soloution through pytz. - Fast dst() call. Drawbacks: - Trying to use this technique with the current API tends to create infinite recursions. It seems to require big API changes. - Slow datetime() instance creation. === Stdlib option 2: A datetime _is_dst flag === By having a flag on the datetime instance that says this is in DST or not the timezone implementation can be kept simpler. You also have to either calculate if the datetime is in a DST or not either when creating it, which demands datetime object creations, and causes infinite recursions, or you have to calculate it when needed, which means you can get Ambiguous date time errors at unexpected times later. Also, when trying to implement this, I get bogged down in the complexities of how tzinfo and datetime is calling each other back and forth, and when to pass in the current is_dst and when to pass in the the desired is_dst, etc. The API and current implementation is not designed with this case in mind, and it gets very tricky. Benefits: - Simpler tzinfo() implementations. Drawbacks: - It seems likely that we must change some API's. - This in turn may affect the pytz implementation. Or not, hard to say. - The DST offset must use slow timezone calculations. However, since datetimes are immutable it can be a cached, lazy, one-time operation. === Stdlib option 3: UTC internal representation === Having UTC as the internal representation makes the whole issue go away. Datetimes are no longer ambiguous, except when creating, so checks need to be done during creation, but that should be possible without datetime creation in this case, resolving the infinite recursion problem. Benefits: - Problem solved. - Minimal API changes. Drawbacks: - Backwards compatibility with pickles. - Possible other backwards
Re: [Python-Dev] python 3 niggle: None 1 raises TypeError
On Tue, Feb 18, 2014 at 5:10 PM, Mark Lawrence breamore...@yahoo.co.uk wrote: Sorry if this has already been suggested, but why not introduce a new singleton to make the database people happier if not happy? To avoid confusion call it dbnull? A reasonable compromise or complete cobblers? :) I think this is possible already, for the database people. The problem is that it will not pass the is None test, which at the very least is not backwards compatible with how they have used it before. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] python 3 niggle: None 1 raises TypeError
On Tue, Feb 18, 2014 at 8:35 AM, Greg Ewing greg.ew...@canterbury.ac.nz wrote:
If you don't want to touch comparison in general,
how about an option to sort()?
results = sorted(invoices, key=attrgetter('duedate'), none='first')
I think this is a much better option.
//Lennart
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] python 3 niggle: None 1 raises TypeError
On Fri, Feb 14, 2014 at 9:04 AM, Chris Withers ch...@simplistix.co.uk wrote:
Hi All,
Sending this to python-dev as I'm wondering if this was considered when the
choice to have objects of different types raise a TypeError when ordered...
So, the concrete I case I have is implementing stable ordering for the
python Range objects that psycopg2 uses. These have 3 attributes that can
either be None or, for sake of argument, a numeric value.
To implement __lt__ in Python 2, I could do:
def __lt__(self, other):
if not isinstance(other, Range):
return True
return ((self._lower, self._upper, self._bounds)
(other._lower, other._upper, other._bounds))
Because None 1 raises a TypeError, in Python 3 I have to do:
def __lt__(self, other):
if not isinstance(other, Range):
return NotImplemented
for attr in '_lower', '_upper', '_bounds':
self_value = getattr(self, attr)
other_value = getattr(other, attr)
if self_value == other_value:
pass
elif self_value is None:
return True
elif other_value is None:
return False
else:
return self_value other_value
return False
Am I missing something? How can I get this method down to a sane size?
It was considered. It's not obvious where you want None to appear in
your ordering, so you will have to implement this by yourself. I can't
come up with anything obviously shorter.
Or, well, this, but it's not exactly pretty:
def __lt__(self, other):
if not isinstance(other, Range):
return NotImplemented
ninf = float('-inf') # So None is lower than anything else.
return ((self._lower if self._lower is not None else ninf,
self._upper if self._upper is not None else ninf,
self._bounds if self._bounds is not None else ninf) (
other._lower if other._lower is not None else ninf,
other._upper if other._upper is not None else ninf,
other._bounds if other._bounds is not None else ninf))
Or maybe:
def __lt__(self, other):
if not isinstance(other, Range):
return NotImplemented
s = (self._lower, self._upper, self._bounds)
if None is s:
return True
o = (other._lower, other._upper, other._bounds)
if None in o:
return False
return s o
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] python 3 niggle: None 1 raises TypeError
What I think is the biggest nitpick here is that you can't make a NULL object so that NULL is None evaluates as True. If you could, you could then easily make this NULL object evaluate as less than everything except itself and None, and use that consistently. But since this NULL is not None, that breaks any backwards compatibility if your SQL library mapped NULL to None in earlier versions. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 460: allowing %d and %f and mojibake
On Sat, Jan 11, 2014 at 8:40 PM, Kristján Valur Jónsson
krist...@ccpgames.com wrote:
Hi there.
How about a compromise?
Personally, I think adding the full complement of integer/float formatting to
bytes is a bit over the top.
How about just supporting two format specifiers?
%b : interpolate a bytes object. If it doesn't have the buffer interface,
error.
%s : interpolate a str object, encoded to ASCII using 'strict' conversion.
This should cover the most common use cases.
In particular, you could do this:
Headers.append('Content-Length: %s'%(len(data),))
And then subsequently:
Packet = b'%b%b'%(bjoin(headers), data)
For more complex formatting, you delegate to the more capable string class,
but benefit from automatic ASCII conversion:
Data = bpercentage = %s % (%4.2f % (value,))
Although nice and clean as principle, I think it makes for somewhat
messy code. I'm in favor of having float and integer specifiers as
well.
I'm also for including %s, because it makes moving from Python 2
easier. But it should definitely error out if you try to feed it a
non-ascii string.
//Lennart
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python3 complexity
On Thu, Jan 9, 2014 at 8:16 AM, Ben Finney ben+pyt...@benfinney.id.au wrote: Nick Coghlan ncogh...@gmail.com writes: Set the mode to rb, process it as binary. Done. Which entails abandoning the stated goal of “just want to parse text files” :-) Only if your definition of text files means it's unicode. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python3 complexity
On Thu, Jan 9, 2014 at 10:06 AM, Kristján Valur Jónsson krist...@ccpgames.com wrote: Do I speak Chinese to my grocer because china is a growing force in the world? Or start every discussion with my children with a negotiation on what language to use? No, because your environment have a default language. And Python has a default encoding. You only get problems when some file doesn't use the default encoding. //Lennart ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python3 complexity
On Fri, Jan 10, 2014 at 2:03 AM, Joao S. O. Bueno jsbu...@python.org.br wrote: On 9 January 2014 04:50, Lennart Regebro rege...@gmail.com wrote: To be honest, you can define text as A stream of bytes that are split up in lines separated by a linefeed, and do some basic text processing like that. Just very *basic*, but still. Replacing characters. Extracting certain lines etc. That is, until you hit a character which has a byte with the same value of ASCII newline in the middle of a multi-byte character. So, this approach is broken to start with. For a very specific definition of broken, yes, namely that it will fail with UTF-16 or EBCDIC. Files that with the above definition of text files are not text files. :-) //Lennart ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python3 complexity
On Thu, Jan 9, 2014 at 1:07 AM, Ben Finney ben+pyt...@benfinney.id.au wrote: Kristján Valur Jónsson krist...@ccpgames.com writes: Believe it or not, sometimes you really don't care about encodings. Sometimes you just want to parse text files. Files don't contain text, they contain bytes. Bytes only become text when filtered through the correct encoding. To be honest, you can define text as A stream of bytes that are split up in lines separated by a linefeed, and do some basic text processing like that. Just very *basic*, but still. Replacing characters. Extracting certain lines etc. This is harder in Python 3, as bytes does not have all the functionality strings has, like formatting. This can probably be fixed in Python 3.5, if the relevant PEP gets finished. For the battery analogy, that's like saying: I want a battery. What kind? It doesn't matter, as long as it's over 5V. //Lennart ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] 2.x vs 3.x survey results
On Sun, Jan 5, 2014 at 5:20 AM, John Yeuk Hon Wong gokoproj...@gmail.com wrote: I think it helps Luca and many others (including myself) if there is a reference of the difference between 2.7 and Python 3.3+. Not specifically for 2.7 and 3.3, no. This is a fairly complete list: http://python3porting.com/differences.html There are PEPs and books, but is there any such long list of references? If not, should we start investing in one? I know the basic one such as xrange and range, items vs iteritems, izip vs zip that sort of uniform syntax/library inclusion difference. If there is such reference available? I'm honestly despairing that people still don't know that there is a free book on the topic. I have no idea how to increase the knowledge on this point. //Lennart ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Fwd: Python 2.x and 3.x usage survey
Yeah, but I'd still expect more people on Python-dev to at least have used Python 3 and possibly be maintainers of libraries that have been ported to Python 3 etc. On Tue, Dec 31, 2013 at 11:04 AM, Steven D'Aprano st...@pearwood.info wrote: On Tue, Dec 31, 2013 at 08:16:33AM +0100, Lennart Regebro wrote: On Tue, Dec 31, 2013 at 6:31 AM, Dan Stromberg drsali...@gmail.com wrote: So far the results are looking good for 3.x. Python-dev probably is a bit special. Why? Most Python-Dev people have day jobs, and the version of Python that they use in their day job is subject to exactly the same outside pressures as everyone else (RedHat ships with Python 2.6, so that's the version we're using). -- Steven ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/regebro%40gmail.com ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Fwd: Python 2.x and 3.x usage survey
On Tue, Dec 31, 2013 at 6:31 AM, Dan Stromberg drsali...@gmail.com wrote: So far the results are looking good for 3.x. Python-dev probably is a bit special. //Lennart ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] PEP-431 non-status
Hi all! If you wonder about the lack of progress reports on pep-431, this is because of a lack of progress. I simply haven't had any time to work on this. I considered make a kickstarter so I could take time off from working, but to be honest, even with that, I still have no time to realistically get this in before beta 1. Sorry about that! //Lennart ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP-431 non-status
On Wed, Oct 2, 2013 at 3:05 PM, Dirkjan Ochtman dirk...@ochtman.nl wrote: On Wed, Oct 2, 2013 at 2:17 PM, Lennart Regebro rege...@gmail.com wrote: If you wonder about the lack of progress reports on pep-431, this is because of a lack of progress. I simply haven't had any time to work on this. I considered make a kickstarter so I could take time off from working, but to be honest, even with that, I still have no time to realistically get this in before beta 1. Is there a list of stuff that needs to be done? I might be able to help out. Yes. 1. Implement the PEP. :-) I looked at it during the PyCon sprint, and I had hoped to be able to mostly move pytz into datetime, but unfortunately it wasn't that easy, because of the way datetime normalization works. The pytz normalization calls methods that would need to call the normalization... :-) //Lennart ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP-431 non-status
That could be a possibility as well. On Wed, Oct 2, 2013 at 6:14 PM, Stuart Bishop stu...@stuartbishop.net wrote: On Wed, Oct 2, 2013 at 9:42 PM, Lennart Regebro rege...@gmail.com wrote: On Wed, Oct 2, 2013 at 3:05 PM, Dirkjan Ochtman dirk...@ochtman.nl wrote: On Wed, Oct 2, 2013 at 2:17 PM, Lennart Regebro rege...@gmail.com wrote: If you wonder about the lack of progress reports on pep-431, this is because of a lack of progress. I simply haven't had any time to work on this. I considered make a kickstarter so I could take time off from working, but to be honest, even with that, I still have no time to realistically get this in before beta 1. Is there a list of stuff that needs to be done? I might be able to help out. Yes. 1. Implement the PEP. :-) I looked at it during the PyCon sprint, and I had hoped to be able to mostly move pytz into datetime, but unfortunately it wasn't that easy, because of the way datetime normalization works. The pytz normalization calls methods that would need to call the normalization... :-) I think the first thing that needs to happen is get the is_dst flag in. Then we can turn pytz into a new, sane API without all that awful normalization stuff :) It doesn't all have to land in the same release. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Use an empty def as a lambda
On Thu, Sep 19, 2013 at 10:54 PM, Ben Gift benhg...@gmail.com wrote: I think the lambda keyword is difficult to understand for many people. It would be more pythonic to use an empty def call instead. I agree, but that ship has sailed, at least until the time when Python 2 is dead. I don't want these kinds of syntax changes for a *long* time now, personally. //Lennart ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python 3 as a Default in Linux Distros
On Thu, Jul 25, 2013 at 12:41 PM, Laurent Gautier lgaut...@gmail.com wrote: - a user is running a python script (he expects to be working), and is using the default /usr/bin/python (formerly Python 2, now Python 3). If the program fails because of obvious Python 2-only idioms, reporting this rather that the SyntaxError is much less confusing. 2to3 will not report this. It will get rid of the SyntaxError (which at least is fairly clear) and give you a completely different and even more obscure error. You have replaced a somewhat unclear error with a very unclear error. If we want to report a problem, then /usr/bin/python should just report that you should use /usr/bin/python2 or /usr/bin/python3. That's clear. //Lennart ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python 3 as a Default in Linux Distros
On Thu, Jul 25, 2013 at 1:53 PM, Laurent Gautier lgaut...@gmail.com wrote: My meaning was the use of 2to3's machinery (and fire a message if a translation occurs) not 2to3 itself, obviously. I don't understand what you mean is the difference. //Lennart ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python 3 as a Default in Linux Distros
On Wed, Jul 24, 2013 at 11:12 AM, Bohuslav Kabrda bkab...@redhat.com wrote: - Should we point /usr/bin/python to Python 3 when we make the move? No. - What should user get after using yum install python? Will a base install include Python 3? If it does, I think yum install python should mean python3, and hence already be installed. //Lennart ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python 3 as a Default in Linux Distros
On Wed, Jul 24, 2013 at 3:54 PM, Lennart Regebro rege...@gmail.com wrote: On Wed, Jul 24, 2013 at 11:12 AM, Bohuslav Kabrda bkab...@redhat.com wrote: - Should we point /usr/bin/python to Python 3 when we make the move? No. To be more explicit. I think it's perfectly fine to not provide a /usr/bin/python at all, but I think pointing it to Python 3 will provide unhelpful error messages. Also having a /usr/bin/python that does nothing but say You should use either /usr/bin/python2 or /usr/bin/python3 would work. But pointing it to Python 3 will make for unhelpful and confusing error messages for anyone having Python scripts using /usr/bin/python, which is the majority of people having python scripts at all. //Lennart ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python 3 as a Default in Linux Distros
On Wed, Jul 24, 2013 at 7:22 PM, Laurent Gautier lgaut...@gmail.com wrote: The wrapper in /usr/bin/python: - could use what is in 2to3. I think that most of the cases are solved there. Only the most trivial cases are solved completely by 2to3, and the error messages you would get would be hard to understand, and running the scripts would be very slow. //Lennart ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Tweaking PEP 8 guidelines for use of leading underscores
On Mon, Jul 15, 2013 at 6:17 AM, Nick Coghlan ncogh...@gmail.com wrote: = Private interfaces Unless explicitly documented otherwise, a leading underscore on any name indicates that it is an internal implementation detail and any backwards compatibility guarantees do not apply. It is strongly encouraged that private APIs (whether modules, classes, functions, attributes or other names) be clearly marked in this way, as Python users may rely on introspection to identify available functionality and may be misled into believing an API without a leading underscore is in fact a public API with the standard backwards compatibility guarantees. Very good. While the explicit use of a leading underscore in the names of private modules is preferred, all test modules and all modules that are not explicitly covered in the documentation are also considered private interfaces, even when their names do not start with a leading underscore and even if they include a module level documentation string. This includes submodules of packages that are documented as if they were a single module. But wait, aren't this about how to use other peoples modules, more than a style guide of how to write your own modules? I don't think this belongs in PEP 8 at all. //Lennart ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python 3.4 and Windows XP: just 45 days until EOL
On Fri, Jul 12, 2013 at 2:11 AM, Steve Dower steve.do...@microsoft.com wrote: +1. And maybe amend PEP 11 to specify whose extended support phase does not expire within 6 months of release? (I picked 6 for no particular reason.) Why have the specification in PEP 11 if we feel we can change the rules arbitrarily when we feel like it? //Lennart ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Purpose of Doctests [Was: Best practices for Enum]
On Mon, May 20, 2013 at 1:51 AM, Gregory P. Smith g...@krypto.org wrote: On May 19, 2013 4:31 PM, Benjamin Peterson benja...@python.org wrote: 2013/5/19 Gregory P. Smith g...@krypto.org: Idea: I don't believe anybody has written a fixer for lib2to3 that applies fixers to doctests. That'd be an interesting project for someone. 2to3 can operate on doctests, though it doesn't do anything different to them than it does to normal sourcecode. Oh cool. I didn't realize that already existed! It won't change any output, though, which still means that they tend to break. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] NoneType(None) raises exception
On Fri, Apr 26, 2013 at 1:09 AM, Ethan Furman et...@stoneleaf.us wrote: We just fixed NoneType() to return None instead of raising an exception. Another use-case for calling NoneType is working with ORMs: result = [] for field in row: type = get_type(field) # returns int, bool, str, NoneType, ... result.append(type(field)) Having it return NoneType there seems a strange thing to do, as have a None field makes no sense. Why would you have a column that can only contain the value None? What is returning NoneType at that point supposed to signify? If it's supposed to signify that the value is NULL, I think the above code is a very strange way of handling that. get_type(field) should reasonably return the type of the column, not the type of the value you got, as that would be just type(field) and doing type(field)(field) is a bit pointless. :-) //Lennart ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Why can't I encode/decode base64 without importing a module?
On Thu, Apr 25, 2013 at 7:43 AM, Antoine Pitrou solip...@pitrou.net wrote: On Thu, 25 Apr 2013 04:19:36 +0200 Lennart Regebro rege...@gmail.com wrote: On Thu, Apr 25, 2013 at 3:54 AM, Stephen J. Turnbull step...@xemacs.org wrote: RFC 4648 repeatedly refers to *characters*, without specifying an encoding for them. [...] Base64 is an encoding that transforms between 8-bit streams. No, it isn't. What Stephen wrote above. Yes it is. Base64 takes 8-bit bytes and transforms them into another 8-bit stream that can be safely transmitted over various channels that would mangle an unencoded 8-bit stream, such as email etc. http://en.wikipedia.org/wiki/Base64 Either you get a LookupError: unknown encoding: base64, which is what you get now, or you get an UnicodeEncodingError if the text is not ASCII. We don't want the latter, because it means that code that looks fine for the developer breaks in real life because the developer was American That's bogus. No, that's real life. By the same argument, we should suppress any encoding which isn't able to represent all possible unicode strings. No, if you explicitly use such an encoding it is because you need to because you are transferring data to a system that needs the encoding in question. Unicode errors are unavoidable at that point, not an unexpected surprise because a conversion happened implicitly that you didn't know about. //Lennart ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Why can't I encode/decode base64 without importing a module?
On Thu, Apr 25, 2013 at 8:57 AM, Stephen J. Turnbull step...@xemacs.org wrote: I think that would be an unfortunate result. These operations on streams are theoretically nicely composable. It would be nice if practice reflected that by having a uniform API for all of these operations (charset translation, encoded text to internal, content transfer encoding, compression ...). I think it would be useful, too, though I can't prove that. But the translation to and from Unicode to some 8-bit encoding is different from the others. It makes sense that they have a different API. If you have a Unicode string you can go: Unicode text - UTF8 - ZIP - BASE64. Or you can go Unicode text - UTF8 - BASE64 - ZIP Although admittedly that makes much less sense. :-) But you can not go: Unicode text - BASE64 - ZIP - UTF8 The str/bytes encoding/decoding is not like the others. //Lennart ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Why can't I encode/decode base64 without importing a module?
On Thu, Apr 25, 2013 at 11:25 AM, Antoine Pitrou solip...@pitrou.net wrote: Le Thu, 25 Apr 2013 08:38:12 +0200, Yes it is. Base64 takes 8-bit bytes and transforms them into another 8-bit stream that can be safely transmitted over various channels that would mangle an unencoded 8-bit stream, such as email etc. http://en.wikipedia.org/wiki/Base64 I don't see anything in that Wikipedia page that validates your opinion. OK, quote me the exact page text from the Wikipedia article or RFC that explains how you map the 31-bit character space of Unicode to Base64. The Wikipedia page does talk about *text* and *characters* for the result of base64 encoding. So are saying that you want the Python implementation of base64 encoding to take 8-bit binary data in bytes format and return a Unicode string containing the Base64 encoded data? I think that would surprise most people, and be of significantly less use than a base64 encoding that returns bytes. Python 3 still views text as Unicode only. Everything else is not text, but binary data. This makes sense, is consistent and makes things easier to handle. This is the whole point of making the str into Unicode in Python 3. No, if you explicitly use such an encoding it is because you need to because you are transferring data to a system that needs the encoding in question. Unicode errors are unavoidable at that point, not an unexpected surprise because a conversion happened implicitly that you didn't know about. I don't know what implicit conversion you are talking about. There's no implicit conversion in a scheme where the result of base64 encoding is a text string. I'm sorry, I thought you were arguing for a base64 encoding taking Unicode strings and returning 8-bit bytes. That position I can understand, although I disagree with it. The position that a base64 encoding should take 8-bit bytes and return Unicode strings is incomprehensible to me. I have no idea why you would want that, how you would use it, how you would implement that API in a reasonable way, nor how you would explain why it is like that. I can't think of any usecase where you would want base64 encoded data unless you intend to transmit it over an 8-bit channel, so why it should return a Unicode string instead of 8-bit bytes is completely beyond my comprehension. Sorry. //Lennart ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Why can't I encode/decode base64 without importing a module?
On Thu, Apr 25, 2013 at 2:57 PM, Antoine Pitrou solip...@pitrou.net wrote: I can think of many usecases where I want to *embed* base64-encoded data in a larger text *before* encoding that text and transmitting it over a 8-bit channel. That still doesn't mean that this should be the default behavior. Just because you *can* represent base64 as Unicode text doesn't mean that it should be. (GPG signatures, binary data embedded in JSON objects, etc.) Is the GPG signature calculated on the *Unicode* data? How is that done? Isn't it done on the encoded message? As I understand it a GPG signature is done on any sort of document. Either me or you have completely misunderstood how GPG works, I think. :-) In the case of JSON objects, they are intended for data exchange, and hence in the end need to be byte strings. So if you have a byte string you want to base64 encode before transmitting it with json, you would just end up transforming it to a unicode string and then back. That doesn't seem useful. One use case where you clearly *do* want the base64 encoded data to be unicode strings is because you want to embed it in a text discussing base64 strings, for a blog or a book or something. That doesn't seem to be a very common usecase. For the most part you base64 encode things because it's going to be transmitted, and hence the natural result of a base64 encoding should be data that is ready to be transmitted, hence byte strings, and not Unicode strings. Python 3 doesn't *view* text as unicode, it *represents* it as unicode. I don't agree that there is a significant difference between those wordings in this context. The end result is the same: Things intended to be handled/seen as textual should be unicode strings, things intended for data exchange should be byte strings. Something that is base64 encoded is primarily intended for data exchange. A base64 encoding should therefore return byte strings, especially since most API's that perform this transmission will take byte strings as input. If you want to include this in textual data, for whatever reason, like printing it in a book, then the conversion is trivial, but that is clearly the less common use case, and should therefore not be the default behavior. //Lennart ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Why can't I encode/decode base64 without importing a module?
On Thu, Apr 25, 2013 at 4:22 PM, MRAB pyt...@mrabarnett.plus.com wrote: The JSON specification says that it's text. Its string literals can contain Unicode codepoints. It needs to be encoded to bytes for transmission and storage, but JSON itself is not a bytestring format. OK, fair enough. base64 is a way of encoding binary data as text. It's a way of encoding binary data using ASCII. There is a subtle but important difference. In Python 3 we're trying to stop mixing binary data (bytestrings) with text (Unicode strings). Yup. And that's why a byte64 encoding shouldn't return Unicode strings. //Lennart ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Why can't I encode/decode base64 without importing a module?
On Thu, Apr 25, 2013 at 5:27 PM, Antoine Pitrou solip...@pitrou.net wrote: Le Thu, 25 Apr 2013 15:34:45 +0200, Lennart Regebro rege...@gmail.com a écrit : I don't agree that there is a significant difference between those wordings in this context. The end result is the same: Things intended to be handled/seen as textual should be unicode strings, things intended for data exchange should be byte strings. I don't think this distinction is meaningful at all. OK, then I think we have found the core of the problem, and the end of the discussion (from my side, that is). In the end, everything is a byte string on a classical computer (including unicode strings displayed on your monitor, obviously). Yes of course. Especially since my monitor is an output device. ;-) If you think the technicalities of an operation should never be hidden or abstracted away, then you're better off with C than Python ;-) The whole point is that Python *does* abstract it away. It abstract the internals of Unicode strings in such a way that they are no longer, conceptually, 8-bit data. This *is* a distinction Python does, and it is a useful distinction. I do not see any reason to remove it. http://regebro.wordpress.com/2011/03/23/unconfusing-unicode-what-is-unicode/ //Lennart ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Why can't I encode/decode base64 without importing a module?
On Thu, Apr 25, 2013 at 3:54 AM, Stephen J. Turnbull step...@xemacs.org wrote:
RFC 4648 repeatedly refers to *characters*, without specifying an
encoding for them. In fact, if you copy accurately, you can write
BASE64 on a napkin and that napkin will accurate transmit the data
(assuming it doesn't run into sleet or gloom of night).
Or Mrs Cake.
What else is that but text in the sense of Py3k?
Text in the sense of Py3k is Unicode. That a 8-bit character stream
(or in this case 6-bit) fits in the 31 bit character space of Unicode
doesn't make it Unicode, and hence not text. (Napkins of course have
even higher bit density than 31 bits per character, unless you write
very small). From the viewpoint of Py3k, bytes data is not text.
This is a very useful way to deal with Unicode. See also
http://regebro.wordpress.com/2011/03/23/unconfusing-unicode-what-is-unicode/
My point is not that Python's base64 codec *should* be bytes-to-str
and back.
Base64 does not convert between a Unicode character stream and an
8-bite byte stream. It converts between a 8-bit byte-stream and an
8-bit byte stream. It therefore should be bytes to bytes. To fit
Unicode text into Base64 you have to first use an encoding on that
Unicode text to convert it to bytes.
What I'm groping toward is an idea of a variable method, so that we
could use .encode and .decode where they are TOOWTDI for people even
though a purely formal interpretation of duck-typing would say but
why is that blue whale quacking, waddling, and flying? In other
words (although I have no idea how best to implement it), I would like
somestring.encode('base64') to fail with I don't know how to do
that (an attribute lookup error?), the same way that
somebytes.encode('utf-8') does in Python 3 today.
There's only two options there. Either you get a LookupError: unknown
encoding: base64, which is what you get now, or you get an
UnicodeEncodingError if the text is not ASCII. We don't want the
latter, because it means that code that looks fine for the developer
breaks in real life because the developer was American and didn't
think of this, but his client happens to have an accent in the name.
Base64 is an encoding that transforms between 8-bit streams. Let it be
that. Don't try to shoehorn it into a completely different kind of
encoding.
//Lennart
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Why can't I encode/decode base64 without importing a module?
On Tue, Apr 23, 2013 at 4:04 AM, Steven D'Aprano st...@pearwood.info wrote: As others have pointed out in the past, repeatedly, the codec system is completely general and can transform bytes-bytes and text-text just as easily as bytes-text. Yes, but the encode()/decode() methods are not, and the fact that you now know what goes in and what comes out means that people get much fewer Decode/EncodeErrors. Which is a good thing. //Lennart ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 435 -- Adding an Enum type to the Python standard library
OK, so I finally got tie to read the PEP. I like it, I really have missed Enums, this is awesome. That's all folks! //Lennart ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] The end of 2.7
On Sun, Apr 7, 2013 at 7:11 AM, mar...@v.loewis.de wrote: Wrt. to the 3.x migration rate: I think this is a self-fulfilling prophecy. Migration rate will certainly increase once we announce an end of 2.7, and then again when the end is actually reached. Well... People are in general *stuck* on Python 2. They are not staying because they want to. So I'm not so sure migration rate will increase because an end is announced or reached. //Lennart ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] The end of 2.7
On Sun, Apr 7, 2013 at 9:51 AM, mar...@v.loewis.de wrote: Quoting Lennart Regebro rege...@gmail.com: Well... People are in general *stuck* on Python 2. They are not staying because they want to. So I'm not so sure migration rate will increase because an end is announced or reached. I assume you say that because people rely on libraries that haven't been ported (correct me if there are other reasons to be stuck). Company policies that mean you are using old distros with no support for Python 3 and not being allowed to install it from source is also a reason, but yes, the main one being libraries/frameworks, yes. With an announced end-of-life, I'm certain that migration rate will increase, because people will now urge their suppliers, pointing to the announcement. The suppliers are often people who are maintaining an open source library of some sort. When I see questions on stackoverflow about support for X on Python 3 I sometimes take a quick look of the state of libraries, check out their mailing list etc. It's *always* a problem of that the maintainers themselves are stuck on Python 2.7 or earlier together with porting being problematic. I think Python 3.3 with the u'' literal is much more important for increased adoption there than the end of life of 2.7 as it often makes porting much easier. But even so sometimes API's needs to be changed, etc, so it takes a big concerted effort of both the maintainers, and the few people that are interested in porting it to Python 3. And when you get one new person asking for Python 3 support every 6 months, that's just not enough people. That's the hangup IMO. Ending Python 2.7 will make no difference there either good or bad, I think. We need to find other ways of improving adoption. As for the company policies, in theory it sounds like a good argument that ending Python 2.7 would be incentive for these to change. But these are often slow moving companies that are happy using outdated software, and are clearly using it already, or they would be on distros that *did* support Python 3 already. :-) Nobody can *really* be stuck on a not-ported dependency, as they could always port it themselves, and even fork if the developer refuses to integrate the port (and you know that this actually happens). Yes, but time/money/knowledge is at a premium. Also self-confidence. A lot of people probably think porting is much harder than it is. :-) //Lennart ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] The end of 2.7
On Sun, Apr 7, 2013 at 10:10 AM, Lennart Regebro rege...@gmail.com wrote: That's the hangup IMO. Ending Python 2.7 will make no difference there either good or bad, I think. We need to find other ways of improving adoption. And to be clear: I am therefore not arguing *not* to end it. I just don't think that doing so will increase Python 3 adoption. I think that's a red herring. I have little opinion on whether to announce an official end or not, nor when. I think the burden of maintaining many branches is a much better argument, and that it therefore probably should be decided by the maintainers. //Lennart ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [RELEASE] Python 2.7.4 release candidate 1
On Tue, Mar 26, 2013 at 11:54 AM, Matthias Klose d...@ubuntu.com wrote: Am 25.03.2013 01:30, schrieb Benjamin Peterson: 2.7.4 will be the latest maintenance release in the Python 2.7 series. I hope it's not (and in the IDLE thread you say so otherwise too). It most certainly will be the latest once it's released. But hopefully not the last. :-) //Lennart ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] built-in Python test runner (was: Python Language Summit at PyCon: Agenda)
On Tue, Mar 5, 2013 at 8:11 AM, Donald Stufft donald.stu...@gmail.com wrote: I don't care much what that mechanism is, but I think the easiest way to get there is to tell people to extend distutils with a test command (or use Distribute) and perhaps add such a command in 3.4 that will do the unittest discover thingy. I remember looking into zope.testrunner hooking into that mechanism as well, but I don't remember what the outcome was. Doesn't setuptools/distribute already have a setup.py test command? Yes, but distutils do not. That seems like the easiest way forward? Yup. Although I can understand people if they want something that is independent of packaging/distribution. //Lennart ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] built-in Python test runner (was: Python Language Summit at PyCon: Agenda)
On Tue, Mar 5, 2013 at 8:13 AM, Robert Collins robe...@robertcollins.net wrote: On 5 March 2013 20:02, Lennart Regebro rege...@gmail.com wrote: What's needed here is not a tool that can run all unittests in existence, but an official way for automated tools to run tests, with the ability for any test and test framework to hook into that, so that you can run any test suite automatically from an automated tool. The, once that mechanism has been identified/implemented, we need to tell everybody to do this. I think the command line is the right place to do that - declare as metadata the command line to run a packages tests. Yeah, that's good and simple solution. //Lennart ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] built-in Python test runner (was: Python Language Summit at PyCon: Agenda)
On Tue, Mar 5, 2013 at 9:25 AM, Nick Coghlan ncogh...@gmail.com wrote: On Tue, Mar 5, 2013 at 5:02 PM, Lennart Regebro rege...@gmail.com wrote: I don't care much what that mechanism is, but I think the easiest way to get there is to tell people to extend distutils with a test command (or use Distribute) and perhaps add such a command in 3.4 that will do the unittest discover thingy. I remember looking into zope.testrunner hooking into that mechanism as well, but I don't remember what the outcome was. There is no easy way forward at this point in time. There just isn't. If people want to dispute that claim, please feel free to solve all the other problems distutils-sig is trying to tackle, so we can pay attention to this one. I have to admit that of all the packaging problems out there, this is one of the easiest ones. ;-) That said, it's not easy. We'll get to this eventually - there are just several other more important things ahead of it in the queue for packaging and distribution infrastructure enhancements (and python-dev is not the group that will solve them). To be honest I'm not sure distutils-sig is the right place for this. It's really only a packaging problem because Setuptools has a test command. :-) Perhaps we can solve this outside distutils-sig so that distutils-sig can concentrate on the harder problems? //Lennart ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] built-in Python test runner (was: Python Language Summit at PyCon: Agenda)
On Tue, Mar 5, 2013 at 1:41 AM, Robert Collins robe...@robertcollins.net wrote: So that is interesting, but its not sufficient to meet the automation need Barry is calling out, unless all test suites can be run by 'python -m unittest discover' with no additional parameters [and a pretty large subset cannot]. But can they be changed so they are? That's gotta be the important bit. What's needed here is not a tool that can run all unittests in existence, but an official way for automated tools to run tests, with the ability for any test and test framework to hook into that, so that you can run any test suite automatically from an automated tool. The, once that mechanism has been identified/implemented, we need to tell everybody to do this. I don't care much what that mechanism is, but I think the easiest way to get there is to tell people to extend distutils with a test command (or use Distribute) and perhaps add such a command in 3.4 that will do the unittest discover thingy. I remember looking into zope.testrunner hooking into that mechanism as well, but I don't remember what the outcome was. //Lennart ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Usage of += on strings in loops in stdlib
On Tue, Feb 12, 2013 at 10:03 PM, Maciej Fijalkowski fij...@gmail.com wrote: Hi We recently encountered a performance issue in stdlib for pypy. It turned out that someone commited a performance fix that uses += for strings instead of .join() that was there before. Can someone show the actual diff? Of this? I'm making a talk about outdated patterns in Python at DjangoCon EU, prompted by this question, and obsessive avoidance of string concatenation. But all the tests I've done show that ''.join() still is faster or as fast, except when you are joining very few strings, like for example two strings, in which case concatenation is faster or as fast. Both under PyPy and CPython. So I'd like to know in which case ''.hoin() is faster on PyPy and += faster on CPython. Code with times x = 10 s1 = 'X'* x s2 = 'X'* x for i in xrange(500): s1 += s2 Python 3.3: 0.049 seconds PyPy 1.9: 24.217 seconds PyPy indeed is much much slower than CPython here. But let's look at the join case: x = 10 s1 = 'X'* x s2 = 'X'* x for i in xrange(500): s1 = ''.join((s1, s2)) Python 3.3: 18.969 seconds PyPy 1.9: 62.539 seconds Here PyPy needs twice the time, and CPython needs 387 times as long time. Both are slower. The best case is of course to make a long list of strings and join them: x = 10 s1 = 'X'* x s2 = 'X'* x l = [s1] for i in xrange(500): l.append(s2) s1 = ''.join(l) Python 3.3: 0.052 seconds PyPy 1.9: 0.117 seconds That's not always feasible though. //Lennart ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Usage of += on strings in loops in stdlib
On Wed, Feb 13, 2013 at 1:10 PM, Serhiy Storchaka storch...@gmail.com wrote: I prefer x = '%s%s%s%s' % (a, b, c, d) when string's number is more than 3 and some of them are literal strings. This has the benefit of being slow both on CPython and PyPy. Although using .format() is even slower. :-) //Lennart ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Usage of += on strings in loops in stdlib
On Wed, Feb 13, 2013 at 3:27 PM, Amaury Forgeot d'Arc
amaur...@gmail.com wrote:
2013/2/13 Lennart Regebro rege...@gmail.com
On Wed, Feb 13, 2013 at 1:10 PM, Serhiy Storchaka storch...@gmail.com
wrote:
I prefer x = '%s%s%s%s' % (a, b, c, d) when string's number is more
than 3
and some of them are literal strings.
This has the benefit of being slow both on CPython and PyPy. Although
using .format() is even slower. :-)
Did you really try it?
Yes.
PyPy is really fast with str.__mod__, when the format string is a constant.
Yes, it's jitted.
Simple concatenation: s1 = s1 + s2
PyPy-1.9 time for 100 concats of 1 length strings = 7.133
CPython time for 100 concats of 1 length strings = 0.005
Making a list of strings and joining after the loop: s1 = ''.join(l)
PyPy-1.9 time for 100 concats of 1 length strings = 0.005
CPython time for 100 concats of 1 length strings = 0.003
Old formatting: s1 = '%s%s' % (s1, s2)
PyPy-1.9 time for 100 concats of 1 length strings = 20.924
CPython time for 100 concats of 1 length strings = 3.787
New formatting: s1 = '{0}{1}'.format(s1, s2)
PyPy-1.9 time for 100 concats of 1 length strings = 13.249
CPython time for 100 concats of 1 length strings = 3.751
I have, by the way, yet to find a usecase where the fastest method in
CPython is not also the fastest in PyPy.
//Lennart
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Usage of += on strings in loops in stdlib
On Wed, Feb 13, 2013 at 3:27 PM, Amaury Forgeot d'Arc amaur...@gmail.com wrote: Yes, it's jitted. Admittedly, I have no idea in which cases the JIT kicks in, and what I should do to make that happen to make sure I have the best possible real-life test cases. //Lennart ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Usage of += on strings in loops in stdlib
On Wed, Feb 13, 2013 at 7:06 PM, Maciej Fijalkowski fij...@gmail.com wrote: I actually wonder. There seems to be the consensus to avoid += (to some extent). Can someone commit the change to urrllib then? I'm talking about reverting http://bugs.python.org/issue1285086 specifically That's unquoting of URLs, strings that aren't particularly long, normally. And it's not in any tight loops. I'm astonished that any change makes any noticeable speed difference here at all. //Lennart ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Usage of += on strings in loops in stdlib
On Wed, Feb 13, 2013 at 4:02 PM, Amaury Forgeot d'Arc amaur...@gmail.com wrote: 2013/2/13 Lennart Regebro rege...@gmail.com On Wed, Feb 13, 2013 at 3:27 PM, Amaury Forgeot d'Arc amaur...@gmail.com wrote: Yes, it's jitted. Admittedly, I have no idea in which cases the JIT kicks in, and what I should do to make that happen to make sure I have the best possible real-life test cases. PyPy JIT kicks in only after 1000 iterations. Actually, my test code mixed iterations and string length up when printing the results, so the tests I showed was not 100 iterations with 10.000 long string, but 10.000 iterations with 100 long strings. No matter what the iteration/string length is .format() is the slowest or second slowest of all string concatenation methods I've tried and '%s%s' % just marginally faster. This both on PyPy and CPython and irrespective of string length. I'll stick my neck out and say that using formatting for concatenation is probably an anti-pattern. //Lennart ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] efficient string concatenation (yep, from 2004)
Something is needed - a patch for PyPy or for the documentation I guess.
Not arguing that it wouldn't be good, but I disagree that it is needed.
This is only an issue when you, as in your proof, have a loop that
does concatenation. This is usually when looping over a list of
strings that should be concatenated together. Doing so in a loop with
concatenation may be the natural way for people new to Python, but the
natural way to do it in Python is with a ''.join() call.
This:
s = ''.join(('X' for x in xrange(x)))
Is more than twice as fast in Python 2.7 than your example. It is in
fact also slower in PyPy 1.9 than Python 2.7, but only with a factor
of two:
Python 2.7:
time for 1000 concats = 0.887
Pypy 1.9:
time for 1000 concats = 1.600
(And of course s = 'X'* x takes only a bout a hundredth of the time,
but that's cheating. ;-)
//Lennart
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] PEP-0431 update 4
Here is the update of PEP-0431. The major changes are that the new class now has a lowercase name, to conform to the rest of the datetime modeule: dsttimezone (i decided against dst_timezone, as non of the other classes have underscores). Also the timezone name sets are gone. Thanks to Nick Coghlan for finding a whole bunch of minor mistakes. //Lennart PEP: 431 Title: Time zone support improvements Version: $Revision$ Last-Modified: $Date$ Author: Lennart Regebro rege...@gmail.com BDFL-Delegate: Barry Warsaw ba...@python.org Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 11-Dec-2012 Post-History: 11-Dec-2012, 28-Dec-2012, 28-Jan-2013 Abstract This PEP proposes the implementation of concrete time zone support in the Python standard library, and also improvements to the time zone API to deal with ambiguous time specifications during DST changes. Proposal Concrete time zone support -- The time zone support in Python has no concrete implementation in the standard library outside of a tzinfo baseclass that supports fixed offsets. To properly support time zones you need to include a database over all time zones, both current and historical, including daylight saving changes. But such information changes frequently, so even if we include the last information in a Python release, that information would be outdated just a few months later. Time zone support has therefore only been available through two third-party modules, ``pytz`` and ``dateutil``, both who include and wrap the zoneinfo database. This database, also called tz or The Olsen database, is the de-facto standard time zone database over time zones, and it is included in most Unix and Unix-like operating systems, including OS X. This gives us the opportunity to include the code that supports the zoneinfo data in the standard library, but by default use the operating system's copy of the data, which typically will be kept updated by the updating mechanism of the operating system or distribution. For those who have an operating system that does not include the zoneinfo database, for example Windows, the Python source distribution will include a copy of the zoneinfo database, and a distribution containing the latest zoneinfo database will also be available at the Python Package Index, so it can be easily installed with the Python packaging tools such as ``easy_install`` or ``pip``. This could also be done on Unices that are no longer receiving updates and therefore have an outdated database. With such a mechanism Python would have full time zone support in the standard library on any platform, and a simple package installation would provide an updated time zone database on those platforms where the zoneinfo database isn't included, such as Windows, or on platforms where OS updates are no longer provided. The time zone support will be implemented by making the ``datetime`` module into a package, and adding time zone support to ``datetime`` based on Stuart Bishop's ``pytz`` module. Getting the local time zone --- On Unix there is no standard way of finding the name of the time zone that is being used. All the information that is available is the time zone abbreviations, such as ``EST`` and ``PDT``, but many of those abbreviations are ambiguous and therefore you can't rely on them to figure out which time zone you are located in. There is however a standard for finding the compiled time zone information since it's located in ``/etc/localtime``. Therefore it is possible to create a local time zone object with the correct time zone information even though you don't know the name of the time zone. A function in ``datetime`` should be provided to return the local time zone. The support for this will be made by integrating Lennart Regebro's ``tzlocal`` module into the new ``datetime`` module. For Windows it will look up the local Windows time zone name, and use a mapping between Windows time zone names and zoneinfo time zone names provided by the Unicode consortium to convert that to a zoneinfo time zone. The mapping should be updated before each major or bugfix release, scripts for doing so will be provided in the ``Tools/`` directory. Ambiguous times --- When changing over from daylight savings time (DST) the clock is turned back one hour. This means that the times during that hour happens twice, once with DST and then once without DST. Similarly, when changing to daylight savings time, one hour goes missing. The current time zone API can not differentiate between the two ambiguous times during a change from DST. For example, in Stockholm the time of 2012-11-28 02:00:00 happens twice, both at UTC 2012-11-28 00:00:00 and also at 2012-11-28 01:00:00. The current time zone API can not disambiguate this and therefore it's unclear which time should be returned:: # This could be either 00:00 or 01:00 UTC: dt = datetime(2012
Re: [Python-Dev] PEP 431 Updates
On Mon, Jan 28, 2013 at 11:26 AM, Antoine Pitrou solip...@pitrou.net wrote: Will non-ambiguous shorthands such as Warsaw and GMT be accepted? pytz accepts GMT, so that will work. Otherwise no. NonExistentTimeError can also be raised for is_dst=True and is_dst=False, no? Or am I misunderstanding the semantics? No, it will only be raised when is_dst=None. The flag is there to say what timezone the resulting time should be if it is ambiguous or missing. It is true that the times that don't exist will also not exist even if you set the flag. 2012-03-25 02:30 is a time that don't exist in Europe/Warsaw, flag or not. But it will not raise an error if you set the flag to False or True, for reasons of backwards compatibility as well as ease of handling. People don't expect that certain times don't exist, and will generally not check for those exceptions. Therefore, the normal behavior is to not raise an error, but happily keep dealing with the time as if it does exist. Only of you explicitly set it to None will you get an error. //Lennart ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 431 Updates
On Sat, Jan 26, 2013 at 10:38 AM, Nick Coghlan ncogh...@gmail.com wrote: 2. Under New class DstTzInfo This will be a subclass of tzinfo rather than zoneinfo (which is not a class). Given that this is a *concrete* subclass, you may want to consider the name DstTimezone, which would be slightly more consistent with the existing timezone fixed offset subclass. (Incidentally, *grumble* at all the existing classes in that module without capitalized names...) Which probably mean it actually rather should be called tztimezone or something. 6. Under New collections Why both lists and sets? Because pytz did it. But yes, you are right, an ordered set is a better solution. Baseing it on OrderedDict seems like a hack, though. I could implement a custom orderedset, of course. //Lennart ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 431 Updates
On Mon, Jan 28, 2013 at 11:17 PM, Steven D'Aprano st...@pearwood.info wrote: On 28/01/13 23:52, Antoine Pitrou wrote: Le Mon, 28 Jan 2013 22:31:29 +1000, Nick Coghlanncogh...@gmail.com a écrit : 6. Under New collections Why both lists and sets? Because pytz did it. But yes, you are right, an ordered set is a better solution. Baseing it on OrderedDict seems like a hack, though. I could implement a custom orderedset, of course. Sets themselves have an honourable history of just being a thin wrapper around dictionaries with all the values set to None (although they're not implemented that way any more). Whether you create an actual OrderedSet class, or just expose the result of calling keys() on an OrderedDict instance is just an implementation detail, though. Why the complication? Just expose a regular set and let users call sorted() if that's what they want. An OrderedSet is not a sorted set. An OrderedSet, like an OrderedDict, remembers *insertion order*, it does not automatically sort the keys. So if datetime needs an ordered set, and I have no opinion on whether or not it really does, calling sorted() on a regular set is not the solution. I think the use case for needing really quick checks if a string is in the list of recognized timezone is rather unusual. What is usually needed is a sorted list of valid time zone names. I'll drop the set. //Lennart ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] PEP-431 updated.
OK, here is the third major incarnation of PEP-431.
I think it addresses all the concerns brought to light so far.
https://raw.github.com/regebro/tz-pep/master/pep-04tz.txt
Is there a simpler way of doing this than thus cut and paste
updating? Is there a way to make pull requests to hg.python.org, for
example?
PEP: 431
Title: Time zone support improvements
Version: $Revision$
Last-Modified: $Date$
Author: Lennart Regebro rege...@gmail.com
BDFL-Delegate: Barry Warsaw ba...@python.org
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 11-Dec-2012
Post-History: 11-Dec-2012, 28-Dec-2012
Abstract
This PEP proposes the implementation of concrete time zone support in the
Python standard library, and also improvements to the time zone API to deal
with ambiguous time specifications during DST changes.
Proposal
Concrete time zone support
--
The time zone support in Python has no concrete implementation in the
standard library outside of a tzinfo baseclass that supports fixed offsets.
To properly support time zones you need to include a database over all time
zones, both current and historical, including daylight saving changes.
But such information changes frequently, so even if we include the last
information in a Python release, that information would be outdated just a
few months later.
Time zone support has therefore only been available through two third-party
modules, ``pytz`` and ``dateutil``, both who include and wrap the zoneinfo
database. This database, also called tz or The Olsen database, is the
de-facto standard time zone database over time zones, and it is included in
most Unix and Unix-like operating systems, including OS X.
This gives us the opportunity to include the code that supports the zoneinfo
data in the standard library, but by default use the operating system's copy
of the data, which typically will be kept updated by the updating mechanism
of the operating system or distribution.
For those who have an operating system that does not include the zoneinfo
database, for example Windows, the Python source distribution will include a
copy of the zoneinfo database, and a distribution containing the latest
zoneinfo database will also be available at the Python Package Index, so it
can be easily installed with the Python packaging tools such as
``easy_install`` or ``pip``. This could also be done on Unices that are no
longer receiving updates and therefore have an outdated database.
With such a mechanism Python would have full time zone support in the
standard library on any platform, and a simple package installation would
provide an updated time zone database on those platforms where the zoneinfo
database isn't included, such as Windows, or on platforms where OS updates
are no longer provided.
The time zone support will be implemented by making the ``datetime`` module
into a package, and adding time zone support to ``datetime`` based on Stuart
Bishop's ``pytz`` module.
Getting the local time zone
---
On Unix there is no standard way of finding the name of the time zone that is
being used. All the information that is available is the time zone
abbreviations, such as ``EST`` and ``PDT``, but many of those abbreviations
are ambiguous and therefore you can't rely on them to figure out which time
zone you are located in.
There is however a standard for finding the compiled time zone information
since it's located in ``/etc/localtime``. Therefore it is possible to create
a local time zone object with the correct time zone information even though
you don't know the name of the time zone. A function in ``datetime`` should
be provided to return the local time zone.
The support for this will be made by integrating Lennart Regebro's
``tzlocal`` module into the new ``datetime`` module.
For Windows it will look up the local Windows time zone name, and use a
mapping between Windows time zone names and zoneinfo time zone names provided
by the Unicode consortium to convert that to a zoneinfo time zone.
The mapping should be updated before each major or bugfix release, scripts
for doing so will be provided in the ``Tools/`` directory.
Ambiguous times
---
When changing over from daylight savings time (DST) the clock is turned back
one hour. This means that the times during that hour happens twice, once
without DST and then once with DST. Similarly, when changing to daylight
savings time, one hour goes missing.
The current time zone API can not differentiate between the two ambiguous
times during a change from DST. For example, in Stockholm the time of
2012-11-28 02:00:00 happens twice, both at UTC 2012-11-28 00:00:00 and also
at 2012-11-28 01:00:00.
The current time zone API can not disambiguate this and therefore it's
unclear which time should be returned::
# This could be either 00:00 or 01:00 UTC:
dt = datetime(2012, 10, 28, 2, 0, tzinfo=zoneinfo('Europe/Stockholm
Re: [Python-Dev] PEP 431 Time zone support improvements - Update
On Fri, Dec 28, 2012 at 10:12 PM, Ronald Oussoren ronaldousso...@mac.comwrote: On 28 Dec, 2012, at 21:23, Lennart Regebro rege...@gmail.com wrote: Happy Holidays! Here is the update of PEP 431 with the changes that emerged after the earlier discussion. Why is the new timezone support added in a submodule of datetime? Adding the new function and exception to datetime itself wouldn't clutter the API that much, and datetime already contains some timezone support (datetime.tzinfo). Putting the API directly into the datetime module does conflict with the new timezone class from Python 3.2. The timezone() function therefore needs to be called something else, or the timezone class must be renamed. Alternative names for the timezone() function is get_timezone(), which has already been rejected, and zoneinfo() which makes it clear that it's only zoneinfo timezones that are relevant. Or the timezone class get's renamed TimeZone (which is more PEP8 anyway). We can allow the timezone() function to take both timezone(offset, [name]) as now, and timezone(name) and return a TimeZone object in the first case and a zoneinfo based timezone in the second case. Or maybe somebody else can come up with more clever options? ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
