How Best to Coerce Python Objects to Integers?
Hi all, I'd suggest that this http://blog.pyspoken.com/2017/01/02/how-best-to-c oerce-python-objects-to-integers/ is not one of the greatest articles ever written about Python exception handling. Other opinions are welcome. Kindest regards. Mark Lawrence. -- https://mail.python.org/mailman/listinfo/python-list
Re: How Best to Coerce Python Objects to Integers?
On 04/01/17 01:10, Steve D'Aprano wrote: > On Wed, 4 Jan 2017 11:22 am, Erik wrote: >> What he *should* have done is just validated his input strings before >> presenting the string to int() - i.e., process the input with knowledge >> that is specific to the problem domain before calling the >> general-purpose function. > > That's the Look Before You Leap solution. But in this case, given the > scenario described (a text file with a few typos), the best way is to ask > for forgiveness rather than permission: Yes, probably, in this case ;) OK, in the case where the function you're calling is sane (and Python's int() is - it won't blindly accept "0x101" as a hex value, for example) then it's probably right that leaping first and then, on failure, processing the value and leaping again is the right thing to do. [I tend to work in an environment where things I'm calling may not be sane (and in some cases I may never know), so I will usually consider LBYL as a way of CMA ;)]. In this whole discussion there has been no mention of what happens when the function returns None, though. > Another thought: if he is receiving human generated input, there is an > argument to be made for accepting "look alikes" -- e.g. maybe the data was > entered by Aunt Tilly, who was a typist in the 1960s and can't break the > habit of using l or I interchangeably for 1, and O for 0. Sure - and that's what I meant by processing the string according to his problem "domain". If he has Aunt Tillys doing his data input, then l->1 and 0->O may be a reasonable thing (I recently did a project where things like 0-> converting Icelandic's Eth and Thorn runic letters to 'D' and 'P' - though 0-> lly wrong ;) - was a reasonable character translation because that's what 0-> le actually typed in). E. -- https://mail.python.org/mailman/listinfo/python-list
Re: How Best to Coerce Python Objects to Integers?
On Wed, 4 Jan 2017 11:22 am, Erik wrote: > On 03/01/17 22:47, Chris Angelico wrote: >> On Wed, Jan 4, 2017 at 9:42 AM, Ethan Furmanwrote: >>> Aside from calling "except Exception" a "naked except" >> >> If you read the comments, you'll see that he originally had an actual >> bare except clause, but then improved the code somewhat in response to >> a recommendation that SystemExit etc not be caught. > > But, as stated at the top of the article, his brief was: "The strings > come from a file that a human has typed in, so even though most of the > values are good, a few will have errors ('25C') that int() will reject.". Right. And from there he starts worrying about the case where the inputs aren't strings at all, or they're weird exotic objects with nasty __int__ methods. That's overkill and can only hide programming errors. > What he *should* have done is just validated his input strings before > presenting the string to int() - i.e., process the input with knowledge > that is specific to the problem domain before calling the > general-purpose function. That's the Look Before You Leap solution. But in this case, given the scenario described (a text file with a few typos), the best way is to ask for forgiveness rather than permission: def int_or_else(value): try: return int(value) else ValueError: pass Why is this better? In the given scenario, errors are rare. Most values are good, with only a few typos, so it is wasteful to parse the string twice, once to validate it and once to generate the int. Besides, there's probably no Python code you can write which will validate an int as fast as the int() function itself. [...] > Instead, he tried to patch around int() rejecting the strings. And then > decided that he'd patch around int() rejecting things that weren't even > strings even though that's not what the function has (apparently) been > specified to receive. Indeed. Another thought: if he is receiving human generated input, there is an argument to be made for accepting "look alikes" -- e.g. maybe the data was entered by Aunt Tilly, who was a typist in the 1960s and can't break the habit of using l or I interchangeably for 1, and O for 0. -- Steve â £Cheer up,â Ø they said, â £things could be worse.â Ø So I cheered up, and sure enough, things got worse. -- https://mail.python.org/mailman/listinfo/python-list
Re: How Best to Coerce Python Objects to Integers?
On 03/01/17 22:47, Chris Angelico wrote: > On Wed, Jan 4, 2017 at 9:42 AM, Ethan Furmanwrote: >> Aside from calling "except Exception" a "naked except" > > If you read the comments, you'll see that he originally had an actual > bare except clause, but then improved the code somewhat in response to > a recommendation that SystemExit etc not be caught. But, as stated at the top of the article, his brief was: "The strings come from a file that a human has typed in, so even though most of the values are good, a few will have errors ('25C') that int() will reject.". What he *should* have done is just validated his input strings before presenting the string to int() - i.e., process the input with knowledge that is specific to the problem domain before calling the general-purpose function. He mentions temperature sensors, so perhaps stripping a trailing 'c' or 'C' is a reasonable thing to do (or even, if there's a trailing 'f' or 'F', performing a numerical conversion after the value is known). Instead, he tried to patch around int() rejecting the strings. And then decided that he'd patch around int() rejecting things that weren't even strings even though that's not what the function has (apparently) been specified to receive. The "bulletproof" result will convert "25C" to None even though 25 is probably a reasonable result for that string in his domain problem domain. E. -- https://mail.python.org/mailman/listinfo/python-list
Re: How Best to Coerce Python Objects to Integers?
On Wed, Jan 4, 2017 at 10:39 AM, Steve D'Apranowrote: > The problem here is not so much the use of try...except but the *intention* > that "Anything whatsoever should be coerced to int". If you have something > like: > > left_margin = int_or_else(ftp_server) > > that's surely a programming error that needs fixing, rather than something > that should just return a default value. Agreed. There are certainly times when you want to say "coerce any *string* to an int", and there are times when you want to say "coerce any string to int or None" (maybe you want to take the average of a whole lot of values, ignoring the ones that say "N/A"), but I don't know of any time I want to say "coerce any programming error to int or None", unless you count the exit code of a process. I used the word "improved" rather loosely. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: How Best to Coerce Python Objects to Integers?
On 01/03/2017 02:47 PM, Chris Angelico wrote: > On Wed, Jan 4, 2017 at 9:42 AM, Ethan Furman wrote: >> Aside from calling "except Exception" a "naked except" > > If you read the comments, you'll see that he originally had an actual > bare except clause, but then improved the code somewhat in response to > a recommendation that SystemExit etc not be caught. I did read the comments and noticed the reasons for the improved code; fixing the article to not use the "naked except" phrase would be another improvement. And, of course, whether or not "except Exception" is too broad depends on the use-case. -- ~Ethan~ -- https://mail.python.org/mailman/listinfo/python-list
Re: How Best to Coerce Python Objects to Integers?
On Wed, 4 Jan 2017 10:09 am, Ethan Furman wrote: > And, of course, whether or not "except Exception" is too broad depends on > the use-case. I'm having trouble thinking of *any* use-case where this would be useful. His intention, it seems, is to write a function which simply cannot fail, presumably so that he can write code which consumes the output and just keeps going no matter what he throws at it: graph = make_graph() incoming_data = [1, 12, 7, "hello", None, [], {}, 5] for obj in incoming_data: graph.draw_point(int_or_else(obj)) But of course that's impossible: class Evil: def __int__(self): os.abort() Or for that matter: class Evil: def __int__(self): time.sleep(2147483647) So his code still can fail under some circumstances. And so it should. Why is my input data including such evil objects? I should find out why. I'm having trouble seeing how it could be anything but a programming bug. He gives the rationale for this function: A scenario thatâ Ös more realistic than the Unintable class might be a class that wraps an industrial sensor. Calling int() on an instance normally returns a value representing pressure or temperature. However, it might reasonably raise a SensorNotReadyError. Okay. Let's suppose it can return either an integer as a string, some arbitrary non-numeric string to indicate a sensor error, or raises SensorNotReadyError. Then this would be an appropriate function to use: def int_or_else(value): try: return int(value) except ValueError: assert isinstance(value, str) # consider logging the error? return None except SensorNotReadyError: return None Now when he connects up to the sensor and starts reading values, it will work, but if his input gets contaminated with arbitrary junk objects (a sign of a programming bug in his code) he will find out about it. One possible use-case might be something like Excel, where there are two data types: numbers, and text, and numeric operations on text will just skip them altogether. If you were to build an object-oriented spreadsheet, where the cells can contain any object not just numbers and text, then you could potentially have a situation like: Column A Row 1: 1 Row 2: ftp_server() Row 3: 2 Row 4: 3 Row 5: =sum(A1:A4) and you (arguably) want the result to be 6 rather than some error. Or do you? I can make a good case for skipping text cells, as Excel does, but I'm not sure that ftp_server should be skipped. So I'll count that as a use-case, but a dubious one. Another possible use-case might be the REPL for an interpreter, where you want the REPL to keep going no matter what exceptions take place. But I don't think this is the right way to do that, and it's not how the Python REPL works either. Other than that, I'm not seeing any use-case where this sort of thing is anything but a bad idea. -- Steve â £Cheer up,â Ø they said, â £things could be worse.â Ø So I cheered up, and sure enough, things got worse. -- https://mail.python.org/mailman/listinfo/python-list
Re: How Best to Coerce Python Objects to Integers?
On Wed, 4 Jan 2017 09:47 am, Chris Angelico wrote: > On Wed, Jan 4, 2017 at 9:42 AM, Ethan Furmanwrote: >> Aside from calling "except Exception" a "naked except" > > If you read the comments, you'll see that he originally had an actual > bare except clause, but then improved the code somewhat in response to > a recommendation that SystemExit etc not be caught. But why? That makes no sense. If his intention is to return None on failure, no matter what happens, then he *should* catch SystemExit. Otherwise: class Unint: def __int__(self): raise SystemExit int_or_else(Unint) will exit instead of returning None. Surely that's not what he wants, given that he wants to cover up other programming errors like NameError and TypeError? The problem here is not so much the use of try...except but the *intention* that "Anything whatsoever should be coerced to int". If you have something like: left_margin = int_or_else(ftp_server) that's surely a programming error that needs fixing, rather than something that should just return a default value. -- Steve â £Cheer up,â Ø they said, â £things could be worse.â Ø So I cheered up, and sure enough, things got worse. -- https://mail.python.org/mailman/listinfo/python-list
Re: How Best to Coerce Python Objects to Integers?
On 01/03/2017 01:41 PM, breamore...@gmail.com wrote: > Hi all, I'd suggest that this [1] is not one of the greatest articles > ever written about Python exception handling. Other opinions are welcome. Aside from calling "except Exception" a "naked except" I think it's decent. He walks through the problem, explains the rationale, and only has one line of code guarded by the except clause. -- ~Ethan~ [1] http://blog.pyspoken.com/2017/01/02/how-best-to-coerce-python-objects-to-in tegers/ -- https://mail.python.org/mailman/listinfo/python-list
Re: How Best to Coerce Python Objects to Integers?
On Wed, Jan 4, 2017 at 9:42 AM, Ethan Furmanwrote: > Aside from calling "except Exception" a "naked except" If you read the comments, you'll see that he originally had an actual bare except clause, but then improved the code somewhat in response to a recommendation that SystemExit etc not be caught. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: How Best to Coerce Python Objects to Integers?
On Wed, Jan 4, 2017 at 8:41 AM, <breamore...@gmail.com> wrote: > Hi all, I'd suggest that this http://blog.pyspoken.com/2017/01/02/how-best-to -coerce-python-objects-to-integers/ is not one of the greatest articles ever written about Python exception handling. Other opinions are welcome. > """ So there you have it. Iâ Öm happy with this function. It feels bulletproof. It contains a naked except, but that only covers one simple line of code thatâ Ös unlikely to hide anything nasty. """ Yep! It's perfect. He has successfully made a function that won't leak any exceptions. Congratulations! The novice believes that the first priority is to stop the program from crashing. The expert understands that crashing (especially with an exception, but even a segfault) is actually very helpful and useful. I love the logic that a bare 'except' is okay as long as it's only covering one single line of code. There's almost enough truth in that to be meaningful, while still completely missing the point that a NameError is almost certainly a bug no matter where it crops up. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: How Best to Coerce Python Objects to Integers?
On 04/01/17 01:10, Steve D'Aprano wrote: On Wed, 4 Jan 2017 11:22 am, Erik wrote: What he *should* have done is just validated his input strings before presenting the string to int() - i.e., process the input with knowledge that is specific to the problem domain before calling the general-purpose function. That's the Look Before You Leap solution. But in this case, given the scenario described (a text file with a few typos), the best way is to ask for forgiveness rather than permission: Yes, probably, in this case ;) OK, in the case where the function you're calling is sane (and Python's int() is - it won't blindly accept "0x101" as a hex value, for example) then it's probably right that leaping first and then, on failure, processing the value and leaping again is the right thing to do. [I tend to work in an environment where things I'm calling may not be sane (and in some cases I may never know), so I will usually consider LBYL as a way of CMA ;)]. In this whole discussion there has been no mention of what happens when the function returns None, though. Another thought: if he is receiving human generated input, there is an argument to be made for accepting "look alikes" -- e.g. maybe the data was entered by Aunt Tilly, who was a typist in the 1960s and can't break the habit of using l or I interchangeably for 1, and O for 0. Sure - and that's what I meant by processing the string according to his problem "domain". If he has Aunt Tillys doing his data input, then l->1 and 0->O may be a reasonable thing (I recently did a project where things like converting Icelandic's Eth and Thorn runic letters to 'D' and 'P' - though morally wrong ;) - was a reasonable character translation because that's what people actually typed in). E. -- https://mail.python.org/mailman/listinfo/python-list
Re: How Best to Coerce Python Objects to Integers?
On Wed, 4 Jan 2017 11:22 am, Erik wrote: > On 03/01/17 22:47, Chris Angelico wrote: >> On Wed, Jan 4, 2017 at 9:42 AM, Ethan Furmanwrote: >>> Aside from calling "except Exception" a "naked except" >> >> If you read the comments, you'll see that he originally had an actual >> bare except clause, but then improved the code somewhat in response to >> a recommendation that SystemExit etc not be caught. > > But, as stated at the top of the article, his brief was: "The strings > come from a file that a human has typed in, so even though most of the > values are good, a few will have errors ('25C') that int() will reject.". Right. And from there he starts worrying about the case where the inputs aren't strings at all, or they're weird exotic objects with nasty __int__ methods. That's overkill and can only hide programming errors. > What he *should* have done is just validated his input strings before > presenting the string to int() - i.e., process the input with knowledge > that is specific to the problem domain before calling the > general-purpose function. That's the Look Before You Leap solution. But in this case, given the scenario described (a text file with a few typos), the best way is to ask for forgiveness rather than permission: def int_or_else(value): try: return int(value) else ValueError: pass Why is this better? In the given scenario, errors are rare. Most values are good, with only a few typos, so it is wasteful to parse the string twice, once to validate it and once to generate the int. Besides, there's probably no Python code you can write which will validate an int as fast as the int() function itself. [...] > Instead, he tried to patch around int() rejecting the strings. And then > decided that he'd patch around int() rejecting things that weren't even > strings even though that's not what the function has (apparently) been > specified to receive. Indeed. Another thought: if he is receiving human generated input, there is an argument to be made for accepting "look alikes" -- e.g. maybe the data was entered by Aunt Tilly, who was a typist in the 1960s and can't break the habit of using l or I interchangeably for 1, and O for 0. -- Steve “Cheer up,” they said, “things could be worse.” So I cheered up, and sure enough, things got worse. -- https://mail.python.org/mailman/listinfo/python-list
Re: How Best to Coerce Python Objects to Integers?
On Wed, 4 Jan 2017 10:09 am, Ethan Furman wrote: > And, of course, whether or not "except Exception" is too broad depends on > the use-case. I'm having trouble thinking of *any* use-case where this would be useful. His intention, it seems, is to write a function which simply cannot fail, presumably so that he can write code which consumes the output and just keeps going no matter what he throws at it: graph = make_graph() incoming_data = [1, 12, 7, "hello", None, [], {}, 5] for obj in incoming_data: graph.draw_point(int_or_else(obj)) But of course that's impossible: class Evil: def __int__(self): os.abort() Or for that matter: class Evil: def __int__(self): time.sleep(2147483647) So his code still can fail under some circumstances. And so it should. Why is my input data including such evil objects? I should find out why. I'm having trouble seeing how it could be anything but a programming bug. He gives the rationale for this function: A scenario that’s more realistic than the Unintable class might be a class that wraps an industrial sensor. Calling int() on an instance normally returns a value representing pressure or temperature. However, it might reasonably raise a SensorNotReadyError. Okay. Let's suppose it can return either an integer as a string, some arbitrary non-numeric string to indicate a sensor error, or raises SensorNotReadyError. Then this would be an appropriate function to use: def int_or_else(value): try: return int(value) except ValueError: assert isinstance(value, str) # consider logging the error? return None except SensorNotReadyError: return None Now when he connects up to the sensor and starts reading values, it will work, but if his input gets contaminated with arbitrary junk objects (a sign of a programming bug in his code) he will find out about it. One possible use-case might be something like Excel, where there are two data types: numbers, and text, and numeric operations on text will just skip them altogether. If you were to build an object-oriented spreadsheet, where the cells can contain any object not just numbers and text, then you could potentially have a situation like: Column A Row 1: 1 Row 2: ftp_server() Row 3: 2 Row 4: 3 Row 5: =sum(A1:A4) and you (arguably) want the result to be 6 rather than some error. Or do you? I can make a good case for skipping text cells, as Excel does, but I'm not sure that ftp_server should be skipped. So I'll count that as a use-case, but a dubious one. Another possible use-case might be the REPL for an interpreter, where you want the REPL to keep going no matter what exceptions take place. But I don't think this is the right way to do that, and it's not how the Python REPL works either. Other than that, I'm not seeing any use-case where this sort of thing is anything but a bad idea. -- Steve “Cheer up,” they said, “things could be worse.” So I cheered up, and sure enough, things got worse. -- https://mail.python.org/mailman/listinfo/python-list
Re: How Best to Coerce Python Objects to Integers?
On 03/01/17 22:47, Chris Angelico wrote: On Wed, Jan 4, 2017 at 9:42 AM, Ethan Furmanwrote: Aside from calling "except Exception" a "naked except" If you read the comments, you'll see that he originally had an actual bare except clause, but then improved the code somewhat in response to a recommendation that SystemExit etc not be caught. But, as stated at the top of the article, his brief was: "The strings come from a file that a human has typed in, so even though most of the values are good, a few will have errors ('25C') that int() will reject.". What he *should* have done is just validated his input strings before presenting the string to int() - i.e., process the input with knowledge that is specific to the problem domain before calling the general-purpose function. He mentions temperature sensors, so perhaps stripping a trailing 'c' or 'C' is a reasonable thing to do (or even, if there's a trailing 'f' or 'F', performing a numerical conversion after the value is known). Instead, he tried to patch around int() rejecting the strings. And then decided that he'd patch around int() rejecting things that weren't even strings even though that's not what the function has (apparently) been specified to receive. The "bulletproof" result will convert "25C" to None even though 25 is probably a reasonable result for that string in his domain problem domain. E. -- https://mail.python.org/mailman/listinfo/python-list
Re: How Best to Coerce Python Objects to Integers?
On Wed, Jan 4, 2017 at 10:39 AM, Steve D'Apranowrote: > The problem here is not so much the use of try...except but the *intention* > that "Anything whatsoever should be coerced to int". If you have something > like: > > left_margin = int_or_else(ftp_server) > > that's surely a programming error that needs fixing, rather than something > that should just return a default value. Agreed. There are certainly times when you want to say "coerce any *string* to an int", and there are times when you want to say "coerce any string to int or None" (maybe you want to take the average of a whole lot of values, ignoring the ones that say "N/A"), but I don't know of any time I want to say "coerce any programming error to int or None", unless you count the exit code of a process. I used the word "improved" rather loosely. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: How Best to Coerce Python Objects to Integers?
On Wed, 4 Jan 2017 09:47 am, Chris Angelico wrote: > On Wed, Jan 4, 2017 at 9:42 AM, Ethan Furmanwrote: >> Aside from calling "except Exception" a "naked except" > > If you read the comments, you'll see that he originally had an actual > bare except clause, but then improved the code somewhat in response to > a recommendation that SystemExit etc not be caught. But why? That makes no sense. If his intention is to return None on failure, no matter what happens, then he *should* catch SystemExit. Otherwise: class Unint: def __int__(self): raise SystemExit int_or_else(Unint) will exit instead of returning None. Surely that's not what he wants, given that he wants to cover up other programming errors like NameError and TypeError? The problem here is not so much the use of try...except but the *intention* that "Anything whatsoever should be coerced to int". If you have something like: left_margin = int_or_else(ftp_server) that's surely a programming error that needs fixing, rather than something that should just return a default value. -- Steve “Cheer up,” they said, “things could be worse.” So I cheered up, and sure enough, things got worse. -- https://mail.python.org/mailman/listinfo/python-list
Re: How Best to Coerce Python Objects to Integers?
On 01/03/2017 02:47 PM, Chris Angelico wrote: On Wed, Jan 4, 2017 at 9:42 AM, Ethan Furman wrote: Aside from calling "except Exception" a "naked except" If you read the comments, you'll see that he originally had an actual bare except clause, but then improved the code somewhat in response to a recommendation that SystemExit etc not be caught. I did read the comments and noticed the reasons for the improved code; fixing the article to not use the "naked except" phrase would be another improvement. And, of course, whether or not "except Exception" is too broad depends on the use-case. -- ~Ethan~ -- https://mail.python.org/mailman/listinfo/python-list
Re: How Best to Coerce Python Objects to Integers?
On Wed, Jan 4, 2017 at 9:42 AM, Ethan Furmanwrote: > Aside from calling "except Exception" a "naked except" If you read the comments, you'll see that he originally had an actual bare except clause, but then improved the code somewhat in response to a recommendation that SystemExit etc not be caught. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: How Best to Coerce Python Objects to Integers?
On 01/03/2017 01:41 PM, breamore...@gmail.com wrote: Hi all, I'd suggest that this [1] is not one of the greatest articles ever written about Python exception handling. Other opinions are welcome. Aside from calling "except Exception" a "naked except" I think it's decent. He walks through the problem, explains the rationale, and only has one line of code guarded by the except clause. -- ~Ethan~ [1] http://blog.pyspoken.com/2017/01/02/how-best-to-coerce-python-objects-to-integers/ -- https://mail.python.org/mailman/listinfo/python-list
Re: How Best to Coerce Python Objects to Integers?
On Wed, Jan 4, 2017 at 8:41 AM, <breamore...@gmail.com> wrote: > Hi all, I'd suggest that this > http://blog.pyspoken.com/2017/01/02/how-best-to-coerce-python-objects-to-integers/ > is not one of the greatest articles ever written about Python exception > handling. Other opinions are welcome. > """ So there you have it. I’m happy with this function. It feels bulletproof. It contains a naked except, but that only covers one simple line of code that’s unlikely to hide anything nasty. """ Yep! It's perfect. He has successfully made a function that won't leak any exceptions. Congratulations! The novice believes that the first priority is to stop the program from crashing. The expert understands that crashing (especially with an exception, but even a segfault) is actually very helpful and useful. I love the logic that a bare 'except' is okay as long as it's only covering one single line of code. There's almost enough truth in that to be meaningful, while still completely missing the point that a NameError is almost certainly a bug no matter where it crops up. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
How Best to Coerce Python Objects to Integers?
Hi all, I'd suggest that this http://blog.pyspoken.com/2017/01/02/how-best-to-coerce-python-objects-to-integers/ is not one of the greatest articles ever written about Python exception handling. Other opinions are welcome. Kindest regards. Mark Lawrence. -- https://mail.python.org/mailman/listinfo/python-list