Re: common mistakes in this simple program

[Ganesh Pal]

On Tue, Mar 1, 2016 at 2:41 AM, Martin A. Brown  wrote:

> Please read below.  I will take a stab at explaining the gaps of
> understanding you seem to have (others have tried already, but I'll
> try, as well).
>
> I am going to give you four different functions which demonstrate
> how to use exceptions.  You may find it instructive to paste these
> functions into an interactive Python shell and try them out, as
> well.
>

Thanks Martin for beautifully demonstrating the use of exception  with
an example that's easy to understand.

Step 1: catch a specific Exception
Step 2: catch a specific Exception
Step 3: catch several different classes of Exception
Step 4: catch many different classes of Exception
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: common mistakes in this simple program

[Steven D'Aprano]

On Tue, 1 Mar 2016 03:29 am, Ian Kelly wrote:

> On Mon, Feb 29, 2016 at 8:18 AM, Ganesh Pal  wrote:
>> Iam on python 2.6
> 
> Python 2.6 has been unsupported since October 2013. Among other
> things, that means it is no longer receiving security updates like
> more recent versions. Unless you have an extremely strong reason for
> wanting to stay to Python 2.6, you should update your Python version.

Python 2.6 is still fully supported by Red Hat.



-- 
Steven

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: common mistakes in this simple program

[Ian Kelly]

On Mon, Feb 29, 2016 at 4:14 PM, Cameron Simpson  wrote:
> Another remark here: if you're going to log, log the exception as well:
>
> logging.error("something went wrong: %s", e)
>
> Ian's example code is nice and simple to illustrate "log and then reraise"
> but few things are as annoying as log files reciting "something went wrong"
> or the equivalent without any accompanying context information.

The proper way to pass in the exception is as a keyword argument:

logging.error("something went wrong", exc_info=e)

But I actually meant to use logging.exception, which handles this automatically.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: common mistakes in this simple program

[Rob Gaddi]

Cameron Simpson wrote:

> On 29Feb2016 10:45, Ian Kelly  wrote:
>>On Mon, Feb 29, 2016 at 10:26 AM, Ganesh Pal  wrote:
>>> On Mon, Feb 29, 2016 at 9:59 PM, Ian Kelly  wrote:
 On Mon, Feb 29, 2016 at 8:18 AM, Ganesh Pal  wrote:
> 1. usage of try- expect

 try-except in every single function is a code smell. You should only
 be using it where you're actually going to handle the exception. If
 you catch an exception just to log it, you generally should also
 reraise it so that something further up the call chain has the
 opportunity to handle it.
>>>
>>> How do we reraise the exception in python ,  I have used raise not
>>> sure how to reraise the exception
>>
>>raise with no arguments will reraise the exception currently being handled.
>>
>>except Exception:
>>logging.error("something went wrong")
>>raise
>
> Another remark here: if you're going to log, log the exception as well:
>
>  logging.error("something went wrong: %s", e)
>
> Ian's example code is nice and simple to illustrate "log and then reraise" 
> but 
> few things are as annoying as log files reciting "something went wrong" or 
> the 
> equivalent without any accompanying context information.
>
> Cheers,
> Cameron Simpson 

Or, for that matter:

  logging.exception('something went wrong')

Which gives you the whole traceback as well and doesn't require you to
explictly grab the exception.

-- 
Rob Gaddi, Highland Technology -- www.highlandtechnology.com
Email address domain is currently out of order.  See above to fix.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: common mistakes in this simple program

[Cameron Simpson]

On 29Feb2016 10:45, Ian Kelly  wrote:

On Mon, Feb 29, 2016 at 10:26 AM, Ganesh Pal  wrote:

On Mon, Feb 29, 2016 at 9:59 PM, Ian Kelly  wrote:

On Mon, Feb 29, 2016 at 8:18 AM, Ganesh Pal  wrote:

1. usage of try- expect


try-except in every single function is a code smell. You should only
be using it where you're actually going to handle the exception. If
you catch an exception just to log it, you generally should also
reraise it so that something further up the call chain has the
opportunity to handle it.


How do we reraise the exception in python ,  I have used raise not
sure how to reraise the exception


raise with no arguments will reraise the exception currently being handled.

except Exception:
   logging.error("something went wrong")
   raise


Another remark here: if you're going to log, log the exception as well:

logging.error("something went wrong: %s", e)

Ian's example code is nice and simple to illustrate "log and then reraise" but 
few things are as annoying as log files reciting "something went wrong" or the 
equivalent without any accompanying context information.


Cheers,
Cameron Simpson 
--
https://mail.python.org/mailman/listinfo/python-list

Re: common mistakes in this simple program

[Martin A. Brown]

Greetings Ganesh,

>> You're falling into the trap of assuming that the only exception you
>> can ever get is the one that you're planning for, and then handling.
>
>Ok sure !

This point is very important, so I'll reiterate it.  I hope the poor 
horse lives.

>> ALL exceptions as though they were that one. Instead catch ONLY the
>> exception that you're expecting to see, and ignore everything else. Do
>> not use a bare "except:" clause, nor even "except Exception:", for
>> this. You will appreciate it later on.
>
>What option do I have now in so ensure that I loop over the For 
>loop. 

>Try except is ruled out ?

No, no not at all!

  #1:  Yes, you (probably) should be using the try-except construct.
  #2:  No, you should not (ever) use a bare try-except construct.

Please read below.  I will take a stab at explaining the gaps of 
understanding you seem to have (others have tried already, but I'll 
try, as well).

I am going to give you four different functions which demonstrate 
how to use exceptions.  You may find it instructive to paste these 
functions into an interactive Python shell and try them out, as 
well.

I will explain a few different cases, which I will show below, but I 
think you'll need to figure out how to apply this to your situation.


Step 1: catch a specific Exception
--

  def catchValueError(val, default=0):
  '''somebody passed in a non-convertible type, return a default'''
  try:
  return int(val)
  except ValueError:
  return default

Now, let's try passing a few values into that function:

  >>> catchValueError(0)
  0
  >>> catchValueError('-3')
  -3
  >>> catchValueError('Catullus')
  0

The last example above is the most interesting.  We fed a string to 
the builtin function int() and it raised a ValueError.  Our little 
try-except block, however, caught the ValueError and performed our 
desired action (in this case, returning a default value).


Step 2: catch a specific Exception
--

But, what about this:

  >>> catchValueError(dict())
  Traceback (most recent call last):
File "", line 1, in 
File "", line 4, in catchValueError
  TypeError: int() argument must be a string or a number, not 'dict'

Now, you can see that there's a different problem.  The exception is 
different if we feed different data to the function.  So, let's try 
again, but maybe we catch the TypeError instead.

  def catchTypeError(val, default=0):
'''somebody passed in a non-convertible type, return a default'''
try:
return int(val)
except TypeError:
return default

This seems to work--but, now, if we try our other values (which 
worked before), they don't work now:

  >>> catchTypeError(dict())
  0
  >>> catchTypeError(0)
  0
  >>> catchTypeError('-3')
  -3
  >>> catchTypeError('Catullus')
  Traceback (most recent call last):
File "", line 1, in 
File "", line 4, in catchTypeError
  ValueError: invalid literal for int() with base 10: 'Catullus'


Step 3: catch several different classes of Exception


You can a single try-except block to catch multiple classes of 
specific Exception (and then perform exception handling).

  def catchTVError(val, default=0):
'''somebody passed in a non-convertible type, return a default'''
try:
return int(val)
except (ValueError, TypeError):
return default

In this little toy demonstration, you can see that this is probably 
the desired functionality.

  >>> catchTVError(dict())
  0
  >>> catchTVError(0)
  0
  >>> catchTVError(-3)
  -3
  >>> catchTVError('Catullus')
  0


Step 4: catch many different classes of Exception
-

Now, suppose you want to deal with each possible exception in a different
manneryou can have different exception-handling behaviour for each class of
exception.

  def altCatchTVError(val, default=42):
  '''somebody passed in a non-convertible type, return a default'''
  try:
  return int(val)
  except ValueError:
  return abs(default)
  except TypeError:
  return 0 - abs(default)

As you can see, this offers a good deal more flexibility for handling specific
exceptions that you might encounter.

  >>> altCatchTVError(0)
  0
  >>> altCatchTVError(22)
  22
  >>> altCatchTVError('17')
  17
  >>> altCatchTVError('-3')
  -3
  >>> altCatchTVError('str')
  42
  >>> altCatchTVError(dict())
  -42


Interlude and recommendation


As you can see, there are many possible Exceptions that can be raised when you
are calling a simple builtin function, int().

Consider now what may happen when you call out to a different program; you
indicated that your run() function calls out to subprocess.Popen().  There are
many more possible errors that can occur, just a few that can come to my mind:

  * locating the program on disk
  * setting 

Re: common mistakes in this simple program

[Marko Rauhamaa]

sohcahto...@gmail.com:

> Every time you say "try-expect", my head wants to explode.
>
> It is called a "try-except" block, because you're using the key words
> "try" and "except" when you make one.

Ah, I remember a Python-based test system idea where the "except"
keyword meant "expect":

try: event()
except user.CloseRequest():
...
except user.SendRequest():
...
except peer.DataIndication():
...
except Timeout():
...

etc.

In the end, though, we came up with something better.


Marko
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: common mistakes in this simple program

[Joel Goldstick]

On Mon, Feb 29, 2016 at 2:49 PM, Ganesh Pal  wrote:

> On Mar 1, 2016 12:06 AM, "Chris Angelico"  wrote
> >
> > You're falling into the trap of assuming that the only exception you
> > can ever get is the one that you're planning for, and then handling.
>
> Ok sure !
>
> > ALL exceptions as though they were that one. Instead catch ONLY the
> > exception that you're expecting to see, and ignore everything else. Do
> > not use a bare "except:" clause, nor even "except Exception:", for
> > this. You will appreciate it later on.
>
> What option do I have now in  so ensure that I loop over the For loop. Try
> except is ruled out ?
>

Its not ruled out.  It is pointless.  Look at the documents.  Here is a
start: https://wiki.python.org/moin/HandlingExceptions

> --
> https://mail.python.org/mailman/listinfo/python-list
>



-- 
Joel Goldstick
http://joelgoldstick.com/ 
http://cc-baseballstats.info/
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: common mistakes in this simple program

[Ganesh Pal]

On Mar 1, 2016 12:06 AM, "Chris Angelico"  wrote
>
> You're falling into the trap of assuming that the only exception you
> can ever get is the one that you're planning for, and then handling.

Ok sure !

> ALL exceptions as though they were that one. Instead catch ONLY the
> exception that you're expecting to see, and ignore everything else. Do
> not use a bare "except:" clause, nor even "except Exception:", for
> this. You will appreciate it later on.

What option do I have now in  so ensure that I loop over the For loop. Try
except is ruled out ?
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: common mistakes in this simple program

[sohcahtoa82]

On Monday, February 29, 2016 at 10:21:57 AM UTC-8, Ganesh Pal wrote:
> >> How do we reraise the exception in python ,  I have used raise not
> >> sure how to reraise the exception
> >
> > raise with no arguments will reraise the exception currently being handled.
> >
> > except Exception:
> > logging.error("something went wrong")
> > raise
> 
> Thanks Ian for taking time and looking into the code ,  o k raise
> keyword for raising exception is fine .
> 
>  assert ret ==0,"ERROR (ret %d): " \
>  " \nout: %s\nerr: %s\n" % (ret, out, err)
>  except Exception as e:
>  print("Failed to run %s got %s" % (cmd, e))
>  return False
>  return True
> 
>  def prep_host():
>  """
>  Prepare clustering
>  """
>  for cmd in ["ls -al",
>  "touch /tmp/file1",
>  "mkdir /tmp/dir1"]:
>  try:
>  if not run_cmd_and_verify(cmd, timeout=3600):
>  return False
>  except:
> >>>
> >>> What exceptions are you expecting this to catch? run_cmd_and_verify
> >>> already catches any expected exceptions that it raises.
> 
> In my case the exception is nothing but the error  example if  we plan
> to run the command  say  #ifconfig -a and the command fails because of
> a type ( say u ran #igconfig -a).
> 
> we will the output as
> 
> # Failed to run igconfig -a got Error (ret=127)
> out :
> error: command not found: igconfig
> 
> So the execption is the error i.e Error (ret=127) out : error: command
> not found: igconfig,  Iam fine with this behaviour.
> 
> 
> >
> > But that exception is already caught by the run_cmd_and_verify
> > function, so what exception are you expecting to be caught *here*?
> 
> I wanted to run the command in a loop  and have a fxn for the pattern
> that repeats in this case the function is run_cmd_and_verify  , the
> only known way to me was using try with expect
> 
> I thought I will use try and have pass in except which  you don't recommend
> 
>for cmd in ["ls -al",
>  "touch /tmp/file1",
> "mkdir /tmp/dir1"]:
>   try:
>if not run_cmd_and_verify(cmd, timeout=3600):
> print "running command failed "
>return False
> except:
> pass
> 
> > You should virtually never just pass in an exception handler. Either
> > handle the exception, or log it and reraise it. If you're going to do
> > neither of those things, then don't use a try-except at all.
> 
> What alternative do I have other than try-expect ? can try - else be
> used  for my case?
> 
> Regards,
> GPal

Every time you say "try-expect", my head wants to explode.

It is called a "try-except" block, because you're using the key words "try" and 
"except" when you make one.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: common mistakes in this simple program

[Chris Angelico]

On Tue, Mar 1, 2016 at 5:21 AM, Ganesh Pal  wrote:
>
> In my case the exception is nothing but the error  example if  we plan
> to run the command  say  #ifconfig -a and the command fails because of
> a type ( say u ran #igconfig -a).
>
> we will the output as
>
> # Failed to run igconfig -a got Error (ret=127)
> out :
> error: command not found: igconfig
>
> So the execption is the error i.e Error (ret=127) out : error: command
> not found: igconfig,  Iam fine with this behaviour.
>

You're falling into the trap of assuming that the only exception you
can ever get is the one that you're planning for, and then handling
ALL exceptions as though they were that one. Instead catch ONLY the
exception that you're expecting to see, and ignore everything else. Do
not use a bare "except:" clause, nor even "except Exception:", for
this. You will appreciate it later on.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: common mistakes in this simple program

[Ganesh Pal]

>> How do we reraise the exception in python ,  I have used raise not
>> sure how to reraise the exception
>
> raise with no arguments will reraise the exception currently being handled.
>
> except Exception:
> logging.error("something went wrong")
> raise

Thanks Ian for taking time and looking into the code ,  o k raise
keyword for raising exception is fine .

 assert ret ==0,"ERROR (ret %d): " \
 " \nout: %s\nerr: %s\n" % (ret, out, err)
 except Exception as e:
 print("Failed to run %s got %s" % (cmd, e))
 return False
 return True

 def prep_host():
 """
 Prepare clustering
 """
 for cmd in ["ls -al",
 "touch /tmp/file1",
 "mkdir /tmp/dir1"]:
 try:
 if not run_cmd_and_verify(cmd, timeout=3600):
 return False
 except:
>>>
>>> What exceptions are you expecting this to catch? run_cmd_and_verify
>>> already catches any expected exceptions that it raises.

In my case the exception is nothing but the error  example if  we plan
to run the command  say  #ifconfig -a and the command fails because of
a type ( say u ran #igconfig -a).

we will the output as

# Failed to run igconfig -a got Error (ret=127)
out :
error: command not found: igconfig

So the execption is the error i.e Error (ret=127) out : error: command
not found: igconfig,  Iam fine with this behaviour.


>
> But that exception is already caught by the run_cmd_and_verify
> function, so what exception are you expecting to be caught *here*?

I wanted to run the command in a loop  and have a fxn for the pattern
that repeats in this case the function is run_cmd_and_verify  , the
only known way to me was using try with expect

I thought I will use try and have pass in except which  you don't recommend

   for cmd in ["ls -al",
 "touch /tmp/file1",
"mkdir /tmp/dir1"]:
  try:
   if not run_cmd_and_verify(cmd, timeout=3600):
print "running command failed "
   return False
except:
pass

> You should virtually never just pass in an exception handler. Either
> handle the exception, or log it and reraise it. If you're going to do
> neither of those things, then don't use a try-except at all.

What alternative do I have other than try-expect ? can try - else be
used  for my case?

Regards,
GPal
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: common mistakes in this simple program

[Ian Kelly]

On Mon, Feb 29, 2016 at 10:26 AM, Ganesh Pal  wrote:
> On Mon, Feb 29, 2016 at 9:59 PM, Ian Kelly  wrote:
>> On Mon, Feb 29, 2016 at 8:18 AM, Ganesh Pal  wrote:
>>> Iam on python 2.6
>
>>> 1. usage of try- expect
>>
>> try-except in every single function is a code smell. You should only
>> be using it where you're actually going to handle the exception. If
>> you catch an exception just to log it, you generally should also
>> reraise it so that something further up the call chain has the
>> opportunity to handle it.
>
> How do we reraise the exception in python ,  I have used raise not
> sure how to reraise the exception

raise with no arguments will reraise the exception currently being handled.

except Exception:
logging.error("something went wrong")
raise

>>> assert ret ==0,"ERROR (ret %d): " \
>>> " \nout: %s\nerr: %s\n" % (ret, out, err)
>>> except Exception as e:
>>> print("Failed to run %s got %s" % (cmd, e))
>>> return False
>>> return True
>>>
>>> def prep_host():
>>> """
>>> Prepare clustering
>>> """
>>> for cmd in ["ls -al",
>>> "touch /tmp/file1",
>>> "mkdir /tmp/dir1"]:
>>> try:
>>> if not run_cmd_and_verify(cmd, timeout=3600):
>>> return False
>>> except:
>>
>> What exceptions are you expecting this to catch? run_cmd_and_verify
>> already catches any expected exceptions that it raises.
>
> This is a wrapper for Popen it runs the command and returns stdout
> ,stderror and returncode .The only exception it can return is a time
> out exception

But that exception is already caught by the run_cmd_and_verify
function, so what exception are you expecting to be caught *here*?

> Here is the modified buggy code ,  Can I have Try and except with Pass
> , how do I modify the  try and expect in the pre-host ?

You should virtually never just pass in an exception handler. Either
handle the exception, or log it and reraise it. If you're going to do
neither of those things, then don't use a try-except at all.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: common mistakes in this simple program

[Ganesh Pal]

On Mon, Feb 29, 2016 at 9:59 PM, Ian Kelly  wrote:
> On Mon, Feb 29, 2016 at 8:18 AM, Ganesh Pal  wrote:
>> Iam on python 2.6

>> 1. usage of try- expect
>
> try-except in every single function is a code smell. You should only
> be using it where you're actually going to handle the exception. If
> you catch an exception just to log it, you generally should also
> reraise it so that something further up the call chain has the
> opportunity to handle it.

How do we reraise the exception in python ,  I have used raise not
sure how to reraise the exception

>
>> def run_cmd_and_verify(cmd, timeout=1000):
>> try:
>> pdb.set_trace()
>> out, err, ret = run(cmd, timeout=timeout)
>
> What is "run"? It's imported like a module above, but here you're
> using it like a function.

Sorry run is a function  which was imported from a library , the
function had to be  # from utility import run ,

>> assert ret ==0,"ERROR (ret %d): " \
>> " \nout: %s\nerr: %s\n" % (ret, out, err)
>> except Exception as e:
>> print("Failed to run %s got %s" % (cmd, e))
>> return False
>> return True
>>
>> def prep_host():
>> """
>> Prepare clustering
>> """
>> for cmd in ["ls -al",
>> "touch /tmp/file1",
>> "mkdir /tmp/dir1"]:
>> try:
>> if not run_cmd_and_verify(cmd, timeout=3600):
>> return False
>> except:
>
> What exceptions are you expecting this to catch? run_cmd_and_verify
> already catches any expected exceptions that it raises.

This is a wrapper for Popen it runs the command and returns stdout
,stderror and returncode .The only exception it can return is a time
out exception

Here is the modified buggy code ,  Can I have Try and except with Pass
, how do I modify the  try and expect in the pre-host ?

#!/usr/bin/env python

"""
bugging code
"""
import logging
from utility import run

def run_cmd_and_verify(cmd, timeout=1000):
try:
out, err, ret = run(cmd, timeout=timeout)
assert ret ==0,"ERROR (ret %d): " \
" \nout: %s\nerr: %s\n" % (ret, out, err)
except Exception as e:
logging.error("Failed to run %s got %s" % (cmd, e))
return False
return True

def prep_host():
"""
Prepare clustering
"""
for cmd in ["ls -al",
"touch /tmp/file1",
"mkdir /tmp/dir1"]:
try:
if not run_cmd_and_verify(cmd, timeout=3600):
logging.info("Preparing cluster failed ...")
return False
except:
pass
logging.info("Preparing Cluster.Done !!!")
return True


def main():
functions = [prep_host]
for func in functions:
try:
func()
except Exception as e:
logging.info(e)
return False
if __name__ == '__main__':
main()


Regards,
Gpal
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: common mistakes in this simple program

[Ian Kelly]

On Mon, Feb 29, 2016 at 8:18 AM, Ganesh Pal  wrote:
> Iam on python 2.6

Python 2.6 has been unsupported since October 2013. Among other
things, that means it is no longer receiving security updates like
more recent versions. Unless you have an extremely strong reason for
wanting to stay to Python 2.6, you should update your Python version.

> 1. usage of try- expect

try-except in every single function is a code smell. You should only
be using it where you're actually going to handle the exception. If
you catch an exception just to log it, you generally should also
reraise it so that something further up the call chain has the
opportunity to handle it.

> 2. Return of True/ False

If you're returning True to mean "This completed without exception"
and False to mean "This raised an exception", then the more Pythonic
thing to do would just be to let the exception propagate, giving the
caller the opportunity to catch, inspect, and handle the exception.

> """
> """

An empty docstring is pointless.

> import os
> import shlex
> import subprocess
> import sys
> import time
> import logging
> import run
> import pdb

Most of these are unused. Only import modules that your code is actually using.

> def run_cmd_and_verify(cmd, timeout=1000):
> try:
> pdb.set_trace()
> out, err, ret = run(cmd, timeout=timeout)

What is "run"? It's imported like a module above, but here you're
using it like a function.

> assert ret ==0,"ERROR (ret %d): " \
> " \nout: %s\nerr: %s\n" % (ret, out, err)
> except Exception as e:
> print("Failed to run %s got %s" % (cmd, e))
> return False
> return True
>
> def prep_host():
> """
> Prepare clustering
> """
> for cmd in ["ls -al",
> "touch /tmp/file1",
> "mkdir /tmp/dir1"]:
> try:
> if not run_cmd_and_verify(cmd, timeout=3600):
> return False
> except:

What exceptions are you expecting this to catch? run_cmd_and_verify
already catches any expected exceptions that it raises.

Also, you should almost never use a bare except like this. Use "except
Exception" instead. A bare except will catch things like
KeyboardInterrupt and SystemExit that should not be caught.

>   print("Error: While preparing cluster !!!")
>   return False
> print("Preparing Cluster.Done !!!")
> return True
>
>
> def main():
> functions = [prep_host]
> for func in functions:
> try:
> func()
> except Exception as e:

As above, what exceptions are you expecting here?

> print(e)
> return False
> if __name__ == '__main__':
> main()
-- 
https://mail.python.org/mailman/listinfo/python-list