Re: percent faster than format()? (was: Re: optomizations)

2013-04-23 Thread Chris Angelico
On Wed, Apr 24, 2013 at 12:36 AM, Steven D'Aprano wrote: > # Using Python 3.3. > > py> from timeit import Timer > py> setup = "a = 'spam'; b = 'ham'; c = 'eggs'" > py> t1 = Timer("'%s, %s and %s for breakfast' % (a, b, c)", setup) > py> t2 = Timer("'{}, {} and {} for breakfast'.format(a, b, c)", s

Re: percent faster than format()? (was: Re: optomizations)

2013-04-23 Thread Steven D'Aprano
On Tue, 23 Apr 2013 09:46:53 +0200, Ulrich Eckhardt wrote: > Am 23.04.2013 06:00, schrieb Steven D'Aprano: >> If it comes down to micro-optimizations to shave a few microseconds >> off, consider using string % formatting rather than the format method. > > Why? I don't see any obvious difference b

Re: optomizations

2013-04-23 Thread Chris Angelico
On Tue, Apr 23, 2013 at 11:53 AM, Roy Smith wrote: > In article , > Rodrick Brown wrote: > >> I would like some feedback on possible solutions to make this script run >> faster. > > If I had to guess, I would think this stuff: > >> line = line.replace('mediacdn.xxx.com', 'med

Re: percent faster than format()? (was: Re: optomizations)

2013-04-23 Thread Chris “Kwpolska” Warrick
On Tue, Apr 23, 2013 at 9:46 AM, Ulrich Eckhardt wrote: > Am 23.04.2013 06:00, schrieb Steven D'Aprano: >> >> If it comes down to micro-optimizations to shave a few microseconds off, >> consider using string % formatting rather than the format method. > > > Why? I don't see any obvious difference

percent faster than format()? (was: Re: optomizations)

2013-04-23 Thread Ulrich Eckhardt
Am 23.04.2013 06:00, schrieb Steven D'Aprano: If it comes down to micro-optimizations to shave a few microseconds off, consider using string % formatting rather than the format method. Why? I don't see any obvious difference between the two... Greetings! Uli -- http://mail.python.org/mailma

Re: optomizations

2013-04-22 Thread Steven D'Aprano
On Tue, 23 Apr 2013 00:20:59 -0400, Rodrick Brown wrote: > Got a doc or URL I have no experience working with python profilers. https://duckduckgo.com/html/?q=python%20profiler This is also good: http://pymotw.com/2/profile/ -- Steven -- http://mail.python.org/mailman/listinfo/python-li

Re: optomizations

2013-04-22 Thread Rodrick Brown
On Apr 22, 2013, at 11:18 PM, Dan Stromberg wrote: On Mon, Apr 22, 2013 at 6:53 PM, Roy Smith wrote: > > So, my real advice to you is to fire up the profiler and see what it > says. I agree. Fire up a line-oriented profiler and only then start trying to improve the hot spots. Got a doc o

Re: optomizations

2013-04-22 Thread Chris Angelico
On Tue, Apr 23, 2013 at 2:00 PM, Steven D'Aprano wrote: > Also, you're searching the file system *for every line* in the input > file. Pull this outside of the loop and have it run once: > > if not os.path.exists(outdir): > os.makedirs(outdir) > > Likewi

Re: optomizations

2013-04-22 Thread Steven D'Aprano
On Mon, 22 Apr 2013 21:19:23 -0400, Rodrick Brown wrote: > I would like some feedback on possible solutions to make this script run > faster. > The system is pegged at 100% CPU and it takes a long time to complete. Have you profiled the app to see where it is spending all its time? What does "a

Re: optomizations

2013-04-22 Thread Dan Stromberg
On Mon, Apr 22, 2013 at 6:53 PM, Roy Smith wrote: > > So, my real advice to you is to fire up the profiler and see what it > says. I agree. Fire up a line-oriented profiler and only then start trying to improve the hot spots. -- http://mail.python.org/mailman/listinfo/python-list

Re: optomizations

2013-04-22 Thread MRAB
On 23/04/2013 02:19, Rodrick Brown wrote: I would like some feedback on possible solutions to make this script run faster. The system is pegged at 100% CPU and it takes a long time to complete. #!/usr/bin/env python import gzip import re import os import sys from datetime import datetime impor

Re: optomizations

2013-04-22 Thread Roy Smith
In article , Rodrick Brown wrote: > I would like some feedback on possible solutions to make this script run > faster. If I had to guess, I would think this stuff: > line = line.replace('mediacdn.xxx.com', 'media.xxx.com') > line = line.replace('staticcd

Re: optomizations

2013-04-22 Thread Chris Angelico
On Tue, Apr 23, 2013 at 11:19 AM, Rodrick Brown wrote: > with gzip.open(args.inputfile) as datafile: > for line in datafile: > outfile = '{}{}{}_combined.log'.format(dateobj.year, > dateobj.month, dateobj.day) > outdir = (args.out

optomizations

2013-04-22 Thread Rodrick Brown
I would like some feedback on possible solutions to make this script run faster. The system is pegged at 100% CPU and it takes a long time to complete. #!/usr/bin/env python import gzip import re import os import sys from datetime import datetime import argparse if __name__ == '__main__': p