Iterating over files of a huge directory

2012-12-17 Thread Gilles Lenfant
Hi, I have googled but did not find an efficient solution to my problem. My customer provides a directory with a hge list of files (flat, potentially 10+) and I cannot reasonably use os.listdir(this_path) unless creating a big memory footprint. So I'm looking for an iterator that

Re: Iterating over files of a huge directory

2012-12-17 Thread Chris Angelico
On Tue, Dec 18, 2012 at 2:28 AM, Gilles Lenfant gilles.lenf...@gmail.com wrote: Hi, I have googled but did not find an efficient solution to my problem. My customer provides a directory with a hge list of files (flat, potentially 10+) and I cannot reasonably use

Re: Iterating over files of a huge directory

2012-12-17 Thread Tim Golden
On 17/12/2012 15:41, Chris Angelico wrote: On Tue, Dec 18, 2012 at 2:28 AM, Gilles Lenfant gilles.lenf...@gmail.com wrote: Hi, I have googled but did not find an efficient solution to my problem. My customer provides a directory with a hge list of files (flat, potentially 10+) and

Re: Iterating over files of a huge directory

2012-12-17 Thread Oscar Benjamin
On 17 December 2012 15:28, Gilles Lenfant gilles.lenf...@gmail.com wrote: I have googled but did not find an efficient solution to my problem. My customer provides a directory with a hge list of files (flat, potentially 10+) and I cannot reasonably use os.listdir(this_path) unless

Re: Iterating over files of a huge directory

2012-12-17 Thread marduk
On Mon, Dec 17, 2012, at 10:28 AM, Gilles Lenfant wrote: Hi, I have googled but did not find an efficient solution to my problem. My customer provides a directory with a hge list of files (flat, potentially 10+) and I cannot reasonably use os.listdir(this_path) unless creating a

Re: Iterating over files of a huge directory

2012-12-17 Thread Gilles Lenfant
Le lundi 17 décembre 2012 16:52:19 UTC+1, Oscar Benjamin a écrit : On 17 December 2012 15:28, Gilles Lenfant ... wrote: In the last couple of months there has been a lot of discussion (on python-list or python-dev - not sure) about creating a library to more efficiently iterate over

Re: Iterating over files of a huge directory

2012-12-17 Thread Paul Rudin
Chris Angelico ros...@gmail.com writes: On Tue, Dec 18, 2012 at 2:28 AM, Gilles Lenfant gilles.lenf...@gmail.com wrote: Hi, I have googled but did not find an efficient solution to my problem. My customer provides a directory with a hge list of files (flat, potentially 10+) and I

Re: Iterating over files of a huge directory

2012-12-17 Thread MRAB
On 2012-12-17 17:27, Paul Rudin wrote: Chris Angelico ros...@gmail.com writes: On Tue, Dec 18, 2012 at 2:28 AM, Gilles Lenfant gilles.lenf...@gmail.com wrote: Hi, I have googled but did not find an efficient solution to my problem. My customer provides a directory with a hge list of

Re: Re: Iterating over files of a huge directory

2012-12-17 Thread Evan Driscoll
On 12/17/2012 09:52 AM, Oscar Benjamin wrote: In the last couple of months there has been a lot of discussion (on python-list or python-dev - not sure) about creating a library to more efficiently iterate over the files in a directory. The result so far is this library on github:

Re: Re: Iterating over files of a huge directory

2012-12-17 Thread Oscar Benjamin
On 17 December 2012 18:40, Evan Driscoll drisc...@cs.wisc.edu wrote: On 12/17/2012 09:52 AM, Oscar Benjamin wrote: https://github.com/benhoyt/betterwalk This is very useful to know about; thanks. I actually wrote something very similar on my own (I wanted to get information about whether

Re: Iterating over files of a huge directory

2012-12-17 Thread Evan Driscoll
On 12/17/2012 01:50 PM, Oscar Benjamin wrote: On 17 December 2012 18:40, Evan Driscoll drisc...@cs.wisc.edu wrote: On 12/17/2012 09:52 AM, Oscar Benjamin wrote: https://github.com/benhoyt/betterwalk This is very useful to know about; thanks. I actually wrote something very similar on my own

Re: Iterating over files of a huge directory

2012-12-17 Thread Chris Angelico
On Tue, Dec 18, 2012 at 5:29 AM, MRAB pyt...@mrabarnett.plus.com wrote: Off topic Years ago I had to deal with an in-house application that was written using a certain database package. The package stored each predefined query in a separate file in the same directory. I found that if I

Re: Iterating over files of a huge directory

2012-12-17 Thread Terry Reedy
On 12/17/2012 10:28 AM, Gilles Lenfant wrote: Hi, I have googled but did not find an efficient solution to my problem. My customer provides a directory with a hge list of files (flat, potentially 10+) and I cannot reasonably use os.listdir(this_path) unless creating a big memory