Do you do it as separate process or thread.
There is https://wiki.python.org/moin/GlobalInterpreterLock
so you need to spawn many processes
Best regards,
Marek Mosiewicz
http://marekmosiewicz.pl
W dniu 23.05.2019 o 20:39, Bob van der Poel pisze:
I've got a short script
On Sun, May 26, 2019 at 11:05 AM Grant Edwards
wrote:
> On 2019-05-23, Chris Angelico wrote:
> > On Fri, May 24, 2019 at 5:37 AM Bob van der Poel
> wrote:
> >>
> >> I've got a short script that loops though a number of files and
> >> processes them one at a time. I had a bit of time today and
On Mon, May 27, 2019 at 4:24 AM Grant Edwards wrote:
>
> On 2019-05-26, Chris Angelico wrote:
>
> > Sometimes, the "simple" and "obvious" code, the part that clearly has
> > no bugs in it, is the part that has the problem. :)
>
> And in this case, the critical part of the code that was actually
On 2019-05-26, Chris Angelico wrote:
> Sometimes, the "simple" and "obvious" code, the part that clearly has
> no bugs in it, is the part that has the problem. :)
And in this case, the critical part of the code that was actually
serializing everything wasn't shown. One strives to post problem
On Mon, May 27, 2019 at 4:06 AM Grant Edwards wrote:
>
> On 2019-05-23, Chris Angelico wrote:
> > On Fri, May 24, 2019 at 5:37 AM Bob van der Poel wrote:
> >>
> >> I've got a short script that loops though a number of files and
> >> processes them one at a time. I had a bit of time today and
On 2019-05-23, Chris Angelico wrote:
> On Fri, May 24, 2019 at 5:37 AM Bob van der Poel wrote:
>>
>> I've got a short script that loops though a number of files and
>> processes them one at a time. I had a bit of time today and figured
>> I'd rewrite the script to process the files 4 at a time
On 24May2019 11:40, bvdp wrote:
Just got a 1 liner working with parallel. Super! All I ended up doing is:
parallel mma {} ::: *mma
+1
Glad to see a nice simple approach.
Cheers,
Cameron Simpson (formerly )
--
https://mail.python.org/mailman/listinfo/python-list
Just got a 1 liner working with parallel. Super! All I ended up doing is:
parallel mma {} ::: *mma
which whizzed through my files in less than 1/4 of the time of my
one-at-a-time script. (In case anyone is wondering, or cares, this is a
bunch of Musical Midi Accompaniment files:
On 5/23/19 6:32 PM, Cameron Simpson wrote:
On 23May2019 17:04, bvdp wrote:
Anyway, yes the problem is that I was naively using command.getoutput()
which blocks until the command is finished. So, of course, only one
process
was being run at one time! Bad me!
I guess I should be looking at
Ahh, 2 really excellent ideas! I'm reading about parallel right now. And, I
know how to use make, so I really should have thought of -j as well. Thanks
for the ideas.
On Fri, May 24, 2019 at 12:02 AM Christian Gollwitzer
wrote:
> Am 23.05.19 um 23:44 schrieb Paul Rubin:
> > Bob van der Poel
Am 23.05.19 um 23:44 schrieb Paul Rubin:
Bob van der Poel writes:
for i in range(0, len(filelist), CPU_COUNT):
for z in range(i, i+CPU_COUNT):
doit( filelist[z])
Write your program to just process one file, then use GNU Parallel
to run the program on your 1200 files, 6 at a
On 5/23/2019 2:39 PM, Bob van der Poel wrote:
I'm processing about 1200 files and my total duration is around 2 minutes.
A followup to my previous response, which has not shown up yet. The
python test suite is over 400 files. You might look at how
test.regrtest runs them in parallel when
On 5/23/2019 2:39 PM, Bob van der Poel wrote:
I've got a short script that loops though a number of files and processes
them one at a time. I had a bit of time today and figured I'd rewrite the
script to process the files 4 at a time by using 4 different instances of
python.
As others have
On 23May2019 17:04, bvdp wrote:
Anyway, yes the problem is that I was naively using command.getoutput()
which blocks until the command is finished. So, of course, only one process
was being run at one time! Bad me!
I guess I should be looking at subprocess.Popen(). Now, a more relevant
On Fri, May 24, 2019 at 10:48 AM MRAB wrote:
>
> On 2019-05-24 01:22, Chris Angelico wrote:
> > What I'd recommend is a thread pool. Broadly speaking, it would look
> > something like this:
> >
> > jobs = [...]
> >
> > def run_jobs():
> > while jobs:
> > try: job = jobs.pop()
> >
On 2019-05-24 01:22, Chris Angelico wrote:
On Fri, May 24, 2019 at 10:07 AM Bob van der Poel wrote:
Thanks all! The sound you are hearing is my head smacking against my hand!
Or is it my hand against my head?
Anyway, yes the problem is that I was naively using command.getoutput()
which
On Fri, May 24, 2019 at 10:07 AM Bob van der Poel wrote:
>
> Thanks all! The sound you are hearing is my head smacking against my hand!
> Or is it my hand against my head?
>
> Anyway, yes the problem is that I was naively using command.getoutput()
> which blocks until the command is finished. So,
Thanks all! The sound you are hearing is my head smacking against my hand!
Or is it my hand against my head?
Anyway, yes the problem is that I was naively using command.getoutput()
which blocks until the command is finished. So, of course, only one process
was being run at one time! Bad me!
I
On 2019-05-23 22:41, Avi Gross via Python-list wrote:
Bob,
As others have noted, you have not made it clear how what you are doing is
running "in parallel."
I have a similar need where I have thousands of folders and need to do an
analysis based on the contents of one at a time and have 8
Bob,
As others have noted, you have not made it clear how what you are doing is
running "in parallel."
I have a similar need where I have thousands of folders and need to do an
analysis based on the contents of one at a time and have 8 cores available
but the process may run for months if run
You really need to give more info on what you're doing in doit() to know what's
going on. Are you using subprocess, threading, multiprocessing, etc?
Going off of what you've put there those nested for loops are being run in the
1 main thread. If doit() kicks off a program and doesn't wait for
On Fri, May 24, 2019 at 5:37 AM Bob van der Poel wrote:
>
> I've got a short script that loops though a number of files and processes
> them one at a time. I had a bit of time today and figured I'd rewrite the
> script to process the files 4 at a time by using 4 different instances of
> python.
22 matches
Mail list logo
