Re: [Tutor] Multiprocessing with many input input parameters
On Fri, Jul 12, 2019 at 11:53:16AM +, Shall, Sydney via Tutor wrote:
> Thanks Mike,
>
> But I am still not clear.
Neither is your question.
> do I write:
>
> def f([x,y,z]) ?
> How exactly do one write the function and how does one ensure that each
> positional argument is accounted for.
Is your function taking three seperate arguments, or a single argument
that must be a list of exactly three items?
(1) Three seperate arguments:
def function(a, b, c):
# Write the body of the function.
...
That's all you need do, as the interpreter will ensure it is only called
with three arguments.
(2) One list argument with three items:
def function(alist):
if isinstance(alist, list):
if len(alist) < 3:
raise ValueError("too few arguments in alist")
elif len(alist) > 3:
raise ValueError("too many arguments in alist")
else:
a, b, c = alist # Unpack the three items from the list.
# Write the body of the function here.
...
else:
raise TypeError('expected a list')
Python will ensure your function is only called with a single argument.
The rest is up to you.
--
Steven
___
Tutor maillist - Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Multiprocessing with many input input parameters
Right, I meant tuple, not list.
a = ('A string')
b = ('A List Member',)
print(a[0])
print(b[0])
The output for this is:
A
A List Member
@mike
-Original Message-
From: Cameron Simpson
Sent: Friday, July 12, 2019 7:59 PM
To: Mike Barnett
Cc: Shall, Sydney ; tutor@python.org
Subject: Re: [Tutor] Multiprocessing with many input input parameters
On 11Jul2019 15:40, Mike Barnett wrote:
>If you're passing parameters as a list, then you need a "," at the end of the
>items. Otherwise if you have something like a string as the only item, the
>list will be the string.
>
>list_with_one_item = ['item one',]
Actually, this isn't true.
This is a one element list, no trailing coma required:
[5]
Mike has probably confused this with tuples. Because tuples are delineated with
parentheses, there is ambiguity between a tuple's parentheses and normal "group
these terms together" parentheses.
So:
x = 5 + 4 * (9 + 7)
Here we just have parentheses causing the assignment "9 + 7" to occur before
the multiplication by 4. And this is also legal:
x = 5 + 4 * (9)
where the parentheses don't add anything special in terma of behaviour.
Here is a 2 element tuple:
(9, 7)
How does one write a one element tuple? Like this:
(9,)
Here the trailing comma is _required_ to syntacticly indicate that we intend a
1 element tuple instead of a plain "9 in parentheses") as in the earlier
assignment statement.
I'm not sure any of this is relevant to Sydney's question though.
Cheers,
Cameron Simpson
___
Tutor maillist - Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Multiprocessing with many input input parameters
On 7/12/19 5:53 AM, Shall, Sydney via Tutor wrote:
> Thanks Mike,
>
> But I am still not clear.
>
> do I write:
>
> def f([x,y,z]) ?
> How exactly do one write the function and how does one ensure that each
> positional argument is accounted for.
The concept of packing will be useful, you can use the * operator to
pack and unpack. A trivial example to get you started:
>>> a = [1, 2, 3, 4]
>>> print(a)
[1, 2, 3, 4]
>>> print(*a)
1 2 3 4
In the first print we print the list, in the second we print the result
of unpacking the list - you see it's now four numbers rather than one
list of four numbers.
In a function definition you can pack with the * operator:
>>> def f(*args):
... print(type(args))
... print(len(args))
... print(args)
...
>>>
>>>
>>> f(1, 2, 3, 4)
4
(1, 2, 3, 4)
Here we called the function with four arguments, but it received those
packed into the one argument args, which is a tuple of length 4.
Python folk conventionally name the argument which packs the positional
args that way - *args - but the name "args" has no magic, its
familiarity just aids in recognition. By packing your positional args
you don't error out if you're not called with the exact number you
expect (or if you want to accept differing numbers of args), and then
you can do what you need to with what you get.
The same packing concept works for dictionaries as well, here the
operator is **.
>>> def fun(a, b, c):
... print(a, b, c)
...
>>> d = {'a':2, 'b':4, 'c':10}
>>> fun(**d)
2 4 10
What happened here is in unpacking, the keys in the dict are matched up
with the names of the function parameters, and the values for those keys
are passed as the parameters. If your dict doesn't match, it fails:
>>> d = {'a':2, 'b':4, 'd':10}
>>> fun(**d)
Traceback (most recent call last):
File "", line 1, in
TypeError: fun() got an unexpected keyword argument 'd'
Dictionary unpacking looks like:
>>> def fun(**kwargs):
... print(f"{kwargs}")
...
>>>
>>> fun(a=1, b=2, c=3)
{'a': 1, 'b': 2, 'c': 3}
again the name 'kwargs' is just convention.
There are rules for how to mix regular positional args, unpacked
positional args (or varargs), and keyword ares but don't want to go on
forever here...
___
Tutor maillist - Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Multiprocessing with many input input parameters
Hi Sydney,
On Wed, 10 Jul 2019 at 16:45, Shall, Sydney via Tutor wrote:
>
> I am a relative beginner.
>
> My program models cell reproduction. I have written a program that models
> this and it works.
>
> Now I want to model a tissue with several types of cells. I did this by
> simply rerunning the program with different inputs (cell characteristics).
> But now I want to send and receive signals between the cells in each
> population. This requires some sort of concurrent processing with halts at
> appropriate points to pass and receive signals.
You say that this "requires some sort of concurrent processing" but I
think that you are mistaken there. I have a lot of experience in
mathematical modelling and dynamic simulation including some
multi-cell models and I have never personally come across a situation
where any significant benefit could be obtained from using concurrent
processing for different parts of a single simulation. Those
situations do exist but you haven't said anything to make me think
that yours is an exceptional case.
A simple way to do this (not the only way!) is something like:
# Some data structure that stores which cells are sending messages
messages_from_cells = {}
for t in range(num_timesteps):
# Calculate new state of all cells based only on the old states of
all cells and messages.
new_cells = {}
for c in range(len(cells)):
new_cells[c] = update_cell(cells[c], messages_from_cells)
# Update all cells synchronously:
cells = new_cells
# Update messages based on new cell states:
for c in range(len(cells)):
messages_from_cells = update_messages(cells[c], messages_from_cells)
You just need to figure out a data structure (I've suggested a dict
above) that would somehow store what messages are being passed between
which cells. You can update each cell based on the current messages
and then update the messages ready for the next timestep.
Concurrent execution is not needed: I have simulated concurrency by
using two separate loops over the cells. The result is as if each cell
was updated concurrently. Another approach is that at each timestep
you can choose a cell randomly and update that one keeping all the
others constant. It really depends what kind of model you are using.
In a simulation context like this there are two different reasons why
you might conceivably want to use concurrent execution:
1. Your simulations are CPU-bound and are slow and you need to make
them run faster by using more cores.
2. Your simulation needs more memory then an individual computer has
and you need to run it over a cluster of many computers.
Python's multiprocessing module can help with the first problem: it
can theoretically make your simulations run faster. However it is hard
to actually achieve any speedup that way. Most likely there are other
ways to make your code run faster that are easier than using
concurrent execution and can deliver bigger gains. Multiprocessing
used well might make your code 10x faster but I will bet that there
are easier ways to make your code 100x faster.
Multiprocessing makes the second problem worse: it actually uses more
memory on each computer. To solve problem 2 is very hard but can be
done. I don't think either problem applies to you though.
There is a situation where I have used multiprocessing to make
simulations faster. In practice I rarely want to do just one
simulation; I want to do thousands with different parameters or
because they are stochastic and I want to average them. Running these
thousands of simulations can be made faster easily with
multiprocessing because it is an "embarrassingly parallel" problem.
You need to get your simulations working without multiprocessing first
though. This is a much easier way to solve problem 1 (in so far as
using more cores can help).
Side note: you've misspelt Professor in your email signature:
> Prodessor. Sydney Shall
--
Oscar
___
Tutor maillist - Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Multiprocessing with many input input parameters
On Sat, Jul 13, 2019 at 09:59:16AM +1000, Cameron Simpson wrote: > Mike has probably confused this with tuples. Because tuples are > delineated with parentheses, there is ambiguity between a tuple's > parentheses and normal "group these terms together" parentheses. There are no "tuple parentheses". Tuples are created by the *comma*, not the parens. The only exception is the empty tuple, since you can't have a comma on its own. x = ()# Zero item tuple. x = 1,# Single item tuple. x = 1, 2 # Two item tuple. Any time you have a tuple, you only need to put parens around it to dismbiguate it from the surrounding syntax: x = 1, 2, (3, 4, 5), 6 # Tuple containing a tuple. function(0, 1, (2, 3), 4) # Tuple as argument to a function. or just to make it more clear to the human reader. > Here is a 2 element tuple: > > (9, 7) > > How does one write a one element tuple? Like this: > > (9,) To be clear, in both cases you could drop the parentheses and still get a tuple: 9, 7 9, provided that wasn't in a context where the comma was interpreted as something with higher syntactic precedence, such as a function call: func(9, 7)# Two integer arguments, not one tuple argument. func((9, 7)) # One tuple argument. -- Steven ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Multiprocessing with many input input parameters
On 11Jul2019 15:40, Mike Barnett wrote: If you're passing parameters as a list, then you need a "," at the end of the items. Otherwise if you have something like a string as the only item, the list will be the string. list_with_one_item = ['item one',] Actually, this isn't true. This is a one element list, no trailing coma required: [5] Mike has probably confused this with tuples. Because tuples are delineated with parentheses, there is ambiguity between a tuple's parentheses and normal "group these terms together" parentheses. So: x = 5 + 4 * (9 + 7) Here we just have parentheses causing the assignment "9 + 7" to occur before the multiplication by 4. And this is also legal: x = 5 + 4 * (9) where the parentheses don't add anything special in terma of behaviour. Here is a 2 element tuple: (9, 7) How does one write a one element tuple? Like this: (9,) Here the trailing comma is _required_ to syntacticly indicate that we intend a 1 element tuple instead of a plain "9 in parentheses") as in the earlier assignment statement. I'm not sure any of this is relevant to Sydney's question though. Cheers, Cameron Simpson ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Multiprocessing with many input input parameters
Thanks Mike, But I am still not clear. do I write: def f([x,y,z]) ? How exactly do one write the function and how does one ensure that each positional argument is accounted for. Dr. Sydney Shall Department of Haematological Medicine King's College London 123 Coldharbour Lane London SE5 9NU ENGLAND E-Mail: sydney.shall (Correspondents outside the College should add @KCL.AC.UK) TEL: +44 (0)208 48 59 01 From: Mike Barnett Sent: 11 July 2019 16:40 To: Shall, Sydney Cc: tutor@python.org Subject: RE: [Tutor] Multiprocessing with many input input parameters If you're passing parameters as a list, then you need a "," at the end of the items. Otherwise if you have something like a string as the only item, the list will be the string. list_with_one_item = ['item one',] @mike -Original Message- From: Shall, Sydney Sent: Wednesday, July 10, 2019 11:44 AM To: tutor@python.org Subject: [Tutor] Multiprocessing with many input input parameters I am using MAC OS X 10.14.5 on a MAC iBook I use Python 3.7.0 from Anaconda, with Spyder 3.3.3 I am a relative beginner. My program models cell reproduction. I have written a program that models this and it works. Now I want to model a tissue with several types of cells. I did this by simply rerunning the program with different inputs (cell characteristics). But now I want to send and receive signals between the cells in each population. This requires some sort of concurrent processing with halts at appropriate points to pass and receive signals. I thought to use multiprocessing. I have read the documentation and reproduced the models in the docs. But I cannot figure out how to feed in the data for multiple parameters. I have tried using Pool and it works fine, but I can only get it to accept 1 input parameter, although multiple data inputs with one parameter works nicely. So, my questions are; 1. Is multiprocessing the suitable choice. 2. if yes, how does one write a function with multiple input parameters. Thank s in advance. Sydney Prodessor. Sydney Shall Department of Haematological Medicine King's College London 123 Coldharbour Lane London SE5 9NU ENGLAND E-Mail: sydney.shall (Correspondents outside the College should add @KCL.AC.UK) TEL: +44 (0)208 48 59 01 ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Multiprocessing with many input input parameters
On 12/07/2019 01:51, DL Neil wrote: > older articles! We haven't discussed hardware. Most modern PC CPUs offer > multiple "cores". Assuming (say) four cores, asyncio is capable of > running up to four processes concurrently - realising attendant > acceleration of the entirety. Just to pick up on this point because I often see it being cited. The number of concurrent processes running to achieve performance gain is only very loosely tied to the number of cores. We ran concurrent processes for many years before multi-core processors were invented with significant gains. Indeed any modern computer runs hundreds of "concurrent" processes on a small umber of cores and the OS switches between them. What the number of cores affects is the number of processes actually executing at the same time. If you just want to chunk up the processing of a large amount of data and run the exact same code multiple times then there is no point in having more than the number of cores. But if your concurrent processes are doing different tasks on different data then the number of cores is basically irrelevant. And especially if they are performing any kind of I/O operations since they are likely to be parked by the OS for most of the time anyway. Of course, there is a point where creating extra processes becomes counter-effective since that is an expensive operation, and especially if the process will be very short lived or only execute for tiny lengths of time (such as handling a network event by passing it on to some other process). But for most real world uses of multiprocessing the number of cores is not a major factor in deciding how many processes to run. I certainly would not hesitate to run 10xN where N is the number of cores. Beyond that you might need to think carefully. In Sydney's scenario is sounds like the processes are different and explicitly halt to perform I/O so the cores issue should not be a problem. -- Alan G Author of the Learn to Program web site http://www.alan-g.me.uk/ http://www.amazon.com/author/alan_gauld Follow my photo-blog on Flickr at: http://www.flickr.com/photos/alangauldphotos ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Multiprocessing with many input input parameters
Sydney, 1 There have been many projects to look at cells, division, multiplication, ... It is worth researching the Python eco-system in the expectation of saving yourself time and effort! 2 The latest releases of Python (such as you quote) offer updated asyncio module(s) for multiprocessing, ie be careful if you are reading older articles! We haven't discussed hardware. Most modern PC CPUs offer multiple "cores". Assuming (say) four cores, asyncio is capable of running up to four processes concurrently - realising attendant acceleration of the entirety. (admittedly, I tend to limit my ambitions to number_of_cores - 1) On 12/07/19 3:40 AM, Mike Barnett wrote: If you're passing parameters as a list, then you need a "," at the end of the items. Otherwise if you have something like a string as the only item, the list will be the string. list_with_one_item = ['item one',] @mike -Original Message- From: Shall, Sydney Sent: Wednesday, July 10, 2019 11:44 AM To: tutor@python.org Subject: [Tutor] Multiprocessing with many input input parameters I am using MAC OS X 10.14.5 on a MAC iBook I use Python 3.7.0 from Anaconda, with Spyder 3.3.3 I am a relative beginner. My program models cell reproduction. I have written a program that models this and it works. Now I want to model a tissue with several types of cells. I did this by simply rerunning the program with different inputs (cell characteristics). But now I want to send and receive signals between the cells in each population. This requires some sort of concurrent processing with halts at appropriate points to pass and receive signals. I thought to use multiprocessing. I have read the documentation and reproduced the models in the docs. But I cannot figure out how to feed in the data for multiple parameters. I have tried using Pool and it works fine, but I can only get it to accept 1 input parameter, although multiple data inputs with one parameter works nicely. So, my questions are; 1. Is multiprocessing the suitable choice. 2. if yes, how does one write a function with multiple input parameters. Thank s in advance. Sydney Prodessor. Sydney Shall Department of Haematological Medicine King's College London 123 Coldharbour Lane London SE5 9NU ENGLAND E-Mail: sydney.shall (Correspondents outside the College should add @KCL.AC.UK) TEL: +44 (0)208 48 59 01 ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor -- Regards =dn ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Multiprocessing with many input input parameters
If you're passing parameters as a list, then you need a "," at the end of the items. Otherwise if you have something like a string as the only item, the list will be the string. list_with_one_item = ['item one',] @mike -Original Message- From: Shall, Sydney Sent: Wednesday, July 10, 2019 11:44 AM To: tutor@python.org Subject: [Tutor] Multiprocessing with many input input parameters I am using MAC OS X 10.14.5 on a MAC iBook I use Python 3.7.0 from Anaconda, with Spyder 3.3.3 I am a relative beginner. My program models cell reproduction. I have written a program that models this and it works. Now I want to model a tissue with several types of cells. I did this by simply rerunning the program with different inputs (cell characteristics). But now I want to send and receive signals between the cells in each population. This requires some sort of concurrent processing with halts at appropriate points to pass and receive signals. I thought to use multiprocessing. I have read the documentation and reproduced the models in the docs. But I cannot figure out how to feed in the data for multiple parameters. I have tried using Pool and it works fine, but I can only get it to accept 1 input parameter, although multiple data inputs with one parameter works nicely. So, my questions are; 1. Is multiprocessing the suitable choice. 2. if yes, how does one write a function with multiple input parameters. Thank s in advance. Sydney Prodessor. Sydney Shall Department of Haematological Medicine King's College London 123 Coldharbour Lane London SE5 9NU ENGLAND E-Mail: sydney.shall (Correspondents outside the College should add @KCL.AC.UK) TEL: +44 (0)208 48 59 01 ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
