Hi Régis,
Thanks for your fast reply.
From what I understand, otwrapy would do the same thing as
sobol.setBlockSize(int(ot.ResourceMap.Get("parallel-threads"))).
So your suggesting to do instead:
import otwrapy as otw
model = otw.Parallelizer(ot.PythonFunction(2, 400, func), n_cpus=10)
If so, what is the benefit of doing it?
This will only be useful when doing sobol.getFirstOrderIndices(0).
What I am trying to do is to perform several analysis as I have a functional
output:
model = ot.PythonFunction(2, 400, func)
sobol = ot.SensitivityAnalysis(sample1, sample2, model)
sobol.setBlockSize(int(ot.ResourceMap.Get("parallel-threads")))
indices = [[], [], []]
for i in range(400):
indices[1].append(np.array(sobol.getFirstOrderIndices(i)))
indices[2].append(np.array(sobol.getTotalOrderIndices(i)))
This work but I want the for loop to be parallel so I tried:
from pathos.multiprocessing import ProcessingPool, cpu_count
model = ot.PythonFunction(2, 400, func)
sobol = ot.SensitivityAnalysis(sample1, sample2, sobol_model)
sobol.setBlockSize(int(ot.ResourceMap.Get("parallel-threads")))
def map_indices(i):
first = np.array(sobol.getFirstOrderIndices(i))
total = np.array(sobol.getTotalOrderIndices(i))
return first, total
pool = ProcessingPool(cpu_count())
results = pool.imap(map_indices, range(400))
first = np.empty(400)
total = np.empty(400)
for i in range(400):
first[i], total[i] = results.next()
But in order for this to work, the map_indice function has to be pickled (here
it uses dill that can serialize close to everything).
Hence the error I get.
Thanks again,
Pamphile ROY
----- Mail original -----
De: "regis lebrun" <[email protected]>
À: "Pamphile ROY" <[email protected]>, "users" <[email protected]>
Envoyé: Samedi 22 Octobre 2016 12:48:29
Objet: Re: [ot-users] Pickling and parallelism
Hi,
I understand that you have a model that has been interfaced with OpenTURNS
using the OpenTURNSPythonFunction class and you want to perform sensitivity
analysis using the SensitivityAnalysis class, and you would like to benefit
from some parallelism in the execution of the analysis.
The correct way to do that is to provide the _exec_sample method. In this
method, you are free to use any multithreading/multiprocessing capability you
want. You may consider either otwrapy (http://felipeam86.github.io/otwrapy/) or
one of the solutions proposed here:
http://openturns.github.io/developer_guide/wrapper_development.html or your
favorite tool. Then, you will get rid of the GIL.
I cannot help for the second point, as it is far beyond my knowledge on python
and serialization.
Cheers
Régis LEBRUN
>________________________________
> De : Pamphile ROY <[email protected]>
>À : [email protected]
>Envoyé le : Vendredi 21 octobre 2016 21h56
>Objet : [ot-users] Pickling and parallelism
>
>
>
>Hi,
>
>
>I would have 2 questions:
>
>
>1. I have seen that the class (and others) allows some multithreading. From my
>understanding, it is based on TBB and only multithreads the tasks.
>Thus it is concerned by the GIL. Is there an automatic set up like for
>multithreading but for multiprocessing instead? Any advice?
>
>
>2. Using pathos for multiprocessing, I am trying to dump an instance of
>SensitivityAnalysis but I cannot get it to work even with dill.
>For information, I am running under macOS sierra and this is OT 1.6 (maybe it
>is coming from here... I am going to upgrade but it means a refactoring on my
>side).
>Here is the following traceback:
>
>
>sobol = ot.SensitivityAnalysis(sample1, sample2, sobol_model)
>_f = dill.dumps(sobol)
>
>File
>"/Users/Pamphile/.virtualenvs/jpod/lib/python2.7/site-packages/dill/dill.py",
>line 243, in dumps
>dump(obj, file, protocol, byref, fmode, recurse)#, strictio)
>File
>"/Users/Pamphile/.virtualenvs/jpod/lib/python2.7/site-packages/dill/dill.py",
>line 236, in dump
>pik.dump(obj)
>File
>"/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py",
> line 224, in dump
>self.save(obj)
>File
>"/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py",
> line 306, in save
>rv = reduce(self.proto)
>File
>"/Applications/OpenTURNS/openturns/lib/python2.7/site-packages/openturns/common.py",
> line 258, in Object___getstate__
>study.add('instance', self)
>File
>"/Applications/OpenTURNS/openturns/lib/python2.7/site-packages/openturns/common.py",
> line 688, in add
>return _common.Study_add(self, *args)
>NotImplementedError: Wrong number or type of arguments for overloaded function
>'Study_add'.
>Possible C/C++ prototypes are:
>OT::Study::add(OT::InterfaceObject const &)
>OT::Study::add(OT::String const &,OT::InterfaceObject const &,OT::Bool)
>OT::Study::add(OT::String const &,OT::InterfaceObject const &)
>OT::Study::add(OT::PersistentObject const &)
>OT::Study::add(OT::String const &,OT::PersistentObject const &,OT::Bool)
>OT::Study::add(OT::String const &,OT::PersistentObject const &)
>
>
>Thank you for your help!
>
>
>
>
>Pamphile
>_______________________________________________
>OpenTURNS users mailing list
>[email protected]
>http://openturns.org/mailman/listinfo/users
>
>
>
_______________________________________________
OpenTURNS users mailing list
[email protected]
http://openturns.org/mailman/listinfo/users
