Re: [Paraview] Connect ParaViewWeb to Catalyst

[Sebastien Jourdain]

Glad to ear that you managed to get something going. If you feel like
sharing your code, we might be able to provide some integration into
ParaView and/or ParaViewWeb proper.

Seb

On Sat, May 21, 2016 at 11:53 AM, Sven Kramer 
wrote:

> Thank you, Seb,
> it seems like I am getting it to work. There are still some difficulties
> because of my limited javascript experience, but with some quick
> workarounds I do have the Catalyst output displayed from ParaViewWeb. As
> you said, nothing else was required than wrapping the C functions in python.
>
> Thanks
> Sven
>
>
> 2016-05-20 18:59 GMT+02:00 Sebastien Jourdain <
> sebastien.jourd...@kitware.com>:
>
>> Hi Sven,
>>
>> I am not aware on any specific developer documentation for Catalyst on
>> the ParaView side (but I haven't worked on that subject for years),
>> although Utkarsh or Andy might know.
>>
>> Usually looking at the code and redoing it in the Python world is what is
>> required, but as I said, I'm not sure of what it means since I don't know
>> that code anymore. ;-)
>>
>> For the timeframe, I can't tell (meaning I don't know), but we can still
>> try to help you or implement the missing piece via support contract if that
>> make sense on your end.
>>
>> Thanks,
>>
>> Seb
>>
>> On Fri, May 20, 2016 at 10:46 AM, Sven Kramer 
>> wrote:
>>
>>> Hello Seb,
>>> glad to read your answer. So the problem is not only that I was unable
>>> to find the relevant information.
>>>
>>> You say, you have not (yet) written the required code. Does "not (yet)"
>>> mean that it is on the agenda maybe within the next half year? Then I could
>>> find a workaround and look forward to the next Paraview updates.
>>>
>>> Otherwise I would try to work on the implementation for connecting
>>> Paraviewweb to Catalyst output. Is there any developer documentation of
>>> Catalysts "specific socket connection"? Or some general Catalyst developer
>>> information? Those would be great to get started.
>>> My first attempt would be to "wrap" the functionality of
>>> Qt/ApplicationComponents/pqCatalystConnectReaction.cxx into pvserver and
>>> make it available to the web client. Or are the necessary steps much more
>>> involved?
>>>
>>> Thank you
>>> Sven
>>>
>>>
>>> 2016-05-20 15:42 GMT+02:00 Sebastien Jourdain <
>>> sebastien.jourd...@kitware.com>:
>>>
 Hi Sven,

 Live catalyst connection require a specific socket connection to the
 in-situ process with some special handling.
 We did not (yet) wrote the Python code to perform that specific action
 for ParaViewWeb so it could be triggered by the web client or configured
 via an argument at launch time.
 I don't see any reason why this couldn't work, but as for now, I also
 don't know an easy step to do it without spending time doing it myself.

 Seb

 On Thu, May 19, 2016 at 1:52 PM, Sven Kramer 
 wrote:

> Hello,
> I am looking for the recommended steps how to display an in-situ
> (live) Catalyst simulation through ParaViewWeb.
>
> After some efforts I am able to run ParaViewWeb successfully and I can
> view a Catalyst-enabled simulation in paraview.
>
> I hope, it is also possible to show the live Catalyst output via
> ParaViewWeb, but I cannot imagine how to achieve this. I tried starting a
> pvserver as mentioned at the end of
> https://blog.kitware.com/introduction-to-paraview-catalyst-live/ and
> giving "--ds-host localhost --ds-port 1" options to
> pv_web_visualizer.py, but it doesn't help. I can't share anything to be
> displayed on the web.
>
> Below you find the (not too detailed) steps I took to have a working
> solution for either Catalyst-enabled simulations or a working web
> visualizer:
>
> Test connection from Catalyst simulation to paraview:
> 1) Build and execute paraview from ParaView-v5.0.1-source.tar.gz (from
> http://www.paraview.org/download/)
> 2) Choose Catalyst->Connect in paraview
> 3) Run CFullExample from
> https://github.com/Kitware/ParaViewCatalystExampleCode
> 4) Pipeline from CFullExample is shown in paraview.
> Success!
>
> Test ParaViewWeb:
> 1) Download ParaViewData-5.0.0.tar.gz from
> http://www.paraview.org/download/, extract in /data
> 2) Run $ParaView_DIR/bin/pvpython
> $ParaView_DIR/lib/site-packages/paraview/web/pv_web_visualizer.py 
> --content
> $ParaView_DIR/www  --data-dir /data --port 8080
> 3) Open the Visualizer App http://localhost:8080/apps/Visualizer/
> Success!
>
> But how do I get Catalyst into the web visualizer? Is there any
> example how to combine them? Anybody who can suggest the required steps?
>
> Regards,
> Sven
>
>
> ___
> Powered by www.kitware.com
>
> Visit other Kitware open-source projects at
> 

Re: [Paraview] Memory leak with Catalyst

[Burlen Loring]

btw, we just had similar problem that was entirely explained by vtk 
writer attempting to gather a bunch of arrays to the root node. Utkarsh 
found a way to disable that behavior. I wonder if you are hitting the same.


On 5/20/2016 4:36 PM, Gallagher, Timothy P wrote:


Well... not going so well.


If I run a small simulation with massif and the Release build of 
paraview, I don't see any growing memory in time. I also get my VTK 
files and images.



If I run the same simulation linked against a Debug build with 
VTK_DEBUG_LEAKS on, there is a segfault inside one of the 
libvtkRendering libraries (it says libvtkRenderingOp but the rest of 
the name is cut off). If I take the image rendering out of my pipeline 
(comment out the RegisterView call) and leave the VTK file writing, it 
runs fine and doesn't report any leaks. I tried to get the segfault to 
drop a core file, but it won't do it.



I'm thinking that there is an issue in the rendering part causing the 
leak. I have no idea what it could be though -- this is the 4.4.0 
version, is anybody aware of any bug fixes between 4.4.0 and current 
versions that might be relevant?



Thanks for the help,


Tim






*From:* ParaView  on behalf of 
Gallagher, Timothy P 

*Sent:* Friday, May 20, 2016 4:17 PM
*To:* Burlen Loring; Andy Bauer
*Cc:* paraview@paraview.org
*Subject:* Re: [Paraview] Memory leak with Catalyst

Thanks for the advice Burlen -- I haven't used Massif before, so this 
is a good opportunity to give it a try.



The initialize/finalize each time approach doesn't work either. I am 
calling the Finalize() routine on the vtkCPProcessor class but then 
when it tries to initialize again, it throws:



Warning: In /data4/ParaView/VTK/Parallel/Core/vtkSocketController.cxx, 
line 50

vtkSocketController (0xa4342f0): Already initialized.


and then crashes. So, hopefully I can sort out the memory leak rather 
than trying to debug why a hacked fix doesn't work!



Tim




*From:* Burlen Loring 
*Sent:* Friday, May 20, 2016 3:46 PM
*To:* Gallagher, Timothy P; Andy Bauer
*Cc:* paraview@paraview.org
*Subject:* Re: [Paraview] Memory leak with Catalyst

VTK_DEBUG_LEAKS, although will catch some serious errors, will not 
catch all kinds of leaks. For example you can have leaks where data 
erroneously accumulates with each time step, but is deleted at program 
termination. VTK_DEBUG_LEAKS will not catch that kind of error. It's 
probably better to use massif to profile your code on a small one node 
run. There's an excellent tool called massif visualizer to aid in 
exploring the data generated.



$0.02


On 05/20/2016 11:56 AM, Gallagher, Timothy P wrote:


Hi Andy,


Thanks for the tips. I will get working on the VTK_DEBUG_LEAKS now 
and see what it turns up.



The initialize/finalize every time is definitely a hack and not for 
long-term use. But, sponsors want a report on Monday and we need to 
be able to visualize things for that -- the simulation is too big to 
write data files and post-process later. So I modified the code to do 
that for now until I can find a proper fix.



I'll let you know if I get stuck with the log file.


Thanks again,


Tim




*From:* Andy Bauer 
*Sent:* Friday, May 20, 2016 2:39 PM
*To:* Gallagher, Timothy P
*Cc:* paraview@paraview.org
*Subject:* Re: [Paraview] Memory leak with Catalyst
Hi Tim,

If you build Catalyst with VTK_DEBUG_LEAKS enabled it is pretty good 
at finding VTK objects that aren't deleted properly. You should be 
able to run this with a small amount of calls to Catalyst as well. If 
you try this and want help understanding the output (if an object 
like a vtkUnstructuredGrid is leaked it will often give a whole bunch 
of false leaks that the unstructured grid is just holding the 
references to), just share your output file and I can take a look at 
it to try and narrow down the culprit. That may be slightly easier to 
use than Valgrind.


Beyond this, you could maybe try the same run but without doing any 
Catalyst work to see what happens then. That may be a lot of compute 
cycles but I'm not sure how else to try and bisect where the memory 
leak is occurring.


Initializing and finalizing Catalyst every time you want output would 
probably work but I'd think there may be significant overhead doing 
it like this. Plus, it's not really solving the problem -- just 
avoiding it.


Best,
Andy


On Fri, May 20, 2016 at 12:57 PM, Gallagher, Timothy P 
> wrote:


Hi,


One of our users is running a very big simulation and writing out
images of two slices (two different views) every 1000 iterations
and writing out the 

Re: [Paraview] capability of ParaView, Catalyst in distributed computing environment ...

[Gallagher, Timothy P]

If you can control which communicator Paraview uses, you might want to look at 
using MPI_INTERCOMM_MERGE, which will take the union of the two disjoint sets 
described by an intercommunicator. So with that, you would create a new 
communicator that has all of your processors as members.

Paraview should be able to use that merged communicator seamlessly because it 
is a single communicator with all of the processor members attached. Meanwhile, 
your individual codes still use their own MPI_COMM_WORLD and the 
intercommunicator when they need to share information. 

That way, the codes do not need to change. Your adapter does the work of 
dealing with the merge process and the managing the merged communicator. 

I haven't tested anything like this of course. Just putting the idea out there.

Tim


From: u.utku.turunco...@be.itu.edu.tr 
Sent: Sunday, May 22, 2016 7:59 AM
To: Andy Bauer
Cc: Ufuk Utku Turuncoglu; Gallagher, Timothy P; paraview@paraview.org
Subject: Re: [Paraview] capability of ParaView,  Catalyst in distributed 
computing environment ...

Thanks for the information. Currently, i am working on two component case
and the initial results show that grid and data information belong to each
model component must be accessible by all the MPI processes (defined in
global MPI_COMM_WORLD) in adaptor side. This makes the implementation very
complex when the 2d decomposition configuration of both model components
(which run in a specific subset of processors) are considered. In this
case, it seems that the easiest way is to interpolate/ redistribute the
data of both components into common grid or creating new 2d decomposition
in adaptor. Another possibility might be to implement MPI sections
specific for each model component (basically having two distinct
MPI_COMM_WORLD inside of global one) to access grid and fields in the
adaptor side but in this case i am not sure ParaView could handle these
kind of information or not. Anyway, it seems that it is a challanging
problem and probably it would be good to have this feature. I'll keep to
continue to try different implementations to test different ideas and keep
you posted about it. In the mean time, if you have any other idea, let me
know.

Regards,

--ufuk

> It may be possible to do this with Catalyst. I would guess that nearly all
> of the complex work would need to be done in the adaptor to integrate this
> properly though.
>
> On Wed, May 18, 2016 at 11:17 AM,  wrote:
>
>> Yes, you are right. In this case, there will be two separate
>> MPI_COMM_WORLD. Plus, one that covers all the resources (let's say that
>> global MPI_COMM_WORLD). Actually, this kind of setup is very common for
>> multi-physics applications such as fluid-structure interaction. So, is
>> it
>> possible to tight this kind of environment with Catalyst? I am not
>> expert
>> about Catalyst but it seems that there might be a problem in the
>> rendering
>> stage even underlying grids and fields are defined without any problem.
>>
>> Regards,
>>
>> --ufuk
>>
>> > I'm not sure if this is exactly what the original user is referring
>> to,
>> > but it is possible to have two separate codes communicate using MPI
>> > through the dynamic processes in MPI-2. Essentially, one program
>> starts
>> up
>> > on N processors and begins running and gets an MPI_COMM_WORLD. It then
>> > spawns another executable on M different processors and that new
>> > executable will call MPI_INIT and also get its own MPI_COMM_WORLD. So
>> you
>> > have two, disjoint MPI_COMM_WORLD's that get linked together through a
>> > newly created intercommunicator.
>> >
>> >
>> > I've used this to couple a structural mechanics code to our fluid
>> dynamics
>> > solver for example. It sounds like that is similar to what is being
>> done
>> > here.
>> >
>> >
>> > How that would interact with coprocessing is beyond my knowledge
>> though.
>> > It does sound like an interesting problem and one I would be very
>> curious
>> > to find out the details.
>> >
>> >
>> > Tim
>> >
>> >
>> > 
>> > From: ParaView  on behalf of Andy Bauer
>> > 
>> > Sent: Wednesday, May 18, 2016 10:52 AM
>> > To: Ufuk Utku Turuncoglu (BE)
>> > Cc: paraview@paraview.org
>> > Subject: Re: [Paraview] capability of ParaView, Catalyst in
>> distributed
>> > computing environment ...
>> >
>> > Hi,
>> >
>> > I'm a bit confused. MPI_COMM_WORLD is the global communicator and as
>> far
>> > as I'm aware, can't be modified which means there can't be two
>> different
>> > communicators.
>> >
>> > Catalyst can be set to use a specific MPI communicator and that's been
>> > done by at least one code (Code_Saturne). I think they have a
>> multiphysics
>> > simulation as well.
>> >
>> > Cheers,
>> > Andy
>> >
>> > On Wed, May 18, 2016 at 5:22 AM, Ufuk Utku Turuncoglu (BE)
>> > 

Re: [Paraview] Problems about calculate the difference between two datasets

[Mathieu Westphal]

Hello

Python calculator is the way to go here

inputs[0].PointData['flux1'] - inputs[1].PointData['flux2']

Beware of the order of the input, they will be ordered randomly by
paraview, but you can reorder them wia the python shell, or modify your
python-calculator script.


Mathieu Westphal

On Sun, May 22, 2016 at 10:31 AM, Feng Jiang  wrote:

> ParaView newbie here ,and sorry to interrupt you for some problems i met.
>
> I have been using two vtk input files which named t1 and t2 for example,
> both input files contained only one dataset for CellData may called flux1
> and flux2.
>
> Exactly the structure and the data t1 and t2 got were same and only make a
> reflect from the surface of the center of x axis.
>
> And I want to make the CellData flux1 to subtrack the flux2.
> However I have tried Using the Group Datasets and the Calculator, or Using
> the Python Calculator or Programmable filter, and all the result I got is
> empty.
>
> The results I tried seems the ParaView only subtracked the data basic on
> the Cell ID.
>
> So is there any way to solve it or some wrong operate I made I wondered,
> or I must make the datasets reflect before I input ParaView ?
>
> Thanks for any guidance,
>
> Feng
>
>
>
> ___
> Powered by www.kitware.com
>
> Visit other Kitware open-source projects at
> http://www.kitware.com/opensource/opensource.html
>
> Please keep messages on-topic and check the ParaView Wiki at:
> http://paraview.org/Wiki/ParaView
>
> Search the list archives at: http://markmail.org/search/?q=ParaView
>
> Follow this link to subscribe/unsubscribe:
> http://public.kitware.com/mailman/listinfo/paraview
>
>
___
Powered by www.kitware.com

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Please keep messages on-topic and check the ParaView Wiki at: 
http://paraview.org/Wiki/ParaView

Search the list archives at: http://markmail.org/search/?q=ParaView

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/paraview

Re: [Paraview] capability of ParaView, Catalyst in distributed computing environment ...

[u . utku . turuncoglu]

Thanks for the information. Currently, i am working on two component case
and the initial results show that grid and data information belong to each
model component must be accessible by all the MPI processes (defined in
global MPI_COMM_WORLD) in adaptor side. This makes the implementation very
complex when the 2d decomposition configuration of both model components
(which run in a specific subset of processors) are considered. In this
case, it seems that the easiest way is to interpolate/ redistribute the
data of both components into common grid or creating new 2d decomposition
in adaptor. Another possibility might be to implement MPI sections
specific for each model component (basically having two distinct
MPI_COMM_WORLD inside of global one) to access grid and fields in the
adaptor side but in this case i am not sure ParaView could handle these
kind of information or not. Anyway, it seems that it is a challanging
problem and probably it would be good to have this feature. I'll keep to
continue to try different implementations to test different ideas and keep
you posted about it. In the mean time, if you have any other idea, let me
know.

Regards,

--ufuk

> It may be possible to do this with Catalyst. I would guess that nearly all
> of the complex work would need to be done in the adaptor to integrate this
> properly though.
>
> On Wed, May 18, 2016 at 11:17 AM,  wrote:
>
>> Yes, you are right. In this case, there will be two separate
>> MPI_COMM_WORLD. Plus, one that covers all the resources (let's say that
>> global MPI_COMM_WORLD). Actually, this kind of setup is very common for
>> multi-physics applications such as fluid-structure interaction. So, is
>> it
>> possible to tight this kind of environment with Catalyst? I am not
>> expert
>> about Catalyst but it seems that there might be a problem in the
>> rendering
>> stage even underlying grids and fields are defined without any problem.
>>
>> Regards,
>>
>> --ufuk
>>
>> > I'm not sure if this is exactly what the original user is referring
>> to,
>> > but it is possible to have two separate codes communicate using MPI
>> > through the dynamic processes in MPI-2. Essentially, one program
>> starts
>> up
>> > on N processors and begins running and gets an MPI_COMM_WORLD. It then
>> > spawns another executable on M different processors and that new
>> > executable will call MPI_INIT and also get its own MPI_COMM_WORLD. So
>> you
>> > have two, disjoint MPI_COMM_WORLD's that get linked together through a
>> > newly created intercommunicator.
>> >
>> >
>> > I've used this to couple a structural mechanics code to our fluid
>> dynamics
>> > solver for example. It sounds like that is similar to what is being
>> done
>> > here.
>> >
>> >
>> > How that would interact with coprocessing is beyond my knowledge
>> though.
>> > It does sound like an interesting problem and one I would be very
>> curious
>> > to find out the details.
>> >
>> >
>> > Tim
>> >
>> >
>> > 
>> > From: ParaView  on behalf of Andy Bauer
>> > 
>> > Sent: Wednesday, May 18, 2016 10:52 AM
>> > To: Ufuk Utku Turuncoglu (BE)
>> > Cc: paraview@paraview.org
>> > Subject: Re: [Paraview] capability of ParaView, Catalyst in
>> distributed
>> > computing environment ...
>> >
>> > Hi,
>> >
>> > I'm a bit confused. MPI_COMM_WORLD is the global communicator and as
>> far
>> > as I'm aware, can't be modified which means there can't be two
>> different
>> > communicators.
>> >
>> > Catalyst can be set to use a specific MPI communicator and that's been
>> > done by at least one code (Code_Saturne). I think they have a
>> multiphysics
>> > simulation as well.
>> >
>> > Cheers,
>> > Andy
>> >
>> > On Wed, May 18, 2016 at 5:22 AM, Ufuk Utku Turuncoglu (BE)
>> > > >>
>> > wrote:
>> > Hi All,
>> >
>> > I just wonder about the capability of ParaView, Catalyst in
>> distributed
>> > computing environment. I have little bit experience in in-situ
>> > visualization but it is hard for me to see the big picture at this
>> point.
>> > So, i decided to ask to the user list to get some suggestion from the
>> > experts. Hypothetically, lets assume that we have two simulation code
>> that
>> > are coupled together (i.e. fluid-structure interaction) and both of
>> them
>> > have their own MPI_COMM_WORLD and run on different processors (model1
>> runs
>> > on MPI rank 0,1,2,3 and model2 runs on 4,5,6,7). What is the correct
>> > design to create integrated in-situ visualization analysis (both model
>> > contributes to same visualization pipeline) in this case? Do you know
>> any
>> > implementation that is similar to this design? At least, is it
>> possible?
>> >
>> > In this case, the adaptor code will need to access to two different
>> > MPI_COMM_WORLD and it could run on all processor (from 0 to 7) or its
>> own
>> > 

[Paraview] Problems about calculate the difference between two datasets

[Feng Jiang]

ParaView newbie here ,and sorry to interrupt you for some problems i met.

I have been using two vtk input files which named t1 and t2 for example,
both input files contained only one dataset for CellData may called flux1
and flux2.

Exactly the structure and the data t1 and t2 got were same and only make a
reflect from the surface of the center of x axis.

And I want to make the CellData flux1 to subtrack the flux2.
However I have tried Using the Group Datasets and the Calculator, or Using
the Python Calculator or Programmable filter, and all the result I got is
empty.

The results I tried seems the ParaView only subtracked the data basic on
the Cell ID.

So is there any way to solve it or some wrong operate I made I wondered, or
I must make the datasets reflect before I input ParaView ?

Thanks for any guidance,

Feng
___
Powered by www.kitware.com

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Please keep messages on-topic and check the ParaView Wiki at: 
http://paraview.org/Wiki/ParaView

Search the list archives at: http://markmail.org/search/?q=ParaView

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/paraview