Re: [opendx-users] Usage model for OpenDX
Drew, Thanks for the info. Actually, I was toying around with the SocketConnect VPE module tonight, and actually found an OpenDX listserv message back from 2000 where the developer mentioned how he utilized this shared memory buffer. So (by coincidence) I was planning to look at that myself too; thanks for the tip! Per some previous correspondence, I was debating on whether I should use this socket interface where DX Objects are streamed to the socket, or just send my raw data (much more compactly) from my application to DX (using my own socket interface), and then formulate the DX Objects in the DX-side module. This may be more efficient in terms of socket throughput. But even so, I may still try this socketconnect interface because it would be quicker to use (since it's already written.) - seems easy enough. Thanks again for your help, Matt University of Texas at Austin (formerly from Owego, NY though - your neck of the woods, Drew) -- Andrew J. Dolgert wrote: Re: [opendx-users] Usage model for OpenDX Hi Matt, I like your approach and Mike's suggestions. His way of embedding OpenDX sounds very nice. As to whether you can use the Image viewer from C calls, I'm not sure, either, because I've never tried running a network through C calls. I recall Pelkie explaining to me that the Image module has a bunch of stuff in it that wrapsother modules like Display and Camera, and I think Image is unduly intimate with the GUI, so it makes sense it might not be available as such, but Mike seems to have a workaround. There is a module in the VPE called socketconnect. I looked and couldn't find it while trying to answer your last message. Glad you found it. It isn't implemented on my dear Windows version, but I looked at the code, and it has a flippin' sweet protocol: 1. integer length of buffer to send 2. the buffer. Then you close the socket. Internally, it treats that buffer as a serialized version of a DX Object and deserializes it. On your client, that means you need to take your data, create a DX Object, and then turn that Object into a string. The command you will use to serialize is Object _dxfExportBin_Buffer(Object o, int* size, void **buffer); It returns NULL on error and your input o on success. I'm telling you about this command b/c I don't recall seeing it in the docs even though it is exported from one of the libs. If you want to implement a shared memory copy, you can start from socketconnect.c. Drew
Re: [opendx-users] Usage model for OpenDX
Drew Mike, Thanks very much for the insight. I am admittedly new to OpenDX, and appreciate your guidance. I am currently taking your advice and ascribing to the strategy of sending my raw data (via a TCP/IP socket) to an OpenDX module that translates the data into DX Objects (Fields, etc.). I will undertake this socket approach because it's obviously expandable to network implementations, though not a fast as using shared memory. I also noticed there is an import via socket module available in the VPE; have you had any experience with this? As Mike mentioned, I observed in the call_module DX samples where it appears the VPE data flow graphs are essentially coded into C: a program-schematic of the data flow, analogous to the VPE implementation. For the data receiver / viewer, I may take this approach because of it's obvious speed increase. This doesn't preclude me from using the DX image viewer, correct? I agree that it seems much easier to use the DX viewer natively rather than creating a middle-man that has to constantly request and receive updates from DX. It seems to me I could still write my custom GUI in Qt (later on), and simply instruct DX what to display / do via C-calls (via the callm lib). This way, I'd still utilize the original DX image / visualization window (for zoom, rotation, etc.), but allow my front-end GUI to manipulate it in terms of which data is displayed, and how, etc. If there's any errors in my plans or assumptions, please let me know. Thanks again, Matt -- Michael Zelenik wrote: Drew correctly points out the difficulty of using the VPE and data flow environment for robust or commercial products. However, all of the DX functionality IS available through the library routines, either directly in an explicit routine, or through the valuable DXCallModule() function, which allows you to run any module just as in the VPE. This also includes Image and Display modules, with event handling. Using this approach, we have created robust commercial products entirely in C. without the dxexec or the VPE. These products make extensive use of the DX functionality and data model inside, and the Display module outside. They also provide excellent user interactivity with complex 3D displays (much better than one can get in the data flow environment). In general, we prototype in the VPE, and then write the C version after we see where things going, often as an iterative process. It has worked very well. In my opinion, this is the ONLY way to write anything that is of large size or complexity, and certainly anything that needs good error handling. But, like everything else with DX, there is a learning curve, but once over that, it becomes very nice to work with (except for memory handling sometimes). If you search back through the archives for my name in the From or Body fields, you can see some discussion about this back in 2000 or so. Mike Andrew J. Dolgert wrote: Hi Matt, You can set variables in a running copy of DX using DXLink, but you are correct that you send them as text. There are helper routines in one of the libraries that let you write a loadable module and external program which talk with each other over sockets. These routines are commented out in the Windows build, by the way, with a big #ifndef WIN32. You would want to combine the two in order to have full control over DX. Send data on the socket; change parameters through the script. If you want to use shared memory, you have to write it yourself as a loadable module in DX. On Windows, you have to roll your own anyway. Writing a module isn't so bad, though. MPI is also available, as I think about it, and implementations of MPI sometimes use shared memory on local machines. I haven't looked into using MPI with DX and suspect it is implemented mostly for parallel execution of modules, not for receiving data from remote processes. If you don't have a running copy of DX, then you can still use library methods to decode arrays and such, but you can't load a DX net. Without a DX net, what good is doing visualization in DX? I think this technique is best for translators to/from DX. For instance, if your program wanted to create a DX Object and then send it to DX, it could use the library to make the relevant Arrays and Objects, then use _dxfExportBin_buffer to turn this object into a byte stream, send the byte stream to DX, and then have a DX module decode it with _dxfImportBin_buffer. More likely, you'll write a DX module that receives data in your preferred format and then creates the Object on the fly. When you control DX from an external program, the simplest way is to allow DX to show its own Image window. If you want the display in your own program, then see the Tk example in the distribution. Every time you try to rotate the image, you have to request that DX rotate the image and return the result. If you want to test whether this will work well enough for you, you could
RE: [opendx-users] Usage model for OpenDX
Matt wrote: there is an import via socket module available in the VPE; have you had any experience with this? I have not used it. As Mike mentioned, I observed in the call_module DX samples where it appears the VPE data flow graphs are essentially coded into C: a program-schematic of the data flow, analogous to the VPE implementation. For the data receiver / viewer, I may take this approach because of it's obvious speed increase. This doesn't preclude me from using the DX image viewer, correct? The following is based on what we currently do, which grew out of our initial use of this approach in 1997. Things may well have changed since then, but I think the basics will be the same. You can use Display, but Image was (is?) not available. This is not the limitation that it might initially sound like, since default rotate/pan/zoom interactors are provided. But I believe they may only be available in software rendering mode, since we end up using our own and I think that was the reason. I just took them directly from the code in Image(). Then, the Supervise* modules provide all that you need for window and event handling, with events tied to whatever functionality you want. So, for example, to interact with one object in a Display of many objects, you can catch a button down event, call Pick to find the object, replace the object with a simpler geometry version (for interaction speed), and then apply a transform to it based on the cursor motion, and then rerender via Display. And then repeat as long as the button is down. Then on button up you can put the original object back in and Display. This all happens in a tight event loop down inside Display/Supervise* so it's very fast. I agree that it seems much easier to use the DX viewer natively rather than creating a middle-man that has to constantly request and receive updates from DX. It seems to me I could still write my custom GUI in Qt (later on), and simply instruct DX what to display / do via C-calls (via the callm lib). This way, I'd still utilize the original DX image / visualization window (for zoom, rotation, etc.), but allow my front-end GUI to manipulate it in terms of which data is displayed, and how, etc. That should work. We generally try to keep the GUI separate from the rest of the code with a functional interface; with the GUI either telling the rest of the code what to do, or displaying results from it. If there's any errors in my plans or assumptions, please let me know. Thanks again, Matt -- Michael Zelenik wrote: Drew correctly points out the difficulty of using the VPE and data flow environment for robust or commercial products. However, all of the DX functionality IS available through the library routines, either directly in an explicit routine, or through the valuable DXCallModule() function, which allows you to run any module just as in the VPE. This also includes Image and Display modules, with event handling. Using this approach, we have created robust commercial products entirely in C. without the dxexec or the VPE. These products make extensive use of the DX functionality and data model inside, and the Display module outside. They also provide excellent user interactivity with complex 3D displays (much better than one can get in the data flow environment). In general, we prototype in the VPE, and then write the C version after we see where things going, often as an iterative process. It has worked very well. In my opinion, this is the ONLY way to write anything that is of large size or complexity, and certainly anything that needs good error handling. But, like everything else with DX, there is a learning curve, but once over that, it becomes very nice to work with (except for memory handling sometimes). If you search back through the archives for my name in the From or Body fields, you can see some discussion about this back in 2000 or so. Mike Andrew J. Dolgert wrote: Hi Matt, You can set variables in a running copy of DX using DXLink, but you are correct that you send them as text. There are helper routines in one of the libraries that let you write a loadable module and external program which talk with each other over sockets. These routines are commented out in the Windows build, by the way, with a big #ifndef WIN32. You would want to combine the two in order to have full control over DX. Send data on the socket; change parameters through the script. If you want to use shared memory, you have to write it yourself as a loadable module in DX. On Windows, you have to roll your own anyway. Writing a module isn't so bad, though. MPI is also available, as I think about it, and implementations of MPI sometimes use shared memory on local machines. I haven't looked into using MPI with DX and suspect it is implemented mostly for parallel execution of modules, not for receiving data from remote processes. If you don't
RE: [opendx-users] Usage model for OpenDX
Title: Re: [opendx-users] Usage model for OpenDX Hi Matt, I like your approach and Mike's suggestions. His way of embedding OpenDX sounds very nice. As to whether you can use the Image viewer from C calls, I'm not sure, either, because I've never tried running a network through C calls. I recall Pelkie explaining to me that the Image module has a bunch of stuff in it that wrapsother modules like Display and Camera, and I think Image is unduly intimate with the GUI, so it makes sense it might not be available as such, but Mike seems to have a workaround. There is a module in the VPE called socketconnect. I looked and couldn't find it while trying to answer your last message. Glad you found it. It isn't implemented on my dear Windows version, but I looked at the code, and it has a flippin' sweet protocol: 1. integer length of buffer to send 2. the buffer. Then you close the socket. Internally, it treats that buffer as a serialized version of a DX Object and deserializes it. On your client, that means you need to take your data, create a DX Object, and then turn that Object into a string. The command you will use to serialize is Object _dxfExportBin_Buffer(Object o, int* size, void **buffer); It returns NULL on error and your input o on success. I'm telling you about this command b/c I don't recall seeing it in the docs even though it is exported from one of the libs. If you want to implement a shared memory copy, you can start from socketconnect.c. Drew
RE: [opendx-users] Usage model for OpenDX
Hi Matt, You can set variables in a running copy of DX using DXLink, but you are correct that you send them as text. There are helper routines in one of the libraries that let you write a loadable module and external program which talk with each other over sockets. These routines are commented out in the Windows build, by the way, with a big #ifndef WIN32. You would want to combine the two in order to have full control over DX. Send data on the socket; change parameters through the script. If you want to use shared memory, you have to write it yourself as a loadable module in DX. On Windows, you have to roll your own anyway. Writing a module isn't so bad, though. MPI is also available, as I think about it, and implementations of MPI sometimes use shared memory on local machines. I haven't looked into using MPI with DX and suspect it is implemented mostly for parallel execution of modules, not for receiving data from remote processes. If you don't have a running copy of DX, then you can still use library methods to decode arrays and such, but you can't load a DX net. Without a DX net, what good is doing visualization in DX? I think this technique is best for translators to/from DX. For instance, if your program wanted to create a DX Object and then send it to DX, it could use the library to make the relevant Arrays and Objects, then use _dxfExportBin_buffer to turn this object into a byte stream, send the byte stream to DX, and then have a DX module decode it with _dxfImportBin_buffer. More likely, you'll write a DX module that receives data in your preferred format and then creates the Object on the fly. When you control DX from an external program, the simplest way is to allow DX to show its own Image window. If you want the display in your own program, then see the Tk example in the distribution. Every time you try to rotate the image, you have to request that DX rotate the image and return the result. If you want to test whether this will work well enough for you, you could expand the Tk example to allow rotations. You may want to do this to prove to yourself that DX can fulfill your technical requirements. I probably have some of this wrong b/c I know better the things I did than everything doable in DX. People have wrenched this program into all sorts of amazing capabilities over the years. While the data model of DX is great and it is a very expressive visual language, it is not good at being embedded in other programs because it runs as a separate process, doesn't have structured error modes to the controlling process, and the executive sometimes dies. All this is probably okay for an academic tool, but, again, it depends on your requirements. Drew Dolgert Cornell Theory Center
RE: [opendx-users] Usage model for OpenDX
Drew correctly points out the difficulty of using the VPE and data flow environment for robust or commercial products. However, all of the DX functionality IS available through the library routines, either directly in an explicit routine, or through the valuable DXCallModule() function, which allows you to run any module just as in the VPE. This also includes Image and Display modules, with event handling. Using this approach, we have created robust commercial products entirely in C. without the dxexec or the VPE. These products make extensive use of the DX functionality and data model inside, and the Display module outside. They also provide excellent user interactivity with complex 3D displays (much better than one can get in the data flow environment). In general, we prototype in the VPE, and then write the C version after we see where things going, often as an iterative process. It has worked very well. In my opinion, this is the ONLY way to write anything that is of large size or complexity, and certainly anything that needs good error handling. But, like everything else with DX, there is a learning curve, but once over that, it becomes very nice to work with (except for memory handling sometimes). If you search back through the archives for my name in the From or Body fields, you can see some discussion about this back in 2000 or so. Mike -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] Behalf Of Andrew J. Dolgert Sent: Tuesday, May 31, 2005 10:07 AM To: opendx2-users@lists.berlios.de Subject: RE: [opendx-users] Usage model for OpenDX Hi Matt, You can set variables in a running copy of DX using DXLink, but you are correct that you send them as text. There are helper routines in one of the libraries that let you write a loadable module and external program which talk with each other over sockets. These routines are commented out in the Windows build, by the way, with a big #ifndef WIN32. You would want to combine the two in order to have full control over DX. Send data on the socket; change parameters through the script. If you want to use shared memory, you have to write it yourself as a loadable module in DX. On Windows, you have to roll your own anyway. Writing a module isn't so bad, though. MPI is also available, as I think about it, and implementations of MPI sometimes use shared memory on local machines. I haven't looked into using MPI with DX and suspect it is implemented mostly for parallel execution of modules, not for receiving data from remote processes. If you don't have a running copy of DX, then you can still use library methods to decode arrays and such, but you can't load a DX net. Without a DX net, what good is doing visualization in DX? I think this technique is best for translators to/from DX. For instance, if your program wanted to create a DX Object and then send it to DX, it could use the library to make the relevant Arrays and Objects, then use _dxfExportBin_buffer to turn this object into a byte stream, send the byte stream to DX, and then have a DX module decode it with _dxfImportBin_buffer. More likely, you'll write a DX module that receives data in your preferred format and then creates the Object on the fly. When you control DX from an external program, the simplest way is to allow DX to show its own Image window. If you want the display in your own program, then see the Tk example in the distribution. Every time you try to rotate the image, you have to request that DX rotate the image and return the result. If you want to test whether this will work well enough for you, you could expand the Tk example to allow rotations. You may want to do this to prove to yourself that DX can fulfill your technical requirements. I probably have some of this wrong b/c I know better the things I did than everything doable in DX. People have wrenched this program into all sorts of amazing capabilities over the years. While the data model of DX is great and it is a very expressive visual language, it is not good at being embedded in other programs because it runs as a separate process, doesn't have structured error modes to the controlling process, and the executive sometimes dies. All this is probably okay for an academic tool, but, again, it depends on your requirements. Drew Dolgert Cornell Theory Center
RE: [opendx-users] Usage model for OpenDX
I wrote: ... all of the DX functionality IS available through the library routines... That is, except for the VPE, data flow, and interactors. But the idea is that you don't need or want the data flow when doing this, since you then have much better control over everything. And in general, one wants to use their own tools for user interactors anyway. Mike
Re: [opendx-users] Usage model for OpenDX
I linked Qt with DX and it always crashed, even though I tried various ways of making sure that DX was in a thread that did not wait a long time. My solution was to connect DX with Qt using CORBA, the free OmniORB. Though it seems complicated to through into the mix yet another tool, it works. Data transfer is with a socket, the Qt - CORBA - DX is for instructions. DX is linked with CORBA and as a separate process Qt is linked with CORBA. best regards, Alan -- Centro di Ricerca, Sviluppo e Studi Superiori in Sardegna Center for Advanced Studies, Research, and Development in Sardinia Postal Address: | Physical Address for FedEx, UPS, DHL: --- | - Alan Scheinine| Alan Scheinine c/o CRS4 | c/o CRS4 C.P. n. 25| Loc. Pixina Manna Edificio 1 09010 Pula (Cagliari), Italy | 09010 Pula (Cagliari), Italy Email: [EMAIL PROTECTED] Phone: 070 9250 238 [+39 070 9250 238] Fax: 070 9250 216 or 220 [+39 070 9250 216 or +39 070 9250 220] Operator at reception: 070 9250 1 [+39 070 9250 1] Mobile phone: 347 7990472 [+39 347 7990472]
