[U2] XLr8 New Release 4.0.14 [ad]
U2logic's announces the new release of XLr8 Tools for Universe and Unidata version 4.0.14. In keeping with our normal release strategy, we have combined our announcement into a single one each month. Only XLr8 tools continually updates it products several times a month in our U2 world. U2logic's is an application creator so we use our tool on average about 8 hours a day so we really do know what the tool need to do and how the perform. Try it yourself, we do have trial licenses. Here is our change log: v4.0.14 - March 6, 2014 Bugzilla [647] - Project Explorer display is not correct for LD and LF files on dictionaries Bugzilla [646] - XLr8Editor does allow direct access from version control (SVN) v4.0.12 - March 5, 2014 Bugzilla [641] - XLr8Licenser error on startup Regards, Doug www.u2logic.com XLr8 Tools for Universe and Unidata ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users
Re: [U2] Multi-threaded phantom processing
Yes of course there are many solutions: rcp, ftp, nfa, http, mapped drives, etc. But the one that has the most security is the socket to socket communications with your own protocol. You get to control who has access and what commands they have access to. And of course you get to add logging as you need. Plus it runs native from U2 so that you can do command like compile, or move just a single attribute or sub-value. My customer needed a tool to keep the developers out of the live data server. By disabling the normal data transfer methods and installing a socket-to-socket approach we made the auditors very happy. David A. Green (480) 201-7953 DAG Consulting -Original Message- From: u2-users-boun...@listserver.u2ug.org [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Rick Nuckolls Sent: Wednesday, March 05, 2014 5:57 PM To: U2 Users List Cc: U2 Users List Subject: Re: [U2] Multi-threaded phantom processing As elegant as some of these ideas are, it might be possible, depending on the number and nature of the transactions, to batch them up every n seconds and use rcp to copy them to an in box on the remote machine. -Rick On Mar 5, 2014, at 4:19 PM, Horacio Pellegrino pellegrin...@gmail.com wrote: Why don't you have one port listening for requests, and as soon as they come in, you create a new phantom to answer that request ( liberating the listener immediately for the next request ), then when the phantom finishes it could log off. Might be you want to keep a pool of phantoms alive for a while, on the assumption they might be reused. That way you are sill using one license per request, but you have the benefit of pooling... You can always set min-max limit for those phantom-workers. Horacio Pellegrino On Tue, Mar 4, 2014 at 11:30 PM, Peter Cheney peter.che...@firstmac.com.auwrote: Thanks David. As they say Bigger than Ben-Hur! That's a great idea. The server I have is listening on a predetermined port. I trolled through /etc/services and a netstat on the remote host to select something that looked unused. So then if it the main phantom spawns 1-n children (presumably more phantoms?), it would also need to select 1-n ports which I suppose must also be predetermined? Or is there a call to the OS that can return an unused port? We're on AIX but the remote host is HPUX. Cheers Peter Peter Cheney Ultracs Developer t 07 3017 8837 | f 07 3002 8400 e peter.che...@firstmac.com.au w firstmac.com.au -Original Message- From: u2-users-boun...@listserver.u2ug.org [mailto: u2-users-boun...@listserver.u2ug.org] On Behalf Of David A. Green Sent: Wednesday, 5 March 2014 13:01 To: 'U2 Users List' Subject: Re: [U2] Multi-threaded phantom processing Peter, first of all let me congratulate you on this achievement. Having just finished a similar feat using UniData I understand the complexities that went into it. The system I'm current working with will be relatively low traffic. But I was pondering on the method of having a server socket that would just spawn a transfer server socket and send the socket address to the calling client. Then the client disconnects from the main server and just uses the transfer server to complete the task. Having 10 open phantoms with sockets will use up 10 licenses. David A. Green (480) 201-7953 DAG Consulting -Original Message- From: u2-users-boun...@listserver.u2ug.org [mailto: u2-users-boun...@listserver.u2ug.org] On Behalf Of Peter Cheney Sent: Tuesday, March 04, 2014 6:18 PM To: U2 Users List (u2-users@listserver.u2ug.org) Subject: [U2] Multi-threaded phantom processing Hi Everyone, A project that I'm currently working on requires transmission of transactional and ancillary data from one UV site to site to another. I have the communications all set up and it is working very well via the UV socket functions with a client at one end and a server at the other. Both the client and the server both run as a UV phantom processes written in UV basic and the data being transferred (also encrypted) is more or less real time with various UV file triggers on the client end creating queue items for transmission based on the data content / transaction type. This set up is a single threaded affair and I am wondering how it will cope and what options I'll have as the volume of data increases. Multi-threading my server process to dish out some of the load might be one possibility and I've seen examples in the past where the last digit of a numeric key (i.e. 0-9) is used to determine which one of 10 phantoms to call thus providing an effective distribution of the work. But even then, reading/writing processing data to a file for each child process seems to me to be the only way to achieve parent - child phantom communications at either end. Is there another way perhaps? It may be that I will need 2 sets of
Re: [U2] Multi-threaded phantom processing
Will you kinda lost me on this approach. Since the Server socket is on a different machine than the client they don't see each other's locks. Different clients might be on different servers too. David A. Green (480) 201-7953 DAG Consulting -Original Message- From: u2-users-boun...@listserver.u2ug.org [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Wjhonson Sent: Wednesday, March 05, 2014 5:26 PM To: u2-users@listserver.u2ug.org Subject: Re: [U2] Multi-threaded phantom processing Peter is the only point of the parent child communications to address points like 1 I'm busy go away; OR 2 I'm available to work If so, you could simply use a lock-semaphore to address both of those issues.\ -Original Message- From: Horacio Pellegrino pellegrin...@gmail.com To: U2 Users List u2-users@listserver.u2ug.org Sent: Wed, Mar 5, 2014 4:19 pm Subject: Re: [U2] Multi-threaded phantom processing Why don't you have one port listening for requests, and as soon as they come in, you create a new phantom to answer that request ( liberating the listener immediately for the next request ), then when the phantom finishes it could log off. Might be you want to keep a pool of phantoms alive for a while, on the assumption they might be reused. That way you are sill using one license per request, but you have the benefit of pooling... You can always set min-max limit for those phantom-workers. Horacio Pellegrino On Tue, Mar 4, 2014 at 11:30 PM, Peter Cheney peter.che...@firstmac.com.auwrote: Thanks David. As they say Bigger than Ben-Hur! That's a great idea. The server I have is listening on a predetermined port. I trolled through /etc/services and a netstat on the remote host to select something that looked unused. So then if it the main phantom spawns 1-n children (presumably more phantoms?), it would also need to select 1-n ports which I suppose must also be predetermined? Or is there a call to the OS that can return an unused port? We're on AIX but the remote host is HPUX. Cheers Peter Peter Cheney Ultracs Developer t 07 3017 8837 | f 07 3002 8400 e peter.che...@firstmac.com.au w firstmac.com.au -Original Message- From: u2-users-boun...@listserver.u2ug.org [mailto: u2-users-boun...@listserver.u2ug.org] On Behalf Of David A. Green Sent: Wednesday, 5 March 2014 13:01 To: 'U2 Users List' Subject: Re: [U2] Multi-threaded phantom processing Peter, first of all let me congratulate you on this achievement. Having just finished a similar feat using UniData I understand the complexities that went into it. The system I'm current working with will be relatively low traffic. But I was pondering on the method of having a server socket that would just spawn a transfer server socket and send the socket address to the calling client. Then the client disconnects from the main server and just uses the transfer server to complete the task. Having 10 open phantoms with sockets will use up 10 licenses. David A. Green (480) 201-7953 DAG Consulting -Original Message- From: u2-users-boun...@listserver.u2ug.org [mailto: u2-users-boun...@listserver.u2ug.org] On Behalf Of Peter Cheney Sent: Tuesday, March 04, 2014 6:18 PM To: U2 Users List (u2-users@listserver.u2ug.org) Subject: [U2] Multi-threaded phantom processing Hi Everyone, A project that I'm currently working on requires transmission of transactional and ancillary data from one UV site to site to another. I have the communications all set up and it is working very well via the UV socket functions with a client at one end and a server at the other. Both the client and the server both run as a UV phantom processes written in UV basic and the data being transferred (also encrypted) is more or less real time with various UV file triggers on the client end creating queue items for transmission based on the data content / transaction type. This set up is a single threaded affair and I am wondering how it will cope and what options I'll have as the volume of data increases. Multi-threading my server process to dish out some of the load might be one possibility and I've seen examples in the past where the last digit of a numeric key (i.e. 0-9) is used to determine which one of 10 phantoms to call thus providing an effective distribution of the work. But even then, reading/writing processing data to a file for each child process seems to me to be the only way to achieve parent - child phantom communications at either end. Is there another way perhaps? It may be that I will need 2 sets of phantoms, 1 for the transactions and one for the ancillary data. Although we run UV on AIX the only other requirement is that I must be able to do it within UV Basic. Does anyone out there on the list know of a better way or any alternatives/advice so I can get a better appreciation of what else might be possible please? Many thanks in advance, Peter
Re: [U2] Multi-threaded phantom processing
I was referring solely to the point of launching phantoms and how they could communicate back to their master who launched them. Not back to a different server. So the thread was mentioning a master-slave relationship on *a* server as I understood it. And the question asked was how do the slaves and master communicate. -Original Message- From: David A. Green dgr...@dagconsulting.com To: 'U2 Users List' u2-users@listserver.u2ug.org Sent: Thu, Mar 6, 2014 5:43 am Subject: Re: [U2] Multi-threaded phantom processing Will you kinda lost me on this approach. Since the Server socket is on a different machine than the client they don't see each other's locks. Different clients might be on different servers too. David A. Green (480) 201-7953 DAG Consulting -Original Message- From: u2-users-boun...@listserver.u2ug.org [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Wjhonson Sent: Wednesday, March 05, 2014 5:26 PM To: u2-users@listserver.u2ug.org Subject: Re: [U2] Multi-threaded phantom processing Peter is the only point of the parent child communications to address points like 1 I'm busy go away; OR 2 I'm available to work If so, you could simply use a lock-semaphore to address both of those issues.\ -Original Message- From: Horacio Pellegrino pellegrin...@gmail.com To: U2 Users List u2-users@listserver.u2ug.org Sent: Wed, Mar 5, 2014 4:19 pm Subject: Re: [U2] Multi-threaded phantom processing Why don't you have one port listening for requests, and as soon as they come in, you create a new phantom to answer that request ( liberating the listener immediately for the next request ), then when the phantom finishes it could log off. Might be you want to keep a pool of phantoms alive for a while, on the assumption they might be reused. That way you are sill using one license per request, but you have the benefit of pooling... You can always set min-max limit for those phantom-workers. Horacio Pellegrino On Tue, Mar 4, 2014 at 11:30 PM, Peter Cheney peter.che...@firstmac.com.auwrote: Thanks David. As they say Bigger than Ben-Hur! That's a great idea. The server I have is listening on a predetermined port. I trolled through /etc/services and a netstat on the remote host to select something that looked unused. So then if it the main phantom spawns 1-n children (presumably more phantoms?), it would also need to select 1-n ports which I suppose must also be predetermined? Or is there a call to the OS that can return an unused port? We're on AIX but the remote host is HPUX. Cheers Peter Peter Cheney Ultracs Developer t 07 3017 8837 | f 07 3002 8400 e peter.che...@firstmac.com.au w firstmac.com.au -Original Message- From: u2-users-boun...@listserver.u2ug.org [mailto: u2-users-boun...@listserver.u2ug.org] On Behalf Of David A. Green Sent: Wednesday, 5 March 2014 13:01 To: 'U2 Users List' Subject: Re: [U2] Multi-threaded phantom processing Peter, first of all let me congratulate you on this achievement. Having just finished a similar feat using UniData I understand the complexities that went into it. The system I'm current working with will be relatively low traffic. But I was pondering on the method of having a server socket that would just spawn a transfer server socket and send the socket address to the calling client. Then the client disconnects from the main server and just uses the transfer server to complete the task. Having 10 open phantoms with sockets will use up 10 licenses. David A. Green (480) 201-7953 DAG Consulting -Original Message- From: u2-users-boun...@listserver.u2ug.org [mailto: u2-users-boun...@listserver.u2ug.org] On Behalf Of Peter Cheney Sent: Tuesday, March 04, 2014 6:18 PM To: U2 Users List (u2-users@listserver.u2ug.org) Subject: [U2] Multi-threaded phantom processing Hi Everyone, A project that I'm currently working on requires transmission of transactional and ancillary data from one UV site to site to another. I have the communications all set up and it is working very well via the UV socket functions with a client at one end and a server at the other. Both the client and the server both run as a UV phantom processes written in UV basic and the data being transferred (also encrypted) is more or less real time with various UV file triggers on the client end creating queue items for transmission based on the data content / transaction type. This set up is a single threaded affair and I am wondering how it will cope and what options I'll have as the volume of data increases. Multi-threading my server process to dish out some of the load might be one possibility and I've seen examples in the past where the last digit of a numeric key (i.e. 0-9) is used to determine which one of 10 phantoms to call thus providing an effective distribution of the work. But even then, reading/writing processing data to a file for each child
Re: [U2] Multi-threaded phantom processing
My thoughts were that the parent will need to allocate the work and in doing so take the incoming data stream, launching a child and somehow pass the work to it. My only current knowledge of doing that within UV basic is writing to a queue and have the child phantom read from that and respond via another queue. Hence my question if there there's another possibly better way of doing it? It is nice to know if a particular phantom is busy or available though so that suggestion is certainly handy thanks. Peter Cheney Ultracs Developer t 07 3017 8837 | f 07 3002 8400 e peter.che...@firstmac.com.au w firstmac.com.au -Original Message- From: u2-users-boun...@listserver.u2ug.org [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Wjhonson Sent: Friday, 7 March 2014 02:52 To: u2-users@listserver.u2ug.org Subject: Re: [U2] Multi-threaded phantom processing I was referring solely to the point of launching phantoms and how they could communicate back to their master who launched them. Not back to a different server. So the thread was mentioning a master-slave relationship on *a* server as I understood it. And the question asked was how do the slaves and master communicate. -Original Message- From: David A. Green dgr...@dagconsulting.com To: 'U2 Users List' u2-users@listserver.u2ug.org Sent: Thu, Mar 6, 2014 5:43 am Subject: Re: [U2] Multi-threaded phantom processing Will you kinda lost me on this approach. Since the Server socket is on a different machine than the client they don't see each other's locks. Different clients might be on different servers too. David A. Green (480) 201-7953 DAG Consulting -Original Message- From: u2-users-boun...@listserver.u2ug.org [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Wjhonson Sent: Wednesday, March 05, 2014 5:26 PM To: u2-users@listserver.u2ug.org Subject: Re: [U2] Multi-threaded phantom processing Peter is the only point of the parent child communications to address points like 1 I'm busy go away; OR 2 I'm available to work If so, you could simply use a lock-semaphore to address both of those issues.\ -Original Message- From: Horacio Pellegrino pellegrin...@gmail.com To: U2 Users List u2-users@listserver.u2ug.org Sent: Wed, Mar 5, 2014 4:19 pm Subject: Re: [U2] Multi-threaded phantom processing Why don't you have one port listening for requests, and as soon as they come in, you create a new phantom to answer that request ( liberating the listener immediately for the next request ), then when the phantom finishes it could log off. Might be you want to keep a pool of phantoms alive for a while, on the assumption they might be reused. That way you are sill using one license per request, but you have the benefit of pooling... You can always set min-max limit for those phantom-workers. Horacio Pellegrino On Tue, Mar 4, 2014 at 11:30 PM, Peter Cheney peter.che...@firstmac.com.auwrote: Thanks David. As they say Bigger than Ben-Hur! That's a great idea. The server I have is listening on a predetermined port. I trolled through /etc/services and a netstat on the remote host to select something that looked unused. So then if it the main phantom spawns 1-n children (presumably more phantoms?), it would also need to select 1-n ports which I suppose must also be predetermined? Or is there a call to the OS that can return an unused port? We're on AIX but the remote host is HPUX. Cheers Peter Peter Cheney Ultracs Developer t 07 3017 8837 | f 07 3002 8400 e peter.che...@firstmac.com.au w firstmac.com.au -Original Message- From: u2-users-boun...@listserver.u2ug.org [mailto: u2-users-boun...@listserver.u2ug.org] On Behalf Of David A. Green Sent: Wednesday, 5 March 2014 13:01 To: 'U2 Users List' Subject: Re: [U2] Multi-threaded phantom processing Peter, first of all let me congratulate you on this achievement. Having just finished a similar feat using UniData I understand the complexities that went into it. The system I'm current working with will be relatively low traffic. But I was pondering on the method of having a server socket that would just spawn a transfer server socket and send the socket address to the calling client. Then the client disconnects from the main server and just uses the transfer server to complete the task. Having 10 open phantoms with sockets will use up 10 licenses. David A. Green (480) 201-7953 DAG Consulting -Original Message- From: u2-users-boun...@listserver.u2ug.org [mailto: u2-users-boun...@listserver.u2ug.org] On Behalf Of Peter Cheney Sent: Tuesday, March 04, 2014 6:18 PM To: U2 Users List (u2-users@listserver.u2ug.org) Subject: [U2] Multi-threaded phantom processing Hi Everyone, A project that I'm currently working on requires transmission of transactional and ancillary data from one UV site to site to another. I have the communications all set up and it is working very well via
Re: [U2] Multi-threaded phantom processing
The simplest approach is to use the queue, but that requires check...wait...check...wait Another way, is to have one master and several always awake slaves listening as devices. The master uses the SEND command to pipe a command directly into the input stream of the listeners. This makes them wake up instantly (more or less) instead of waiting a second or a minute to check the queue. -Original Message- From: Peter Cheney peter.che...@firstmac.com.au To: 'U2 Users List' u2-users@listserver.u2ug.org Sent: Thu, Mar 6, 2014 4:05 pm Subject: Re: [U2] Multi-threaded phantom processing My thoughts were that the parent will need to allocate the work and in doing so take the incoming data stream, launching a child and somehow pass the work to it. My only current knowledge of doing that within UV basic is writing to a queue and have the child phantom read from that and respond via another queue. Hence my question if there there's another possibly better way of doing it? It is nice to know if a particular phantom is busy or available though so that suggestion is certainly handy thanks. Peter Cheney Ultracs Developer t 07 3017 8837 | f 07 3002 8400 e peter.che...@firstmac.com.au w firstmac.com.au -Original Message- From: u2-users-boun...@listserver.u2ug.org [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Wjhonson Sent: Friday, 7 March 2014 02:52 To: u2-users@listserver.u2ug.org Subject: Re: [U2] Multi-threaded phantom processing I was referring solely to the point of launching phantoms and how they could communicate back to their master who launched them. Not back to a different server. So the thread was mentioning a master-slave relationship on *a* server as I understood it. And the question asked was how do the slaves and master communicate. -Original Message- From: David A. Green dgr...@dagconsulting.com To: 'U2 Users List' u2-users@listserver.u2ug.org Sent: Thu, Mar 6, 2014 5:43 am Subject: Re: [U2] Multi-threaded phantom processing Will you kinda lost me on this approach. Since the Server socket is on a different machine than the client they don't see each other's locks. Different clients might be on different servers too. David A. Green (480) 201-7953 DAG Consulting -Original Message- From: u2-users-boun...@listserver.u2ug.org [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Wjhonson Sent: Wednesday, March 05, 2014 5:26 PM To: u2-users@listserver.u2ug.org Subject: Re: [U2] Multi-threaded phantom processing Peter is the only point of the parent child communications to address points like 1 I'm busy go away; OR 2 I'm available to work If so, you could simply use a lock-semaphore to address both of those issues.\ -Original Message- From: Horacio Pellegrino pellegrin...@gmail.com To: U2 Users List u2-users@listserver.u2ug.org Sent: Wed, Mar 5, 2014 4:19 pm Subject: Re: [U2] Multi-threaded phantom processing Why don't you have one port listening for requests, and as soon as they come in, you create a new phantom to answer that request ( liberating the listener immediately for the next request ), then when the phantom finishes it could log off. Might be you want to keep a pool of phantoms alive for a while, on the assumption they might be reused. That way you are sill using one license per request, but you have the benefit of pooling... You can always set min-max limit for those phantom-workers. Horacio Pellegrino On Tue, Mar 4, 2014 at 11:30 PM, Peter Cheney peter.che...@firstmac.com.auwrote: Thanks David. As they say Bigger than Ben-Hur! That's a great idea. The server I have is listening on a predetermined port. I trolled through /etc/services and a netstat on the remote host to select something that looked unused. So then if it the main phantom spawns 1-n children (presumably more phantoms?), it would also need to select 1-n ports which I suppose must also be predetermined? Or is there a call to the OS that can return an unused port? We're on AIX but the remote host is HPUX. Cheers Peter Peter Cheney Ultracs Developer t 07 3017 8837 | f 07 3002 8400 e peter.che...@firstmac.com.au w firstmac.com.au -Original Message- From: u2-users-boun...@listserver.u2ug.org [mailto: u2-users-boun...@listserver.u2ug.org] On Behalf Of David A. Green Sent: Wednesday, 5 March 2014 13:01 To: 'U2 Users List' Subject: Re: [U2] Multi-threaded phantom processing Peter, first of all let me congratulate you on this achievement. Having just finished a similar feat using UniData I understand the complexities that went into it. The system I'm current working with will be relatively low traffic. But I was pondering on the method of having a server socket that would just spawn a transfer server socket and send the socket address to the calling client. Then the client disconnects from the main server and just uses the transfer server to complete the task. Having 10
Re: [U2] Multi-threaded phantom processing
Thanks for that suggestion too. I did not know the SEND command could be used like that. So I suppose the listeners would use the openpipe and wait then? Must try that. This also reminded me of a time during mid-late 90's programming on a Sequoia pick system. That version of basic had a PAUSE command that you could use to pause a phantom indefinitely. Then wake it up again via another command in another program (POKE I think) directed at its PID. It was a very handy tool back then but have not seen it or an equivalent since. Peter Cheney Ultracs Developer t 07 3017 8837 | f 07 3002 8400 e peter.che...@firstmac.com.au w firstmac.com.au -Original Message- From: u2-users-boun...@listserver.u2ug.org [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Wjhonson Sent: Friday, 7 March 2014 10:20 To: u2-users@listserver.u2ug.org Subject: Re: [U2] Multi-threaded phantom processing The simplest approach is to use the queue, but that requires check...wait...check...wait Another way, is to have one master and several always awake slaves listening as devices. The master uses the SEND command to pipe a command directly into the input stream of the listeners. This makes them wake up instantly (more or less) instead of waiting a second or a minute to check the queue. -Original Message- From: Peter Cheney peter.che...@firstmac.com.au To: 'U2 Users List' u2-users@listserver.u2ug.org Sent: Thu, Mar 6, 2014 4:05 pm Subject: Re: [U2] Multi-threaded phantom processing My thoughts were that the parent will need to allocate the work and in doing so take the incoming data stream, launching a child and somehow pass the work to it. My only current knowledge of doing that within UV basic is writing to a queue and have the child phantom read from that and respond via another queue. Hence my question if there there's another possibly better way of doing it? It is nice to know if a particular phantom is busy or available though so that suggestion is certainly handy thanks. Peter Cheney Ultracs Developer t 07 3017 8837 | f 07 3002 8400 e peter.che...@firstmac.com.au w firstmac.com.au -Original Message- From: u2-users-boun...@listserver.u2ug.org [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Wjhonson Sent: Friday, 7 March 2014 02:52 To: u2-users@listserver.u2ug.org Subject: Re: [U2] Multi-threaded phantom processing I was referring solely to the point of launching phantoms and how they could communicate back to their master who launched them. Not back to a different server. So the thread was mentioning a master-slave relationship on *a* server as I understood it. And the question asked was how do the slaves and master communicate. -Original Message- From: David A. Green dgr...@dagconsulting.com To: 'U2 Users List' u2-users@listserver.u2ug.org Sent: Thu, Mar 6, 2014 5:43 am Subject: Re: [U2] Multi-threaded phantom processing Will you kinda lost me on this approach. Since the Server socket is on a different machine than the client they don't see each other's locks. Different clients might be on different servers too. David A. Green (480) 201-7953 DAG Consulting -Original Message- From: u2-users-boun...@listserver.u2ug.org [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Wjhonson Sent: Wednesday, March 05, 2014 5:26 PM To: u2-users@listserver.u2ug.org Subject: Re: [U2] Multi-threaded phantom processing Peter is the only point of the parent child communications to address points like 1 I'm busy go away; OR 2 I'm available to work If so, you could simply use a lock-semaphore to address both of those issues.\ -Original Message- From: Horacio Pellegrino pellegrin...@gmail.com To: U2 Users List u2-users@listserver.u2ug.org Sent: Wed, Mar 5, 2014 4:19 pm Subject: Re: [U2] Multi-threaded phantom processing Why don't you have one port listening for requests, and as soon as they come in, you create a new phantom to answer that request ( liberating the listener immediately for the next request ), then when the phantom finishes it could log off. Might be you want to keep a pool of phantoms alive for a while, on the assumption they might be reused. That way you are sill using one license per request, but you have the benefit of pooling... You can always set min-max limit for those phantom-workers. Horacio Pellegrino On Tue, Mar 4, 2014 at 11:30 PM, Peter Cheney peter.che...@firstmac.com.auwrote: Thanks David. As they say Bigger than Ben-Hur! That's a great idea. The server I have is listening on a predetermined port. I trolled through /etc/services and a netstat on the remote host to select something that looked unused. So then if it the main phantom spawns 1-n children (presumably more phantoms?), it would also need to select 1-n ports which I suppose must also be predetermined? Or is there a call to the OS that can return an unused port? We're on AIX but the remote host is HPUX. Cheers Peter
Re: [U2] Multi-threaded phantom processing
Hi Peter Sounds like an interested piece of work, just a couple of questions for you: - what sort of data volumes and update frequencies are involved? - do you need to know if the data has been successfully committed to the remote UV database? - did you consider SQL BCI or even UV/Net (read the different UV versions is an issue and so this may be a constraint) Gregor mentioned SQL BCI as an alternative interface - I've used that before to exchange data between systems and it works well. If you had the same UV version on both hosts - then UV/Net is simple effect way of transmitting data between UV systems with only licenses required on the remote system. With BCI you could utilise the Stored Procedure facility via an ExecDirect() call to push the data to the remote system in your UV file trigger. With UV/Net you could push the items to be copied the by your UV file trigger - thus the updates being written directly onto the remote host. BCI is free/included in UV and of course will utilise a license on the remote host. You need to write your own custom code. Probably a bit easier to maintain than any TCP/IP sockets based interface. Definitely a lot easier to troubleshoot ;-) ! YMMV. UV/Net is licensed on the number user connections on the remote host. Hence the above alternative as designed to eliminate the file-based queue - which in some instances by be desirable to maintain - depending on business, technical or operational needs/objections. Cheers, David -Original Message- From: u2-users-boun...@listserver.u2ug.org [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Peter Cheney Sent: Wednesday, 5 March 2014 12:18 PM To: U2 Users List (u2-users@listserver.u2ug.org) Subject: [U2] Multi-threaded phantom processing Hi Everyone, A project that I’m currently working on requires transmission of transactional and ancillary data from one UV site to site to another. I have the communications all set up and it is working very well via the UV socket functions with a client at one end and a server at the other. Both the client and the server both run as a UV phantom processes written in UV basic and the data being transferred (also encrypted) is more or less real time with various UV file triggers on the client end creating queue items for transmission based on the data content / transaction type. This set up is a single threaded affair and I am wondering how it will cope and what options I’ll have as the volume of data increases. Multi-threading my server process to dish out some of the load might be one possibility and I’ve seen examples in the past where the last digit of a numeric key (i.e. 0-9) is used to determine which one of 10 phantoms to call thus providing an effective distribution of the work. But even then, reading/writing processing data to a file for each child process seems to me to be the only way to achieve parent - child phantom communications at either end. Is there another way perhaps? It may be that I will need 2 sets of phantoms, 1 for the transactions and one for the ancillary data. Although we run UV on AIX the only other requirement is that I must be able to do it within UV Basic. Does anyone out there on the list know of a better way or any alternatives/advice so I can get a better appreciation of what else might be possible please? Many thanks in advance, Peter Peter Cheney Ultracs Developer t 07 3017 8837 | f 07 3002 8400 e peter.che...@firstmac.com.au w firstmac.com.au --- Note: This email (inc all attachments) is for the use of the intended recipient(s) only. Privileged or confidential information may be contained in this communication. If you have received this email in error, please notify the sender immediately and then delete all copies of this message from your computer network. If you are not the intended recipient, you must not keep, use, disclose, copy or distribute this email without the author's prior permission. If you are the intended recipient and you do not wish to receive similar electronic messages from us in future, then please respond to the sender to this effect. We have taken precautions to minimise the risk of transmitting software viruses, but advise you to carry out your own virus checks on this email and its attachments. We do not accept liability for any loss or damage caused by software viruses and do not represent that this transmission is free from viruses or other defects. Firstmac Limited (ABN 59 094 145 963) (AFSL 290600) --- ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users ** IMPORTANT MESSAGE * This e-mail message is intended only for the addressee(s) and contains
Re: [U2] Handle multi request
Hi Christo, Thankyou for your reply and welcome to the list! I hope you enjoy this fantastic community of ours. With some small differences, you have described almost exactly what I have created. I am using file triggers on the client end to place items in a queue. Once the queued items are processed, indexes on the various (translated) keys are created they’re moved to a log. The communications is done by UV basic socket functions: opensocket(), writesocket(), readsocket(), and closesocket(). Other functions used are: protocollogging(), getsocketinformation(), setsocketoptions(), and getsocketoptions(). The client/server communications also have a “heartbeat” function to determine if the link is alive or not. Most of the data we send back and forth is XML but there is one case where raw attribute data direct from a UV file is transmitted. This was made necessary by the complex structure of the fields within some files data resulting in it being too hard to convert to XML. The communications is also en/decrypted via the encrypt function. I will also need to audit our results and create various reports and this is yet to be done but I have most of the structure in place so just have to write the code. I’m releasing a beta version to test on 17th and there’s still plenty of work left for me to do. Cheers and thanks. Peter Peter Cheney Ultracs Developer t 07 3017 8837 | f 07 3002 8400 e peter.che...@firstmac.com.au w firstmac.com.au From: Van Rooyen, C. (Christo) [mailto:chris...@nedbank.co.za] Sent: Friday, 7 March 2014 15:11 To: Peter Cheney Subject: Handle multi request Peter, This is the first I respond on this mailing list. I have developed a method where I use shared folders to receive requests for information. It works on the basis of a “request” folder with a “commit” folder. To return the result I place it in a “result” folder and commit it in the “done” folder. I use D3 on RedHat Linux. I have a Phantom that poll the commit folder for any file placed. Using the filename it read the request in the request folder with the same name. The process then removes the filename from the commit folder. After processing it, it will place the result in the “result” folder, and let the incoming requestor know by committing it in the “done” folder. I have a clean-up process that removes any unclaimed results older than a week. For audit purpose, the requests are archived. In all cases the filename is the same. I have a header in the request telling the process what to do and use an inscription section for any data passed. With this method you can have any client system in any environment that can either use FTP, SAMBA, NFS, Apache with PHP, e.c.t. to place request to the folders. I use this method also for outgaining requests to a VB app running on a Windows server that handle my PDF mailing of invoices and statements. It is fast, and as long as the folders are accessible the requests can be placed. I use a controlling phantom that spawn a phantom for each requests received. If your phantom is not running, you won’t miss out on requests coming in. it will just process them the moment the phantom restart. We use this for most of the communication between the systems. Mainframe to UNIX to Windows and so on. Regards, Christo van Rooyen Systems Analyst / Designer II - Nedbank Limited (South Africa) Nedbank Limited Reg No 1951/09/06. The following link displays the names of the Nedbank Board of Directors and Company Secretary. [ http://www.nedbank.co.za/terms/DirectorsNedbank.htm ] This email is confidential and is intended for the addressee only. The following link will take you to Nedbank's legal notice. [ http://www.nedbank.co.za/terms/EmailDisclaimer.htm ] --- Note: This email (inc all attachments) is for the use of the intended recipient(s) only. Privileged or confidential information may be contained in this communication. If you have received this email in error, please notify the sender immediately and then delete all copies of this message from your computer network. If you are not the intended recipient, you must not keep, use, disclose, copy or distribute this email without the author's prior permission. If you are the intended recipient and you do not wish to receive similar electronic messages from us in future, then please respond to the sender to this effect. We have taken precautions to minimise the risk of transmitting software viruses, but advise you to carry out your own virus checks on this email and its attachments. We do not accept liability for any loss or damage caused by software viruses and do not represent that this transmission is free from viruses or other defects. Firstmac Limited (ABN 59 094 145 963) (AFSL 290600) ---
Re: [U2] Multi-threaded phantom processing
Initially data volume will be small - probably about 20 customers per day and about 6 records per customer but it is likely to grow very quickly up to some unknown amount. We'll have to see I guess. The records per customer are likely to be the high volume stuff. There is transaction logging on the remote system that I can use and am relying on for data integrity but I did not know about the SQL stuff until this week. Not sure if it would work until I can find time to play with it and see how it might suit us. Some of the other constraints were: no access allowed to the remote unirpc port, the remote UV is below 10.3 and different from ours, neither site has a UV/Net licence in prod. So I think I'm pretty much stuck with file queuing and multiple phantoms for now at least. I also was able to get a successful remote file process going on our test system while waiting for confirmation of UV/Net accessibility. It is very nice and might have been a nice solution too. Then I did some experimenting with the performance requirements on the file triggers and found there to be too much of a lag for the triggers to fire directly to the other end. This would've caused unacceptable problems in testing not to mention going live so a file queuing process had to be devised. In all the file queuing works quite well and can be coded for rock solid reliability so it has a lot of pluses. I can also fine tune the phantom loop times and socket timeouts on the fly at both ends once the data starts flowing and we get some idea of what needs changing. Fingers, toes, arms, legs (and eyes!) all currently crossed. Cheers Peter Peter Cheney Ultracs Developer t 07 3017 8837 | f 07 3002 8400 e peter.che...@firstmac.com.au w firstmac.com.au -Original Message- From: u2-users-boun...@listserver.u2ug.org [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Hona, David Sent: Friday, 7 March 2014 13:41 To: U2 Users List Subject: Re: [U2] Multi-threaded phantom processing Hi Peter Sounds like an interested piece of work, just a couple of questions for you: - what sort of data volumes and update frequencies are involved? - do you need to know if the data has been successfully committed to the remote UV database? - did you consider SQL BCI or even UV/Net (read the different UV versions is an issue and so this may be a constraint) Gregor mentioned SQL BCI as an alternative interface - I've used that before to exchange data between systems and it works well. If you had the same UV version on both hosts - then UV/Net is simple effect way of transmitting data between UV systems with only licenses required on the remote system. With BCI you could utilise the Stored Procedure facility via an ExecDirect() call to push the data to the remote system in your UV file trigger. With UV/Net you could push the items to be copied the by your UV file trigger - thus the updates being written directly onto the remote host. BCI is free/included in UV and of course will utilise a license on the remote host. You need to write your own custom code. Probably a bit easier to maintain than any TCP/IP sockets based interface. Definitely a lot easier to troubleshoot ;-) ! YMMV. UV/Net is licensed on the number user connections on the remote host. Hence the above alternative as designed to eliminate the file-based queue - which in some instances by be desirable to maintain - depending on business, technical or operational needs/objections. Cheers, David -Original Message- From: u2-users-boun...@listserver.u2ug.org [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Peter Cheney Sent: Wednesday, 5 March 2014 12:18 PM To: U2 Users List (u2-users@listserver.u2ug.org) Subject: [U2] Multi-threaded phantom processing Hi Everyone, A project that I’m currently working on requires transmission of transactional and ancillary data from one UV site to site to another. I have the communications all set up and it is working very well via the UV socket functions with a client at one end and a server at the other. Both the client and the server both run as a UV phantom processes written in UV basic and the data being transferred (also encrypted) is more or less real time with various UV file triggers on the client end creating queue items for transmission based on the data content / transaction type. This set up is a single threaded affair and I am wondering how it will cope and what options I’ll have as the volume of data increases. Multi-threading my server process to dish out some of the load might be one possibility and I’ve seen examples in the past where the last digit of a numeric key (i.e. 0-9) is used to determine which one of 10 phantoms to call thus providing an effective distribution of the work. But even then, reading/writing processing data to a file for each child process seems to me to be the only way to achieve parent - child phantom communications at either end. Is
