Hi Peter

Clearly, you have no other option than to go the direction you have. Sounds 
like you're using the correct API and approach for the task at hand and with 
the constraints you have. Sounds like you have considered various implications 
and you're constrain by UV versions and UniRPC being blocked (a good thing if 
you're connectivity to third-parties and over unsecure/public networks :))...

BTW: UV File triggers are light-weight triggers available in UV11.x - without 
the overhead and complexity of UV SQL Triggers.

The MQSeries interface is good, but it's learning curve can be steep and is 
only feasible if both sites are licensed users. I've used it a lot, but the 
underlying complexity of the MQ setup and API can be up there with TCP/IP 
sockets :(... if you're new to it all!

SQL BCI requires UniRPC access so that would be ruled out for your use as well. 

All of the higher-level APIs just save you the hassle of writing a complex 
low-level interface which could be hard to develop, debug and maintain in the 
future...

In my experience, it is difficult to optimise any interface unless you have 
some minimum, average and maximum loads or throughput requirements to be met. 
It'll be difficult to optimise the interface other than setting minimum and 
maximum phantom processes - as you're and other have already mentioned. I guess 
time will tell how it works and what tweaking (aka optimisation) will be 
required...nothing like real-world usage to sort things out...

Another potential consideration you need to consider or at least have test 
cases is loss of connectivity due to abnormal termination by the client, 
network (local/carrier/remote) or remote server. All these events may cause 
various issues that can impact UV locally, remotely and your TCP connectivity. 
Troubleshooting network and application connectivity is never easy - hence 
working through "worse case scenarios" in advance is a worthwhile - and 
somewhat very tedious exercise. Such issues have are applicable to all 
applications - of course. Such network issues impact the native UV APIs as 
well...but I am sure you'll get to know and loathe errors such as FIN_WAIT, 
etc., etc. as you get more usage and issues, etc.

Cheers,
David


-----Original Message-----
From: u2-users-boun...@listserver.u2ug.org 
[mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Peter Cheney
Sent: Friday, 7 March 2014 6:00 PM
To: 'U2 Users List'
Subject: Re: [U2] Multi-threaded phantom processing

Initially data volume will be small - probably about 20 customers per day and 
about 6 records per customer but it is likely to grow very quickly up to some 
unknown amount. We'll have to see I guess. The records per customer are likely 
to be the high volume stuff.

There is transaction logging on the remote system that I can use and am relying 
on for data integrity but I did not know about the SQL stuff until this week. 
Not sure if it would work until I can find time to play with it and see how it 
might suit us.

Some of the other constraints were: no access allowed to the remote unirpc 
port, the remote UV is below 10.3 and different from ours, neither site has a 
UV/Net licence in prod. So I think I'm pretty much stuck with file queuing and 
multiple phantoms for now at least.

I also was able to get a successful remote file process going on our test 
system while waiting for confirmation of UV/Net accessibility. It is very nice 
and might have been a nice solution too.

Then I did some experimenting with the performance requirements on the file 
triggers and found there to be too much of a lag for the triggers to fire 
directly to the other end. This would've caused unacceptable problems in 
testing not to mention going live so a file queuing process had to be devised.

In all the file queuing works quite well and can be coded for rock solid 
reliability so it has a lot of pluses. I can also fine tune the phantom loop 
times and socket timeouts on the fly at both ends once the data starts flowing 
and we get some idea of what needs changing.

Fingers, toes, arms, legs (and eyes!) all currently crossed.
Cheers
Peter





Peter Cheney
Ultracs Developer
t 07 3017 8837 | f 07 3002 8400
e peter.che...@firstmac.com.au
w firstmac.com.au


-----Original Message-----
From: u2-users-boun...@listserver.u2ug.org 
[mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Hona, David
Sent: Friday, 7 March 2014 13:41
To: U2 Users List
Subject: Re: [U2] Multi-threaded phantom processing

Hi Peter

Sounds like an interested piece of work, just a couple of questions for you:

- what sort of data volumes and update frequencies are involved?

- do you need to know if the data has been successfully committed to the remote 
UV database?

- did you consider SQL BCI or even UV/Net (read the different UV versions is an 
issue and so this may be a constraint)

Gregor mentioned SQL BCI as an alternative interface - I've used that before to 
exchange data between systems and it works well. If you had the same UV version 
on both hosts - then UV/Net is simple effect way of transmitting data between 
UV systems with only licenses required on the remote system.

With BCI you could utilise the Stored Procedure facility via an ExecDirect() 
call to push the data to the remote system in your UV file trigger.

With UV/Net you could "push" the items to be copied the by your UV file trigger 
- thus the updates being written directly onto the remote host.

BCI is free/included in UV and of course will utilise a license on the remote 
host. You need to write your own custom code. Probably a bit easier to maintain 
than any TCP/IP sockets based interface. Definitely a lot easier to 
troubleshoot ;-) ! YMMV.

UV/Net is licensed on the number user connections on the remote host.

Hence the above alternative as designed to eliminate the file-based queue - 
which in some instances by be desirable to maintain - depending on business, 
technical or operational needs/objections.

Cheers,
David


-----Original Message-----
From: u2-users-boun...@listserver.u2ug.org 
[mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Peter Cheney
Sent: Wednesday, 5 March 2014 12:18 PM
To: U2 Users List (u2-users@listserver.u2ug.org)
Subject: [U2] Multi-threaded phantom processing

Hi Everyone,

A project that I’m currently working on requires transmission of transactional 
and ancillary data from one UV site to site to another.
I have the communications all set up and it is working very well via the UV 
socket functions with a client at one end and a server at the other.
Both the client and the server both run as a UV phantom processes written in UV 
basic and the data being transferred (also encrypted) is more or less real time 
with various UV file triggers on the client end creating queue items for 
transmission based on the data content / transaction type.

This set up is a single threaded affair and I am wondering how it will cope and 
what options I’ll have as the volume of data increases.

Multi-threading my server process to dish out some of the load might be one 
possibility and I’ve seen examples in the past where the last digit of a 
numeric key (i.e. 0-9) is used to determine which one of 10 phantoms to call 
thus providing an effective distribution of the work.

But even then, reading/writing processing data to a file for each child process 
seems to me to be the only way to achieve parent <-> child phantom 
communications at either end. Is there another way perhaps? It may be that I 
will need 2 sets of phantoms, 1 for the transactions and one for the ancillary 
data. Although we run UV on AIX the only other requirement is that I must be 
able to do it within UV Basic.

Does anyone out there on the list know of a better way or any 
alternatives/advice so I can get a better appreciation of what else might be 
possible please?

Many thanks in advance,
Peter


Peter Cheney
Ultracs Developer
t 07 3017 8837 | f 07 3002 8400
e peter.che...@firstmac.com.au
w firstmac.com.au


 -------------------------------------------------------------------------------
Note:
This email (inc all attachments) is for the use of the intended recipient(s) 
only.
Privileged or confidential information may be contained in this communication. 
If you have received this email in error, please notify the sender immediately 
and then delete all copies of this message from your computer network. If you 
are not the intended recipient, you must not keep, use, disclose, copy or 
distribute this email without the author's prior permission. If you are the 
intended recipient and you do not wish to receive similar electronic messages 
from us in future, then please respond to the sender to this effect.
We have taken precautions to minimise the risk of transmitting software 
viruses, but advise you to carry out your own virus checks on this email and 
its attachments. We do not accept liability for any loss or damage caused by 
software viruses and do not represent that this transmission is free from 
viruses or other defects.
Firstmac Limited (ABN 59 094 145 963) (AFSL 290600)
 -------------------------------------------------------------------------------
_______________________________________________
U2-Users mailing list
U2-Users@listserver.u2ug.org
http://listserver.u2ug.org/mailman/listinfo/u2-users

************** IMPORTANT MESSAGE ***************************** This e-mail 
message is intended only for the addressee(s) and contains information which 
may be confidential.
If you are not the intended recipient please advise the sender by return email, 
do not use or disclose the contents, and delete the message and any attachments 
from your system. Unless specifically indicated, this email does not constitute 
formal advice or commitment by the sender or the Commonwealth Bank of Australia 
(ABN 48 123 123 124) or its subsidiaries.
We can be contacted through our web site: commbank.com.au.
If you no longer wish to receive commercial electronic messages from us, please 
reply to this e-mail by typing Unsubscribe in the subject line.
**************************************************************


_______________________________________________
U2-Users mailing list
U2-Users@listserver.u2ug.org
http://listserver.u2ug.org/mailman/listinfo/u2-users

 -------------------------------------------------------------------------------
Note: 
This email (inc all attachments) is for the use of the intended recipient(s) 
only.
Privileged or confidential information may be contained in this communication. 
If you have received this email in error, please notify the sender immediately 
and then delete all copies of this message from your computer network. If you 
are not the intended recipient, you must not keep, use, disclose, copy or 
distribute this email without the author's prior permission. If you are the 
intended recipient and you do not wish to receive similar electronic messages 
from us in future, then please respond to the sender to this effect. 
We have taken precautions to minimise the risk of transmitting software 
viruses, but advise you to carry out your own virus checks on this email and 
its attachments. We do not accept liability for any loss or damage caused by 
software viruses and do not represent that this transmission is free from 
viruses or other defects. 
Firstmac Limited (ABN 59 094 145 963) (AFSL 290600)
 -------------------------------------------------------------------------------
_______________________________________________
U2-Users mailing list
U2-Users@listserver.u2ug.org
http://listserver.u2ug.org/mailman/listinfo/u2-users

************** IMPORTANT MESSAGE *****************************       
This e-mail message is intended only for the addressee(s) and contains 
information which may be
confidential. 
If you are not the intended recipient please advise the sender by return email, 
do not use or
disclose the contents, and delete the message and any attachments from your 
system. Unless
specifically indicated, this email does not constitute formal advice or 
commitment by the sender
or the Commonwealth Bank of Australia (ABN 48 123 123 124) or its subsidiaries. 
We can be contacted through our web site: commbank.com.au. 
If you no longer wish to receive commercial electronic messages from us, please 
reply to this
e-mail by typing Unsubscribe in the subject line. 
**************************************************************


_______________________________________________
U2-Users mailing list
U2-Users@listserver.u2ug.org
http://listserver.u2ug.org/mailman/listinfo/u2-users

Reply via email to