RE: [U2] Fastest Bi-Directional data transfer btwn MV and non MV dbms
My understanding is that a raw socket, using ANY of the protocols mentioned or the home-brew options you suggest, will fail if there is nothing on the other end of the pipe. This is where the additional layers of products like MQ come into play -- sockets may form part of the plumbing, but they are far from a complete/robust solution in their own right Ross Ferris Stamina Software Visage Better by Design! -Original Message- From: [EMAIL PROTECTED] [mailto:owner-u2- [EMAIL PROTECTED] On Behalf Of Jeff Schasny Sent: Wednesday, 24 October 2007 11:28 PM To: u2-users@listserver.u2ug.org Subject: Re: [U2] Fastest Bi-Directional data transfer btwn MV and non MV dbms Sockets are just the pipe you push/pull things through. One of the features they bring to the table is that you get to (yes, I actually said that) design your own protocol for using them reliably or pick one of the already available protocols. FTP, HTTP, RCP, and telnet (and pretty much everything else in your local /etc/services file) are all sockets based protocols. Oh, and I think you could apply your statement to just about any data communications methodology and be correct. Ross Ferris wrote: And SUCK if one side happens to go down you need to resync (massive amounts of) data I believe Ross Ferris Stamina Software Visage Better by Design! -Original Message- From: [EMAIL PROTECTED] [mailto:owner-u2- [EMAIL PROTECTED] On Behalf Of Jeff Schasny Sent: Wednesday, 24 October 2007 2:36 AM To: u2-users@listserver.u2ug.org Subject: Re: [U2] Fastest Bi-Directional data transfer btwn MV and non MV dbms humorIs it just me or does it smell like troll in here all of a sudden/humor Since the question as defined in the sentence below is pretty generic I'll respond in kind. Sockets. Inter process communication across disparate platforms and applications is just what they were made for. Low overhead, high throughput, and completely neutral as to data format. Baker Hughes wrote: Hey, What is the fastest and lowest cpu overhead method of transferring data between U2/MV databases, and other data sources? [snip] -- - -- - Jeff Schasny - Denver, Co, USA jeff at schasny dot com - -- - --- u2-users mailing list u2-users@listserver.u2ug.org To unsubscribe please visit http://listserver.u2ug.org/ --- u2-users mailing list u2-users@listserver.u2ug.org To unsubscribe please visit http://listserver.u2ug.org/ -- --- - Jeff Schasny - Denver, Co, USA jeff at schasny dot com --- - --- u2-users mailing list u2-users@listserver.u2ug.org To unsubscribe please visit http://listserver.u2ug.org/ --- u2-users mailing list u2-users@listserver.u2ug.org To unsubscribe please visit http://listserver.u2ug.org/
RE: [U2] Fastest Bi-Directional data transfer btwn MV and non-MV dbms
Baker, Given the scenario you have just outlined, and my imaginings of the way that each of the 3 systems SHOULD work, you have no likelihood of deadlock collisions (this could be guaranteed with possibly relatively minor tweaks to all sides of the equation). If you want a solution, I just need a few more FACTS (guestimates AOK for numbers) - what is the database behind the WCS system - does the WCS have automated/robotic picking, manual/RF or a combination - average number of line items on a transaction originating from the Universe system - average line items for an order from the web portal - assume you want LIVE inventory on the portal (may be reasons why this is BAD, but that is another story) - peak transactions/hr from OLTP web portal Baker, I know you mean well, but I'm just questioning the need for Fastest in this scenario, unless I see some seriously LARGE numbers for some of the above :-) Ross Ferris Stamina Software Visage Better by Design! [ad] BTW, we also do applications, covering areas like web ordering, warehousing, distribution etc just for the record, and have had to tackle issues like this before [/ad] -Original Message- From: [EMAIL PROTECTED] [mailto:owner-u2- [EMAIL PROTECTED] On Behalf Of Baker Hughes Sent: Thursday, 25 October 2007 12:20 AM To: u2-users@listserver.u2ug.org Subject: RE: [U2] Fastest Bi-Directional data transfer btwn MV and non- MV dbms Ross, Yes, there is a real-world application to the question, at least one where I may try to 'sell' the solution after the theory is worked out. 3 Different systems play with the same live Inventory of products: a UniVerse based OLTP, a MS SQL db based web-order portal, and a Warehouse Control System which fills the orders and receives stock. At night we batch the daily stock receipts from WCS up to UniVerse, update the Avail to Sell qty for the OLTP and allocate Order Reserve Qty to backorders. Then UV sends the updated ATS to the web database (which is always 24 hours behind). Ross has asked the most astute question in all this, that of data collisions, where the same product is updated on 2 or 3 sides at once. This is perhaps the question that looms largest and keeps people (like us) in batch mode rather than real-time. Thanks everyone for the very worthy contributions to this science. -Baker -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Ross Ferris Sent: Wednesday, October 24, 2007 1:58 AM To: u2-users@listserver.u2ug.org Subject: RE: [U2] Fastest Bi-Directional data transfer btwn MV and non MV dbms Baker, How live and active is this bi-directional transfer likely to be? Do you need to consider the possibility of data collisions (ie: will someone change a record in your UV database that could also be changed on the other end) OR are the discrete changes somewhat atomic transactions, with no chance of duplication Are both systems running live databases? What are you REALLY trying to do (your question is nearly as big as Texas) ... if you have some specific goal in mind, then some potential road blocks may be removed (or emerge) Ross Ferris Stamina Software Visage Better by Design! --- u2-users mailing list u2-users@listserver.u2ug.org To unsubscribe please visit http://listserver.u2ug.org/ --- u2-users mailing list u2-users@listserver.u2ug.org To unsubscribe please visit http://listserver.u2ug.org/
RE: [U2] Fastest Bi-Directional data transfer btwn MV and non-MV dbms
That would be an [ad] ... but there are some people that may convince us to port Visage.DRS to UV as well -- slight tweak would go a long way to a proving parts of a solution for this scenario (but how did you know we were looking at replication to foreign databases?) Ross Ferris Stamina Software Visage Better by Design! You should ask Ross how he managed to get live data replication working with D3. g --- u2-users mailing list u2-users@listserver.u2ug.org To unsubscribe please visit http://listserver.u2ug.org/
RE: [U2] Fastest Bi-Directional data transfer btwn MV and non-MV dbms
So, now the question becomes how much (time/effort/$) is it worth to make this work -- I think you have all of the necessary pieces now clearly identified, and the information flows are somewhat obvious (though I would still like to see those data volumes) Ross Ferris Stamina Software Visage Better by Design! -Original Message- From: [EMAIL PROTECTED] [mailto:owner-u2- [EMAIL PROTECTED] On Behalf Of Baker Hughes Sent: Thursday, 25 October 2007 3:11 AM To: u2-users@listserver.u2ug.org Subject: RE: [U2] Fastest Bi-Directional data transfer btwn MV and non- MV dbms Glen, snip You always get the good ones, doncha? What does the warehouse control system use as a database? How many different O/S are we looking at here? The first idea that comes to mind is transactional queuing and inspection. A FIFO updating situation will not work, since you have logic that is updating data on all sides of the triangle, potentially at the same time. You need all 3 facets of the system to contact and update a single data store. I would recommend either another MS SQL server or a MySQL server. With UV, you can access SQL tables as MV files, right?(MV triggers to update SQL maybe) Can you get the warehouse control system to read/write from that SQL database, too? As far as MS SQL, is there a reason that will prevent you from linking the existing MS SQL table structures to a central data store that will contain the (moved) data? I don't think you're going to get out of this without some major internal tweaking, data relocation, and application code changes. /snip Well, I was actually only tasked with making the ATS more real time for our Asia sales office. Since they're entering orders during our night, they always get the short end of the stick, ATS may or may not be right; the batch updates are all timed/tuned for US timezone (and work pretty accurately heretofore). Anyway... we all hate partial solutions ... so one gets to contemplating, and one thing leads to another. I don't have enough clout around here to convince them to add yet another data store to their enterprise, but I agree with you - you must establish referential integrity to go real time with this cluster. UniVerse is presently the quasi-reference point [UniVerse is the center of our world - figure that out isaac asimov], but it's only perfectly accurate once a day. Answers to your questions: a) WCS uses an ISAM db b) Unix Windows I like your solution to the puzzle Glen. ... clear thinking as always. rgds, -Baker --- u2-users mailing list u2-users@listserver.u2ug.org To unsubscribe please visit http://listserver.u2ug.org/ --- u2-users mailing list u2-users@listserver.u2ug.org To unsubscribe please visit http://listserver.u2ug.org/
RE: [U2] Fastest Bi-Directional data transfer btwn MV and non-MV dbms
Well, I was actually only tasked with making the ATS more real time for our Asia sales office. Since they're entering orders during our night, they always get the short end of the stick, ATS may or may not be right; the batch updates are all timed/tuned for US timezone (and work pretty accurately heretofore). Anyway... we all hate partial solutions ... so one gets to contemplating, and one thing leads to another. I don't have enough clout around here to convince them to add yet another data store to their enterprise, but I agree with you - you must establish referential integrity to go real time with this cluster. UniVerse is presently the quasi-reference point [UniVerse is the center of our world - figure that out isaac asimov], but it's only perfectly accurate once a day. OK. So use the existing MS SQL server as your reference point. You don't _have_ to put another data store in the mix. I'm definately no SQL expert, but can't SQL views and stored procedures be used to blend tables and provide the proper updating and reference paths/points for the UV and the ISAM DB by themselves? The suggestion of the additional MySQL/MS SQL server was to serve as a live multi-point data store, but now that I think about it you would be able to get the same results with what you have. Answers to your questions: a) WCS uses an ISAM db b) Unix Windows I like your solution to the puzzle Glen. ... clear thinking as always. rgds, -Baker Thanks, I'm glad that you actually understood what I typed. I'm not known for clarity in my postings. g Glen --- u2-users mailing list u2-users@listserver.u2ug.org To unsubscribe please visit http://listserver.u2ug.org/
RE: [U2] Fastest Bi-Directional data transfer btwn MV and non-MV dbms
-Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] Behalf Of Ross Ferris Sent: Thursday, October 25, 2007 4:42 AM To: u2-users@listserver.u2ug.org Subject: RE: [U2] Fastest Bi-Directional data transfer btwn MV and non-MV dbms That would be an [ad] ... but there are some people that may convince us to port Visage.DRS to UV as well -- slight tweak would go a long way to a proving parts of a solution for this scenario (but how did you know we were looking at replication to foreign databases?) Hrm.. Didn't we discuss something of this nature @ Spectrum Long Beach? Albeit, there were no definate plans in place at the time. I didn't mean to provoke attention in the wrong place, if that's the case. #;) Ross Ferris Stamina Software Visage Better by Design! You should ask Ross how he managed to get live data replication working with D3. g Glen --- u2-users mailing list u2-users@listserver.u2ug.org To unsubscribe please visit http://listserver.u2ug.org/
RE: [U2] Fastest Bi-Directional data transfer btwn MV and non MV dbms [AD}
Hi Baker, In response to: make an almost convincing argument to do it on the dedicated target If the target is another shared system, then it is often not a good candidate for this type of thing. I guess what I've often seen is a dedicated processor on a workstation that was used to transfer data. If this system uses multithreading so you don't get a store-and-forward end-to-end delay then it becomes an excellent candidate for doing the transformation. In response to: I honestly raised an eyebrow at your thought that non-MV DB could transform MV data better/faster. I wouldn't say another DB. [AD]I've written extensive Java, C++ and C# (even assembler in my day) code to process MultiValue data, and if you can avoid the overhead of an immutable string problem it is possible to get really great performance. There are some interesting gotchas, but I've had to figure them out a long time ago, and I have a robust, mature library of code that I use for this.[/AD] Robert -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Baker Hughes Sent: Wednesday, October 24, 2007 3:04 PM To: u2-users@listserver.u2ug.org Subject: RE: [U2] Fastest Bi-Directional data transfer btwn MV and non MV dbms [AD} Thank you Robert and Janet. Overly kind of you Robert to take the time to distill some insights into this reply. You give more consideration to the overhead of data Transformation and make an almost convincing argument to do it on the dedicated target, assumedly something relational/non-MV. The anecdote you give is an interesting one about the benchmark attempt, which sounded half-baked by the MV programmers. I'd still be interested to see a real comparative benchmark with thorough transformation done on the MV side before jettison. [Ad] I've written and extensive ETL myself that was used to normalize/extract MV data from 27 UniData systems [due to their untimely merger-induced demise]. I even used WRITESEQ's instead of WRITEBLK and it was still extremely fast. [/Ad] Most of us have a long history of transformation if we've been doing EDI - flattening our dimensioned data into the ANSI standards. I honestly raised an eyebrow at your thought that non-MV DB could transform MV data better/faster. But you've done a good bit of it and apparently written some things to accomplish it, and I revere your experience at this. hmmm ... maybe the transformation issue (and others you've outlined to a lesser extent) is why it's such a long leap for MV-based BI tools to mash disparate data stores. Sincere regards, -Baker -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Janet Bond Sent: Wednesday, October 24, 2007 1:35 PM To: u2-users@listserver.u2ug.org Subject: RE: [U2] Fastest Bi-Directional data transfer btwn MV and non MV dbms [AD} As promised here is Robert Houben's input to your question Baker!!! :) For anyone who doesn't know me, I was the lead designer and developer of the PK Harmony product which we demoed at PC Labs at the Spectrum show in 1986 (over 20 years ago!) I've been involved in data communications since the early 1980's and I'm still intimately involved in it, so I think that I have some expertise in the matter! ;) I put the ad marker in so the moderators won't flip. I don't believe that anyone markets PK Harmony anymore (that was another company) so I shouldn't need it for that, but just in case... Also, I may accidentally reference some products that I worked on that my present company markets, so we'll have to comply! ;) What I say here can be applied to any product currently on the market. There are several factors that affect throughput and performance when transferring data between systems (any systems). I'll detail these and then go through them, with some special emphasis for how they are impacted by MultiValue processing. I use SQL Server as the example target. In some cases your target is different, but most of what I say is either still relevant or at the very least, worth thinking about: - I/O bandwidth and contention - CPU speed and contention - Disk bandwidth and contention - Synchronization - End to end latency - Transformation I/O Bandwidth and Contention: = The first thing to look at is I/O bandwidth and contention. There are products that you can get that will allow you to set up two endpoints and push data through, and measure the throughput. If you have a 10MBit LAN, you will never exceed 10 MBits. If you have a busy network, and your two endpoints need to go through multiple routers, you will undoubtedly have less than 10 MBits (or 100MBits) to work with. There is a hard limit, determined by your network environment, to how much data you can push through. Although this is not usually the most limiting factor, I've been amazed when people who had smoking throughput pushing data between two applications on the same machine, are surprised when they lose
RE: [U2] Fastest Bi-Directional data transfer btwn MV and non-MV dbms (AD)
Hello Baker, We have a customer who is processing tens of thousands of transactions a day. These transactions are centralized on SQL Server so that the Oracle ESB, UniVerse and Web Systems can share the data. The key LOB Application is on Universe, so it drives the live process. Every weekend they transfer millions of records in a large batch to ensure that everything is synchronized. The data flows both ways to SQL and Oracle. This is a Worldwide 24x7 company that is experiencing massive monthly growth, the transactions generate a serious amount of revenue. The environment needs to be fast, stable and scalable. The technology (Legacy to SQL Bridge) can access remote databases from Universe. The tables are viewed as if they are Universe files, records as items and fields as attributes. This lets Universe read, write and select data from the remote databases as if they were Universe files. On our demonstration environment here are the numbers. Using the Legacy to SQL Bridge to transfer data from SQL Server into PICK took about 1.2 seconds for 10,000 rows. Thatbs over 8,000 rows per second. Going the other ways, we were able to get, in the end, about 250 rows per second, as I recall. A better SQL Server configuration would probably have helped. These are actually very modest numbers, when you consider the configuration that was running: b Everything was running on a Lenovo Laptop: o Intel Centrino Duo o 2 GB RAM o 100 GB Disk (very full, fragmentation moderate) b Windows XP Professional b SQL Server 2005 b Microsoft Virtual PC 2004, running: o Red Hat Enterprise Linux 3: o' Universe 12 might have been 11? o' FusionWare Integration Server with the Legacy to SQL Bridge Everything was vying for CPU and I/O on one system, and we had the overhead of Microsoftbs Virtual environment (not known to be best of breed at this point). So, in an ideal tuned environment, the numbers could be much better. Then again, in a real-world environment where both your MultiValue and your SQL systems are shared, overloaded, hardworking systems, these numbers may still be about right. Hope that is useful. Janet /AD -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Ross Ferris Sent: Thursday, October 25, 2007 12:54 AM To: u2-users@listserver.u2ug.org Subject: RE: [U2] Fastest Bi-Directional data transfer btwn MV and non-MV dbms Baker, Given the scenario you have just outlined, and my imaginings of the way that each of the 3 systems SHOULD work, you have no likelihood of deadlock collisions (this could be guaranteed with possibly relatively minor tweaks to all sides of the equation). If you want a solution, I just need a few more FACTS (guestimates AOK for numbers) - what is the database behind the WCS system - does the WCS have automated/robotic picking, manual/RF or a combination - average number of line items on a transaction originating from the Universe system - average line items for an order from the web portal - assume you want LIVE inventory on the portal (may be reasons why this is BAD, but that is another story) - peak transactions/hr from OLTP web portal Baker, I know you mean well, but I'm just questioning the need for Fastest in this scenario, unless I see some seriously LARGE numbers for some of the above :-) Ross Ferris Stamina Software Visage Better by Design! [ad] BTW, we also do applications, covering areas like web ordering, warehousing, distribution etc just for the record, and have had to tackle issues like this before [/ad] -Original Message- From: [EMAIL PROTECTED] [mailto:owner-u2- [EMAIL PROTECTED] On Behalf Of Baker Hughes Sent: Thursday, 25 October 2007 12:20 AM To: u2-users@listserver.u2ug.org Subject: RE: [U2] Fastest Bi-Directional data transfer btwn MV and non- MV dbms Ross, Yes, there is a real-world application to the question, at least one where I may try to 'sell' the solution after the theory is worked out. 3 Different systems play with the same live Inventory of products: a UniVerse based OLTP, a MS SQL db based web-order portal, and a Warehouse Control System which fills the orders and receives stock. At night we batch the daily stock receipts from WCS up to UniVerse, update the Avail to Sell qty for the OLTP and allocate Order Reserve Qty to backorders. Then UV sends the updated ATS to the web database (which is always 24 hours behind). Ross has asked the most astute question in all this, that of data collisions, where the same product is updated on 2 or 3 sides at once. This is perhaps the question that looms largest and keeps people (like us) in batch mode rather than real-time. Thanks everyone for the very worthy contributions to this science. -Baker -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Ross Ferris Sent: Wednesday, October 24, 2007 1:58 AM To: u2-users
RE: [U2] Fastest Bi-Directional data transfer btwn MV and non-MV dbms (AD)
Robert, Janet, Thanks for these mind expanding contributions to the discussion. I rejoice with you at these successes, as we do for all our MV colleagues. I trust I and others will benefit from your insights as we get the opportunity, with similarly challenging projects, hopefully in the not too distant future. And it wouldn't be a bad thing if someone gave your product a hard look when they are faced with that project... ;-) -Baker -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Janet Bond Sent: Thursday, October 25, 2007 1:22 PM To: u2-users@listserver.u2ug.org Subject: RE: [U2] Fastest Bi-Directional data transfer btwn MV and non-MV dbms (AD) Hello Baker, We have a customer who is processing tens of thousands of transactions a day. These transactions are centralized on SQL Server so that the Oracle ESB, UniVerse and Web Systems can share the data. The key LOB Application is on Universe, so it drives the live process. Every weekend they transfer millions of records in a large batch to ensure that everything is synchronized. The data flows both ways to SQL and Oracle. This is a Worldwide 24x7 company that is experiencing massive monthly growth, the transactions generate a serious amount of revenue. The environment needs to be fast, stable and scalable. The technology (Legacy to SQL Bridge) can access remote databases from Universe. The tables are viewed as if they are Universe files, records as items and fields as attributes. This lets Universe read, write and select data from the remote databases as if they were Universe files. On our demonstration environment here are the numbers. Using the Legacy to SQL Bridge to transfer data from SQL Server into PICK took about 1.2 seconds for 10,000 rows. Thatbs over 8,000 rows per second. Going the other ways, we were able to get, in the end, about 250 rows per second, as I recall. A better SQL Server configuration would probably have helped. These are actually very modest numbers, when you consider the configuration that was running: b Everything was running on a Lenovo Laptop: o Intel Centrino Duo o 2 GB RAM o 100 GB Disk (very full, fragmentation moderate) b Windows XP Professional b SQL Server 2005 b Microsoft Virtual PC 2004, running: o Red Hat Enterprise Linux 3: o' Universe 12 might have been 11? o' FusionWare Integration Server with the Legacy to SQL Bridge Everything was vying for CPU and I/O on one system, and we had the overhead of Microsoftbs Virtual environment (not known to be best of breed at this point). So, in an ideal tuned environment, the numbers could be much better. Then again, in a real-world environment where both your MultiValue and your SQL systems are shared, overloaded, hardworking systems, these numbers may still be about right. Hope that is useful. Janet /AD -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Ross Ferris Sent: Thursday, October 25, 2007 12:54 AM To: u2-users@listserver.u2ug.org Subject: RE: [U2] Fastest Bi-Directional data transfer btwn MV and non-MV dbms Baker, Given the scenario you have just outlined, and my imaginings of the way that each of the 3 systems SHOULD work, you have no likelihood of deadlock collisions (this could be guaranteed with possibly relatively minor tweaks to all sides of the equation). If you want a solution, I just need a few more FACTS (guestimates AOK for numbers) - what is the database behind the WCS system - does the WCS have automated/robotic picking, manual/RF or a combination - average number of line items on a transaction originating from the Universe system - average line items for an order from the web portal - assume you want LIVE inventory on the portal (may be reasons why this is BAD, but that is another story) - peak transactions/hr from OLTP web portal Baker, I know you mean well, but I'm just questioning the need for Fastest in this scenario, unless I see some seriously LARGE numbers for some of the above :-) Ross Ferris Stamina Software Visage Better by Design! [ad] BTW, we also do applications, covering areas like web ordering, warehousing, distribution etc just for the record, and have had to tackle issues like this before [/ad] -Original Message- From: [EMAIL PROTECTED] [mailto:owner-u2- [EMAIL PROTECTED] On Behalf Of Baker Hughes Sent: Thursday, 25 October 2007 12:20 AM To: u2-users@listserver.u2ug.org Subject: RE: [U2] Fastest Bi-Directional data transfer btwn MV and non- MV dbms Ross, Yes, there is a real-world application to the question, at least one where I may try to 'sell' the solution after the theory is worked out. 3 Different systems play with the same live Inventory of products: a UniVerse based OLTP, a MS SQL db based web-order portal, and a Warehouse Control System which fills the orders and receives stock. At night we batch the daily stock receipts from WCS up
RE: [U2] Fastest Bi-Directional data transfer btwn MV and non-MV dbms
Glen, Good thoughts here as always, thanks. Right now, the UV db is not fully replicated to the MSSQL tables, rather the other way around (for the most part). The ISAM WCS tables also update UV. Therefore, at present, the UV db comes closest to being the reference, and it still runs the business. The main thing, as mentioned, is trying to bring some batch processes into real time, especially those that hinder global commerce. Some of this will have to be resolved with code, others may be helped by some of the good contributions to this thread. thanks much, -Baker snip OK. So use the existing MS SQL server as your reference point. You don't _have_ to put another data store in the mix. I'm definately no SQL expert, but can't SQL views and stored procedures be used to blend tables and provide the proper updating and reference paths/points for the UV and the ISAM DB by themselves? The suggestion of the additional MySQL/MS SQL server was to serve as a live multi-point data store, but now that I think about it you would be able to get the same results with what you have. /snip --- u2-users mailing list u2-users@listserver.u2ug.org To unsubscribe please visit http://listserver.u2ug.org/
RE: [U2] Fastest Bi-Directional data transfer btwn MV and non MV dbms
And SUCK if one side happens to go down you need to resync (massive amounts of) data I believe Ross Ferris Stamina Software Visage Better by Design! -Original Message- From: [EMAIL PROTECTED] [mailto:owner-u2- [EMAIL PROTECTED] On Behalf Of Jeff Schasny Sent: Wednesday, 24 October 2007 2:36 AM To: u2-users@listserver.u2ug.org Subject: Re: [U2] Fastest Bi-Directional data transfer btwn MV and non MV dbms humorIs it just me or does it smell like troll in here all of a sudden/humor Since the question as defined in the sentence below is pretty generic I'll respond in kind. Sockets. Inter process communication across disparate platforms and applications is just what they were made for. Low overhead, high throughput, and completely neutral as to data format. Baker Hughes wrote: Hey, What is the fastest and lowest cpu overhead method of transferring data between U2/MV databases, and other data sources? [snip] -- --- - Jeff Schasny - Denver, Co, USA jeff at schasny dot com --- - --- u2-users mailing list u2-users@listserver.u2ug.org To unsubscribe please visit http://listserver.u2ug.org/ --- u2-users mailing list u2-users@listserver.u2ug.org To unsubscribe please visit http://listserver.u2ug.org/
RE: [U2] Fastest Bi-Directional data transfer btwn MV and non MV dbms
Baker, How live and active is this bi-directional transfer likely to be? Do you need to consider the possibility of data collisions (ie: will someone change a record in your UV database that could also be changed on the other end) OR are the discrete changes somewhat atomic transactions, with no chance of duplication Are both systems running live databases? What are you REALLY trying to do (your question is nearly as big as Texas) ... if you have some specific goal in mind, then some potential road blocks may be removed (or emerge) Ross Ferris Stamina Software Visage Better by Design! -Original Message- From: [EMAIL PROTECTED] [mailto:owner-u2- [EMAIL PROTECTED] On Behalf Of Baker Hughes Sent: Wednesday, 24 October 2007 5:05 AM To: u2-users@listserver.u2ug.org Subject: RE: [U2] Fastest Bi-Directional data transfer btwn MV and non MV dbms Thanks Jerry for the introduction. Pleased to meet you Jeff. I don't post everyday, sort of ebbs and flows with workload, so you may not have seen me before... and I probably post more questions than answers ... just glad to be a part. humor sort of refreshing to be called a troll, I stand 6'8 and usually draw other names from folks grin/humor Jeff - your response about sockets, along with Kevins, are noted. Thanks. Have a great day, -Baker Hughes -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Jerry Banker Sent: Tuesday, October 23, 2007 12:10 PM To: u2-users@listserver.u2ug.org Subject: RE: [U2] Fastest Bi-Directional data transfer btwn MV and non MV dbms Not likely, Baker Hughes has been in the PICK arena for many moons and is one of the founders of the Texas users group. -Original Message- From: Jeff Schasny [mailto:[EMAIL PROTECTED] humorIs it just me or does it smell like troll in here all of a sudden/humor --- u2-users mailing list u2-users@listserver.u2ug.org To unsubscribe please visit http://listserver.u2ug.org/ --- u2-users mailing list u2-users@listserver.u2ug.org To unsubscribe please visit http://listserver.u2ug.org/ --- u2-users mailing list u2-users@listserver.u2ug.org To unsubscribe please visit http://listserver.u2ug.org/
Re: [U2] Fastest Bi-Directional data transfer btwn MV and non MV dbms
Sockets are just the pipe you push/pull things through. One of the features they bring to the table is that you get to (yes, I actually said that) design your own protocol for using them reliably or pick one of the already available protocols. FTP, HTTP, RCP, and telnet (and pretty much everything else in your local /etc/services file) are all sockets based protocols. Oh, and I think you could apply your statement to just about any data communications methodology and be correct. Ross Ferris wrote: And SUCK if one side happens to go down you need to resync (massive amounts of) data I believe Ross Ferris Stamina Software Visage Better by Design! -Original Message- From: [EMAIL PROTECTED] [mailto:owner-u2- [EMAIL PROTECTED] On Behalf Of Jeff Schasny Sent: Wednesday, 24 October 2007 2:36 AM To: u2-users@listserver.u2ug.org Subject: Re: [U2] Fastest Bi-Directional data transfer btwn MV and non MV dbms humorIs it just me or does it smell like troll in here all of a sudden/humor Since the question as defined in the sentence below is pretty generic I'll respond in kind. Sockets. Inter process communication across disparate platforms and applications is just what they were made for. Low overhead, high throughput, and completely neutral as to data format. Baker Hughes wrote: Hey, What is the fastest and lowest cpu overhead method of transferring data between U2/MV databases, and other data sources? [snip] -- --- - Jeff Schasny - Denver, Co, USA jeff at schasny dot com --- - --- u2-users mailing list u2-users@listserver.u2ug.org To unsubscribe please visit http://listserver.u2ug.org/ --- u2-users mailing list u2-users@listserver.u2ug.org To unsubscribe please visit http://listserver.u2ug.org/ -- Jeff Schasny - Denver, Co, USA jeff at schasny dot com --- u2-users mailing list u2-users@listserver.u2ug.org To unsubscribe please visit http://listserver.u2ug.org/
RE: [U2] Fastest Bi-Directional data transfer btwn MV and non-MV dbms
Ross, Yes, there is a real-world application to the question, at least one where I may try to 'sell' the solution after the theory is worked out. 3 Different systems play with the same live Inventory of products: a UniVerse based OLTP, a MS SQL db based web-order portal, and a Warehouse Control System which fills the orders and receives stock. At night we batch the daily stock receipts from WCS up to UniVerse, update the Avail to Sell qty for the OLTP and allocate Order Reserve Qty to backorders. Then UV sends the updated ATS to the web database (which is always 24 hours behind). Ross has asked the most astute question in all this, that of data collisions, where the same product is updated on 2 or 3 sides at once. This is perhaps the question that looms largest and keeps people (like us) in batch mode rather than real-time. Thanks everyone for the very worthy contributions to this science. -Baker -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Ross Ferris Sent: Wednesday, October 24, 2007 1:58 AM To: u2-users@listserver.u2ug.org Subject: RE: [U2] Fastest Bi-Directional data transfer btwn MV and non MV dbms Baker, How live and active is this bi-directional transfer likely to be? Do you need to consider the possibility of data collisions (ie: will someone change a record in your UV database that could also be changed on the other end) OR are the discrete changes somewhat atomic transactions, with no chance of duplication Are both systems running live databases? What are you REALLY trying to do (your question is nearly as big as Texas) ... if you have some specific goal in mind, then some potential road blocks may be removed (or emerge) Ross Ferris Stamina Software Visage Better by Design! --- u2-users mailing list u2-users@listserver.u2ug.org To unsubscribe please visit http://listserver.u2ug.org/
RE: [U2] Fastest Bi-Directional data transfer btwn MV and non-MV dbms
-Original Message- From: [EMAIL PROTECTED] [mailto:owner-u2- [EMAIL PROTECTED] On Behalf Of Baker Hughes Sent: Wednesday, October 24, 2007 10:20 AM To: u2-users@listserver.u2ug.org Subject: RE: [U2] Fastest Bi-Directional data transfer btwn MV and non-MV dbms Ross, Yes, there is a real-world application to the question, at least one where I may try to 'sell' the solution after the theory is worked out. 3 Different systems play with the same live Inventory of products: a UniVerse based OLTP, a MS SQL db based web-order portal, and a Warehouse Control System which fills the orders and receives stock. At night we batch the daily stock receipts from WCS up to UniVerse, update the Avail to Sell qty for the OLTP and allocate Order Reserve Qty to backorders. Then UV sends the updated ATS to the web database (which is always 24 hours behind). Baker, You always get the good ones, doncha? What does the warehouse control system use as a database? How many different O/S are we looking at here? The first idea that comes to mind is transactional queuing and inspection. A FIFO updating situation will not work, since you have logic that is updating data on all sides of the triangle, potentially at the same time. You need all 3 facets of the system to contact and update a single data store. I would recommend either another MS SQL server or a MySQL server. With UV, you can access SQL tables as MV files, right?(MV triggers to update SQL maybe) Can you get the warehouse control system to read/write from that SQL database, too? As far as MS SQL, is there a reason that will prevent you from linking the existing MS SQL table structures to a central data store that will contain the (moved) data? I don't think you're going to get out of this without some major internal tweaking, data relocation, and application code changes. Ross has asked the most astute question in all this, that of data collisions, where the same product is updated on 2 or 3 sides at once. This is perhaps the question that looms largest and keeps people (like us) in batch mode rather than real-time. You should ask Ross how he managed to get live data replication working with D3. g [chop] Glen Batchelor IT Director All-Spec Industries phone: (910) 332-0424 fax: (910) 763-5664 E-mail: [EMAIL PROTECTED] Web: http://www.all-spec.com Blog: http://blog.all-spec.com --- u2-users mailing list u2-users@listserver.u2ug.org To unsubscribe please visit http://listserver.u2ug.org/
RE: [U2] Fastest Bi-Directional data transfer btwn MV and non-MV dbms
Glen, snip You always get the good ones, doncha? What does the warehouse control system use as a database? How many different O/S are we looking at here? The first idea that comes to mind is transactional queuing and inspection. A FIFO updating situation will not work, since you have logic that is updating data on all sides of the triangle, potentially at the same time. You need all 3 facets of the system to contact and update a single data store. I would recommend either another MS SQL server or a MySQL server. With UV, you can access SQL tables as MV files, right?(MV triggers to update SQL maybe) Can you get the warehouse control system to read/write from that SQL database, too? As far as MS SQL, is there a reason that will prevent you from linking the existing MS SQL table structures to a central data store that will contain the (moved) data? I don't think you're going to get out of this without some major internal tweaking, data relocation, and application code changes. /snip Well, I was actually only tasked with making the ATS more real time for our Asia sales office. Since they're entering orders during our night, they always get the short end of the stick, ATS may or may not be right; the batch updates are all timed/tuned for US timezone (and work pretty accurately heretofore). Anyway... we all hate partial solutions ... so one gets to contemplating, and one thing leads to another. I don't have enough clout around here to convince them to add yet another data store to their enterprise, but I agree with you - you must establish referential integrity to go real time with this cluster. UniVerse is presently the quasi-reference point [UniVerse is the center of our world - figure that out isaac asimov], but it's only perfectly accurate once a day. Answers to your questions: a) WCS uses an ISAM db b) Unix Windows I like your solution to the puzzle Glen. ... clear thinking as always. rgds, -Baker --- u2-users mailing list u2-users@listserver.u2ug.org To unsubscribe please visit http://listserver.u2ug.org/
RE: [U2] Fastest Bi-Directional data transfer btwn MV and non MV dbms [AD}
their MultiValue data, and pushed it raw to a file on disk at the other end. Then they tried to compare that to what we were doing. The problem with that approach was that they had MultiValues and SubValue marks, they had dates, times, masked decimals and other unusual constructs that were meaningless to any non-MultiValue target that they could have chosen. Needless to say, their home-grown benchmark app outperformed our product. It also happened to be a meaningless comparison. [/AD] Someone has to process the MultiValues, SubValues and data types. Doing it in BASIC, which on all MultiValue systems is a stack-based language has performance issues associated with it. If you are familiar with the Immutable string issue in Java and .NET and the reason why you use StringBuilder or StringBuffer classes to process changing strings in these languages, MultiValue BASIC actually has the same issue under the covers. It also garbage collects, so the comparison is amazingly accurate. Doing this on the MultiValue side causes performance problems. Evolution of MultiValue Data Transfer: == So, in the evolution of data transfer products that I've been involved in over the years, a number of milestones have been reached, and these are some of them: Serial I/O Replaced with TCP/IP: The original PK Harmony (and even original ODBC) products allowed you to use Serial I/O to communicate with the MultiValue systems. In many cases, that was the only available way at the time. There were problems with buffer sizes, and lossy boundaries in Serial I/O, that required you to have an error correcting packeting structure at both ends. This meant that you were doing this type of stuff in MultiValue/BASIC. Yuck!!! The move to TCP/IP for communications allowed us to stop worrying about these things and just stream the data out with minimal packeting structure. ANSI SQL: = Relational products require a relational engine. That engine must reside on the database. The transformation effort of taking a complex ANSI compliant SQL statement and translating it to run *correctly* on a MultiValue system often overshadows all other performance characteristics. Some products in the past have taken shortcuts. These shortcuts result in SQL Statements that return inconsistent results, depending on the fields you reference (MultiValue/SubValue counts change). If you don't take the shortcuts, you get hit with performance. Sometimes you just can't win... :( Shared Resources vs. Dedicated: === [AD]We finally made a decision to produce a product set that did not require ANSI SQL, that allowed us to push the raw data and a metadata record (from our mapping tool) to the dedicated resource, so that the dedicated resource could do the heavy lifting. This was our Direct product set. We feel that this hits the sweet spot.[/AD] The Sweet Spot: === Over my more-than 20 years of MultiValue data communications, I've come to see a certain set of characteristics as a sweet spot. Here, for what it's worth, are those characteristics of a data transfer solution: - Favor dedicated resources to shared - Do transformation on the dedicated resource - Streaming I/O using transport layer - As little packeting structure as possible - Avoid imposing ANSI SQL on MultiValue - recognize the differences and get over them - Think about synchronization issues - they may be unavoidable, but where they aren't they can cost you big time - Use multi-threading to mitigate end-to-end delay Robert Houben CTO Logo: FusionWare Corporation - Enterprise Service Bus (ESB), Service-Oriented Architecture (SOA) 604-633-9891 #158 mailto:[EMAIL PROTECTED] http://www.fusionware.net /AD -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Baker Hughes Sent: Tuesday, October 23, 2007 12:15 PM To: u2-users@listserver.u2ug.org Subject: RE: [U2] Fastest Bi-Directional data transfer btwn MV and non MV dbms Janet, snip/ I can setup a conference call with one of Developers. We have been in the transferring MultiValue data to other data sources since the early 80's (PK Harmony to start with, anyone remember). We may have some good input for you. /snip I'm not in a position to buy anything, really just trying to think through the questions posted. It would be lovely to have your developer join the thread and describe how PKH/FW does it's magic. Not expecting him to share code, of course, just a few thoughts about your approach is all. Sorry to draw you into the cross fire, that's why I said what I did about ads; maybe I should've put it at the top though. sincere regards, -Baker --- u2-users mailing list u2-users@listserver.u2ug.org To unsubscribe please visit http://listserver.u2ug.org/ --- u2-users mailing list u2-users@listserver.u2ug.org To unsubscribe please visit http
RE: [U2] Fastest Bi-Directional data transfer btwn MV and non MV dbms [AD}
Thank you Robert and Janet. Overly kind of you Robert to take the time to distill some insights into this reply. You give more consideration to the overhead of data Transformation and make an almost convincing argument to do it on the dedicated target, assumedly something relational/non-MV. The anecdote you give is an interesting one about the benchmark attempt, which sounded half-baked by the MV programmers. I'd still be interested to see a real comparative benchmark with thorough transformation done on the MV side before jettison. [Ad] I've written and extensive ETL myself that was used to normalize/extract MV data from 27 UniData systems [due to their untimely merger-induced demise]. I even used WRITESEQ's instead of WRITEBLK and it was still extremely fast. [/Ad] Most of us have a long history of transformation if we've been doing EDI - flattening our dimensioned data into the ANSI standards. I honestly raised an eyebrow at your thought that non-MV DB could transform MV data better/faster. But you've done a good bit of it and apparently written some things to accomplish it, and I revere your experience at this. hmmm ... maybe the transformation issue (and others you've outlined to a lesser extent) is why it's such a long leap for MV-based BI tools to mash disparate data stores. Sincere regards, -Baker -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Janet Bond Sent: Wednesday, October 24, 2007 1:35 PM To: u2-users@listserver.u2ug.org Subject: RE: [U2] Fastest Bi-Directional data transfer btwn MV and non MV dbms [AD} As promised here is Robert Houben's input to your question Baker!!! :) For anyone who doesn't know me, I was the lead designer and developer of the PK Harmony product which we demoed at PC Labs at the Spectrum show in 1986 (over 20 years ago!) I've been involved in data communications since the early 1980's and I'm still intimately involved in it, so I think that I have some expertise in the matter! ;) I put the ad marker in so the moderators won't flip. I don't believe that anyone markets PK Harmony anymore (that was another company) so I shouldn't need it for that, but just in case... Also, I may accidentally reference some products that I worked on that my present company markets, so we'll have to comply! ;) What I say here can be applied to any product currently on the market. There are several factors that affect throughput and performance when transferring data between systems (any systems). I'll detail these and then go through them, with some special emphasis for how they are impacted by MultiValue processing. I use SQL Server as the example target. In some cases your target is different, but most of what I say is either still relevant or at the very least, worth thinking about: - I/O bandwidth and contention - CPU speed and contention - Disk bandwidth and contention - Synchronization - End to end latency - Transformation I/O Bandwidth and Contention: = The first thing to look at is I/O bandwidth and contention. There are products that you can get that will allow you to set up two endpoints and push data through, and measure the throughput. If you have a 10MBit LAN, you will never exceed 10 MBits. If you have a busy network, and your two endpoints need to go through multiple routers, you will undoubtedly have less than 10 MBits (or 100MBits) to work with. There is a hard limit, determined by your network environment, to how much data you can push through. Although this is not usually the most limiting factor, I've been amazed when people who had smoking throughput pushing data between two applications on the same machine, are surprised when they lose a ton of performance when they move one of these application to another system and they suddenly run into a bottleneck on the network. CPU Speed and Contention: = The other thing to consider is CPU speed and contention. On a typical MultiValue system, you will find yourself disk constrained, but if you are doing a lot of transformation (we'll look at that later) then you may find that this is a limiting factor. The other thing to consider is that whenever you can push processing from a shared CPU resource (your MultiValue system) to a dedicated resource (the client's desktop), you can significantly increase performance. Disk Bandwidth and Contention: == Next up is Disk bandwidth and contention. This can be a hugely significant factor. If you look at most OLTP type, MultiValue applications, you will see that the CPU sits mostly idle (seems over the years to average about 10%). Not all of this is file access, BTW, in many cases what you are encountering is context switches and internal program space being managed in virtual memory. Again, as with CPU, moving as much of that from the shared resource to the dedicated resource as you can will ALWAYS be a good thing for performance
Re: [U2] Fastest Bi-Directional data transfer btwn MV and non MV dbms
humorIs it just me or does it smell like troll in here all of a sudden/humor Since the question as defined in the sentence below is pretty generic I'll respond in kind. Sockets. Inter process communication across disparate platforms and applications is just what they were made for. Low overhead, high throughput, and completely neutral as to data format. Baker Hughes wrote: Hey, What is the fastest and lowest cpu overhead method of transferring data between U2/MV databases, and other data sources? [snip] -- Jeff Schasny - Denver, Co, USA jeff at schasny dot com --- u2-users mailing list u2-users@listserver.u2ug.org To unsubscribe please visit http://listserver.u2ug.org/
RE: [U2] Fastest Bi-Directional data transfer btwn MV and non MV dbms
Baker, There is a happy medium between CPU overhead and bandwidth usage. If you use compression of some kind, then CPU overhead is going to be higher than an uncompressed stream. Cached disk writes are normally going to be faster than network writes, unless you have a fiber backbone or dedicated gigabit link. I can't give benchmarks for any of the integration I've done here, but I can say that: 1) XML is really wasteful all-around, but it can make your data considerably more portable. 2) SQL is a norm these days and some servers optimize on the query language and storage mechanism relating to it. MySQL is quite fast, in my limited experience. I can't afford to play with MS SQL Server. SQL implementations in MV may still leave a lot to be desired, though. 3) ASCII data from MV can be highly compressed, but once again there's the CPU usage. Raw/compressed ASCII is extremely portable, if the means and will are there to implement a custom communication protocol. (see my unfinished RFC for a stepping stone [http://mvdevcentral.com/MV-RFC.txt]) Some people may think I'm wierd, but I'm quite fond of SMTP services for large-block data transferring. It's not exactly real-time bi-directional, so I'm not sure if it fits your question. You decide. MTA servers like Postfix were written with efficiency in mind and have built-in routing logic and access control(mail boxes and aliases). A pop client can easily be written in most BASIC flavors, or a CLI client can be downloaded for free, which you can use to obtain new mail. MIME envelopes don't have to always contain text, photos, and HTML. They can contain a compressed image of an entire file's worth of MV data(within mail storage limitations) or the last 5 minute's transactional changes. Food for thought, even though I didn't exactly spell out a solution. Glen Batchelor IT Director All-Spec Industries phone: (910) 332-0424 fax: (910) 763-5664 E-mail: [EMAIL PROTECTED] Web: http://www.all-spec.com Blog: http://blog.all-spec.com -Original Message- From: [EMAIL PROTECTED] [mailto:owner-u2- [EMAIL PROTECTED] On Behalf Of Baker Hughes Sent: Tuesday, October 23, 2007 11:58 AM To: u2-users@listserver.u2ug.org Subject: [U2] Fastest Bi-Directional data transfer btwn MV and non MV dbms Hey, What is the fastest and lowest cpu overhead method of transferring data between U2/MV databases, and other data sources? Assumed: a) no restrictions on underlying OS - whichever best facilitates your proposed method. b) no restrictions on Database of choice - I know this is a U2 list and we all have commitments to it, but if someone else (QM, Cache, D3...) is doing something that we ought to be doing in U2, name the db and feat they are accomplishing. C) Production system - the system must also support your OLTP users, not dedicated to data serving The primary concern is throughput, screaming fast throughput. Dogs that won't hunt: 1) if you want to take a potshot at methods that are in your estimation - tired dogs - take aim. 2) beneficent and ruthless honesty - so that this doesn't descend into a religious war please be honest yet charitable. 3) No Ads - don't respond with [just] a product name, tell what it does, the underlying method / technology We are after the technically superior destination, emotions aside, barring past development investments. I'm trying to take a clean white board approach to this question and appreciate your help in answering it. A few to consider [just as starters]: i) old jacks - WRITE/READBLK from MV triggered by whatever signal/method ii) Ajax - Async jscript Xml iii) ODBC / JDBC Thanks so much, -Baker --- u2-users mailing list u2-users@listserver.u2ug.org To unsubscribe please visit http://listserver.u2ug.org/ --- u2-users mailing list u2-users@listserver.u2ug.org To unsubscribe please visit http://listserver.u2ug.org/
Re: [U2] Fastest Bi-Directional data transfer btwn MV and non MV dbms
Sockets. If you have a consumer that can accept a socket connection, I believe that'll provide the most efficient throughput. --- u2-users mailing list u2-users@listserver.u2ug.org To unsubscribe please visit http://listserver.u2ug.org/
RE: [U2] Fastest Bi-Directional data transfer btwn MV and non MV dbms
Not likely, Baker Hughes has been in the PICK arena for many moons and is one of the founders of the Texas users group. -Original Message- From: Jeff Schasny [mailto:[EMAIL PROTECTED] humorIs it just me or does it smell like troll in here all of a sudden/humor --- u2-users mailing list u2-users@listserver.u2ug.org To unsubscribe please visit http://listserver.u2ug.org/
RE: [U2] Fastest Bi-Directional data transfer btwn MV and non MV dbms [AD]
Hello Baker, Please email me. I can setup a conference call with one of Developers. We have been in the transferring MultiValue data to other data sources since the early 80's (PK Harmony to start with, anyone remember). We may have some good input for you. Thanks, Janet Bond FusionWare Corporation Sales Operation Manager 1.866.266.2326 x159 -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Baker Hughes Sent: Tuesday, October 23, 2007 8:58 AM To: u2-users@listserver.u2ug.org Subject: [U2] Fastest Bi-Directional data transfer btwn MV and non MV dbms Hey, What is the fastest and lowest cpu overhead method of transferring data between U2/MV databases, and other data sources? Assumed: a) no restrictions on underlying OS - whichever best facilitates your proposed method. b) no restrictions on Database of choice - I know this is a U2 list and we all have commitments to it, but if someone else (QM, Cache, D3...) is doing something that we ought to be doing in U2, name the db and feat they are accomplishing. C) Production system - the system must also support your OLTP users, not dedicated to data serving The primary concern is throughput, screaming fast throughput. Dogs that won't hunt: 1) if you want to take a potshot at methods that are in your estimation - tired dogs - take aim. 2) beneficent and ruthless honesty - so that this doesn't descend into a religious war please be honest yet charitable. 3) No Ads - don't respond with [just] a product name, tell what it does, the underlying method / technology We are after the technically superior destination, emotions aside, barring past development investments. I'm trying to take a clean white board approach to this question and appreciate your help in answering it. A few to consider [just as starters]: i) old jacks - WRITE/READBLK from MV triggered by whatever signal/method ii) Ajax - Async jscript Xml iii) ODBC / JDBC Thanks so much, -Baker --- u2-users mailing list u2-users@listserver.u2ug.org To unsubscribe please visit http://listserver.u2ug.org/ --- u2-users mailing list u2-users@listserver.u2ug.org To unsubscribe please visit http://listserver.u2ug.org/
Re: [U2] Fastest Bi-Directional data transfer btwn MV and non MV dbms [AD]
Janet, When posting as a vendor or service provider, please use [AD] brackets [/AD] in your response as a courtesy to others. - Charles Barouch, Moderator Janet Bond wrote: I can setup a conference call with one of Developers. We have been in the transferring MultiValue data to other data sources since the early 80's (PK Harmony to start with, anyone remember). We may have some good input for you. --- u2-users mailing list u2-users@listserver.u2ug.org To unsubscribe please visit http://listserver.u2ug.org/
RE: [U2] Fastest Bi-Directional data transfer btwn MV and non MV dbms [AD]
I added it in the Subject should it be somewhere else? Please accept my apologies if I have offended anyone. Janet -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Moderator Sent: Tuesday, October 23, 2007 11:15 AM To: u2-users@listserver.u2ug.org Subject: Re: [U2] Fastest Bi-Directional data transfer btwn MV and non MV dbms [AD] Janet, When posting as a vendor or service provider, please use [AD] brackets [/AD] in your response as a courtesy to others. - Charles Barouch, Moderator Janet Bond wrote: I can setup a conference call with one of Developers. We have been in the transferring MultiValue data to other data sources since the early 80's (PK Harmony to start with, anyone remember). We may have some good input for you. --- u2-users mailing list u2-users@listserver.u2ug.org To unsubscribe please visit http://listserver.u2ug.org/ --- u2-users mailing list u2-users@listserver.u2ug.org To unsubscribe please visit http://listserver.u2ug.org/
RE: [U2] Fastest Bi-Directional data transfer btwn MV and non MV dbms
Thanks Jerry for the introduction. Pleased to meet you Jeff. I don't post everyday, sort of ebbs and flows with workload, so you may not have seen me before... and I probably post more questions than answers ... just glad to be a part. humor sort of refreshing to be called a troll, I stand 6'8 and usually draw other names from folks grin/humor Jeff - your response about sockets, along with Kevins, are noted. Thanks. Have a great day, -Baker Hughes -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Jerry Banker Sent: Tuesday, October 23, 2007 12:10 PM To: u2-users@listserver.u2ug.org Subject: RE: [U2] Fastest Bi-Directional data transfer btwn MV and non MV dbms Not likely, Baker Hughes has been in the PICK arena for many moons and is one of the founders of the Texas users group. -Original Message- From: Jeff Schasny [mailto:[EMAIL PROTECTED] humorIs it just me or does it smell like troll in here all of a sudden/humor --- u2-users mailing list u2-users@listserver.u2ug.org To unsubscribe please visit http://listserver.u2ug.org/ --- u2-users mailing list u2-users@listserver.u2ug.org To unsubscribe please visit http://listserver.u2ug.org/
RE: [U2] Fastest Bi-Directional data transfer btwn MV and non MV dbms
Janet, snip/ I can setup a conference call with one of Developers. We have been in the transferring MultiValue data to other data sources since the early 80's (PK Harmony to start with, anyone remember). We may have some good input for you. /snip I'm not in a position to buy anything, really just trying to think through the questions posted. It would be lovely to have your developer join the thread and describe how PKH/FW does it's magic. Not expecting him to share code, of course, just a few thoughts about your approach is all. Sorry to draw you into the cross fire, that's why I said what I did about ads; maybe I should've put it at the top though. sincere regards, -Baker --- u2-users mailing list u2-users@listserver.u2ug.org To unsubscribe please visit http://listserver.u2ug.org/
RE: [U2] Fastest Bi-Directional data transfer btwn MV and non MV dbms
What is the fastest and lowest cpu overhead method of transferring data between U2/MV databases, and other data sources? Our esteemed colleague and moderator Chuck Barouch has/had a product called Zeus which does/did transfers like this. If he can avoid self-censorship for a moment I would invite him to respond to this inquiry with his own experience in this area. As time permits I have been writing demos to show an MV DBMS populating SQL Server. As I said in another post today, the communications pipe is unimportant as is the DBMS. The same demo can be used for any MV DBMS and for any RDBMS, all someone needs to do is change the SQL query or stored procedure, and match the RDBMS fields to fields retrieved from the MV side. It comes down to a simple bit of X-to-Y mapping of fields and ensuring data types are properly managed. I'm sorry that I do not yet have a video of this on our website but I did show this demo and others along with code in my recent presentation to SAPUG. Maybe I can kick off a discussion of numbers: Without any optimization and on an over-burdened 2GHz laptop I'm getting about 200 records per second extracted from MV and Inserted into SQL Server. SQL Updates on the same data is roughly about 1/2 that. Amongst other business responsibilities I'm trying to find time to optimize the queries on both sides and run on a system which is more likely to generate production-quality numbers. I have a 3GHz dual core Athlon that should yield some good numbers soon. I'd appreciate it if someone else can provide some other realistic numbers so that I have a goal for acceptability. As a low-priority development project, and driven by common requests we find here, we have a package which moves data from any MV environment to any RDBMS. It allows for construction of a query string for about 15 different relational sources and that X/Y mapping of fields and data types. I wish this were further along so that I could provide some numbers about transfers from U2 to Excel, QM to Oracle, D3 to DB2, Reality to Interbase, etc. I would welcome discussion with someone who can help to drive this development. minor_rant If that's too subtle, and at the risk of soliciting, we can't proactively develop tools like this unless we get financing, and you only get people to fund projects if you ask. That's not an ad, it's a fact of business. More solutions would be available for common problems if this community were more willing to match supply/talent with the demand of ongoing business needs. As it is, for years we continue to see the same data transfer problems presented every week as though it's the first time someone is asking the question. / Tony Gravagno Nebula Research and Development TG@ removethisNebula-RnD.com Active updates of local fires in southern California, with map: removeNebula-RnD.com/blog/cosmos/earth/2007/10/local-fires1.html --- u2-users mailing list u2-users@listserver.u2ug.org To unsubscribe please visit http://listserver.u2ug.org/
RE: [U2] Fastest Bi-Directional data transfer btwn MV and non MV dbms [AD]
Gee Janet all you had to do was put This is not an ad, it's a fact of business in your posting and you would have been alright. grs -Original Message- From: [EMAIL PROTECTED] [mailto:owner-u2- [EMAIL PROTECTED] On Behalf Of Janet Bond Sent: Tuesday, October 23, 2007 1:37 PM To: u2-users@listserver.u2ug.org Subject: RE: [U2] Fastest Bi-Directional data transfer btwn MV and non MV dbms [AD] I added it in the Subject should it be somewhere else? Please accept my apologies if I have offended anyone. Janet -Original Message- From: [EMAIL PROTECTED] [mailto:owner-u2- [EMAIL PROTECTED] On Behalf Of Moderator Sent: Tuesday, October 23, 2007 11:15 AM To: u2-users@listserver.u2ug.org Subject: Re: [U2] Fastest Bi-Directional data transfer btwn MV and non MV dbms [AD] Janet, When posting as a vendor or service provider, please use [AD] brackets [/AD] in your response as a courtesy to others. - Charles Barouch, Moderator Janet Bond wrote: I can setup a conference call with one of Developers. We have been in the transferring MultiValue data to other data sources since the early 80's (PK Harmony to start with, anyone remember). We may have some good input for you. --- u2-users mailing list u2-users@listserver.u2ug.org To unsubscribe please visit http://listserver.u2ug.org/ --- u2-users mailing list u2-users@listserver.u2ug.org To unsubscribe please visit http://listserver.u2ug.org/ --- u2-users mailing list u2-users@listserver.u2ug.org To unsubscribe please visit http://listserver.u2ug.org/
RE: [U2] Fastest Bi-Directional data transfer btwn MV and non MV dbms [AD]
[AD] Okay, let's play nice Chuck did say he missed the last letters of the subject. I will have Robert/Antoon Houben provide a comment for you all as they have a wealth of information and history with MultiValue systems. /AD Thank you, Janet -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of George R Smith Sent: Tuesday, October 23, 2007 3:13 PM To: u2-users@listserver.u2ug.org Subject: RE: [U2] Fastest Bi-Directional data transfer btwn MV and non MV dbms [AD] Gee Janet all you had to do was put This is not an ad, it's a fact of business in your posting and you would have been alright. grs -Original Message- From: [EMAIL PROTECTED] [mailto:owner-u2- [EMAIL PROTECTED] On Behalf Of Janet Bond Sent: Tuesday, October 23, 2007 1:37 PM To: u2-users@listserver.u2ug.org Subject: RE: [U2] Fastest Bi-Directional data transfer btwn MV and non MV dbms [AD] I added it in the Subject should it be somewhere else? Please accept my apologies if I have offended anyone. Janet -Original Message- From: [EMAIL PROTECTED] [mailto:owner-u2- [EMAIL PROTECTED] On Behalf Of Moderator Sent: Tuesday, October 23, 2007 11:15 AM To: u2-users@listserver.u2ug.org Subject: Re: [U2] Fastest Bi-Directional data transfer btwn MV and non MV dbms [AD] Janet, When posting as a vendor or service provider, please use [AD] brackets [/AD] in your response as a courtesy to others. - Charles Barouch, Moderator Janet Bond wrote: I can setup a conference call with one of Developers. We have been in the transferring MultiValue data to other data sources since the early 80's (PK Harmony to start with, anyone remember). We may have some good input for you. --- u2-users mailing list u2-users@listserver.u2ug.org To unsubscribe please visit http://listserver.u2ug.org/ --- u2-users mailing list u2-users@listserver.u2ug.org To unsubscribe please visit http://listserver.u2ug.org/ --- u2-users mailing list u2-users@listserver.u2ug.org To unsubscribe please visit http://listserver.u2ug.org/ --- u2-users mailing list u2-users@listserver.u2ug.org To unsubscribe please visit http://listserver.u2ug.org/
