Re: [U2] UPLOAD not DOWNLOAD
I did mention in my initial posting Cederville's DOWNLOAD Which extracts data *from* Universe. It would be counter-intuitive to call a system which loads data *into* Universe also a Download. Rather it must be called an Upload. -Original Message- From: Tony Gravagno <3xk547...@sneakemail.com> To: u2-users Sent: Sat, Oct 27, 2012 12:23 pm Subject: Re: [U2] UPLOAD not DOWNLOAD Well, what you say below is exactly opposite of what I described. Normally I would encourage anyone else to re-read what I wrote and cite specific text which leads to the conclusions below - in this case... nah. The data source never needs to know anything about the target - that would be silly. And no, I didn't describe a process that "goes the wrong way" - you said "*into* Universe (not out from it)" which is what I described. Perhaps what confused you is that the code for going in both directions looks exactly the same - which is one of the neat aspects of the approach. BTW, someone might suggest from my code that the retrieval from the source is done in bulk while posting to the U2 target seems to be done with individual transactions. That could be the case and for large loads I'd modify the code to aggregate the data into a collection and then post that to U2 in a single transaction. BTW, the terms UPload and DOWNload are hugely ambiguous, and I believe you have defined the terms backwards here. When I download to my PC, it is TO my PC, not away from it. When I upload to a web server, the data is moving out. And yet you define "UPLOAD not DOWNLOAD" as meaning "*into* Universe (not out from it)". The terms IMport and EXport are more explicit. Your reference to the Cedarville Download utility is a clue to the confusion. Cedarville is using Download in the context of the recipient of the information downloading information to a more familiar format from the Unidata server. From the perspective of everyone in this group, that would be an Export, and going in the opposite direction would be an Import. So what you're describing is an Import into U2 ... traditionally described as a download. T > From: Wjhonson > > That goes the wrong way. > Download does not depend on knowing who is consuming what to > create it's results. > > So Upload should not need to have interactive consumption details > either. > That's my take. > > It's reads a source and decides on its own without the sources > assistance where to put what. > > You're coming at it more from a point of view of both sides *agreeing* > on what data is in what position. ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users
Re: [U2] UPLOAD not DOWNLOAD
Well, what you say below is exactly opposite of what I described. Normally I would encourage anyone else to re-read what I wrote and cite specific text which leads to the conclusions below - in this case... nah. The data source never needs to know anything about the target - that would be silly. And no, I didn't describe a process that "goes the wrong way" - you said "*into* Universe (not out from it)" which is what I described. Perhaps what confused you is that the code for going in both directions looks exactly the same - which is one of the neat aspects of the approach. BTW, someone might suggest from my code that the retrieval from the source is done in bulk while posting to the U2 target seems to be done with individual transactions. That could be the case and for large loads I'd modify the code to aggregate the data into a collection and then post that to U2 in a single transaction. BTW, the terms UPload and DOWNload are hugely ambiguous, and I believe you have defined the terms backwards here. When I download to my PC, it is TO my PC, not away from it. When I upload to a web server, the data is moving out. And yet you define "UPLOAD not DOWNLOAD" as meaning "*into* Universe (not out from it)". The terms IMport and EXport are more explicit. Your reference to the Cedarville Download utility is a clue to the confusion. Cedarville is using Download in the context of the recipient of the information downloading information to a more familiar format from the Unidata server. From the perspective of everyone in this group, that would be an Export, and going in the opposite direction would be an Import. So what you're describing is an Import into U2 ... traditionally described as a download. T > From: Wjhonson > > That goes the wrong way. > Download does not depend on knowing who is consuming what to > create it's results. > > So Upload should not need to have interactive consumption details > either. > That's my take. > > It's reads a source and decides on its own without the sources > assistance where to put what. > > You're coming at it more from a point of view of both sides *agreeing* > on what data is in what position. ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users
Re: [U2] UPLOAD not DOWNLOAD
Did you read Brian Leachs document? I think the way he solved this was rather clever, using a TCL command to associate the columns to attributes, then once you've done it once per definition, you can set it and forget it. Much more clever than writing an import routine for every new type of document. -Original Message- From: George Gallen To: U2 Users List Sent: Sat, Oct 27, 2012 6:52 am Subject: Re: [U2] UPLOAD not DOWNLOAD Will, if you don't want interactive (manual mapping), there is no other way aside from using DICT items that can be used to automatically map from an input file to your UV file, unless you feel like re-programming and compiling each time to change which columns go to which attributes. And you still may wind up having to manually map anyway, if your DICT name/description don't match column headers at all. I'm sure you have converted your share of client supplied files, and they are rarely the same, in column heading names or order of columns, even the formatting of dates, countries, states, phone numbers. Outputting is easier, it's usually, here's my data, now you deal with it to import it George Gallen Senior Programmer/Analyst Accounting/Data Division ggal...@wyanokegroup.com ph:856.848.9005 Ext 220 The Wyanoke Group http://www.wyanokegroup.com From: u2-users-boun...@listserver.u2ug.org [u2-users-boun...@listserver.u2ug.org] On Behalf Of Wjhonson [wjhon...@aol.com] Sent: Friday, October 26, 2012 11:40 PM To: u2-users@listserver.u2ug.org Subject: Re: [U2] UPLOAD not DOWNLOAD That goes the wrong way. Download does not depend on knowing who is consuming what to create it's results. So Upload should not need to have interactive consumption details either. That's my take. It's reads a source and decides on its own without the sources assistance where to put what. You're coming at it more from a point of view of both sides *agreeing* on what data is in what position. ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users
Re: [U2] UPLOAD not DOWNLOAD
Will, if you don't want interactive (manual mapping), there is no other way aside from using DICT items that can be used to automatically map from an input file to your UV file, unless you feel like re-programming and compiling each time to change which columns go to which attributes. And you still may wind up having to manually map anyway, if your DICT name/description don't match column headers at all. I'm sure you have converted your share of client supplied files, and they are rarely the same, in column heading names or order of columns, even the formatting of dates, countries, states, phone numbers. Outputting is easier, it's usually, here's my data, now you deal with it to import it George Gallen Senior Programmer/Analyst Accounting/Data Division ggal...@wyanokegroup.com ph:856.848.9005 Ext 220 The Wyanoke Group http://www.wyanokegroup.com From: u2-users-boun...@listserver.u2ug.org [u2-users-boun...@listserver.u2ug.org] On Behalf Of Wjhonson [wjhon...@aol.com] Sent: Friday, October 26, 2012 11:40 PM To: u2-users@listserver.u2ug.org Subject: Re: [U2] UPLOAD not DOWNLOAD That goes the wrong way. Download does not depend on knowing who is consuming what to create it's results. So Upload should not need to have interactive consumption details either. That's my take. It's reads a source and decides on its own without the sources assistance where to put what. You're coming at it more from a point of view of both sides *agreeing* on what data is in what position. ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users
Re: [U2] UPLOAD not DOWNLOAD
hi John, Depending what you are looking for, - a generic "ETL" tool where you define 1) Extract data from a defineable source (remote sql, csv-xls-dbf-xml-json files, webservice result , ...) with extendable method 2) Transform the received clob to internal UV data representation (iconv, mv'ed representation, build multi-part unique key, ...) 3) Load into U2 files (update one or more uvfiles into a Transaction (db transaction + application transaction)) That's not a "routine" that's complex tool with a lot of params and subroutines :-p I have it, writen in SB+, require SB+ at runtime, but it's not free. Please let me know if you can be interested. - - Anyway, If you search routines to import, write one record on one uvfile per "received record", from some specific data source I'm ready to share my experience (sql, dbf, csv, mdb, xls, xml, ...). Please put some detail of the source. regards Manu Wjhonson a écrit : Does anyone have a robust ETL routine for loading data *into* Universe (not out from it). Something like Cedarville's DOWNLOAD but in reverse ? Before I roll my own. ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users
