Re: [U2] Sparse array population in Pick
U2 has built in XML handleing - of course XML can be very complicated multi level data structures as well but you have to strip it in such a way to make it fit, u2 is not an xml database nor a json database - so for both you are going to have to extract at a particular level - or traverse it in a DOM kind of way. From: u2-users-boun...@listserver.u2ug.org [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Kevin King Sent: 11 December 2010 23:29 To: U2 Users List Subject: Re: [U2] Sparse array population in Pick We also do a lot with JSON but a JSON string can go far beyond attributes, values, and subvalues. Would be interested in Doug's take on how to extract something out of a JSON string without parsing it from scratch on every extraction, and how he would represent 4 or more dimensions in a MV item? Building JSON is simple in MV Basic. Parsing it can be a bit more difficult. ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users _ No virus found in this message. Checked by AVG - www.avg.com Version: 10.0.1170 / Virus Database: 426/3309 - Release Date: 12/11/10 ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users
Re: [U2] Sparse array population in Pick
The problem as I see it Mecki is that we have effectively two kinds of arrays in JSON; name/value pairs and sequential arrays. So perhaps I'm over-complicating but it seems problematic to represent scalars, NVPs, and sequential arrays with dynamic arrays - at least in a way that would allow the input to be decoded and re-encoded into an identical JSON string. I understand the concept of not going deeper than 4 dimensions but we've found that we can harness the power of the server to generate a lot of information very quickly (think of it like multiple simultaneous report requests) and then send it all back in one data stream for rendering via the web server. This is smokin' fast compared to trying to read each record over the connection and do the reporting logic from the PHP side. Our menu structure itself is a nested PHP object that can go to any number of levels. We can build such a thing easily in BASIC and send it to the web server no problem but if the web server wanted to move that depth of information into Unidata, well, it's a bit more involved. And as to why JSON vs. XML, especially when Unidata already supports XML, I gotta go back to one of my fundamental TCP/IP lessons: If you want data to move faster on the wire, move less of it! JSON can move the same information as XML (structurally speaking) but with a lot less bandwidth. ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users
Re: [U2] Sparse array population in Pick
Kevin, I would split the data into hierarchically organized files on the U2 end if need be. In my experience anything past the multi-value level becomes awkward in any Pick system if you want to use the data for reporting. Sub values are still reasonably well handled with Basic but after that it gets more and more messy. And even if you can nest data far deeper than 4 dimensions using XML or JSON, do you really want to do it, though? Just because you can doesn't always mean you should. just my 2 pennies Mecki On 12/12/2010 03:42, Kevin King wrote: In our JSON implementation (moving information between U2 and PHP) we've artificially limited the data coming into U2 so that it doesn't go deeper than (effectively) multivalues. We allow the data going out of U2, however, to be nested significantly deeper. For this product the limit isn't a problem but I would love to find a sensible way to parse a JSON string of any depth or complexity into a U2 structure that can be easily consumed by a program. ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users
Re: [U2] Sparse array population in Pick
We also do a lot with JSON but a JSON string can go far beyond attributes, values, and subvalues. Would be interested in Doug's take on how to extract something out of a JSON string without parsing it from scratch on every extraction, and how he would represent 4 or more dimensions in a MV item? Building JSON is simple in MV Basic. Parsing it can be a bit more difficult. ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users
Re: [U2] Sparse array population in Pick
On 11/12/10 23:29, Kevin King wrote: We also do a lot with JSON but a JSON string can go far beyond attributes, values, and subvalues. Would be interested in Doug's take on how to extract something out of a JSON string without parsing it from scratch on every extraction, and how he would represent 4 or more dimensions in a MV item? iirc some versions of MV can store up to 7 dimensions? FM, VM, SM, TM then there's two (or three?) more available at the program level? Pain in the neck to manipulate, I think, but there is some support there. Cheers, Wol ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users
Re: [U2] Sparse array population in Pick
To see what you can use in any particular implementation, try RAISE() and LOWER() to see how far they go (at least those who support these functions). I've used them before but they are indeed a pain to manipulate. Array reference statementsdon't recognize themand English/JQL/Retrieve/etc. don't handle them well. You can store them, though, then pull the data out and raise it. Charlie Noah Charles W. Noah Associates cwn...@comcast.net The views and opinions expressed herein are my own (Charlie Noah) and do not necessarily reflect the views, positions or policies of any of my former, current or future employers, employees, clients, friends, enemies or anyone else who might take exception to them. On 12-11-2010 6:03 PM, Wols Lists wrote: On 11/12/10 23:29, Kevin King wrote: We also do a lot with JSON but a JSON string can go far beyond attributes, values, and subvalues. Would be interested in Doug's take on how to extract something out of a JSON string without parsing it from scratch on every extraction, and how he would represent 4 or more dimensions in a MV item? iirc some versions of MV can store up to 7 dimensions? FM, VM, SM, TM then there's two (or three?) more available at the program level? Pain in the neck to manipulate, I think, but there is some support there. Cheers, Wol ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users
Re: [U2] Sparse array population in Pick
In our JSON implementation (moving information between U2 and PHP) we've artificially limited the data coming into U2 so that it doesn't go deeper than (effectively) multivalues. We allow the data going out of U2, however, to be nested significantly deeper. For this product the limit isn't a problem but I would love to find a sensible way to parse a JSON string of any depth or complexity into a U2 structure that can be easily consumed by a program. ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users
Re: [U2] Sparse array population in Pick
Isnt this just a name value pair construct ? i do this a lot and just have 2 mv attributes 1 = name, 2 = value If you want mv data in the values then you can always LOWER it. From: u2-users-boun...@listserver.u2ug.org [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of fft2...@aol.com Sent: 09 December 2010 22:10 To: u2-users@listserver.u2ug.org Subject: [U2] Sparse array population in Pick Years ago I had written a system, far predating XML, where the element-tags were unpredictable. Essentially the user was allowed to create any tags they wished, and any number of tags they wished, attached to another item. Each tag had an associated value. So far example Zip Code = 95062 You could not however predict what tags a person would use, they were all free-form and user-supplied, but you still had to store the tag with their associated value. At the time I developed two ideas for how to do this in a Pick item TAGS1052 001 Make = Chrysler 002 Model = LeBaron 003 Age = 7 years and 3 days TAGS1052 001 Make]Model]Age 002 Chrysler 003 LeBaron 004 7 years and 3 days The first model is clear. Anyone with no programming background at all, can easily understand it, and also easily edit it. It suffers from requiring more elaborate programming than the second model, as you have to parse every element. The second model is not quite as clear. You determine the attibute position of the value by locate the tag in attribute 1 and then adding 1 to it. That gives you the attribute number where the value lives. Alternately you could simply pre-fill attribute 1 with an initial null to push everything forward 1 place, then you wouldn't have to add 1 after your locate. Comments? Critiques? Nasty cat-calls and grimaces? Will Johnson ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users _ No virus found in this message. Checked by AVG - www.avg.com Version: 10.0.1170 / Virus Database: 426/3305 - Release Date: 12/09/10 ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users
Re: [U2] Sparse array population in Pick
That is why I was thinking a real name-value store might be in order. As MV professionals, not every solution has to be a Pick solution. On Fri, Dec 10, 2010 at 4:10 AM, Symeon Breen syme...@gmail.com wrote: Isnt this just a name value pair construct ? i do this a lot and just have 2 mv attributes 1 = name, 2 = value If you want mv data in the values then you can always LOWER it. ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users
Re: [U2] Sparse array population in Pick
Consider replacing age with year or purchase date. --Bill -Original Message- From: u2-users-boun...@listserver.u2ug.org [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of fft2...@aol.com Sent: Thursday, December 09, 2010 5:10 PM To: u2-users@listserver.u2ug.org Subject: [U2] Sparse array population in Pick Years ago I had written a system, far predating XML, where the element-tags were unpredictable. Essentially the user was allowed to create any tags they wished, and any number of tags they wished, attached to another item. Each tag had an associated value. So far example Zip Code = 95062 You could not however predict what tags a person would use, they were all free-form and user-supplied, but you still had to store the tag with their associated value. At the time I developed two ideas for how to do this in a Pick item TAGS1052 001 Make = Chrysler 002 Model = LeBaron 003 Age = 7 years and 3 days TAGS1052 001 Make]Model]Age 002 Chrysler 003 LeBaron 004 7 years and 3 days The first model is clear. Anyone with no programming background at all, can easily understand it, and also easily edit it. It suffers from requiring more elaborate programming than the second model, as you have to parse every element. The second model is not quite as clear. You determine the attibute position of the value by locate the tag in attribute 1 and then adding 1 to it. That gives you the attribute number where the value lives. Alternately you could simply pre-fill attribute 1 with an initial null to push everything forward 1 place, then you wouldn't have to add 1 after your locate. Comments? Critiques? Nasty cat-calls and grimaces? Will Johnson ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users
Re: [U2] Sparse array population in Pick
Yep, we do it all of time.
-Original Message-
From: u2-users-boun...@listserver.u2ug.org
[mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Rob Sobers
Sent: Thursday, December 09, 2010 6:53 PM
To: U2 Users List
Subject: Re: [U2] Sparse array population in Pick
Doug,
I love JSON and use it all the time, mainly in JavaScript and Python, but
how do you propose persisting it to a U2 database? Or are you proposing
that one would persist data in the native attribute, value, sub-value format
and then de-serialize into JSON for the purpose of transporting?
-Rob
On Thu, Dec 9, 2010 at 7:41 PM, Doug dave...@hotmail.com wrote:
Read up about JSON (JavaScript Object Notation). At least JSON is
compatible with Java, JavaScript, PHP, Python, and Ruby to name a few,
see http://json.org. Your field names can be dictionaries or meta names.
Most other languages besides Unidata or Universe are not happy with
dots in the named fields so you need to replace them with some other
character like a underline. You will have to write a parser in
UniBasic but you can do in about 30 lines of code. This syntax
supports multi-values or even sub-values. Here are some simple examples:
Here is a single valued record:
[{IdCode:A,Desc:Active,ReportColor:#CC,No_AR:}]
Here is a multivalued record:
[{IdCode:178*MAINT,CustomerNo:178,G1EffectiveDate:[[10/01/20
07],[ 12/31/2010]],G1BillRate:[[0.00],[1.00]]}]
JavaScript Reference would be for the first line if it was loaded as
an object called oData as:
Single array reference:
oData[0].IdCode would be 178*MAINT
Multiple array reference:
oData[0].G1EffectiveDate[1][0] would be 12/31/2010
We use this format in our U2WebLink middleware running under UniObjects
for
Java for the Web. The data is much more compact than XML and very
readable. Most of the languages have the ability to parse this in a
single function. In JavaScript it takes around 7 to 10 milliseconds
to parse a pretty big JSON string into a JSON array.
Regards,
Doug
___
U2-Users mailing list
U2-Users@listserver.u2ug.org
http://listserver.u2ug.org/mailman/listinfo/u2-users
Re: [U2] Sparse array population in Pick
In a message dated 12/10/2010 6:56:38 AM Pacific Standard Time, slestak...@gmail.com writes: That is why I was thinking a real name-value store might be in order. As MV professionals, not every solution has to be a Pick solution. On Fri, Dec 10, 2010 at 4:10 AM, Symeon Breen syme...@gmail.com wrote: Isnt this just a name value pair construct ? i do this a lot and just have 2 mv attributes 1 = name, 2 = value If you want mv data in the values then you can always LOWER it. Not sure why you would lower it. You could have all the tags in attribute one and all the values in attribute 2, as multi-values. Of course if the user decides to create 800 tags, that can get pretty darn ugly to look at in the EDitor. And it's true that not every solution has to be a Pick solution, but if you have to reference it many times, from within the Pick environment, why make it external? ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users
Re: [U2] Sparse array population in Pick
that's what EV is for (unless you use another editor for data than AE) ;-) On 10/12/2010 18:06, fft2...@aol.com wrote: In a message dated 12/10/2010 6:56:38 AM Pacific Standard Time, slestak...@gmail.com writes: That is why I was thinking a real name-value store might be in order. As MV professionals, not every solution has to be a Pick solution. On Fri, Dec 10, 2010 at 4:10 AM, Symeon Breen syme...@gmail.com wrote: Isnt this just a name value pair construct ? i do this a lot and just have 2 mv attributes 1 = name, 2 = value If you want mv data in the values then you can always LOWER it. Not sure why you would lower it. You could have all the tags in attribute one and all the values in attribute 2, as multi-values. Of course if the user decides to create 800 tags, that can get pretty darn ugly to look at in the EDitor. And it's true that not every solution has to be a Pick solution, but if you have to reference it many times, from within the Pick environment, why make it external? ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users
Re: [U2] Sparse array population in Pick
I think Symeon was saying that if the tag values are multi-valued, then lowering it makes them multi-sub-valued and won't corrupt the structure. I use this lower/raise trick for name/value pairs in 'named common' for passing parameters between processes except that I use a lower/lower (when stored) and a raise/raise (when extracted) so I can store items with attribute marks in my named pairs. -Original Message- From: u2-users-boun...@listserver.u2ug.org [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of fft2...@aol.com Sent: Friday, December 10, 2010 10:07 AM To: u2-users@listserver.u2ug.org Subject: Re: [U2] Sparse array population in Pick Not sure why you would lower it. _ Scanned by IBM Email Security Management Services powered by MessageLabs. For more information please visit http://www.ers.ibm.com This email is intended only for the use of the party to which it is addressed and may contain information that is privileged, confidential, or protected by law. If you are not the intended recipient you are hereby notified that any dissemination, copying or distribution of the email or its contents is strictly prohibited. If you have received this message in error, please notify us immediately, by replying to the message and deleting it from your computer. WARNING: Internet communications are not assured to be secure or clear of inaccuracies as information could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or contain viruses. Therefore, we do not accept responsibility for any errors or omissions that are present in this email, or any attachment, that have arisen as a result of e-mail transmission. _ ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users
Re: [U2] Sparse array population in Pick
E-mail client from issue and this post never made it, so I'm forwarding my response again. Glen Batchelor IT Director/CIO/CTO All-Spec Industries phone: (910) 332-0424 fax: (910) 763-5664 E-mail: webmas...@all-spec.com Web: http://www.all-spec.com Blog: http://blog.all-spec.com -Original Message- From: Glen Batchelor [mailto:webmas...@allspec.com] Sent: Friday, December 10, 2010 2:38 PM To: 'U2 Users List' Subject: RE: [U2] Sparse array population in Pick You can also store the metadata outside of the data item in another item using a standardized item naming structure. If a specific file contains the same types of data formats then you can store a global default in a locally or globally unique key. When creating new records the default layout is pulled up. This allows you more formatting and storage flexibility, IMO, but always requires two normal reads instead of one to obtain the data and metadata. In the case of a new data item, 2 reads will still be required. One to determine the item doesn't exist and a second to read the default meta so a new item can be created. CARS $$METADEFAULT 001 Make 002 Model 003 Age $$META.TAGS1052 001 Make 002 Model 003 Age 004 Comments TAGS1052 001 Chrysler 002 LeBaron 003 2558 D (2558 days) 004 This item is a non-default data test If you required multiple formats in the same data file, then that's OK. You can always store additional non-standard formats in additional $$METADEFAULT. items. would be the format number and you can store descriptions in the dictionary regarding what the various format codes are meant to be used for. Another, more compact, option is to just store the description of the metadata in attribute 001 and shift the data down one attribute. CARS $$METADEFAULT 001 Normal inventory filings for autos 002 Make 003 Model 004 Age $$METADEFAULT.0001 001 Collision inventory related filings for autos 002 Incident Date 003 Make 004 Model 005 Age 006 Insurer 007 Policy Number 008 Notes $$META.TAGS1052 001 TAGS1052 Created on 10/12/2010 by GB 002 Make 003 Model 004 Age 005 Comments TAGS1052 001 Chrysler 002 LeBaron 003 2558 D (2558 days) 004 This item is a non-default data test I could make up more sample data, but I hope that's enough to understand to structures. Regards, Glen Batchelor IT Director/CIO/CTO All-Spec Industries phone: (910) 332-0424 fax: (910) 763-5664 E-mail: webmas...@all-spec.com Web: http://www.all-spec.com Blog: http://blog.all-spec.com -Original Message- From: u2-users-boun...@listserver.u2ug.org [mailto:u2-users- boun...@listserver.u2ug.org] On Behalf Of Bill Brutzman Sent: Friday, December 10, 2010 10:56 AM To: U2 Users List Subject: Re: [U2] Sparse array population in Pick Consider replacing age with year or purchase date. --Bill -Original Message- From: u2-users-boun...@listserver.u2ug.org [mailto:u2-users- boun...@listserver.u2ug.org] On Behalf Of fft2...@aol.com Sent: Thursday, December 09, 2010 5:10 PM To: u2-users@listserver.u2ug.org Subject: [U2] Sparse array population in Pick Years ago I had written a system, far predating XML, where the element- tags were unpredictable. Essentially the user was allowed to create any tags they wished, and any number of tags they wished, attached to another item. Each tag had an associated value. So far example Zip Code = 95062 You could not however predict what tags a person would use, they were all free-form and user-supplied, but you still had to store the tag with their associated value. At the time I developed two ideas for how to do this in a Pick item TAGS1052 001 Make = Chrysler 002 Model = LeBaron 003 Age = 7 years and 3 days TAGS1052 001 Make]Model]Age 002 Chrysler 003 LeBaron 004 7 years and 3 days The first model is clear. Anyone with no programming background at all, can easily understand it, and also easily edit it. It suffers from requiring more elaborate programming than the second model, as you have to parse every element. The second model is not quite as clear. You determine the attibute position of the value by locate the tag in attribute 1 and then adding 1 to it. That gives you the attribute number where the value lives. Alternately you could simply pre-fill attribute 1 with an initial null to push everything forward 1 place, then you wouldn't have to add 1 after your locate. Comments? Critiques? Nasty cat-calls and grimaces? Will Johnson
[U2] Sparse array population in Pick
Years ago I had written a system, far predating XML, where the element-tags were unpredictable. Essentially the user was allowed to create any tags they wished, and any number of tags they wished, attached to another item. Each tag had an associated value. So far example Zip Code = 95062 You could not however predict what tags a person would use, they were all free-form and user-supplied, but you still had to store the tag with their associated value. At the time I developed two ideas for how to do this in a Pick item TAGS1052 001 Make = Chrysler 002 Model = LeBaron 003 Age = 7 years and 3 days TAGS1052 001 Make]Model]Age 002 Chrysler 003 LeBaron 004 7 years and 3 days The first model is clear. Anyone with no programming background at all, can easily understand it, and also easily edit it. It suffers from requiring more elaborate programming than the second model, as you have to parse every element. The second model is not quite as clear. You determine the attibute position of the value by locate the tag in attribute 1 and then adding 1 to it. That gives you the attribute number where the value lives. Alternately you could simply pre-fill attribute 1 with an initial null to push everything forward 1 place, then you wouldn't have to add 1 after your locate. Comments? Critiques? Nasty cat-calls and grimaces? Will Johnson ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users
Re: [U2] Sparse array population in Pick
Will, I used the second model (originally about 20 years ago) with the initial empty value. Neither model is really very simple, but with the second model we created one or two I-descriptor subroutines that can allow dictionaries to be tailored to any tag you want to look for and select for all values associated with that tag quite nicely. Display routines are also not too daunting. There are certainly worse file structures to deal with than the second model. But maybe I'm biased, I 'invented' its use at a previous company, and I and the rest of the company worked with it for at least 15 years. Best Regards, Richard Lewis On Thu, Dec 9, 2010 at 3:09 PM, fft2...@aol.com wrote: Years ago I had written a system, far predating XML, where the element-tags were unpredictable. Essentially the user was allowed to create any tags they wished, and any number of tags they wished, attached to another item. Each tag had an associated value. So far example Zip Code = 95062 You could not however predict what tags a person would use, they were all free-form and user-supplied, but you still had to store the tag with their associated value. At the time I developed two ideas for how to do this in a Pick item TAGS1052 001 Make = Chrysler 002 Model = LeBaron 003 Age = 7 years and 3 days TAGS1052 001 Make]Model]Age 002 Chrysler 003 LeBaron 004 7 years and 3 days The first model is clear. Anyone with no programming background at all, can easily understand it, and also easily edit it. It suffers from requiring more elaborate programming than the second model, as you have to parse every element. The second model is not quite as clear. You determine the attibute position of the value by locate the tag in attribute 1 and then adding 1 to it. That gives you the attribute number where the value lives. Alternately you could simply pre-fill attribute 1 with an initial null to push everything forward 1 place, then you wouldn't have to add 1 after your locate. Comments? Critiques? Nasty cat-calls and grimaces? Will Johnson ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users
Re: [U2] Sparse array population in Pick
If you can restrict the tag names to valid dictionary names you could do something like: TAGS DICT: 1052_MAKE 1052_MODEL 1052_AGE TAGS DATA: 001 Chrysler 002 LeBaron 003 7 years and 3 days David A. Green (480) 813-1725 DAG Consulting -Original Message- From: u2-users-boun...@listserver.u2ug.org [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of fft2...@aol.com Sent: Thursday, December 09, 2010 3:10 PM To: u2-users@listserver.u2ug.org Subject: [U2] Sparse array population in Pick Years ago I had written a system, far predating XML, where the element-tags were unpredictable. Essentially the user was allowed to create any tags they wished, and any number of tags they wished, attached to another item. Each tag had an associated value. So far example Zip Code = 95062 You could not however predict what tags a person would use, they were all free-form and user-supplied, but you still had to store the tag with their associated value. At the time I developed two ideas for how to do this in a Pick item TAGS1052 001 Make = Chrysler 002 Model = LeBaron 003 Age = 7 years and 3 days TAGS1052 001 Make]Model]Age 002 Chrysler 003 LeBaron 004 7 years and 3 days The first model is clear. Anyone with no programming background at all, can easily understand it, and also easily edit it. It suffers from requiring more elaborate programming than the second model, as you have to parse every element. The second model is not quite as clear. You determine the attibute position of the value by locate the tag in attribute 1 and then adding 1 to it. That gives you the attribute number where the value lives. Alternately you could simply pre-fill attribute 1 with an initial null to push everything forward 1 place, then you wouldn't have to add 1 after your locate. Comments? Critiques? Nasty cat-calls and grimaces? Will Johnson ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users
Re: [U2] Sparse array population in Pick
What's the motivation for stuffing an XML-like data structure into a U2 database? Is this just a thought experiment? -Rob On Thu, Dec 9, 2010 at 5:35 PM, David A. Green dgr...@dagconsulting.comwrote: If you can restrict the tag names to valid dictionary names you could do something like: TAGS DICT: 1052_MAKE 1052_MODEL 1052_AGE TAGS DATA: 001 Chrysler 002 LeBaron 003 7 years and 3 days David A. Green (480) 813-1725 DAG Consulting -Original Message- From: u2-users-boun...@listserver.u2ug.org [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of fft2...@aol.com Sent: Thursday, December 09, 2010 3:10 PM To: u2-users@listserver.u2ug.org Subject: [U2] Sparse array population in Pick Years ago I had written a system, far predating XML, where the element-tags were unpredictable. Essentially the user was allowed to create any tags they wished, and any number of tags they wished, attached to another item. Each tag had an associated value. So far example Zip Code = 95062 You could not however predict what tags a person would use, they were all free-form and user-supplied, but you still had to store the tag with their associated value. At the time I developed two ideas for how to do this in a Pick item TAGS1052 001 Make = Chrysler 002 Model = LeBaron 003 Age = 7 years and 3 days TAGS1052 001 Make]Model]Age 002 Chrysler 003 LeBaron 004 7 years and 3 days The first model is clear. Anyone with no programming background at all, can easily understand it, and also easily edit it. It suffers from requiring more elaborate programming than the second model, as you have to parse every element. The second model is not quite as clear. You determine the attibute position of the value by locate the tag in attribute 1 and then adding 1 to it. That gives you the attribute number where the value lives. Alternately you could simply pre-fill attribute 1 with an initial null to push everything forward 1 place, then you wouldn't have to add 1 after your locate. Comments? Critiques? Nasty cat-calls and grimaces? Will Johnson ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users
Re: [U2] Sparse array population in Pick
Doug,
I love JSON and use it all the time, mainly in JavaScript and Python, but
how do you propose persisting it to a U2 database? Or are you proposing
that one would persist data in the native attribute, value, sub-value format
and then de-serialize into JSON for the purpose of transporting?
-Rob
On Thu, Dec 9, 2010 at 7:41 PM, Doug dave...@hotmail.com wrote:
Read up about JSON (JavaScript Object Notation). At least JSON is
compatible with Java, JavaScript, PHP, Python, and Ruby to name a few, see
http://json.org. Your field names can be dictionaries or meta names.
Most other languages besides Unidata or Universe are not happy with dots in
the named fields so you need to replace them with some other character like
a underline. You will have to write a parser in UniBasic but you can do in
about 30 lines of code. This syntax supports multi-values or even
sub-values. Here are some simple examples:
Here is a single valued record:
[{IdCode:A,Desc:Active,ReportColor:#CC,No_AR:}]
Here is a multivalued record:
[{IdCode:178*MAINT,CustomerNo:178,G1EffectiveDate:[[10/01/2007],[
12/31/2010]],G1BillRate:[[0.00],[1.00]]}]
JavaScript Reference would be for the first line if it was loaded as an
object called oData as:
Single array reference:
oData[0].IdCode would be 178*MAINT
Multiple array reference:
oData[0].G1EffectiveDate[1][0] would be 12/31/2010
We use this format in our U2WebLink middleware running under UniObjects for
Java for the Web. The data is much more compact than XML and very
readable. Most of the languages have the ability to parse this in a single
function. In JavaScript it takes around 7 to 10 milliseconds to parse a
pretty big JSON string into a JSON array.
Regards,
Doug
www.u2logic.com/tools.html
-Original Message-
From: u2-users-boun...@listserver.u2ug.org
[mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Rob Sobers
Sent: Thursday, December 09, 2010 4:05 PM
To: U2 Users List
Subject: Re: [U2] Sparse array population in Pick
What's the motivation for stuffing an XML-like data structure into a U2
database? Is this just a thought experiment?
-Rob
On Thu, Dec 9, 2010 at 5:35 PM, David A. Green
dgr...@dagconsulting.comwrote:
If you can restrict the tag names to valid dictionary names you could
do something like:
TAGS DICT:
1052_MAKE
1052_MODEL
1052_AGE
TAGS DATA:
001 Chrysler
002 LeBaron
003 7 years and 3 days
David A. Green
(480) 813-1725
DAG Consulting
-Original Message-
From: u2-users-boun...@listserver.u2ug.org
[mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of
fft2...@aol.com
Sent: Thursday, December 09, 2010 3:10 PM
To: u2-users@listserver.u2ug.org
Subject: [U2] Sparse array population in Pick
Years ago I had written a system, far predating XML, where the
element-tags were unpredictable. Essentially the user was allowed to
create any tags they wished, and any number of tags they wished, attached
to another item.
Each tag had an associated value. So far example
Zip Code = 95062
You could not however predict what tags a person would use, they were
all free-form and user-supplied, but you still had to store the tag
with their associated value.
At the time I developed two ideas for how to do this in a Pick item
TAGS1052
001 Make = Chrysler
002 Model = LeBaron
003 Age = 7 years and 3 days
TAGS1052
001 Make]Model]Age
002 Chrysler
003 LeBaron
004 7 years and 3 days
The first model is clear. Anyone with no programming background at
all, can easily understand it, and also easily edit it. It suffers
from requiring more elaborate programming than the second model, as
you have to parse every
element.
The second model is not quite as clear. You determine the attibute
position of the value by locate the tag in attribute 1 and then
adding 1 to it.
That gives you the attribute number where the value lives.
Alternately you could simply pre-fill attribute 1 with an initial null
to push everything forward 1 place, then you wouldn't have to add 1 after
your locate.
Comments? Critiques? Nasty cat-calls and grimaces?
Will Johnson
___
U2-Users mailing list
U2-Users@listserver.u2ug.org
http://listserver.u2ug.org/mailman/listinfo/u2-users
___
U2-Users mailing list
U2-Users@listserver.u2ug.org
http://listserver.u2ug.org/mailman/listinfo/u2-users
___
U2-Users mailing list
U2-Users@listserver.u2ug.org
http://listserver.u2ug.org/mailman/listinfo/u2-users
___
U2-Users mailing list
U2-Users@listserver.u2ug.org
http://listserver.u2ug.org/mailman/listinfo/u2-users
___
U2-Users mailing list
U2-Users@listserver.u2ug.org
http
Re: [U2] Sparse array population in Pick
No David you cannot restrict the tag names to valid Dict names, they can be anything the user could possibly think up. And the user's can think up a lot of things! Think of the tags that you can apply on Amazon to products. You are not forced to pick your tags from a list, you can type in anything you want as a tag. And to Rob, the motivation for storing this in a U2 database, is because U2 needs to access it for further processing. Will ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users
Re: [U2] Sparse array population in Pick
Wouldn't be difficult to make a json encoder/decoder in Unibasic. Something I have been toying with is using Redis with Unidata. I wrapped Set and Get in pcperform statements. Works quite well. I do not propose this for permanent storage, but could have some use in report generation and caching. On 12/9/2010 8:53 PM, Rob Sobers wrote: Doug, I love JSON and use it all the time, mainly in JavaScript and Python, but how do you propose persisting it to a U2 database? Or are you proposing that one would persist data in the native attribute, value, sub-value format and then de-serialize into JSON for the purpose of transporting? -Rob ttp://listserver.u2ug.org/mailman/listinfo/u2-users ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users
Re: [U2] Sparse array population in Pick
You could quite easily come up with an escaping scheme to convert any tag name into a valid escaped dictionary name. -Original Message- From: u2-users-boun...@listserver.u2ug.org [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of fft2...@aol.com Sent: Friday, 10 December 2010 1:08 PM To: u2-users@listserver.u2ug.org Subject: Re: [U2] Sparse array population in Pick No David you cannot restrict the tag names to valid Dict names, they can be anything the user could possibly think up. And the user's can think up a lot of things! Think of the tags that you can apply on Amazon to products. You are not forced to pick your tags from a list, you can type in anything you want as a tag. And to Rob, the motivation for storing this in a U2 database, is because U2 needs to access it for further processing. Will ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users __ This email has been scanned by the MessageLabs Email Security System. For more information please visit http://www.messagelabs.com/email __ ### The information transmitted in this message and attachments (if any) is intended only for the person or entity to which it is addressed. The message may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of or taking of any action in reliance upon this information by persons or entities other than the intended recipient is prohibited. If you received this in error, please contact the sender and delete the material from any computer. The intended recipient of this e-mail may only use, reproduce, disclose or distribute the information contained in this e-mail and any attached files with the permission of IMB. ### ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users
Re: [U2] Sparse array population in Pick
In a message dated 12/9/2010 6:18:24 PM Pacific Standard Time, dmc...@imb.com.au writes: You could quite easily come up with an escaping scheme to convert any tag name into a valid escaped dictionary name. Could you explain this more clearly? I don't understand what you mean. ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users
Re: [U2] Sparse array population in Pick
Escaping/Encoding. As in how strings in C, C++, PHP, etc use \n to indicate a new line. How you escape can be arbitrarily determined if you are writing the code. For example, reserve the $ sign as the escape character. $$ gives you a literal $ in the tag name, $S gives you a space, $Q gives you a quote, etc. Do this for each character the user can enter that isn't a valid char for a dictionary name. Heck, you could even just Base64 Encode it you really wanted to go down that path and then you could fully support Unicode tag names. Having said that, the solution could come down to how you need to process the data. You could, for instance, organise the file layout differently: Record ID: Unique ID (like using a GUID in SQL, we have our own SR to do that here) 1: Item ID 2: Tag Name 3: Tag Value Using this method it is trivial to select/sort via Item Id, Tag Name or Tag Value, but sacrifices lookup speed for individual items. Even that could be partially alleviated though, by using a secondary index file keyed on Item ID with Attr1 a MV'ed list of all its records in the previous file. Regards, Dan -Original Message- From: u2-users-boun...@listserver.u2ug.org [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of fft2...@aol.com Sent: Friday, 10 December 2010 1:28 PM To: u2-users@listserver.u2ug.org Subject: Re: [U2] Sparse array population in Pick In a message dated 12/9/2010 6:18:24 PM Pacific Standard Time, dmc...@imb.com.au writes: You could quite easily come up with an escaping scheme to convert any tag name into a valid escaped dictionary name. Could you explain this more clearly? I don't understand what you mean. ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users __ This email has been scanned by the MessageLabs Email Security System. For more information please visit http://www.messagelabs.com/email __ ### The information transmitted in this message and attachments (if any) is intended only for the person or entity to which it is addressed. The message may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of or taking of any action in reliance upon this information by persons or entities other than the intended recipient is prohibited. If you received this in error, please contact the sender and delete the material from any computer. The intended recipient of this e-mail may only use, reproduce, disclose or distribute the information contained in this e-mail and any attached files with the permission of IMB. ### ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users
Re: [U2] Sparse array population in Pick
But Dan you're just flattening the array. B This is Pick! The point of a method of populating a sparse array is to address the efficient storage of data when your array is sparse. In your method, you'd have to do several disk reads in order to collect all the tags for one item in one spot. The method I displayed, is a single disk read :) ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users
Re: [U2] Sparse array population in Pick
100% agree, but this is pick doesn't mean disregard all rules of good DB design. Fair enough, MVDBs are designed around not subscribing to the 1st norm form rule; we should still present out data in the most structured manner possible, for this will give us the greatest flexibility if/when the system needs to be expanded. As I said though, it all comes down to how you need to work with your data. My form is not the most efficient for random access of individual items, but it does give you the flexibility of being able to sort and select on tag/value pairs. Additional benefits are that I could also index on tag names if required, which neither of the your original proposals can. I reiterate though, design all comes down to the intended purpose. Similar situations happen in SQL where you need determine if/where/how much you should denormalize your data. Work out the most important requirements/future requirements for your system and develop accordingly. -Original Message- From: u2-users-boun...@listserver.u2ug.org [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of fft2...@aol.com Sent: Friday, 10 December 2010 2:00 PM To: u2-users@listserver.u2ug.org Subject: Re: [U2] Sparse array population in Pick But Dan you're just flattening the array. B This is Pick! The point of a method of populating a sparse array is to address the efficient storage of data when your array is sparse. In your method, you'd have to do several disk reads in order to collect all the tags for one item in one spot. The method I displayed, is a single disk read :) ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users __ This email has been scanned by the MessageLabs Email Security System. For more information please visit http://www.messagelabs.com/email __ ### The information transmitted in this message and attachments (if any) is intended only for the person or entity to which it is addressed. The message may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of or taking of any action in reliance upon this information by persons or entities other than the intended recipient is prohibited. If you received this in error, please contact the sender and delete the material from any computer. The intended recipient of this e-mail may only use, reproduce, disclose or distribute the information contained in this e-mail and any attached files with the permission of IMB. ### ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users
Re: [U2] Sparse array population in Pick
I should probably expand on the good DB design statement. Take, for example, your second suggestion with the tags in Attr 1 and values in Attr 2 onwards. Once this system is implemented, how do you add and future fields that need to be catered for, such as created/modified timestamps for items, or worse, each tag in the item? Since you have already assigned all possible attributes 1-n you need to either create another file or rework all programs that use it. In the flattened solution I offer, it is a trivial exercise. I say this because we have a 15 year old system here that suffers from this issue and it causes as many headaches. I wish we had a system that used pick style where and only where it was appropriate. -Original Message- From: u2-users-boun...@listserver.u2ug.org [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Dan McGrath Sent: Friday, 10 December 2010 3:18 PM To: U2 Users List Subject: Re: [U2] Sparse array population in Pick 100% agree, but this is pick doesn't mean disregard all rules of good DB design. Fair enough, MVDBs are designed around not subscribing to the 1st norm form rule; we should still present out data in the most structured manner possible, for this will give us the greatest flexibility if/when the system needs to be expanded. As I said though, it all comes down to how you need to work with your data. My form is not the most efficient for random access of individual items, but it does give you the flexibility of being able to sort and select on tag/value pairs. Additional benefits are that I could also index on tag names if required, which neither of the your original proposals can. I reiterate though, design all comes down to the intended purpose. Similar situations happen in SQL where you need determine if/where/how much you should denormalize your data. Work out the most important requirements/future requirements for your system and develop accordingly. -Original Message- From: u2-users-boun...@listserver.u2ug.org [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of fft2...@aol.com Sent: Friday, 10 December 2010 2:00 PM To: u2-users@listserver.u2ug.org Subject: Re: [U2] Sparse array population in Pick But Dan you're just flattening the array. B This is Pick! The point of a method of populating a sparse array is to address the efficient storage of data when your array is sparse. In your method, you'd have to do several disk reads in order to collect all the tags for one item in one spot. The method I displayed, is a single disk read :) ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users __ This email has been scanned by the MessageLabs Email Security System. For more information please visit http://www.messagelabs.com/email __ ### The information transmitted in this message and attachments (if any) is intended only for the person or entity to which it is addressed. The message may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of or taking of any action in reliance upon this information by persons or entities other than the intended recipient is prohibited. If you received this in error, please contact the sender and delete the material from any computer. The intended recipient of this e-mail may only use, reproduce, disclose or distribute the information contained in this e-mail and any attached files with the permission of IMB. ### ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users __ This email has been scanned by the MessageLabs Email Security System. For more information please visit http://www.messagelabs.com/email __ __ This email has been scanned by the MessageLabs Email Security System. For more information please visit http://www.messagelabs.com/email __ ### The information transmitted in this message and attachments (if any) is intended only for the person or entity to which it is addressed. The message may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of or taking of any action in reliance upon this information by persons
Re: [U2] Sparse array population in Pick
In a message dated 12/9/2010 8:18:12 PM Pacific Standard Time, dmc...@imb.com.au writes: My form is not the most efficient for random access of individual items, but it does give you the flexibility of being able to sort and select on tag/value pairs. Additional benefits are that I could also index on tag names if required, which neither of the your original proposals can. I believe in the second proposal, you can index the first attribute, which is multi-valued, and it will work just fine as an index of the tag names. You cannot however index the values. ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users
