Re: [U2] General guidelines on indexing

[Bill Haskett]

t you had, otherwise the same 
message.

UD 5.2 on Solaris 5.2 (I think).
I know it's old, but that's all they give me.

Symeon Breen wrote:
Ok I am very jealous now of all these quick times, when my one 
crashed out
Can someone remind me (its been years since i did any udt config 
stuff) what

should i be looking at to fix this error

Thanks

PS - i like this thread i think this can be of benefit to people to 
fine

tune both code and their os and u2 configuration !



-Original Message-
From: Symeon Breen [mailto:syme...@gmail.com] Sent: 09 July 2009 14:04
To: 'U2 Users List'
Subject: RE: [U2] General guidelines on indexing

Unidata 7.1 (32bit) on redhat EL 3 64bit, on XEON E5335 with 4 gig 
ram. \


Error when creating a shared memory segment (size=35682416), errno=22

Lol

Hmmm ?


___
U2-Users mailing list
U2-Users@listserver.u2ug.org
http://listserver.u2ug.org/mailman/listinfo/u2-users

Re: [U2] General guidelines on indexing

[Bill Haskett]

I have a "UniData Troubleshooting Guide 2004.pdf" I got from Wally at an 
IBM conference several years ago.  I think it is dated Aug 2004.  It was 
part of a "Survival Guide" session and is an indispensable piece of 
documentation for UniData memory (and other) issues.


I couldn't find it on the internet or on IBM's site.  Let me know if you 
need it; I think it's ok to send it to you.


Bill

Mecki Foerthmann said the following on 7/10/2009 4:31 AM:
Well, if it makes you feel better, my test crashed out with a similar 
error message as well.
The size shown was about half of what you had, otherwise the same 
message.

UD 5.2 on Solaris 5.2 (I think).
I know it's old, but that's all they give me.

Symeon Breen wrote:
Ok I am very jealous now of all these quick times, when my one 
crashed out
Can someone remind me (its been years since i did any udt config 
stuff) what

should i be looking at to fix this error

Thanks

PS - i like this thread i think this can be of benefit to people to fine
tune both code and their os and u2 configuration !



-Original Message-
From: Symeon Breen [mailto:syme...@gmail.com] Sent: 09 July 2009 14:04
To: 'U2 Users List'
Subject: RE: [U2] General guidelines on indexing

Unidata 7.1 (32bit) on redhat EL 3 64bit, on XEON E5335 with 4 gig ram.

Error when creating a shared memory segment (size=35682416), errno=22

Lol

Hmmm ?



-Original Message-
From: u2-users-boun...@listserver.u2ug.org
[mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Ron Hutchings
Sent: 09 July 2009 13:40
To: u2-users@listserver.u2ug.org
Subject: Re: [U2] General guidelines on indexing


IBM BOX (OLDER)
PICK.FORMAT
10.0.11
6.9377 seconds

Linux box
PICK.FORMAT
10.2.0
0.3522 seconds

 

Date: Thu, 9 Jul 2009 11:20:25 +0800
From: adrian.wom...@rac.com.au
To: u2-users@listserver.u2ug.org
Subject: Re: [U2] General guidelines on indexing


I just tried this example on Universe 10.2.6 - it took 0.0665 seconds -
basically instant, can't complain about that.
 


-Original Message-
From: u2-users-boun...@listserver.u2ug.org
[mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Martin
Phillips
Sent: Thursday, 9 July 2009 12:59 AM
To: U2 Users List
Subject: Re: [U2] General guidelines on indexing


By way of a simple expample, I just tried the following program...
   s = ''
   z = str('*', 1000)
   t1 = time()
   for i = 1 to 10
  s<-1> = z
   next i
   t2 = time()
   crt t2 - t1

This took six seconds on QM but 32 minutes on UniVerse.


___
U2-Users mailing list
U2-Users@listserver.u2ug.org
http://listserver.u2ug.org/mailman/listinfo/u2-users

Re: [U2] General guidelines on indexing

[Anthony W. Youngman]

In message 
,

 Edward Brown  writes

After indexing, we made a lot more use of the SETINDEX and READFWD

logic
in our programs.

I find this curious / disappointing - is it really the case that unidata
can't take the mix of indexed / unindexed dictionary items and do just
as efficient a job as the code you're writing?

Also, the performance of dynamic arrays need not be as much an issue as
you've found. If they're built up with -1 rather than a counter then the
speed penalty of adding items to a very large list is much the same as a
tiny one.


Is it?

Depends on how the memory is managed, but I understood that adding items 
required a memcpy every now and then. The bigger your dynamic array, the 
more expensive is the memcpy when it's required.




The only real issue with dynamic arrays is if the machine does not have
the physical memory available to hold the variable.

Been there, done that ... 32 users on a 16meg EXL 7330. It spent most of 
its time in swap ...


(Finally talked the company into upgrading the ram when a 1meg stick 
went bad. Bought 32meg for that machine, and used its 16 to upgrade the 
other two machines to 24 each - Prime replaced the 1meg free as part of 
the upgrade deal  :-)


Cheers,
Wol
--
Anthony W. Youngman 
'Yings, yow graley yin! Suz ae rikt dheu,' said the blue man, taking the
thimble. 'What *is* he?' said Magrat. 'They're gnomes,' said Nanny. The man
lowered the thimble. 'Pictsies!' Carpe Jugulum, Terry Pratchett 1998
Visit the MaVerick web-site -  Open Source Pick
___
U2-Users mailing list
U2-Users@listserver.u2ug.org
http://listserver.u2ug.org/mailman/listinfo/u2-users

Re: [U2] General guidelines on indexing

[jpb-u2ug]

I have 2 RH Linux AS3 64bit servers one on a Dell 6800 with 4 processors and
16GB memory with UV 10.1.12 and 117 users. The other is a Dell 2500 with 2
processors and 16GB memory with UV 10.2.4 and 6 users. The first takes
0.1876 seconds and the second is 0.1106 seconds. The time varies with the
amount of activity and if I run them consecutively but I haven't had any
trouble with memory or going over 1 second.

Jerry Banker

-Original Message-
From: u2-users-boun...@listserver.u2ug.org
[mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Symeon Breen
Sent: Friday, July 10, 2009 4:15 AM
To: 'U2 Users List'
Subject: Re: [U2] General guidelines on indexing

Ok I am very jealous now of all these quick times, when my one crashed out 

Can someone remind me (its been years since i did any udt config stuff) what
should i be looking at to fix this error 


Thanks



PS - i like this thread i think this can be of benefit to people to fine
tune both code and their os and u2 configuration !



-Original Message-
From: Symeon Breen [mailto:syme...@gmail.com] 
Sent: 09 July 2009 14:04
To: 'U2 Users List'
Subject: RE: [U2] General guidelines on indexing

Unidata 7.1 (32bit) on redhat EL 3 64bit, on XEON E5335 with 4 gig ram.


Error when creating a shared memory segment (size=35682416), errno=22



Lol

Hmmm ?




-Original Message-
From: u2-users-boun...@listserver.u2ug.org
[mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Ron Hutchings
Sent: 09 July 2009 13:40
To: u2-users@listserver.u2ug.org
Subject: Re: [U2] General guidelines on indexing


IBM BOX (OLDER)
PICK.FORMAT
10.0.11
6.9377 seconds

Linux box
PICK.FORMAT
10.2.0
0.3522 seconds

> Date: Thu, 9 Jul 2009 11:20:25 +0800
> From: adrian.wom...@rac.com.au
> To: u2-users@listserver.u2ug.org
> Subject: Re: [U2] General guidelines on indexing
> 
> 
> I just tried this example on Universe 10.2.6 - it took 0.0665 seconds -
> basically instant, can't complain about that.
>  
> 
> -Original Message-
> From: u2-users-boun...@listserver.u2ug.org
> [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Martin
> Phillips
> Sent: Thursday, 9 July 2009 12:59 AM
> To: U2 Users List
> Subject: Re: [U2] General guidelines on indexing
> 
> 
> By way of a simple expample, I just tried the following program...
>s = ''
>z = str('*', 1000)
>t1 = time()
>for i = 1 to 10
>   s<-1> = z
>next i
>t2 = time()
>crt t2 - t1
> 
> This took six seconds on QM but 32 minutes on UniVerse. 
> 
> 
> 
> 
> DISCLAIMER:
> Disclaimer.  This e-mail is private and confidential. If you are not the
intended recipient, please advise us by return e-mail immediately, and
delete the e-mail and any attachments without using or disclosing the
contents in any way. The views expressed in this e-mail are those of the
author, and do not represent those of this company unless this is clearly
indicated. You should scan this e-mail and any attachments for viruses. This
company accepts no liability for any direct or indirect damage or loss
resulting from the use of any attachments to this e-mail.
> ___
> U2-Users mailing list
> U2-Users@listserver.u2ug.org
> http://listserver.u2ug.org/mailman/listinfo/u2-users

_
HotmailR has ever-growing storage! Don't worry about storage limits. 
http://windowslive.com/Tutorial/Hotmail/Storage?ocid=TXT_TAGLM_WL_HM_Tutoria
l_Storage_062009
___
U2-Users mailing list
U2-Users@listserver.u2ug.org
http://listserver.u2ug.org/mailman/listinfo/u2-users

___
U2-Users mailing list
U2-Users@listserver.u2ug.org
http://listserver.u2ug.org/mailman/listinfo/u2-users

___
U2-Users mailing list
U2-Users@listserver.u2ug.org
http://listserver.u2ug.org/mailman/listinfo/u2-users

Re: [U2] General guidelines on indexing

[Jeff Schasny]

Me too. Blew up immediately.

IBM P570 AIX 5.3 UV 10.2.4

Mecki Foerthmann wrote:
Well, if it makes you feel better, my test crashed out with a similar 
error message as well.
The size shown was about half of what you had, otherwise the same 
message.

UD 5.2 on Solaris 5.2 (I think).
I know it's old, but that's all they give me.

Symeon Breen wrote:
Ok I am very jealous now of all these quick times, when my one 
crashed out
Can someone remind me (its been years since i did any udt config 
stuff) what

should i be looking at to fix this error

Thanks



PS - i like this thread i think this can be of benefit to people to fine
tune both code and their os and u2 configuration !



-Original Message-
From: Symeon Breen [mailto:syme...@gmail.com] Sent: 09 July 2009 14:04
To: 'U2 Users List'
Subject: RE: [U2] General guidelines on indexing

Unidata 7.1 (32bit) on redhat EL 3 64bit, on XEON E5335 with 4 gig ram.


Error when creating a shared memory segment (size=35682416), errno=22



Lol

Hmmm ?




-Original Message-
From: u2-users-boun...@listserver.u2ug.org
[mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Ron Hutchings
Sent: 09 July 2009 13:40
To: u2-users@listserver.u2ug.org
Subject: Re: [U2] General guidelines on indexing


IBM BOX (OLDER)
PICK.FORMAT
10.0.11
6.9377 seconds

Linux box
PICK.FORMAT
10.2.0
0.3522 seconds

 

Date: Thu, 9 Jul 2009 11:20:25 +0800
From: adrian.wom...@rac.com.au
To: u2-users@listserver.u2ug.org
Subject: Re: [U2] General guidelines on indexing


I just tried this example on Universe 10.2.6 - it took 0.0665 seconds -
basically instant, can't complain about that.
 


-Original Message-
From: u2-users-boun...@listserver.u2ug.org
[mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Martin
Phillips
Sent: Thursday, 9 July 2009 12:59 AM
To: U2 Users List
Subject: Re: [U2] General guidelines on indexing


By way of a simple expample, I just tried the following program...
   s = ''
   z = str('*', 1000)
   t1 = time()
   for i = 1 to 10
  s<-1> = z
   next i
   t2 = time()
   crt t2 - t1

This took six seconds on QM but 32 minutes on UniVerse.



DISCLAIMER:
Disclaimer.  This e-mail is private and confidential. If you are not 
the


intended recipient, please advise us by return e-mail immediately, and
delete the e-mail and any attachments without using or disclosing the
contents in any way. The views expressed in this e-mail are those of the
author, and do not represent those of this company unless this is 
clearly
indicated. You should scan this e-mail and any attachments for 
viruses. This

company accepts no liability for any direct or indirect damage or loss
resulting from the use of any attachments to this e-mail.
 

___
U2-Users mailing list
U2-Users@listserver.u2ug.org
http://listserver.u2ug.org/mailman/listinfo/u2-users



_
HotmailR has ever-growing storage! Don't worry about storage limits. 
http://windowslive.com/Tutorial/Hotmail/Storage?ocid=TXT_TAGLM_WL_HM_Tutoria 


l_Storage_062009
___
U2-Users mailing list
U2-Users@listserver.u2ug.org
http://listserver.u2ug.org/mailman/listinfo/u2-users

___
U2-Users mailing list
U2-Users@listserver.u2ug.org
http://listserver.u2ug.org/mailman/listinfo/u2-users

  

___
U2-Users mailing list
U2-Users@listserver.u2ug.org
http://listserver.u2ug.org/mailman/listinfo/u2-users



--

Jeff Schasny - Denver, Co, USA
jschasny at gmail dot com

___
U2-Users mailing list
U2-Users@listserver.u2ug.org
http://listserver.u2ug.org/mailman/listinfo/u2-users

Re: [U2] General guidelines on indexing

[Joshua Gallant]

It sounds to me like you guys are hitting a limit on the size of the
memory segment.  If that's true its tuned as a kernel parameter.  I'd
check there first and then try other ideas.  

- Josh


-Original Message-
From: u2-users-boun...@listserver.u2ug.org
[mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Mecki
Foerthmann
Sent: Friday, July 10, 2009 7:31 AM
To: U2 Users List
Subject: Re: [U2] General guidelines on indexing

Well, if it makes you feel better, my test crashed out with a similar 
error message as well.
The size shown was about half of what you had, otherwise the same
message.
UD 5.2 on Solaris 5.2 (I think).
I know it's old, but that's all they give me.

Symeon Breen wrote:
> Ok I am very jealous now of all these quick times, when my one crashed
out 
>
> Can someone remind me (its been years since i did any udt config
stuff) what
> should i be looking at to fix this error 
>
>
> Thanks
>
>
>
> PS - i like this thread i think this can be of benefit to people to
fine
> tune both code and their os and u2 configuration !
>
>
>
> -Original Message-
> From: Symeon Breen [mailto:syme...@gmail.com] 
> Sent: 09 July 2009 14:04
> To: 'U2 Users List'
> Subject: RE: [U2] General guidelines on indexing
>
> Unidata 7.1 (32bit) on redhat EL 3 64bit, on XEON E5335 with 4 gig
ram.
>
>
> Error when creating a shared memory segment (size=35682416), errno=22
>
>
>
> Lol
>
> Hmmm ?
>
>
>
>
> -Original Message-
> From: u2-users-boun...@listserver.u2ug.org
> [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Ron
Hutchings
> Sent: 09 July 2009 13:40
> To: u2-users@listserver.u2ug.org
> Subject: Re: [U2] General guidelines on indexing
>
>
> IBM BOX (OLDER)
> PICK.FORMAT
> 10.0.11
> 6.9377 seconds
>
> Linux box
> PICK.FORMAT
> 10.2.0
> 0.3522 seconds
>
>   
>> Date: Thu, 9 Jul 2009 11:20:25 +0800
>> From: adrian.wom...@rac.com.au
>> To: u2-users@listserver.u2ug.org
>> Subject: Re: [U2] General guidelines on indexing
>>
>>
>> I just tried this example on Universe 10.2.6 - it took 0.0665 seconds
-
>> basically instant, can't complain about that.
>>  
>>
>> -----Original Message-
>> From: u2-users-boun...@listserver.u2ug.org
>> [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Martin
>> Phillips
>> Sent: Thursday, 9 July 2009 12:59 AM
>> To: U2 Users List
>> Subject: Re: [U2] General guidelines on indexing
>>
>>
>> By way of a simple expample, I just tried the following program...
>>s = ''
>>z = str('*', 1000)
>>t1 = time()
>>for i = 1 to 10
>>   s<-1> = z
>>next i
>>t2 = time()
>>crt t2 - t1
>>
>> This took six seconds on QM but 32 minutes on UniVerse. 
>>
>>
>>
>>
>> DISCLAIMER:
>> Disclaimer.  This e-mail is private and confidential. If you are not
the
>> 
> intended recipient, please advise us by return e-mail immediately, and
> delete the e-mail and any attachments without using or disclosing the
> contents in any way. The views expressed in this e-mail are those of
the
> author, and do not represent those of this company unless this is
clearly
> indicated. You should scan this e-mail and any attachments for
viruses. This
> company accepts no liability for any direct or indirect damage or loss
> resulting from the use of any attachments to this e-mail.
>   
>> ___
>> U2-Users mailing list
>> U2-Users@listserver.u2ug.org
>> http://listserver.u2ug.org/mailman/listinfo/u2-users
>> 
>
> _
> HotmailR has ever-growing storage! Don't worry about storage limits. 
>
http://windowslive.com/Tutorial/Hotmail/Storage?ocid=TXT_TAGLM_WL_HM_Tut
oria
> l_Storage_062009
> ___
> U2-Users mailing list
> U2-Users@listserver.u2ug.org
> http://listserver.u2ug.org/mailman/listinfo/u2-users
>
> ___
> U2-Users mailing list
> U2-Users@listserver.u2ug.org
> http://listserver.u2ug.org/mailman/listinfo/u2-users
>
>   
___
U2-Users mailing list
U2-Users@listserver.u2ug.org
http://listserver.u2ug.org/mailman/listinfo/u2-users
___
U2-Users mailing list
U2-Users@listserver.u2ug.org
http://listserver.u2ug.org/mailman/listinfo/u2-users

Re: [U2] General guidelines on indexing

[Mecki Foerthmann]

Well, if it makes you feel better, my test crashed out with a similar 
error message as well.

The size shown was about half of what you had, otherwise the same message.
UD 5.2 on Solaris 5.2 (I think).
I know it's old, but that's all they give me.

Symeon Breen wrote:
Ok I am very jealous now of all these quick times, when my one crashed out 


Can someone remind me (its been years since i did any udt config stuff) what
should i be looking at to fix this error 



Thanks



PS - i like this thread i think this can be of benefit to people to fine
tune both code and their os and u2 configuration !



-Original Message-
From: Symeon Breen [mailto:syme...@gmail.com] 
Sent: 09 July 2009 14:04

To: 'U2 Users List'
Subject: RE: [U2] General guidelines on indexing

Unidata 7.1 (32bit) on redhat EL 3 64bit, on XEON E5335 with 4 gig ram.


Error when creating a shared memory segment (size=35682416), errno=22



Lol

Hmmm ?




-Original Message-
From: u2-users-boun...@listserver.u2ug.org
[mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Ron Hutchings
Sent: 09 July 2009 13:40
To: u2-users@listserver.u2ug.org
Subject: Re: [U2] General guidelines on indexing


IBM BOX (OLDER)
PICK.FORMAT
10.0.11
6.9377 seconds

Linux box
PICK.FORMAT
10.2.0
0.3522 seconds

  

Date: Thu, 9 Jul 2009 11:20:25 +0800
From: adrian.wom...@rac.com.au
To: u2-users@listserver.u2ug.org
Subject: Re: [U2] General guidelines on indexing


I just tried this example on Universe 10.2.6 - it took 0.0665 seconds -
basically instant, can't complain about that.
 


-Original Message-
From: u2-users-boun...@listserver.u2ug.org
[mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Martin
Phillips
Sent: Thursday, 9 July 2009 12:59 AM
To: U2 Users List
Subject: Re: [U2] General guidelines on indexing


By way of a simple expample, I just tried the following program...
   s = ''
   z = str('*', 1000)
   t1 = time()
   for i = 1 to 10
  s<-1> = z
   next i
   t2 = time()
   crt t2 - t1

This took six seconds on QM but 32 minutes on UniVerse. 





DISCLAIMER:
Disclaimer.  This e-mail is private and confidential. If you are not the


intended recipient, please advise us by return e-mail immediately, and
delete the e-mail and any attachments without using or disclosing the
contents in any way. The views expressed in this e-mail are those of the
author, and do not represent those of this company unless this is clearly
indicated. You should scan this e-mail and any attachments for viruses. This
company accepts no liability for any direct or indirect damage or loss
resulting from the use of any attachments to this e-mail.
  

___
U2-Users mailing list
U2-Users@listserver.u2ug.org
http://listserver.u2ug.org/mailman/listinfo/u2-users



_
HotmailR has ever-growing storage! Don't worry about storage limits. 
http://windowslive.com/Tutorial/Hotmail/Storage?ocid=TXT_TAGLM_WL_HM_Tutoria

l_Storage_062009
___
U2-Users mailing list
U2-Users@listserver.u2ug.org
http://listserver.u2ug.org/mailman/listinfo/u2-users

___
U2-Users mailing list
U2-Users@listserver.u2ug.org
http://listserver.u2ug.org/mailman/listinfo/u2-users

  

___
U2-Users mailing list
U2-Users@listserver.u2ug.org
http://listserver.u2ug.org/mailman/listinfo/u2-users

Re: [U2] General guidelines on indexing

[Symeon Breen]

Ok I am very jealous now of all these quick times, when my one crashed out 

Can someone remind me (its been years since i did any udt config stuff) what
should i be looking at to fix this error 


Thanks



PS - i like this thread i think this can be of benefit to people to fine
tune both code and their os and u2 configuration !



-Original Message-
From: Symeon Breen [mailto:syme...@gmail.com] 
Sent: 09 July 2009 14:04
To: 'U2 Users List'
Subject: RE: [U2] General guidelines on indexing

Unidata 7.1 (32bit) on redhat EL 3 64bit, on XEON E5335 with 4 gig ram.


Error when creating a shared memory segment (size=35682416), errno=22



Lol

Hmmm ?




-Original Message-
From: u2-users-boun...@listserver.u2ug.org
[mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Ron Hutchings
Sent: 09 July 2009 13:40
To: u2-users@listserver.u2ug.org
Subject: Re: [U2] General guidelines on indexing


IBM BOX (OLDER)
PICK.FORMAT
10.0.11
6.9377 seconds

Linux box
PICK.FORMAT
10.2.0
0.3522 seconds

> Date: Thu, 9 Jul 2009 11:20:25 +0800
> From: adrian.wom...@rac.com.au
> To: u2-users@listserver.u2ug.org
> Subject: Re: [U2] General guidelines on indexing
> 
> 
> I just tried this example on Universe 10.2.6 - it took 0.0665 seconds -
> basically instant, can't complain about that.
>  
> 
> -Original Message-
> From: u2-users-boun...@listserver.u2ug.org
> [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Martin
> Phillips
> Sent: Thursday, 9 July 2009 12:59 AM
> To: U2 Users List
> Subject: Re: [U2] General guidelines on indexing
> 
> 
> By way of a simple expample, I just tried the following program...
>s = ''
>z = str('*', 1000)
>t1 = time()
>for i = 1 to 10
>   s<-1> = z
>next i
>t2 = time()
>crt t2 - t1
> 
> This took six seconds on QM but 32 minutes on UniVerse. 
> 
> 
> 
> 
> DISCLAIMER:
> Disclaimer.  This e-mail is private and confidential. If you are not the
intended recipient, please advise us by return e-mail immediately, and
delete the e-mail and any attachments without using or disclosing the
contents in any way. The views expressed in this e-mail are those of the
author, and do not represent those of this company unless this is clearly
indicated. You should scan this e-mail and any attachments for viruses. This
company accepts no liability for any direct or indirect damage or loss
resulting from the use of any attachments to this e-mail.
> ___
> U2-Users mailing list
> U2-Users@listserver.u2ug.org
> http://listserver.u2ug.org/mailman/listinfo/u2-users

_
HotmailR has ever-growing storage! Don't worry about storage limits. 
http://windowslive.com/Tutorial/Hotmail/Storage?ocid=TXT_TAGLM_WL_HM_Tutoria
l_Storage_062009
___
U2-Users mailing list
U2-Users@listserver.u2ug.org
http://listserver.u2ug.org/mailman/listinfo/u2-users

___
U2-Users mailing list
U2-Users@listserver.u2ug.org
http://listserver.u2ug.org/mailman/listinfo/u2-users

Re: [U2] General guidelines on indexing

[JPB-U2UG]

I tried it on my system UV 10.1.12 on Linux AS3 0.643 seconds.

--
From: "Martin Phillips" 
Sent: Thursday, July 09, 2009 3:38 AM
To: "U2 Users List" 
Subject: Re: [U2] General guidelines on indexing


Hi Adrian,


I just tried this example on Universe 10.2.6 - it took 0.0665
seconds basically instant, can't complain about that.


I am sure that something must have been wrong with your test. This 
probably isn't long enough to do even the empty loop with no string copy.


I have just repeated my test on a different system running UV 10.2.8 and 
the test took 38 minutes.



Martin Phillips
Ladybridge Systems Ltd
17b Coldstream Lane, Hardingstone, Northampton, NN4 6DB
+44-(0)1604-709200



___
U2-Users mailing list
U2-Users@listserver.u2ug.org
http://listserver.u2ug.org/mailman/listinfo/u2-users


___
U2-Users mailing list
U2-Users@listserver.u2ug.org
http://listserver.u2ug.org/mailman/listinfo/u2-users

Re: [U2] General guidelines on indexing

[Marc Harbeson]

Dell Poweredge 2900, 4 core, 2 cpu Xeon - 16gb ram, UDT 6.1 - Windows
2003 R2

s = ''
z = STR('*', 1000)
t1 = SYSTEM(12)
FOR i = 1 TO 10
   s<-1> = z
NEXT i
t2 = SYSTEM(12)
CRT t2 - t1

Output: 1219
___
U2-Users mailing list
U2-Users@listserver.u2ug.org
http://listserver.u2ug.org/mailman/listinfo/u2-users

Re: [U2] General guidelines on indexing

[Craig McDonald]

Ran this on IBM AIX P570 8-core with 16gb UV 10.1.17 and 300 users logged
on.

0.095 seconds

-Craig


>>>s = ''
>>>z = str('*', 1000)
>>>t1 = time()
>>>for i = 1 to 10
>>>   s<-1> = z
>>>next i
>>>t2 = time()
>>>crt t2 - t1

-Original Message-
From: u2-users-boun...@listserver.u2ug.org
[mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Joshua Gallant
Sent: Thursday, July 09, 2009 12:58 PM
To: U2 Users List
Subject: Re: [U2] General guidelines on indexing

0.1239 seconds running on a Solaris machine with UV 10.2.2 and 250 other
users.

- Josh




___
U2-Users mailing list
U2-Users@listserver.u2ug.org
http://listserver.u2ug.org/mailman/listinfo/u2-users

Re: [U2] General guidelines on indexing

[Joshua Gallant]

0.1239 seconds running on a Solaris machine with UV 10.2.2 and 250 other
users.

- Josh


-Original Message-
From: u2-users-boun...@listserver.u2ug.org
[mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Bill Haskett
Sent: Thursday, July 09, 2009 1:56 PM
To: U2 Users List
Subject: Re: [U2] General guidelines on indexing

UD v7.2 on Dell 1435 server w/2GB RAM and 180 SATA  RAID 1 drive.

  s = ''
  z = STR('*', 1000)
  t1 = SYSTEM(12)
  FOR i = 1 TO 10
 s<-1> = z
  NEXT i
  t2 = SYSTEM(12)
  CRT t2 - t1 : " milliseconds."

-> RUN BP TEST
1625 milliseconds

Bill

Steve Romanow said the following on 7/9/2009 6:04 AM:
> IBM 2x1.8g power5 (i think) 100 users logged in
> 2.5 seconds
>
> Ron Hutchings wrote:
>> IBM BOX (OLDER)
>> PICK.FORMAT
>> 10.0.11
>> 6.9377 seconds
>>
>> Linux box
>> PICK.FORMAT
>> 10.2.0
>> 0.3522 seconds
>>
>>  
>>> Date: Thu, 9 Jul 2009 11:20:25 +0800
>>> From: adrian.wom...@rac.com.au
>>> To: u2-users@listserver.u2ug.org
>>> Subject: Re: [U2] General guidelines on indexing
>>>
>>>
>>> I just tried this example on Universe 10.2.6 - it took 0.0665
seconds -
>>> basically instant, can't complain about that.
>>>  
>>>
>>> -Original Message-
>>> From: u2-users-boun...@listserver.u2ug.org
>>> [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Martin
>>> Phillips
>>> Sent: Thursday, 9 July 2009 12:59 AM
>>> To: U2 Users List
>>> Subject: Re: [U2] General guidelines on indexing
>>>
>>>
>>> By way of a simple expample, I just tried the following program...
>>>s = ''
>>>z = str('*', 1000)
>>>t1 = time()
>>>for i = 1 to 10
>>>   s<-1> = z
>>>next i
>>>t2 = time()
>>>crt t2 - t1
>>>
>>> This took six seconds on QM but 32 minutes on UniVerse.
>>>
___
U2-Users mailing list
U2-Users@listserver.u2ug.org
http://listserver.u2ug.org/mailman/listinfo/u2-users
___
U2-Users mailing list
U2-Users@listserver.u2ug.org
http://listserver.u2ug.org/mailman/listinfo/u2-users

Re: [U2] General guidelines on indexing

[Bill Haskett]

UD v7.2 on Dell 1435 server w/2GB RAM and 180 SATA  RAID 1 drive.

 s = ''
 z = STR('*', 1000)
 t1 = SYSTEM(12)
 FOR i = 1 TO 10
s<-1> = z
 NEXT i
 t2 = SYSTEM(12)
 CRT t2 - t1 : " milliseconds."

-> RUN BP TEST
1625 milliseconds

Bill

Steve Romanow said the following on 7/9/2009 6:04 AM:

IBM 2x1.8g power5 (i think) 100 users logged in
2.5 seconds

Ron Hutchings wrote:

IBM BOX (OLDER)
PICK.FORMAT
10.0.11
6.9377 seconds

Linux box
PICK.FORMAT
10.2.0
0.3522 seconds

 

Date: Thu, 9 Jul 2009 11:20:25 +0800
From: adrian.wom...@rac.com.au
To: u2-users@listserver.u2ug.org
Subject: Re: [U2] General guidelines on indexing


I just tried this example on Universe 10.2.6 - it took 0.0665 seconds -
basically instant, can't complain about that.
 


-Original Message-
From: u2-users-boun...@listserver.u2ug.org
[mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Martin
Phillips
Sent: Thursday, 9 July 2009 12:59 AM
To: U2 Users List
Subject: Re: [U2] General guidelines on indexing


By way of a simple expample, I just tried the following program...
   s = ''
   z = str('*', 1000)
   t1 = time()
   for i = 1 to 10
  s<-1> = z
   next i
   t2 = time()
   crt t2 - t1

This took six seconds on QM but 32 minutes on UniVerse.


___
U2-Users mailing list
U2-Users@listserver.u2ug.org
http://listserver.u2ug.org/mailman/listinfo/u2-users

Re: [U2] General guidelines on indexing

[Martin Phillips]

Hi Marco,


The dynamic array test took 30 minutes.

I compiled the same program on jBASE 4.1 on the same laptop
and it completed in 0 seconds!


This will be because jBase effectively converts the program to C. Most C 
compilers are so good at optimisation that they will work out what the 
program does and compile it as simply setting the string in one step.



Martin Phillips
Ladybridge Systems Ltd
17b Coldstream Lane, Hardingstone, Northampton, NN4 6DB
+44-(0)1604-709200 


___
U2-Users mailing list
U2-Users@listserver.u2ug.org
http://listserver.u2ug.org/mailman/listinfo/u2-users

Re: [U2] General guidelines on indexing

[David Ward]

Perhaps our approach was different. We created a hash of sorts and the size
isn't an issue. Here's the mod I made to the code. One of the developers I
work with came up with this. Works great.

DIM LOC.ARY(4000)
MAT LOC.ARY=''
Z=STR('*',1000)
T1=TIME()
FOR X=1 TO 10
  BUCKET=MOD(CHECKSUM(X),4000)+1
  LOC.ARY(BUCKET)<-1>=Z
NEXT X
T2=TIME()
CRT OCONV(T2-T1,'MTS')
Elapsed Time 00:00:00 (0.172)

-Original Message-
From: u2-users-boun...@listserver.u2ug.org
[mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Baakkonen, Rodney
A (Rod) 46K
Sent: Thursday, July 09, 2009 10:36 AM
To: U2 Users List
Subject: Re: [U2] General guidelines on indexing


 We used to us the DIM approach as well. But having the process blow up a
couple of years later because the DIM array was not large enough to
accommodate the data caused us to go the work file method. The DIM array was
fast, but needed to much upkeep. And after Sarbanes-Oxley, making changes to
programs took a whole new perspective. But that is another rant for another
time.

-Original Message-
From: u2-users-boun...@listserver.u2ug.org
[mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of David Ward
Sent: Thursday, July 09, 2009 9:24 AM
To: 'U2 Users List'
Subject: Re: [U2] General guidelines on indexing

Original test and a couple of other variations. We use the DIM approach in
several cases where variable data is large and temporary.


Test system run on
WindowsXP
Single DuoCore @2.20ghz
2GB Ram

Original version - <-1> variable append
Elapsed Time 00:25:04

Altered version - Using WRITESEQ - Same System
1.203 seconds

Altered version - Using DIM Variable - Same System
0.172 seconds


___
U2-Users mailing list
U2-Users@listserver.u2ug.org
http://listserver.u2ug.org/mailman/listinfo/u2-users






--
CONFIDENTIALITY NOTICE: If you have received this email in error, please
immediately notify the sender by e-mail at the address shown.  This email
transmission may contain confidential information.  This information is
intended only for the use of the individual(s) or entity to whom it is
intended even if addressed incorrectly.  Please delete it from your files if
you are not the intended recipient.  Thank you for your compliance.
Copyright 2009 CIGNA

==

___
U2-Users mailing list
U2-Users@listserver.u2ug.org
http://listserver.u2ug.org/mailman/listinfo/u2-users

___
U2-Users mailing list
U2-Users@listserver.u2ug.org
http://listserver.u2ug.org/mailman/listinfo/u2-users

Re: [U2] General guidelines on indexing

[Baakkonen, Rodney A (Rod) 46K]

 We used to us the DIM approach as well. But having the process blow up
a couple of years later because the DIM array was not large enough to
accommodate the data caused us to go the work file method. The DIM array
was fast, but needed to much upkeep. And after Sarbanes-Oxley, making
changes to programs took a whole new perspective. But that is another
rant for another time.

-Original Message-
From: u2-users-boun...@listserver.u2ug.org
[mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of David Ward
Sent: Thursday, July 09, 2009 9:24 AM
To: 'U2 Users List'
Subject: Re: [U2] General guidelines on indexing

Original test and a couple of other variations. We use the DIM approach
in
several cases where variable data is large and temporary.


Test system run on
WindowsXP
Single DuoCore @2.20ghz
2GB Ram

Original version - <-1> variable append
Elapsed Time 00:25:04

Altered version - Using WRITESEQ - Same System
1.203 seconds

Altered version - Using DIM Variable - Same System
0.172 seconds


___
U2-Users mailing list
U2-Users@listserver.u2ug.org
http://listserver.u2ug.org/mailman/listinfo/u2-users





--
CONFIDENTIALITY NOTICE: If you have received this email in error, please 
immediately notify the sender by e-mail at the address shown.  This email 
transmission may contain confidential information.  This information is 
intended only for the use of the individual(s) or entity to whom it is intended 
even if addressed incorrectly.  Please delete it from your files if you are not 
the intended recipient.  Thank you for your compliance.  Copyright 2009 CIGNA
==

___
U2-Users mailing list
U2-Users@listserver.u2ug.org
http://listserver.u2ug.org/mailman/listinfo/u2-users

Re: [U2] General guidelines on indexing

[Edward Brown]

These performance postings are interesting and show the very wide range
between different flavours of Pick and the underlying operating system.

Would anyone be interested in helping put together / running a
performance suite with a view, perhaps, of adding a page to the wiki
showing how different systems compare to each other? This may help
people in the future when deciding which system to adopt, but also be a
way of seeing which methods perform best on particular setups OR across
the board, for those of us who target multiple platforms from a single
code base.

Ed

-Original Message-
From: u2-users-boun...@listserver.u2ug.org
[mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Marco
Manyevere
Sent: 09 July 2009 16:28
To: U2 Users List
Subject: Re: [U2] General guidelines on indexing


On my DualCore WindowsXP laptop with 3GB ram @2.16GHZ, (UV 10.0.04):

The dynamic array test took 30 minutes.

I compiled the same program on jBASE 4.1 on the same laptop and it
completed in 0 seconds! I had to print LEN(S) before and after to
confirm that the program was indeed running!




- Original Message 
From: David Ward 
To: U2 Users List 
Sent: Thursday, 9 July, 2009 16:24:17
Subject: Re: [U2] General guidelines on indexing

Original test and a couple of other variations. We use the DIM approach
in
several cases where variable data is large and temporary.


Test system run on
WindowsXP
Single DuoCore @2.20ghz
2GB Ram

Original version - <-1> variable append
Elapsed Time 00:25:04

Altered version - Using WRITESEQ - Same System
1.203 seconds

Altered version - Using DIM Variable - Same System
0.172 seconds


___
U2-Users mailing list
U2-Users@listserver.u2ug.org
http://listserver.u2ug.org/mailman/listinfo/u2-users



  
___
U2-Users mailing list
U2-Users@listserver.u2ug.org
http://listserver.u2ug.org/mailman/listinfo/u2-users

---
Please remember to recycle wherever possible. 
Reduce, reuse, recycle, think do you need to print this e-mail?
---
This e-mail and any attachment(s), is confidential and may be legally 
privileged. It is intended solely for the addressee. If you are not the 
addressee, dissemination, copying or use of this e-mail or any of its content 
is prohibited and may be unlawful. If you are not the intended recipient please 
inform the sender immediately and destroy the e-mail, any attachment(s) and any 
copies. All liability for viruses is excluded to the fullest extent permitted 
by law. It is your responsibility to scan or otherwise check this email and any 
attachment(s). Unless otherwise stated (i) views expressed in this message are 
those of the individual sender (ii) no contract may be construed by this 
e-mail. Emails may be monitored and you are taken to consent to this 
monitoring.  

Civica Services Limited, Company No. 02374268; Civica UK Limited, Company No. 
01628868
Both companies are registered in England and Wales and each has its registered 
office at 2 Burston Road, Putney, London, SW15 6AR.
---

___
U2-Users mailing list
U2-Users@listserver.u2ug.org
http://listserver.u2ug.org/mailman/listinfo/u2-users

Re: [U2] General guidelines on indexing

[Marco Manyevere]

On my DualCore WindowsXP laptop with 3GB ram @2.16GHZ, (UV 10.0.04):

The dynamic array test took 30 minutes.

I compiled the same program on jBASE 4.1 on the same laptop and it completed in 
0 seconds! I had to print LEN(S) before and after to confirm that the program 
was indeed running!




- Original Message 
From: David Ward 
To: U2 Users List 
Sent: Thursday, 9 July, 2009 16:24:17
Subject: Re: [U2] General guidelines on indexing

Original test and a couple of other variations. We use the DIM approach in
several cases where variable data is large and temporary.


Test system run on
WindowsXP
Single DuoCore @2.20ghz
2GB Ram

Original version - <-1> variable append
Elapsed Time 00:25:04

Altered version - Using WRITESEQ - Same System
1.203 seconds

Altered version - Using DIM Variable - Same System
0.172 seconds


___
U2-Users mailing list
U2-Users@listserver.u2ug.org
http://listserver.u2ug.org/mailman/listinfo/u2-users



  
___
U2-Users mailing list
U2-Users@listserver.u2ug.org
http://listserver.u2ug.org/mailman/listinfo/u2-users

Re: [U2] General guidelines on indexing

[David Ward]

Original test and a couple of other variations. We use the DIM approach in
several cases where variable data is large and temporary.


Test system run on
WindowsXP
Single DuoCore @2.20ghz
2GB Ram

Original version - <-1> variable append
Elapsed Time 00:25:04

Altered version - Using WRITESEQ - Same System
1.203 seconds

Altered version - Using DIM Variable - Same System
0.172 seconds


___
U2-Users mailing list
U2-Users@listserver.u2ug.org
http://listserver.u2ug.org/mailman/listinfo/u2-users

Re: [U2] General guidelines on indexing

[Brian Leach]

Martin

> Hi all,
> 
> I wish I hadn't started this!!
> 

Not at all - it was a useful post!

For those interested in such things, last month's PC Plus
geek-at-the-back-of-the-magazine article was all about just this subject,
and covered the concept of 'ropes' - basically handling strings internally
as a balanced tree to improve efficiency of append and insert operations.

Though sadly it didn't give any code samples..

Brian

> -Original Message-
> From: u2-users-boun...@listserver.u2ug.org 
> [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of 
> Martin Phillips
> Sent: 09 July 2009 14:56
> To: U2 Users List
> Subject: Re: [U2] General guidelines on indexing
> 
> I did my tests on Windows. It looks like I need to try other 
> platforms when time permits.
> 
> 
> Martin Phillips
> Ladybridge Systems Ltd
> 17b Coldstream Lane, Hardingstone, Northampton, NN4 6DB
> +44-(0)1604-709200 
> 
> ___
> U2-Users mailing list
> U2-Users@listserver.u2ug.org
> http://listserver.u2ug.org/mailman/listinfo/u2-users
> 

___
U2-Users mailing list
U2-Users@listserver.u2ug.org
http://listserver.u2ug.org/mailman/listinfo/u2-users

Re: [U2] General guidelines on indexing

[Martin Phillips]

Hi all,

I wish I hadn't started this!!

I did my tests on Windows. It looks like I need to try other platforms when 
time permits.



Martin Phillips
Ladybridge Systems Ltd
17b Coldstream Lane, Hardingstone, Northampton, NN4 6DB
+44-(0)1604-709200 


___
U2-Users mailing list
U2-Users@listserver.u2ug.org
http://listserver.u2ug.org/mailman/listinfo/u2-users

Re: [U2] General guidelines on indexing

[bradley . schrag]

Thanks for cleaning up my mess, Brian. You won't have to do it again, at 
least not because of me.  :-)

> In the meantime, I've copied those new sections on indexing into the new
> wiki and kept the attribution.

Brad Schrag
Application Consultant
Commercial Leasing Development
U.S. Bank
EP-MN-BGF
2751 Shepard Road
St. Paul, MN  55416
651-205-3074 direct
sdg
U.S. BANCORP made the following annotations
-
Electronic Privacy Notice. This e-mail, and any attachments, contains 
information that is, or may be, covered by electronic communications privacy 
laws, and is also confidential and proprietary in nature. If you are not the 
intended recipient, please be advised that you are legally prohibited from 
retaining, using, copying, distributing, or otherwise disclosing this 
information in any manner. Instead, please reply to the sender that you have 
received this communication in error, and then immediately delete it. Thank you 
in advance for your cooperation.



-

___
U2-Users mailing list
U2-Users@listserver.u2ug.org
http://listserver.u2ug.org/mailman/listinfo/u2-users

Re: [U2] General guidelines on indexing

[Robert Porter]

Ran the same code (well after adding and END to it) on 10.1.20 on HP-UX 11i (4x 
550Mhz PA-64 procs) ... 3 seconds.
 
Robert F. Porter, MCSE, CCNA, ZCE
Sr. Programmer / Analyst
Laboratory Information Services
Ochsner Health System
(504) 842 - 5185
 
 
 
This transmission (including any attachments) may contain confidential 
information, privileged material (including material protected by the 
solicitor-client or other applicable privileges), or constitute non-public 
information. Any use of this information by anyone other than the intended 
recipient is prohibited. If you have received this transmission in error, 
please immediately reply to the sender and delete this information from your 
system. Use, dissemination, distribution, or reproduction of this transmission 
by unintended recipients is not authorized and may be unlawful.


>>> "Martin Phillips"  7/9/2009 3:38 AM >>>
Hi Adrian,

> I just tried this example on Universe 10.2.6 - it took 0.0665
> seconds basically instant, can't complain about that.

I am sure that something must have been wrong with your test. This probably 
isn't long enough to do even the empty loop with no string copy.

I have just repeated my test on a different system running UV 10.2.8 and the 
test took 38 minutes.


Martin Phillips
Ladybridge Systems Ltd
17b Coldstream Lane, Hardingstone, Northampton, NN4 6DB
+44-(0)1604-709200



___
U2-Users mailing list
U2-Users@listserver.u2ug.org 
http://listserver.u2ug.org/mailman/listinfo/u2-users
___
U2-Users mailing list
U2-Users@listserver.u2ug.org
http://listserver.u2ug.org/mailman/listinfo/u2-users

Re: [U2] General guidelines on indexing

[Steve Romanow]

IBM 2x1.8g power5 (i think) 100 users logged in
2.5 seconds

Ron Hutchings wrote:

IBM BOX (OLDER)
PICK.FORMAT
10.0.11
6.9377 seconds

Linux box
PICK.FORMAT
10.2.0
0.3522 seconds

  

Date: Thu, 9 Jul 2009 11:20:25 +0800
From: adrian.wom...@rac.com.au
To: u2-users@listserver.u2ug.org
Subject: Re: [U2] General guidelines on indexing


I just tried this example on Universe 10.2.6 - it took 0.0665 seconds -
basically instant, can't complain about that.
 


-Original Message-
From: u2-users-boun...@listserver.u2ug.org
[mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Martin
Phillips
Sent: Thursday, 9 July 2009 12:59 AM
To: U2 Users List
Subject: Re: [U2] General guidelines on indexing


By way of a simple expample, I just tried the following program...
   s = ''
   z = str('*', 1000)
   t1 = time()
   for i = 1 to 10
  s<-1> = z
   next i
   t2 = time()
   crt t2 - t1

This took six seconds on QM but 32 minutes on UniVerse. 





DISCLAIMER:
Disclaimer.  This e-mail is private and confidential. If you are not the 
intended recipient, please advise us by return e-mail immediately, and delete 
the e-mail and any attachments without using or disclosing the contents in any 
way. The views expressed in this e-mail are those of the author, and do not 
represent those of this company unless this is clearly indicated. You should 
scan this e-mail and any attachments for viruses. This company accepts no 
liability for any direct or indirect damage or loss resulting from the use of 
any attachments to this e-mail.
___
U2-Users mailing list
U2-Users@listserver.u2ug.org
http://listserver.u2ug.org/mailman/listinfo/u2-users



_
Hotmail® has ever-growing storage! Don’t worry about storage limits. 
http://windowslive.com/Tutorial/Hotmail/Storage?ocid=TXT_TAGLM_WL_HM_Tutorial_Storage_062009

___
U2-Users mailing list
U2-Users@listserver.u2ug.org
http://listserver.u2ug.org/mailman/listinfo/u2-users
  


___
U2-Users mailing list
U2-Users@listserver.u2ug.org
http://listserver.u2ug.org/mailman/listinfo/u2-users

Re: [U2] General guidelines on indexing

[Symeon Breen]

Unidata 7.1 (32bit) on redhat EL 3 64bit, on XEON E5335 with 4 gig ram.


Error when creating a shared memory segment (size=35682416), errno=22



Lol

Hmmm ?




-Original Message-
From: u2-users-boun...@listserver.u2ug.org
[mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Ron Hutchings
Sent: 09 July 2009 13:40
To: u2-users@listserver.u2ug.org
Subject: Re: [U2] General guidelines on indexing


IBM BOX (OLDER)
PICK.FORMAT
10.0.11
6.9377 seconds

Linux box
PICK.FORMAT
10.2.0
0.3522 seconds

> Date: Thu, 9 Jul 2009 11:20:25 +0800
> From: adrian.wom...@rac.com.au
> To: u2-users@listserver.u2ug.org
> Subject: Re: [U2] General guidelines on indexing
> 
> 
> I just tried this example on Universe 10.2.6 - it took 0.0665 seconds -
> basically instant, can't complain about that.
>  
> 
> -Original Message-
> From: u2-users-boun...@listserver.u2ug.org
> [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Martin
> Phillips
> Sent: Thursday, 9 July 2009 12:59 AM
> To: U2 Users List
> Subject: Re: [U2] General guidelines on indexing
> 
> 
> By way of a simple expample, I just tried the following program...
>s = ''
>z = str('*', 1000)
>t1 = time()
>for i = 1 to 10
>   s<-1> = z
>next i
>t2 = time()
>crt t2 - t1
> 
> This took six seconds on QM but 32 minutes on UniVerse. 
> 
> 
> 
> 
> DISCLAIMER:
> Disclaimer.  This e-mail is private and confidential. If you are not the
intended recipient, please advise us by return e-mail immediately, and
delete the e-mail and any attachments without using or disclosing the
contents in any way. The views expressed in this e-mail are those of the
author, and do not represent those of this company unless this is clearly
indicated. You should scan this e-mail and any attachments for viruses. This
company accepts no liability for any direct or indirect damage or loss
resulting from the use of any attachments to this e-mail.
> ___
> U2-Users mailing list
> U2-Users@listserver.u2ug.org
> http://listserver.u2ug.org/mailman/listinfo/u2-users

_
HotmailR has ever-growing storage! Don't worry about storage limits. 
http://windowslive.com/Tutorial/Hotmail/Storage?ocid=TXT_TAGLM_WL_HM_Tutoria
l_Storage_062009
___
U2-Users mailing list
U2-Users@listserver.u2ug.org
http://listserver.u2ug.org/mailman/listinfo/u2-users

___
U2-Users mailing list
U2-Users@listserver.u2ug.org
http://listserver.u2ug.org/mailman/listinfo/u2-users

Re: [U2] General guidelines on indexing

[Ron Hutchings]

IBM BOX (OLDER)
PICK.FORMAT
10.0.11
6.9377 seconds

Linux box
PICK.FORMAT
10.2.0
0.3522 seconds

> Date: Thu, 9 Jul 2009 11:20:25 +0800
> From: adrian.wom...@rac.com.au
> To: u2-users@listserver.u2ug.org
> Subject: Re: [U2] General guidelines on indexing
> 
> 
> I just tried this example on Universe 10.2.6 - it took 0.0665 seconds -
> basically instant, can't complain about that.
>  
> 
> -Original Message-
> From: u2-users-boun...@listserver.u2ug.org
> [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Martin
> Phillips
> Sent: Thursday, 9 July 2009 12:59 AM
> To: U2 Users List
> Subject: Re: [U2] General guidelines on indexing
> 
> 
> By way of a simple expample, I just tried the following program...
>s = ''
>z = str('*', 1000)
>t1 = time()
>for i = 1 to 10
>   s<-1> = z
>next i
>t2 = time()
>crt t2 - t1
> 
> This took six seconds on QM but 32 minutes on UniVerse. 
> 
> 
> 
> 
> DISCLAIMER:
> Disclaimer.  This e-mail is private and confidential. If you are not the 
> intended recipient, please advise us by return e-mail immediately, and delete 
> the e-mail and any attachments without using or disclosing the contents in 
> any way. The views expressed in this e-mail are those of the author, and do 
> not represent those of this company unless this is clearly indicated. You 
> should scan this e-mail and any attachments for viruses. This company accepts 
> no liability for any direct or indirect damage or loss resulting from the use 
> of any attachments to this e-mail.
> ___
> U2-Users mailing list
> U2-Users@listserver.u2ug.org
> http://listserver.u2ug.org/mailman/listinfo/u2-users

_
Hotmail® has ever-growing storage! Don’t worry about storage limits. 
http://windowslive.com/Tutorial/Hotmail/Storage?ocid=TXT_TAGLM_WL_HM_Tutorial_Storage_062009
___
U2-Users mailing list
U2-Users@listserver.u2ug.org
http://listserver.u2ug.org/mailman/listinfo/u2-users

Re: [U2] General guidelines on indexing

[Brian Leach]

All

PLEASE don't use that wiki page - it's the old wiki ..

The live wiki is accessed from the main U2UG website (log in and select Wiki
from the menu).
The old wiki will be removed some day soon when I get the time to do so
cleanly and check that everything has been migrated across.

Why?

Because the old wiki is an off-the-shelf python product that is actually
very good, but gets hacked frequently and I have to clean up the mess
afterwards. It obviously advertises its presence because in the days
following a post, all kinds of bad stuff hits the website, and it starts
generating pages.. That's probably enough said.

The live wiki is UniVerse based :) and linked into the rest of the site. So
far is only open to members, who are of course far too sensible to post crap
all over it. Sadly, that's why it insists you log in before accessing it:
I'll see about streamlining that better now I remember.

In the meantime, I've copied those new sections on indexing into the new
wiki and kept the attribution.

Thanks all


Brian

> -Original Message-
> From: u2-users-boun...@listserver.u2ug.org 
> [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of 
> Baker Hughes
> Sent: 08 July 2009 15:30
> To: U2 Users List
> Subject: Re: [U2] General guidelines on indexing
> 
> Maybe one of you would be willing to pull all these good 
> posts on indexing into a wiki article?  It would be handy 
> reference material and could save someone from hours of 
> pain can't get this stuff from reading the system docs.
> 
> http://212.241.202.162/U2UGWiki/moin.cgi/ForDevelopers
> 
> -Baker
> 
> 
> This communication, its contents and any file attachments 
> transmitted with it are intended solely for the addressee(s) 
> and may contain confidential proprietary information.
> Access by any other party without the express written 
> permission of the sender is STRICTLY PROHIBITED.
> If you have received this communication in error you may not 
> copy, distribute or use the contents, attachments or 
> information in any way.  Please destroy it and contact the sender.
> ___
> U2-Users mailing list
> U2-Users@listserver.u2ug.org
> http://listserver.u2ug.org/mailman/listinfo/u2-users
> 

___
U2-Users mailing list
U2-Users@listserver.u2ug.org
http://listserver.u2ug.org/mailman/listinfo/u2-users

Re: [U2] General guidelines on indexing

[Womack, Adrian]

The code looks fine to me.

I added "crt len(s)" to the end of the program and it printed 10009
(that's what I'd have expected).

I'm running on an HP-UX ia64 box - maybe you're on Windows?
 

-Original Message-
From: u2-users-boun...@listserver.u2ug.org
[mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Martin
Phillips
Sent: Thursday, 9 July 2009 4:39 PM
To: U2 Users List
Subject: Re: [U2] General guidelines on indexing

Hi Adrian,

> I just tried this example on Universe 10.2.6 - it took 0.0665 seconds 
> basically instant, can't complain about that.

I am sure that something must have been wrong with your test. This
probably isn't long enough to do even the empty loop with no string
copy.

I have just repeated my test on a different system running UV 10.2.8 and
the test took 38 minutes.


Martin Phillips
Ladybridge Systems Ltd
17b Coldstream Lane, Hardingstone, Northampton, NN4 6DB
+44-(0)1604-709200

 

___
U2-Users mailing list
U2-Users@listserver.u2ug.org
http://listserver.u2ug.org/mailman/listinfo/u2-users


DISCLAIMER:
Disclaimer.  This e-mail is private and confidential. If you are not the 
intended recipient, please advise us by return e-mail immediately, and delete 
the e-mail and any attachments without using or disclosing the contents in any 
way. The views expressed in this e-mail are those of the author, and do not 
represent those of this company unless this is clearly indicated. You should 
scan this e-mail and any attachments for viruses. This company accepts no 
liability for any direct or indirect damage or loss resulting from the use of 
any attachments to this e-mail.
___
U2-Users mailing list
U2-Users@listserver.u2ug.org
http://listserver.u2ug.org/mailman/listinfo/u2-users

Re: [U2] General guidelines on indexing

[Martin Phillips]

Hi Adrian,


I just tried this example on Universe 10.2.6 - it took 0.0665
seconds basically instant, can't complain about that.


I am sure that something must have been wrong with your test. This probably 
isn't long enough to do even the empty loop with no string copy.


I have just repeated my test on a different system running UV 10.2.8 and the 
test took 38 minutes.



Martin Phillips
Ladybridge Systems Ltd
17b Coldstream Lane, Hardingstone, Northampton, NN4 6DB
+44-(0)1604-709200



___
U2-Users mailing list
U2-Users@listserver.u2ug.org
http://listserver.u2ug.org/mailman/listinfo/u2-users

Re: [U2] General guidelines on indexing

[Womack, Adrian]

I just tried this example on Universe 10.2.6 - it took 0.0665 seconds -
basically instant, can't complain about that.
 

-Original Message-
From: u2-users-boun...@listserver.u2ug.org
[mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Martin
Phillips
Sent: Thursday, 9 July 2009 12:59 AM
To: U2 Users List
Subject: Re: [U2] General guidelines on indexing


By way of a simple expample, I just tried the following program...
   s = ''
   z = str('*', 1000)
   t1 = time()
   for i = 1 to 10
  s<-1> = z
   next i
   t2 = time()
   crt t2 - t1

This took six seconds on QM but 32 minutes on UniVerse. 




DISCLAIMER:
Disclaimer.  This e-mail is private and confidential. If you are not the 
intended recipient, please advise us by return e-mail immediately, and delete 
the e-mail and any attachments without using or disclosing the contents in any 
way. The views expressed in this e-mail are those of the author, and do not 
represent those of this company unless this is clearly indicated. You should 
scan this e-mail and any attachments for viruses. This company accepts no 
liability for any direct or indirect damage or loss resulting from the use of 
any attachments to this e-mail.
___
U2-Users mailing list
U2-Users@listserver.u2ug.org
http://listserver.u2ug.org/mailman/listinfo/u2-users

[U2] General guidelines on indexing - Wiki updated

[bradley . schrag]

I took a stab at extracting and compiling index-related info from this 
thread. It's not pretty, but it's a start. Feel very free to update, 
restructure or change as seems appropriate.

http://212.241.202.162/U2UGWiki/moin.cgi/ForDevelopers


Brad.
U.S. BANCORP made the following annotations
-
Electronic Privacy Notice. This e-mail, and any attachments, contains 
information that is, or may be, covered by electronic communications privacy 
laws, and is also confidential and proprietary in nature. If you are not the 
intended recipient, please be advised that you are legally prohibited from 
retaining, using, copying, distributing, or otherwise disclosing this 
information in any manner. Instead, please reply to the sender that you have 
received this communication in error, and then immediately delete it. Thank you 
in advance for your cooperation.



-

___
U2-Users mailing list
U2-Users@listserver.u2ug.org
http://listserver.u2ug.org/mailman/listinfo/u2-users

Re: [U2] General guidelines on indexing

[Martin Phillips]

Hi Symeon,


Ha Ha a dev from IBM letting secrets out to ladybridge  ;)


Yes, it was a bit much to hope!!!


Martin Phillips
Ladybridge Systems Ltd
17b Coldstream Lane, Hardingstone, Northampton, NN4 6DB
+44-(0)1604-709200
___
U2-Users mailing list
U2-Users@listserver.u2ug.org
http://listserver.u2ug.org/mailman/listinfo/u2-users

Re: [U2] General guidelines on indexing

[Symeon Breen]

Ha Ha a dev from IBM letting secrets out to ladybridge  ;)

-Original Message-
From: u2-users-boun...@listserver.u2ug.org
[mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Martin Phillips
Sent: 08 July 2009 20:57
To: U2 Users List
Subject: Re: [U2] General guidelines on indexing

Hi Ed,

> Your test program on unidata 7.1 takes 1137 milliseconds
> - 1.1 seconds.

Whilst this is not a totally valid comparison since your test was done on 
different hardware (mine was on my laptop), it does show that Unidata almost

certainly cannot be using contiguous strings as it would take longer than 
this to copy the data. I have been intrigued by some of the Unidata 
performance figures I have obtained in the past where some things that 
should take a long time are actually very fast while sometimes the opposite 
is true. I am left wondering if there are some clever optimisations being 
done in the compiler. Perhaps someone from the development group will let a 
few secrets out.


Martin Phillips
Ladybridge Systems Ltd
17b Coldstream Lane, Hardingstone, Northampton, NN4 6DB
+44-(0)1604-709200

___
U2-Users mailing list
U2-Users@listserver.u2ug.org
http://listserver.u2ug.org/mailman/listinfo/u2-users

___
U2-Users mailing list
U2-Users@listserver.u2ug.org
http://listserver.u2ug.org/mailman/listinfo/u2-users

Re: [U2] General guidelines on indexing

[Martin Phillips]

Hi again Ed,


SELECT file WITH PRINT.DATE GT "01/04/2008" AND WITH ADDR LIKE "'A'0X"


This query can be resolved with an index. Also, the optimiser will shuffle 
the clauses to make best use of indices. Unidata had query optimisation 
before UniVerse but I believe that essentially the same algorithms were 
ported across for UV 10. I have played games trying to defeat the UV 
optimiser and I have to say the I am very impressed by it.




SELECT file WITH PRINT.DATE GT "01/04/2008" OR WITH ADDR LIKE "'A'0X"


This cannot be resolved with an index because the second condition cannot be 
determined without reading all the records.



Also, my indexes look to be out of date, as the non-indexed and
indexed selects return different numbers.


That is worrying since the index system is supposed to guarantee consistency 
between the data file and the index. The only time that they should ever be 
able to get out of step is after a system failure or killing a process in 
the middle of a write.



Martin Phillips
Ladybridge Systems Ltd
17b Coldstream Lane, Hardingstone, Northampton, NN4 6DB
+44-(0)1604-709200 


___
U2-Users mailing list
U2-Users@listserver.u2ug.org
http://listserver.u2ug.org/mailman/listinfo/u2-users

Re: [U2] General guidelines on indexing

[Martin Phillips]

Hi Ed,


Your test program on unidata 7.1 takes 1137 milliseconds
- 1.1 seconds.


Whilst this is not a totally valid comparison since your test was done on 
different hardware (mine was on my laptop), it does show that Unidata almost 
certainly cannot be using contiguous strings as it would take longer than 
this to copy the data. I have been intrigued by some of the Unidata 
performance figures I have obtained in the past where some things that 
should take a long time are actually very fast while sometimes the opposite 
is true. I am left wondering if there are some clever optimisations being 
done in the compiler. Perhaps someone from the development group will let a 
few secrets out.



Martin Phillips
Ladybridge Systems Ltd
17b Coldstream Lane, Hardingstone, Northampton, NN4 6DB
+44-(0)1604-709200

___
U2-Users mailing list
U2-Users@listserver.u2ug.org
http://listserver.u2ug.org/mailman/listinfo/u2-users

Re: [U2] General guidelines on indexing

[Brian Leach]

All

Since we're already going off on a huge tangent re. indexing, it's worth
pointing out that if you're writing client side code in .Net the same
applies: strings are immutable and every change or append copies the string
in memory. That's why there is a StringBuilder class for appending to
strings, rather than using the String type.

Brian 

> 
> Different multivalue products approach string management in 
> varying ways. In UniVerse, strings are stored as contiguous 
> memory. If I write a statement such as
>X<-1> = 'ABC'
> this run machine has to work out how big the new string will 
> be, allocate memory, copy the old value of X to the new area 
> appending ABC to it, and then release the original memory used by X.

___
U2-Users mailing list
U2-Users@listserver.u2ug.org
http://listserver.u2ug.org/mailman/listinfo/u2-users

Re: [U2] General guidelines on indexing

[Edward Brown]

I want to move to one of my earlier questions in this thread regarding
mixing indexed and non-indexed dictionary items, and if unidata is able
to use the indexed items at all in this circumstance:

Simple test:

One fairly big file:

Items - 473,000
Item size - 695 bytes on average
Indexed dictionary items - PRINT.DATE

1. Selection using an indexed dictionary item - repeated three times,
average score:

SELECT file WITH PRINT.DATE GT "01/04/2008"
34,993 records selected
190ms


2. Selection using NO-INDEX: (this very slow, I killed my test program
after one pass)

SELECT file NO.INDEX WITH PRINT.DATE GT "01/04/2008"
55174 records selected
234 seconds

Selection using indexed and non-indexed dict item, ANDed together - this
should be a simple optimisation case:
SELECT file WITH PRINT.DATE GT "01/04/2008" AND WITH ADDR LIKE "'A'0X"
109 records selected
945ms


3. Selection using indexed and non-indexed dict item, ORed together -
there would not be much point using the index here as every record needs
to be read in anyway:

SELECT file WITH PRINT.DATE GT "01/04/2008" OR WITH ADDR LIKE "'A'0X"
56752 records selected
146 seconds

This is interesting! It proves that on unidata 7.1 at least the database
engine is able to optimise queries that are built of indexed and
non-indexed dictionary items. It means that there isn't much point doing
an initial selection based on indexed dictionary items and then further
reductions in code, where that is done simply to avoid a mix of the two.
It also means that indexing a subset of frequently used dictionary items
*is* worthwhile, even if the majority of select commands performed
day-to-day use additional dictionary items - so long as they are AND'd
together.

Other observations - the file is large but clearly getting cached by the
OS as it is repeatedly scanned, which is why the last set of selects is
faster than the no-index select.
Also, my indexes look to be out of date, as the non-indexed and indexed
selects return different numbers.

Ed

---
Please remember to recycle wherever possible. 
Reduce, reuse, recycle, think do you need to print this e-mail?
---
This e-mail and any attachment(s), is confidential and may be legally 
privileged. It is intended solely for the addressee. If you are not the 
addressee, dissemination, copying or use of this e-mail or any of its content 
is prohibited and may be unlawful. If you are not the intended recipient please 
inform the sender immediately and destroy the e-mail, any attachment(s) and any 
copies. All liability for viruses is excluded to the fullest extent permitted 
by law. It is your responsibility to scan or otherwise check this email and any 
attachment(s). Unless otherwise stated (i) views expressed in this message are 
those of the individual sender (ii) no contract may be construed by this 
e-mail. Emails may be monitored and you are taken to consent to this 
monitoring.  

Civica Services Limited, Company No. 02374268; Civica UK Limited, Company No. 
01628868
Both companies are registered in England and Wales and each has its registered 
office at 2 Burston Road, Putney, London, SW15 6AR.
---

___
U2-Users mailing list
U2-Users@listserver.u2ug.org
http://listserver.u2ug.org/mailman/listinfo/u2-users

Re: [U2] General guidelines on indexing

[Edward Brown]

Martin

Your test program on unidata 7.1 takes 1137 milliseconds - 1.1 seconds.

I changed it to use system(12), this is a better resolution clock on
unidata than TIME().

Interesting commentary on chunking. I believe (and I might be talking
out of my ar*e here) that chunking done with (system memory) page-sized
blocks could be made to appear contiguous to software sitting above the
operating system by taking advantage of the hardware vm / memory
controller. I would not be surprised if unidata benefitted from
something like this.

Ed





-Original Message-
From: u2-users-boun...@listserver.u2ug.org
[mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Martin
Phillips
Sent: 08 July 2009 17:59
To: U2 Users List
Subject: Re: [U2] General guidelines on indexing

Hi all,

>> I don't agree. Disk access is inherently slower than RAM access.

I think that this discussion started for Unidata and then got UniVerse 
involved too but it might have been the other way around. Sadly, there
is no 
internals training material for Unidata so we have to guess what goes
on.

Different multivalue products approach string management in varying
ways. In 
UniVerse, strings are stored as contiguous memory. If I write a
statement 
such as
   X<-1> = 'ABC'
this run machine has to work out how big the new string will be,
allocate 
memory, copy the old value of X to the new area appending ABC to it, and

then release the original memory used by X.

As you append successive fields, the string to be moved gets longer and 
longer. We tend to think of computers as being blindingly fast but
copying a 
big string is still a slow process. If I have a string that starts empty
and 
I add a million fields, each of 3 bytes plus the delimiter, I will end
up 
copying a total of 1,999,998,000,000 bytes - hardly an insignificant
task.

>From my own experiments some time ago, I believe that Unidata also uses

contiguous strings but I have no direct proof of this. The alternative 
(adopted by our QM product, by PI/open, Information and perhaps others)
is 
to use "chunked strings" where a string is stored as a series of chunks.
In 
this model, appending a field requires only addition of a new chunk or,
for 
better performance, replacement of the final chunk.

Of course, the performance gain of chunked strings in this example may
be 
offset by their decreased performance for things like substring
extraction 
which is now more complex than a simple indexing operation.

By way of a simple expample, I just tried the following program...
   s = ''
   z = str('*', 1000)
   t1 = time()
   for i = 1 to 10
  s<-1> = z
   next i
   t2 = time()
   crt t2 - t1

This took six seconds on QM but 32 minutes on UniVerse. I do not have a 
Unidata system available at the moment to try. To be fair, I am sure
that I 
could construct an example that reversed the performance difference.

Writing to a sequential file is somewhat similar to the chunked string
model 
as it buffers data until it has a good sized chunk and then writes it
out, 
continuing with an empty buffer.


Martin Phillips
Ladybridge Systems Ltd
17b Coldstream Lane, Hardingstone, Northampton, NN4 6DB
+44-(0)1604-709200 

___
U2-Users mailing list
U2-Users@listserver.u2ug.org
http://listserver.u2ug.org/mailman/listinfo/u2-users

---
Please remember to recycle wherever possible. 
Reduce, reuse, recycle, think do you need to print this e-mail?
---
This e-mail and any attachment(s), is confidential and may be legally 
privileged. It is intended solely for the addressee. If you are not the 
addressee, dissemination, copying or use of this e-mail or any of its content 
is prohibited and may be unlawful. If you are not the intended recipient please 
inform the sender immediately and destroy the e-mail, any attachment(s) and any 
copies. All liability for viruses is excluded to the fullest extent permitted 
by law. It is your responsibility to scan or otherwise check this email and any 
attachment(s). Unless otherwise stated (i) views expressed in this message are 
those of the individual sender (ii) no contract may be construed by this 
e-mail. Emails may be monitored and you are taken to consent to this 
monitoring.  

Civica Services Limited, Company No. 02374268; Civica UK Limited, Company No. 
01628868
Both companies are registered in England and Wales and each has its registered 
office at 2 Burston Road, Putney, London, SW15 6AR.
---

___
U2-Users mailing list
U2-Users@listserver.u2ug.org
http://listserver.u2ug.org/mailman/listinfo/u2-users

Re: [U2] General guidelines on indexing

[Martin Phillips]

Hi all,


I don't agree. Disk access is inherently slower than RAM access.


I think that this discussion started for Unidata and then got UniVerse 
involved too but it might have been the other way around. Sadly, there is no 
internals training material for Unidata so we have to guess what goes on.


Different multivalue products approach string management in varying ways. In 
UniVerse, strings are stored as contiguous memory. If I write a statement 
such as

  X<-1> = 'ABC'
this run machine has to work out how big the new string will be, allocate 
memory, copy the old value of X to the new area appending ABC to it, and 
then release the original memory used by X.


As you append successive fields, the string to be moved gets longer and 
longer. We tend to think of computers as being blindingly fast but copying a 
big string is still a slow process. If I have a string that starts empty and 
I add a million fields, each of 3 bytes plus the delimiter, I will end up 
copying a total of 1,999,998,000,000 bytes - hardly an insignificant task.


From my own experiments some time ago, I believe that Unidata also uses 
contiguous strings but I have no direct proof of this. The alternative 
(adopted by our QM product, by PI/open, Information and perhaps others) is 
to use "chunked strings" where a string is stored as a series of chunks. In 
this model, appending a field requires only addition of a new chunk or, for 
better performance, replacement of the final chunk.


Of course, the performance gain of chunked strings in this example may be 
offset by their decreased performance for things like substring extraction 
which is now more complex than a simple indexing operation.


By way of a simple expample, I just tried the following program...
  s = ''
  z = str('*', 1000)
  t1 = time()
  for i = 1 to 10
 s<-1> = z
  next i
  t2 = time()
  crt t2 - t1

This took six seconds on QM but 32 minutes on UniVerse. I do not have a 
Unidata system available at the moment to try. To be fair, I am sure that I 
could construct an example that reversed the performance difference.


Writing to a sequential file is somewhat similar to the chunked string model 
as it buffers data until it has a good sized chunk and then writes it out, 
continuing with an empty buffer.



Martin Phillips
Ladybridge Systems Ltd
17b Coldstream Lane, Hardingstone, Northampton, NN4 6DB
+44-(0)1604-709200 


___
U2-Users mailing list
U2-Users@listserver.u2ug.org
http://listserver.u2ug.org/mailman/listinfo/u2-users

Re: [U2] General guidelines on indexing

[bradley . schrag]

Excellent idea, Baker. I'll work on that after this thread settles down. I 
won't include the "dynamic array vs. work file" bits since that's not 
directly related to indexing.

Brad Schrag

> 
> Maybe one of you would be willing to pull all these good posts on 
> indexing into a wiki article?  It would be handy reference material 
> and could save someone from hours of pain can't get this stuff 
> from reading the system docs.


U.S. BANCORP made the following annotations
-
Electronic Privacy Notice. This e-mail, and any attachments, contains 
information that is, or may be, covered by electronic communications privacy 
laws, and is also confidential and proprietary in nature. If you are not the 
intended recipient, please be advised that you are legally prohibited from 
retaining, using, copying, distributing, or otherwise disclosing this 
information in any manner. Instead, please reply to the sender that you have 
received this communication in error, and then immediately delete it. Thank you 
in advance for your cooperation.



-

___
U2-Users mailing list
U2-Users@listserver.u2ug.org
http://listserver.u2ug.org/mailman/listinfo/u2-users

Re: [U2] General guidelines on indexing

[jpb-u2ug]

It's been a long time since I've been in hardware so I may be all off on
this, but, this could be possible because after it sends the WRITESEQ it's
basically up to the disk subsystem to actually do the writing. The written
sequential record get's put in the queue and the subsystem sends back a
signal that it's been done and to continue. However that doesn't necessarily
mean it has completed the write. By the time the process is done the
sequential file may not have completed writing all of the records in the
queue but the process will think it has. I know I have noticed this happen
when writing to a networked file system and it makes the file unusable until
the writing has completed.

Jerry Banker

-Original Message-
From: u2-users-boun...@listserver.u2ug.org
[mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Marco Manyevere
Sent: Wednesday, July 08, 2009 10:44 AM
To: U2 Users List
Subject: Re: [U2] General guidelines on indexing


In one test I did a couple of months back, I found that appending IDs to the
end of a dynamic array perfomed _much_ _much_ slower than a WRITESEQ to the
end of a disk file and the dynamic array wasnt even a 100 000 records long.
We were able to reduce the time required to produce a report from over 30
minutes to less than 2 minutes by removing the dynamic array operations and
replacing them with WRITESEQs. We tried all the different syntaxes of
appending to the array with no noticeable difference in poor performance
once the array got large.



- Original Message 
From: "Baakkonen, Rodney A (Rod) 46K" 
To: U2 Users List 
Sent: Wednesday, 8 July, 2009 17:15:57
Subject: Re: [U2] General guidelines on indexing

In theory, I would have to agree with you.  Who knows how all this stuff
really works under the hood. You have Unidata Shared memory management
and shared basic code server. We also have a huge SAN with a lots of
cache. Measuring performance and impacting it is different these days.
There are so many layers  and events, finding the bottleneck can be
tricky.

All I can tell you is that we did extensive testing and we would not be
doing what we are unless it worked. Maybe on you site, on your server
and your database, these techniques don't enhance performance. I'm
telling you what we found, your mileage may vary.

We have been told to use concatenate with a @AM to build our arrays, as
Unidata has a pointer to the end of a variable. We have been told to use
REMOVE when pulling data out of the array. We know that. We still had
performance gains that went from days to hours when we changed to a well
sized work file.

There are many reasons why we use the work file. Typically this is used
in the selection part of the process. The sorting and reporting is
after. Sometimes the work file contains more than keys ( vs. having
multiple dynamic arrays). Sometimes multiple reports are generated from
the same work file.  

-Original Message-
From: u2-users-boun...@listserver.u2ug.org
[mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Edward Brown
Sent: Wednesday, July 08, 2009 6:47 AM
To: U2 Users List
Subject: Re: [U2] General guidelines on indexing

I don't agree. Disk access is inherently slower than RAM access.
Therefore a process that makes efficient use of RAM will be faster than
an equivalent algorithm making efficient use of disk.

In your case, it's just a matter of scale:

50 million records at (lets say) 14 bytes per ID plus the multivalue
marker needed to build up the dynamic array.

15 * 50,000,000 = 750,000,000 bytes.

That's 732,422KB,  715MB

If your process is running on a modern server then this kind of op
becomes practical.

Assumptions:
- that the dynamic array isn't using Unicode. If it is then memory
reqirements double.
- That you select every record - normally (presumably) it would be just
a fraction?


In fact isn't all of this theoretical? Using the index select / readfwd
/ own tests method, there's no need to build workfiles or dynamic arrays
at all - simply do the tests as each record is retrieved with readfwd
and then create the report / do the processing all within the same loop?

Ed



-Original Message-
From: u2-users-boun...@listserver.u2ug.org
[mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Baakkonen,
Rodney A (Rod) 46K
Sent: 08 July 2009 12:30
To: U2 Users List
Subject: Re: [U2] General guidelines on indexing


When you have a  file with 50 million records, it does not matter how
you build the or parse the dynamic array. A well sized work file will
run circles around the dynamic array. 



-Original Message-
From: u2-users-boun...@listserver.u2ug.org
[mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Edward Brown
Sent: Wednesday, July 08, 2009 6:12 AM
To: U2 Users List
Subject: Re: [U2] General guidelines on indexing

> After indexing, we made a lot more use of the SETINDEX and READFWD
logic

Re: [U2] General guidelines on indexing

[Edward Brown]

Marco Manyevere wrote:
> In one test I did a couple of months back, I found that appending IDs
to the end of a dynamic array perfomed _much_ _much_ slower than a
WRITESEQ to the end of a disk file and the dynamic array wasnt even a
100 000 records long. We were able to reduce the time required to
produce a report from over 30 minutes to less than 2 minutes by removing
the dynamic array operations and replacing them with WRITESEQs. We tried
all the different syntaxes of appending to the array with no noticeable
difference in poor performance once the array got large.


Quick test on my development PC, on unidata 7.1

Source:

BIG.REC = ""
FOR ZEROS = 1 TO 10
  RUN.TO = 1:STR("0",ZEROS)
  BIG.REC = ""
  START.TIME = SYSTEM(12)
  FOR A = 1 TO RUN.TO
BIG.REC<-1> = "test string"
  NEXT A
  END.TIME = SYSTEM(12)
  CRT RUN.TO:" iterations = ":END.TIME - START.TIME)"MR2,":" ms
":((END.TIME-START.TIME)/RUN.TO):" ms per iteration"
NEXT ZEROS


Output:

10 iterations = 1 ms  0.1 ms per iteration
100 iterations = 0 ms  0 ms per iteration 
1000 iterations = 0 ms  0 ms per iteration
1 iterations = 3 ms  0.0003 ms per iteration  
10 iterations = 34 ms  0.0003 ms per iteration
100 iterations = 363 ms  0.0004 ms per iteration  
1000 iterations = 3710 ms  0.0004 ms per iteration
Error when attaching shm (-1476392888, 0, 0), errno=8

The above does show that <-1> gives fast appending and that there isn't
much of a penalty as sizes increase. The only issue would be running out
of memory :-)

This seems fast to me - is it totally out of line with your tests?

Ed

---
Please remember to recycle wherever possible. 
Reduce, reuse, recycle, think do you need to print this e-mail?
---
This e-mail and any attachment(s), is confidential and may be legally 
privileged. It is intended solely for the addressee. If you are not the 
addressee, dissemination, copying or use of this e-mail or any of its content 
is prohibited and may be unlawful. If you are not the intended recipient please 
inform the sender immediately and destroy the e-mail, any attachment(s) and any 
copies. All liability for viruses is excluded to the fullest extent permitted 
by law. It is your responsibility to scan or otherwise check this email and any 
attachment(s). Unless otherwise stated (i) views expressed in this message are 
those of the individual sender (ii) no contract may be construed by this 
e-mail. Emails may be monitored and you are taken to consent to this 
monitoring.  

Civica Services Limited, Company No. 02374268; Civica UK Limited, Company No. 
01628868
Both companies are registered in England and Wales and each has its registered 
office at 2 Burston Road, Putney, London, SW15 6AR.
---

___
U2-Users mailing list
U2-Users@listserver.u2ug.org
http://listserver.u2ug.org/mailman/listinfo/u2-users

Re: [U2] General guidelines on indexing

[Robert Porter]

Sorry yes, I was speaking of UV. There the data for the index is stored in 
I_(filename) by default.
But inside the header of the actual file, there is a pointer to where the 
indexes live. 
So the indexes could live somewhere else by using the SET.INDEX command, if you 
wanted to
move them for performance reasons.  You can use SET.INDEX (filename) INFORM to 
see where 
the index resides.
 
So when you restore a file, both the real and the restored files have pointers 
to /accountpath/I_(filename)...
provided it wasn't moved. Either way they will both point to the same place. 
Using the restored file without 
SET.INDEX (filename) TO NULL  can (and often will) corrupt the index.  Once you 
do this once, you won't 
forget about it again.
 
Robert

>>> Steve Romanow  7/8/2009 10:43 AM >>>
Robert Porter wrote:
> Just a quick note on indexes... and kind of goes to the EMC question.
>  
> The location of the index is stored in header of the file itself!  This can 
> lead to a problem if you have a copy of the data somewhere else
> and try to use it. For example, we'll occasionally want to see how a 
> file/record looked from a few days ago. We'll restore it to
> an alternate path. The first thing you MUST DO before you touch that file is 
> SET.INDEX (filename) TO NULL   -- If you don't and you 
> start accessing it (iirc, even a read can make it go check the indexes) it 
> will start updating the live file's index with the restored files
> data.  Best part if you may not even know, until things (like SELECTs just 
> don't work in your production account.
>  
> Robert
>  
>  
>  
> Robert F. Porter, MCSE, CCNA, ZCE
> Sr. Programmer / Analyst
> Laboratory Information Services
> Ochsner Health System
> (504) 842 - 5185
>  
>  
>  
> This transmission (including any attachments) may contain confidential 
> information, privileged material (including material protected by the 
> solicitor-client or other applicable privileges), or constitute non-public 
> information. Any use of this information by anyone other than the intended 
> recipient is prohibited. If you have received this transmission in error, 
> please immediately reply to the sender and delete this information from your 
> system. Use, dissemination, distribution, or reproduction of this 
> transmission by unintended recipients is not authorized and may be unlawful.
> ___
> U2-Users mailing list
> U2-Users@listserver.u2ug.org 
> http://listserver.u2ug.org/mailman/listinfo/u2-users 
>   
Note that this is for Universe only, correct Robert?  In Unidata, the 
index is in the present working dir as X_{filename}
___
U2-Users mailing list
U2-Users@listserver.u2ug.org 
http://listserver.u2ug.org/mailman/listinfo/u2-users
___
U2-Users mailing list
U2-Users@listserver.u2ug.org
http://listserver.u2ug.org/mailman/listinfo/u2-users

Re: [U2] General guidelines on indexing

[Marco Manyevere]

In one test I did a couple of months back, I found that appending IDs to the 
end of a dynamic array perfomed _much_ _much_ slower than a WRITESEQ to the end 
of a disk file and the dynamic array wasnt even a 100 000 records long. We were 
able to reduce the time required to produce a report from over 30 minutes to 
less than 2 minutes by removing the dynamic array operations and replacing them 
with WRITESEQs. We tried all the different syntaxes of appending to the array 
with no noticeable difference in poor performance once the array got large.



- Original Message 
From: "Baakkonen, Rodney A (Rod) 46K" 
To: U2 Users List 
Sent: Wednesday, 8 July, 2009 17:15:57
Subject: Re: [U2] General guidelines on indexing

In theory, I would have to agree with you.  Who knows how all this stuff
really works under the hood. You have Unidata Shared memory management
and shared basic code server. We also have a huge SAN with a lots of
cache. Measuring performance and impacting it is different these days.
There are so many layers  and events, finding the bottleneck can be
tricky.

All I can tell you is that we did extensive testing and we would not be
doing what we are unless it worked. Maybe on you site, on your server
and your database, these techniques don't enhance performance. I'm
telling you what we found, your mileage may vary.

We have been told to use concatenate with a @AM to build our arrays, as
Unidata has a pointer to the end of a variable. We have been told to use
REMOVE when pulling data out of the array. We know that. We still had
performance gains that went from days to hours when we changed to a well
sized work file.

There are many reasons why we use the work file. Typically this is used
in the selection part of the process. The sorting and reporting is
after. Sometimes the work file contains more than keys ( vs. having
multiple dynamic arrays). Sometimes multiple reports are generated from
the same work file.  

-Original Message-
From: u2-users-boun...@listserver.u2ug.org
[mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Edward Brown
Sent: Wednesday, July 08, 2009 6:47 AM
To: U2 Users List
Subject: Re: [U2] General guidelines on indexing

I don't agree. Disk access is inherently slower than RAM access.
Therefore a process that makes efficient use of RAM will be faster than
an equivalent algorithm making efficient use of disk.

In your case, it's just a matter of scale:

50 million records at (lets say) 14 bytes per ID plus the multivalue
marker needed to build up the dynamic array.

15 * 50,000,000 = 750,000,000 bytes.

That's 732,422KB,  715MB

If your process is running on a modern server then this kind of op
becomes practical.

Assumptions:
- that the dynamic array isn't using Unicode. If it is then memory
reqirements double.
- That you select every record - normally (presumably) it would be just
a fraction?


In fact isn't all of this theoretical? Using the index select / readfwd
/ own tests method, there's no need to build workfiles or dynamic arrays
at all - simply do the tests as each record is retrieved with readfwd
and then create the report / do the processing all within the same loop?

Ed



-Original Message-
From: u2-users-boun...@listserver.u2ug.org
[mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Baakkonen,
Rodney A (Rod) 46K
Sent: 08 July 2009 12:30
To: U2 Users List
Subject: Re: [U2] General guidelines on indexing


When you have a  file with 50 million records, it does not matter how
you build the or parse the dynamic array. A well sized work file will
run circles around the dynamic array. 



-Original Message-
From: u2-users-boun...@listserver.u2ug.org
[mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Edward Brown
Sent: Wednesday, July 08, 2009 6:12 AM
To: U2 Users List
Subject: Re: [U2] General guidelines on indexing

> After indexing, we made a lot more use of the SETINDEX and READFWD
logic
in our programs. 

I find this curious / disappointing - is it really the case that unidata
can't take the mix of indexed / unindexed dictionary items and do just
as efficient a job as the code you're writing?

Also, the performance of dynamic arrays need not be as much an issue as
you've found. If they're built up with -1 rather than a counter then the
speed penalty of adding items to a very large list is much the same as a
tiny one. Then, when you loop through them for further processing use
the REMOVE (or FORMLIST) command rather than a counter.

The only real issue with dynamic arrays is if the machine does not have
the physical memory available to hold the variable.

Ed


---
Please remember to recycle wherever possible. Reduce, reuse, recycle,
think do you need to print this e-mail?

Re: [U2] General guidelines on indexing

[Steve Romanow]

Robert Porter wrote:

Just a quick note on indexes... and kind of goes to the EMC question.
 
The location of the index is stored in header of the file itself!  This can lead to a problem if you have a copy of the data somewhere else

and try to use it. For example, we'll occasionally want to see how a 
file/record looked from a few days ago. We'll restore it to
an alternate path. The first thing you MUST DO before you touch that file is SET.INDEX (filename) TO NULL   -- If you don't and you 
start accessing it (iirc, even a read can make it go check the indexes) it will start updating the live file's index with the restored files

data.  Best part if you may not even know, until things (like SELECTs just 
don't work in your production account.
 
Robert
 
 
 
Robert F. Porter, MCSE, CCNA, ZCE

Sr. Programmer / Analyst
Laboratory Information Services
Ochsner Health System
(504) 842 - 5185
 
 
 
This transmission (including any attachments) may contain confidential information, privileged material (including material protected by the solicitor-client or other applicable privileges), or constitute non-public information. Any use of this information by anyone other than the intended recipient is prohibited. If you have received this transmission in error, please immediately reply to the sender and delete this information from your system. Use, dissemination, distribution, or reproduction of this transmission by unintended recipients is not authorized and may be unlawful.

___
U2-Users mailing list
U2-Users@listserver.u2ug.org
http://listserver.u2ug.org/mailman/listinfo/u2-users
  
Note that this is for Universe only, correct Robert?  In Unidata, the 
index is in the present working dir as X_{filename}

___
U2-Users mailing list
U2-Users@listserver.u2ug.org
http://listserver.u2ug.org/mailman/listinfo/u2-users

Re: [U2] General guidelines on indexing

[Baakkonen, Rodney A (Rod) 46K]

In theory, I would have to agree with you.  Who knows how all this stuff
really works under the hood. You have Unidata Shared memory management
and shared basic code server. We also have a huge SAN with a lots of
cache. Measuring performance and impacting it is different these days.
There are so many layers  and events, finding the bottleneck can be
tricky.

 All I can tell you is that we did extensive testing and we would not be
doing what we are unless it worked. Maybe on you site, on your server
and your database, these techniques don't enhance performance. I'm
telling you what we found, your mileage may vary.

We have been told to use concatenate with a @AM to build our arrays, as
Unidata has a pointer to the end of a variable. We have been told to use
REMOVE when pulling data out of the array. We know that. We still had
performance gains that went from days to hours when we changed to a well
sized work file.

There are many reasons why we use the work file. Typically this is used
in the selection part of the process. The sorting and reporting is
after. Sometimes the work file contains more than keys ( vs. having
multiple dynamic arrays). Sometimes multiple reports are generated from
the same work file.  

-Original Message-
From: u2-users-boun...@listserver.u2ug.org
[mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Edward Brown
Sent: Wednesday, July 08, 2009 6:47 AM
To: U2 Users List
Subject: Re: [U2] General guidelines on indexing

I don't agree. Disk access is inherently slower than RAM access.
Therefore a process that makes efficient use of RAM will be faster than
an equivalent algorithm making efficient use of disk.

In your case, it's just a matter of scale:

50 million records at (lets say) 14 bytes per ID plus the multivalue
marker needed to build up the dynamic array.

15 * 50,000,000 = 750,000,000 bytes.

That's 732,422KB,  715MB

If your process is running on a modern server then this kind of op
becomes practical.

Assumptions:
 - that the dynamic array isn't using Unicode. If it is then memory
reqirements double.
 - That you select every record - normally (presumably) it would be just
a fraction?


In fact isn't all of this theoretical? Using the index select / readfwd
/ own tests method, there's no need to build workfiles or dynamic arrays
at all - simply do the tests as each record is retrieved with readfwd
and then create the report / do the processing all within the same loop?

Ed



-Original Message-
From: u2-users-boun...@listserver.u2ug.org
[mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Baakkonen,
Rodney A (Rod) 46K
Sent: 08 July 2009 12:30
To: U2 Users List
Subject: Re: [U2] General guidelines on indexing


 When you have a  file with 50 million records, it does not matter how
you build the or parse the dynamic array. A well sized work file will
run circles around the dynamic array. 



-Original Message-
From: u2-users-boun...@listserver.u2ug.org
[mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Edward Brown
Sent: Wednesday, July 08, 2009 6:12 AM
To: U2 Users List
Subject: Re: [U2] General guidelines on indexing

> After indexing, we made a lot more use of the SETINDEX and READFWD
logic
in our programs. 

I find this curious / disappointing - is it really the case that unidata
can't take the mix of indexed / unindexed dictionary items and do just
as efficient a job as the code you're writing?

Also, the performance of dynamic arrays need not be as much an issue as
you've found. If they're built up with -1 rather than a counter then the
speed penalty of adding items to a very large list is much the same as a
tiny one. Then, when you loop through them for further processing use
the REMOVE (or FORMLIST) command rather than a counter.

The only real issue with dynamic arrays is if the machine does not have
the physical memory available to hold the variable.

Ed


---
Please remember to recycle wherever possible. Reduce, reuse, recycle,
think do you need to print this e-mail?

---
This e-mail and any attachment(s), is confidential and may be legally
privileged. It is intended solely for the addressee. If you are not the
addressee, dissemination, copying or use of this e-mail or any of its
content is prohibited and may be unlawful. If you are not the intended
recipient please inform the sender immediately and destroy the e-mail,
any attachment(s) and any copies. All liability for viruses is excluded
to the fullest extent permitted by law. It is your responsibility to
scan or otherwise check this email and any attachment(s). Unless
otherwise stated (i) views expressed in this message are those of the
individual sender (ii) no contract may be construed by this e-mail.
Emails may be monitored and

Re: [U2] General guidelines on indexing

[Baker Hughes]

Maybe one of you would be willing to pull all these good posts on indexing into 
a wiki article?  It would be handy reference material and could save someone 
from hours of pain can't get this stuff from reading the system docs.

http://212.241.202.162/U2UGWiki/moin.cgi/ForDevelopers

-Baker


This communication, its contents and any file attachments transmitted with it 
are intended solely for the addressee(s) and may contain confidential 
proprietary information.
Access by any other party without the express written permission of the sender 
is STRICTLY PROHIBITED.
If you have received this communication in error you may not copy, distribute 
or use the contents, attachments or information in any way.  Please destroy it 
and contact the sender.
___
U2-Users mailing list
U2-Users@listserver.u2ug.org
http://listserver.u2ug.org/mailman/listinfo/u2-users

Re: [U2] General guidelines on indexing

[Robert Porter]

Just a quick note on indexes... and kind of goes to the EMC question.
 
The location of the index is stored in header of the file itself!  This can 
lead to a problem if you have a copy of the data somewhere else
and try to use it. For example, we'll occasionally want to see how a 
file/record looked from a few days ago. We'll restore it to
an alternate path. The first thing you MUST DO before you touch that file is 
SET.INDEX (filename) TO NULL   -- If you don't and you 
start accessing it (iirc, even a read can make it go check the indexes) it will 
start updating the live file's index with the restored files
data.  Best part if you may not even know, until things (like SELECTs just 
don't work in your production account.
 
Robert
 
 
 
Robert F. Porter, MCSE, CCNA, ZCE
Sr. Programmer / Analyst
Laboratory Information Services
Ochsner Health System
(504) 842 - 5185
 
 
 
This transmission (including any attachments) may contain confidential 
information, privileged material (including material protected by the 
solicitor-client or other applicable privileges), or constitute non-public 
information. Any use of this information by anyone other than the intended 
recipient is prohibited. If you have received this transmission in error, 
please immediately reply to the sender and delete this information from your 
system. Use, dissemination, distribution, or reproduction of this transmission 
by unintended recipients is not authorized and may be unlawful.
___
U2-Users mailing list
U2-Users@listserver.u2ug.org
http://listserver.u2ug.org/mailman/listinfo/u2-users

Re: [U2] General guidelines on indexing

[Steve Romanow]

Edward Brown wrote:

I don't agree. Disk access is inherently slower than RAM access.
Therefore a process that makes efficient use of RAM will be faster than
an equivalent algorithm making efficient use of disk.

In your case, it's just a matter of scale:

50 million records at (lets say) 14 bytes per ID plus the multivalue
marker needed to build up the dynamic array.

15 * 50,000,000 = 750,000,000 bytes.

That's 732,422KB,  715MB

If your process is running on a modern server then this kind of op
becomes practical.

Assumptions:
 - that the dynamic array isn't using Unicode. If it is then memory
reqirements double.
 - That you select every record - normally (presumably) it would be just
a fraction?


In fact isn't all of this theoretical? Using the index select / readfwd
/ own tests method, there's no need to build workfiles or dynamic arrays
at all - simply do the tests as each record is retrieved with readfwd
and then create the report / do the processing all within the same loop?

Ed

  
The only real reason I can see for _really_ needing the extra data 
structure or workfile is if you have post sorting, maybe on a calculated 
value, say sorting on a totaled value.  You need a pass through the data 
first to get that value.

___
U2-Users mailing list
U2-Users@listserver.u2ug.org
http://listserver.u2ug.org/mailman/listinfo/u2-users

Re: [U2] General guidelines on indexing

[Edward Brown]

I don't agree. Disk access is inherently slower than RAM access.
Therefore a process that makes efficient use of RAM will be faster than
an equivalent algorithm making efficient use of disk.

In your case, it's just a matter of scale:

50 million records at (lets say) 14 bytes per ID plus the multivalue
marker needed to build up the dynamic array.

15 * 50,000,000 = 750,000,000 bytes.

That's 732,422KB,  715MB

If your process is running on a modern server then this kind of op
becomes practical.

Assumptions:
 - that the dynamic array isn't using Unicode. If it is then memory
reqirements double.
 - That you select every record - normally (presumably) it would be just
a fraction?


In fact isn't all of this theoretical? Using the index select / readfwd
/ own tests method, there's no need to build workfiles or dynamic arrays
at all - simply do the tests as each record is retrieved with readfwd
and then create the report / do the processing all within the same loop?

Ed



-Original Message-
From: u2-users-boun...@listserver.u2ug.org
[mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Baakkonen,
Rodney A (Rod) 46K
Sent: 08 July 2009 12:30
To: U2 Users List
Subject: Re: [U2] General guidelines on indexing


 When you have a  file with 50 million records, it does not matter how
you build the or parse the dynamic array. A well sized work file will
run circles around the dynamic array. 



-Original Message-
From: u2-users-boun...@listserver.u2ug.org
[mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Edward Brown
Sent: Wednesday, July 08, 2009 6:12 AM
To: U2 Users List
Subject: Re: [U2] General guidelines on indexing

> After indexing, we made a lot more use of the SETINDEX and READFWD
logic
in our programs. 

I find this curious / disappointing - is it really the case that unidata
can't take the mix of indexed / unindexed dictionary items and do just
as efficient a job as the code you're writing?

Also, the performance of dynamic arrays need not be as much an issue as
you've found. If they're built up with -1 rather than a counter then the
speed penalty of adding items to a very large list is much the same as a
tiny one. Then, when you loop through them for further processing use
the REMOVE (or FORMLIST) command rather than a counter.

The only real issue with dynamic arrays is if the machine does not have
the physical memory available to hold the variable.

Ed


---
Please remember to recycle wherever possible. Reduce, reuse, recycle,
think do you need to print this e-mail?

---
This e-mail and any attachment(s), is confidential and may be legally
privileged. It is intended solely for the addressee. If you are not the
addressee, dissemination, copying or use of this e-mail or any of its
content is prohibited and may be unlawful. If you are not the intended
recipient please inform the sender immediately and destroy the e-mail,
any attachment(s) and any copies. All liability for viruses is excluded
to the fullest extent permitted by law. It is your responsibility to
scan or otherwise check this email and any attachment(s). Unless
otherwise stated (i) views expressed in this message are those of the
individual sender (ii) no contract may be construed by this e-mail.
Emails may be monitored and you are taken to consent to this monitoring.


Civica Services Limited, Company No. 02374268; Civica UK Limited,
Company No. 01628868 Both companies are registered in England and Wales
and each has its registered office at 2 Burston Road, Putney, London,
SW15 6AR.

---

___
U2-Users mailing list
U2-Users@listserver.u2ug.org
http://listserver.u2ug.org/mailman/listinfo/u2-users





___
U2-Users mailing list
U2-Users@listserver.u2ug.org
http://listserver.u2ug.org/mailman/listinfo/u2-users
___
U2-Users mailing list
U2-Users@listserver.u2ug.org
http://listserver.u2ug.org/mailman/listinfo/u2-users

Re: [U2] General guidelines on indexing

[Baakkonen, Rodney A (Rod) 46K]

 When you have a  file with 50 million records, it does not matter how
you build the or parse the dynamic array. A well sized work file will
run circles around the dynamic array. 



-Original Message-
From: u2-users-boun...@listserver.u2ug.org
[mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Edward Brown
Sent: Wednesday, July 08, 2009 6:12 AM
To: U2 Users List
Subject: Re: [U2] General guidelines on indexing

> After indexing, we made a lot more use of the SETINDEX and READFWD
logic
in our programs. 

I find this curious / disappointing - is it really the case that unidata
can't take the mix of indexed / unindexed dictionary items and do just
as efficient a job as the code you're writing?

Also, the performance of dynamic arrays need not be as much an issue as
you've found. If they're built up with -1 rather than a counter then the
speed penalty of adding items to a very large list is much the same as a
tiny one. Then, when you loop through them for further processing use
the REMOVE (or FORMLIST) command rather than a counter.

The only real issue with dynamic arrays is if the machine does not have
the physical memory available to hold the variable.

Ed


---
Please remember to recycle wherever possible. Reduce, reuse, recycle,
think do you need to print this e-mail?

---
This e-mail and any attachment(s), is confidential and may be legally
privileged. It is intended solely for the addressee. If you are not the
addressee, dissemination, copying or use of this e-mail or any of its
content is prohibited and may be unlawful. If you are not the intended
recipient please inform the sender immediately and destroy the e-mail,
any attachment(s) and any copies. All liability for viruses is excluded
to the fullest extent permitted by law. It is your responsibility to
scan or otherwise check this email and any attachment(s). Unless
otherwise stated (i) views expressed in this message are those of the
individual sender (ii) no contract may be construed by this e-mail.
Emails may be monitored and you are taken to consent to this monitoring.


Civica Services Limited, Company No. 02374268; Civica UK Limited,
Company No. 01628868 Both companies are registered in England and Wales
and each has its registered office at 2 Burston Road, Putney, London,
SW15 6AR.

---

___
U2-Users mailing list
U2-Users@listserver.u2ug.org
http://listserver.u2ug.org/mailman/listinfo/u2-users





___
U2-Users mailing list
U2-Users@listserver.u2ug.org
http://listserver.u2ug.org/mailman/listinfo/u2-users

Re: [U2] General guidelines on indexing

[Edward Brown]

> After indexing, we made a lot more use of the SETINDEX and READFWD
logic
in our programs. 

I find this curious / disappointing - is it really the case that unidata
can't take the mix of indexed / unindexed dictionary items and do just
as efficient a job as the code you're writing?

Also, the performance of dynamic arrays need not be as much an issue as
you've found. If they're built up with -1 rather than a counter then the
speed penalty of adding items to a very large list is much the same as a
tiny one. Then, when you loop through them for further processing use
the REMOVE (or FORMLIST) command rather than a counter.

The only real issue with dynamic arrays is if the machine does not have
the physical memory available to hold the variable.

Ed

---
Please remember to recycle wherever possible. 
Reduce, reuse, recycle, think do you need to print this e-mail?
---
This e-mail and any attachment(s), is confidential and may be legally 
privileged. It is intended solely for the addressee. If you are not the 
addressee, dissemination, copying or use of this e-mail or any of its content 
is prohibited and may be unlawful. If you are not the intended recipient please 
inform the sender immediately and destroy the e-mail, any attachment(s) and any 
copies. All liability for viruses is excluded to the fullest extent permitted 
by law. It is your responsibility to scan or otherwise check this email and any 
attachment(s). Unless otherwise stated (i) views expressed in this message are 
those of the individual sender (ii) no contract may be construed by this 
e-mail. Emails may be monitored and you are taken to consent to this 
monitoring.  

Civica Services Limited, Company No. 02374268; Civica UK Limited, Company No. 
01628868
Both companies are registered in England and Wales and each has its registered 
office at 2 Burston Road, Putney, London, SW15 6AR.
---

___
U2-Users mailing list
U2-Users@listserver.u2ug.org
http://listserver.u2ug.org/mailman/listinfo/u2-users

Re: [U2] General guidelines on indexing

[Baakkonen, Rodney A (Rod) 46K]

 We are on Unidata 7.1 on Sun Solaris

We have used indexing extensively in our database. We have one file with
10 indexes. The file is currently using around 80 gig. (35 dat, 9 idx
and 1 Over parts files). I am not sure how to weigh the update
performance difference these indexes make. But we feel whatever is lost
updating is more than gained reporting wise.

After indexing, we made a lot more use of the SETINDEX and READFWD logic
in our programs. This technique is used when there are lots of
dictionary items used in evaluating what to select. Rather than do
'SELECT FILE WITH START.DATE > "01/01/09" AND WITH ORG.STATE = "MN" AND
WITH DOLLAR.AMT > "2500.00" AND WITH REGION > "1" AND WITH QUALIFIER <
"10", we would use the index on START.DATE to position us in the index
at "01/01/09". READFWD then reads records into your program in
START.DATE order. The rest of the conditions can then be evaluated in
Unibasic to see if records should be selected. Records that meet the
conditions are written to a well sized work file. DON'T USE A DYNAMIC
ARRAY to build a list on a large file. A well size work file will run
circles around a huge dynamic array. We have had processes that built
arrays using READFWD to hold the keys take days to run. Switching to a
well sized work file to hold the keys cut the process to hours.


I have used BUILD.INDEX ONLINE on smaller files. IT takes a long time to
build an index with people in it. But it does seem to work.

The only data consistency problems we have had were due to crashes and
indexes were missing data. There is a guide index verb to try and catch
these problems. You can also find them using a SELECT with NO.INDEX to
compare results to see if a problem is there. We have very little
problem with this.

We don't have Sub Values indexed. But we have values indexed all over.
It seems to work fine if your are aware of using EVERY, EACH and a
couple other Multi-value verbs.

Unidata does not have that header problem that Universe has. If you move
the file around, Unidata's indexes still work fine.

Indexes can get you a lot of performance gain. It will cost you some
disk space on large files. And your programmers need to understand when
to do a EXECUTE SELECT and when to use SETINDEX/RFWD

If you have other questions let me know. - Rod
-Original Message-
From: u2-users-boun...@listserver.u2ug.org
[mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of
bradley.sch...@usbank.com
Sent: Tuesday, July 07, 2009 1:57 PM
To: U2 Users List
Subject: [U2] General guidelines on indexing

Our primary application hasn't needed the performance gains offered by
indexing, but our database has grown large and complex enough that we're
looking at it seriously. Having only dabbled with indexing in test
environments, I've got a few general and best practice questions. I've
seen some comments in the archives on some of the pitfalls, e.g. the
dangers of indexing remote files and indexing on correlatives with
subroutines. But posts I found are at least a year old. I'm guessing
there may be updated information out there. Feel free to point out that
everything I'm asking is in the archives and send me there.

Some questions:

* Are there guidelines for how many indices is too many for one file
(assuming disk space isn't an issue)?
* Does BUILD.INDEX with the ONLINE parameter work as advertised? Can it
really be run while the file is being updated?
* How about data consistency? I seem to remember there being concerns in
earlier days of an index not always being updated correctly. 
* How about indexing multi/subvalued fields? I don't know that we'd want
to, but is it advisable? Valuable?
* We use EMC to clone our production account so we can run nightly
reports off-line. The account is renamed in this process. Might that
cause any index issues? The clone is read-only, so there are no updates,
just queries.
* I've seen mention of index corruption. Is it obvious when an index is
corrupt or can it be subtle? If subtle, are there ways to detect issues
before our users do?
* Performance is what we're after, but are there benefits to indexing
other than performance? 


AIX 5.3
UniData 7.1


All thoughts, comments, observations are most appreciated.

Brad Schrag

U.S. BANCORP made the following annotations
-
Electronic Privacy Notice. This e-mail, and any attachments, contains
information that is, or may be, covered by electronic communications
privacy laws, and is also confidential and proprietary in nature. If you
are not the intended recipient, please be advised that you are legally
prohibited from retaining, using, copying, distributing, or otherwise
disclosing this information in any manner. Instead, pleas

Re: [U2] General guidelines on indexing

[Steve Romanow]

Be careful not to index on a foreign key, i.e. do not index a field that 
is a tlookup or trans to another file.


An example is if custno is on the order header,and you xlate it to 
order.line,  do not index order.line by custno.  It will not update 
properly.


U2 indexing is such a breath of fresh air over the D3 indexing I was 
using for a few years recently.


bradley.sch...@usbank.com wrote:
Our primary application hasn't needed the performance gains offered by 
indexing, but our database has grown large and complex enough that we're 
looking at it seriously. Having only dabbled with indexing in test 
environments, I've got a few general and best practice questions. I've 
seen some comments in the archives on some of the pitfalls, e.g. the 
dangers of indexing remote files and indexing on correlatives with 
subroutines. But posts I found are at least a year old. I'm guessing there 
may be updated information out there. Feel free to point out that 
everything I'm asking is in the archives and send me there.


Some questions:

* Are there guidelines for how many indices is too many for one file 
(assuming disk space isn't an issue)?
* Does BUILD.INDEX with the ONLINE parameter work as advertised? Can it 
really be run while the file is being updated?
* How about data consistency? I seem to remember there being concerns in 
earlier days of an index not always being updated correctly. 
* How about indexing multi/subvalued fields? I don't know that we'd want 
to, but is it advisable? Valuable?
* We use EMC to clone our production account so we can run nightly reports 
off-line. The account is renamed in this process. Might that cause any 
index issues? The clone is read-only, so there are no updates, just 
queries.
* I've seen mention of index corruption. Is it obvious when an index is 
corrupt or can it be subtle? If subtle, are there ways to detect issues 
before our users do?
* Performance is what we're after, but are there benefits to indexing 
other than performance? 



AIX 5.3
UniData 7.1


All thoughts, comments, observations are most appreciated.

Brad Schrag

U.S. BANCORP made the following annotations
-
Electronic Privacy Notice. This e-mail, and any attachments, contains 
information that is, or may be, covered by electronic communications privacy 
laws, and is also confidential and proprietary in nature. If you are not the 
intended recipient, please be advised that you are legally prohibited from 
retaining, using, copying, distributing, or otherwise disclosing this 
information in any manner. Instead, please reply to the sender that you have 
received this communication in error, and then immediately delete it. Thank you 
in advance for your cooperation.



-

___
U2-Users mailing list
U2-Users@listserver.u2ug.org
http://listserver.u2ug.org/mailman/listinfo/u2-users
  

___
U2-Users mailing list
U2-Users@listserver.u2ug.org
http://listserver.u2ug.org/mailman/listinfo/u2-users

Re: [U2] General guidelines on indexing

[asvin . dattani]

Hi,

A general point on indexes. You should be careful in choosing the data 
that you index on. Indexing on a data field where a substantial number of 
records in the file have the same value in that field can cause 
performance problems.

An extreme example is a field that can contain Y or N. But it can happen 
with other fields, for example a field that contains state codes, or even 
a field that contains customer codes, but many of the record are for the 
same customer.

Look at the distribution of the records for each value in the field that 
you are indexing on. Also consider using the NONULLs option when you build 
the index.

As far as your EMC question is concerned, if you change the pathname of 
the file or the index files then you will get problems. You will need to 
use the SET.INDEX verb to get around that.

cheers,

asvin




bradley.sch...@usbank.com 
Sent by: u2-users-boun...@listserver.u2ug.org
Jul 07 2009 19:57

Mail Size: 8099

Please respond to
U2 Users List 


To
"U2 Users List" 
cc

Subject
[U2] General guidelines on indexing

  Entity
   Investment Banking Europe - IBEU



Our primary application hasn't needed the performance gains offered by 
indexing, but our database has grown large and complex enough that we're 
looking at it seriously. Having only dabbled with indexing in test 
environments, I've got a few general and best practice questions. I've 
seen some comments in the archives on some of the pitfalls, e.g. the 
dangers of indexing remote files and indexing on correlatives with 
subroutines. But posts I found are at least a year old. I'm guessing there 

may be updated information out there. Feel free to point out that 
everything I'm asking is in the archives and send me there.

Some questions:

* Are there guidelines for how many indices is too many for one file 
(assuming disk space isn't an issue)?
* Does BUILD.INDEX with the ONLINE parameter work as advertised? Can it 
really be run while the file is being updated?
* How about data consistency? I seem to remember there being concerns in 
earlier days of an index not always being updated correctly. 
* How about indexing multi/subvalued fields? I don't know that we'd want 
to, but is it advisable? Valuable?
* We use EMC to clone our production account so we can run nightly reports 

off-line. The account is renamed in this process. Might that cause any 
index issues? The clone is read-only, so there are no updates, just 
queries.
* I've seen mention of index corruption. Is it obvious when an index is 
corrupt or can it be subtle? If subtle, are there ways to detect issues 
before our users do?
* Performance is what we're after, but are there benefits to indexing 
other than performance? 


AIX 5.3
UniData 7.1


All thoughts, comments, observations are most appreciated.

Brad Schrag

U.S. BANCORP made the following annotations
-
Electronic Privacy Notice. This e-mail, and any attachments, contains 
information that is, or may be, covered by electronic communications 
privacy laws, and is also confidential and proprietary in nature. If you 
are not the intended recipient, please be advised that you are legally 
prohibited from retaining, using, copying, distributing, or otherwise 
disclosing this information in any manner. Instead, please reply to the 
sender that you have received this communication in error, and then 
immediately delete it. Thank you in advance for your cooperation.



-

___
U2-Users mailing list
U2-Users@listserver.u2ug.org
http://listserver.u2ug.org/mailman/listinfo/u2-users




HSBC Bank plc may be solicited in the course of its placement efforts for 
a new issue, by investment clients of the firm for whom the Bank as a firm 
already provides other services. It may equally decide to allocate to its 
own proprietary book or with an associate of HSBC Group. This represents a 
potential conflict of interest. HSBC Bank plc has internal arrangements 
designed to ensure that the firm would give unbiased and full advice to 
the corporate finance client about the valuation and pricing of the 
offering as well as internal systems, controls and procedures to identify 
and manage conflicts of interest.

HSBC Bank plc
Registered Office: 8 Canada Square, London E14 5HQ, United Kingdom
Registered in England - Number 14259
Authorised and regulated by the Financial Services Authority.



-
SAVE PAPER - THINK BEFORE YOU PRINT!

This transmission has been issued by a member of the HSBC Group
"HSBC" for the information of the addressee only and should not be
reproduced and/or distributed to any other person

Re: [U2] General guidelines on indexing

[Edward Brown]

We've recently implemented indexing into our application as a
replacement for our own custom indexes, so I'm at least a little bit
qualified to answer some of your questions:


* Number of indexes - no guidelines afaik. The update process seems fast
enough - certainly, every write triggers an index update even if the
indexed attribute(s) hasn't changed. I would say the cost in update time
vs index benefit will be file-specific, depending on how much the file
is written to and how often it is searched.

* Never used build.index / online.

* Consistency - no problems yet.

* Indexing mv / svs - no problems here. We've used some fairly complex
dictionary items to build up the index keys. Basically it doesn't matter
how the index key is built up providing it comes from @RECORD - although
we've used remote files in one or two cases in the knowledge that they
are unlikely to change.

* ECM - never used it

* Corruption - not hit this one yet. But it's obviously a concern.

* Benefits - as I said above, the indexes are replacing an amount of
existing code, although I've had to write more code to fully integrate
the indexes with our search front end. However overall it's lead to a
simplification in the code base and faster / better search behaviour,
with various quirks in the original code gone.

We've also considered using indexes in a more general sense for selects
and there's an idea floating around to measure the speed of the selects
and the select commands as our system is used out in the wild; we can
then use this information to index those specific dictionary items that
would give the biggest improvements.

Hth

Ed


---
Please remember to recycle wherever possible. 
Reduce, reuse, recycle, think do you need to print this e-mail?
---
This e-mail and any attachment(s), is confidential and may be legally 
privileged. It is intended solely for the addressee. If you are not the 
addressee, dissemination, copying or use of this e-mail or any of its content 
is prohibited and may be unlawful. If you are not the intended recipient please 
inform the sender immediately and destroy the e-mail, any attachment(s) and any 
copies. All liability for viruses is excluded to the fullest extent permitted 
by law. It is your responsibility to scan or otherwise check this email and any 
attachment(s). Unless otherwise stated (i) views expressed in this message are 
those of the individual sender (ii) no contract may be construed by this 
e-mail. Emails may be monitored and you are taken to consent to this 
monitoring.  

Civica Services Limited, Company No. 02374268; Civica UK Limited, Company No. 
01628868
Both companies are registered in England and Wales and each has its registered 
office at 2 Burston Road, Putney, London, SW15 6AR.
---

___
U2-Users mailing list
U2-Users@listserver.u2ug.org
http://listserver.u2ug.org/mailman/listinfo/u2-users

[U2] General guidelines on indexing

[bradley . schrag]

Our primary application hasn't needed the performance gains offered by 
indexing, but our database has grown large and complex enough that we're 
looking at it seriously. Having only dabbled with indexing in test 
environments, I've got a few general and best practice questions. I've 
seen some comments in the archives on some of the pitfalls, e.g. the 
dangers of indexing remote files and indexing on correlatives with 
subroutines. But posts I found are at least a year old. I'm guessing there 
may be updated information out there. Feel free to point out that 
everything I'm asking is in the archives and send me there.

Some questions:

* Are there guidelines for how many indices is too many for one file 
(assuming disk space isn't an issue)?
* Does BUILD.INDEX with the ONLINE parameter work as advertised? Can it 
really be run while the file is being updated?
* How about data consistency? I seem to remember there being concerns in 
earlier days of an index not always being updated correctly. 
* How about indexing multi/subvalued fields? I don't know that we'd want 
to, but is it advisable? Valuable?
* We use EMC to clone our production account so we can run nightly reports 
off-line. The account is renamed in this process. Might that cause any 
index issues? The clone is read-only, so there are no updates, just 
queries.
* I've seen mention of index corruption. Is it obvious when an index is 
corrupt or can it be subtle? If subtle, are there ways to detect issues 
before our users do?
* Performance is what we're after, but are there benefits to indexing 
other than performance? 


AIX 5.3
UniData 7.1


All thoughts, comments, observations are most appreciated.

Brad Schrag

U.S. BANCORP made the following annotations
-
Electronic Privacy Notice. This e-mail, and any attachments, contains 
information that is, or may be, covered by electronic communications privacy 
laws, and is also confidential and proprietary in nature. If you are not the 
intended recipient, please be advised that you are legally prohibited from 
retaining, using, copying, distributing, or otherwise disclosing this 
information in any manner. Instead, please reply to the sender that you have 
received this communication in error, and then immediately delete it. Thank you 
in advance for your cooperation.



-

___
U2-Users mailing list
U2-Users@listserver.u2ug.org
http://listserver.u2ug.org/mailman/listinfo/u2-users