it take to
initially populate the file.
John
-Original Message-
From: u2-users-boun...@listserver.u2ug.org
[mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Charles Stevenson
Sent: Thursday, July 05, 2012 5:41 PM
To: U2 Users List
Subject: Re: [U2] RESIZE - dynamic files
Chris
On 05/07/12 23:58, Rick Nuckolls wrote:
Oops, I would of thought that if a file had, say 100,000 bytes, @ 70 percent
full, there would be 30,000 bytes empty or dead. Are you suggesting the
there would be 70,000 bytes of data and 42,000 bytes of dead space?
Do you mean 100,000 bytes of disk
you gain anything by reducing overflow?
Chris
To: u2-users@listserver.u2ug.org
From: wjhon...@aol.com
Date: Thu, 5 Jul 2012 20:12:21 -0400
Subject: Re: [U2] RESIZE - dynamic files
A BASIC SELECT cannot use criteria at all.
It is going to walk through every record in the file, in order
-users@listserver.u2ug.org
Sent: Fri, Jul 6, 2012 10:23 am
Subject: Re: [U2] RESIZE - dynamic files
o is there a performance increase in BASIC SELECTS by reducing overflow? Some
eople are saying to reduce disk space to speed up the BASIC SELECT
hile others say to reduce overflow.. I'm a bit
ignore the bs and use the defaults. :)
-Rick
-Original Message-
From: u2-users-boun...@listserver.u2ug.org
[mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Chris Austin
Sent: Friday, July 06, 2012 9:56 AM
To: u2-users@listserver.u2ug.org
Subject: Re: [U2] RESIZE - dynamic files
-
From: u2-users-boun...@listserver.u2ug.org
[mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Chris Austin
Sent: 07/06/2012 12:56 PM
To: u2-users@listserver.u2ug.org
Subject: Re: [U2] RESIZE - dynamic files
So is there a performance increase in BASIC SELECTS by reducing overflow?
Some
Subject: Re: [U2] RESIZE - dynamic files
Logically, the graphed solution to varying the split.load value with an
-axis=modulus, y-axis=time_to_select__read_the_whole_file is going to be
arabolic, having very slow performance at modulus=1 and modulus = # of records.
If you actually want to find
, I think
this is all starting to make sense.
Chris
From: ro...@stamina.com.au
To: u2-users@listserver.u2ug.org
Date: Wed, 4 Jul 2012 01:36:26 +
Subject: Re: [U2] RESIZE - dynamic files
I would suggest that then actual goal is to achieve maximum performance for
your system, so knowing
...@datacom.co.nz
To: u2-users@listserver.u2ug.org
Date: Wed, 4 Jul 2012 14:05:02 +1200
Subject: Re: [U2] RESIZE - dynamic files
Doug may have had a key bounce in his input
Let's do the math:
258687736 (Record Size)
192283300 (Key Size)
The key size is actually 19283300
] On Behalf Of Chris Austin
Sent: 05 July 2012 15:19
To: u2-users@listserver.u2ug.org
Subject: Re: [U2] RESIZE - dynamic files
I was able to drop from 30% overflow to 12% by making 2 changes:
1) changed the split from 80% to 70% (that alone reduce 10% overflow)
2) changed the MINIMUM.MODULUS
[mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Chris Austin
Sent: 05 July 2012 15:19
To: u2-users@listserver.u2ug.org
Subject: Re: [U2] RESIZE - dynamic files
I was able to drop from 30% overflow to 12% by making 2 changes:
1) changed the split from 80% to 70% (that alone
the performance I need to write a ton of records and then capture the
output and compare the output using timestamps.
Chris
From: r...@lynden.com
To: u2-users@listserver.u2ug.org
Date: Thu, 5 Jul 2012 09:22:02 -0700
Subject: Re: [U2] RESIZE - dynamic files
Chis,
I still am wondering what
, July 05, 2012 11:21 AM
To: u2-users@listserver.u2ug.org
Subject: Re: [U2] RESIZE - dynamic files
Rick,
You are correct, I should be using the smaller size (I just haven't changed it
yet). Based on the reading I have done you should
only use the larger group size when the average record size
...@lynden.com
To: 'U2 Users List' u2-users@listserver.u2ug.org
Sent: Thu, Jul 5, 2012 2:29 pm
Subject: Re: [U2] RESIZE - dynamic files
Chris,
For the type of use that you described earlier; BASIC selects and reads,
educing overflow will have negligible performance benefit, especially compared
o
Chris,
I can appreciate what you are doing as an academic exercise.
You seem happy how it looks at this moment, where, because you set
MINIMUM.MODULUS 118681, you ended up with a current load of 63%.
But think about it: as you add records, the load will reach 70%, per
SPLIT.LOAD 70, then
On 05/07/12 16:12, Martin Phillips wrote:
A file without overflow is not necessarily the best solution. Winding the
split load down to 70% means that at least 30% of the file
is dead space. The implication of this is that the file is larger and will
take more disk reads to process
On 05/07/12 14:49, Chris Austin wrote:
Disk space is not a factor, as we are a smaller shop and disk space comes
cheap. However, one thing I did notice is when I increased the modulus to a
very large
number which then increased my disk space to about 3-4x of my record data, my
SELECT
-
From: Rick Nuckolls r...@lynden.com
To: 'U2 Users List' u2-users@listserver.u2ug.org
Sent: Thu, Jul 5, 2012 2:29 pm
Subject: Re: [U2] RESIZE - dynamic files
Chris,
For the type of use that you described earlier; BASIC selects and reads,
educing overflow will have negligible performance benefit
: Thu, Jul 5, 2012 4:43 pm
Subject: Re: [U2] RESIZE - dynamic files
Most disks and disk systems cache huge amounts of information these days, and,
epending on 20 factors or so, one solution will be better than another for a
iven file.
For the wholesale, SELECT F WITH, The fewest disk records
...@listserver.u2ug.org] On Behalf Of Wjhonson
Sent: Thursday, July 05, 2012 5:12 PM
To: u2-users@listserver.u2ug.org
Subject: Re: [U2] RESIZE - dynamic files
A BASIC SELECT cannot use criteria at all.
It is going to walk through every record in the file, in order.
And that's the sticky wicket. That whole
.
Chris
To: u2-users@listserver.u2ug.org
From: wjhon...@aol.com
Date: Thu, 5 Jul 2012 20:12:21 -0400
Subject: Re: [U2] RESIZE - dynamic files
A BASIC SELECT cannot use criteria at all.
It is going to walk through every record in the file, in order.
And that's the sticky wicket. That whole
All the other groups effectively get 1 added to their number
Not exactly.
Sorry to those who already know this, but maybe it's time to go over linear
hashing in theory ..
Linear hashing was a system devised by Litwin and originally only for
in-memory lists. In fact there's some good
Good explanation, Brian!
To anyone who skipped it because it looked long: read it anyway.
cds
On 7/4/2012 5:26 AM, Brian Leach wrote:
Sorry to those who already know this, but maybe it's time to go over linear
hashing in theory ..
Linear hashing was a system devised by Litwin and originally
This makes it sound as if you might need to search two groups for a record,
which is not correct. If the initial hash is based on the larger modulo, and
the group exists, then the key will be in the higher number group. If the
result of the first hash is larger than the modulus of the of the
SMAT -d (or ANALYZE.SHM -d) see uv/bin/smat[.exe]
uv/bin/analyze.shm[.exe]
Dynamic Files:
Slot # Inode Device Ref Count Htype Split Merge Curmod Basemod
Largerec Filesp Selects Nextsplit
0 1285128087 209307792516208050 4001
2048 3267 2782736
On 04/07/12 11:26, Brian Leach wrote:
All the other groups effectively get 1 added to their number
Not exactly.
Sorry to those who already know this, but maybe it's time to go over linear
hashing in theory ..
Linear hashing was a system devised by Litwin and originally only for
in-memory
I believe PiOpen used a directory with two files in it ‘$0’ and ‘$1’
corresponding to DATA.30 and OVER.30. If the numbers went up from there, I
think that they corresponded to alternate keys, ie ‘$2’ and ‘$3’ represented
DATA.30 and OVER.30 for the first alternate key.
I do not think that
On 04/07/12 19:59, Rick Nuckolls wrote:
I believe PiOpen used a directory with two files in it ‘$0’ and ‘$1’
corresponding to DATA.30 and OVER.30. If the numbers went up from there, I
think that they corresponded to alternate keys, ie ‘$2’ and ‘$3’
represented DATA.30 and OVER.30 for the
On 04/07/12 17:44, Charles Stevenson wrote:
SMAT -d (or ANALYZE.SHM -d) see uv/bin/smat[.exe]
uv/bin/analyze.shm[.exe]
Dynamic Files:
Slot # Inode Device Ref Count Htype Split Merge Curmod Basemod
Largerec Filesp Selects Nextsplit
0 1285128087 209307792516
Almost. Though the file will look after itself, it may not do so very well.
Dynamic files, for best performance, do sometimes need periodic resizing.
Having said that it is true that some never resize Dynamic files.
If the minimum modulo is much lower than the actual, then this will cause
above I'm just trying to get an idea of what these should be
set at.
Thanks,
Chris
From: cjausti...@hotmail.com
To: u2-users@listserver.u2ug.org
Date: Tue, 3 Jul 2012 10:28:17 -0500
Subject: Re: [U2] RESIZE - dynamic files
Doug,
When I do the math I come up with a different # (see
Yep, I added an extra 2 in the key value. Oh, the perils of cut and
paste...
___
U2-Users mailing list
U2-Users@listserver.u2ug.org
http://listserver.u2ug.org/mailman/listinfo/u2-users
to be bigger than the number of groups (92,776
in this case)?
Chris
Date: Tue, 3 Jul 2012 11:04:53 -0600
From: dave...@gmail.com
To: u2-users@listserver.u2ug.org
Subject: Re: [U2] RESIZE - dynamic files
Yep, I added an extra 2 in the key value. Oh, the perils of cut and
paste
See comment interspersed...
Using the record above, how would I calculate the following?
1) MINIMUM.MODULUS (Is there a formula to use or should I add 20% to the
current number)?
Should be less the the current size, if you want the file to merge
2) SPLIT - would 90% seem about right?
on the table above I'm just trying to get an idea of what these should be
set at.
Thanks,
Chris
From: cjausti...@hotmail.com
To: u2-users@listserver.u2ug.org
Date: Tue, 3 Jul 2012 10:28:17 -0500
Subject: Re: [U2] RESIZE - dynamic files
Doug,
When I do the math I come up
: Tue, 3 Jul 2012 10:21:16 -0700
Subject: Re: [U2] RESIZE - dynamic files
(record + id / 4096 or 2048)
You need to factor in overhead the split factor: (records + ids) * 1.1 *
1.25 / 4096(for 80%)
If you use a 20% merge factor and a 80% split factor, the file will start
merging
Doug,
The data is growing over time with this file. Does that mean I should ignore
the formula? Or should I still use a lower MINIMUM.MODULO than the
actual modulo #..
Is the idea to reduce overflow by lowering the split? What is this 'overflow'
referring to?
2) SPLIT - would 90% seem
records.
Chris
From: r...@lynden.com
To: u2-users@listserver.u2ug.org
Date: Tue, 3 Jul 2012 10:21:16 -0700
Subject: Re: [U2] RESIZE - dynamic files
(record + id / 4096 or 2048)
You need to factor in overhead the split factor: (records + ids) * 1.1 *
1.25 / 4096(for 80
.
Chris
From: r...@lynden.com
To: u2-users@listserver.u2ug.org
Date: Tue, 3 Jul 2012 10:21:16 -0700
Subject: Re: [U2] RESIZE - dynamic files
(record + id / 4096 or 2048)
You need to factor in overhead the split factor: (records + ids) * 1.1 * 1.25 / 4096(for 80%)
If you use a 20
@listserver.u2ug.org
Subject: Re: [U2] RESIZE - dynamic files
Using the formula below, and changing the split to 90% I get the following:
File name .. GENACCTRN_POSTED
Pathname ... GENACCTRN_POSTED
File type .. DYNAMIC
File style and revision
of record IDs ... 21508521 bytes
Unused space ... 213343528 bytes
Total space for records 522252288 bytes
From: r...@lynden.com
To: u2-users@listserver.u2ug.org
Date: Tue, 3 Jul 2012 13:10:43 -0700
Subject: Re: [U2] RESIZE - dynamic files
The split load is not affecting
: r...@lynden.com
To: u2-users@listserver.u2ug.org
Date: Tue, 3 Jul 2012 13:10:43 -0700
Subject: Re: [U2] RESIZE - dynamic files
The split load is not affecting anything here, since it is more than the
actual load. What your overflow suggests is that you lower the split.load
value to 70
group, not
unreasonably shallow, and it's likely to be a while before you have to resize
again.
From: cjausti...@hotmail.com
To: u2-users@listserver.u2ug.org
Date: Tue, 3 Jul 2012 15:23:23 -0500
Subject: Re: [U2] RESIZE - dynamic files
I guess what I need to know is what's
...@hotmail.com
To: u2-users@listserver.u2ug.org
Date: Tue, 3 Jul 2012 16:57:34 -0400
Subject: Re: [U2] RESIZE - dynamic files
One rule of thumb is to make sure that you have an average of 10 or less
items in each group. Going by that, you'd want a minimum mod of 130k or more.
I've also noticed
Austin
Sent: Tuesday, July 03, 2012 2:15 PM
To: u2-users@listserver.u2ug.org
Subject: Re: [U2] RESIZE - dynamic files
Dan,
I changed the MINIMUM.MODULUS to the value of 23 as you suggested and my
Actual Load has really gone down (as well as overflow). See below for the
results:
File name
I should have said 60% more disk records, to be clear.
-Original Message-
From: u2-users-boun...@listserver.u2ug.org
[mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Rick Nuckolls
Sent: Tuesday, July 03, 2012 2:24 PM
To: 'U2 Users List'
Subject: Re: [U2] RESIZE - dynamic files
Austin
Sent: Tuesday, July 03, 2012 2:15 PM
To: u2-users@listserver.u2ug.org
Subject: Re: [U2] RESIZE - dynamic files
Dan,
I changed the MINIMUM.MODULUS to the value of 23 as you suggested and my
Actual Load has really gone down (as well as overflow). See below for the
results:
File
Disks get bigger much faster than the rate they get faster.
So the overflow is the thing to minimize.
-Original Message-
From: Chris Austin cjausti...@hotmail.com
To: u2-users u2-users@listserver.u2ug.org
Sent: Tue, Jul 3, 2012 2:38 pm
Subject: Re: [U2] RESIZE - dynamic files
his
, 2012 2:38 PM
To: u2-users@listserver.u2ug.org
Subject: Re: [U2] RESIZE - dynamic files
This is why I'm confused.. Is the goal here to reduce 'overflow' or to
keep the 'Total size' of the disk down? If the goal is to keep the total
disk size down then it would appear
you would want your actual
, Inc.
-Original Message-
From: u2-users-boun...@listserver.u2ug.org
[mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Chris Austin
Sent: 07/03/2012 5:38 PM
To: u2-users@listserver.u2ug.org
Subject: Re: [U2] RESIZE - dynamic files
This is why I'm confused.. Is the goal here to reduce
the split.load? What should my SPLIT.LOAD be around,
and how do you come up with that %?
Chris
From: r...@lynden.com
To: u2-users@listserver.u2ug.org
Date: Tue, 3 Jul 2012 14:45:28 -0700
Subject: Re: [U2] RESIZE - dynamic files
37% is a very low load. Reading disk records takes much longer than
To: u2-users@listserver.u2ug.org
Subject: Re: [U2] RESIZE - dynamic files
I set the split load based on what Dan suggested:
I'd take the merge down a little, to maybe 30% or even less, and maybe knock
the split up a bit - say, 90% - to cut down on the splitting.
I thought this would cut down
Chris,
Let's back way up. I take it your original question is a general one,
not specific to one poorly performing problematic file. Is that right?
If so, generally speaking, you just don't get a lot out of fine-tuning
dynamic files.
Tweaking the default parameters doesn't usually make a
: [U2] RESIZE - dynamic files
Chris,
Let's back way up. I take it your original question is a general one,
not specific to one poorly performing problematic file. Is that right?
If so, generally speaking, you just don't get a lot out of fine-tuning
dynamic files.
Tweaking the default
Better by Design!
-Original Message-
From: u2-users-boun...@listserver.u2ug.org
[mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Chris Austin
Sent: Wednesday, 4 July 2012 7:38 AM
To: u2-users@listserver.u2ug.org
Subject: Re: [U2] RESIZE - dynamic files
This is why I'm
Doug may have had a key bounce in his input
Let's do the math:
258687736 (Record Size)
192283300 (Key Size)
The key size is actually 19283300 in Chris' figures
Regarding 68,063 being less than the current modulus of 82,850. I think the
answer may lie in the splitting process.
Hi Chris,
The whole point of dynamic files is that you don't do RESIZE. The file will
look after itself, automatically responding to
variations in the volume of data.
There are knobs to twiddle but in most cases they can safely be left at their
defaults. A dynamic file will never perform as
The dynamic file I'm working with is below. What do 'overflowed' and 'badly'
refer to under MODULUS? Is the goal of the RESIZE to eliminate that
overflow? Any ideas what I should change to achieve this?
File name .. TEST_FILE
Pathname ... TEST_FILE
File type
Group size appears adequate (although anytime anything hashes into the group(s)
with the largest record [3267b], you'll split: 3267 is 79.8% of 4096, so if you
have a lot of records up in the 3K range, you may want to increase group size
and decrease min modulus accordingly), but the minimum
tuning a dynamic file?
Chris
From: martinphill...@ladybridge.com
To: u2-users@listserver.u2ug.org
Date: Mon, 2 Jul 2012 20:56:40 +0100
Subject: Re: [U2] RESIZE - dynamic files
Hi Chris,
The whole point of dynamic files is that you don't do RESIZE. The file will
look after itself
Hi Chris:
You cannot get away with not resizing dynamic files in my experience. The
files do not split and merge like we are led to believe. The separator is
not used on dynamic files. Your Universe file is badly sized. The math
below will get you reasonably file size.
Let's do the math:
...@listserver.u2ug.org] On Behalf Of Chris Austin
Sent: Monday, July 02, 2012 2:07 PM
To: u2-users@listserver.u2ug.org
Subject: Re: [U2] RESIZE - dynamic files
The dynamic file I'm working with is below. What do 'overflowed' and 'badly'
refer to under MODULUS? Is the goal of the RESIZE to eliminate
Should I put [AD] in the subject line for an unsolicited testimonial? :)
The best advice I can give you is to buy a product called FAST:
http://www.fitzlong.com/
A great tool for analyzing and resizing files, be they dynamic or standard
hashed files. Excellent support from excellent people at
[EMAIL PROTECTED] wrote on 06/12/2006 10:24:51 AM:
Does anyone have any tech tips on how to select parameters when
resizing dynamic files ?
The following is from a published tech tip. It provides guidelines, but
of course the nature of MV files makes it difficult to predict optimal
sizing.
What does the guide -r option do ?
We have been using the -a option.
Timothy Snyder [EMAIL PROTECTED] wrote:
[EMAIL PROTECTED] wrote on 06/12/2006 10:24:51 AM:
Does anyone have any tech tips on how to select parameters when
resizing dynamic files ?
The following is from a published
We have used the product here before.
I think our license on it lapsed.
I have been using the guide for several years instead of using fast.
Hennessey, Mark F. [EMAIL PROTECTED] wrote:
Should I put [AD] in the subject line for an unsolicited testimonial? :)
The best advice I can
[EMAIL PROTECTED] wrote on 06/12/2006 12:57:03 PM:
What does the guide -r option do ?
We have been using the -a option.
The -r option sends guide output to a hashed file. This makes it very easy
to select for files that are undersized, or that have corruption. So I'll
often do a
67 matches
Mail list logo
