Sent: Monday, November 04, 2002 4:58
AM
Subject: RE: Creating Large ECWs - RAM
requirements
Adam,
A
180GB mosaic should be no problem at all, so something is going wrong
here. The config
you
outline is pretty much what I use, and it works OK for 2TB mosaics for me,
which is the
largest I have tried on that config. And I note
you have been able to compress larger
images.
I
suggest you have a talk with ER Mapper technical support and look into the
issue, however here
are
some thoughts:
1.
When compressing, make sure you select "compress for internet usage". This
will
reduce compression memory
requirements.
How big is the compression wizard
reporting the input to be, in both total GB and in
width and
height?
Is it possible one of the inputs
is a much smaller cell size to the rest, so the effective
image
mosaic size is much larger
180GB?
Are the input files "compressed
for Internet Usage" (which also means "use less RAM")?
At what point is it failing?
Does it start to show any compression progress at all?
Is your mosaic very wide and not
very high? (e.g. 1,000,000 pixels or more wide?) ECW
memory
usage is based on line width,
regardless of how high the image is. So memory usage to
compress
an image 1,000,000 x 10,000 pixels
is pretty much the same as compressing an image that
is 1,000,000 x 1,000,000 pixels in
size, and much more than needed to compress an image
10,000 x 1,000,000 in
size.
2. Memory or VM is unlikely to be a
problem with only 180GB being compressed, but check the
Virtual Memory size for the
compression process. In the Task Manager, go to the Processes
tab,
and using the View->Select
Columns menu, turn on Memory Usage, Peak Memory Usage
and
Virtual Memory Size columns.
The last column (VM size) is really the only one we care about
here.
When it is running, see if
VM size gets to near 2GB. If so, that suggests running into a
memory
problem. But this is
unlikely to be an issue until you get above TB size
mosaics.
3. It is perhaps possible that your
system is not allocating enough VM for processes, perhaps
because
the volume being used for VM swap
is nearly full.
Have a quick look at My
Computer->Properties, under the Advanced tab, under Performance
Options..,
under the Change..
button.
I set page size on my machine to
4000 (4GB) for both initial size and maximum. This is faster,
as
the page file is always allocated,
and it also enables a large process with say 2GB of VM to run,
whilst
still leaving another 2GB for the
OS and other processes (although a single process can not
be
larger than 2GB VM under W2K, the
total for all processes can be).
4. It may be that a disk error has
corrupted an ECW file. As they are pyramidal in nature, you
may
not pick this error up until you
try and compress, which must read the entire image, rather than
just
an overview.
To check for this, save the
algorithm for the mosaic as a virtual dataset, then display it, with a
simple
convolution filter, with "process
at dataset resolution" checked. It will take forever to display, as
it
must read the entire 180GB mosaic,
however it will force reading the entire input files, which is
what
we want in this
case.
5. Check CPU temperature. I see you
are running AMD CPUs (so do I). The trouble I have,
with
two of my machines, is the AMD MP
CPUs are *very* temperature sensive, and one of my
machines
has trouble running for more than
an hour or so at full CPU load on both CPUs, without locking
up.
Compression can cause inherent
system faults to be highlighted, as it really stresses the
CPUs,
as both will run flat out, with
lots of FPU work. Which can cause a machine with a thermal
problem
to start
failing.
6. Is there anything particularly
complex about the algorithm? I think that the memory usage
is
slightly higher if you are running
multiple surfaces, so if you have each image in its own
surface,
instead of all in one surface,
more RAM would be required (but still, not all that much
more).
7. Have a look at the ECW Cache
Monitor (Compression toolbar). It shows ECW files being
read & cache information; it
might give a clue if the problem is related to the input files. It
will
also give you a low-level monitor
of the input images being read.
What
puzzles me is the PC lockup; this should not be happening.
If
you can check the actual compression dimensions, and CPU temperature, and VM,
and
if
any of the input files are corrupt, hopefully one of these will be the clue to
the problem.
As
an aside, I know that R&D put a lot of work was put into 6.3 to handle
very large mosaics
(mainly to deal with mosaics with 10,000's of input
files in the mosaics), so they are going
to
be very interested in hearing about your problem and what the resolution to it
is.
Kind
regards
Stuart
Hi All,
I'm running into problems creating an 18Gb
greyscale ECW air photo mosaic. I'm running ERM 6.3 on Win2000 with
SP2, Dual 1.9Mhz Athlon processors and 1Gb of 266Mhz DDR RAM. I have
15 greyscale ECW files compressed at 1:15 (they total 17.1
Gb) that represents 263Gb of uncompressed data. I want to
combine these files into one ECW file compressed at 1:15. I've tried
running the compression several times, but in every case my PC freezes
up. I have created mosaics RGB mosaics of this size before on slower
machines, so I'm fairly confident that I have the necessary hardware.
Any suggestions would be appreciated.
Regards,
Adam
