I would second that, with an extension: it would be most helpful to have some guidance on memory management overall.

Many see the total addressable space afforded by 64-bit systems as "Cool, I can use a 16-terabyte array!"

But as we've seen in bits and pieces here and elsewhere, total addressable memory is not what LC will necessarily use, or can use, for many tasks. And it some cases it may not be what we really want to do.

OS APIs impose their own limits, and many other factors can come into play as well.

It would be useful to have at least some general guidance on what those limitations are, both in terms of LC as it is today and in terms where it bumps into system limits and general memory management challenges.

Often these implications aren't obvious.

For example, this morning I noted the difficulty with exporting an image of a 5000 X 5000 px button on Ubuntu. Later I tried a set of much smaller objects but spread out much more broadly (10k px and more), and LC remained very responsive throughout - even exported the snapshot fine.

So I'm guessing from this that the difficulty with making unusually large standard controls was specific to how Linux' GDK handles them internally. Indeed, making LC-handled objects, like graphics, apparently have no problem being rendered and buffered at such sizes.

Rather than make this an attempt at an exhaustive study of memory implications (though I'd gladly read it, it would take too long to write), it might be nice if Mark could draw from examples he's seen in the community over the years to outline some of the more commonly requested scenarios.

--
 Richard Gaskin


hh wrote:

Please post this "Split it!"- answer, as it is, in LC's blog.
This is good even for real beginners.

Large files or large data shouldn't be a reason for _incomplete_ 64Bit
implementations that would make once again LC Script slower.

Mark Waddingham wrote:
>
> Tom Glod wrote:
> I will... if u wanna replicate...put an image on a stack..make it
> 32k x 32k
> ..... and try and do a export snapshot of the image,  LC goes POOF...
> Trevor
> said tha last version of 8 (8.13)  had some memory issues solves, so i
> will try to test is there too.

Currently, the engine implements 'export snapshot' by allocating a
raster (32-bits per pixel) of the size of the snapshot you are making,
rendering into it and then compressing it.

So really the maximum size you could hope to snapshot is 16k x* 16k
pixels as that requires 2Gb - the engine in general uses signed integer
indicies, so the maximum memory buffer it can manipulate is 2Gb bytes. A
32-bit process would probably struggle to do that (due to only having
around 2-3Gb of user address space to use) - as there is overhead in
rasterization and then compression; but a 64-bit process should be fine.

There is a bug here as (at least in this specific case) the engine
should fail gracefully (as we know there is a hard limit in the size of
an image the engine can process).

As you correctly point out 32k x 32k comes in at 1Gb pixels - which at
24-bit RGB comes out at 4Gb of data. No 'normal' 32-bit application
which isn't explicitly designed for manipulating huge images will be
able to deal with something that size. I would expect applications such
as Photoshop to be able to deal with them though since I believe their
native raw storage format for images pages from disk as required (so you
never have the 'whole thing' in memory at once - just the bit you are
looking at / editing).

One important thing to remember is that the amount of memory required to
take a snapshot is (SOMECONSTANT * 4 * the number of pixels) in the rect
of the snapshot (I've not checked but I would estimate 0 < SOMECONSTANT
< 2) which means that you can get LiveCode to generate very large
images, but you have to break the problem down by splitting up the
snapshot into bands and probably use an external (command-line) tool to
compress the image into your format of choice (how big an image such a
tool can process, again, will be dependent on whether it needs to load
the entire image into memory to compress it or not).

Rough idea:

     repeat with i = 0 to pHeight / kBandSize
        import snapshot from rect (0, pWidth, i * kBandSize, kBandSize)
        write the imageData of the last image to file tOutputFile
        delete the last image
     end repeat

After this you will have a very large file with the raw xRGB data in it,
so you need to find a tool which can take raw 32-bit xRGB data (with
specified order of the RGB), the width and height and process it into
jpg or png or whatever format you require (I'm hoping others who know
more about such things might be able to chime in here - ImageMagick has
an arsenal of command-line tools, for example).

Warmest Regards,

Mark.

P.S. There is a hard limit of co-ordinate magnitude in LC and thus the
size of any object - 32767 pixels on any side of anything.


_______________________________________________
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode

Reply via email to