Re: Garbage collection (crashing on Windows)

[Mark Waddingham]

On 2016-08-22 14:47, Ben Rubinstein wrote:

https://blogs.technet.microsoft.com/askperf/2007/03/23/memory-management-demystifying-3gb/

This looks like a great tip - before I go into the ring with the
client's IT dept (always a tricky exercise) can I just check that
LiveCode does have the IMAGE_FILE_LARGE_ADDRESS_AWARE flag set in the
image header as described in that article? And do you know from which
version that is true?


That is a good question... It probably isn't (looking at the linker 
flags) - however you can set it on your standalone exe yourself:


http://stackoverflow.com/questions/1346480/how-to-make-a-net-application-large-address-aware

I'm pretty sure the LC codebase will be fine for use under this mode 
although it hasn't been explicitly tested.



But does 'freed' literally release the memory, or just mark the object
as available? Surely you still need to do some kind of garbage
collection in order to collapse what may be isolated fragments of
'free' memory?


When a value is no longer referenced, the area(s) of memory occupied by 
it are returned to the heap for future allocation. Note that LiveCode 
relies on the standard memory management functions (malloc / free) on 
the various OSes - they handle any coalescing of free regions (and 
handing unused pages back to the OS - some OSes are better at this than 
others).



How can we do this? As noted hasMemory is defunct; heapSpace is Mac
only; is there a method I can use to profile the memory usage?


On Mac there is - yes. The 'heap' command line tool lets you poke 
another process for its current memory usage (what is and isn't 
allocated in the C heap). On Windows, I think ProcExplorer can do a 
similar thing - 
https://technet.microsoft.com/en-us/sysinternals/processexplorer.aspx 
(if not I'm almost 100% certain there is a sysinternals tool which 
*does* give the usage information of any C heaps - win apps can have 
more than one - allocated by a process).


Hope this helps,

Warmest Regards,

Mark.

--
Mark Waddingham ~ m...@livecode.com ~ http://www.livecode.com/
LiveCode: Everyone can create apps

___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode

Re: Garbage collection (crashing on Windows)

[Ben Rubinstein]

Mark,

Thanks so much for this detailed and very useful response. A few quick 
follow-up questions.



1)
> 
https://blogs.technet.microsoft.com/askperf/2007/03/23/memory-management-demystifying-3gb/
This looks like a great tip - before I go into the ring with the client's IT 
dept (always a tricky exercise) can I just check that LiveCode does have the 
IMAGE_FILE_LARGE_ADDRESS_AWARE flag set in the image header as described in 
that article? And do you know from which version that is true?



(not a question)
> We actually changed this mechanism to make it less conservative in
> 6.7.11, 7.1.4 and 8.0 onwards. Previously, deleted objects wouldn't get
> actually freed until the root event loop runs

Aha!  Sadly I just updated the app to 6.7.11 without improving the situation 
(and on an experimental only basis, to 8.1 also, similarly without improvement).


2)

LiveCode doesn't use what is generally referred to as 'garbage collection'
as it generally frees 'things' up as soon as they are no longer referenced.


But does 'freed' literally release the memory, or just mark the object as 
available? Surely you still need to do some kind of garbage collection in 
order to collapse what may be isolated fragments of 'free' memory?


3)
In another thread (20/08/2016 19:44) you just wrote:
> For optimization purposes the best approach is to measure the amount of 
memory *actually* in use before and after any particular operation you perform 
- just as you do with time when profiling for speed (rather than memory 
footprint).


How can we do this? As noted hasMemory is defunct; heapSpace is Mac only; is 
there a method I can use to profile the memory usage?



Many thanks,

Ben

On 19/08/2016 18:42, Mark Waddingham wrote:

Hi Ben,

When I got to the end of this email I remembered something quite pertinent -
you mentioned that the limit you were hitting was 2Gb... One thing to check is
that the install of Windows you are running on cannot be poked to actually
raise this limit to 3Gb:

https://blogs.technet.microsoft.com/askperf/2007/03/23/memory-management-demystifying-3gb/


Perhaps other's with more insider Windows knowledge can chip in there. It will
depend on the machine, the version of Windows and probably lots of other
factors. Given that 'hardware is cheap' compared to rewriting software - if
the windows install being used currently does not use that 'trick', and can
be, you'll probably find you get a fair bit of mileage with a bit of computer
configuration - rather than coding!

Assuming that cannot be done then...

On 2016-08-17 19:52, Ben Rubinstein wrote:

Please refresh my memory: is there any way to cause/allow garbage to
be collected without ending all script running?


LiveCode doesn't use what is generally referred to as 'garbage collection' as
it generally frees 'things' up as soon as they are no longer referenced. Now I
say 'generally' because things fall into two classes:

   1) Values (strings, arrays, data, numbers)

   2) Objects (stacks, cards, buttons etc.)

I'll deal with Objects first:

Objects are deleted as soon as they can be relative to the requirements of the
engine. We actually changed this mechanism to make it less conservative in
6.7.11, 7.1.4 and 8.0 onwards. Previously, deleted objects wouldn't get
actually freed until the root event loop runs (i.e. when there is no script
running); now they will generally get freed much closer to when they are
deleted, especially if they were created 'at the same level or above' where
the object is deleted. e.g.

   on foo
 create control bar
 delete control bar
   end foo

Here the delete will free the object immediately (as the engine knows that it
cannot have any internal references to it internally - in particular on the C
stack).

It sounds like the problem you are having (assuming you aren't creating and
deleting lots of controls) is to do with values and so...

Values are freed *as soon as* there is no longer any reference to them. In 6.7
and before that would be whenever a variable is changed (the old value was
released immediately), or whenever the variable goes out of scope (e.g. locals
in a handler get released when the handler ends, script locals are released
when the object is deleted). In 7.0+ this happens as soon as there are no
variables referencing the same instance of the value. e.g.

  (1) local tVariable1, tVariable2
  (2) put "foo" & "bar" into tVariable1
  (3) put tVariable1 into tVariable2
  (4) put empty into tVariable1

After step (3), tVariable1 and tVariable2 will reference the same value. At
step (4) the reference tVariable1 holds will be removed, but the value will
not be deleted (from memory) until tVariable2 changes, or goes out of scope.
The general mechanism is that values are shared when copied into different
variables, and are only copied when a variable is mutated. e.g.

  (1) local tVariable1, tVariable2
  (2) put "foo" & "bar" into tVariable1
  (3) put tVariable1 into tVariable2
  (4) put "baz" after 

Re: Garbage collection (crashing on Windows)

[Monte Goulding]

> On 21 Aug 2016, at 2:33 AM, Dr. Hawkins  wrote:
> 
> In that case, what are the several second pauses in the IDE, particularly
> the editor (versions 5 & 7, maybe others)

Do you happen to have bug report numbers for these pauses because they can 
probably be fixed? I’ve recently done some work on two functions of the script 
editor that had quite inefficient and roughly 10 year old code. Indenting and 
searching entire stacks for strings will be *much* faster in 8.1. Searching all 
stacks and their stackFiles in the the IDE (something I do multiple times a day 
at the moment) has gone from a “get it going and go and get a coffee” thing to 
a “get it going and quickly look to see if you have any new email” thing.

Both of these huge improvements involved taking much more care with things that 
would require parsing entire blobs of text to add or extract one line. 

Cheers

Monte


___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode

Re: Garbage collection (crashing on Windows)

[Mark Waddingham]

Most likely the time it takes to layout the text for display - 'typesetting' 
(the process that converts a styled string into a list of glyphs and positions) 
is a very complicated and time consuming process.

Mark.

Sent from my iPhone

> On 20 Aug 2016, at 17:33, Dr. Hawkins  wrote:
> 
>> On Fri, Aug 19, 2016 at 10:42 AM, Mark Waddingham  wrote:
>> 
>> LiveCode doesn't use what is generally referred to as 'garbage collection'
>> as it generally frees 'things' up as soon as they are no longer referenced.
> 
> 
> In that case, what are the several second pauses in the IDE, particularly
> the editor (versions 5 & 7, maybe others)
> 
> 
> 
> -- 
> Dr. Richard E. Hawkins, Esq.
> (702) 508-8462
> ___
> use-livecode mailing list
> use-livecode@lists.runrev.com
> Please visit this url to subscribe, unsubscribe and manage your subscription 
> preferences:
> http://lists.runrev.com/mailman/listinfo/use-livecode

___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode

Re: Garbage collection (crashing on Windows)

[Dr. Hawkins]

On Fri, Aug 19, 2016 at 10:42 AM, Mark Waddingham  wrote:

> LiveCode doesn't use what is generally referred to as 'garbage collection'
> as it generally frees 'things' up as soon as they are no longer referenced.


In that case, what are the several second pauses in the IDE, particularly
the editor (versions 5 & 7, maybe others)



-- 
Dr. Richard E. Hawkins, Esq.
(702) 508-8462
___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode

Re: Garbage collection (crashing on Windows)

[Dr. Hawkins]

On Thu, Aug 18, 2016 at 11:09 AM, Jeanne A. E. DeVoto <
revolut...@jaedworks.com> wrote:

> I've had ugly crashes with large or complex arrays (that weren't anywhere
> near the 4G limit).


It's not just LiveCode where I trip over obscure bugs . .

That's actually how I found met buy in Cray's Fortran compiler (sold as
Absoft or some such at the time for Linux).

Apparently, I was the first person ever to attempt to allocate a dynamic
array over 256 megs with it.  As, for reasons beyond my comprehension, such
arrays are bit-addressed, this rolled over the 32 bit addressing, and
landed me in their next beta . . .

-- 
Dr. Richard E. Hawkins, Esq.
(702) 508-8462
___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode

Re: Garbage collection (crashing on Windows)

[Mark Talluto]

Split would works for simple data sets. This made up example would support 
split just fine.
 
> On Aug 19, 2016, at 8:29 PM, Monte Goulding  wrote:
> 
> Mark you could use the split command here if you don’t mind numerically 
> indexed arrays.
> 
>> On 20 Aug 2016, at 9:07 AM, Mark Talluto  wrote:
>> 
>> command createArray
>>local tListOfRecordIDs, tTableID
>> 
>>put fld "recordID data" into tListOfRecordIDs
>>put uuid() into tTableID
>> 
>>--FOUND IT FASTER TO WALK THROUGH LONG LISTS
>>--WHEN WE PUT THEM IN AN ARRAY LIKE THIS
>>--ONCE THE ARRAY IS MADE, IT WILL BE USED MANY TIMES OVER
>>--NEGATING THE COST OF MAKING THIS ARRAY
>>repeat for each line xRecordID in tListOfRecordIDs
>> put empty into tArrayA[tTableID][xRecordID]
>>end repeat
>> 
>>processArray tArrayA
>> end createArray
> 
> ___
> use-livecode mailing list
> use-livecode@lists.runrev.com
> Please visit this url to subscribe, unsubscribe and manage your subscription 
> preferences:
> http://lists.runrev.com/mailman/listinfo/use-livecode


___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode

Re: Garbage collection (crashing on Windows)

[Mark Waddingham]

Ah! Using repeat for each key is essentially using the keys of an array as a 
'proper list' - it is probably the fastest way currently to iterate over a 
sequence of strings.

In this case the value of each element in that array is immaterial - so the 
performance benefit comes iterating over the keys in this manner, rather than 
the way that array was constructed.

Mark.

Sent from my iPhone

> On 20 Aug 2016, at 00:07, Mark Talluto  wrote:
> 
> I do not remember how much of a performance boost we got from manipulating 
> long lists this way, but if memory serves, it was quite beneficial.
> This code sample is just something I made up on the fly. Hopefully it 
> demonstrates the value of this method.
> 
> command createArray
> local tListOfRecordIDs, tTableID
> 
> put fld "recordID data" into tListOfRecordIDs
> put uuid() into tTableID
> 
> --FOUND IT FASTER TO WALK THROUGH LONG LISTS
> --WHEN WE PUT THEM IN AN ARRAY LIKE THIS
> --ONCE THE ARRAY IS MADE, IT WILL BE USED MANY TIMES OVER
> --NEGATING THE COST OF MAKING THIS ARRAY
> repeat for each line xRecordID in tListOfRecordIDs
>  put empty into tArrayA[tTableID][xRecordID]
> end repeat
> 
> processArray tArrayA
> end createArray
> 
> 
> command processArray @rArrayA
> put the keys of rArrayA into tTableID
> 
> --WALK THROUGH ARRAY LIST
> repeat for each key xRecordID in rArrayA[tTableID]
>  --do something with xRecordID here
>  --we might modify something in this list
> end repeat
> 
> --WALK THROUGH ARRAY LIST
> repeat for each key xRecordID in rArrayA[tTableID]
>  --do something with this modified list from previous repeat
> end repeat
> end processArray
> 
> 
> -Mark
> 
> 
> 
>> On Aug 19, 2016, at 12:22 PM, Mark Waddingham  wrote:
>> 
>> With the current implementation of the engine, the only reason using empty 
>> is faster than using pValue is because the latter requires a variable 
>> lookup, the former is a constant.
>> 
>> What are you using sValueCache for?
>> 
>> Sent from my iPhone
>> 
>>> On 19 Aug 2016, at 19:37, Mark Talluto  wrote:
>>> 
>>> 
 On Aug 19, 2016, at 10:42 AM, Mark Waddingham  wrote:
 
 command shareAndStoreKey @xArray, pKey, pValue
  set the caseSensitive to true -- this is assuming your values are 
 sensitive to case
  if pValue is not among the keys of sValueCache then
  put pValue into sValueCache[pValue]
  end if
  put sValueCache[pValue] into xArray[pKey]
 end command
>>> 
>>> 
>>> Hi Mark,
>>> 
>>> We have been doing:
>>> put empty into sValueCache[pValue]--as an optimization. 
>>> 
>>> How does this compare to:
>>> put pValue into sValueCache[pValue]--in terms of performance?
>>> 
>>> 
>>> Best regards,
>>> 
>>> Mark Talluto
>>> livecloud.io
>>> canelasoftware.com
>>> 
>>> 
>>> 
>>> 
>>> ___
>>> use-livecode mailing list
>>> use-livecode@lists.runrev.com
>>> Please visit this url to subscribe, unsubscribe and manage your 
>>> subscription preferences:
>>> http://lists.runrev.com/mailman/listinfo/use-livecode
>> 
>> ___
>> use-livecode mailing list
>> use-livecode@lists.runrev.com
>> Please visit this url to subscribe, unsubscribe and manage your subscription 
>> preferences:
>> http://lists.runrev.com/mailman/listinfo/use-livecode
> 
> 
> ___
> use-livecode mailing list
> use-livecode@lists.runrev.com
> Please visit this url to subscribe, unsubscribe and manage your subscription 
> preferences:
> http://lists.runrev.com/mailman/listinfo/use-livecode

___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode

Re: Garbage collection (crashing on Windows)

[Monte Goulding]

Mark you could use the split command here if you don’t mind numerically indexed 
arrays.

> On 20 Aug 2016, at 9:07 AM, Mark Talluto  wrote:
> 
> command createArray
> local tListOfRecordIDs, tTableID
> 
> put fld "recordID data" into tListOfRecordIDs
> put uuid() into tTableID
> 
> --FOUND IT FASTER TO WALK THROUGH LONG LISTS
> --WHEN WE PUT THEM IN AN ARRAY LIKE THIS
> --ONCE THE ARRAY IS MADE, IT WILL BE USED MANY TIMES OVER
> --NEGATING THE COST OF MAKING THIS ARRAY
> repeat for each line xRecordID in tListOfRecordIDs
>  put empty into tArrayA[tTableID][xRecordID]
> end repeat
> 
> processArray tArrayA
> end createArray

___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode

Re: Garbage collection (crashing on Windows)

[Mark Talluto]

I do not remember how much of a performance boost we got from manipulating long 
lists this way, but if memory serves, it was quite beneficial.
This code sample is just something I made up on the fly. Hopefully it 
demonstrates the value of this method.

command createArray
 local tListOfRecordIDs, tTableID
 
 put fld "recordID data" into tListOfRecordIDs
 put uuid() into tTableID
 
 --FOUND IT FASTER TO WALK THROUGH LONG LISTS
 --WHEN WE PUT THEM IN AN ARRAY LIKE THIS
 --ONCE THE ARRAY IS MADE, IT WILL BE USED MANY TIMES OVER
 --NEGATING THE COST OF MAKING THIS ARRAY
 repeat for each line xRecordID in tListOfRecordIDs
  put empty into tArrayA[tTableID][xRecordID]
 end repeat
 
 processArray tArrayA
end createArray


command processArray @rArrayA
 put the keys of rArrayA into tTableID
 
 --WALK THROUGH ARRAY LIST
 repeat for each key xRecordID in rArrayA[tTableID]
  --do something with xRecordID here
  --we might modify something in this list
 end repeat
 
 --WALK THROUGH ARRAY LIST
 repeat for each key xRecordID in rArrayA[tTableID]
  --do something with this modified list from previous repeat
 end repeat
end processArray


-Mark



> On Aug 19, 2016, at 12:22 PM, Mark Waddingham  wrote:
> 
> With the current implementation of the engine, the only reason using empty is 
> faster than using pValue is because the latter requires a variable lookup, 
> the former is a constant.
> 
> What are you using sValueCache for?
> 
> Sent from my iPhone
> 
>> On 19 Aug 2016, at 19:37, Mark Talluto  wrote:
>> 
>> 
>>> On Aug 19, 2016, at 10:42 AM, Mark Waddingham  wrote:
>>> 
>>> command shareAndStoreKey @xArray, pKey, pValue
>>>   set the caseSensitive to true -- this is assuming your values are 
>>> sensitive to case
>>>   if pValue is not among the keys of sValueCache then
>>>   put pValue into sValueCache[pValue]
>>>   end if
>>>   put sValueCache[pValue] into xArray[pKey]
>>> end command
>> 
>> 
>> Hi Mark,
>> 
>> We have been doing:
>> put empty into sValueCache[pValue]--as an optimization. 
>> 
>> How does this compare to:
>> put pValue into sValueCache[pValue]--in terms of performance?
>> 
>> 
>> Best regards,
>> 
>> Mark Talluto
>> livecloud.io
>> canelasoftware.com
>> 
>> 
>> 
>> 
>> ___
>> use-livecode mailing list
>> use-livecode@lists.runrev.com
>> Please visit this url to subscribe, unsubscribe and manage your subscription 
>> preferences:
>> http://lists.runrev.com/mailman/listinfo/use-livecode
> 
> ___
> use-livecode mailing list
> use-livecode@lists.runrev.com
> Please visit this url to subscribe, unsubscribe and manage your subscription 
> preferences:
> http://lists.runrev.com/mailman/listinfo/use-livecode


___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode

Re: Garbage collection (crashing on Windows)

[mwieder]

Ah, the wonders of paged memory.



-
-- 
 Mark Wieder
 ahsoftw...@gmail.com
--
View this message in context: 
http://runtime-revolution.278305.n4.nabble.com/Garbage-collection-crashing-on-Windows-tp4707560p4707624.html
Sent from the Revolution - User mailing list archive at Nabble.com.

___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode

Re: Garbage collection (crashing on Windows)

[Mark Waddingham]

With the current implementation of the engine, the only reason using empty is 
faster than using pValue is because the latter requires a variable lookup, the 
former is a constant.

What are you using sValueCache for?

Sent from my iPhone

> On 19 Aug 2016, at 19:37, Mark Talluto  wrote:
> 
> 
>> On Aug 19, 2016, at 10:42 AM, Mark Waddingham  wrote:
>> 
>> command shareAndStoreKey @xArray, pKey, pValue
>>set the caseSensitive to true -- this is assuming your values are 
>> sensitive to case
>>if pValue is not among the keys of sValueCache then
>>put pValue into sValueCache[pValue]
>>end if
>>put sValueCache[pValue] into xArray[pKey]
>>  end command
> 
> 
> Hi Mark,
> 
> We have been doing:
> put empty into sValueCache[pValue]--as an optimization. 
> 
> How does this compare to:
> put pValue into sValueCache[pValue]--in terms of performance?
> 
> 
> Best regards,
> 
> Mark Talluto
> livecloud.io
> canelasoftware.com
> 
> 
> 
> 
> ___
> use-livecode mailing list
> use-livecode@lists.runrev.com
> Please visit this url to subscribe, unsubscribe and manage your subscription 
> preferences:
> http://lists.runrev.com/mailman/listinfo/use-livecode

___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode

Re: Garbage collection (crashing on Windows)

[Mark Talluto]

> On Aug 19, 2016, at 10:42 AM, Mark Waddingham  wrote:
> 
> command shareAndStoreKey @xArray, pKey, pValue
> set the caseSensitive to true -- this is assuming your values are 
> sensitive to case
> if pValue is not among the keys of sValueCache then
> put pValue into sValueCache[pValue]
> end if
> put sValueCache[pValue] into xArray[pKey]
>   end command


Hi Mark,

We have been doing:
put empty into sValueCache[pValue]--as an optimization. 

How does this compare to:
put pValue into sValueCache[pValue]--in terms of performance?


Best regards,

Mark Talluto
livecloud.io
canelasoftware.com




___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode

Re: Garbage collection (crashing on Windows)

[Mark Waddingham]

Hi Ben,

When I got to the end of this email I remembered something quite 
pertinent - you mentioned that the limit you were hitting was 2Gb... One 
thing to check is that the install of Windows you are running on cannot 
be poked to actually raise this limit to 3Gb:


https://blogs.technet.microsoft.com/askperf/2007/03/23/memory-management-demystifying-3gb/

Perhaps other's with more insider Windows knowledge can chip in there. 
It will depend on the machine, the version of Windows and probably lots 
of other factors. Given that 'hardware is cheap' compared to rewriting 
software - if the windows install being used currently does not use that 
'trick', and can be, you'll probably find you get a fair bit of mileage 
with a bit of computer configuration - rather than coding!


Assuming that cannot be done then...

On 2016-08-17 19:52, Ben Rubinstein wrote:

Please refresh my memory: is there any way to cause/allow garbage to
be collected without ending all script running?


LiveCode doesn't use what is generally referred to as 'garbage 
collection' as it generally frees 'things' up as soon as they are no 
longer referenced. Now I say 'generally' because things fall into two 
classes:


   1) Values (strings, arrays, data, numbers)

   2) Objects (stacks, cards, buttons etc.)

I'll deal with Objects first:

Objects are deleted as soon as they can be relative to the requirements 
of the engine. We actually changed this mechanism to make it less 
conservative in 6.7.11, 7.1.4 and 8.0 onwards. Previously, deleted 
objects wouldn't get actually freed until the root event loop runs (i.e. 
when there is no script running); now they will generally get freed much 
closer to when they are deleted, especially if they were created 'at the 
same level or above' where the object is deleted. e.g.


   on foo
 create control bar
 delete control bar
   end foo

Here the delete will free the object immediately (as the engine knows 
that it cannot have any internal references to it internally - in 
particular on the C stack).


It sounds like the problem you are having (assuming you aren't creating 
and deleting lots of controls) is to do with values and so...


Values are freed *as soon as* there is no longer any reference to them. 
In 6.7 and before that would be whenever a variable is changed (the old 
value was released immediately), or whenever the variable goes out of 
scope (e.g. locals in a handler get released when the handler ends, 
script locals are released when the object is deleted). In 7.0+ this 
happens as soon as there are no variables referencing the same instance 
of the value. e.g.


  (1) local tVariable1, tVariable2
  (2) put "foo" & "bar" into tVariable1
  (3) put tVariable1 into tVariable2
  (4) put empty into tVariable1

After step (3), tVariable1 and tVariable2 will reference the same value. 
At step (4) the reference tVariable1 holds will be removed, but the 
value will not be deleted (from memory) until tVariable2 changes, or 
goes out of scope. The general mechanism is that values are shared when 
copied into different variables, and are only copied when a variable is 
mutated. e.g.


  (1) local tVariable1, tVariable2
  (2) put "foo" & "bar" into tVariable1
  (3) put tVariable1 into tVariable2
  (4) put "baz" after tVariable2
  (5) put empty into tVariable1

Here, at step (4), the value referenced by tVariable2 will be copied 
(and so tVariable1 and tVariable2 will no longer reference the same 
value), and then changed. This means that at step (5) the value 
previously referenced by tVariable1 *will* be freed, because it is not 
shared with tVariable2 (obviously - because tVariable2 is no longer the 
same value!).


The reason I was being so paedagogic in the above is that it opens an 
opportunity for you to potentially reduce the memory footprint of your 
dataset (which sounds like it is what is causing the problem) by doing 
some pre-processing and exploiting the fact that values are not copied 
until they are modified. Of course, I don't know what the structure of 
the data you are processing is - so I'm going to assume you are loading 
in lots of text files and breaking them up into pieces, presumably 
storing in arrays with the individual array elements being numbers and 
strings.


In this case there are a few interesting things to note about the 
engine's implementation of values...


Array keys are *always* shared (up to case). When you do:

   put tElement into tArray[tKey]

The engine first 'uniques' tKey - this means it ensures that there is 
only one copy of tKey (up to case differences) in memory. So - for every 
single array in memory which contains a key "foo", the value 
representing the key "foo" will not be copied, just referenced from all 
the arrays. Note that "foo" and "Foo", whilst referencing the same value 
(unless caseSensitive is true), will be stored in memory as different 
values which leads to memory optimization tip 1:


   When constructing arrays from 

Re: Garbage collection (crashing on Windows)

[Ben Rubinstein]

On 18/08/2016 19:09, Jeanne A. E. DeVoto wrote:

At 3:28 PM +0100 8/18/2016, Ben Rubinstein wrote:

The real problem is that the script was written originally 13 years ago, and
the basic architecture was to load everything into large global arrays for
cleanliness.



I've had ugly crashes with large or complex arrays (that weren't anywhere near
the 4G limit). There are (or perhaps were - this may have been fixed in the
8.x cycle) some problems with the way such arrays were structured in memory,
IIRC.

Which engine version are you using? (Apologies if you already said that and I
missed this.)


Every version from 2.4 to 8.1!

One problem of course is that over this long my memory fades - checking the 
script I see that it has a number of instances of
  wait for 1 second with messages -- theory that this might improve garbage 
collection


which is why I thought I'd established that this doesn't work; although the 
latest problem is arising earlier in the process than previously, so I'll 
review to see whether I could also do this earlier, in case it does in fact help.


Ben


___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode

Re: Garbage collection (crashing on Windows)

[Jeanne A. E. DeVoto]

At 3:28 PM +0100 8/18/2016, Ben Rubinstein wrote:
The real problem is that the script was written originally 13 years 
ago, and the basic architecture was to load everything into large 
global arrays for cleanliness.



I've had ugly crashes with large or complex arrays (that weren't 
anywhere near the 4G limit). There are (or perhaps were - this may 
have been fixed in the 8.x cycle) some problems with the way such 
arrays were structured in memory, IIRC.


Which engine version are you using? (Apologies if you already said 
that and I missed this.)


___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode

Re: Garbage collection (crashing on Windows)

[Richard Gaskin]

Ben Rubinstein wrote:

> The real problem is that the script was written originally 13 years
> ago, and the basic architecture was to load everything into large
> global arrays for cleanliness. Over that period, the size of the data
> has multiplied over the years by 1-2 orders of magnitude

Just how big is the data you need to work with in RAM at any given time?


> My two biggest problems are:
>
> - LC just in effect crashes/halts, rather than throwing an error so
> the app could at least report the problem

The challenge with low-memory conditions are that if there isn't enough 
RAM to operate normally there may not be enough RAM to report that 
through normal error-handling mechanisms.


It would certainly be nice to have reliable out-of-memory reporting, but 
I've never seen in it any xTalk yet.



> - as far as I can tell, when I empty globals etc the memory isn't
> being recovered.  This is really the root of my problem.

At first glance that sounds like a bug.  If you can find a way to create 
a simple test case to demonstrate a global not freeing up RAM after 
being emptied I'll bet the team could fix it quickly.


Grasping at straws here, I wonder if introducing an idle after the 
emptying might help, e.g.:


  put empty into gSomeVar
  wait 0 with messages

That should normally surrender control to the OS event loop, and if 
that's needed to complete the clearing of the RAM you may have a 
workaround for now.


I'd still call it a workaround, though, since we can't expect newcomers 
to be able to guess about such things; ideally the engine would clear 
memory when we ask it to.


If "put empty..." doesn't work, I wonder if this might trigger something 
more complete:


  delete variable gSomeVar

--
 Richard Gaskin
 Fourth World Systems
 Software Design and Development for the Desktop, Mobile, and the Web
 
 ambassa...@fourthworld.comhttp://www.FourthWorld.com

___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode

Re: Garbage collection (crashing on Windows)

[Mike Bonner]

Or you can probably delete the whole variable directly.

On Thu, Aug 18, 2016 at 8:59 AM, Mike Bonner  wrote:

> If you switch to using an array key with your global variable, and then "
> delete global gArray[yourkey]" it should release the memory.
>
>
> On Thu, Aug 18, 2016 at 8:28 AM, Ben Rubinstein 
> wrote:
>
>> Thanks Monte, Richard.
>>
>> As it happens this script isn't doing with properties, which is good (or
>> bad since that potentially easy fix isn't available to me!) but that's a
>> useful tip to remember. I've already moved some very large functions to
>> passing by reference.
>>
>> The real problem is that the script was written originally 13 years ago,
>> and the basic architecture was to load everything into large global arrays
>> for cleanliness. Over that period, the size of the data has multiplied over
>> the years by 1-2 orders of magnitude; and meanwhile the processing has been
>> modified and warped almost continuously, so it's a really complicated rats
>> nest. The whole project is reaching its sunset, so the client isn't keen to
>> invest in major refactoring... but just at this point we seem to be
>> regularly hitting this limit.
>>
>> My two biggest problems are:
>>
>> - LC just in effect crashes/halts, rather than throwing an error so the
>> app could at least report the problem
>>
>> - as far as I can tell, when I empty globals etc the memory isn't being
>> recovered.  This is really the root of my problem.
>>
>> Ben
>>
>>
>> On 18/08/2016 00:26, Monte Goulding wrote:
>>
>>> There are certainly a number of tricks you can use to reduce the memory
>>> usage of your app. One interesting one I learnt recently after an internal
>>> discussion was because LiveCode copies values that have multiple reference
>>> when they are mutated and custom properties are arrays linked to the object
>>> you can save some memory by doing the following:
>>>
>>> put the myProp of me into tVar
>>> set the myProp of me to empty — here we are dereferencing the value so
>>> LiveCode doesn’t do a deep copy
>>> MutateVar tVar
>>> set the myProp of me to tVar
>>>
>>> Obviously there’s also passing by reference etc to reduce copies.
>>>
>>> The other thing you want to look at doing here is processing data in
>>> chunks rather than reading it all in.
>>>
>>> Cheers
>>>
>>> Monte
>>>
>>> On 18 Aug 2016, at 3:52 AM, Ben Rubinstein  wrote:

 Please refresh my memory: is there any way to cause/allow garbage to be
 collected without ending all script running?

 I have an app which process large amounts of data. It runs fine on Mac,
 but on Windows intermittently (but not frequently as the application data
 source, which fluctuates, is gradually increasing) as the total memory in
 use approaches 2GB, the app faults.

 Obviously this is very distressing for me and my client, because the
 app has lost control at this point and can't even tell us what's happening
 (it normally runs on a schedule on a VM, so it often takes days to discover
 that the system downstream hasn't been updated for a while, and trace back
 to find that the LC app is halted).

 I have managed to reduce the incidence of this by breaking the
 processing into two parts, but this is undesirable for various reasons, and
 only palliative. What I really want is a way for my main function to invoke
 garbage collection after calling some of the larger processing steps.

 Is there any way to do this? Am I right in thinking that the issue is
 related to LC's total memory usage reaching 2GB on Windows? Are there any
 useful functions (e.g. the "hasMemory" function is documented as "only
 partially implemented, and may not return useful values on some platforms",
 which makes it seem pretty pointless...)?

>>>
>>> ___
>>> use-livecode mailing list
>>> use-livecode@lists.runrev.com
>>> Please visit this url to subscribe, unsubscribe and manage your
>>> subscription preferences:
>>> http://lists.runrev.com/mailman/listinfo/use-livecode
>>>
>>>
>>
>> ___
>> use-livecode mailing list
>> use-livecode@lists.runrev.com
>> Please visit this url to subscribe, unsubscribe and manage your
>> subscription preferences:
>> http://lists.runrev.com/mailman/listinfo/use-livecode
>
>
>
___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode

Re: Garbage collection (crashing on Windows)

[Mike Bonner]

If you switch to using an array key with your global variable, and then "
delete global gArray[yourkey]" it should release the memory.


On Thu, Aug 18, 2016 at 8:28 AM, Ben Rubinstein  wrote:

> Thanks Monte, Richard.
>
> As it happens this script isn't doing with properties, which is good (or
> bad since that potentially easy fix isn't available to me!) but that's a
> useful tip to remember. I've already moved some very large functions to
> passing by reference.
>
> The real problem is that the script was written originally 13 years ago,
> and the basic architecture was to load everything into large global arrays
> for cleanliness. Over that period, the size of the data has multiplied over
> the years by 1-2 orders of magnitude; and meanwhile the processing has been
> modified and warped almost continuously, so it's a really complicated rats
> nest. The whole project is reaching its sunset, so the client isn't keen to
> invest in major refactoring... but just at this point we seem to be
> regularly hitting this limit.
>
> My two biggest problems are:
>
> - LC just in effect crashes/halts, rather than throwing an error so the
> app could at least report the problem
>
> - as far as I can tell, when I empty globals etc the memory isn't being
> recovered.  This is really the root of my problem.
>
> Ben
>
>
> On 18/08/2016 00:26, Monte Goulding wrote:
>
>> There are certainly a number of tricks you can use to reduce the memory
>> usage of your app. One interesting one I learnt recently after an internal
>> discussion was because LiveCode copies values that have multiple reference
>> when they are mutated and custom properties are arrays linked to the object
>> you can save some memory by doing the following:
>>
>> put the myProp of me into tVar
>> set the myProp of me to empty — here we are dereferencing the value so
>> LiveCode doesn’t do a deep copy
>> MutateVar tVar
>> set the myProp of me to tVar
>>
>> Obviously there’s also passing by reference etc to reduce copies.
>>
>> The other thing you want to look at doing here is processing data in
>> chunks rather than reading it all in.
>>
>> Cheers
>>
>> Monte
>>
>> On 18 Aug 2016, at 3:52 AM, Ben Rubinstein  wrote:
>>>
>>> Please refresh my memory: is there any way to cause/allow garbage to be
>>> collected without ending all script running?
>>>
>>> I have an app which process large amounts of data. It runs fine on Mac,
>>> but on Windows intermittently (but not frequently as the application data
>>> source, which fluctuates, is gradually increasing) as the total memory in
>>> use approaches 2GB, the app faults.
>>>
>>> Obviously this is very distressing for me and my client, because the app
>>> has lost control at this point and can't even tell us what's happening (it
>>> normally runs on a schedule on a VM, so it often takes days to discover
>>> that the system downstream hasn't been updated for a while, and trace back
>>> to find that the LC app is halted).
>>>
>>> I have managed to reduce the incidence of this by breaking the
>>> processing into two parts, but this is undesirable for various reasons, and
>>> only palliative. What I really want is a way for my main function to invoke
>>> garbage collection after calling some of the larger processing steps.
>>>
>>> Is there any way to do this? Am I right in thinking that the issue is
>>> related to LC's total memory usage reaching 2GB on Windows? Are there any
>>> useful functions (e.g. the "hasMemory" function is documented as "only
>>> partially implemented, and may not return useful values on some platforms",
>>> which makes it seem pretty pointless...)?
>>>
>>
>> ___
>> use-livecode mailing list
>> use-livecode@lists.runrev.com
>> Please visit this url to subscribe, unsubscribe and manage your
>> subscription preferences:
>> http://lists.runrev.com/mailman/listinfo/use-livecode
>>
>>
>
> ___
> use-livecode mailing list
> use-livecode@lists.runrev.com
> Please visit this url to subscribe, unsubscribe and manage your
> subscription preferences:
> http://lists.runrev.com/mailman/listinfo/use-livecode
___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode

Re: Garbage collection (crashing on Windows)

[Ben Rubinstein]

Thanks Monte, Richard.

As it happens this script isn't doing with properties, which is good (or bad 
since that potentially easy fix isn't available to me!) but that's a useful 
tip to remember. I've already moved some very large functions to passing by 
reference.


The real problem is that the script was written originally 13 years ago, and 
the basic architecture was to load everything into large global arrays for 
cleanliness. Over that period, the size of the data has multiplied over the 
years by 1-2 orders of magnitude; and meanwhile the processing has been 
modified and warped almost continuously, so it's a really complicated rats 
nest. The whole project is reaching its sunset, so the client isn't keen to 
invest in major refactoring... but just at this point we seem to be regularly 
hitting this limit.


My two biggest problems are:

- LC just in effect crashes/halts, rather than throwing an error so the app 
could at least report the problem


- as far as I can tell, when I empty globals etc the memory isn't being 
recovered.  This is really the root of my problem.


Ben


On 18/08/2016 00:26, Monte Goulding wrote:

There are certainly a number of tricks you can use to reduce the memory usage 
of your app. One interesting one I learnt recently after an internal discussion 
was because LiveCode copies values that have multiple reference when they are 
mutated and custom properties are arrays linked to the object you can save some 
memory by doing the following:

put the myProp of me into tVar
set the myProp of me to empty — here we are dereferencing the value so LiveCode 
doesn’t do a deep copy
MutateVar tVar
set the myProp of me to tVar

Obviously there’s also passing by reference etc to reduce copies.

The other thing you want to look at doing here is processing data in chunks 
rather than reading it all in.

Cheers

Monte


On 18 Aug 2016, at 3:52 AM, Ben Rubinstein  wrote:

Please refresh my memory: is there any way to cause/allow garbage to be 
collected without ending all script running?

I have an app which process large amounts of data. It runs fine on Mac, but on 
Windows intermittently (but not frequently as the application data source, 
which fluctuates, is gradually increasing) as the total memory in use 
approaches 2GB, the app faults.

Obviously this is very distressing for me and my client, because the app has 
lost control at this point and can't even tell us what's happening (it normally 
runs on a schedule on a VM, so it often takes days to discover that the system 
downstream hasn't been updated for a while, and trace back to find that the LC 
app is halted).

I have managed to reduce the incidence of this by breaking the processing into 
two parts, but this is undesirable for various reasons, and only palliative. 
What I really want is a way for my main function to invoke garbage collection 
after calling some of the larger processing steps.

Is there any way to do this? Am I right in thinking that the issue is related to LC's total memory 
usage reaching 2GB on Windows? Are there any useful functions (e.g. the "hasMemory" 
function is documented as "only partially implemented, and may not return useful values on 
some platforms", which makes it seem pretty pointless...)?


___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode




___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode

Re: Garbage collection (crashing on Windows)

[Richard Gaskin]

Ben Rubinstein wrote:

Please refresh my memory: is there any way to cause/allow garbage to be
collected without ending all script running?

I have an app which process large amounts of data. It runs fine on Mac, but on
Windows intermittently (but not frequently as the application data source,
which fluctuates, is gradually increasing) as the total memory in use
approaches 2GB, the app faults.

Obviously this is very distressing for me and my client, because the app has
lost control at this point and can't even tell us what's happening (it
normally runs on a schedule on a VM, so it often takes days to discover that
the system downstream hasn't been updated for a while, and trace back to find
that the LC app is halted).

I have managed to reduce the incidence of this by breaking the processing into
two parts, but this is undesirable for various reasons, and only palliative.
What I really want is a way for my main function to invoke garbage collection
after calling some of the larger processing steps.

Is there any way to do this? Am I right in thinking that the issue is related
to LC's total memory usage reaching 2GB on Windows? Are there any useful
functions (e.g. the "hasMemory" function is documented as "only partially
implemented, and may not return useful values on some platforms", which makes
it seem pretty pointless...)?


LC's hasMemory function was useful in Mac Classic where it was first 
introduced to the xTalk world in SuperCard, but with modern memory 
management it would be difficult if not impossible to try to make it 
work as it used to, allowing an app to modify the system's control of 
memory.


Monte's tips were great (thanks Monte, love learning engine details like 
that!), but since memory is only problematic on Windows but not Mac, I 
wonder if this may be merely a leak that could be fixed?


Are you deleting variables when they're no longer needed?  Does 
introducing a periodic idle help?


Can you tell us a bit about what the app is working on, particularly the 
size of the data and what it's doing with it?


If it's a leak we should be able to hone in on what's causing it.  And 
if it's an expected behavior that needs tailoring for your algo, I'll 
bet we can find a way to make it work as well on Win as it does on Mac.


--
 Richard Gaskin
 Fourth World Systems
 Software Design and Development for the Desktop, Mobile, and the Web
 
 ambassa...@fourthworld.comhttp://www.FourthWorld.com

___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode

Re: Garbage collection (crashing on Windows)

[Monte Goulding]

There are certainly a number of tricks you can use to reduce the memory usage 
of your app. One interesting one I learnt recently after an internal discussion 
was because LiveCode copies values that have multiple reference when they are 
mutated and custom properties are arrays linked to the object you can save some 
memory by doing the following:

put the myProp of me into tVar
set the myProp of me to empty — here we are dereferencing the value so LiveCode 
doesn’t do a deep copy
MutateVar tVar
set the myProp of me to tVar

Obviously there’s also passing by reference etc to reduce copies.

The other thing you want to look at doing here is processing data in chunks 
rather than reading it all in.

Cheers

Monte

> On 18 Aug 2016, at 3:52 AM, Ben Rubinstein  wrote:
> 
> Please refresh my memory: is there any way to cause/allow garbage to be 
> collected without ending all script running?
> 
> I have an app which process large amounts of data. It runs fine on Mac, but 
> on Windows intermittently (but not frequently as the application data source, 
> which fluctuates, is gradually increasing) as the total memory in use 
> approaches 2GB, the app faults.
> 
> Obviously this is very distressing for me and my client, because the app has 
> lost control at this point and can't even tell us what's happening (it 
> normally runs on a schedule on a VM, so it often takes days to discover that 
> the system downstream hasn't been updated for a while, and trace back to find 
> that the LC app is halted).
> 
> I have managed to reduce the incidence of this by breaking the processing 
> into two parts, but this is undesirable for various reasons, and only 
> palliative. What I really want is a way for my main function to invoke 
> garbage collection after calling some of the larger processing steps.
> 
> Is there any way to do this? Am I right in thinking that the issue is related 
> to LC's total memory usage reaching 2GB on Windows? Are there any useful 
> functions (e.g. the "hasMemory" function is documented as "only partially 
> implemented, and may not return useful values on some platforms", which makes 
> it seem pretty pointless...)?

___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode

Garbage collection (crashing on Windows)

[Ben Rubinstein]

Please refresh my memory: is there any way to cause/allow garbage to be 
collected without ending all script running?


I have an app which process large amounts of data. It runs fine on Mac, but on 
Windows intermittently (but not frequently as the application data source, 
which fluctuates, is gradually increasing) as the total memory in use 
approaches 2GB, the app faults.


Obviously this is very distressing for me and my client, because the app has 
lost control at this point and can't even tell us what's happening (it 
normally runs on a schedule on a VM, so it often takes days to discover that 
the system downstream hasn't been updated for a while, and trace back to find 
that the LC app is halted).


I have managed to reduce the incidence of this by breaking the processing into 
two parts, but this is undesirable for various reasons, and only palliative. 
What I really want is a way for my main function to invoke garbage collection 
after calling some of the larger processing steps.


Is there any way to do this? Am I right in thinking that the issue is related 
to LC's total memory usage reaching 2GB on Windows? Are there any useful 
functions (e.g. the "hasMemory" function is documented as "only partially 
implemented, and may not return useful values on some platforms", which makes 
it seem pretty pointless...)?


TIA,

Ben

___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode