Re: how to clear residual garbage in a stack?
> I haven't actually tried it but would it work to just put empty into the > field before deleting the card? That would be sort of a "clean as you go" > approach. That’s a very neat workaround which should leave at most a couple of bytes of uncollected garbage, until the bug is squashed. Of course most people wouldn’t need to worry about the problem at all, the memory leak involved would hardly be noticeable for most stacks, it’s just that it accumulates over time. I have submitted the bug, and also the Windows 10 saving problem which is really my concern, to QualityControl. Neville Smythe ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode
Re: how to clear residual garbage in a stack?
I haven't actually tried it but would it work to just put empty into the field before deleting the card? That would be sort of a "clean as you go" approach. -- Jacqueline Landman Gay | jac...@hyperactivesw.com HyperActive Software | http://www.hyperactivesw.com On May 25, 2018 9:26:21 PM Neville Smythe via use-livecode wrote: So now we have confirmed the “delete-card-with-unshared-textfield-leaves-unreferenced-text-which-compact-does-not-remove” bug (thanks to Brian and Richard) the question remains how to actually remove the garbage. As Brian says deleting the background groups will remove the residual text; you then have the task of replacing the groups. I have found that a simpler method: turning on sharedText, then turning it back off and replacing text also works. So, a recipe is 1. Copy the target stack, don’t work on the original 2. Go through the stack (and its substacks) to find background groups which have fields with sharedText false (this is an outer loop for each stack) 3. Find all cards which such groups 4. Save the text from every found field from every found card into an array 5. Turn on sharedText for each field 6. Visit each card and put empty into each found field (not sure if this step is necessary, but I think it is) 7. compact the stack 8. Turn off sharedText for each field 9. Visit each card to replace the individual text saved in the array 10.Save the stack. Sounds complex but the process took only a couple of seconds to clean up my 9MB stack with 325 cards + lots of substacks and lots of groups. It saved 1.5MB, not as much as I was hoping but still worthwhile. That’s 15% off the time Windows 10 users have wait for save to finish. This is not a workaround for the bug. It is a fix for affected stacks. Neville Smythe ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode
Re: how to clear residual garbage in a stack?
So now we have confirmed the “delete-card-with-unshared-textfield-leaves-unreferenced-text-which-compact-does-not-remove” bug (thanks to Brian and Richard) the question remains how to actually remove the garbage. As Brian says deleting the background groups will remove the residual text; you then have the task of replacing the groups. I have found that a simpler method: turning on sharedText, then turning it back off and replacing text also works. So, a recipe is 1. Copy the target stack, don’t work on the original 2. Go through the stack (and its substacks) to find background groups which have fields with sharedText false (this is an outer loop for each stack) 3. Find all cards which such groups 4. Save the text from every found field from every found card into an array 5. Turn on sharedText for each field 6. Visit each card and put empty into each found field (not sure if this step is necessary, but I think it is) 7. compact the stack 8. Turn off sharedText for each field 9. Visit each card to replace the individual text saved in the array 10.Save the stack. Sounds complex but the process took only a couple of seconds to clean up my 9MB stack with 325 cards + lots of substacks and lots of groups. It saved 1.5MB, not as much as I was hoping but still worthwhile. That’s 15% off the time Windows 10 users have wait for save to finish. This is not a workaround for the bug. It is a fix for affected stacks. Neville Smythe ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode
Re: how to clear residual garbage in a stack?
Brian Milby wrote: Ok, I can summarize what is happening here. When a card that contains a background with shared text disabled on a field, the data is indeed retained. The extra data survives saves and "compact this stack" as long as the shared group remains on any card. If you remove the group from the last card and then do a compact, then all of the extra space is recovered. When you place the group back on a card, the fields are now empty. As implied by the documentation, if you remove the group from the last card and do a save from the menu, the space is also reclaimed. Impressive sleuthing, Brian. Well done! -- Richard Gaskin Fourth World Systems Software Design and Development for the Desktop, Mobile, and the Web ambassa...@fourthworld.comhttp://www.FourthWorld.com ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode
Re: how to clear residual garbage in a stack?
Ok, I can summarize what is happening here. When a card that contains a background with shared text disabled on a field, the data is indeed retained. The extra data survives saves and "compact this stack" as long as the shared group remains on any card. If you remove the group from the last card and then do a compact, then all of the extra space is recovered. When you place the group back on a card, the fields are now empty. As implied by the documentation, if you remove the group from the last card and do a save from the menu, the space is also reclaimed. I'm inclined to call this a bug although I'm not sure where to place the blame. It could be in the "remove group" process (which would happen when deleting the card) - if the share text is false and the group is on more than one card (current + n>=1), the current text should be discarded (undo doesn't work, so that isn't a reason to keep the data around). It could be in the "compact this stack" process - extra data for background fields should be removed. My guess is that for performance, the second one will take the blame here though. Would be good for someone from LC to respond (Ali or Monte probably) with their thoughts. I'm not sure where that lives in the source code. It does make me curious about how text field data is stored for shared groups. On Wed, May 23, 2018 at 4:53 PM, Brian Milby wrote: > I’ll need to test further. I created a stack that ends up with a bunch of > groups and could delete them after. Have not tried putting text in a field > like you described though. > > On Wed, May 23, 2018 at 4:29 PM Neville Smythe < > neville.smy...@optusnet.com.au> wrote: > >> That’s if you use copy-paste to create the cards, because each group will >> have lost its original card. The groups are still available in the Place >> menu (and the browser?) and OK I suppose this is not a bug, but that there >> is no mention of this in the Dictionary for Delete is a serious deficiency. >> >> But if you use new to create the cards, there is only one background >> stack for the whole group, and it is still in card 1 after deleting the >> other cards. In the case that the background field does not share text, the >> bug is that the text for all the deleted cards is still in the stack, but >> with no way to reference it (I believe), and is not removed by compact. >> [Even in the copy case, I wonder of the shared text is still there… will >> have to check that, but not today…] >> >> >> On 24 May 2018, at 1:32 am, Brian Milby wrote: >> >> You have to go to the Object menu and choose Place Group to see all of >> the orphaned groups. All I did to replicate was put the big field inside a >> group and made it a background. I’m sure it would be possible to delete >> them, but I would need to look for the process. >> On Wed, May 23, 2018 at 7:12 AM Neville Smythe via use-livecode < >> use-livecode@lists.runrev.com> wrote: >> >>> Thanks Richard. >>> >>> Disregard my nonsense about the EOF at 28KB, I misread the BBEdit >>> output. The file did indeed have extra garbage after deleting all but 1 >>> card. >>> >>> The extraneous data actually consists of the text of the background >>> field from the deleted cards: “Lorem ipsum” occurs 2024 in the hex dump, >>> but Find and Replace can only find 18 instances, all in the single card >>> remaining. Very similar to the unplaced background groups left over as you >>> reported if the cards are created by copy-and-paste, but in this case I >>> created them with new card (with a field in a background group). This bug >>> is worse because there is no handle to the orphaned text in the browser or >>> menus! Note that I’m still thinking of this as a bug, rather than a problem >>> with my system :-) - it happens in Windows as well. >>> >>> Brian Milby tells me that he could not replicate the bug in LC 9 if the >>> cards are created with with copy-and-paste and no background groups. I’ll >>> do some more testing with LC9, with and without a background group and the >>> two methods of creating cards (and on Windows and Linux). My guess is that >>> compact stack is not working correctly in the case of deleting cards with >>> background fields in LC8, and this may or may not have been fixed in LC9. >>> >>> Brian does also confirm the slow saving in Windows 10, in fact his times >>> are much worse. Are there no longer any Windows 10 users of LC? (to be fair >>> my stacks compiled under LC 8.1.x do perform well under Windows 10 except >>> for the terrible saving problem). >>> >>> Neville Smythe >>> ___ >>> use-livecode mailing list >>> use-livecode@lists.runrev.com >>> Please visit this url to subscribe, unsubscribe and manage your >>> subscription preferences: >>> http://lists.runrev.com/mailman/listinfo/use-livecode >> >> >> ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage
Re: how to clear residual garbage in a stack?
You have to go to the Object menu and choose Place Group to see all of the orphaned groups. All I did to replicate was put the big field inside a group and made it a background. I’m sure it would be possible to delete them, but I would need to look for the process. On Wed, May 23, 2018 at 7:12 AM Neville Smythe via use-livecode < use-livecode@lists.runrev.com> wrote: > Thanks Richard. > > Disregard my nonsense about the EOF at 28KB, I misread the BBEdit output. > The file did indeed have extra garbage after deleting all but 1 card. > > The extraneous data actually consists of the text of the background field > from the deleted cards: “Lorem ipsum” occurs 2024 in the hex dump, but Find > and Replace can only find 18 instances, all in the single card remaining. > Very similar to the unplaced background groups left over as you reported if > the cards are created by copy-and-paste, but in this case I created them > with new card (with a field in a background group). This bug is worse > because there is no handle to the orphaned text in the browser or menus! > Note that I’m still thinking of this as a bug, rather than a problem with > my system :-) - it happens in Windows as well. > > Brian Milby tells me that he could not replicate the bug in LC 9 if the > cards are created with with copy-and-paste and no background groups. I’ll > do some more testing with LC9, with and without a background group and the > two methods of creating cards (and on Windows and Linux). My guess is that > compact stack is not working correctly in the case of deleting cards with > background fields in LC8, and this may or may not have been fixed in LC9. > > Brian does also confirm the slow saving in Windows 10, in fact his times > are much worse. Are there no longer any Windows 10 users of LC? (to be fair > my stacks compiled under LC 8.1.x do perform well under Windows 10 except > for the terrible saving problem). > > Neville Smythe > ___ > use-livecode mailing list > use-livecode@lists.runrev.com > Please visit this url to subscribe, unsubscribe and manage your > subscription preferences: > http://lists.runrev.com/mailman/listinfo/use-livecode ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode
Re: how to clear residual garbage in a stack?
Thanks Richard. Disregard my nonsense about the EOF at 28KB, I misread the BBEdit output. The file did indeed have extra garbage after deleting all but 1 card. The extraneous data actually consists of the text of the background field from the deleted cards: “Lorem ipsum” occurs 2024 in the hex dump, but Find and Replace can only find 18 instances, all in the single card remaining. Very similar to the unplaced background groups left over as you reported if the cards are created by copy-and-paste, but in this case I created them with new card (with a field in a background group). This bug is worse because there is no handle to the orphaned text in the browser or menus! Note that I’m still thinking of this as a bug, rather than a problem with my system :-) - it happens in Windows as well. Brian Milby tells me that he could not replicate the bug in LC 9 if the cards are created with with copy-and-paste and no background groups. I’ll do some more testing with LC9, with and without a background group and the two methods of creating cards (and on Windows and Linux). My guess is that compact stack is not working correctly in the case of deleting cards with background fields in LC8, and this may or may not have been fixed in LC9. Brian does also confirm the slow saving in Windows 10, in fact his times are much worse. Are there no longer any Windows 10 users of LC? (to be fair my stacks compiled under LC 8.1.x do perform well under Windows 10 except for the terrible saving problem). Neville Smythe ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode
Re: how to clear residual garbage in a stack?
I can reproduce the time issue (100 cards - 8.5MB file - 71s), but not the space issue. Have a button copy a card and paste it 100 times and then “save this stack”. Another button deletes the last card until there is only one left and then does the same. File size goes to 85kb in 673ms. This is on Win10, LC9 Business. Stack size largely due to a single field with a bunch of text in it. On Tue, May 22, 2018 at 12:18 PM Richard Gaskin via use-livecode < use-livecode@lists.runrev.com> wrote: > Neville Smythe wrote: > > > Actually I just did a hex dump of the 8MB file saved from the 1 card > > stack; and lo! it has an EOF at about the 28KB position, at which > > point BBEdit stops dumping the file; up to that point it appears to be > > the correct data for a 1 card stack, no extraneous garbage. But the > > system reports it as an 8MB file. Doesn’t that mean the file was not > > closed correctly by the save stack command, or that compact stack is > > not working correctly to reduce the internal memory and save stack is > > blindly writing bytes beyond the EOF, so either way it is a bug? > > Possibly. Or it may be an error in the disk driver or a corruption in > the file system. > > EOF is not normally written to a file (at least not since the old CP/M > days). My understanding is that modern file systems allow the OS to > know where the file ends by the length record in the inode; EOF is sent > only as a flag by the OS during read operations. > > I wonder if BBEdit's issue is that it's reading the file as text rather > than binary, in which case it may be interpreting some binary elements > as the OS-defined EOF constant. > > If you read the stack file as binary in LC what is the length of the data? > > -- > Richard Gaskin > Fourth World Systems > Software Design and Development for the Desktop, Mobile, and the Web > > ambassa...@fourthworld.comhttp://www.FourthWorld.com > > ___ > use-livecode mailing list > use-livecode@lists.runrev.com > Please visit this url to subscribe, unsubscribe and manage your > subscription preferences: > http://lists.runrev.com/mailman/listinfo/use-livecode ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode
Re: how to clear residual garbage in a stack?
Neville Smythe wrote: > Actually I just did a hex dump of the 8MB file saved from the 1 card > stack; and lo! it has an EOF at about the 28KB position, at which > point BBEdit stops dumping the file; up to that point it appears to be > the correct data for a 1 card stack, no extraneous garbage. But the > system reports it as an 8MB file. Doesn’t that mean the file was not > closed correctly by the save stack command, or that compact stack is > not working correctly to reduce the internal memory and save stack is > blindly writing bytes beyond the EOF, so either way it is a bug? Possibly. Or it may be an error in the disk driver or a corruption in the file system. EOF is not normally written to a file (at least not since the old CP/M days). My understanding is that modern file systems allow the OS to know where the file ends by the length record in the inode; EOF is sent only as a flag by the OS during read operations. I wonder if BBEdit's issue is that it's reading the file as text rather than binary, in which case it may be interpreting some binary elements as the OS-defined EOF constant. If you read the stack file as binary in LC what is the length of the data? -- Richard Gaskin Fourth World Systems Software Design and Development for the Desktop, Mobile, and the Web ambassa...@fourthworld.comhttp://www.FourthWorld.com ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode
Re: how to clear residual garbage in a stack?
Did you attempt to do a “save as” on the stack? From what you described it sounds like LC wrote the file in place but did not truncate what was there after the EOF so extra data was there on the end. Need to try this on Mac/Linux to see if it does the same thing. On Tue, May 22, 2018 at 9:14 AM Neville Smythe via use-livecode < use-livecode@lists.runrev.com> wrote: > Hmm, excellent suggestions from Phil and Richard, but evidently something > else is going on. > > > Use "compact stack" to get rid of space formerly used by now-deleted > > objects. > > compact stack did not recover the 8 MB after deleting all but the first > card from my test stack to reduce it to what should be a 28KB file. > > > Does your card contain a group with its backgroundBehavior set to true? > > > > If so, and if the cards are created with copy and paste, I believe you > > may find a large number of unplaced groups in the stack. > > The extra cards were created using create, not copy-and-paste, there is > only 1 group in the stack before and after deleting extra cards. > > Actually I just did a hex dump of the 8MB file saved from the 1 card > stack; and lo! it has an EOF at about the 28KB position, at which point > BBEdit stops dumping the file; up to that point it appears to be the > correct data for a 1 card stack, no extraneous garbage. But the system > reports it as an 8MB file. Doesn’t that mean the file was not closed > correctly by the save stack command, or that compact stack is not working > correctly to reduce the internal memory and save stack is blindly writing > bytes beyond the EOF, so either way it is a bug? > > All of which is peripheral to my original aim of finding out what is going > on with LC and Windows 10 > > > Neville Smythe > ___ > use-livecode mailing list > use-livecode@lists.runrev.com > Please visit this url to subscribe, unsubscribe and manage your > subscription preferences: > http://lists.runrev.com/mailman/listinfo/use-livecode ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode
Re: how to clear residual garbage in a stack?
Hmm, excellent suggestions from Phil and Richard, but evidently something else is going on. > Use "compact stack" to get rid of space formerly used by now-deleted > objects. compact stack did not recover the 8 MB after deleting all but the first card from my test stack to reduce it to what should be a 28KB file. > Does your card contain a group with its backgroundBehavior set to true? > > If so, and if the cards are created with copy and paste, I believe you > may find a large number of unplaced groups in the stack. The extra cards were created using create, not copy-and-paste, there is only 1 group in the stack before and after deleting extra cards. Actually I just did a hex dump of the 8MB file saved from the 1 card stack; and lo! it has an EOF at about the 28KB position, at which point BBEdit stops dumping the file; up to that point it appears to be the correct data for a 1 card stack, no extraneous garbage. But the system reports it as an 8MB file. Doesn’t that mean the file was not closed correctly by the save stack command, or that compact stack is not working correctly to reduce the internal memory and save stack is blindly writing bytes beyond the EOF, so either way it is a bug? All of which is peripheral to my original aim of finding out what is going on with LC and Windows 10 Neville Smythe ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode
Re: how to clear residual garbage in a stack?
Thanks Phil, I knew there had to be a command. Of all the synonyms I tried I got close with compress ;-) Turns out a stack of mine used by national organisation for 20 years had bloated from 3MB to 9MB. > Use "compact stack" to get rid of space formerly used by now-deleted > objects. Neville Smythe ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode
Re: how to clear residual garbage in a stack?
Does your card contain a group with its backgroundBehavior set to true? If so, and if the cards are created with copy and paste, I believe you may find a large number of unplaced groups in the stack. In the IDE menus see Object -> Place Group If that submenu is long, you've found your culprit. Rather than copy and paste, when creating new cards with a common background group can be done with the clone or create commands the cards will all share a common group, reducing size even before running the delete loop. -- Richard Gaskin Fourth World Systems Neville Smythe wrote: > I am constructing a large stack as a test for a bug report (to report > the slow saving of large stacks for a standalone Windows 10 app under > LC 8.x). I start with a stack with a single card which saves as a 28 > KB file. I then use a script to create 299 copies of card 1. The > resulting stack saves to a 8.2 MB file. As expected. > > I then deleted all but the first card, and re-ran the script so it > again had 300 cards. This saves to a 16.4 MB file! Repeating the > delete-and-create gives a 24 MB file. Evidently the data for the > deleted cards has not been removed. I can think of efficiency reasons > why this might be useful in the IDE, but there must/should be a way to > reduce the file size to a minimum, otherwise stacks which continually > delete and create data will just grow in a very user-unfriendly way. > > I can’t find any command such as clean or purge or free or compress or > release garbage which will reduce the file size to its actual > requirements. I have a vague memory of this being raised some years > ago but I can’t find the reference. Surely there must be a built-in > command? Because the only way I can see to reduce the size would be to > create a new stack and copy over all cards and scripts and substacks > before saving, which is ridiculous. > > BTW The bug: The 8.2MB stack saves on a Mac in 0.27 seconds. On > Windows 10 saving time is 6.4 seconds. Other apps on Windows save an 8 > MB file in under a second so the problem is not with the Windows OS or > antivirus virus software or the particular machine. Possibly this bug > is related to the reported unusablility of LC 9 IDE on Windows 10, of > which I have heard no more since it was raised in this forum a few > weeks ago, although my context is a standalone compiled under LC 8.x. > > Neville Smythe ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode
Re: how to clear residual garbage in a stack?
Neville, Use "compact stack" to get rid of space formerly used by now-deleted objects. See the docs. Phil Davis On 5/21/18 5:42 PM, Neville Smythe via use-livecode wrote: I am constructing a large stack as a test for a bug report (to report the slow saving of large stacks for a standalone Windows 10 app under LC 8.x). I start with a stack with a single card which saves as a 28 KB file. I then use a script to create 299 copies of card 1. The resulting stack saves to a 8.2 MB file. As expected. I then deleted all but the first card, and re-ran the script so it again had 300 cards. This saves to a 16.4 MB file! Repeating the delete-and-create gives a 24 MB file. Evidently the data for the deleted cards has not been removed. I can think of efficiency reasons why this might be useful in the IDE, but there must/should be a way to reduce the file size to a minimum, otherwise stacks which continually delete and create data will just grow in a very user-unfriendly way. I can’t find any command such as clean or purge or free or compress or release garbage which will reduce the file size to its actual requirements. I have a vague memory of this being raised some years ago but I can’t find the reference. Surely there must be a built-in command? Because the only way I can see to reduce the size would be to create a new stack and copy over all cards and scripts and substacks before saving, which is ridiculous. BTW The bug: The 8.2MB stack saves on a Mac in 0.27 seconds. On Windows 10 saving time is 6.4 seconds. Other apps on Windows save an 8 MB file in under a second so the problem is not with the Windows OS or antivirus virus software or the particular machine. Possibly this bug is related to the reported unusablility of LC 9 IDE on Windows 10, of which I have heard no more since it was raised in this forum a few weeks ago, although my context is a standalone compiled under LC 8.x. Neville Smythe ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode -- Phil Davis ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode
