Re: Sorting text is *VERY* slow in LC9 on Windows (Re: Accumulating text is *VERY* slow in LC9 on Windows)
Thanks Mark, now I get it! Ben On 12/09/2021 12:09, Mark Waddingham via use-livecode wrote: On 2021-09-10 19:10, Ben Rubinstein via use-livecode wrote: The other was The only caveat is that it might cause apps mutating lots of medium->large strings concurrently to take up more memory in general... > ... (and any issue arising from that could be resolved by moving to the 64-bit windows engine). I have been doing my tests on the 64-bit Windows engine. What am I missing? The change I am making to the buffer extension rules means that mutable strings (i.e. those which the last action was modifying them, rather than copying them / fetching them) will take up more space than they did before to allow for possible extension (previously the maximum 'just in case' space which was allocated was 63 bytes). This means that applications may take up more memory as a result (if they happen to have lots of large mutable strings in play at any one time). 32-bit windows engines have a maximum address space of 3Gb - so if an app was already approaching that limit, this might tip them over to start causing out of memory errors. However, in this case, switching to use the 64-bit windows engine would resolve the problem as 64-bit address space is substantially larger. Warmest Regards, Mark. ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode
Re: Sorting text is *VERY* slow in LC9 on Windows (Re: Accumulating text is *VERY* slow in LC9 on Windows)
On 2021-09-10 19:10, Ben Rubinstein via use-livecode wrote: The other was The only caveat is that it might cause apps mutating lots of medium->large strings concurrently to take up more memory in general... > ... (and any issue arising from that could be resolved by moving to the 64-bit windows engine). I have been doing my tests on the 64-bit Windows engine. What am I missing? The change I am making to the buffer extension rules means that mutable strings (i.e. those which the last action was modifying them, rather than copying them / fetching them) will take up more space than they did before to allow for possible extension (previously the maximum 'just in case' space which was allocated was 63 bytes). This means that applications may take up more memory as a result (if they happen to have lots of large mutable strings in play at any one time). 32-bit windows engines have a maximum address space of 3Gb - so if an app was already approaching that limit, this might tip them over to start causing out of memory errors. However, in this case, switching to use the 64-bit windows engine would resolve the problem as 64-bit address space is substantially larger. Warmest Regards, Mark. -- Mark Waddingham ~ m...@livecode.com ~ http://www.livecode.com/ LiveCode: Everyone can create apps ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode
Re: Sorting text is *VERY* slow in LC9 on Windows (Re: Accumulating text is *VERY* slow in LC9 on Windows)
Ah yes, the cheese thing. I so wish I had saved that thread, but I may have needed an extra backup drive to do it. So listen, and ye shall hear. Long, long ago in the olden days, before there was a forum, before 64-bit apps, back when the QCC reports numbered in the lower hundreds, someone on the mailing list mentioned a preference for a cheese. I don't recall who or what, because at the time it seemed so innocuous. Someone else agreed, someone else disagreed, some did both. Discussions went on about which country produced the best cheese, opinions which were disputed by others who felt their own country should be nominated or at least take second place. Others had opinions about countries who did or did not produce cheese. Different varieties of cheese were analyzed, and opined upon. The same type of cheese was compared between one country and another. Suggestions were made to visit a country in order to try a cheese, which would presumably convince the doubters. It was implied that one's own country had good enough cheese, if not the best. This went on for a very long time. Of course, eventually the uninvolved, or those who were indifferent to cheese, or perhaps those who did not like cheese at all, asked the participants to desist and stay on topic. But by then the thread had taken on a life of its own and continued rolling on, gathering momentum like a stone thrown downhill and perhaps subsequently tumbling down river. Being a polite mailing list, no one actually said, "shut up" but at times you could read between the lines. Other readers suffered in stoic silence and made good use of their Delete key. There were occasional mutterings about slightly changing subject lines in posts, which prevented filtering out cheese curds from on-topic posts. The conversation continued for months. Or if it didn't, it seemed like it. Finally List Mom appeared with a gentle but firm pronouncement that in addition to politics, religion, and other such topics, Cheese was now banned on the list. Some were resigned, others sighed in relief. Occasionally you will still see oblique references to the dairy product but in general we speak of it only by inferred reference. I trust I will not be banned for being so explicit, but sometimes one must speak plainly. -- Jacqueline Landman Gay | jac...@hyperactivesw.com HyperActive Software | http://www.hyperactivesw.com On September 11, 2021 7:27:13 PM Lagi Pittas via use-livecode wrote: I thought Mark was misremembering Emmental when he said Edam (they do start with an E) but I believe Venezuelan Beaver Cheeses is probably a better analogy as it has many holes caused by the beavers' sharp incisors. and is a bit of a mystery (very difficult to acquire) as is the workings of windows buffer allocation . Can we say Cheese? https://www.youtube.com/watch?v=Hz1JWzyvv8A=3s lagi p.s Jacques - when did this Cheese thing start - it is certainly before my time. On Fri, 10 Sept 2021 at 19:16, J. Landman Gay via use-livecode < use-livecode@lists.runrev.com> wrote: You're talking about cheese. I'm telling Mom. Nyah. -- Jacqueline Landman Gay | jac...@hyperactivesw.com HyperActive Software | http://www.hyperactivesw.com On September 10, 2021 8:42:48 AM Trevor DeVore via use-livecode wrote: On Fri, Sep 10, 2021 at 8:15 AM Mark Waddingham via use-livecode < use-livecode@lists.runrev.com> wrote: On 2021-09-10 14:06, Mark Waddingham via use-livecode wrote: The windows heap is much more prudent than UNIXy counterparts it would seem - where UNIX heaps will happily leave plenty of free space (which the heaps know about and thus can re-use), Windows appears to avoid that like the plague (which I'm sure is the case for lots of historical reasons and backwards compatibility). [ To give a very rough analogy, the map of used space in a heap on windows is like a block of cheddar; whereas on UNIXy systems it will be like a block of edam ]. I of course meant 'Swiss', not 'Edam'! Thanks for the clarification. I feel like I have a decent understanding of cheeses, but I couldn’t figure out how cheddar and edam were different in this analogy :-) -- Trevor DeVore ScreenSteps ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode -- KIndest Regards Lagi ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode
Re: Sorting text is *VERY* slow in LC9 on Windows (Re: Accumulating text is *VERY* slow in LC9 on Windows)
I thought Mark was misremembering Emmental when he said Edam (they do start with an E) but I believe Venezuelan Beaver Cheeses is probably a better analogy as it has many holes caused by the beavers' sharp incisors. and is a bit of a mystery (very difficult to acquire) as is the workings of windows buffer allocation . Can we say Cheese? https://www.youtube.com/watch?v=Hz1JWzyvv8A=3s lagi p.s Jacques - when did this Cheese thing start - it is certainly before my time. On Fri, 10 Sept 2021 at 19:16, J. Landman Gay via use-livecode < use-livecode@lists.runrev.com> wrote: > You're talking about cheese. I'm telling Mom. Nyah. > > -- > Jacqueline Landman Gay | jac...@hyperactivesw.com > HyperActive Software | http://www.hyperactivesw.com > On September 10, 2021 8:42:48 AM Trevor DeVore via use-livecode > wrote: > > > On Fri, Sep 10, 2021 at 8:15 AM Mark Waddingham via use-livecode < > > use-livecode@lists.runrev.com> wrote: > > > >> On 2021-09-10 14:06, Mark Waddingham via use-livecode wrote: > >> > The windows heap is much more prudent than UNIXy counterparts it would > >> > seem - where UNIX heaps will happily leave plenty of free space (which > >> > the heaps know about and thus can re-use), Windows appears to avoid > >> > that like the plague (which I'm sure is the case for lots of > >> > historical reasons and backwards compatibility). [ To give a very > >> > rough analogy, the map of used space in a heap on windows is like a > >> > block of cheddar; whereas on UNIXy systems it will be like a block of > >> > edam ]. > >> > >> I of course meant 'Swiss', not 'Edam'! > > > > > > Thanks for the clarification. I feel like I have a decent understanding > of > > cheeses, but I couldn’t figure out how cheddar and edam were different in > > this analogy :-) > > > > -- > > Trevor DeVore > > ScreenSteps > > > >> > > ___ > > use-livecode mailing list > > use-livecode@lists.runrev.com > > Please visit this url to subscribe, unsubscribe and manage your > > subscription preferences: > > http://lists.runrev.com/mailman/listinfo/use-livecode > > > > > ___ > use-livecode mailing list > use-livecode@lists.runrev.com > Please visit this url to subscribe, unsubscribe and manage your > subscription preferences: > http://lists.runrev.com/mailman/listinfo/use-livecode > -- KIndest Regards Lagi ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode
Re: Sorting text is *VERY* slow in LC9 on Windows (Re: Accumulating text is *VERY* slow in LC9 on Windows)
Yes, it did mention cheese... but it was entirely on topic and about LiveCode. (How could Mom not approve.) -- Scott Morrow Elementary Software (Now with 20% less chalk dust!) web https://elementarysoftware.com/ email sc...@elementarysoftware.com booth1-360-734-4701 -- > On Sep 10, 2021, at 11:15 AM, J. Landman Gay via use-livecode > wrote: > > You're talking about cheese. I'm telling Mom. Nyah. > > -- > Jacqueline Landman Gay | jac...@hyperactivesw.com > HyperActive Software | http://www.hyperactivesw.com > On September 10, 2021 8:42:48 AM Trevor DeVore via use-livecode > wrote: > >> On Fri, Sep 10, 2021 at 8:15 AM Mark Waddingham via use-livecode < >> use-livecode@lists.runrev.com> wrote: >> >>> On 2021-09-10 14:06, Mark Waddingham via use-livecode wrote: >>> > The windows heap is much more prudent than UNIXy counterparts it would >>> > seem - where UNIX heaps will happily leave plenty of free space (which >>> > the heaps know about and thus can re-use), Windows appears to avoid >>> > that like the plague (which I'm sure is the case for lots of >>> > historical reasons and backwards compatibility). [ To give a very >>> > rough analogy, the map of used space in a heap on windows is like a >>> > block of cheddar; whereas on UNIXy systems it will be like a block of >>> > edam ]. >>> >>> I of course meant 'Swiss', not 'Edam'! >> >> >> Thanks for the clarification. I feel like I have a decent understanding of >> cheeses, but I couldn’t figure out how cheddar and edam were different in >> this analogy :-) >> >> -- >> Trevor DeVore >> ScreenSteps >> >>> >> ___ >> use-livecode mailing list >> use-livecode@lists.runrev.com >> Please visit this url to subscribe, unsubscribe and manage your subscription >> preferences: >> http://lists.runrev.com/mailman/listinfo/use-livecode ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode
Re: Sorting text is *VERY* slow in LC9 on Windows (Re: Accumulating text is *VERY* slow in LC9 on Windows)
I’m going to propound on the politics of holey cheese makers in a moment! Sent from my iPhone > On Sep 10, 2021, at 11:16, J. Landman Gay via use-livecode > wrote: > > You're talking about cheese. I'm telling Mom. Nyah. > > -- > Jacqueline Landman Gay | jac...@hyperactivesw.com > HyperActive Software | http://www.hyperactivesw.com >> On September 10, 2021 8:42:48 AM Trevor DeVore via use-livecode >> wrote: >> >> On Fri, Sep 10, 2021 at 8:15 AM Mark Waddingham via use-livecode < >> use-livecode@lists.runrev.com> wrote: >> On 2021-09-10 14:06, Mark Waddingham via use-livecode wrote: >>> > The windows heap is much more prudent than UNIXy counterparts it would >>> > seem - where UNIX heaps will happily leave plenty of free space (which >>> > the heaps know about and thus can re-use), Windows appears to avoid >>> > that like the plague (which I'm sure is the case for lots of >>> > historical reasons and backwards compatibility). [ To give a very >>> > rough analogy, the map of used space in a heap on windows is like a >>> > block of cheddar; whereas on UNIXy systems it will be like a block of >>> > edam ]. >>> >>> I of course meant 'Swiss', not 'Edam'! >> >> >> Thanks for the clarification. I feel like I have a decent understanding of >> cheeses, but I couldn’t figure out how cheddar and edam were different in >> this analogy :-) >> >> -- >> Trevor DeVore >> ScreenSteps >> >>> >> ___ >> use-livecode mailing list >> use-livecode@lists.runrev.com >> Please visit this url to subscribe, unsubscribe and manage your subscription >> preferences: >> http://lists.runrev.com/mailman/listinfo/use-livecode > > > > > ___ > use-livecode mailing list > use-livecode@lists.runrev.com > Please visit this url to subscribe, unsubscribe and manage your subscription > preferences: > http://lists.runrev.com/mailman/listinfo/use-livecode ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode
Re: Sorting text is *VERY* slow in LC9 on Windows (Re: Accumulating text is *VERY* slow in LC9 on Windows)
You're talking about cheese. I'm telling Mom. Nyah. -- Jacqueline Landman Gay | jac...@hyperactivesw.com HyperActive Software | http://www.hyperactivesw.com On September 10, 2021 8:42:48 AM Trevor DeVore via use-livecode wrote: On Fri, Sep 10, 2021 at 8:15 AM Mark Waddingham via use-livecode < use-livecode@lists.runrev.com> wrote: On 2021-09-10 14:06, Mark Waddingham via use-livecode wrote: > The windows heap is much more prudent than UNIXy counterparts it would > seem - where UNIX heaps will happily leave plenty of free space (which > the heaps know about and thus can re-use), Windows appears to avoid > that like the plague (which I'm sure is the case for lots of > historical reasons and backwards compatibility). [ To give a very > rough analogy, the map of used space in a heap on windows is like a > block of cheddar; whereas on UNIXy systems it will be like a block of > edam ]. I of course meant 'Swiss', not 'Edam'! Thanks for the clarification. I feel like I have a decent understanding of cheeses, but I couldn’t figure out how cheddar and edam were different in this analogy :-) -- Trevor DeVore ScreenSteps ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode
Re: Sorting text is *VERY* slow in LC9 on Windows (Re: Accumulating text is *VERY* slow in LC9 on Windows)
Hi Mark, Thank you for this, very promising. Only two things puzzled me. One you've already addressed when you corrected the specified cheese. The other was The only caveat is that it might cause apps mutating lots of medium->large strings concurrently to take up more memory in general... > ... (and any issue arising from that could be resolved by moving to the 64-bit windows engine). I have been doing my tests on the 64-bit Windows engine. What am I missing? TIA (and T as always for all your work), Ben On 10/09/2021 14:06, Mark Waddingham via use-livecode wrote: On 2021-09-02 18:38, Mark Waddingham via use-livecode wrote: We will endeavour to fix for 9.6.5-rc-1 (due 'real soon now'!). So I have been prodding the windows 'accumulating large strings' speed problem this week (in amongst other things). It is definitely memory allocation causing the problem. The rule for extending a string buffer when inserting/appending more text is pretty much the same as 6.7 (the amount of 'extra space' it asks for each time is a little less - but changing it to the constant used in 6.7 makes no difference). What is different between 6.7 and 7.0+ is the memory allocation patterns of the engine itself and I think it is this which is invariably causing a whole new buffer having to be allocated when appending to large strings, rather than the existing one being extended. The windows heap is much more prudent than UNIXy counterparts it would seem - where UNIX heaps will happily leave plenty of free space (which the heaps know about and thus can re-use), Windows appears to avoid that like the plague (which I'm sure is the case for lots of historical reasons and backwards compatibility). [ To give a very rough analogy, the map of used space in a heap on windows is like a block of cheddar; whereas on UNIXy systems it will be like a block of edam ]. So anyway, long story short, making a string buffer grow in proportion to its size appears to mitigate the problem - I still need to finesse things a bit though to ensure that memory starvation doesn't occur when you have a couple of large mutating strings but all being well we should be able to get something together for 9.6.5-rc-1. The only caveat is that it might cause apps mutating lots of medium->large strings concurrently to take up more memory in general (i.e. in that extra unused 'just in case' space at the end of each buffer) but no more than the same apps would on any other (UNIX-based) platform (and any issue arising from that could be resolved by moving to the 64-bit windows engine). For reference, the bug about the string accumulation problem as posed by Ben is https://quality.livecode.com/show_bug.cgi?id=23319 (this will also fix the sort speed too - as the final step the sort command performs internally is re-accumulating the string in the new order). Warmest Regards, Mark. P.S. I cannot really say whether this will help with the various 'IDE can be like treacle' on Windows problems - but it definitely can't hurt! ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode
Re: Sorting text is *VERY* slow in LC9 on Windows (Re: Accumulating text is *VERY* slow in LC9 on Windows)
Cheddar, Edam and Swiss! Very daring references on this list. ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode
Re: Sorting text is *VERY* slow in LC9 on Windows (Re: Accumulating text is *VERY* slow in LC9 on Windows)
On Fri, Sep 10, 2021 at 8:15 AM Mark Waddingham via use-livecode < use-livecode@lists.runrev.com> wrote: > On 2021-09-10 14:06, Mark Waddingham via use-livecode wrote: > > The windows heap is much more prudent than UNIXy counterparts it would > > seem - where UNIX heaps will happily leave plenty of free space (which > > the heaps know about and thus can re-use), Windows appears to avoid > > that like the plague (which I'm sure is the case for lots of > > historical reasons and backwards compatibility). [ To give a very > > rough analogy, the map of used space in a heap on windows is like a > > block of cheddar; whereas on UNIXy systems it will be like a block of > > edam ]. > > I of course meant 'Swiss', not 'Edam'! Thanks for the clarification. I feel like I have a decent understanding of cheeses, but I couldn’t figure out how cheddar and edam were different in this analogy :-) -- Trevor DeVore ScreenSteps > ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode
Re: Sorting text is *VERY* slow in LC9 on Windows (Re: Accumulating text is *VERY* slow in LC9 on Windows)
On 2021-09-10 14:06, Mark Waddingham via use-livecode wrote: The windows heap is much more prudent than UNIXy counterparts it would seem - where UNIX heaps will happily leave plenty of free space (which the heaps know about and thus can re-use), Windows appears to avoid that like the plague (which I'm sure is the case for lots of historical reasons and backwards compatibility). [ To give a very rough analogy, the map of used space in a heap on windows is like a block of cheddar; whereas on UNIXy systems it will be like a block of edam ]. I of course meant 'Swiss', not 'Edam'! -- Mark Waddingham ~ m...@livecode.com ~ http://www.livecode.com/ LiveCode: Everyone can create apps ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode
Re: Sorting text is *VERY* slow in LC9 on Windows (Re: Accumulating text is *VERY* slow in LC9 on Windows)
On 2021-09-02 18:38, Mark Waddingham via use-livecode wrote: We will endeavour to fix for 9.6.5-rc-1 (due 'real soon now'!). So I have been prodding the windows 'accumulating large strings' speed problem this week (in amongst other things). It is definitely memory allocation causing the problem. The rule for extending a string buffer when inserting/appending more text is pretty much the same as 6.7 (the amount of 'extra space' it asks for each time is a little less - but changing it to the constant used in 6.7 makes no difference). What is different between 6.7 and 7.0+ is the memory allocation patterns of the engine itself and I think it is this which is invariably causing a whole new buffer having to be allocated when appending to large strings, rather than the existing one being extended. The windows heap is much more prudent than UNIXy counterparts it would seem - where UNIX heaps will happily leave plenty of free space (which the heaps know about and thus can re-use), Windows appears to avoid that like the plague (which I'm sure is the case for lots of historical reasons and backwards compatibility). [ To give a very rough analogy, the map of used space in a heap on windows is like a block of cheddar; whereas on UNIXy systems it will be like a block of edam ]. So anyway, long story short, making a string buffer grow in proportion to its size appears to mitigate the problem - I still need to finesse things a bit though to ensure that memory starvation doesn't occur when you have a couple of large mutating strings but all being well we should be able to get something together for 9.6.5-rc-1. The only caveat is that it might cause apps mutating lots of medium->large strings concurrently to take up more memory in general (i.e. in that extra unused 'just in case' space at the end of each buffer) but no more than the same apps would on any other (UNIX-based) platform (and any issue arising from that could be resolved by moving to the 64-bit windows engine). For reference, the bug about the string accumulation problem as posed by Ben is https://quality.livecode.com/show_bug.cgi?id=23319 (this will also fix the sort speed too - as the final step the sort command performs internally is re-accumulating the string in the new order). Warmest Regards, Mark. P.S. I cannot really say whether this will help with the various 'IDE can be like treacle' on Windows problems - but it definitely can't hurt! -- Mark Waddingham ~ m...@livecode.com ~ http://www.livecode.com/ LiveCode: Everyone can create apps ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode
Re: Sorting text is *VERY* slow in LC9 on Windows (Re: Accumulating text is *VERY* slow in LC9 on Windows)
I'm very much hoping that Mark W might magically fix this in 9.6.5. But in the meantime FWIW, the place where this was really hurting (in a script that took 8 minutes under LC6, was taking 8 hours under LC9, but I've gradually tamed it down to under an hour by buffering the large accumulations) was a single sort command, on 70 MB of data in approx 223,000 lines. I've replaced this line: sort lines of tNewTable by item iSortCol of each which took 1 second on Mac, 2063 seconds (i.e. 34 minutes) on Windows, with a call to this command command sortLinesByTabbedColumn @tTable, iSortCol local aTable, tSortTable, iARcounter, tARbuffer, tRow, k -- load table into an array for fast access by line number put tTable into aTable split aTable using return -- compile index of just the column to sort on, and line number set the itemDelimiter to tab repeat for each key k in aTable get (item iSortCol of aTable[k]) && k appendRow it, iARcounter, tARbuffer, tSortTable end repeat put tARbuffer after tSortTable -- sort it sort lines of tSortTable -- rebuild table out of array, in sorted order put empty into tARbuffer put empty into tTable repeat for each line tRow in tSortTable put last word of tRow into k appendRow aTable[k], iARcounter, tARbuffer, tTable end repeat put tARbuffer after tTable end sortLinesByTabbedColumn which takes 25 seconds on Windows (to my surprise, most of that time was in the final 'rebuild' loop). On 02/09/2021 23:53, Bob Sneidar via use-livecode wrote: I am going to say no, because you still have to traverse the file once to get it into sqLite, then do the sort, then write out the file when done. I might be mistaken, the subsequent SQL sort may make up for lost time. Using a memory SQL really shines when you need to make multiple passes at the data using different queries. One pass may not impress you much. For instance, I have a File Management module built into my application. A file can belong to a customer, and also to a site, and also to a device. Like so: custid siteid deviceidfilepath 123 disk/folder/file1 456 098 disk/folder/file2 789 765 432 disk/folder/file3 Note all have a custid, some have a siteid as well, and some also have a deviceid. So rather than query mySQL for the files for each site or device as I select them, I instead, upon selecting a customer, query mySQL for ALL the file records for that customer, (which of course contain the file records for all the sites and devices), then store that in a memory database. Then when a different site or device belonging to that customer is selected, I query the memory database for those belonging to that site, or that device in those modules respectively. The performance enhancement is significant. Another way I apply this is to get the objects on a card passing a list of properties I'm interested in, then store the data in a memory database. I can then query for objects with certain properties without having to iterate through all the objects on a card in a repeat loop. For instance, the farthest left, top, right and bottom object whose visible is true in 4 memory db queries, giving me the total rect of all the visible objects without grouping/ungrouping and the hell that can ensue. Bob S On Sep 2, 2021, at 11:22 , Bernard Devlin via use-livecode wrote: Whilst waiting for a fix, would a temporary solution be to use sqlite to create an in-memory database and let sqlite do the sorting for you? Regards, Bernard. On Mon, Aug 30, 2021 at 8:23 PM Ben Rubinstein via use-livecode < use-livecode@lists.runrev.com> wrote: Thanks to Mark Waddingham's advice about using a buffer var when accumulating a large text variabel in stages, I've now got a script that took 8 hours under LC9, and (8 minutes under LC6) down by stages to just under 1 hour under LC9. However I have some remaining issues not amenable to this approach; of which the most significant relates to the sort command. In all cases it seems to take much longer under LC9 than it did under LC6; although the factor is quite variable. The most dramatic is one instance, in which this statement: sort lines of tNewTable by item iSortCol of each takes 35 minutes to execute. `tNewTable` is a variable consisting of some 223,000 lines of text; approx 70MB. The exact same statement with the same data on the same computer in LC6 takes just 1 second. Has anyone else noticed something of this sort? As I said, the effect varies: e.g. 54 seconds versus 1 second; 22 seconds versus 1 second. So it may not be so noticeable in all cases.
Re: Sorting text is *VERY* slow in LC9 on Windows (Re: Accumulating text is *VERY* slow in LC9 on Windows)
I am going to say no, because you still have to traverse the file once to get it into sqLite, then do the sort, then write out the file when done. I might be mistaken, the subsequent SQL sort may make up for lost time. Using a memory SQL really shines when you need to make multiple passes at the data using different queries. One pass may not impress you much. For instance, I have a File Management module built into my application. A file can belong to a customer, and also to a site, and also to a device. Like so: custid siteid deviceidfilepath 123 disk/folder/file1 456 098 disk/folder/file2 789 765 432 disk/folder/file3 Note all have a custid, some have a siteid as well, and some also have a deviceid. So rather than query mySQL for the files for each site or device as I select them, I instead, upon selecting a customer, query mySQL for ALL the file records for that customer, (which of course contain the file records for all the sites and devices), then store that in a memory database. Then when a different site or device belonging to that customer is selected, I query the memory database for those belonging to that site, or that device in those modules respectively. The performance enhancement is significant. Another way I apply this is to get the objects on a card passing a list of properties I'm interested in, then store the data in a memory database. I can then query for objects with certain properties without having to iterate through all the objects on a card in a repeat loop. For instance, the farthest left, top, right and bottom object whose visible is true in 4 memory db queries, giving me the total rect of all the visible objects without grouping/ungrouping and the hell that can ensue. Bob S > On Sep 2, 2021, at 11:22 , Bernard Devlin via use-livecode > wrote: > > Whilst waiting for a fix, would a temporary solution be to use sqlite to > create an in-memory database and let sqlite do the sorting for you? > > Regards, Bernard. > > On Mon, Aug 30, 2021 at 8:23 PM Ben Rubinstein via use-livecode < > use-livecode@lists.runrev.com> wrote: > >> Thanks to Mark Waddingham's advice about using a buffer var when >> accumulating >> a large text variabel in stages, I've now got a script that took 8 hours >> under >> LC9, and (8 minutes under LC6) down by stages to just under 1 hour under >> LC9. >> >> However I have some remaining issues not amenable to this approach; of >> which >> the most significant relates to the sort command. >> >> In all cases it seems to take much longer under LC9 than it did under LC6; >> although the factor is quite variable. The most dramatic is one instance, >> in >> which this statement: >> >>sort lines of tNewTable by item iSortCol of each >> >> takes 35 minutes to execute. `tNewTable` is a variable consisting of some >> 223,000 lines of text; approx 70MB. The exact same statement with the same >> data on the same computer in LC6 takes just 1 second. >> >> Has anyone else noticed something of this sort? As I said, the effect >> varies: >> e.g. 54 seconds versus 1 second; 22 seconds versus 1 second. So it may not >> be >> so noticeable in all cases. >> >> TIA, >> >> Ben >> ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode
Re: Sorting text is *VERY* slow in LC9 on Windows (Re: Accumulating text is *VERY* slow in LC9 on Windows)
Whilst waiting for a fix, would a temporary solution be to use sqlite to create an in-memory database and let sqlite do the sorting for you? Regards, Bernard. On Mon, Aug 30, 2021 at 8:23 PM Ben Rubinstein via use-livecode < use-livecode@lists.runrev.com> wrote: > Thanks to Mark Waddingham's advice about using a buffer var when > accumulating > a large text variabel in stages, I've now got a script that took 8 hours > under > LC9, and (8 minutes under LC6) down by stages to just under 1 hour under > LC9. > > However I have some remaining issues not amenable to this approach; of > which > the most significant relates to the sort command. > > In all cases it seems to take much longer under LC9 than it did under LC6; > although the factor is quite variable. The most dramatic is one instance, > in > which this statement: > > sort lines of tNewTable by item iSortCol of each > > takes 35 minutes to execute. `tNewTable` is a variable consisting of some > 223,000 lines of text; approx 70MB. The exact same statement with the same > data on the same computer in LC6 takes just 1 second. > > Has anyone else noticed something of this sort? As I said, the effect > varies: > e.g. 54 seconds versus 1 second; 22 seconds versus 1 second. So it may not > be > so noticeable in all cases. > > TIA, > > Ben > > ___ > use-livecode mailing list > use-livecode@lists.runrev.com > Please visit this url to subscribe, unsubscribe and manage your > subscription preferences: > http://lists.runrev.com/mailman/listinfo/use-livecode > ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode
Re: Sorting text is *VERY* slow in LC9 on Windows (Re: Accumulating text is *VERY* slow in LC9 on Windows)
On 2021-08-30 20:22, Ben Rubinstein via use-livecode wrote: Thanks to Mark Waddingham's advice about using a buffer var when accumulating a large text variabel in stages, I've now got a script that took 8 hours under LC9, and (8 minutes under LC6) down by stages to just under 1 hour under LC9. Has anyone else noticed something of this sort? As I said, the effect varies: e.g. 54 seconds versus 1 second; 22 seconds versus 1 second. So it may not be so noticeable in all cases. It will undoubtedly be the same problem as your accumulation slowness - as the sort routines use (essentially) `put after` to reconstruct the string after sort. So fixing the accumulation slowness will fix sort. Indeed, there's a further optimization to be had in sort now I come to think of it (probably relatively minor after the accumulation problem is sorted) - the sorted buffer size (in chars) will be the same as the input buffer size as its just a permutation of the lines - so the output buffer can be pre-allocated at the right size. We will endeavour to fix for 9.6.5-rc-1 (due 'real soon now'!). Warmest Regards, Mark. -- Mark Waddingham ~ m...@livecode.com ~ http://www.livecode.com/ LiveCode: Everyone can create apps ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode
Sorting text is *VERY* slow in LC9 on Windows (Re: Accumulating text is *VERY* slow in LC9 on Windows)
Thanks to Mark Waddingham's advice about using a buffer var when accumulating a large text variabel in stages, I've now got a script that took 8 hours under LC9, and (8 minutes under LC6) down by stages to just under 1 hour under LC9. However I have some remaining issues not amenable to this approach; of which the most significant relates to the sort command. In all cases it seems to take much longer under LC9 than it did under LC6; although the factor is quite variable. The most dramatic is one instance, in which this statement: sort lines of tNewTable by item iSortCol of each takes 35 minutes to execute. `tNewTable` is a variable consisting of some 223,000 lines of text; approx 70MB. The exact same statement with the same data on the same computer in LC6 takes just 1 second. Has anyone else noticed something of this sort? As I said, the effect varies: e.g. 54 seconds versus 1 second; 22 seconds versus 1 second. So it may not be so noticeable in all cases. TIA, Ben ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode