Re: Sorting text is *VERY* slow in LC9 on Windows (Re: Accumulating text is *VERY* slow in LC9 on Windows)
Thanks Mark, now I get it! Ben On 12/09/2021 12:09, Mark Waddingham via use-livecode wrote: On 2021-09-10 19:10, Ben Rubinstein via use-livecode wrote: The other was The only caveat is that it might cause apps mutating lots of medium->large strings concurrently to take up more memory in general... > ... (and any issue arising from that could be resolved by moving to the 64-bit windows engine). I have been doing my tests on the 64-bit Windows engine. What am I missing? The change I am making to the buffer extension rules means that mutable strings (i.e. those which the last action was modifying them, rather than copying them / fetching them) will take up more space than they did before to allow for possible extension (previously the maximum 'just in case' space which was allocated was 63 bytes). This means that applications may take up more memory as a result (if they happen to have lots of large mutable strings in play at any one time). 32-bit windows engines have a maximum address space of 3Gb - so if an app was already approaching that limit, this might tip them over to start causing out of memory errors. However, in this case, switching to use the 64-bit windows engine would resolve the problem as 64-bit address space is substantially larger. Warmest Regards, Mark. ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode
Re: Sorting text is *VERY* slow in LC9 on Windows (Re: Accumulating text is *VERY* slow in LC9 on Windows)
On 2021-09-10 19:10, Ben Rubinstein via use-livecode wrote: The other was The only caveat is that it might cause apps mutating lots of medium->large strings concurrently to take up more memory in general... > ... (and any issue arising from that could be resolved by moving to the 64-bit windows engine). I have been doing my tests on the 64-bit Windows engine. What am I missing? The change I am making to the buffer extension rules means that mutable strings (i.e. those which the last action was modifying them, rather than copying them / fetching them) will take up more space than they did before to allow for possible extension (previously the maximum 'just in case' space which was allocated was 63 bytes). This means that applications may take up more memory as a result (if they happen to have lots of large mutable strings in play at any one time). 32-bit windows engines have a maximum address space of 3Gb - so if an app was already approaching that limit, this might tip them over to start causing out of memory errors. However, in this case, switching to use the 64-bit windows engine would resolve the problem as 64-bit address space is substantially larger. Warmest Regards, Mark. -- Mark Waddingham ~ m...@livecode.com ~ http://www.livecode.com/ LiveCode: Everyone can create apps ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode
Re: Sorting text is *VERY* slow in LC9 on Windows (Re: Accumulating text is *VERY* slow in LC9 on Windows)
Ah yes, the cheese thing. I so wish I had saved that thread, but I may have needed an extra backup drive to do it. So listen, and ye shall hear. Long, long ago in the olden days, before there was a forum, before 64-bit apps, back when the QCC reports numbered in the lower hundreds, someone on the mailing list mentioned a preference for a cheese. I don't recall who or what, because at the time it seemed so innocuous. Someone else agreed, someone else disagreed, some did both. Discussions went on about which country produced the best cheese, opinions which were disputed by others who felt their own country should be nominated or at least take second place. Others had opinions about countries who did or did not produce cheese. Different varieties of cheese were analyzed, and opined upon. The same type of cheese was compared between one country and another. Suggestions were made to visit a country in order to try a cheese, which would presumably convince the doubters. It was implied that one's own country had good enough cheese, if not the best. This went on for a very long time. Of course, eventually the uninvolved, or those who were indifferent to cheese, or perhaps those who did not like cheese at all, asked the participants to desist and stay on topic. But by then the thread had taken on a life of its own and continued rolling on, gathering momentum like a stone thrown downhill and perhaps subsequently tumbling down river. Being a polite mailing list, no one actually said, "shut up" but at times you could read between the lines. Other readers suffered in stoic silence and made good use of their Delete key. There were occasional mutterings about slightly changing subject lines in posts, which prevented filtering out cheese curds from on-topic posts. The conversation continued for months. Or if it didn't, it seemed like it. Finally List Mom appeared with a gentle but firm pronouncement that in addition to politics, religion, and other such topics, Cheese was now banned on the list. Some were resigned, others sighed in relief. Occasionally you will still see oblique references to the dairy product but in general we speak of it only by inferred reference. I trust I will not be banned for being so explicit, but sometimes one must speak plainly. -- Jacqueline Landman Gay | jac...@hyperactivesw.com HyperActive Software | http://www.hyperactivesw.com On September 11, 2021 7:27:13 PM Lagi Pittas via use-livecode wrote: I thought Mark was misremembering Emmental when he said Edam (they do start with an E) but I believe Venezuelan Beaver Cheeses is probably a better analogy as it has many holes caused by the beavers' sharp incisors. and is a bit of a mystery (very difficult to acquire) as is the workings of windows buffer allocation . Can we say Cheese? https://www.youtube.com/watch?v=Hz1JWzyvv8A=3s lagi p.s Jacques - when did this Cheese thing start - it is certainly before my time. On Fri, 10 Sept 2021 at 19:16, J. Landman Gay via use-livecode < use-livecode@lists.runrev.com> wrote: You're talking about cheese. I'm telling Mom. Nyah. -- Jacqueline Landman Gay | jac...@hyperactivesw.com HyperActive Software | http://www.hyperactivesw.com On September 10, 2021 8:42:48 AM Trevor DeVore via use-livecode wrote: On Fri, Sep 10, 2021 at 8:15 AM Mark Waddingham via use-livecode < use-livecode@lists.runrev.com> wrote: On 2021-09-10 14:06, Mark Waddingham via use-livecode wrote: The windows heap is much more prudent than UNIXy counterparts it would seem - where UNIX heaps will happily leave plenty of free space (which the heaps know about and thus can re-use), Windows appears to avoid that like the plague (which I'm sure is the case for lots of historical reasons and backwards compatibility). [ To give a very rough analogy, the map of used space in a heap on windows is like a block of cheddar; whereas on UNIXy systems it will be like a block of edam ]. I of course meant 'Swiss', not 'Edam'! Thanks for the clarification. I feel like I have a decent understanding of cheeses, but I couldn’t figure out how cheddar and edam were different in this analogy :-) -- Trevor DeVore ScreenSteps ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode -- KIndest Regards Lagi ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode
Re: Sorting text is *VERY* slow in LC9 on Windows (Re: Accumulating text is *VERY* slow in LC9 on Windows)
I thought Mark was misremembering Emmental when he said Edam (they do start with an E) but I believe Venezuelan Beaver Cheeses is probably a better analogy as it has many holes caused by the beavers' sharp incisors. and is a bit of a mystery (very difficult to acquire) as is the workings of windows buffer allocation . Can we say Cheese? https://www.youtube.com/watch?v=Hz1JWzyvv8A=3s lagi p.s Jacques - when did this Cheese thing start - it is certainly before my time. On Fri, 10 Sept 2021 at 19:16, J. Landman Gay via use-livecode < use-livecode@lists.runrev.com> wrote: > You're talking about cheese. I'm telling Mom. Nyah. > > -- > Jacqueline Landman Gay | jac...@hyperactivesw.com > HyperActive Software | http://www.hyperactivesw.com > On September 10, 2021 8:42:48 AM Trevor DeVore via use-livecode > wrote: > > > On Fri, Sep 10, 2021 at 8:15 AM Mark Waddingham via use-livecode < > > use-livecode@lists.runrev.com> wrote: > > > >> On 2021-09-10 14:06, Mark Waddingham via use-livecode wrote: > >> > The windows heap is much more prudent than UNIXy counterparts it would > >> > seem - where UNIX heaps will happily leave plenty of free space (which > >> > the heaps know about and thus can re-use), Windows appears to avoid > >> > that like the plague (which I'm sure is the case for lots of > >> > historical reasons and backwards compatibility). [ To give a very > >> > rough analogy, the map of used space in a heap on windows is like a > >> > block of cheddar; whereas on UNIXy systems it will be like a block of > >> > edam ]. > >> > >> I of course meant 'Swiss', not 'Edam'! > > > > > > Thanks for the clarification. I feel like I have a decent understanding > of > > cheeses, but I couldn’t figure out how cheddar and edam were different in > > this analogy :-) > > > > -- > > Trevor DeVore > > ScreenSteps > > > >> > > ___ > > use-livecode mailing list > > use-livecode@lists.runrev.com > > Please visit this url to subscribe, unsubscribe and manage your > > subscription preferences: > > http://lists.runrev.com/mailman/listinfo/use-livecode > > > > > ___ > use-livecode mailing list > use-livecode@lists.runrev.com > Please visit this url to subscribe, unsubscribe and manage your > subscription preferences: > http://lists.runrev.com/mailman/listinfo/use-livecode > -- KIndest Regards Lagi ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode
Re: Sorting text is *VERY* slow in LC9 on Windows (Re: Accumulating text is *VERY* slow in LC9 on Windows)
Yes, it did mention cheese... but it was entirely on topic and about LiveCode. (How could Mom not approve.) -- Scott Morrow Elementary Software (Now with 20% less chalk dust!) web https://elementarysoftware.com/ email sc...@elementarysoftware.com booth1-360-734-4701 -- > On Sep 10, 2021, at 11:15 AM, J. Landman Gay via use-livecode > wrote: > > You're talking about cheese. I'm telling Mom. Nyah. > > -- > Jacqueline Landman Gay | jac...@hyperactivesw.com > HyperActive Software | http://www.hyperactivesw.com > On September 10, 2021 8:42:48 AM Trevor DeVore via use-livecode > wrote: > >> On Fri, Sep 10, 2021 at 8:15 AM Mark Waddingham via use-livecode < >> use-livecode@lists.runrev.com> wrote: >> >>> On 2021-09-10 14:06, Mark Waddingham via use-livecode wrote: >>> > The windows heap is much more prudent than UNIXy counterparts it would >>> > seem - where UNIX heaps will happily leave plenty of free space (which >>> > the heaps know about and thus can re-use), Windows appears to avoid >>> > that like the plague (which I'm sure is the case for lots of >>> > historical reasons and backwards compatibility). [ To give a very >>> > rough analogy, the map of used space in a heap on windows is like a >>> > block of cheddar; whereas on UNIXy systems it will be like a block of >>> > edam ]. >>> >>> I of course meant 'Swiss', not 'Edam'! >> >> >> Thanks for the clarification. I feel like I have a decent understanding of >> cheeses, but I couldn’t figure out how cheddar and edam were different in >> this analogy :-) >> >> -- >> Trevor DeVore >> ScreenSteps >> >>> >> ___ >> use-livecode mailing list >> use-livecode@lists.runrev.com >> Please visit this url to subscribe, unsubscribe and manage your subscription >> preferences: >> http://lists.runrev.com/mailman/listinfo/use-livecode ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode
Re: Sorting text is *VERY* slow in LC9 on Windows (Re: Accumulating text is *VERY* slow in LC9 on Windows)
I’m going to propound on the politics of holey cheese makers in a moment! Sent from my iPhone > On Sep 10, 2021, at 11:16, J. Landman Gay via use-livecode > wrote: > > You're talking about cheese. I'm telling Mom. Nyah. > > -- > Jacqueline Landman Gay | jac...@hyperactivesw.com > HyperActive Software | http://www.hyperactivesw.com >> On September 10, 2021 8:42:48 AM Trevor DeVore via use-livecode >> wrote: >> >> On Fri, Sep 10, 2021 at 8:15 AM Mark Waddingham via use-livecode < >> use-livecode@lists.runrev.com> wrote: >> On 2021-09-10 14:06, Mark Waddingham via use-livecode wrote: >>> > The windows heap is much more prudent than UNIXy counterparts it would >>> > seem - where UNIX heaps will happily leave plenty of free space (which >>> > the heaps know about and thus can re-use), Windows appears to avoid >>> > that like the plague (which I'm sure is the case for lots of >>> > historical reasons and backwards compatibility). [ To give a very >>> > rough analogy, the map of used space in a heap on windows is like a >>> > block of cheddar; whereas on UNIXy systems it will be like a block of >>> > edam ]. >>> >>> I of course meant 'Swiss', not 'Edam'! >> >> >> Thanks for the clarification. I feel like I have a decent understanding of >> cheeses, but I couldn’t figure out how cheddar and edam were different in >> this analogy :-) >> >> -- >> Trevor DeVore >> ScreenSteps >> >>> >> ___ >> use-livecode mailing list >> use-livecode@lists.runrev.com >> Please visit this url to subscribe, unsubscribe and manage your subscription >> preferences: >> http://lists.runrev.com/mailman/listinfo/use-livecode > > > > > ___ > use-livecode mailing list > use-livecode@lists.runrev.com > Please visit this url to subscribe, unsubscribe and manage your subscription > preferences: > http://lists.runrev.com/mailman/listinfo/use-livecode ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode
Re: Sorting text is *VERY* slow in LC9 on Windows (Re: Accumulating text is *VERY* slow in LC9 on Windows)
You're talking about cheese. I'm telling Mom. Nyah. -- Jacqueline Landman Gay | jac...@hyperactivesw.com HyperActive Software | http://www.hyperactivesw.com On September 10, 2021 8:42:48 AM Trevor DeVore via use-livecode wrote: On Fri, Sep 10, 2021 at 8:15 AM Mark Waddingham via use-livecode < use-livecode@lists.runrev.com> wrote: On 2021-09-10 14:06, Mark Waddingham via use-livecode wrote: > The windows heap is much more prudent than UNIXy counterparts it would > seem - where UNIX heaps will happily leave plenty of free space (which > the heaps know about and thus can re-use), Windows appears to avoid > that like the plague (which I'm sure is the case for lots of > historical reasons and backwards compatibility). [ To give a very > rough analogy, the map of used space in a heap on windows is like a > block of cheddar; whereas on UNIXy systems it will be like a block of > edam ]. I of course meant 'Swiss', not 'Edam'! Thanks for the clarification. I feel like I have a decent understanding of cheeses, but I couldn’t figure out how cheddar and edam were different in this analogy :-) -- Trevor DeVore ScreenSteps ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode
Re: Sorting text is *VERY* slow in LC9 on Windows (Re: Accumulating text is *VERY* slow in LC9 on Windows)
Hi Mark, Thank you for this, very promising. Only two things puzzled me. One you've already addressed when you corrected the specified cheese. The other was The only caveat is that it might cause apps mutating lots of medium->large strings concurrently to take up more memory in general... > ... (and any issue arising from that could be resolved by moving to the 64-bit windows engine). I have been doing my tests on the 64-bit Windows engine. What am I missing? TIA (and T as always for all your work), Ben On 10/09/2021 14:06, Mark Waddingham via use-livecode wrote: On 2021-09-02 18:38, Mark Waddingham via use-livecode wrote: We will endeavour to fix for 9.6.5-rc-1 (due 'real soon now'!). So I have been prodding the windows 'accumulating large strings' speed problem this week (in amongst other things). It is definitely memory allocation causing the problem. The rule for extending a string buffer when inserting/appending more text is pretty much the same as 6.7 (the amount of 'extra space' it asks for each time is a little less - but changing it to the constant used in 6.7 makes no difference). What is different between 6.7 and 7.0+ is the memory allocation patterns of the engine itself and I think it is this which is invariably causing a whole new buffer having to be allocated when appending to large strings, rather than the existing one being extended. The windows heap is much more prudent than UNIXy counterparts it would seem - where UNIX heaps will happily leave plenty of free space (which the heaps know about and thus can re-use), Windows appears to avoid that like the plague (which I'm sure is the case for lots of historical reasons and backwards compatibility). [ To give a very rough analogy, the map of used space in a heap on windows is like a block of cheddar; whereas on UNIXy systems it will be like a block of edam ]. So anyway, long story short, making a string buffer grow in proportion to its size appears to mitigate the problem - I still need to finesse things a bit though to ensure that memory starvation doesn't occur when you have a couple of large mutating strings but all being well we should be able to get something together for 9.6.5-rc-1. The only caveat is that it might cause apps mutating lots of medium->large strings concurrently to take up more memory in general (i.e. in that extra unused 'just in case' space at the end of each buffer) but no more than the same apps would on any other (UNIX-based) platform (and any issue arising from that could be resolved by moving to the 64-bit windows engine). For reference, the bug about the string accumulation problem as posed by Ben is https://quality.livecode.com/show_bug.cgi?id=23319 (this will also fix the sort speed too - as the final step the sort command performs internally is re-accumulating the string in the new order). Warmest Regards, Mark. P.S. I cannot really say whether this will help with the various 'IDE can be like treacle' on Windows problems - but it definitely can't hurt! ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode
Re: Sorting text is *VERY* slow in LC9 on Windows (Re: Accumulating text is *VERY* slow in LC9 on Windows)
Cheddar, Edam and Swiss! Very daring references on this list. ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode
Re: Sorting text is *VERY* slow in LC9 on Windows (Re: Accumulating text is *VERY* slow in LC9 on Windows)
On Fri, Sep 10, 2021 at 8:15 AM Mark Waddingham via use-livecode < use-livecode@lists.runrev.com> wrote: > On 2021-09-10 14:06, Mark Waddingham via use-livecode wrote: > > The windows heap is much more prudent than UNIXy counterparts it would > > seem - where UNIX heaps will happily leave plenty of free space (which > > the heaps know about and thus can re-use), Windows appears to avoid > > that like the plague (which I'm sure is the case for lots of > > historical reasons and backwards compatibility). [ To give a very > > rough analogy, the map of used space in a heap on windows is like a > > block of cheddar; whereas on UNIXy systems it will be like a block of > > edam ]. > > I of course meant 'Swiss', not 'Edam'! Thanks for the clarification. I feel like I have a decent understanding of cheeses, but I couldn’t figure out how cheddar and edam were different in this analogy :-) -- Trevor DeVore ScreenSteps > ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode
Re: Sorting text is *VERY* slow in LC9 on Windows (Re: Accumulating text is *VERY* slow in LC9 on Windows)
On 2021-09-10 14:06, Mark Waddingham via use-livecode wrote: The windows heap is much more prudent than UNIXy counterparts it would seem - where UNIX heaps will happily leave plenty of free space (which the heaps know about and thus can re-use), Windows appears to avoid that like the plague (which I'm sure is the case for lots of historical reasons and backwards compatibility). [ To give a very rough analogy, the map of used space in a heap on windows is like a block of cheddar; whereas on UNIXy systems it will be like a block of edam ]. I of course meant 'Swiss', not 'Edam'! -- Mark Waddingham ~ m...@livecode.com ~ http://www.livecode.com/ LiveCode: Everyone can create apps ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode
Re: Sorting text is *VERY* slow in LC9 on Windows (Re: Accumulating text is *VERY* slow in LC9 on Windows)
On 2021-09-02 18:38, Mark Waddingham via use-livecode wrote: We will endeavour to fix for 9.6.5-rc-1 (due 'real soon now'!). So I have been prodding the windows 'accumulating large strings' speed problem this week (in amongst other things). It is definitely memory allocation causing the problem. The rule for extending a string buffer when inserting/appending more text is pretty much the same as 6.7 (the amount of 'extra space' it asks for each time is a little less - but changing it to the constant used in 6.7 makes no difference). What is different between 6.7 and 7.0+ is the memory allocation patterns of the engine itself and I think it is this which is invariably causing a whole new buffer having to be allocated when appending to large strings, rather than the existing one being extended. The windows heap is much more prudent than UNIXy counterparts it would seem - where UNIX heaps will happily leave plenty of free space (which the heaps know about and thus can re-use), Windows appears to avoid that like the plague (which I'm sure is the case for lots of historical reasons and backwards compatibility). [ To give a very rough analogy, the map of used space in a heap on windows is like a block of cheddar; whereas on UNIXy systems it will be like a block of edam ]. So anyway, long story short, making a string buffer grow in proportion to its size appears to mitigate the problem - I still need to finesse things a bit though to ensure that memory starvation doesn't occur when you have a couple of large mutating strings but all being well we should be able to get something together for 9.6.5-rc-1. The only caveat is that it might cause apps mutating lots of medium->large strings concurrently to take up more memory in general (i.e. in that extra unused 'just in case' space at the end of each buffer) but no more than the same apps would on any other (UNIX-based) platform (and any issue arising from that could be resolved by moving to the 64-bit windows engine). For reference, the bug about the string accumulation problem as posed by Ben is https://quality.livecode.com/show_bug.cgi?id=23319 (this will also fix the sort speed too - as the final step the sort command performs internally is re-accumulating the string in the new order). Warmest Regards, Mark. P.S. I cannot really say whether this will help with the various 'IDE can be like treacle' on Windows problems - but it definitely can't hurt! -- Mark Waddingham ~ m...@livecode.com ~ http://www.livecode.com/ LiveCode: Everyone can create apps ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode
Re: Sorting text is *VERY* slow in LC9 on Windows (Re: Accumulating text is *VERY* slow in LC9 on Windows)
I'm very much hoping that Mark W might magically fix this in 9.6.5. But in the meantime FWIW, the place where this was really hurting (in a script that took 8 minutes under LC6, was taking 8 hours under LC9, but I've gradually tamed it down to under an hour by buffering the large accumulations) was a single sort command, on 70 MB of data in approx 223,000 lines. I've replaced this line: sort lines of tNewTable by item iSortCol of each which took 1 second on Mac, 2063 seconds (i.e. 34 minutes) on Windows, with a call to this command command sortLinesByTabbedColumn @tTable, iSortCol local aTable, tSortTable, iARcounter, tARbuffer, tRow, k -- load table into an array for fast access by line number put tTable into aTable split aTable using return -- compile index of just the column to sort on, and line number set the itemDelimiter to tab repeat for each key k in aTable get (item iSortCol of aTable[k]) && k appendRow it, iARcounter, tARbuffer, tSortTable end repeat put tARbuffer after tSortTable -- sort it sort lines of tSortTable -- rebuild table out of array, in sorted order put empty into tARbuffer put empty into tTable repeat for each line tRow in tSortTable put last word of tRow into k appendRow aTable[k], iARcounter, tARbuffer, tTable end repeat put tARbuffer after tTable end sortLinesByTabbedColumn which takes 25 seconds on Windows (to my surprise, most of that time was in the final 'rebuild' loop). On 02/09/2021 23:53, Bob Sneidar via use-livecode wrote: I am going to say no, because you still have to traverse the file once to get it into sqLite, then do the sort, then write out the file when done. I might be mistaken, the subsequent SQL sort may make up for lost time. Using a memory SQL really shines when you need to make multiple passes at the data using different queries. One pass may not impress you much. For instance, I have a File Management module built into my application. A file can belong to a customer, and also to a site, and also to a device. Like so: custid siteid deviceidfilepath 123 disk/folder/file1 456 098 disk/folder/file2 789 765 432 disk/folder/file3 Note all have a custid, some have a siteid as well, and some also have a deviceid. So rather than query mySQL for the files for each site or device as I select them, I instead, upon selecting a customer, query mySQL for ALL the file records for that customer, (which of course contain the file records for all the sites and devices), then store that in a memory database. Then when a different site or device belonging to that customer is selected, I query the memory database for those belonging to that site, or that device in those modules respectively. The performance enhancement is significant. Another way I apply this is to get the objects on a card passing a list of properties I'm interested in, then store the data in a memory database. I can then query for objects with certain properties without having to iterate through all the objects on a card in a repeat loop. For instance, the farthest left, top, right and bottom object whose visible is true in 4 memory db queries, giving me the total rect of all the visible objects without grouping/ungrouping and the hell that can ensue. Bob S On Sep 2, 2021, at 11:22 , Bernard Devlin via use-livecode wrote: Whilst waiting for a fix, would a temporary solution be to use sqlite to create an in-memory database and let sqlite do the sorting for you? Regards, Bernard. On Mon, Aug 30, 2021 at 8:23 PM Ben Rubinstein via use-livecode < use-livecode@lists.runrev.com> wrote: Thanks to Mark Waddingham's advice about using a buffer var when accumulating a large text variabel in stages, I've now got a script that took 8 hours under LC9, and (8 minutes under LC6) down by stages to just under 1 hour under LC9. However I have some remaining issues not amenable to this approach; of which the most significant relates to the sort command. In all cases it seems to take much longer under LC9 than it did under LC6; although the factor is quite variable. The most dramatic is one instance, in which this statement: sort lines of tNewTable by item iSortCol of each takes 35 minutes to execute. `tNewTable` is a variable consisting of some 223,000 lines of text; approx 70MB. The exact same statement with the same data on the same computer in LC6 takes just 1 second. Has anyone else noticed something of this sort? As I said, the effect varies: e.g. 54 seconds versus 1 second; 22 seconds versus 1 second. So it may not be so noticeable in all cases.
Re: Sorting text is *VERY* slow in LC9 on Windows (Re: Accumulating text is *VERY* slow in LC9 on Windows)
I am going to say no, because you still have to traverse the file once to get it into sqLite, then do the sort, then write out the file when done. I might be mistaken, the subsequent SQL sort may make up for lost time. Using a memory SQL really shines when you need to make multiple passes at the data using different queries. One pass may not impress you much. For instance, I have a File Management module built into my application. A file can belong to a customer, and also to a site, and also to a device. Like so: custid siteid deviceidfilepath 123 disk/folder/file1 456 098 disk/folder/file2 789 765 432 disk/folder/file3 Note all have a custid, some have a siteid as well, and some also have a deviceid. So rather than query mySQL for the files for each site or device as I select them, I instead, upon selecting a customer, query mySQL for ALL the file records for that customer, (which of course contain the file records for all the sites and devices), then store that in a memory database. Then when a different site or device belonging to that customer is selected, I query the memory database for those belonging to that site, or that device in those modules respectively. The performance enhancement is significant. Another way I apply this is to get the objects on a card passing a list of properties I'm interested in, then store the data in a memory database. I can then query for objects with certain properties without having to iterate through all the objects on a card in a repeat loop. For instance, the farthest left, top, right and bottom object whose visible is true in 4 memory db queries, giving me the total rect of all the visible objects without grouping/ungrouping and the hell that can ensue. Bob S > On Sep 2, 2021, at 11:22 , Bernard Devlin via use-livecode > wrote: > > Whilst waiting for a fix, would a temporary solution be to use sqlite to > create an in-memory database and let sqlite do the sorting for you? > > Regards, Bernard. > > On Mon, Aug 30, 2021 at 8:23 PM Ben Rubinstein via use-livecode < > use-livecode@lists.runrev.com> wrote: > >> Thanks to Mark Waddingham's advice about using a buffer var when >> accumulating >> a large text variabel in stages, I've now got a script that took 8 hours >> under >> LC9, and (8 minutes under LC6) down by stages to just under 1 hour under >> LC9. >> >> However I have some remaining issues not amenable to this approach; of >> which >> the most significant relates to the sort command. >> >> In all cases it seems to take much longer under LC9 than it did under LC6; >> although the factor is quite variable. The most dramatic is one instance, >> in >> which this statement: >> >>sort lines of tNewTable by item iSortCol of each >> >> takes 35 minutes to execute. `tNewTable` is a variable consisting of some >> 223,000 lines of text; approx 70MB. The exact same statement with the same >> data on the same computer in LC6 takes just 1 second. >> >> Has anyone else noticed something of this sort? As I said, the effect >> varies: >> e.g. 54 seconds versus 1 second; 22 seconds versus 1 second. So it may not >> be >> so noticeable in all cases. >> >> TIA, >> >> Ben >> ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode
Re: Sorting text is *VERY* slow in LC9 on Windows (Re: Accumulating text is *VERY* slow in LC9 on Windows)
Whilst waiting for a fix, would a temporary solution be to use sqlite to create an in-memory database and let sqlite do the sorting for you? Regards, Bernard. On Mon, Aug 30, 2021 at 8:23 PM Ben Rubinstein via use-livecode < use-livecode@lists.runrev.com> wrote: > Thanks to Mark Waddingham's advice about using a buffer var when > accumulating > a large text variabel in stages, I've now got a script that took 8 hours > under > LC9, and (8 minutes under LC6) down by stages to just under 1 hour under > LC9. > > However I have some remaining issues not amenable to this approach; of > which > the most significant relates to the sort command. > > In all cases it seems to take much longer under LC9 than it did under LC6; > although the factor is quite variable. The most dramatic is one instance, > in > which this statement: > > sort lines of tNewTable by item iSortCol of each > > takes 35 minutes to execute. `tNewTable` is a variable consisting of some > 223,000 lines of text; approx 70MB. The exact same statement with the same > data on the same computer in LC6 takes just 1 second. > > Has anyone else noticed something of this sort? As I said, the effect > varies: > e.g. 54 seconds versus 1 second; 22 seconds versus 1 second. So it may not > be > so noticeable in all cases. > > TIA, > > Ben > > ___ > use-livecode mailing list > use-livecode@lists.runrev.com > Please visit this url to subscribe, unsubscribe and manage your > subscription preferences: > http://lists.runrev.com/mailman/listinfo/use-livecode > ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode
Re: Sorting text is *VERY* slow in LC9 on Windows (Re: Accumulating text is *VERY* slow in LC9 on Windows)
On 2021-08-30 20:22, Ben Rubinstein via use-livecode wrote: Thanks to Mark Waddingham's advice about using a buffer var when accumulating a large text variabel in stages, I've now got a script that took 8 hours under LC9, and (8 minutes under LC6) down by stages to just under 1 hour under LC9. Has anyone else noticed something of this sort? As I said, the effect varies: e.g. 54 seconds versus 1 second; 22 seconds versus 1 second. So it may not be so noticeable in all cases. It will undoubtedly be the same problem as your accumulation slowness - as the sort routines use (essentially) `put after` to reconstruct the string after sort. So fixing the accumulation slowness will fix sort. Indeed, there's a further optimization to be had in sort now I come to think of it (probably relatively minor after the accumulation problem is sorted) - the sorted buffer size (in chars) will be the same as the input buffer size as its just a permutation of the lines - so the output buffer can be pre-allocated at the right size. We will endeavour to fix for 9.6.5-rc-1 (due 'real soon now'!). Warmest Regards, Mark. -- Mark Waddingham ~ m...@livecode.com ~ http://www.livecode.com/ LiveCode: Everyone can create apps ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode
Sorting text is *VERY* slow in LC9 on Windows (Re: Accumulating text is *VERY* slow in LC9 on Windows)
Thanks to Mark Waddingham's advice about using a buffer var when accumulating a large text variabel in stages, I've now got a script that took 8 hours under LC9, and (8 minutes under LC6) down by stages to just under 1 hour under LC9. However I have some remaining issues not amenable to this approach; of which the most significant relates to the sort command. In all cases it seems to take much longer under LC9 than it did under LC6; although the factor is quite variable. The most dramatic is one instance, in which this statement: sort lines of tNewTable by item iSortCol of each takes 35 minutes to execute. `tNewTable` is a variable consisting of some 223,000 lines of text; approx 70MB. The exact same statement with the same data on the same computer in LC6 takes just 1 second. Has anyone else noticed something of this sort? As I said, the effect varies: e.g. 54 seconds versus 1 second; 22 seconds versus 1 second. So it may not be so noticeable in all cases. TIA, Ben ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode
Re: Accumulating text is *VERY* slow in LC9 on Windows
Thank you Mark! Using a buffer indeed essentially solves the problem (at least for this example!). Because I may have to use this in a bunch of places, I made my life easier by putting it in a command using variables passed by reference. Because I'm almost invariably appending a line at a time, and usually have a row counter going anyway, I'm deciding when to dump the buffer based on number of lines rather than length of the buffer (which may occasionally mean the buffer has to be resized up). command appendRow tRow, @iRowCounter, @tBuffer, @tOutput put tRow & return after tBuffer add 1 to iRowCounter if (iRowCounter mod siBuffSize) = 0 then put tBuffer after tOutput if sbDeleteBuffer then delete codeunit 1 to -1 of tBuffer else put empty into tBuffer end if end if end appendRow I initially wrote this emptying the buffer just by putting empty into it. Then I borrowed your line of deleting the contents, which I presume makes it more likely to be retained in the heap. In general inline is a bit faster than using the command; and deleting the code points of the buffer is a bit faster than emptying - but it's not completely clear cut. (This is one set of runs, not averaging repeated ones.) command command inline inline buffer size empty delete empty delete 500 lines 33 31 29 29 1500 lines 16 13 7 14 2500 lines 14 10 7 11 4000 lines 18 11 8 9 5000 lines 19 7 11 9 (All times in seconds on Windows, LC 9.6.3.) And given that not using a buffer takes 589 seconds, none of the above variations really matter! So my remaining question for you Mark is this: > P.S. This is an engine issue - we'll need to look into why there's such a > difference with 6.7 - as, I'm pretty sure I kept the rules about extending > buffers pretty much the same in string concatenation. Do you have a sense of (a) how likely it is you'll have an "aha!" moment and (b) any idea of when you might have a chance to look at this. I'm just asking purely selfishly, because if it's soon I might get away without having to address my 5,000 line code jungle nightmare...! Many thanks for this tip-off, Ben On 26/08/2021 12:23, Mark Waddingham via use-livecode wrote: On 2021-08-25 18:15, Ben Rubinstein via use-livecode wrote: I simplified it down to this (pointless) loop which just rebuilds a table one line at a time: local tNewTable repeat for each line tRow in tWorkTable put tRow & return after tNewTable end repeat with these results: 6.7.11 MacOS 8 seconds 6.7.11 Win32 7 seconds 9.6.3 MacOS 0 seconds 9.6.3 Win32 591 seconds Using a buffer var should workaround the performance issue (which is related to the windows heap manager not being very good at continually re-extending a buffer): on mouseUp local tLine repeat 256 times put "*" after tLine end repeat local tTime put the millisecs into tTime local tBuffer local tText repeat 257000 times put tLine & return after tBuffer if the number of codeunits in tBuffer > 50 then put tBuffer after tText delete codeunit 1 to -1 of tBuffer end if end repeat put tBuffer after tText answer (the number of codeunits in tText) & return & (the millisecs - tTime) end mouseUp In the original loop, tNewTable is continually extended internally, something which appears to cause O(n^2) performance on Windows. In the revised loop, an intermediate buffer var is used which (after first time getting 'full') will have a backing store of 500k ish - meaning tText is extended much less often. (Playing with the value of 50 up or down will affect the resulting speed - there will always be a sweet spot). On my Windows VM - the above loop (which generates about 68mb of text or so, takes about 3s. Hope this helps! Mark. P.S. This is an engine issue - we'll need to look into why there's such a difference with 6.7 - as, I'm pretty sure I kept the rules about extending buffers pretty much the same in string concatenation. ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode
Re: Accumulating text is *VERY* slow in LC9 on Windows
On 2021-08-25 18:15, Ben Rubinstein via use-livecode wrote: I simplified it down to this (pointless) loop which just rebuilds a table one line at a time: local tNewTable repeat for each line tRow in tWorkTable put tRow & return after tNewTable end repeat with these results: 6.7.11 MacOS 8 seconds 6.7.11 Win32 7 seconds 9.6.3 MacOS 0 seconds 9.6.3 Win32591 seconds Using a buffer var should workaround the performance issue (which is related to the windows heap manager not being very good at continually re-extending a buffer): on mouseUp local tLine repeat 256 times put "*" after tLine end repeat local tTime put the millisecs into tTime local tBuffer local tText repeat 257000 times put tLine & return after tBuffer if the number of codeunits in tBuffer > 50 then put tBuffer after tText delete codeunit 1 to -1 of tBuffer end if end repeat put tBuffer after tText answer (the number of codeunits in tText) & return & (the millisecs - tTime) end mouseUp In the original loop, tNewTable is continually extended internally, something which appears to cause O(n^2) performance on Windows. In the revised loop, an intermediate buffer var is used which (after first time getting 'full') will have a backing store of 500k ish - meaning tText is extended much less often. (Playing with the value of 50 up or down will affect the resulting speed - there will always be a sweet spot). On my Windows VM - the above loop (which generates about 68mb of text or so, takes about 3s. Hope this helps! Mark. P.S. This is an engine issue - we'll need to look into why there's such a difference with 6.7 - as, I'm pretty sure I kept the rules about extending buffers pretty much the same in string concatenation. -- Mark Waddingham ~ m...@livecode.com ~ http://www.livecode.com/ LiveCode: Everyone can create apps ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode
Re: Accumulating text is *VERY* slow in LC9 on Windows
Thanks Curry. In fact (apologies I didn't make this clear in my already too-long email) it really is just the accumulating of text, at least in this instance, that's the issue. After I added a test on each extracted (i.e. looped over) line to see if it was "strictly a binary string", and counting the results; but then commented out the line that appended the text, the loop completed in a second or less. Uncommenting the line put it back to 8 minutes or so. So of your items below, it really is just item 3. The difference between the Mac and Windows machines I happened to be using isn't really relevant. The point is that on one Windows machine, running the same script in LC9 is about 50 times slower than the same script on the same machine in LC6. On a given Mac machine, it's many times faster in LC9 than LC6. Something is awry on Windows. BUT - I've only tested on one Windows machine, that's all I've got access to at the moment. So if anyone has a chance to see if they get similar results, that would be very helpful. Ben On 26/08/2021 02:27, Curry Kenworthy via use-livecode wrote: Ben: > But the key thing is: for this task, LC9 is > dramatically slower on Windows!) > Have others seen something like this? Generally: Yep, I've seen plenty of slowdowns. On both Windows and Mac, depending on the task. Specifically: I'd like to test this later; still recovering from two back-to-back illnesses. Suggestion: although your code may look short, you're potentially testing up to 5 different issues: 1. Loop speed 2. Text line parse speed 3. Text concat speed 4. Variable/memory speed 5. Mac to Win text It would be good to narrow your test and see which area has the biggest impact, or eliminate some of those possible factors. Best wishes, Curry Kenworthy Custom Software Development "Better Methods, Better Results" LiveCode Training and Consulting http://livecodeconsulting.com/ ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode
Re: Accumulating text is *VERY* slow in LC9 on Windows
Hi Alex, Thanks for responding. It's an interesting idea and I may have to try it. But I've got 5,000 lines of script that has evolved over 20 years, so I really don't want to have to rewrite it all! It seems to me that there is an identifiable problem here; something that is happening on (at least one) Windows machine, which was fine under LC6 but is broken under LC9; whereas on Mac it's not the case. My hope is that there is some simple solution to fix this! Ben On 25/08/2021 19:48, Alex Tweedly via use-livecode wrote: Crazy idea - totally untried (sorry, I don't have a Win machine) put 1 into tLineCount repeat for each line tRow in tWorkTable put tRow into tNewTable[tLineCount] add 1 to tLineCount end repeat combine tNewTable using CR Alex. On 25/08/2021 18:15, Ben Rubinstein via use-livecode wrote: Some 20 months ago, I reported that I was in a situation where an app written in 6.7 needed to be updated to access 64bit drivers, which meant updating to 9.5 - which displayed horrifying increase in processing time. In fact I was able to put off the evil day - but now it has returned, and can be put off no longer. A process that normally takes 2 hours is currently taking 9. The core processing stage has gone from around ten minutes to over six hours. After way too long, I've finally got down to at least one smoking gun; which is as simple as can be. Part of what took me so long is a confusion; in production the process runs on Windows, but I develop on Mac. Although on Mac the overall process does take about a third longer in LC9 than LC6, the simple tests I've finally isolated actually run much _quicker_ in LC9 than LC6. So switching between LC6 and LC9 on Mac as I tried to isolate the issue was giving confusing signals. But unmistakeably it's *much* slower on Windows. A simple routine which loops over a load of tab and return formatted data loaded from a TSV file, to truncate a particular field, had the following results processing a 70MB file of approximately 257,000 rows: 6.7.11 MacOS 9 seconds 6.7.11 Win32 10 seconds 9.6.3 MacOS 2 seconds 9.6.3 Win32 498 seconds I simplified it down to this (pointless) loop which just rebuilds a table one line at a time: local tNewTable repeat for each line tRow in tWorkTable put tRow & return after tNewTable end repeat with these results: 6.7.11 MacOS 8 seconds 6.7.11 Win32 7 seconds 9.6.3 MacOS 0 seconds 9.6.3 Win32 591 seconds (there's obviously a lot of variability in these - both were running in IDE, on a logged-in computer, so stuff was probably going on in the background; but I know the overall effect is similar when built as standalone and running by schedule on an unattended machine. But the key thing is: for this task, LC9 is dramatically slower on Windows!) Have others seen something like this? When I posted about this before (thread: "OMG text processing performance 6.7 - 9.5") Mark Waddingham suggested that it might be to do with a hidden cost of binary<->text transforms. That makes some sense; but given that the text already exists, I'm wondering whether taking a line out of text would cause it to be transformed, only to be transformed again when appending? And in particular, why this would affect Windows only. I have also added tests using "is strictly a binary string" in the code above, and this was true for neither input 'tWorkTable', nor the output 'tNewTable', nor any of the 257,00 extracted lines. However it is definitely the accumulating of text that is the issue - simply looping over the lines - even with testing each one to see if it is "strictly a binary string" - is a second or less on Windows in LC9. Has anyone had similar experiences? Suggestions for how this could be avoided? Many thanks in advance, Ben ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode
Re: Accumulating text is *VERY* slow in LC9 on Windows
Ben: > But the key thing is: for this task, LC9 is > dramatically slower on Windows!) > Have others seen something like this? Generally: Yep, I've seen plenty of slowdowns. On both Windows and Mac, depending on the task. Specifically: I'd like to test this later; still recovering from two back-to-back illnesses. Suggestion: although your code may look short, you're potentially testing up to 5 different issues: 1. Loop speed 2. Text line parse speed 3. Text concat speed 4. Variable/memory speed 5. Mac to Win text It would be good to narrow your test and see which area has the biggest impact, or eliminate some of those possible factors. Best wishes, Curry Kenworthy Custom Software Development "Better Methods, Better Results" LiveCode Training and Consulting http://livecodeconsulting.com/ ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode
Re: Accumulating text is *VERY* slow in LC9 on Windows
Crazy idea - totally untried (sorry, I don't have a Win machine) put 1 into tLineCount repeat for each line tRow in tWorkTable put tRow into tNewTable[tLineCount] add 1 to tLineCount end repeat combine tNewTable using CR Alex. On 25/08/2021 18:15, Ben Rubinstein via use-livecode wrote: Some 20 months ago, I reported that I was in a situation where an app written in 6.7 needed to be updated to access 64bit drivers, which meant updating to 9.5 - which displayed horrifying increase in processing time. In fact I was able to put off the evil day - but now it has returned, and can be put off no longer. A process that normally takes 2 hours is currently taking 9. The core processing stage has gone from around ten minutes to over six hours. After way too long, I've finally got down to at least one smoking gun; which is as simple as can be. Part of what took me so long is a confusion; in production the process runs on Windows, but I develop on Mac. Although on Mac the overall process does take about a third longer in LC9 than LC6, the simple tests I've finally isolated actually run much _quicker_ in LC9 than LC6. So switching between LC6 and LC9 on Mac as I tried to isolate the issue was giving confusing signals. But unmistakeably it's *much* slower on Windows. A simple routine which loops over a load of tab and return formatted data loaded from a TSV file, to truncate a particular field, had the following results processing a 70MB file of approximately 257,000 rows: 6.7.11 MacOS 9 seconds 6.7.11 Win32 10 seconds 9.6.3 MacOS 2 seconds 9.6.3 Win32 498 seconds I simplified it down to this (pointless) loop which just rebuilds a table one line at a time: local tNewTable repeat for each line tRow in tWorkTable put tRow & return after tNewTable end repeat with these results: 6.7.11 MacOS 8 seconds 6.7.11 Win32 7 seconds 9.6.3 MacOS 0 seconds 9.6.3 Win32 591 seconds (there's obviously a lot of variability in these - both were running in IDE, on a logged-in computer, so stuff was probably going on in the background; but I know the overall effect is similar when built as standalone and running by schedule on an unattended machine. But the key thing is: for this task, LC9 is dramatically slower on Windows!) Have others seen something like this? When I posted about this before (thread: "OMG text processing performance 6.7 - 9.5") Mark Waddingham suggested that it might be to do with a hidden cost of binary<->text transforms. That makes some sense; but given that the text already exists, I'm wondering whether taking a line out of text would cause it to be transformed, only to be transformed again when appending? And in particular, why this would affect Windows only. I have also added tests using "is strictly a binary string" in the code above, and this was true for neither input 'tWorkTable', nor the output 'tNewTable', nor any of the 257,00 extracted lines. However it is definitely the accumulating of text that is the issue - simply looping over the lines - even with testing each one to see if it is "strictly a binary string" - is a second or less on Windows in LC9. Has anyone had similar experiences? Suggestions for how this could be avoided? Many thanks in advance, Ben ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode
Accumulating text is *VERY* slow in LC9 on Windows
Some 20 months ago, I reported that I was in a situation where an app written in 6.7 needed to be updated to access 64bit drivers, which meant updating to 9.5 - which displayed horrifying increase in processing time. In fact I was able to put off the evil day - but now it has returned, and can be put off no longer. A process that normally takes 2 hours is currently taking 9. The core processing stage has gone from around ten minutes to over six hours. After way too long, I've finally got down to at least one smoking gun; which is as simple as can be. Part of what took me so long is a confusion; in production the process runs on Windows, but I develop on Mac. Although on Mac the overall process does take about a third longer in LC9 than LC6, the simple tests I've finally isolated actually run much _quicker_ in LC9 than LC6. So switching between LC6 and LC9 on Mac as I tried to isolate the issue was giving confusing signals. But unmistakeably it's *much* slower on Windows. A simple routine which loops over a load of tab and return formatted data loaded from a TSV file, to truncate a particular field, had the following results processing a 70MB file of approximately 257,000 rows: 6.7.11 MacOS 9 seconds 6.7.11 Win32 10 seconds 9.6.3 MacOS 2 seconds 9.6.3 Win32498 seconds I simplified it down to this (pointless) loop which just rebuilds a table one line at a time: local tNewTable repeat for each line tRow in tWorkTable put tRow & return after tNewTable end repeat with these results: 6.7.11 MacOS 8 seconds 6.7.11 Win32 7 seconds 9.6.3 MacOS 0 seconds 9.6.3 Win32591 seconds (there's obviously a lot of variability in these - both were running in IDE, on a logged-in computer, so stuff was probably going on in the background; but I know the overall effect is similar when built as standalone and running by schedule on an unattended machine. But the key thing is: for this task, LC9 is dramatically slower on Windows!) Have others seen something like this? When I posted about this before (thread: "OMG text processing performance 6.7 - 9.5") Mark Waddingham suggested that it might be to do with a hidden cost of binary<->text transforms. That makes some sense; but given that the text already exists, I'm wondering whether taking a line out of text would cause it to be transformed, only to be transformed again when appending? And in particular, why this would affect Windows only. I have also added tests using "is strictly a binary string" in the code above, and this was true for neither input 'tWorkTable', nor the output 'tNewTable', nor any of the 257,00 extracted lines. However it is definitely the accumulating of text that is the issue - simply looping over the lines - even with testing each one to see if it is "strictly a binary string" - is a second or less on Windows in LC9. Has anyone had similar experiences? Suggestions for how this could be avoided? Many thanks in advance, Ben ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode
