Re: Livecode performance problem
Paul Dupuis wrote: > For strange legacy application reasons, when lines get added to the > set of fields (a frequency action by users - sometimes adding hundreds > of lines a day), the data has to be repackaged into this tab delimited > structure in a single variable to run some procedures on. > > So a user driven action of: > > 1) Add a line (or a few) to the fields > 2) repackage data and run a few procedures > 3) User repeats starting at step 1 > > Can become very slow as the number of lines in the fields gets large > - as you all have noted! I'd just bite the bullet and replace those columnar fields with one delimited field. -- Richard Gaskin Fourth World Systems Software Design and Development for the Desktop, Mobile, and the Web ambassa...@fourthworld.comhttp://www.FourthWorld.com ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode
Re: Livecode performance problem
Wow, what a great example. I decided to work through the various suggestions just so I could get a better grip on some of LC’s more sophisticated data handling commands. My original run using Paul’s example was 8 seconds (2000 lines per field, 1100 chars per line) Applying Brian’s simple suggestions of using “after” instead of “into” cut that in half (4 seconds) Splitting the lines and using an array drove the time down to 23 milliseconds. Amazing! Between those 2 that's 8000 msec -> 23 msec or about 350 times faster. I will definitely pay more attention to “split” in future. Great discussion. Mark > On 20 Aug 2022, at 12:50 am, Paul Dupuis via use-livecode > wrote: > > On 8/19/2022 7:40 PM, Mark Wieder via use-livecode wrote: >> On 8/19/22 16:31, Alex Tweedly via use-livecode wrote: >> >>> to trim about another 15% off the time (for my sample data, 24ms down to >>> 20ms.) >> >> Nice. >> Note, of course, that we're all going on the assumption that all four fields >> contain the same number of lines. >> >> > > Thank you all for the example of improving performance over my clunky code. > > Yes, all the fields contain the same number of lines. > > For strange legacy application reasons, when lines get added to the set of > fields (a frequency action by users - sometimes adding hundreds of lines a > day), the data has to be repackaged into this tab delimited structure in a > single variable to run some procedures on. > > So a user driven action of: > > 1) Add a line (or a few) to the fields > 2) repackage data and run a few procedures > 3) User repeats starting at step 1 > > Can become very slow as the number of lines in the fields gets large - as you > all have noted! > > Thank you all again! > > > > ___ > use-livecode mailing list > use-livecode@lists.runrev.com > Please visit this url to subscribe, unsubscribe and manage your subscription > preferences: > http://lists.runrev.com/mailman/listinfo/use-livecode ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode
Re: Livecode performance problem
Short version: inserting between is a much slower process than after, as the entire field/string/whatever from the post of insertion has to be moved or otherwise handled. ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode
Re: Livecode performance problem
Thank you! The prior solutions have dramatically reduced the time - the WHOLE analytics process in its entirety because of the slowness of my old code was taking HOURS (like 6-10) - and is now down to 30-35 minutes with the prior solutions. I can't wait to tr the combine by columns method to see how much more (40%! WOW!) it reduced the overall process time further! On 8/20/2022 5:59 PM, Alex Tweedly via use-livecode wrote: I can't answer for anyone else, but for me - because I never thought about it :-) Combine by column - never used it before, so never thought of it. It's significantly faster, roughly 40% improvement over the previous best. Note, the spec requires that each line begins with the line number, so you need to add something like: local tmp repeat with i = 1 to the number of lines in g1 put i after tmp end repeat put tmp into x[1] put g1 into x[2] put g2 into x[3] put g3 into x[4] put g4 into x[5] and finish off with combine x by column put cr after x to fully match the earlier results. Alex. On 20/08/2022 18:45, David Epstein via use-livecode wrote: I didn’t text the speed, but why not put fld A into x[1] put fld B into x[2] put fld C into x[3] put fld D into x[4] combine x by column return x I have a set of fields, call them A, B, C, and D. Each has the same number of lines. Each field has different text (per line) I need to combine the data - frequently - into a form that look like: ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode
Re: Livecode performance problem
I can't answer for anyone else, but for me - because I never thought about it :-) Combine by column - never used it before, so never thought of it. It's significantly faster, roughly 40% improvement over the previous best. Note, the spec requires that each line begins with the line number, so you need to add something like: local tmp repeat with i = 1 to the number of lines in g1 put i after tmp end repeat put tmp into x[1] put g1 into x[2] put g2 into x[3] put g3 into x[4] put g4 into x[5] and finish off with combine x by column put cr after x to fully match the earlier results. Alex. On 20/08/2022 18:45, David Epstein via use-livecode wrote: I didn’t text the speed, but why not put fld A into x[1] put fld B into x[2] put fld C into x[3] put fld D into x[4] combine x by column return x I have a set of fields, call them A, B, C, and D. Each has the same number of lines. Each field has different text (per line) I need to combine the data - frequently - into a form that look like: ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode
Re: Livecode performance problem
I didn’t text the speed, but why not put fld A into x[1] put fld B into x[2] put fld C into x[3] put fld D into x[4] combine x by column return x > > I have a set of fields, call them A, B, C, and D. Each has the same > number of lines. Each field has different text (per line) > > I need to combine the data - frequently - into a form that look like: > > C> ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode
Re: Livecode performance problem
On 8/19/2022 7:40 PM, Mark Wieder via use-livecode wrote: On 8/19/22 16:31, Alex Tweedly via use-livecode wrote: to trim about another 15% off the time (for my sample data, 24ms down to 20ms.) Nice. Note, of course, that we're all going on the assumption that all four fields contain the same number of lines. Thank you all for the example of improving performance over my clunky code. Yes, all the fields contain the same number of lines. For strange legacy application reasons, when lines get added to the set of fields (a frequency action by users - sometimes adding hundreds of lines a day), the data has to be repackaged into this tab delimited structure in a single variable to run some procedures on. So a user driven action of: 1) Add a line (or a few) to the fields 2) repackage data and run a few procedures 3) User repeats starting at step 1 Can become very slow as the number of lines in the fields gets large - as you all have noted! Thank you all again! ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode
Re: Livecode performance problem
On 20/08/2022 00:03, Bob Sneidar via use-livecode wrote: It's probably a lot of text. The engine has to start from the beginning of every string then scan through for every cr or lf or cr/lf or whatever counts as a line break, until if finds the nth one. The more lines, the longer the scan takes each time, and the more text per line the exponentially more time it takes. Multiply that by 4 times plus the combinination of all of them as the code progresses *4 for the output string and you have the makings of a mountain that keeps getting steeper the higher you go. Yes, for all the input strings. For the output string, it's just a "put ... after ...", so there is no need to count or scan the output string; LC already keeps (effectively) a pointer to the end of a string, and optimizes the straightforward extension at the end of a string. Alex. ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode
Re: Livecode performance problem
On 8/19/22 16:03, Bob Sneidar via use-livecode wrote: It's probably a lot of text. The engine has to start from the beginning of every string then scan through for every cr or lf or cr/lf or whatever counts as a line break, until if finds the nth one. The more lines, the longer the scan takes each time, and the more text per line the exponentially more time it takes. Multiply that by 4 times plus the combinination of all of them as the code progresses *4 for the output string and you have the makings of a mountain that keeps getting steeper the higher you go. My tests were based on 2174 lines of text in each field. Got nowhere near 20 seconds. Or even one. -- Mark Wieder ahsoftw...@gmail.com ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode
Re: Livecode performance problem
On 8/19/22 16:31, Alex Tweedly via use-livecode wrote: to trim about another 15% off the time (for my sample data, 24ms down to 20ms.) Nice. Note, of course, that we're all going on the assumption that all four fields contain the same number of lines. -- Mark Wieder ahsoftw...@gmail.com ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode
Re: Livecode performance problem
On 19/08/2022 23:32, Mark Wieder via use-livecode wrote: It is indeed faster. Here's what I came up with. I'm not sure why 2000 lines of text in four fields should take that long, I came up with original code: 320 ms array version: 21 ms put empty into vCombined put fld "A" into v1 put fld "B" into v2 put fld "C" into v3 put fld "D" into v4 split v1 by cr split v2 by cr split v3 by cr split v4 by cr put 1 into i repeat for each element tLine in v1 put i[i][i][i][i] after vCombined add 1 to i end repeat which is already quick enough that any further improvement is mainly academic. But for the record: "repeat for each line " is very efficient for a single variable, o you can avoid one of the four 'split's, and use the line variable to count your loop. -- split v1 by cr split v2 by cr split v3 by cr split v4 by cr put 1 into i repeat for each line tLine in v1 put i& tLine & v2[i] & v3[i] & v4[i] & cr after vCombined add 1 to i end repeat to trim about another 15% off the time (for my sample data, 24ms down to 20ms.) (and if you know that one of the four fields typically contains much more text (i.e. longer lines) than the others, you would choose it to not be split). Alex. ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode
Re: Livecode performance problem
It's probably a lot of text. The engine has to start from the beginning of every string then scan through for every cr or lf or cr/lf or whatever counts as a line break, until if finds the nth one. The more lines, the longer the scan takes each time, and the more text per line the exponentially more time it takes. Multiply that by 4 times plus the combinination of all of them as the code progresses *4 for the output string and you have the makings of a mountain that keeps getting steeper the higher you go. That is my understanding at least. Bob S On Aug 19, 2022, at 15:32 , Mark Wieder via use-livecode mailto:use-livecode@lists.runrev.com>> wrote: It is indeed faster. Here's what I came up with. I'm not sure why 2000 lines of text in four fields should take that long, I came up with ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode
Re: Livecode performance problem
On 8/19/22 15:07, Bob Sneidar via use-livecode wrote: Off the top of my head: split v1 by tab split v2 by tab split v3 by tab split v4 by tab put the keys of v1 into tKeyList sort tKeyList ascending numeric repeat for each line tKey in tKeyList put tKey & tab & v1 [tKey] & tab & v2 [tKey] & tab & v3 [tKey] & tab & v4 [tKey] & cr after tCombined end repeat Not sure if this will be faster, but every time you refer to a line of a string the engine has to parse out where that line is from the beginning. It is indeed faster. Here's what I came up with. I'm not sure why 2000 lines of text in four fields should take that long, I came up with original code: 320 ms array version: 21 ms put empty into vCombined put fld "A" into v1 put fld "B" into v2 put fld "C" into v3 put fld "D" into v4 split v1 by cr split v2 by cr split v3 by cr split v4 by cr put 1 into i repeat for each element tLine in v1 put i[i][i][i][i] after vCombined add 1 to i end repeat -- Mark Wieder ahsoftw...@gmail.com ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode
Re: Livecode performance problem
Based on what Bob said, here is my version of that handler: on mouseUp local tA, tB, tC, tD, tAll, tStart, tEnd put fld "A" into tA put fld "B" into tB put fld "C" into tC put fld "D" into tD -- put the milliseconds into tStart split tA by cr split tB by cr split tC by cr split tD by cr repeat with i=1 to the number of lines of the keys of tA put i & tab & tA[i] & tab & tB[i] & tab & tC[i] & tab & tD[i] & cr after tAll end repeat delete the last char of tAll put the milliseconds into tEnd put (tEnd - tStart) & cr & tAll into fld "result" end mouseUp I used very short lines, but this way was 3/4ms; my initial method was 6/7ms; your original method was 15/16ms. On Fri, Aug 19, 2022 at 6:09 PM Bob Sneidar via use-livecode < use-livecode@lists.runrev.com> wrote: > Off the top of my head: > > split v1 by tab > split v2 by tab > split v3 by tab > split v4 by tab > > put the keys of v1 into tKeyList > sort tKeyList ascending numeric > > repeat for each line tKey in tKeyList >put tKey & tab & v1 [tKey] & tab & v2 [tKey] & tab & v3 [tKey] & tab & > v4 [tKey] & cr after tCombined > end repeat > > Not sure if this will be faster, but every time you refer to a line of a > string the engine has to parse out where that line is from the beginning. > Parsing times will increase exponentially (is that the right term?) the > more lines you are parsing in both the source strings AND the destination > string. > > But arrays are indexed memory locations so the engine knows exactly where > every bit of text is. And putting something after a string does not have to > parse to find the end, the engine knows where that is in memory. It might > be even faster though if you build an array as output and then use the > combine command to convert it all at once to a string. > > I am assuming of course that splitting does not put lines of text in > random order in the array. Arrays can be dicey that way. > > Bob S > > > > > On Aug 19, 2022, at 14:09 , Paul Dupuis via use-livecode < > use-livecode@lists.runrev.com> wrote: > > > > I have a set of fields, call them A, B, C, and D. Each has the same > number of lines. Each field has different text (per line) > > > > I need to combine the data - frequently - into a form that look like: > > > > C> > > > > For the number of lines in the set of fields. Currently I do this as > follows: > > > > put empty into vCombined > > put fld A into v1 > > put fld B into v2 > > put fld C into v3 > > put fld D into v4 > > repeat with i=1 to the number of lines in v1 > > put i & line i of v1 & line i of v2 & line i of v3 & > line i of v4 into line i of vCombined > > end repeat > > > > I put the field contents into variable before the loop to combine them > as my understanding is variable access is faster than field access in a loop > > > > My question to the Hivemind is: Is there a noticeably faster way to > covert these field to the tab delimited structure as show above than the > code above? > > > > This current takes about 20 seconds for 2000 lines or about a second per > 100 lines (very roughly, I was not using a timer in my code, just my > wristwatch seconds in the IDE debugger between breakpoints) > > > > Thank you for any pointers in advance, > > > > Paul Dupuis > > Researchware > > > > > > ___ > > use-livecode mailing list > > use-livecode@lists.runrev.com > > Please visit this url to subscribe, unsubscribe and manage your > subscription preferences: > > http://lists.runrev.com/mailman/listinfo/use-livecode > > > ___ > use-livecode mailing list > use-livecode@lists.runrev.com > Please visit this url to subscribe, unsubscribe and manage your > subscription preferences: > http://lists.runrev.com/mailman/listinfo/use-livecode > ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode
Re: Livecode performance problem
First optimization would be to put … & cr after vCombined and then delete the trailing from when done. Sent from my iPhone > On Aug 19, 2022, at 5:11 PM, Paul Dupuis via use-livecode > wrote: > > I have a set of fields, call them A, B, C, and D. Each has the same number > of lines. Each field has different text (per line) > > I need to combine the data - frequently - into a form that look like: > > C> > > For the number of lines in the set of fields. Currently I do this as follows: > > put empty into vCombined > put fld A into v1 > put fld B into v2 > put fld C into v3 > put fld D into v4 > repeat with i=1 to the number of lines in v1 > put i & line i of v1 & line i of v2 & line i of v3 & line i > of v4 into line i of vCombined > end repeat > > I put the field contents into variable before the loop to combine them as my > understanding is variable access is faster than field access in a loop > > My question to the Hivemind is: Is there a noticeably faster way to covert > these field to the tab delimited structure as show above than the code above? > > This current takes about 20 seconds for 2000 lines or about a second per 100 > lines (very roughly, I was not using a timer in my code, just my wristwatch > seconds in the IDE debugger between breakpoints) > > Thank you for any pointers in advance, > > Paul Dupuis > Researchware > > > ___ > use-livecode mailing list > use-livecode@lists.runrev.com > Please visit this url to subscribe, unsubscribe and manage your subscription > preferences: > http://lists.runrev.com/mailman/listinfo/use-livecode ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode
Re: Livecode performance problem
Off the top of my head: split v1 by tab split v2 by tab split v3 by tab split v4 by tab put the keys of v1 into tKeyList sort tKeyList ascending numeric repeat for each line tKey in tKeyList put tKey & tab & v1 [tKey] & tab & v2 [tKey] & tab & v3 [tKey] & tab & v4 [tKey] & cr after tCombined end repeat Not sure if this will be faster, but every time you refer to a line of a string the engine has to parse out where that line is from the beginning. Parsing times will increase exponentially (is that the right term?) the more lines you are parsing in both the source strings AND the destination string. But arrays are indexed memory locations so the engine knows exactly where every bit of text is. And putting something after a string does not have to parse to find the end, the engine knows where that is in memory. It might be even faster though if you build an array as output and then use the combine command to convert it all at once to a string. I am assuming of course that splitting does not put lines of text in random order in the array. Arrays can be dicey that way. Bob S > On Aug 19, 2022, at 14:09 , Paul Dupuis via use-livecode > wrote: > > I have a set of fields, call them A, B, C, and D. Each has the same number of > lines. Each field has different text (per line) > > I need to combine the data - frequently - into a form that look like: > > C> > > For the number of lines in the set of fields. Currently I do this as follows: > > put empty into vCombined > put fld A into v1 > put fld B into v2 > put fld C into v3 > put fld D into v4 > repeat with i=1 to the number of lines in v1 > put i & line i of v1 & line i of v2 & line i of v3 & line i > of v4 into line i of vCombined > end repeat > > I put the field contents into variable before the loop to combine them as my > understanding is variable access is faster than field access in a loop > > My question to the Hivemind is: Is there a noticeably faster way to covert > these field to the tab delimited structure as show above than the code above? > > This current takes about 20 seconds for 2000 lines or about a second per 100 > lines (very roughly, I was not using a timer in my code, just my wristwatch > seconds in the IDE debugger between breakpoints) > > Thank you for any pointers in advance, > > Paul Dupuis > Researchware > > > ___ > use-livecode mailing list > use-livecode@lists.runrev.com > Please visit this url to subscribe, unsubscribe and manage your subscription > preferences: > http://lists.runrev.com/mailman/listinfo/use-livecode ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode
