Monte, Would it be possible to precalculate (ish) how big a string buffer is required from the size (memalloc) of the array? Then, if later down the process, it works out it doesn't have enough, it can add a bunch more memory to the buffer, thus reducing the frequency of buffer resizing. Or maybe do it in kind-of blocks of n bytes. I'm just spit-balling but I guess you get my thinking.
Sean On Thu, 11 Mar 2021 at 00:52, Monte Goulding via use-livecode < use-livecode@lists.runrev.com> wrote: > It’s probably most likely to do with the string buffer needing to be > constantly resized as the array is iterated for combine. Some googling > suggests Windows may have issues with this. Our strategy for growing string > buffers at the moment is to allocate just enough for the string. Mark would > need to chime in on whether growing the buffer exponentially would be > suitable. It would result in more memory being allocated than necessary but > much less frequent allocations so it depends on what’s most costly as > memory gets cheaper. > > Cheers > > Monte > > > On 11 Mar 2021, at 11:34 am, Sean Cole (Pi) via use-livecode < > use-livecode@lists.runrev.com> wrote: > > > > The code for 'Split': > > > > void MCArraysExecSplit(MCExecContext& ctxt, MCStringRef p_string, > > MCStringRef p_element_delimiter, MCStringRef p_key_delimiter, MCArrayRef& > > r_array) > > { > > if (MCStringSplit(p_string, p_element_delimiter, p_key_delimiter, ctxt . > > GetStringComparisonType(), r_array)) > > return; > > > > ctxt . Throw(); > > } > > > > > > vs > > > > The code for 'Combine' (// comments added by me): > > > > void MCArraysExecCombine(MCExecContext& ctxt, MCArrayRef p_array, > > MCStringRef p_element_delimiter, MCStringRef p_key_delimiter, > MCStringRef& > > r_string) > > { > > bool t_success; > > t_success = true; // Create a register to check progress success > > > > uindex_t t_count; // Create a new (t)emp counter for indices > > t_count = MCArrayGetCount(p_array); // Find out how many arrays there > are > > > > MCAutoStringRef t_string; // Create (t)emp string to store the result > > if (t_success) > > > > t_success = MCStringCreateMutable(0, &t_string); // t_success is always > > true here initially and is reset as true if t_string is now mutable, > false > > if not > > > > > > combine_array_t t_lisctxt; // create a new array object > > t_lisctxt . elements = nil; // initialise t_lisctxt array > > if (t_success) > > > > t_success = MCMemoryNewArray(t_count, t_lisctxt . elements); // make sure > > the array was created > > > > > > if (t_success) > > { > > > > t_lisctxt . index = 0; > > > > MCArrayApply(p_array, list_array_elements, &t_lisctxt); > > > > qsort(t_lisctxt . elements, t_count, sizeof(array_element_t), > > compare_array_element); // sort the elements > > > > for(uindex_t i = 0; i < t_count; i++) > > > > { // Loop through all indices > > > > MCAutoStringRef t_value_as_string; // create a (t)emp string for element > > value > > > > > > > > t_success = ctxt . ConvertToString(t_lisctxt . elements[i] . value, > > &t_value_as_string); // convert array value to string > > > > if (!t_success) > > > > break; skip if unable to convert to string > > > > > > t_success = > > > > (p_key_delimiter == nil || > > > > (MCStringAppend(*t_string, MCNameGetString(t_lisctxt . elements[i] . > key)) > > && > > > > MCStringAppend(*t_string, p_key_delimiter)))&& > > > > MCStringAppend(*t_string, *t_value_as_string) && > > > > (i == t_count - 1 || > > > > MCStringAppend(*t_string, p_element_delimiter)); // t_success is true if > > the array element and values are added correctly > > > > > > if (!t_success) > > > > break; // skip if unable to add value > > > > } > > > > } > > > > if (t_success) > > > > t_success = MCStringCopy(*t_string, r_string); // Copies the (t)emp > string > > into the (r)eturn string > > > > > > MCMemoryDeleteArray(t_lisctxt . elements); > > > > if (t_success) > > > > return; > > > > > > // Throw the current error code (since last library call returned false). > > ctxt . Throw(); > > } > > > > > > Following on from Bob's VM comment, there is reference to > > 'MCMemoryNewArray(t_count, > > t_lisctxt . elements)' which does highlight that some memory management > for > > the arrays is necessary in the combine command. This only creates a > > temporary copy of the array for working through. How this plays out > > differently for Windows vs Mac/Linux and why this would be increasing the > > time necessary by a factor of about 4:1 I can't see. > > > > I've tested as far back as LC7 > > (Times - Read into memory, Split to array, Combine from array) > > LC9.5.0 Win64 - 0.437s, 0.516s, 3m 1.378s > > LC9.0.5 Win32 - 0.446s, 0.547s, 3m 27.9s > > LC8.2.0 DP2 - 0.543s, 0.577s, 3m 30.208s > > LC8.0.0 - 0.542s, 0.545s, 3m 30.815s > > LC7.0.0 - 0.827s, 0.460s , 3m 37.896s > > > > On mac all times are less than 1sec, 3 sec total. > > > > Sean > > > > > > On Wed, 10 Mar 2021 at 17:08, Bob Sneidar via use-livecode < > > use-livecode@lists.runrev.com> wrote: > > > >> Now THAT is fascinating, considering the Windows performance issues with > >> file access reported in the past. Could it be that combine is somehow > >> caching data to virtual memory? > >> > >> Bob S > >> > >> > >> On Mar 9, 2021, at 1:05 PM, Sean Cole (Pi) via use-livecode < > >> use-livecode@lists.runrev.com<mailto:use-livecode@lists.runrev.com>> > >> wrote: > >> > >> It's looking to be a Windows only issue. I need to see how far this goes > >> back and then I'll post a bug report. It's making a process that should > >> only take 30s max on a single thread 2GHz remote Win server take 16mins > to > >> process 2 of these files, so it will be good to find a solution for > this. > >> > >> Thanks everyone for confirming and providing your input. > >> > >> Regards > >> Sean > >> > >> _______________________________________________ > >> use-livecode mailing list > >> use-livecode@lists.runrev.com > >> Please visit this url to subscribe, unsubscribe and manage your > >> subscription preferences: > >> http://lists.runrev.com/mailman/listinfo/use-livecode > >> > > _______________________________________________ > > use-livecode mailing list > > use-livecode@lists.runrev.com > > Please visit this url to subscribe, unsubscribe and manage your > subscription preferences: > > http://lists.runrev.com/mailman/listinfo/use-livecode > > > _______________________________________________ > use-livecode mailing list > use-livecode@lists.runrev.com > Please visit this url to subscribe, unsubscribe and manage your > subscription preferences: > http://lists.runrev.com/mailman/listinfo/use-livecode > _______________________________________________ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode