Some 20 months ago, I reported that I was in a situation where an app written
in 6.7 needed to be updated to access 64bit drivers, which meant updating to
9.5 - which displayed horrifying increase in processing time.
In fact I was able to put off the evil day - but now it has returned, and can
be put off no longer. A process that normally takes 2 hours is currently
taking 9. The core processing stage has gone from around ten minutes to over
six hours.
After way too long, I've finally got down to at least one smoking gun; which
is as simple as can be.
Part of what took me so long is a confusion; in production the process runs on
Windows, but I develop on Mac. Although on Mac the overall process does take
about a third longer in LC9 than LC6, the simple tests I've finally isolated
actually run much _quicker_ in LC9 than LC6. So switching between LC6 and LC9
on Mac as I tried to isolate the issue was giving confusing signals. But
unmistakeably it's *much* slower on Windows.
A simple routine which loops over a load of tab and return formatted data
loaded from a TSV file, to truncate a particular field, had the following
results processing a 70MB file of approximately 257,000 rows:
6.7.11 MacOS 9 seconds
6.7.11 Win32 10 seconds
9.6.3 MacOS 2 seconds
9.6.3 Win32 498 seconds
I simplified it down to this (pointless) loop which just rebuilds a table one
line at a time:
local tNewTable
repeat for each line tRow in tWorkTable
put tRow & return after tNewTable
end repeat
with these results:
6.7.11 MacOS 8 seconds
6.7.11 Win32 7 seconds
9.6.3 MacOS 0 seconds
9.6.3 Win32 591 seconds
(there's obviously a lot of variability in these - both were running in IDE,
on a logged-in computer, so stuff was probably going on in the background; but
I know the overall effect is similar when built as standalone and running by
schedule on an unattended machine. But the key thing is: for this task, LC9 is
dramatically slower on Windows!)
Have others seen something like this?
When I posted about this before (thread: "OMG text processing performance 6.7
- 9.5") Mark Waddingham suggested that it might be to do with a hidden cost of
binary<->text transforms. That makes some sense; but given that the text
already exists, I'm wondering whether taking a line out of text would cause it
to be transformed, only to be transformed again when appending? And in
particular, why this would affect Windows only.
I have also added tests using "is strictly a binary string" in the code above,
and this was true for neither input 'tWorkTable', nor the output 'tNewTable',
nor any of the 257,00 extracted lines.
However it is definitely the accumulating of text that is the issue - simply
looping over the lines - even with testing each one to see if it is "strictly
a binary string" - is a second or less on Windows in LC9.
Has anyone had similar experiences? Suggestions for how this could be avoided?
Many thanks in advance,
Ben
_______________________________________________
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode
