player bug

[Neville via use-livecode]

Has this been reported already?

if I try to play a non-existent audio file in the player object (LC 9.5.1 on 
Mac) LC stop responding to events - no menus or input work except for moving 
windows ; not exactly a hang, since the Activity Monitor says LiveCode is still 
working. Force Quit is the only way out.

Neville
___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode

RE: OMG text processing performance 6.7 - 9.5 - correction

[Neville via use-livecode]

Belay my claim about the offsets found from using an offset search on raw text 
and on the utf-8 version of that text giving exactly the same offset numbers 
for corresponding hits - they don’t of course. The offsets reported in the raw 
text are binary 8-bit character offsets, the offsets reported in the utf-8 
encoded text are unicode character offsets, as they must be.

Apologies, I was reading my data incorrectly.

Neville


___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode

RE: OMG text processing performance 6.7 - 9.5

[Neville via use-livecode]

One further comment … when  talking about long text and not using lineOffset I 
really do mean long. 

Using source text the first 500 KB of Les Miserables, the times for simple 
Parse 1 offset search with skip for *both* raw text and utf-8 was 1 ms, and for 
lineOffset 10 ms and 14 ms respectively, not worth worrying about doing 
anything fancy.

Neville


___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode

RE: OMG text processing performance 6.7 - 9.5

[Neville via use-livecode]

Richard, here is a link to my test stack

https://www.dropbox.com/sh/bbpe12p8bf56ofe/AADbhV2LavLP4Y3CZ8Ab8NGia?dl=0 


The LesMiserables.txt file is included for convenience; it should be placed in 
your Documents directory. The algorithms are all in the script for the `Run` 
button.

I am still mystified that the character offset searches give the same number 
for each hit for the utf8 text as for the raw text. Surely `char x of 
theUTF8Text` returns the unicode character at offset x, `char x of theRawText` 
returns the 8-bit ascii character of the raw text? How then can x be the same 
for the corresponding hit, when I know there are some multibyte unicode 
characters in the text (eg e-acute in Miserables)? Indeed just what does 
textDecode(theRawText,`UTF-8`) do, does it modify the actual text at all or 
just set a property flag?

Another mystery: I decided to extend the search algorithms by adding in 
matchChunk. In this case I use the regular expression `(?m)(?i)(Valjean)` to 
get the start and end offsets of the first match, and then truncate the initial 
section as per Parse2. As expected this search is much slower than any of the 
others on the raw text, it has a lot more to do. I then expected to get around 
the same time for the search on utf8 text rather than an exponentially worse 
time, since matchChunk is presumably encoding-blind. But it is actually 15% 
faster than on the raw text, in fact it is the fastest for finding offsets of 
all the algorithms if you must* search utf8 text ! How can this be? I don’t 
believe the utf8 text is 15% smaller than the raw text!

searches on raw text
matchChunk3059 ms
filter  16 ms
parse0  10 ms
parse18 ms
parse3  2244 ms
parse2671 ms
parse4682 ms

searches on utf-8 text
matchChunk utf8  2492 ms
filter utf8   1954 ms
parse0 utf8   3788 ms
parse1 utf8   223254 ms
parse3 utf8   634423 ms
parse2 utf8   3409 ms
parse4 utf8   7166 ms

*As I mentioned in most case character offset searching the raw text should be 
OK if you are searching for 7-bit ascii strings of length say>2. But I think 
the lineOffset and filter operations could give false positives, if there is a 
multibyte character contains OD as a component byte in the text.

Neville


___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode

RE: OMG text processing performance 6.7 - 9.5

[Neville via use-livecode]

Just for interest, and to see just how slow lineOffset is, I added a couple of 
more tests to the search for occurrences of “Valjean” in the Gutenberg English 
translation of Les Miserables. I also wanted find how filter performs.

The searches were first applied to the raw binary text as read from the utf-8 
encoded file, without using textDecode; then on the text utf-8 decoded

Parse 0 : using itemdelimiter  ‘Valjean’ (case insensitive)
Parse 1: using offset with skips
Parse 2: using offset, truncating the text and 0 skip
Parse 3: use lineOffset with skips
Parse 4: use lineOffset, truncating the text and 0 skip
filter: use filter to find lines containing '*Valjean*'

Parse 1 and 2 produced 1120 hits. Parse 0 gave 1121 hits, the extra one being a 
false positive at the end of the file, which needs to be accounted for in an 
implementation. I was slightly surprised that the character offsets produced 
were the same for raw and for utf-8 text, I guess I was expecting the latter to 
give the unicode character offset. Parse 3 and 4 and filter all output 1099 
lines. 

Results:

searches on raw text:

parse0  11 ms
parse19 ms
parse2751 ms
parse3  2551 ms
parse4753 ms
filter  16 ms

searches on utf-8 text:

parse04386 ms
parse1224367 ms
parse23461 ms
parse3636554 ms —— 
parse47242 ms
filter2258 ms

So for long texts it is best to use raw binary text and search with character 
offset(pSt,pSrc,skip) [Parse 1]. If you have to search on utf-8 encode text 
then use Parse 2, deleting initial sections of the text as you go. Never use 
lineOffset (except for small text) even if that means extra code to find line 
endings on either side of the character offset when you really want the found 
line. If you don’t actually need the offset of the hits in the original file - 
for example for editing the original - then filter is the fastest on long text 
and just as fast on short text, but depending on your needs you probably have 
to do another search on the filtered text; but this would be a viable approach 
if the number of lines produced is itself small.

Neville



___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode

RE: OMG text processing performance 6.7 - 9.5

[Neville via use-livecode]

The recent testing of the Parse1 and Parse2  algorithms I think must have been 
on ascii not utf-8 text

I tested on the English translation of Les Miserables, to ensure at least a 
sprinkling of multi-bite characters in the text, and a longish file: 3.4 MB. I 
tested for the search string ‘Valjean’ which obviously occurs very frequently.

The searches were first applied to the raw binary text as read from the utf-8 
encoded file, without decoding; then on the text utf-8 decoded

Parse 0 : using itemdelimiter  ‘Valjean’ (case insensitive)

Parse 1: using offset with skips

Parse 2: using offset, truncating the text and 0 skip

Results:

searches on raw text
parse0 10 ms
parse1 9 ms
parse2 708 ms

searches on utf-8text
parse0 4402 ms
parse1 225469 ms
parse2 3453 ms


The winner for long utf-8 text is Parse 2; for raw text Parse1 and Parse 0 are 
equivalent The results dramatically demonstrate the exponential decay in 
performance with long utf-8 text. 

For most searches I would think one could use the raw text as long as one was 
searching for an ascii string, false positives where the string of single bytes 
occurs inside multibyte characters would be extremely unlikely.

Neville



___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode

Re: OMG text processing performance 6.7 - 9.5

[Neville via use-livecode]

Are you perchance using lineOffset searches? I have found that lineOffset 
performance on utf8 text degrades exponentially with the length of the file, 
presumably as it searches for line breaks. Use offset instead which remains 
fast (and much faster still if you can search on the raw text before 
textencoding, then utf8 encode the found chunks)
___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode

Re: windows standalone performance

[Neville via use-livecode]

> My user who initially reported the bug to me has acquired a new PC on which 
> the bug does not seem to happen

I spoke too soon … my user's whiz-bang PC still takes 3 seconds to save my 
app's 9 MB data stack, when his older PC takes 10 seconds. Both should be 
saving in a very small fraction of a second.


___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode

Re: windows standalone performance

[Neville via use-livecode]

> Sean wrote: From what I can tell from QC is that it has not been confirmed 
> and been marked as pending due to not being able to reproduce.

The bug is clearly marked as status CONFIRMED.

There was a stage when QC could not reproduce the bug on their machines, but 
Panos did confirm it “on older machines”. Brian Milby has also seen the bug, as 
well as of course our latest correspondent.

I have just confirmed the bug still exists under LC9.5.1, in both standalone 
and the IDE, in Parallels Desktop on a Mac Pro running the latest Windows 10, 
typically taking 7 seconds to save a 8MB stack — of course that is emulating 
the OS but I believe speeds on native hardware are comparable (or worse).  

My user who initially reported the bug to me has acquired a new PC on which the 
bug does not seem to happen (tho’ it is a blindingly fast machine!) so as most 
of my users are Mac people I lost interest in the bug until this discussion. 
Not being a Windows user myself I am not going to research the actual processes 
and cannot assess on what hardware configuration the bug occurs --- clearly not 
all PC’s are affected.

If you want to test on your own PC a test stack (actually a wrapper standalone 
slowSave.app and a data stack data.livecode, together with the source 
slowSave.livecode) can be downloaded from

https://www.dropbox.com/sh/cb2r9jbohxqv6bp/AAAQ1weLLlzrKYQ21yn1apf9a?dl=0 





___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode

Re: windows standalone performance

[Neville via use-livecode]

Hi Sean

The slow saving of stacks under LC9 occurs in Windows 10, native and using 
Parallels Desktop.
https://quality.livecode.com/show_bug.cgi?id=21305 


Mac, Linux and previous versions of Windows are not affected.

The bug was confirmed by Quality Control although not initially; so  it is 
evidently related to some particular configuration of Windows 10 - although my 
Mac Parallels setup is out-of-the-box as was the native Windows machine of the 
user who reported it to me. It is not affected by turning off Windows Defender 
and AV software. There is nothing special I can think of for the particular 
stack being saved (except it is large, to demonstrate the saving speed); the 
example stack is in the bug report. When the "save stack” handler activates you 
can see the temporary file created on the desktop of Explorer immediately, but 
then a long wait for the file write to complete.

Saving a file from LC, rather than a stack, is not any slower than for other 
Windows apps (so sqlite performance is presumably not affected).

Neville
___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode

Re: Windows Standalone Performance

[Neville via use-livecode]

This bug was reported some time ago 
https://quality.livecode.com/show_bug.cgi?id=21305 


Saving a 9 megabyte stack in Windows 10 can take 10 seconds or more.

It is not caused by Windows Defender or anti-virus software.
___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode

Catalina mail recovery stack

[Neville via use-livecode]

After installing Catalina macOS I (along with lots of other people) lost a 
whole swag of mail; the problem seems to occur after moving a message between 
mailboxes, possibly before Mail had finished updating itself, but may have 
other triggers. Many messages simply went totally missing, and lots of message 
(in custom mailboxes) are shown with a header but no content. Remedies 
suggested in various fora haven’t worked: rebuilding mailboxes didn’t help, 
even with manual removal of the Mail.app Envelope databases, and Time Machine 
cannot be used at least in its in its simple mode to recover mail. With over 
85000 emails and the byzantine directory structure of the email messages, there 
is no way I can manually search for the missing email in the backups.

I am writing a stack to create a custom sqlite database of messages from the 
standard Mac ~/Library/Mail location, and from Time Machine backup locations, 
and include tools to compare and find emails. So far it seems to be successful 
in finding lost messages; I have yet to add tools to reimport to Mail.app.

If anyone has this lost mail problem (and has a Time Machine or other backup) 
and would like my stack please email me. The stack is rough for my own use at 
the moment but I can polish it for general use.

neville.smy...@optusnet.com .au
___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode