Re: math, bigly
On 08/09/2017 11:05 AM, Mark Waddingham via use-livecode wrote: Fortunately, there is a potential carrot to go with the stick - we can expand the range of the bit operations (and baseConvert) to 52-bits without too much difficulty I think; and add error checking at the same time. Which would potentially make code which is currently broken 'at the edges' (i.e. rarely touched) not ever encounter an error. As always, I'm awed by the brilliance of your proposed solution. I can't see any downside to this other than the time it would take to implement and test. -- Mark Wieder ahsoftw...@gmail.com ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode
Re: math, bigly
On 08/09/2017 10:15 AM, Mark Waddingham via use-livecode wrote: ...although that scale is relatively common in 'high-end' cryptography settings (factoring large numbers for the purposes of generating keys and such). Heh. And that, of course, is exactly what I was working on, and what got me into trouble. In the future if you're going to continue reading my mind, could you please stop me *before* I go down these rabbit holes? -- Mark Wieder ahsoftw...@gmail.com ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode
Re: math, bigly
Is it a good idea to implement a bignum library in LC Builder? ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode
Re: math, bigly
On 2017-08-09 18:50, Richard Gaskin via use-livecode wrote: Imagine the frustration of passing an invalid object reference to something like the delete command, and rather than reporting that it couldn't do what it asked you it just arbitrarily deleted some other object instead without notification. ;) Indeed - that would be frustrating... However, everyone writes code which relies on 'undefined behavior' (which not throwing errors generally results in) at some point - so there is a real issue with adding errors for things after the fact... Particularly as this has been unchecked for the entire history of the engine so a purely stick approach generally just causes user frustration (and the occasional rant) [ this is not to say that bounds-checking in this case would not be better in all ways one could potential imagine ]. Fortunately, there is a potential carrot to go with the stick - we can expand the range of the bit operations (and baseConvert) to 52-bits without too much difficulty I think; and add error checking at the same time. Which would potentially make code which is currently broken 'at the edges' (i.e. rarely touched) not ever encounter an error. Warmest Regards, Mark. -- Mark Waddingham ~ m...@livecode.com ~ http://www.livecode.com/ LiveCode: Everyone can create apps ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode
Re: math, bigly
Mark Wieder wrote: > But what annoys me isn't the fact that baseconvert, etc (bitXor is the > one that bit me) only work on integers less than 2^32. It's more the > fact that the input data was crunched down to something the functions > could handle, and then give me the wrong result with no warning. I got bit by that once myself. Sure, we can read all notes for all Dict entries for each token we use, but most of the time the engine does a good job of informing us when we're using invalid inputs, so it seems reasonable that it would do the same here. Would adding a size check and error handling impair performance much? Also, yes, it would be nice to see it handle larger values. But regardless of size, being a scripting engine, and one with generally good error reporting in most other contexts, it would be nice if invalid inputs were reported with baseConvert. Imagine the frustration of passing an invalid object reference to something like the delete command, and rather than reporting that it couldn't do what it asked you it just arbitrarily deleted some other object instead without notification. ;) -- Richard Gaskin Fourth World Systems Software Design and Development for the Desktop, Mobile, and the Web ambassa...@fourthworld.comhttp://www.FourthWorld.com ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode
Re: math, bigly
On 2017-08-09 19:15, Bob Sneidar via use-livecode wrote: OIC. I was under the impression that the C math libraries deal with numbers of high precision better than Livecode does, perhaps themselves generating errors on overflow. Maybe that is not the case. C itself does no error checking/bounds checking at all. It is only relatively recently that compiler intrinsics have been available on all platforms we support which would allow an overflow check to be done efficiently (access to the carry flag). Now, that would be fine if we used integers - which we don't - we use IEEE doubles. Of course, the IEEE standard has long had in it the 'exceptions' model for detecting overflow/inexactness and such - however using *that* to efficiently do anything is nigh-on impossible universally. Performance of using the FP exceptions (even non-signalling) is highly variable across processor models and OS combinations. Ideally LiveCode would use decimal arithmetic everywhere from the outside at least (with a fixed controllable precision - number of digits after the decimal point - for anything 'inexact'), and internally use infinite precision integers when there is nothing after the decimal point. That's more an engineering problem in terms of ensuring that in moving to such a model, performance of numeric operations (in general) does not decrease too much (there would be an overhead, its just trying to minimise it which causes all the hard work). Generally C math libraries are infinite precision or nothing - although IBM has some nifty IEEE Decimal emulation code out in the wild blending that with other numeric representations is still not an easy task. Warmest Regards, Mark. -- Mark Waddingham ~ m...@livecode.com ~ http://www.livecode.com/ LiveCode: Everyone can create apps ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode
Re: math, bigly
On 2017-08-09 19:11, Mark Wieder via use-livecode wrote: The problem I've run into is more at a higher level, dealing with number parsing and runtime overflow detection before truncating input parameters down to unsigned integers and calling the math library. Well there's two things here... 1) An 'easy' language should tell you when you are doing something wrong 2) An 'easy' language shouldn't force you to have to write reasonable things yourself Of course, in the absence of (2), (1) is much more important ;) Warmest Regards, Mark. -- Mark Waddingham ~ m...@livecode.com ~ http://www.livecode.com/ LiveCode: Everyone can create apps ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode
Re: math, bigly
On 08/09/2017 09:32 AM, Mark Waddingham via use-livecode wrote: Please do file a bug about it - it is, at the very least, an anomaly. Done. http://quality.livecode.com/show_bug.cgi?id=20254 -- Mark Wieder ahsoftw...@gmail.com ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode
Re: math, bigly
OIC. I was under the impression that the C math libraries deal with numbers of high precision better than Livecode does, perhaps themselves generating errors on overflow. Maybe that is not the case. Bob S > On Aug 9, 2017, at 10:11 , Mark Wieder via use-livecode > wrote: > > On 08/09/2017 09:56 AM, Bob Sneidar via use-livecode wrote: >> I wonder how much leveraging SQLite can help. SQLite has only basic math >> functions, and you cannot use stored functions, but you CAN map functions to >> a C API. And if you can do THAT, then someone has written a math API, and >> maybe that is a good place to start looking. > > The problem isn't with the math libraries. There's nothing wrong with the way > the engine deals with math at that level, and I wouldn't expect SQLite to use > anything other than the standard math libraries. > > The problem I've run into is more at a higher level, dealing with number > parsing and runtime overflow detection before truncating input parameters > down to unsigned integers and calling the math library. > > -- > Mark Wieder > ahsoftw...@gmail.com ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode
Re: math, bigly
On 2017-08-09 18:56, Bob Sneidar via use-livecode wrote: Btw, I should mention that the way numbers work in LiveCode has become one of my biggest 'pet peeves' over the last year or so. The current way things work is too simplistic really - it hides the very important distinction between 'exact' and 'inexact' numeric representations, for example. Unfortunately, there's a fair amount of work involved to fix the situation to my satisfaction level - although I think I have a handle on how to do so without hugely adversely affecting general numeric performance. *sigh* It isn't doing the math that is hard - there are plenty of libraries which do general precision arithmetic. Even standard 'high-school' algorithms work well for reasonable magnitudes and you only really need to get out the big guns - e.g. the Strassen algorithm - for very large numbers, although that scale is relatively common in 'high-end' cryptography settings (factoring large numbers for the purposes of generating keys and such). The main issue is working out how to do it so that the ability to handle large numeric precision does not impact the performance of the normal sized numeric precision operations. There's also the exact <-> inexact boundary issue too (going from integer/rational -> irrational) - realistically modern processors are only really good at (64-bit) integers, or (64-bit) IEEE floating point values. What you really want is to use machine sized integers for all computations which don't require anything greater than that (i.e < 64-bits these days) *without* penalty; but still allow code to be written which can handle large integers too. I think I've figured out a reasonable way to do that - but it requires type / range analysis. The exact / inexact thing is still troublesome though - in other languages 10 means something different from 10.0. Maybe that's actually reasonable, but part of me feels it is a little subtle. Certainly infinite precision integers would be exceptionally handy - as you also get arbitrary length bitsets like that too (there's an equivalence between dense bitsets and integers, unsurprisingly, you can use the two interchangeably). In any case, the first step before doing anything in this regard is to do a little bit more work on our 'Number' representation internally - and make the engine abstract through its API for various operations rather than just using +/- etc. directly in the implementation of syntax - then we would be able to more easily experiment with internal representations. Warmest Regards, Mark. -- Mark Waddingham ~ m...@livecode.com ~ http://www.livecode.com/ LiveCode: Everyone can create apps ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode
Re: math, bigly
On 08/09/2017 09:56 AM, Bob Sneidar via use-livecode wrote: I wonder how much leveraging SQLite can help. SQLite has only basic math functions, and you cannot use stored functions, but you CAN map functions to a C API. And if you can do THAT, then someone has written a math API, and maybe that is a good place to start looking. The problem isn't with the math libraries. There's nothing wrong with the way the engine deals with math at that level, and I wouldn't expect SQLite to use anything other than the standard math libraries. The problem I've run into is more at a higher level, dealing with number parsing and runtime overflow detection before truncating input parameters down to unsigned integers and calling the math library. -- Mark Wieder ahsoftw...@gmail.com ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode
Re: math, bigly
I wonder how much leveraging SQLite can help. SQLite has only basic math functions, and you cannot use stored functions, but you CAN map functions to a C API. And if you can do THAT, then someone has written a math API, and maybe that is a good place to start looking. Bob S https://stackoverflow.com/questions/2108870/how-to-create-custom-functions-in-sqlite https://sqlite.org/c3ref/create_function.html > On Aug 9, 2017, at 09:42 , Mark Waddingham via use-livecode > wrote: > > Btw, I should mention that the way numbers work in LiveCode has become one of > my biggest 'pet peeves' over the last year or so. The current way things work > is too simplistic really - it hides the very important distinction between > 'exact' and 'inexact' numeric representations, for example. > > Unfortunately, there's a fair amount of work involved to fix the situation to > my satisfaction level - although I think I have a handle on how to do so > without hugely adversely affecting general numeric performance. *sigh* > > Warmest Regards, > > Mark. > > -- > Mark Waddingham ~ m...@livecode.com ~ http://www.livecode.com/ > LiveCode: Everyone can create apps ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode
Re: math, bigly
On 2017-08-09 18:29, Mark Wieder via use-livecode wrote: Looking into the engine code, by the time you get to the actual bitXor code, the input arguments are already unsigned integers. Since I was passing 20-hex-digit values to bitXor, I would have expected some kind of runtime warning or error or something. It's not just a loss of precision here, it's a totally wrong answer with no indication that anything has gone astray. Writing a string-oriented function to handle this wasn't a problem - realizing that something had gone wrong was. Btw, I should mention that the way numbers work in LiveCode has become one of my biggest 'pet peeves' over the last year or so. The current way things work is too simplistic really - it hides the very important distinction between 'exact' and 'inexact' numeric representations, for example. Unfortunately, there's a fair amount of work involved to fix the situation to my satisfaction level - although I think I have a handle on how to do so without hugely adversely affecting general numeric performance. *sigh* Warmest Regards, Mark. -- Mark Waddingham ~ m...@livecode.com ~ http://www.livecode.com/ LiveCode: Everyone can create apps ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode
Re: math, bigly
Hi Mark. I remember an old trick to determine if a math function was overflowing. Run the equasion backwards and see if that result equals the initial value. This will reveal rounding errors too. Bob S > On Aug 8, 2017, at 18:38 , Mark Wieder via use-livecode > wrote: > > Phooey... > > put baseconvert("10001",16,10) > > ...and yes, I understand what's going on under the hood, but I'm annoyed that > the IDE doesn't give me an overflow warning if I try to perform an operation > on a number that's bigger than 0x. Instead, it just happily trims it > down to an unsigned integer and proceeds to give me a wrong result. I came > across this yesterday handing two large numbers to the bitXor operand which > threw no errors and went on with the rest of the calculation. I had to > backtrack to figure out what was going on, and finally ended up writing my > own bitwise operators to deal with this. > > -- > Mark Wieder > ahsoftw...@gmail.com ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode
Re: math, bigly
On 2017-08-09 18:29, Mark Wieder via use-livecode wrote: Looking into the engine code, by the time you get to the actual bitXor code, the input arguments are already unsigned integers. Since I was passing 20-hex-digit values to bitXor, I would have expected some kind of runtime warning or error or something. It's not just a loss of precision here, it's a totally wrong answer with no indication that anything has gone astray. Writing a string-oriented function to handle this wasn't a problem - realizing that something had gone wrong was. Indeed, I don't disagree - but its how things have ALWAYS been, which means that throwing an error will likely break someone's app somewhere which has come to rely on the behavior the way it is... Please do file a bug about it - it is, at the very least, an anomaly. Warmest Regards, Mark. -- Mark Waddingham ~ m...@livecode.com ~ http://www.livecode.com/ LiveCode: Everyone can create apps ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode
Re: math, bigly
On 2017-08-09 18:14, Mark Wieder via use-livecode wrote: Oops. OK - point sheepishly taken. Is that the definition of integer in the dictionary? The "integer" entry just says "if the value contains digits (and an optional minus sign) and no other characters, it is an integer", which is as simplisitic a definition as the entry for "hex". Technically an integer is a number which has no fractional part... So that definition is mostly correct (10e100 is also an integer - just a very large one). The 'problem' in LC is that numbers are internally represented as doubles. Technically doubles can hold integer values up to 2^53-1 faithfully - any larger than that (like 10e100) then they cease to be exact (as the magnitude increases, so do the number of values which are represented the same way). But what annoys me isn't the fact that baseconvert, etc (bitXor is the one that bit me) only work on integers less than 2^32. It's more the fact that the input data was crunched down to something the functions could handle, and then give me the wrong result with no warning. We certainly could make all operations which only work *exclusively* on integers (bit*) throw an error when out of bounds (and extend them to 2^53-1 max), but the engine never has so this comes with a backwards-compatibility risk - perhaps one we should consider though. The arithmetic operations are not possible to bounds-check in this way as you get a 'graceful' degradation of precision when using them. I'm pretty sure BaseConvert could be made to be an entirely string->string operation as I said previously - not having any bounds limit at all... Although you'd still not be able to do something like base2 -> base10, bit operations, base10 -> base2 if your bit sequence > 52 bits long. So how worth-while that is, I'm not sure. Warmest Regards, Mark. -- Mark Waddingham ~ m...@livecode.com ~ http://www.livecode.com/ LiveCode: Everyone can create apps ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode
Re: math, bigly
On 08/09/2017 02:58 AM, Mark Waddingham via use-livecode wrote: The engine hasn't ever done range checks on integers - mainly because numbers are doubles so (in most ases) you just lose accuracy / precision as the numbers get larger. In this case it *could* because it is actually using integers. I just used baseconvert as an example to show the problem. The one that caused the problem for me was bitXor. It wasn't a big deal to write a string-oriented version that could handle any inputs, but the problem for me is more the way the engine parses the data. Looking into the engine code, by the time you get to the actual bitXor code, the input arguments are already unsigned integers. Since I was passing 20-hex-digit values to bitXor, I would have expected some kind of runtime warning or error or something. It's not just a loss of precision here, it's a totally wrong answer with no indication that anything has gone astray. Writing a string-oriented function to handle this wasn't a problem - realizing that something had gone wrong was. -- Mark Wieder ahsoftw...@gmail.com ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode
Re: math, bigly
I agree. At least the result should contain something about an overflow. Bob S > On Aug 9, 2017, at 09:14 , Mark Wieder via use-livecode > wrote: > > But what annoys me isn't the fact that baseconvert, etc (bitXor is the one > that bit me) only work on integers less than 2^32. It's more the fact that > the input data was crunched down to something the functions could handle, and > then give me the wrong result with no warning. > > -- > Mark Wieder > ahsoftw...@gmail.com ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode
Re: math, bigly
On 08/09/2017 02:36 AM, hh via use-livecode wrote: Mark wrote: but I'm annoyed that the IDE doesn't give me an overflow warning if I try to perform an operation on a number that's bigger than 0x. Instead, it just happily trims it down to an unsigned integer and proceeds to give me a wrong result. Hey Mark. Don't complain because *you* are easily able to work around your "not-reading-the-dictionary" ;-)) From the baseconvert entry of the dictionary: baseConvert(number, originalBase, destinationBase) The number must be an integer between zero and 4,294,967,295 (2^32 - 1). Oops. OK - point sheepishly taken. Is that the definition of integer in the dictionary? The "integer" entry just says "if the value contains digits (and an optional minus sign) and no other characters, it is an integer", which is as simplisitic a definition as the entry for "hex". But what annoys me isn't the fact that baseconvert, etc (bitXor is the one that bit me) only work on integers less than 2^32. It's more the fact that the input data was crunched down to something the functions could handle, and then give me the wrong result with no warning. -- Mark Wieder ahsoftw...@gmail.com ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode
Re: math, bigly
On 2017-08-09 11:36, hh via use-livecode wrote: Mark wrote: but I'm annoyed that the IDE doesn't give me an overflow warning if I try to perform an operation on a number that's bigger than 0x. Instead, it just happily trims it down to an unsigned integer and proceeds to give me a wrong result. Hey Mark. Don't complain because *you* are easily able to work around your "not-reading-the-dictionary" ;-)) From the baseconvert entry of the dictionary: baseConvert(number, originalBase, destinationBase) The number must be an integer between zero and 4,294,967,295 (2^32 - 1). The engine hasn't ever done range checks on integers - mainly because numbers are doubles so (in most ases) you just lose accuracy / precision as the numbers get larger. In this case it *could* because it is actually using integers. Although, perhaps, baseConvert should be a purely string->string operation - I think that can be done relatively easily without ever having to convert to an integer. This would mean you would be able to base-convert any sized number, although you wouldn't necessarily be able to perform non-precision losing arithmetic operations on the result. Warmest Regards, Mark. -- Mark Waddingham ~ m...@livecode.com ~ http://www.livecode.com/ LiveCode: Everyone can create apps ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode
Re: math, bigly
> Mark wrote: > but I'm annoyed that the IDE doesn't give me an overflow warning if I try > to perform an operation on a number that's bigger than 0x. > Instead, it just happily trims it down to an unsigned integer and proceeds > to give me a wrong result. Hey Mark. Don't complain because *you* are easily able to work around your "not-reading-the-dictionary" ;-)) >From the baseconvert entry of the dictionary: baseConvert(number, originalBase, destinationBase) The number must be an integer between zero and 4,294,967,295 (2^32 - 1). ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode
math, bigly
Phooey... put baseconvert("10001",16,10) ...and yes, I understand what's going on under the hood, but I'm annoyed that the IDE doesn't give me an overflow warning if I try to perform an operation on a number that's bigger than 0x. Instead, it just happily trims it down to an unsigned integer and proceeds to give me a wrong result. I came across this yesterday handing two large numbers to the bitXor operand which threw no errors and went on with the rest of the calculation. I had to backtrack to figure out what was going on, and finally ended up writing my own bitwise operators to deal with this. -- Mark Wieder ahsoftw...@gmail.com ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode