Re: [IronPython] Type analysis of expression
Hi Jeff, I know that the return type of a dynamic expression cannot be determined in general. But what would be a pragmatic way to guess the return type of an expression? Can you give some examples of expressions you need to infer? arithmetic operations: a + b / c functions: sqrt(a) logical expressions: a if (b or c) else d+e I would also like to allow for enumerables in expressions: sum(a) sum([sqrt(x) for x in a if x0])**2 ___ Users mailing list Users@lists.ironpython.com http://lists.ironpython.com/listinfo.cgi/users-ironpython.com
Re: [IronPython] Type analysis of expression
On Sat, Oct 24, 2009 at 3:22 PM, Christian Schmidt christian2.schm...@gmx.de wrote: Hi Jeff, I'm going to assume that you have a way of determing the types of the input variables a, b, and c. If not, well, I'm not sure how much help this will be. You can use the IronPython parser classes to convert the expressions into an AST; the tricky part is doing the actual inference. Can you give some examples of expressions you need to infer? arithmetic operations: a + b / c functions: sqrt(a) These are mostly straightforward - Python's type conversion rules for arithmetic operators are well-defined. Presumably, sqrt is defined in your code (or is just Math.sqrt) - thus the return type is the same as the input type. logical expressions: a if (b or c) else d+e These are more difficult because of the conditional, if 'a' and 'd+e' are different types. You could just give an error if that's the case, or see if they have a common 'supertype' (i.e. if 'a' is int and 'd+e' is float, use float). I would also like to allow for enumerables in expressions: sum(a) sum([sqrt(x) for x in a if x0])**2 I would just make some assumptions - perhaps sum will always be either int or float? A full, accurate type inference engine for Python would be quite a challenge, but you only need part of one. I don't think IPy has anything builtin to help you, though. - Jeff ___ Users mailing list Users@lists.ironpython.com http://lists.ironpython.com/listinfo.cgi/users-ironpython.com
Re: [IronPython] Type analysis of expression
On Mon, Oct 19, 2009 at 1:03 AM, Christian Schmidt christian2.schm...@gmx.de wrote: I know that the return type of a dynamic expression cannot be determined in general. But what would be a pragmatic way to guess the return type of an expression? Can you give some examples of expressions you need to infer? - Jeff ___ Users mailing list Users@lists.ironpython.com http://lists.ironpython.com/listinfo.cgi/users-ironpython.com
Re: [IronPython] Type analysis of expression
Hi Vernon, Maybe I'm missing the point -- are you trying to load data into an existing data table, or to DEFINE a new data table? Sorry for not clarifying this. I'm able to export to _new_ tables, which will be created at runtime. For existing tables we could easily work out with the .NET-Convert mechanisms. If you are trying to create Data Definition Language to create a new table, then you have a real challenge. DDL is not well standardized, and you will either need to have a great deal of knowledge about the capabilities of your underlying database, or use only the simplest of data types. (Int, float, and var-string would be about it.) We have restricted ourselves to Oracle, SQL Server and partly Access and we only use simple types (int, double, decimal Datetime, strings of varying size, ...) Introspection of a Python expression would be the least of your problems. If columns must be nailed down to some specific system data type, then, I suspect, the user will have to make the decision for you, somewhat like a spreadsheet user does. That's the point. In Excel the user does not have to worry about data types. Excel does it - although the result is not always satisfactory (e.g. with zip codes). And one further weakness of Excel is that the cells within a column do not have equal types. Christian ___ Users mailing list Users@lists.ironpython.com http://lists.ironpython.com/listinfo.cgi/users-ironpython.com
[IronPython] Type analysis of expression
Hello, we are using IronPython embedded into our application to evaluate user defined expression. The variables used in the expressions are strongly typed and the results need to be written back into a database. I know that the return type of a dynamic expression cannot be determined in general. But what would be a pragmatic way to guess the return type of an expression? Thanks, Christian ___ Users mailing list Users@lists.ironpython.com http://lists.ironpython.com/listinfo.cgi/users-ironpython.com
Re: [IronPython] Type analysis of expression
Christian Schmidt wrote: Hello, we are using IronPython embedded into our application to evaluate user defined expression. The variables used in the expressions are strongly typed and the results need to be written back into a database. I know that the return type of a dynamic expression cannot be determined in general. But what would be a pragmatic way to guess the return type of an expression? Hehe. If you're evaluating arbitrary functions provided by the user then I don't know of any way of 'inferring' the return type - beyond parsing it and modelling the type flow through the expression (which would be a lot of work). All the best, Michael Thanks, Christian ___ Users mailing list Users@lists.ironpython.com http://lists.ironpython.com/listinfo.cgi/users-ironpython.com -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog ___ Users mailing list Users@lists.ironpython.com http://lists.ironpython.com/listinfo.cgi/users-ironpython.com
Re: [IronPython] Type analysis of expression
What about treating the return type of the python expression as object then coerce the object into the type required by the database? If the coercion fails, then that could be treated as a type error. For example, after using python to evaluate the expression '2+(4*5)', the returned object could be converted to an integer using System.Convert.ToInt32() or to a string with System.Convert.ToString(). System.Convert throws InvalidCastException for those cases that cannot be converted. -Original Message- From: users-boun...@lists.ironpython.com [mailto:users-boun...@lists.ironpython.com] On Behalf Of Michael Foord Sent: Monday, October 19, 2009 4:21 AM To: Discussion of IronPython Subject: Re: [IronPython] Type analysis of expression Christian Schmidt wrote: Hello, we are using IronPython embedded into our application to evaluate user defined expression. The variables used in the expressions are strongly typed and the results need to be written back into a database. I know that the return type of a dynamic expression cannot be determined in general. But what would be a pragmatic way to guess the return type of an expression? Hehe. If you're evaluating arbitrary functions provided by the user then I don't know of any way of 'inferring' the return type - beyond parsing it and modelling the type flow through the expression (which would be a lot of work). All the best, Michael Thanks, Christian ___ Users mailing list Users@lists.ironpython.com http://lists.ironpython.com/listinfo.cgi/users-ironpython.com -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog ___ Users mailing list Users@lists.ironpython.com http://lists.ironpython.com/listinfo.cgi/users-ironpython.com ___ Users mailing list Users@lists.ironpython.com http://lists.ironpython.com/listinfo.cgi/users-ironpython.com
Re: [IronPython] Type analysis of expression
Hey all, Stephen, I infer that Christian can't evaluate the expression yet. (If he could, then deriving the type of the result would be trivial) Presumably the variables in the expression don't yet have known values. So he has an expression which cannot be evaluated, but he wants to guess what type the return value would be if the variables in the expression were known. The variables in the expression do have known types. So I wonder if you could pick a prototypical value for each variable, (eg. set all floats to 1.0, and all integers to 1) and then evaluate the expression to see what type the result is? Obviously this won't work for all expressions (eg. x / (x - 1)), but maybe it would work enough of the time, depending on what form your expressions take. Am I misunderstanding entirely, or only partially? Jonathan Lepisto, Stephen P wrote: What about treating the return type of the python expression as object then coerce the object into the type required by the database? If the coercion fails, then that could be treated as a type error. For example, after using python to evaluate the expression '2+(4*5)', the returned object could be converted to an integer using System.Convert.ToInt32() or to a string with System.Convert.ToString(). System.Convert throws InvalidCastException for those cases that cannot be converted. -Original Message- From: users-boun...@lists.ironpython.com [mailto:users-boun...@lists.ironpython.com] On Behalf Of Michael Foord Sent: Monday, October 19, 2009 4:21 AM To: Discussion of IronPython Subject: Re: [IronPython] Type analysis of expression Christian Schmidt wrote: Hello, we are using IronPython embedded into our application to evaluate user defined expression. The variables used in the expressions are strongly typed and the results need to be written back into a database. I know that the return type of a dynamic expression cannot be determined in general. But what would be a pragmatic way to guess the return type of an expression? Hehe. If you're evaluating arbitrary functions provided by the user then I don't know of any way of 'inferring' the return type - beyond parsing it and modelling the type flow through the expression (which would be a lot of work). All the best, Michael Thanks, Christian ___ Users mailing list Users@lists.ironpython.com http://lists.ironpython.com/listinfo.cgi/users-ironpython.com -- Jonathan Hartley Made of meat. http://tartley.com tart...@tartley.com +44 7737 062 225 twitter/skype: tartley ___ Users mailing list Users@lists.ironpython.com http://lists.ironpython.com/listinfo.cgi/users-ironpython.com
Re: [IronPython] Type analysis of expression
Christian: While Python is type agnostic, most databases are pretty fussy about what you feed them. I presume that you need to convert the user's data into a form acceptable to the database. You do not specify what tool you are using to write your results to the database, so I will provide several answers to your question... 1) If you are writing to the database using Python, and you are using a fully PEP 249 compliant database module: The database module should attempt the conversion for you, giving an exception if it cannot. You will need to do the most obvious conversions yourself such as making a number into a string, so an if except construct should do handily. As far as I know, only the adodbapi module is both fully PEP 249 compliant and implemented in IronPython. http://sourceforge.net/projects/adodbapi 2) You can use Python's isinstance built in function to see what the user sent you: code impert types a = theOutputFromYourUser() if isinstance(a,types.StringType): print it was a string if isinstance(a, (types.FloatType, types.IntType)): print it was a number /code 3) You can take advantage of python's duck type capability by simply trying the user's output to see if it will work as a given type... code numericResult = None stringResult = None a = theOutputFromYourUser() try: numericResult = float(a) except: try: stringResult = str(a) except: pass if numericResult != None: # your numeric stuff here elif stringResult != None: # your string stuff here else: # your error handling here. /code 4) Use some combination of the above. For example: adodbapi uses a dictionary dispatch table to chose which conversion to call, which is a variation on option 2, but then uses exception handling to catch errors within the specific code for some conversions. YMMV -- Vernon Cole On Mon, Oct 19, 2009 at 5:20 AM, Michael Foord fuzzy...@voidspace.org.ukwrote: Christian Schmidt wrote: Hello, we are using IronPython embedded into our application to evaluate user defined expression. The variables used in the expressions are strongly typed and the results need to be written back into a database. I know that the return type of a dynamic expression cannot be determined in general. But what would be a pragmatic way to guess the return type of an expression? Hehe. If you're evaluating arbitrary functions provided by the user then I don't know of any way of 'inferring' the return type - beyond parsing it and modelling the type flow through the expression (which would be a lot of work). All the best, Michael Thanks, Christian ___ Users mailing list Users@lists.ironpython.com http://lists.ironpython.com/listinfo.cgi/users-ironpython.com -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog ___ Users mailing list Users@lists.ironpython.com http://lists.ironpython.com/listinfo.cgi/users-ironpython.com ___ Users mailing list Users@lists.ironpython.com http://lists.ironpython.com/listinfo.cgi/users-ironpython.com
Re: [IronPython] Type analysis of expression
Jonathan, Even if the variables going into the expression are strongly typed, python will evaluate the expression however it can, with the result being some type based on the coercion python applied to each variable. However, I read Christian's problem as one where he needed to get the value into a database and to me that was the deciding factor as to which type to coerce the result of the expression, regardless of the original types of the variables going into the expression. -Original Message- From: users-boun...@lists.ironpython.com [mailto:users-boun...@lists.ironpython.com] On Behalf Of Jonathan Hartley Sent: Monday, October 19, 2009 7:19 AM To: Discussion of IronPython Subject: Re: [IronPython] Type analysis of expression Hey all, Stephen, I infer that Christian can't evaluate the expression yet. (If he could, then deriving the type of the result would be trivial) Presumably the variables in the expression don't yet have known values. So he has an expression which cannot be evaluated, but he wants to guess what type the return value would be if the variables in the expression were known. The variables in the expression do have known types. So I wonder if you could pick a prototypical value for each variable, (eg. set all floats to 1.0, and all integers to 1) and then evaluate the expression to see what type the result is? Obviously this won't work for all expressions (eg. x / (x - 1)), but maybe it would work enough of the time, depending on what form your expressions take. Am I misunderstanding entirely, or only partially? Jonathan Lepisto, Stephen P wrote: What about treating the return type of the python expression as object then coerce the object into the type required by the database? If the coercion fails, then that could be treated as a type error. For example, after using python to evaluate the expression '2+(4*5)', the returned object could be converted to an integer using System.Convert.ToInt32() or to a string with System.Convert.ToString(). System.Convert throws InvalidCastException for those cases that cannot be converted. -Original Message- From: users-boun...@lists.ironpython.com [mailto:users-boun...@lists.ironpython.com] On Behalf Of Michael Foord Sent: Monday, October 19, 2009 4:21 AM To: Discussion of IronPython Subject: Re: [IronPython] Type analysis of expression Christian Schmidt wrote: Hello, we are using IronPython embedded into our application to evaluate user defined expression. The variables used in the expressions are strongly typed and the results need to be written back into a database. I know that the return type of a dynamic expression cannot be determined in general. But what would be a pragmatic way to guess the return type of an expression? Hehe. If you're evaluating arbitrary functions provided by the user then I don't know of any way of 'inferring' the return type - beyond parsing it and modelling the type flow through the expression (which would be a lot of work). All the best, Michael Thanks, Christian ___ Users mailing list Users@lists.ironpython.com http://lists.ironpython.com/listinfo.cgi/users-ironpython.com -- Jonathan Hartley Made of meat. http://tartley.com tart...@tartley.com +44 7737 062 225 twitter/skype: tartley ___ Users mailing list Users@lists.ironpython.com http://lists.ironpython.com/listinfo.cgi/users-ironpython.com ___ Users mailing list Users@lists.ironpython.com http://lists.ironpython.com/listinfo.cgi/users-ironpython.com
Re: [IronPython] Type analysis of expression
Fair enough. I had perhaps wrongly assumed that since Christian wanted to figure out the type of his expression's result, he must have differently typed database columns (or whatever) for different types of result. Best regards all round. Vernon Cole wrote: Stephen said what I was trying to say. It does not matter what the user sends you, you must try to coerce his result into something the database will accept. This is most easily done dynamically at run time. Also forgive my typo of impert for import ... I reread my answer three times and still missed it. -- Vernon Cole On Mon, Oct 19, 2009 at 8:29 AM, Lepisto, Stephen P stephen.p.lepi...@intel.com mailto:stephen.p.lepi...@intel.com wrote: Jonathan, Even if the variables going into the expression are strongly typed, python will evaluate the expression however it can, with the result being some type based on the coercion python applied to each variable. However, I read Christian's problem as one where he needed to get the value into a database and to me that was the deciding factor as to which type to coerce the result of the expression, regardless of the original types of the variables going into the expression. -Original Message- From: users-boun...@lists.ironpython.com mailto:users-boun...@lists.ironpython.com [mailto:users-boun...@lists.ironpython.com mailto:users-boun...@lists.ironpython.com] On Behalf Of Jonathan Hartley Sent: Monday, October 19, 2009 7:19 AM To: Discussion of IronPython Subject: Re: [IronPython] Type analysis of expression Hey all, Stephen, I infer that Christian can't evaluate the expression yet. (If he could, then deriving the type of the result would be trivial) Presumably the variables in the expression don't yet have known values. So he has an expression which cannot be evaluated, but he wants to guess what type the return value would be if the variables in the expression were known. The variables in the expression do have known types. So I wonder if you could pick a prototypical value for each variable, (eg. set all floats to 1.0, and all integers to 1) and then evaluate the expression to see what type the result is? Obviously this won't work for all expressions (eg. x / (x - 1)), but maybe it would work enough of the time, depending on what form your expressions take. Am I misunderstanding entirely, or only partially? Jonathan Lepisto, Stephen P wrote: What about treating the return type of the python expression as object then coerce the object into the type required by the database? If the coercion fails, then that could be treated as a type error. For example, after using python to evaluate the expression '2+(4*5)', the returned object could be converted to an integer using System.Convert.ToInt32() or to a string with System.Convert.ToString(). System.Convert throws InvalidCastException for those cases that cannot be converted. -Original Message- From: users-boun...@lists.ironpython.com mailto:users-boun...@lists.ironpython.com [mailto:users-boun...@lists.ironpython.com mailto:users-boun...@lists.ironpython.com] On Behalf Of Michael Foord Sent: Monday, October 19, 2009 4:21 AM To: Discussion of IronPython Subject: Re: [IronPython] Type analysis of expression Christian Schmidt wrote: Hello, we are using IronPython embedded into our application to evaluate user defined expression. The variables used in the expressions are strongly typed and the results need to be written back into a database. I know that the return type of a dynamic expression cannot be determined in general. But what would be a pragmatic way to guess the return type of an expression? Hehe. If you're evaluating arbitrary functions provided by the user then I don't know of any way of 'inferring' the return type - beyond parsing it and modelling the type flow through the expression (which would be a lot of work). All the best, Michael Thanks, Christian ___ Users mailing list Users@lists.ironpython.com mailto:Users@lists.ironpython.com http://lists.ironpython.com/listinfo.cgi/users-ironpython.com -- Jonathan Hartley Made of meat. http://tartley.com tart...@tartley.com mailto:tart...@tartley.com +44 7737 062 225 twitter/skype: tartley ___ Users mailing list Users@lists.ironpython.com mailto:Users@lists.ironpython.com http://lists.ironpython.com/listinfo.cgi/users-ironpython.com
Re: [IronPython] Type analysis of expression
Hi all,
thanks for your discussion.
Actually I'm having an in-memory table of strongly typed columns. The
user can provide per-row (python-)expressions as additional columns. Now
if the user wants his result to be exported to a database (e.g. SQL
Server or Oracle) I need to set a type for each column - also for the
expression columns.
I thought there might be a way to figure out the return type in a
similar way for example Boo (boo.codehaus.org) does at compile time.
When an expression is parsed at runtime, the interpreter also needs to
decide which .NET-functions to call. For strongly typed input these
functions should normally have typed return values... Wouldn't this work
somehow?
If the Boo way is not possible then the only option is evaluating the
expressions for some random rows and coerce to a common type. What would
be the rules? int-float-string is trivial, but what about decimal,
int64, double, ...? I assume python must have implemented these rules
somewhere. How would one have to implement the general function:
Type GetCoercedType(IEnumerableobject list) { ... }
Thanks,
Christian
___
Users mailing list
Users@lists.ironpython.com
http://lists.ironpython.com/listinfo.cgi/users-ironpython.com
Re: [IronPython] Type analysis of expression
Christian,
Would it be possible to leave the columns as text then coerce at the time the
data is read from the database? In other words, convert the result of the
python expression to a string and store the string in the database. Then, when
the result is needed, coerce the string to a more suitable type. In other
words, defer when the type needs to be known until the value is used.
-Original Message-
From: users-boun...@lists.ironpython.com
[mailto:users-boun...@lists.ironpython.com] On Behalf Of Christian Schmidt
Sent: Monday, October 19, 2009 12:12 PM
To: Discussion of IronPython
Subject: Re: [IronPython] Type analysis of expression
Hi all,
thanks for your discussion.
Actually I'm having an in-memory table of strongly typed columns. The
user can provide per-row (python-)expressions as additional columns. Now
if the user wants his result to be exported to a database (e.g. SQL
Server or Oracle) I need to set a type for each column - also for the
expression columns.
I thought there might be a way to figure out the return type in a
similar way for example Boo (boo.codehaus.org) does at compile time.
When an expression is parsed at runtime, the interpreter also needs to
decide which .NET-functions to call. For strongly typed input these
functions should normally have typed return values... Wouldn't this work
somehow?
If the Boo way is not possible then the only option is evaluating the
expressions for some random rows and coerce to a common type. What would
be the rules? int-float-string is trivial, but what about decimal,
int64, double, ...? I assume python must have implemented these rules
somewhere. How would one have to implement the general function:
Type GetCoercedType(IEnumerableobject list) { ... }
Thanks,
Christian
___
Users mailing list
Users@lists.ironpython.com
http://lists.ironpython.com/listinfo.cgi/users-ironpython.com
___
Users mailing list
Users@lists.ironpython.com
http://lists.ironpython.com/listinfo.cgi/users-ironpython.com
Re: [IronPython] Type analysis of expression
Christian: Maybe I'm missing the point -- are you trying to load data into an existing data table, or to DEFINE a new data table? I was assuming the former, a fairly easy case. If you are trying to create Data Definition Language to create a new table, then you have a real challenge. DDL is not well standardized, and you will either need to have a great deal of knowledge about the capabilities of your underlying database, or use only the simplest of data types. (Int, float, and var-string would be about it.) Introspection of a Python expression would be the least of your problems. If columns must be nailed down to some specific system data type, then, I suspect, the user will have to make the decision for you, somewhat like a spreadsheet user does. -- Vernon On Mon, Oct 19, 2009 at 1:52 PM, Christian Schmidt christian2.schm...@gmx.de wrote: Stephen, Would it be possible to leave the columns as text then coerce at the time the data is read from the database? In other words, convert the result of the python expression to a string and store the string in the database. Then, when the result is needed, coerce the string to a more suitable type. In other words, defer when the type needs to be known until the value is used. This would not be an option, because a reading system is normally not pythonic. And it would not be very user friendly. In normal circumstances the result type should be very easily identifyable - at least for a programmer user... ___ Users mailing list Users@lists.ironpython.com http://lists.ironpython.com/listinfo.cgi/users-ironpython.com ___ Users mailing list Users@lists.ironpython.com http://lists.ironpython.com/listinfo.cgi/users-ironpython.com
Re: [IronPython] Type analysis of expression
On Mon, Oct 19, 2009 at 12:12 PM, Christian Schmidt christian2.schm...@gmx.de wrote: When an expression is parsed at runtime, the interpreter also needs to decide which .NET-functions to call. For strongly typed input these functions should normally have typed return values... Wouldn't this work somehow? Sure. That's how you create the intellisense experience. I don't know if there are any open source Python intellisense engines, though, and even if there were, it would need to be adapted to understand .NET. -- Curt Hagenlocher c...@hagenlocher.org ___ Users mailing list Users@lists.ironpython.com http://lists.ironpython.com/listinfo.cgi/users-ironpython.com
