gaogaotiantian commented on PR #56327:
URL: https://github.com/apache/spark/pull/56327#issuecomment-4625375770
I have a question - how correct and maintainable we want this to be? Python
`ast` modules changes from version to version (not like huge changes but they
do change. New nodes will be created and old nodes can be deprecated). We need
to support a wide range of Python versions, do we build multiple versions of
`ast` transpilers for different python versions?
Also, do we expect the users to "try it out" and see if it kind of works, or
we aim to be "always correct or raising an error"? There's a huge difference.
For example:
```python
case ast.Compare():
# All comparison operators produce bool.
return True
```
Nope, `pd.Series([1]) < pd.Series([0])` returns a `pd.Series`.
```python
case ast.BoolOp():
# and / or of booleans produces bool.
return True
```
Nope, `0 or 2 == 2`
The non-boolean check for operations are not fully valid either - any user
defined type can return a `True` for `+` - which is not that uncommon.
Like I mentioned when this SPIP was firstly discussed - Python is a super
dynamic language and having a transpiler that is always correct on where it
claims to be correct is not easy. So I want to confirm the purpose of the SPIP
- do we want to be always correct, or correct most of the time to cover more
transpilable UDFs, with the cost that we could produce wrong result?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
