Github user davies commented on the pull request:
https://github.com/apache/spark/pull/4521#issuecomment-74773330
The `long` in JVM will be converted into `int` in Python by Pyrolite, this
change sounds reasonable.
`long` in Python is actually BigInteger, it's much slower than `int`, also
it's slow to pickle/unpickle.
I think it's better to convert the Integer into Long in
SQLContext.applySchemaToPythonRDD(), you could have a special case in
needConversion() and EvaluatePython.fromJava(). We had do something for
ByteType/Short, it's easy to add a case for (Int, LongType).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
