GitHub user rxin opened a pull request:
https://github.com/apache/spark/pull/11880
[SPARK-14063][SQL] SQLContext.range should return Dataset[java.lang.Long]
## What changes were proposed in this pull request?
This patch changed the return type for SQLContext.range from Dataset[Long]
(Scala primitive) to Dataset[java.lang.Long] (Java boxed long).
Previously, SPARK-13894 changed the return type of range from Dataset[Row]
to Dataset[Long]. The problem is that due to
https://issues.scala-lang.org/browse/SI-4388, Scala compiles primitive types in
generics into just Object, i.e. range at bytecode level now just returns
Dataset[Object]. This is really bad for Java users because they are losing type
safety and also need to add a type cast every time they use range.
Talked to Jason Zaugg who suggested the best approach is to return
Dataset[java.lang.Long]. The small downside is that when Scala users want to
explicitly type a closure used on the dataset returned by range, they would
need to use java.lang.Long instead of the Scala Long.
## How was this patch tested?
The signature change should be covered by existing unit tests and API
tests. I also added a new test case in DatasetSuite for range.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/rxin/spark SPARK-14063
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/11880.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #11880
----
commit 8272a1fc8204a1124ef0aad40a3077caedb72d10
Author: Reynold Xin <[email protected]>
Date: 2016-03-22T07:24:21Z
[SPARK-14063][SQL] SQLContext.range should return Dataset[java.lang.Long]
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
