[GitHub] spark pull request: [SPARK-1460] Returning SchemaRDD instead of no...

2014-05-16 Thread pwendell
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/448#issuecomment-42393027 Jenkins, retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-1460] Returning SchemaRDD instead of no...

2014-05-14 Thread pwendell
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/448#issuecomment-42451285 Thanks for updating this. I'm merging it. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

[GitHub] spark pull request: [SPARK-1460] Returning SchemaRDD instead of no...

2014-05-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/448#issuecomment-42393186 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-1460] Returning SchemaRDD instead of no...

2014-05-07 Thread pwendell
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/448#issuecomment-42382227 Jenkins, retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-1460] Returning SchemaRDD instead of no...

2014-05-07 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/448#issuecomment-42397473 Merged build finished. All automated tests passed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well.

[GitHub] spark pull request: [SPARK-1460] Returning SchemaRDD instead of no...

2014-05-07 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/448#issuecomment-42381669 Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/14746/ --- If your project is set up for it, you can

[GitHub] spark pull request: [SPARK-1460] Returning SchemaRDD instead of no...

2014-05-07 Thread kanzhang
Github user kanzhang commented on the pull request: https://github.com/apache/spark/pull/448#issuecomment-42385119 @pwendell the build didn't seem to start? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

[GitHub] spark pull request: [SPARK-1460] Returning SchemaRDD instead of no...

2014-05-07 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/448#issuecomment-42397475 All automated tests passed. Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/14762/ --- If your project

[GitHub] spark pull request: [SPARK-1460] Returning SchemaRDD instead of no...

2014-05-06 Thread pwendell
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/448#issuecomment-42272515 @kanzhang this is out of date with master - mind merging it? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as

[GitHub] spark pull request: [SPARK-1460] Returning SchemaRDD instead of no...

2014-05-06 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/448#issuecomment-42350396 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-1460] Returning SchemaRDD instead of no...

2014-05-06 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/448#issuecomment-42350425 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-1460] Returning SchemaRDD instead of no...

2014-05-06 Thread kanzhang
Github user kanzhang commented on the pull request: https://github.com/apache/spark/pull/448#issuecomment-42350963 @pwendell did a rebase just now and pushed --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request: [SPARK-1460] Returning SchemaRDD instead of no...

2014-05-06 Thread mridulm
Github user mridulm commented on a diff in the pull request: https://github.com/apache/spark/pull/448#discussion_r12346057 --- Diff: graphx/src/main/scala/org/apache/spark/graphx/EdgeRDD.scala --- @@ -51,18 +51,12 @@ class EdgeRDD[@specialized ED: ClassTag]( override

[GitHub] spark pull request: [SPARK-1460] Returning SchemaRDD instead of no...

2014-05-06 Thread mridulm
Github user mridulm commented on a diff in the pull request: https://github.com/apache/spark/pull/448#discussion_r12345988 --- Diff: core/src/main/scala/org/apache/spark/rdd/RDD.scala --- @@ -138,7 +138,7 @@ abstract class RDD[T: ClassTag]( * it is computed. This can only

[GitHub] spark pull request: [SPARK-1460] Returning SchemaRDD instead of no...

2014-05-06 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/448#discussion_r12346651 --- Diff: core/src/main/scala/org/apache/spark/rdd/RDD.scala --- @@ -138,7 +138,7 @@ abstract class RDD[T: ClassTag]( * it is computed. This can only

[GitHub] spark pull request: [SPARK-1460] Returning SchemaRDD instead of no...

2014-05-06 Thread mridulm
Github user mridulm commented on a diff in the pull request: https://github.com/apache/spark/pull/448#discussion_r12347791 --- Diff: core/src/main/scala/org/apache/spark/rdd/RDD.scala --- @@ -138,7 +138,7 @@ abstract class RDD[T: ClassTag]( * it is computed. This can only

[GitHub] spark pull request: [SPARK-1460] Returning SchemaRDD instead of no...

2014-05-06 Thread mridulm
Github user mridulm commented on a diff in the pull request: https://github.com/apache/spark/pull/448#discussion_r12347808 --- Diff: graphx/src/main/scala/org/apache/spark/graphx/EdgeRDD.scala --- @@ -51,18 +51,12 @@ class EdgeRDD[@specialized ED: ClassTag]( override

[GitHub] spark pull request: [SPARK-1460] Returning SchemaRDD instead of no...

2014-05-06 Thread mridulm
Github user mridulm commented on a diff in the pull request: https://github.com/apache/spark/pull/448#discussion_r12348272 --- Diff: core/src/main/scala/org/apache/spark/rdd/RDD.scala --- @@ -138,7 +138,7 @@ abstract class RDD[T: ClassTag]( * it is computed. This can only

[GitHub] spark pull request: [SPARK-1460] Returning SchemaRDD instead of no...

2014-05-06 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/448#discussion_r12346614 --- Diff: graphx/src/main/scala/org/apache/spark/graphx/EdgeRDD.scala --- @@ -51,18 +51,12 @@ class EdgeRDD[@specialized ED: ClassTag]( override

[GitHub] spark pull request: [SPARK-1460] Returning SchemaRDD instead of no...

2014-05-06 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/448#issuecomment-42360847 Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/14731/ --- If your project is set up for it, you can

[GitHub] spark pull request: [SPARK-1460] Returning SchemaRDD instead of no...

2014-05-06 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/448#issuecomment-42360844 Merged build finished. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-1460] Returning SchemaRDD instead of no...

2014-05-06 Thread kanzhang
Github user kanzhang commented on a diff in the pull request: https://github.com/apache/spark/pull/448#discussion_r12351929 --- Diff: core/src/main/scala/org/apache/spark/rdd/RDD.scala --- @@ -138,7 +138,7 @@ abstract class RDD[T: ClassTag]( * it is computed. This can only

[GitHub] spark pull request: [SPARK-1460] Returning SchemaRDD instead of no...

2014-05-06 Thread pwendell
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/448#issuecomment-42365832 Jenkins, retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-1460] Returning SchemaRDD instead of no...

2014-05-06 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/448#issuecomment-42366216 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-1460] Returning SchemaRDD instead of no...

2014-05-06 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/448#issuecomment-42366235 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-1460] Returning SchemaRDD instead of no...

2014-05-06 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/448#issuecomment-42371680 Merged build finished. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-1460] Returning SchemaRDD instead of no...

2014-05-06 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/448#issuecomment-42371682 Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/14738/ --- If your project is set up for it, you can

[GitHub] spark pull request: [SPARK-1460] Returning SchemaRDD instead of no...

2014-05-06 Thread pwendell
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/448#issuecomment-42373741 @kanzhang hey you'll need to silence some of the binary compatibility checks in project/MimaBuild.scala: ```

[GitHub] spark pull request: [SPARK-1460] Returning SchemaRDD instead of no...

2014-05-06 Thread mridulm
Github user mridulm commented on a diff in the pull request: https://github.com/apache/spark/pull/448#discussion_r12357557 --- Diff: core/src/main/scala/org/apache/spark/rdd/RDD.scala --- @@ -138,7 +138,7 @@ abstract class RDD[T: ClassTag]( * it is computed. This can only

[GitHub] spark pull request: [SPARK-1460] Returning SchemaRDD instead of no...

2014-05-06 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/448#issuecomment-42377607 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-1460] Returning SchemaRDD instead of no...

2014-05-06 Thread kanzhang
Github user kanzhang commented on the pull request: https://github.com/apache/spark/pull/448#issuecomment-42377588 @pwendell thanks for the heads-up. made those changes, let's see how it goes. --- If your project is set up for it, you can reply to this email and have your reply

[GitHub] spark pull request: [SPARK-1460] Returning SchemaRDD instead of no...

2014-05-05 Thread kanzhang
Github user kanzhang commented on the pull request: https://github.com/apache/spark/pull/448#issuecomment-42217278 @marmbrus @mateiz Just a heads-up, changing to use this.type as return type may break 1.0 public API if we don't get this change in (see comment above). --- If your

[GitHub] spark pull request: [SPARK-1460] Returning SchemaRDD instead of no...

2014-05-05 Thread marmbrus
Github user marmbrus commented on the pull request: https://github.com/apache/spark/pull/448#issuecomment-42224442 @kanzhang, Good point about the public APIs. I talked with @pwendell and we are a little concerned that this.type will make binary computability hard long term, mostly

[GitHub] spark pull request: [SPARK-1460] Returning SchemaRDD instead of no...

2014-05-05 Thread mateiz
Github user mateiz commented on the pull request: https://github.com/apache/spark/pull/448#issuecomment-42248461 Yeah, let's look at that. If it's just Object we may be fine. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as

[GitHub] spark pull request: [SPARK-1460] Returning SchemaRDD instead of no...

2014-05-05 Thread kanzhang
Github user kanzhang commented on the pull request: https://github.com/apache/spark/pull/448#issuecomment-42248502 @marmbrus my untrained eyes couldn't spot any difference (you may want to verify it yourself just to make sure :). For example, RDD.unpersist() with RDD[T]

[GitHub] spark pull request: [SPARK-1460] Returning SchemaRDD instead of no...

2014-05-05 Thread marmbrus
Github user marmbrus commented on a diff in the pull request: https://github.com/apache/spark/pull/448#discussion_r12303586 --- Diff: graphx/src/main/scala/org/apache/spark/graphx/VertexRDD.scala --- @@ -71,18 +71,18 @@ class VertexRDD[@specialized VD: ClassTag]( override

[GitHub] spark pull request: [SPARK-1460] Returning SchemaRDD instead of no...

2014-05-05 Thread marmbrus
Github user marmbrus commented on a diff in the pull request: https://github.com/apache/spark/pull/448#discussion_r12303651 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/api/java/JavaSchemaRDD.scala --- @@ -45,4 +48,141 @@ class JavaSchemaRDD( override def

[GitHub] spark pull request: [SPARK-1460] Returning SchemaRDD instead of no...

2014-05-05 Thread kanzhang
Github user kanzhang commented on the pull request: https://github.com/apache/spark/pull/448#issuecomment-42249374 Actually, let me run with -s option shortly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request: [SPARK-1460] Returning SchemaRDD instead of no...

2014-05-05 Thread kanzhang
Github user kanzhang commented on a diff in the pull request: https://github.com/apache/spark/pull/448#discussion_r12304013 --- Diff: graphx/src/main/scala/org/apache/spark/graphx/VertexRDD.scala --- @@ -71,18 +71,18 @@ class VertexRDD[@specialized VD: ClassTag]( override

[GitHub] spark pull request: [SPARK-1460] Returning SchemaRDD instead of no...

2014-05-05 Thread kanzhang
Github user kanzhang commented on a diff in the pull request: https://github.com/apache/spark/pull/448#discussion_r12304047 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/api/java/JavaSchemaRDD.scala --- @@ -45,4 +48,141 @@ class JavaSchemaRDD( override def

[GitHub] spark pull request: [SPARK-1460] Returning SchemaRDD instead of no...

2014-05-05 Thread marmbrus
Github user marmbrus commented on a diff in the pull request: https://github.com/apache/spark/pull/448#discussion_r12304083 --- Diff: graphx/src/main/scala/org/apache/spark/graphx/VertexRDD.scala --- @@ -71,18 +71,18 @@ class VertexRDD[@specialized VD: ClassTag]( override

[GitHub] spark pull request: [SPARK-1460] Returning SchemaRDD instead of no...

2014-05-05 Thread kanzhang
Github user kanzhang commented on the pull request: https://github.com/apache/spark/pull/448#issuecomment-42252573 The -s option isn't very interesting. The couple lines added are identical in both cases. --- If your project is set up for it, you can reply to this email and have

[GitHub] spark pull request: [SPARK-1460] Returning SchemaRDD instead of no...

2014-05-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/448#issuecomment-42253947 Build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] spark pull request: [SPARK-1460] Returning SchemaRDD instead of no...

2014-05-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/448#issuecomment-42253939 Build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-1460] Returning SchemaRDD instead of no...

2014-05-05 Thread kanzhang
Github user kanzhang commented on the pull request: https://github.com/apache/spark/pull/448#issuecomment-42254028 Pushed an update based on @marmbrus comments. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request: [SPARK-1460] Returning SchemaRDD instead of no...

2014-05-05 Thread kanzhang
Github user kanzhang commented on a diff in the pull request: https://github.com/apache/spark/pull/448#discussion_r12305806 --- Diff: graphx/src/main/scala/org/apache/spark/graphx/VertexRDD.scala --- @@ -71,18 +71,18 @@ class VertexRDD[@specialized VD: ClassTag]( override

[GitHub] spark pull request: [SPARK-1460] Returning SchemaRDD instead of no...

2014-05-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/448#issuecomment-42257630 Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/14687/ --- If your project is set up for it, you can

[GitHub] spark pull request: [SPARK-1460] Returning SchemaRDD instead of no...

2014-05-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/448#issuecomment-42257628 Build finished. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] spark pull request: [SPARK-1460] Returning SchemaRDD instead of no...

2014-05-05 Thread pwendell
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/448#issuecomment-42262939 @marmbrus @kanzhang I'm a bit confused how the bytecode could be totally unchanged. Let's say I create a new class `FooRDD.scala` in a totally different package

[GitHub] spark pull request: [SPARK-1460] Returning SchemaRDD instead of no...

2014-05-05 Thread marmbrus
Github user marmbrus commented on the pull request: https://github.com/apache/spark/pull/448#issuecomment-42264403 I don't think it's very obvious :) My guess would be that this.type is included in a scala specific signature somewhere, and the scala compiler adds casts after

[GitHub] spark pull request: [SPARK-1460] Returning SchemaRDD instead of no...

2014-05-05 Thread pwendell
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/448#issuecomment-42264759 @marmbrus I did some local byte code inspection and I think it's injecting this information into the scalasig of the class (in our case, the RDD class). I made a simple

[GitHub] spark pull request: [SPARK-1460] Returning SchemaRDD instead of no...

2014-05-05 Thread pwendell
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/448#issuecomment-42264854 @marmbrus anyways I don't see this tying our hands much in the future, so seems like a good idea. It will potentially force people to re-compile coming from pre-1.0

[GitHub] spark pull request: [SPARK-1460] Returning SchemaRDD instead of no...

2014-05-05 Thread kanzhang
Github user kanzhang commented on the pull request: https://github.com/apache/spark/pull/448#issuecomment-42266085 @pwendell yes, scalap shows it. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-1460] Returning SchemaRDD instead of no...

2014-05-05 Thread pwendell
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/448#issuecomment-42268837 LGTM then - I looked around a bit and it seems like this use case is exactly what `this.type` is for in Scala. @marmbrus @mateiz any further feedback? --- If

[GitHub] spark pull request: [SPARK-1460] Returning SchemaRDD instead of no...

2014-05-05 Thread marmbrus
Github user marmbrus commented on the pull request: https://github.com/apache/spark/pull/448#issuecomment-42268877 LGTM --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] spark pull request: [SPARK-1460] Returning SchemaRDD instead of no...

2014-05-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/448#issuecomment-42115307 Build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-1460] Returning SchemaRDD instead of no...

2014-05-03 Thread kanzhang
Github user kanzhang commented on the pull request: https://github.com/apache/spark/pull/448#issuecomment-42115441 Added Python API changes. Skipped filter() as I don't see a way to translate Python functions into Java land. Py4j supports implementing Java interfaces in Python

[GitHub] spark pull request: [SPARK-1460] Returning SchemaRDD instead of no...

2014-05-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/448#issuecomment-42116989 Build finished. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] spark pull request: [SPARK-1460] Returning SchemaRDD instead of no...

2014-05-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/448#issuecomment-42116990 Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/14634/ --- If your project is set up for it, you can

[GitHub] spark pull request: [SPARK-1460] Returning SchemaRDD instead of no...

2014-05-01 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/448#issuecomment-41965097 Build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] spark pull request: [SPARK-1460] Returning SchemaRDD instead of no...

2014-05-01 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/448#issuecomment-41965083 Build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-1460] Returning SchemaRDD instead of no...

2014-05-01 Thread kanzhang
Github user kanzhang commented on the pull request: https://github.com/apache/spark/pull/448#issuecomment-41966458 @marmbrus @mateiz Here's an update on using this.type as return type. See following shell output. The result type for setName() method has changed from

[GitHub] spark pull request: [SPARK-1460] Returning SchemaRDD instead of no...

2014-05-01 Thread kanzhang
Github user kanzhang commented on the pull request: https://github.com/apache/spark/pull/448#issuecomment-41968161 PS. Subclasses that override those methods may have to be updated and recompiled (like what I did in EdgeRDD, VertexRDD). Better ideas? --- If your project is set up

[GitHub] spark pull request: [SPARK-1460] Returning SchemaRDD instead of no...

2014-04-26 Thread kanzhang
Github user kanzhang commented on a diff in the pull request: https://github.com/apache/spark/pull/448#discussion_r12023704 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/SchemaRDD.scala --- @@ -313,4 +315,82 @@ class SchemaRDD( } } } + +

[GitHub] spark pull request: [SPARK-1460] Returning SchemaRDD instead of no...

2014-04-26 Thread kanzhang
Github user kanzhang commented on a diff in the pull request: https://github.com/apache/spark/pull/448#discussion_r12023710 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/api/java/JavaSchemaRDD.scala --- @@ -45,4 +50,146 @@ class JavaSchemaRDD( override def

[GitHub] spark pull request: [SPARK-1460] Returning SchemaRDD instead of no...

2014-04-25 Thread marmbrus
Github user marmbrus commented on a diff in the pull request: https://github.com/apache/spark/pull/448#discussion_r12021763 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/SchemaRDD.scala --- @@ -313,4 +315,82 @@ class SchemaRDD( } } } + +

[GitHub] spark pull request: [SPARK-1460] Returning SchemaRDD instead of no...

2014-04-25 Thread marmbrus
Github user marmbrus commented on a diff in the pull request: https://github.com/apache/spark/pull/448#discussion_r12021785 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/api/java/JavaSchemaRDD.scala --- @@ -45,4 +50,146 @@ class JavaSchemaRDD( override def

[GitHub] spark pull request: [SPARK-1460] Returning SchemaRDD instead of no...

2014-04-25 Thread marmbrus
Github user marmbrus commented on the pull request: https://github.com/apache/spark/pull/448#issuecomment-41452343 Looking pretty good. Thanks again for working on this! Jenkins, test this please. --- If your project is set up for it, you can reply to this email and have

[GitHub] spark pull request: [SPARK-1460] Returning SchemaRDD instead of no...

2014-04-25 Thread mateiz
Github user mateiz commented on a diff in the pull request: https://github.com/apache/spark/pull/448#discussion_r12022387 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/SchemaRDD.scala --- @@ -313,4 +315,82 @@ class SchemaRDD( } } } + +

[GitHub] spark pull request: [SPARK-1460] Returning SchemaRDD instead of no...

2014-04-23 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/448#issuecomment-41230289 Build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-1460] Returning SchemaRDD instead of no...

2014-04-23 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/448#issuecomment-41230295 Build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] spark pull request: [SPARK-1460] Returning SchemaRDD instead of no...

2014-04-23 Thread kanzhang
Github user kanzhang commented on the pull request: https://github.com/apache/spark/pull/448#issuecomment-41230336 Hey, just pushed an update on Scala and Java API. Wanted to get some feedback before I move on to Python. Pls pay attention to signatures of filter, intersection and

[GitHub] spark pull request: [SPARK-1460] Returning SchemaRDD instead of no...

2014-04-23 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/448#issuecomment-41234927 Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/14414/ --- If your project is set up for it, you can

[GitHub] spark pull request: [SPARK-1460] Returning SchemaRDD instead of no...

2014-04-23 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/448#issuecomment-41234925 Build finished. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] spark pull request: [SPARK-1460] Returning SchemaRDD instead of no...

2014-04-21 Thread marmbrus
Github user marmbrus commented on a diff in the pull request: https://github.com/apache/spark/pull/448#discussion_r11813558 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/SchemaRDD.scala --- @@ -313,4 +314,46 @@ class SchemaRDD( } } } + +

[GitHub] spark pull request: [SPARK-1460] Returning SchemaRDD instead of no...

2014-04-21 Thread marmbrus
Github user marmbrus commented on a diff in the pull request: https://github.com/apache/spark/pull/448#discussion_r11813669 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/SchemaRDD.scala --- @@ -313,4 +314,46 @@ class SchemaRDD( } } } + +

[GitHub] spark pull request: [SPARK-1460] Returning SchemaRDD instead of no...

2014-04-21 Thread marmbrus
Github user marmbrus commented on the pull request: https://github.com/apache/spark/pull/448#issuecomment-40949024 How about cache(), persist(), unpersist()? Good catch! Btw, I don't see PythonSchemaRDD in the code base yet, can I leave out Python for now?

[GitHub] spark pull request: [SPARK-1460] Returning SchemaRDD instead of no...

2014-04-21 Thread kanzhang
Github user kanzhang commented on a diff in the pull request: https://github.com/apache/spark/pull/448#discussion_r11816456 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/SchemaRDD.scala --- @@ -313,4 +314,46 @@ class SchemaRDD( } } } + +

[GitHub] spark pull request: [SPARK-1460] Returning SchemaRDD instead of no...

2014-04-21 Thread kanzhang
Github user kanzhang commented on a diff in the pull request: https://github.com/apache/spark/pull/448#discussion_r11816864 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/SchemaRDD.scala --- @@ -313,4 +314,46 @@ class SchemaRDD( } } } + +

[GitHub] spark pull request: [SPARK-1460] Returning SchemaRDD instead of no...

2014-04-21 Thread marmbrus
Github user marmbrus commented on a diff in the pull request: https://github.com/apache/spark/pull/448#discussion_r11818451 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/SchemaRDD.scala --- @@ -313,4 +314,46 @@ class SchemaRDD( } } } + +

[GitHub] spark pull request: [SPARK-1460] Returning SchemaRDD instead of no...

2014-04-21 Thread kanzhang
Github user kanzhang commented on a diff in the pull request: https://github.com/apache/spark/pull/448#discussion_r11819317 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/SchemaRDD.scala --- @@ -313,4 +314,46 @@ class SchemaRDD( } } } + +

[GitHub] spark pull request: [SPARK-1460] Returning SchemaRDD instead of no...

2014-04-20 Thread kanzhang
Github user kanzhang commented on the pull request: https://github.com/apache/spark/pull/448#issuecomment-40909814 Thanks for your suggestions. I'll update. Btw, I don't see PythonSchemaRDD in the code base yet, can I leave out Python for now? --- If your project is set up

[GitHub] spark pull request: [SPARK-1460] Returning SchemaRDD instead of no...

2014-04-20 Thread kanzhang
Github user kanzhang commented on a diff in the pull request: https://github.com/apache/spark/pull/448#discussion_r11800178 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/SchemaRDD.scala --- @@ -313,4 +314,46 @@ class SchemaRDD( } } } + +

[GitHub] spark pull request: [SPARK-1460] Returning SchemaRDD instead of no...

2014-04-20 Thread kanzhang
Github user kanzhang commented on a diff in the pull request: https://github.com/apache/spark/pull/448#discussion_r11800221 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/SchemaRDD.scala --- @@ -313,4 +314,46 @@ class SchemaRDD( } } } + +

[GitHub] spark pull request: [SPARK-1460] Returning SchemaRDD instead of no...

2014-04-20 Thread kanzhang
Github user kanzhang commented on the pull request: https://github.com/apache/spark/pull/448#issuecomment-40916074 @marmbrus you are right, I can't override randomSplit() due to invariance of Array. How about cache(), persist(), unpersist()? --- If your project is set up

[GitHub] spark pull request: [SPARK-1460] Returning SchemaRDD instead of no...

2014-04-19 Thread mateiz
Github user mateiz commented on the pull request: https://github.com/apache/spark/pull/448#issuecomment-40880763 I agree with leaving union out and adding repartition, coalesce and the other version of distinct. Also these should definitely be added to Java and Python too. --- If

[GitHub] spark pull request: [SPARK-1460] Returning SchemaRDD instead of no...

2014-04-18 Thread kanzhang
GitHub user kanzhang opened a pull request: https://github.com/apache/spark/pull/448 [SPARK-1460] Returning SchemaRDD instead of normal RDD on Set operations... ... that do not change schema You can merge this pull request into a Git repository by running: $ git pull

[GitHub] spark pull request: [SPARK-1460] Returning SchemaRDD instead of no...

2014-04-18 Thread kanzhang
Github user kanzhang commented on the pull request: https://github.com/apache/spark/pull/448#issuecomment-40850230 First try, pls comment. Not very comfortable with methods that take other RDDs, like intersect, subtract and union, since caller has to make sure they of the same

[GitHub] spark pull request: [SPARK-1460] Returning SchemaRDD instead of no...

2014-04-18 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/448#issuecomment-40850347 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request: [SPARK-1460] Returning SchemaRDD instead of no...

2014-04-18 Thread marmbrus
Github user marmbrus commented on the pull request: https://github.com/apache/spark/pull/448#issuecomment-40854278 ok to test --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark pull request: [SPARK-1460] Returning SchemaRDD instead of no...

2014-04-18 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/448#issuecomment-40854403 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-1460] Returning SchemaRDD instead of no...

2014-04-18 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/448#issuecomment-40854395 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-1460] Returning SchemaRDD instead of no...

2014-04-18 Thread marmbrus
Github user marmbrus commented on the pull request: https://github.com/apache/spark/pull/448#issuecomment-40855861 Thanks for doing this! I think we are actually okay for `intersect` and `subtract` as anything in the result must be a row that was in the original RDD and thus

[GitHub] spark pull request: [SPARK-1460] Returning SchemaRDD instead of no...

2014-04-18 Thread marmbrus
Github user marmbrus commented on a diff in the pull request: https://github.com/apache/spark/pull/448#discussion_r11791420 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/SchemaRDD.scala --- @@ -313,4 +314,46 @@ class SchemaRDD( } } } + +

[GitHub] spark pull request: [SPARK-1460] Returning SchemaRDD instead of no...

2014-04-18 Thread marmbrus
Github user marmbrus commented on the pull request: https://github.com/apache/spark/pull/448#issuecomment-40856180 Also, we should make the same changes to the Java and Python API if possible. --- If your project is set up for it, you can reply to this email and have your reply

[GitHub] spark pull request: [SPARK-1460] Returning SchemaRDD instead of no...

2014-04-18 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/448#issuecomment-40856762 Merged build finished. All automated tests passed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well.

[GitHub] spark pull request: [SPARK-1460] Returning SchemaRDD instead of no...

2014-04-18 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/448#issuecomment-40856764 All automated tests passed. Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/14248/ --- If your project