Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/9889
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enab
Github user marmbrus commented on the pull request:
https://github.com/apache/spark/pull/9889#issuecomment-161057518
Thanks, merging to master and 1.6.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does
Github user gatorsmile commented on the pull request:
https://github.com/apache/spark/pull/9889#issuecomment-160872230
@marmbrus Please check the latest changes. Feel free to let me know if we
need more changes. Thank you!
---
If your project is set up for it, you can reply to this e
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9889#issuecomment-160846047
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/4
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9889#issuecomment-160846044
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9889#issuecomment-160845838
**[Test build #46925 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46925/consoleFull)**
for PR 9889 at commit
[`b8d287a`](https://git
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9889#issuecomment-160823405
**[Test build #46925 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46925/consoleFull)**
for PR 9889 at commit
[`b8d287a`](https://gith
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/9889#discussion_r46231420
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/CacheSuite.scala ---
@@ -0,0 +1,89 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF
Github user gatorsmile commented on the pull request:
https://github.com/apache/spark/pull/9889#issuecomment-160803517
Thank you! @marmbrus
Will do the changes soon.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as we
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/9889#discussion_r46222399
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/CacheSuite.scala ---
@@ -0,0 +1,89 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF)
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/9889#discussion_r46222418
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/CacheSuite.scala ---
@@ -0,0 +1,89 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF)
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/9889#discussion_r46222335
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/SQLContext.scala ---
@@ -339,6 +339,15 @@ class SQLContext private[sql](
}
/**
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/9889#discussion_r46222305
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/CacheSuite.scala ---
@@ -0,0 +1,89 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF)
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/9889#discussion_r4664
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/CacheSuite.scala ---
@@ -0,0 +1,89 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF)
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/9889#discussion_r46000945
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala ---
@@ -579,11 +580,50 @@ class Dataset[T] private[sql](
*/
def takeAsList
Github user gatorsmile commented on the pull request:
https://github.com/apache/spark/pull/9889#issuecomment-159969380
@marmbrus Not sure if the latest code changes resolve all your concerns.
Please let me know if you have any suggestion. Thank you!
Have a good Thanksgiving D
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9889#issuecomment-159665597
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9889#issuecomment-159665598
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/4
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9889#issuecomment-159665401
**[Test build #46689 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46689/consoleFull)**
for PR 9889 at commit
[`92ede39`](https://git
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9889#issuecomment-159623533
**[Test build #46689 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46689/consoleFull)**
for PR 9889 at commit
[`92ede39`](https://gith
Github user gatorsmile commented on the pull request:
https://github.com/apache/spark/pull/9889#issuecomment-159620126
retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this
Github user gatorsmile commented on the pull request:
https://github.com/apache/spark/pull/9889#issuecomment-159618199
please retest it
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this fe
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9889#issuecomment-159614171
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/4
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9889#issuecomment-159614168
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project
Github user gatorsmile commented on the pull request:
https://github.com/apache/spark/pull/9889#issuecomment-159527582
@rxin Sure, will do the Python testing at first. Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as wel
Github user rxin commented on the pull request:
https://github.com/apache/spark/pull/9889#issuecomment-159527365
@gatorsmile just fyi if you have time, the python tests stuff is probably
much more important than the more complicated case of caching.
---
If your project is set up for
Github user gatorsmile commented on the pull request:
https://github.com/apache/spark/pull/9889#issuecomment-159526861
Now, I understood your concern. Thank you for the example! I added your
example into the newly created testcase suite `CacheSuite`. I saw the failure
and thus used `i
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/9889#discussion_r45836772
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala ---
@@ -579,11 +580,50 @@ class Dataset[T] private[sql](
*/
def takeAsList(num
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/9889#discussion_r45836459
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala ---
@@ -579,11 +580,50 @@ class Dataset[T] private[sql](
*/
def takeAsLi
Github user marmbrus commented on the pull request:
https://github.com/apache/spark/pull/9889#issuecomment-159473750
I wouldn't block merging an initial version of this feature on this, but it
would also be nice if we could support the following (this might be hard
though):
`
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/9889#discussion_r45824732
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala ---
@@ -17,6 +17,8 @@
package org.apache.spark.sql
+import org.apach
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/9889#discussion_r45824715
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala ---
@@ -579,11 +580,50 @@ class Dataset[T] private[sql](
*/
def takeAsList
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/9889#discussion_r45824631
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/DatasetSuite.scala
---
@@ -213,6 +215,56 @@ class DatasetSuite extends QueryTest with
SharedSQLContex
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/9889#discussion_r45824550
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/Queryable.scala ---
@@ -27,6 +28,7 @@ private[sql] trait Queryable {
def schema: Stru
Github user marmbrus commented on the pull request:
https://github.com/apache/spark/pull/9889#issuecomment-159471621
It would be great to also have thats that ensure that things like
`.as[Class]` do not break caching.
---
If your project is set up for it, you can reply to this email
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/9889#discussion_r45824400
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/CacheManager.scala ---
@@ -75,12 +75,12 @@ private[sql] class CacheManager extends Logging {
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/9889#discussion_r45824320
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/DatasetSuite.scala
---
@@ -213,6 +215,56 @@ class DatasetSuite extends QueryTest with
SharedSQLContex
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/9889#discussion_r45824255
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/DatasetSuite.scala
---
@@ -213,6 +215,56 @@ class DatasetSuite extends QueryTest with
SharedSQLContex
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/9889#discussion_r45824179
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/DatasetSuite.scala
---
@@ -213,6 +215,56 @@ class DatasetSuite extends QueryTest with
SharedSQLContex
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/9889#discussion_r45824083
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala ---
@@ -579,11 +580,50 @@ class Dataset[T] private[sql](
*/
def takeAsList
Github user rxin commented on the pull request:
https://github.com/apache/spark/pull/9889#issuecomment-159444815
cc @marmbrus
I will let you merge this one.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If yo
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9889#issuecomment-158860879
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9889#issuecomment-158860880
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/4
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9889#issuecomment-158860840
**[Test build #46510 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46510/consoleFull)**
for PR 9889 at commit
[`251`](https://git
Github user gatorsmile commented on the pull request:
https://github.com/apache/spark/pull/9889#issuecomment-158851730
@marmbrus Do these newly added test cases resolve your concerns?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitH
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9889#issuecomment-158851741
**[Test build #46510 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46510/consoleFull)**
for PR 9889 at commit
[`251`](https://gith
Github user gatorsmile commented on the pull request:
https://github.com/apache/spark/pull/9889#issuecomment-158789433
I see, will make a try. Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does
Github user marmbrus commented on the pull request:
https://github.com/apache/spark/pull/9889#issuecomment-158787958
I'm worried the existing caching mechanisms might not work on dataset
operations. Do we have a good notion of equality for encoders and lambda
functions? Can you add
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9889#issuecomment-158705760
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9889#issuecomment-158705761
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/4
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9889#issuecomment-158705743
**[Test build #46485 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46485/consoleFull)**
for PR 9889 at commit
[`c135e1f`](https://git
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9889#issuecomment-158698890
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9889#issuecomment-158698891
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/4
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9889#issuecomment-158698821
**[Test build #46485 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46485/consoleFull)**
for PR 9889 at commit
[`c135e1f`](https://gith
54 matches
Mail list logo
