[GitHub] spark pull request: [SPARK-15097][SQL] make Dataset.sqlContext a s...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/12877


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15097][SQL] make Dataset.sqlContext a s...

[andrewor14]

Github user andrewor14 commented on the pull request:

https://github.com/apache/spark/pull/12877#issuecomment-216713635
  
Thanks merging into master 2.0


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15097][SQL] make Dataset.sqlContext a s...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/12877#issuecomment-216697634
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/57679/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15097][SQL] make Dataset.sqlContext a s...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/12877#issuecomment-216697630
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15097][SQL] make Dataset.sqlContext a s...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/12877#issuecomment-216697465
  
**[Test build #57679 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57679/consoleFull)**
 for PR 12877 at commit 
[`1672ed7`](https://github.com/apache/spark/commit/1672ed76abe6798b529ef560de2d8105f8f26d38).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15097][SQL] make Dataset.sqlContext a s...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/12877#issuecomment-216692547
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15097][SQL] make Dataset.sqlContext a s...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/12877#issuecomment-216692550
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/57675/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15097][SQL] make Dataset.sqlContext a s...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/12877#issuecomment-216692365
  
**[Test build #57675 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57675/consoleFull)**
 for PR 12877 at commit 
[`62a5ce8`](https://github.com/apache/spark/commit/62a5ce8f2fbc615e1fc14bdcddf236091c67298a).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15097][SQL] make Dataset.sqlContext a s...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/12877#issuecomment-216687286
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15097][SQL] make Dataset.sqlContext a s...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/12877#issuecomment-216687288
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/57666/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15097][SQL] make Dataset.sqlContext a s...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/12877#issuecomment-216686843
  
**[Test build #57666 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57666/consoleFull)**
 for PR 12877 at commit 
[`3804b53`](https://github.com/apache/spark/commit/3804b53d849ede69aea74b4dfe309bf76d0b2cda).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15097][SQL] make Dataset.sqlContext a s...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/12877#issuecomment-216681554
  
**[Test build #57679 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57679/consoleFull)**
 for PR 12877 at commit 
[`1672ed7`](https://github.com/apache/spark/commit/1672ed76abe6798b529ef560de2d8105f8f26d38).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15097][SQL] make Dataset.sqlContext a s...

[koertkuipers]

Github user koertkuipers commented on the pull request:

https://github.com/apache/spark/pull/12877#issuecomment-216678197
  
yup needs to be transient, will fix

On Tue, May 3, 2016 at 5:58 PM, andrewor14  wrote:

> I think it's OK for it to be lazy; just wanted to understand why. But it
> should be transient though since sparkSession is also transient.
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly or view it on GitHub
> 
>



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15097][SQL] make Dataset.sqlContext a s...

[andrewor14]

Github user andrewor14 commented on the pull request:

https://github.com/apache/spark/pull/12877#issuecomment-216677161
  
I think it's OK for it to be lazy; just wanted to understand why. But it 
should be transient though since `sparkSession` is also transient.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15097][SQL] make Dataset.sqlContext a s...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/12877#issuecomment-216676070
  
**[Test build #57675 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57675/consoleFull)**
 for PR 12877 at commit 
[`62a5ce8`](https://github.com/apache/spark/commit/62a5ce8f2fbc615e1fc14bdcddf236091c67298a).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15097][SQL] make Dataset.sqlContext a s...

[koertkuipers]

Github user koertkuipers commented on the pull request:

https://github.com/apache/spark/pull/12877#issuecomment-216675245
  
if a SparkSession sits inside a Dataset does that mean _wrapped is always
already initialized (because you cannot have a Dataset without a
SparkContext)? if so, i should probably make it a val instead of lazy val

On Tue, May 3, 2016 at 5:31 PM, Koert Kuipers  wrote:

> i made it lazy val since SparkSession.wrapped is effectively lazy too:
>   protected[sql] def wrapped: SQLContext = {
> if (_wrapped == null) {
>   _wrapped = new SQLContext(self, isRootContext = false)
> }
> _wrapped
>   }
>
>
> On Tue, May 3, 2016 at 5:29 PM, Koert Kuipers  wrote:
>
>> oh since since sparkSession is just a normal val i guess it can also be
>>
>> On Tue, May 3, 2016 at 5:25 PM, andrewor14 
>> wrote:
>>
>>> Looks good otherwise.
>>>
>>> —
>>> You are receiving this because you authored the thread.
>>> Reply to this email directly or view it on GitHub
>>> 
>>>
>>
>>
>



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15097][SQL] make Dataset.sqlContext a s...

[koertkuipers]

Github user koertkuipers commented on the pull request:

https://github.com/apache/spark/pull/12877#issuecomment-216670925
  
i made it lazy val since SparkSession.wrapped is effectively lazy too:
  protected[sql] def wrapped: SQLContext = {
if (_wrapped == null) {
  _wrapped = new SQLContext(self, isRootContext = false)
}
_wrapped
  }


On Tue, May 3, 2016 at 5:29 PM, Koert Kuipers  wrote:

> oh since since sparkSession is just a normal val i guess it can also be
>
> On Tue, May 3, 2016 at 5:25 PM, andrewor14 
> wrote:
>
>> Looks good otherwise.
>>
>> —
>> You are receiving this because you authored the thread.
>> Reply to this email directly or view it on GitHub
>> 
>>
>
>



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15097][SQL] make Dataset.sqlContext a s...

[koertkuipers]

Github user koertkuipers commented on the pull request:

https://github.com/apache/spark/pull/12877#issuecomment-216670423
  
oh since since sparkSession is just a normal val i guess it can also be

On Tue, May 3, 2016 at 5:25 PM, andrewor14  wrote:

> Looks good otherwise.
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly or view it on GitHub
> 
>



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15097][SQL] make Dataset.sqlContext a s...

[andrewor14]

Github user andrewor14 commented on the pull request:

https://github.com/apache/spark/pull/12877#issuecomment-216668953
  
Looks good otherwise.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15097][SQL] make Dataset.sqlContext a s...

[andrewor14]

Github user andrewor14 commented on a diff in the pull request:

https://github.com/apache/spark/pull/12877#discussion_r61958695
  
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala ---
@@ -211,7 +211,7 @@ class Dataset[T] private[sql](
 
   private implicit def classTag = unresolvedTEncoder.clsTag
 
-  def sqlContext: SQLContext = sparkSession.wrapped
+  lazy val sqlContext: SQLContext = sparkSession.wrapped
--- End diff --

also, any particular reason why this is lazy?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15097][SQL] make Dataset.sqlContext a s...

[andrewor14]

Github user andrewor14 commented on a diff in the pull request:

https://github.com/apache/spark/pull/12877#discussion_r61958574
  
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala ---
@@ -211,7 +211,7 @@ class Dataset[T] private[sql](
 
   private implicit def classTag = unresolvedTEncoder.clsTag
 
-  def sqlContext: SQLContext = sparkSession.wrapped
+  lazy val sqlContext: SQLContext = sparkSession.wrapped
--- End diff --

also please add a comment explaining that it must be a val because stable 
identifier is expected when you import the implicits


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15097][SQL] make Dataset.sqlContext a s...

[andrewor14]

Github user andrewor14 commented on a diff in the pull request:

https://github.com/apache/spark/pull/12877#discussion_r61958468
  
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala ---
@@ -211,7 +211,7 @@ class Dataset[T] private[sql](
 
   private implicit def classTag = unresolvedTEncoder.clsTag
 
-  def sqlContext: SQLContext = sparkSession.wrapped
+  lazy val sqlContext: SQLContext = sparkSession.wrapped
--- End diff --

this needs to be `@transient`!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15097][SQL] make Dataset.sqlContext a s...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/12877#issuecomment-216659629
  
**[Test build #57666 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57666/consoleFull)**
 for PR 12877 at commit 
[`3804b53`](https://github.com/apache/spark/commit/3804b53d849ede69aea74b4dfe309bf76d0b2cda).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15097][SQL] make Dataset.sqlContext a s...

[rxin]

Github user rxin commented on the pull request:

https://github.com/apache/spark/pull/12877#issuecomment-216659182
  
cc @andrewor14 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15097][SQL] make Dataset.sqlContext a s...

[koertkuipers]

GitHub user koertkuipers opened a pull request:

https://github.com/apache/spark/pull/12877

[SPARK-15097][SQL] make Dataset.sqlContext a stable identifier for imports

## What changes were proposed in this pull request?
Make Dataset.sqlContext a lazy val so that its a stable identifier and can 
be used for imports.
Now this works again:
import someDataset.sqlContext.implicits._

## How was this patch tested?
Add unit test to DatasetSuite that uses the import show above.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tresata/spark feat-sqlcontext-stable-import

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/12877.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #12877


commit 3804b53d849ede69aea74b4dfe309bf76d0b2cda
Author: Koert Kuipers 
Date:   2016-05-03T20:36:09Z

make Dataset.sqlContext a lazy val so that its a stable identifier and can 
be used for imports (e.g. import someDataset.sqlContext.implicits._)




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org