Hey Koert,
thanks for explanation. I did not recall that every rdd/df/dataset has a
"parent" context/sqlContext. When I think about this it kinda make sense.
J.
On Fri, Oct 14, 2016 at 11:54 PM, Koert Kuipers <ko...@tresata.com> wrote:
> about the stackoverflow question, do this:
>
> def validateAndTransform(df: DataFrame) : DataFrame = {
>
> import df.sparkSession.implicits._
>
> ...
> }
>
>
>
> On Fri, Oct 14, 2016 at 5:51 PM, Koert Kuipers <ko...@tresata.com> wrote:
>
>> b
>> asically the implicit conversiosn that need it are rdd => dataset and
>> seq => dataset
>>
>> On Fri, Oct 14, 2016 at 5:47 PM, Koert Kuipers <ko...@tresata.com> wrote:
>>
>>> for example when do you Seq(1,2,3).toDF("a") it needs to get the
>>> SparkSession from somewhere. by importing the implicits from
>>> spark.implicits._ they have access to a SparkSession for operations like
>>> this.
>>>
>>> On Fri, Oct 14, 2016 at 4:42 PM, Jakub Dubovsky <
>>> spark.dubovsky.ja...@gmail.com> wrote:
>>>
>>>> Hey community,
>>>>
>>>> I would like to *educate* myself about why all *sql implicits* (most
>>>> notably conversion to Dataset API) are imported from *instance* of
>>>> SparkSession and not using static imports.
>>>>
>>>> Having this design one runs into problems like this
>>>> <http://stackoverflow.com/questions/32453886/spark-sql-dataframe-import-sqlcontext-implicits>.
>>>> It requires the presence of SparkSession instance (the only one we have) in
>>>> many parts of code. This makes code structuring harder.
>>>>
>>>> I assume that there is a *reason* why this design was *chosen*. Can
>>>> somebody please point me to a resource or explain why is this?
>>>> What is an advantage of this approach?
>>>> Or why it is not possible to implement it with static imports?
>>>>
>>>> Thanks a lot!
>>>>
>>>> Jakub
>>>>
>>>>
>>>
>>
>
