Re: [SQL] A bug with withColumn?

[Jacek Laskowski]

On Thu, Mar 31, 2016 at 5:47 PM, Jacek Laskowski  wrote:

> It means that it's not only possible to rename a column using
> withColumnRenamed, but also replace the content of a column (in one
> shot) using withColumn with an existing column name. I can live with
> that :)

Hi,

Not only can I live with that, but I did use it today few times
because it was so nice! I used this when I needed to `cast` a column
and I could do it with just `withColumn`. Awesome! You should've seen
the faces of the people I've presented the solution to :)

Pozdrawiam,
Jacek Laskowski

https://medium.com/@jaceklaskowski/
Mastering Apache Spark http://bit.ly/mastering-apache-spark
Follow me at https://twitter.com/jaceklaskowski

-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org

Re: [SQL] A bug with withColumn?

[Jacek Laskowski]

Hi,

Thanks Ted.

It means that it's not only possible to rename a column using
withColumnRenamed, but also replace the content of a column (in one
shot) using withColumn with an existing column name. I can live with
that :)

Pozdrawiam,
Jacek Laskowski

https://medium.com/@jaceklaskowski/
Mastering Apache Spark http://bit.ly/mastering-apache-spark
Follow me at https://twitter.com/jaceklaskowski


On Thu, Mar 31, 2016 at 5:14 PM, Ted Yu  wrote:
> Looks like this is result of the following check:
>
> val shouldReplace = output.exists(f => resolver(f.name, colName))
> if (shouldReplace) {
>
> where existing column, text, was replaced.
>
> On Thu, Mar 31, 2016 at 12:08 PM, Jacek Laskowski  wrote:
>>
>> Hi,
>>
>> Just ran into the following. Is this a bug?
>>
>> scala> df.join(nums, df("id") === nums("id")).withColumn("TEXT2",
>> lit(5)).show
>> +---+---+---+-+-+
>> | id|   text| id| text|TEXT2|
>> +---+---+---+-+-+
>> |  0|  hello|  0|  two|5|
>> |  1|swiecie|  1|three|5|
>> +---+---+---+-+-+
>>
>>
>> scala> df.join(nums, df("id") === nums("id")).withColumn("TEXT",
>> lit(5)).show
>> +---++---++
>> | id|TEXT| id|TEXT|
>> +---++---++
>> |  0|   5|  0|   5|
>> |  1|   5|  1|   5|
>> +---++---++
>>
>> Pozdrawiam,
>> Jacek Laskowski
>> 
>> https://medium.com/@jaceklaskowski/
>> Mastering Apache Spark http://bit.ly/mastering-apache-spark
>> Follow me at https://twitter.com/jaceklaskowski
>>
>> -
>> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
>> For additional commands, e-mail: user-h...@spark.apache.org
>>
>

-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org

Re: [SQL] A bug with withColumn?

[Ted Yu]

Looks like this is result of the following check:

val shouldReplace = output.exists(f => resolver(f.name, colName))
if (shouldReplace) {

where existing column, text, was replaced.

On Thu, Mar 31, 2016 at 12:08 PM, Jacek Laskowski  wrote:

> Hi,
>
> Just ran into the following. Is this a bug?
>
> scala> df.join(nums, df("id") === nums("id")).withColumn("TEXT2",
> lit(5)).show
> +---+---+---+-+-+
> | id|   text| id| text|TEXT2|
> +---+---+---+-+-+
> |  0|  hello|  0|  two|5|
> |  1|swiecie|  1|three|5|
> +---+---+---+-+-+
>
>
> scala> df.join(nums, df("id") === nums("id")).withColumn("TEXT",
> lit(5)).show
> +---++---++
> | id|TEXT| id|TEXT|
> +---++---++
> |  0|   5|  0|   5|
> |  1|   5|  1|   5|
> +---++---++
>
> Pozdrawiam,
> Jacek Laskowski
> 
> https://medium.com/@jaceklaskowski/
> Mastering Apache Spark http://bit.ly/mastering-apache-spark
> Follow me at https://twitter.com/jaceklaskowski
>
> -
> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
> For additional commands, e-mail: user-h...@spark.apache.org
>
>

[SQL] A bug with withColumn?

[Jacek Laskowski]

Hi,

Just ran into the following. Is this a bug?

scala> df.join(nums, df("id") === nums("id")).withColumn("TEXT2", lit(5)).show
+---+---+---+-+-+
| id|   text| id| text|TEXT2|
+---+---+---+-+-+
|  0|  hello|  0|  two|5|
|  1|swiecie|  1|three|5|
+---+---+---+-+-+


scala> df.join(nums, df("id") === nums("id")).withColumn("TEXT", lit(5)).show
+---++---++
| id|TEXT| id|TEXT|
+---++---++
|  0|   5|  0|   5|
|  1|   5|  1|   5|
+---++---++

Pozdrawiam,
Jacek Laskowski

https://medium.com/@jaceklaskowski/
Mastering Apache Spark http://bit.ly/mastering-apache-spark
Follow me at https://twitter.com/jaceklaskowski

-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org