[
https://issues.apache.org/jira/browse/SPARK-22231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17072212#comment-17072212
]
fqaiser94 commented on SPARK-22231:
---
Makes sense, thanks!
> Support of map, filter, withColumn,
[
https://issues.apache.org/jira/browse/SPARK-22231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17071969#comment-17071969
]
DB Tsai commented on SPARK-22231:
-
[~fqaiser94] Thanks for continuing this work. We implemented this
[
https://issues.apache.org/jira/browse/SPARK-22231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17071945#comment-17071945
]
fqaiser94 commented on SPARK-22231:
---
Excellent, thanks Reynold.
If somebody could assign this ticket
[
https://issues.apache.org/jira/browse/SPARK-22231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17070688#comment-17070688
]
Reynold Xin commented on SPARK-22231:
-
[~fqaiser94] thanks for your persistence and my apologies for
[
https://issues.apache.org/jira/browse/SPARK-22231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17023919#comment-17023919
]
fqaiser94 commented on SPARK-22231:
---
@rxin no problems, take your time.
Apparently Spark 3.0 code
[
https://issues.apache.org/jira/browse/SPARK-22231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17015705#comment-17015705
]
Reynold Xin commented on SPARK-22231:
-
Hey sorry. Been pretty busy. I will take a look this week.
>
[
https://issues.apache.org/jira/browse/SPARK-22231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17014252#comment-17014252
]
fqaiser94 commented on SPARK-22231:
---
[~rxin] do you have any other questions/thoughts on this?
>
[
https://issues.apache.org/jira/browse/SPARK-22231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17007543#comment-17007543
]
fqaiser94 commented on SPARK-22231:
---
Just to be complete, there is a third option here.
We don't
[
https://issues.apache.org/jira/browse/SPARK-22231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17007215#comment-17007215
]
Reynold Xin commented on SPARK-22231:
-
[~fqaiser94] you convinced me with #2. It'd be very verbose
[
https://issues.apache.org/jira/browse/SPARK-22231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17007191#comment-17007191
]
fqaiser94 commented on SPARK-22231:
---
I feel strongly that {{withField}} and {{withFieldRenamed}}
[
https://issues.apache.org/jira/browse/SPARK-22231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17007131#comment-17007131
]
Henry Davidge commented on SPARK-22231:
---
I agree that these methods make more sense on DataFrame
[
https://issues.apache.org/jira/browse/SPARK-22231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17007117#comment-17007117
]
Reynold Xin commented on SPARK-22231:
-
Makes sense. One question (I've asked about this before):
[
https://issues.apache.org/jira/browse/SPARK-22231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17007116#comment-17007116
]
fqaiser94 commented on SPARK-22231:
---
Hi folks, I can personally affirm that this would be a valuable
[
https://issues.apache.org/jira/browse/SPARK-22231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16734160#comment-16734160
]
Timothy Pharo commented on SPARK-22231:
---
[~dbtsai] and [~jeremyrsmith], this all looks great and
[
https://issues.apache.org/jira/browse/SPARK-22231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16252611#comment-16252611
]
DB Tsai commented on SPARK-22231:
-
Thanks [~jeremyrsmith] for adding more details. There are couple
[
https://issues.apache.org/jira/browse/SPARK-22231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16206143#comment-16206143
]
Jeremy Smith commented on SPARK-22231:
--
[~viirya] - It is confusing that the lambda is {{Column =>
[
https://issues.apache.org/jira/browse/SPARK-22231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16199789#comment-16199789
]
Liang-Chi Hsieh commented on SPARK-22231:
-
[~Jeremy Smith] Thanks for the context. Regarding the
[
https://issues.apache.org/jira/browse/SPARK-22231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16198965#comment-16198965
]
Jeremy Smith commented on SPARK-22231:
--
I implemented these at Netflix, so I wanted to provide some
[
https://issues.apache.org/jira/browse/SPARK-22231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16198861#comment-16198861
]
Nathan Kronenfeld commented on SPARK-22231:
---
One couple related concerns...
# I think
[
https://issues.apache.org/jira/browse/SPARK-22231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16198286#comment-16198286
]
DB Tsai commented on SPARK-22231:
-
[~viirya] Thanks. I fixed the typo :) Yes, {{mapItems}} will work on
[
https://issues.apache.org/jira/browse/SPARK-22231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16198301#comment-16198301
]
Reynold Xin commented on SPARK-22231:
-
For drop columns - why not just df.drop("items.b")?
>
[
https://issues.apache.org/jira/browse/SPARK-22231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16198306#comment-16198306
]
Reynold Xin commented on SPARK-22231:
-
Can you say more? I can't think of a case in which you'd want
[
https://issues.apache.org/jira/browse/SPARK-22231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16198305#comment-16198305
]
DB Tsai commented on SPARK-22231:
-
[~rxin] {{df.drop("items.b")}} works too. We also want to have
[
https://issues.apache.org/jira/browse/SPARK-22231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16198146#comment-16198146
]
Liang-Chi Hsieh commented on SPARK-22231:
-
Btw, the capacity to work on nested data types looks
[
https://issues.apache.org/jira/browse/SPARK-22231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16198096#comment-16198096
]
Liang-Chi Hsieh commented on SPARK-22231:
-
Looks like `mapItems` is an API can work on any Array
[
https://issues.apache.org/jira/browse/SPARK-22231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16198082#comment-16198082
]
Liang-Chi Hsieh commented on SPARK-22231:
-
I think there is a typo in the second example to add a
26 matches
Mail list logo
