Yikun Jiang created SPARK-35173:
-----------------------------------
Summary: Support with_columns in PySpark.dataframe
Key: SPARK-35173
URL: https://issues.apache.org/jira/browse/SPARK-35173
Project: Spark
Issue Type: Bug
Components: PySpark, SQL
Affects Versions: 3.2.0
Reporter: Yikun Jiang
Now, the pyspark can only use withColumn to do column adding a column or
replacing the existing column that has the same name. There was already have a
withColumns api in scala and java to help user adding columns. [1]
Before this added, the user can only use withColumn again and again like:
{code:java}
self.df.withColumn("key1", col("key1")).withColumn("key2",
col("key2")).withColumn("key3", col("key3")){code}
After the support, you user can use the with_columns complete batch operations:
{code:java}
self.df.with_columns(["key1", "key2", "key3"], [col("key1"), col("key2"),
col("key3")]){code}
[1]
https://github.com/apache/spark/blob/b5241c97b17a1139a4ff719bfce7f68aef094d95/sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala#L2402
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
