Xi Wang created SPARK-20681:
-------------------------------
Summary: DataFram.Drop doesn't take effective, neither does error
Key: SPARK-20681
URL: https://issues.apache.org/jira/browse/SPARK-20681
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 2.1.0
Reporter: Xi Wang
Priority: Critical
I am running the following code trying to drop nested columns, but it doesn't
work, it doesn't return error either.
*I read the DF from this json:*
{'parent':{'child':{'grandchild':{'val':'1',' val_to_be_deleted':'0'}}}}
scala> spark.read.format("json").load("c:/tmp/spark_issue.json")
res0: org.apache.spark.sql.DataFrame = [father: struct<child:
struct<grandchild: struct<val: bigint, val_to_be_deleted: bigint>>>]
*read the df:*
scala> res0.printSchema
root
|-- parent: struct (nullable = true)
| |-- child: struct (nullable = true)
| | |-- grandchild: struct (nullable = true)
| | | |-- val: long (nullable = true)
| | | |-- val_to_be_deleted: long (nullable = true)
*drop the column (I tried different ways, "quote", `back-tick`, col(object)
...) column remains anyway:*
scala> res0.drop(col("father.child.grandchild.val_to_be_deleted")).printSchema
root
|-- father: struct (nullable = true)
| |-- child: struct (nullable = true)
| | |-- grandchild: struct (nullable = true)
| | | |-- val: long (nullable = true)
| | | |-- val_to_be_deleted: long (nullable = true)
scala> res0.drop("father.child.grandchild.val_to_be_deleted").printSchema
root
|-- father: struct (nullable = true)
| |-- child: struct (nullable = true)
| | |-- grandchild: struct (nullable = true)
| | | |-- val: long (nullable = true)
| | | |-- val_to_be_deleted: long (nullable = true)
Any help is appreciated.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
