[
https://issues.apache.org/jira/browse/SPARK-28266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17014048#comment-17014048
]
Dongjoon Hyun commented on SPARK-28266:
---------------------------------------
In Apache Spark 2.4.4, I also cannot reproduce this issue.
{code}
scala> spark.version
res0: String = 2.4.4
scala> sql("create table t as select 1 as id")
res1: org.apache.spark.sql.DataFrame = []
scala> sql("desc extended t").show(false)
+----------------------------+----------------------------------------------------------+-------+
|col_name |data_type
|comment|
+----------------------------+----------------------------------------------------------+-------+
|id |int
|null |
| |
| |
|# Detailed Table Information|
| |
|Database |default
| |
|Table |t
| |
|Owner |dongjoon
| |
|Created Time |Sun Jan 12 22:26:18 PST 2020
| |
|Last Access |Wed Dec 31 16:00:00 PST 1969
| |
|Created By |Spark 2.4.4
| |
|Type |MANAGED
| |
|Provider |hive
| |
|Table Properties |[transient_lastDdlTime=1578896780]
| |
|Statistics |2 bytes
| |
|Location |file:/Users/dongjoon/spark-warehouse/t
| |
|Serde Library
|org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe | |
|InputFormat |org.apache.hadoop.mapred.TextInputFormat
| |
|OutputFormat
|org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat| |
|Storage Properties |[serialization.format=1]
| |
|Partition Provider |Catalog
| |
+----------------------------+----------------------------------------------------------+-------+
scala> sql("ALTER TABLE t SET SERDEPROPERTIES (
'path'='file:/Users/dongjoon/spark-warehouse/t' )")
res3: org.apache.spark.sql.DataFrame = []
scala> sql("select * from t").show
+---+
| id|
+---+
| 1|
+---+
{code}
> data duplication when `path` serde property is present
> ------------------------------------------------------
>
> Key: SPARK-28266
> URL: https://issues.apache.org/jira/browse/SPARK-28266
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 2.2.0, 2.2.1, 2.2.2, 2.2.3, 2.3.0, 2.3.1, 2.3.2, 2.3.3,
> 2.3.4, 2.4.0, 2.4.1, 2.4.2, 2.4.3, 2.4.4, 3.0.0
> Reporter: Ruslan Dautkhanov
> Priority: Major
> Labels: correctness
>
> Spark duplicates returned datasets when `path` serde is present in a parquet
> table.
> Confirmed versions affected: Spark 2.2, Spark 2.3, Spark 2.4.
> Confirmed unaffected versions: Spark 2.1 and earlier (tested with Spark 1.6
> at least).
> Reproducer:
> {code:python}
> >>> spark.sql("create table ruslan_test.test55 as select 1 as id")
> DataFrame[]
> >>> spark.table("ruslan_test.test55").explain()
> == Physical Plan ==
> HiveTableScan [id#16], HiveTableRelation `ruslan_test`.`test55`,
> org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe, [id#16]
> >>> spark.table("ruslan_test.test55").count()
> 1
> {code}
> (all is good at this point, now exist session and run in Hive for example - )
> {code:sql}
> ALTER TABLE ruslan_test.test55 SET SERDEPROPERTIES (
> 'path'='hdfs://epsdatalake/hivewarehouse/ruslan_test.db/test55' )
> {code}
> So LOCATION and serde `path` property would point to the same location.
> Now see count returns two records instead of one:
> {code:python}
> >>> spark.table("ruslan_test.test55").count()
> 2
> >>> spark.table("ruslan_test.test55").explain()
> == Physical Plan ==
> *(1) FileScan parquet ruslan_test.test55[id#9] Batched: true, Format:
> Parquet, Location:
> InMemoryFileIndex[hdfs://epsdatalake/hivewarehouse/ruslan_test.db/test55,
> hdfs://epsdatalake/hive..., PartitionFilters: [], PushedFilters: [],
> ReadSchema: struct<id:int>
> >>>
> {code}
> Also notice that the presence of `path` serde property makes TABLE location
> show up twice -
> {quote}
> InMemoryFileIndex[hdfs://epsdatalake/hivewarehouse/ruslan_test.db/test55,
> hdfs://epsdatalake/hive...,
> {quote}
> We have some applications that create parquet tables in Hive with `path`
> serde property
> and it makes data duplicate in query results.
> Hive, Impala etc and Spark version 2.1 and earlier read such tables fine, but
> not Spark 2.2 and later releases.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
