[
https://issues.apache.org/jira/browse/SPARK-28266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17014041#comment-17014041
]
Dongjoon Hyun commented on SPARK-28266:
---------------------------------------
Hi, [~Tagar].
For now, I cannot reproduce this in `3.0.0-preview` and `3.0.0-preview2`. How
did you test this for master branch?
{code}
spark-sql> select version();
3.0.0 007c873ae34f58651481ccba30e8e2ba38a692c4
Time taken: 3.309 seconds, Fetched 1 row(s)
spark-sql> create table t as select 1 as id;
Time taken: 1.303 seconds
spark-sql> select * from t;
1
Time taken: 0.247 seconds, Fetched 1 row(s)
spark-sql> desc formatted t;
id int NULL
# Detailed Table Information
Database default
Table t
Owner dongjoon
Created Time Sun Jan 12 21:50:55 PST 2020
Last Access UNKNOWN
Created By Spark 3.0.0-preview
Type MANAGED
Provider hive
Table Properties [transient_lastDdlTime=1578894656]
Statistics 2 bytes
Location
file:/Users/dongjoon/APACHE/spark-release/spark-3.0.0-preview-bin-hadoop3.2/spark-warehouse/t
Serde Library org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
InputFormat org.apache.hadoop.mapred.TextInputFormat
OutputFormat org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
Storage Properties [serialization.format=1]
Partition Provider Catalog
Time taken: 0.098 seconds, Fetched 19 row(s)
spark-sql> ALTER TABLE t SET SERDEPROPERTIES (
'path'='file:/Users/dongjoon/APACHE/spark-release/spark-3.0.0-preview-bin-hadoop3.2/spark-warehouse/t'
);
Time taken: 0.093 seconds
spark-sql> select * from t;
1
Time taken: 0.092 seconds, Fetched 1 row(s)
spark-sql> explain select * from t;
== Physical Plan ==
Scan hive default.t [id#29], HiveTableRelation `default`.`t`,
org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe, [id#29]
Time taken: 0.064 seconds, Fetched 1 row(s)
{code}
> data duplication when `path` serde property is present
> ------------------------------------------------------
>
> Key: SPARK-28266
> URL: https://issues.apache.org/jira/browse/SPARK-28266
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 2.2.0, 2.2.1, 2.2.2, 2.2.3, 2.3.0, 2.3.1, 2.3.2, 2.3.3,
> 2.3.4, 2.4.0, 2.4.1, 2.4.2, 2.4.3, 2.4.4, 3.0.0
> Reporter: Ruslan Dautkhanov
> Priority: Major
> Labels: correctness
>
> Spark duplicates returned datasets when `path` serde is present in a parquet
> table.
> Confirmed versions affected: Spark 2.2, Spark 2.3, Spark 2.4.
> Confirmed unaffected versions: Spark 2.1 and earlier (tested with Spark 1.6
> at least).
> Reproducer:
> {code:python}
> >>> spark.sql("create table ruslan_test.test55 as select 1 as id")
> DataFrame[]
> >>> spark.table("ruslan_test.test55").explain()
> == Physical Plan ==
> HiveTableScan [id#16], HiveTableRelation `ruslan_test`.`test55`,
> org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe, [id#16]
> >>> spark.table("ruslan_test.test55").count()
> 1
> {code}
> (all is good at this point, now exist session and run in Hive for example - )
> {code:sql}
> ALTER TABLE ruslan_test.test55 SET SERDEPROPERTIES (
> 'path'='hdfs://epsdatalake/hivewarehouse/ruslan_test.db/test55' )
> {code}
> So LOCATION and serde `path` property would point to the same location.
> Now see count returns two records instead of one:
> {code:python}
> >>> spark.table("ruslan_test.test55").count()
> 2
> >>> spark.table("ruslan_test.test55").explain()
> == Physical Plan ==
> *(1) FileScan parquet ruslan_test.test55[id#9] Batched: true, Format:
> Parquet, Location:
> InMemoryFileIndex[hdfs://epsdatalake/hivewarehouse/ruslan_test.db/test55,
> hdfs://epsdatalake/hive..., PartitionFilters: [], PushedFilters: [],
> ReadSchema: struct<id:int>
> >>>
> {code}
> Also notice that the presence of `path` serde property makes TABLE location
> show up twice -
> {quote}
> InMemoryFileIndex[hdfs://epsdatalake/hivewarehouse/ruslan_test.db/test55,
> hdfs://epsdatalake/hive...,
> {quote}
> We have some applications that create parquet tables in Hive with `path`
> serde property
> and it makes data duplicate in query results.
> Hive, Impala etc and Spark version 2.1 and earlier read such tables fine, but
> not Spark 2.2 and later releases.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
