[
https://issues.apache.org/jira/browse/SPARK-35123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon resolved SPARK-35123.
----------------------------------
Resolution: Duplicate
> read partitioned parquet: my_col=NOW replaced by <current_date> on read()
> -------------------------------------------------------------------------
>
> Key: SPARK-35123
> URL: https://issues.apache.org/jira/browse/SPARK-35123
> Project: Spark
> Issue Type: Bug
> Components: PySpark
> Affects Versions: 3.1.1
> Reporter: Killian
> Priority: Major
>
> When reading parquet file partitioned with a column containing the value
> "NOW", The value is interpreted as now() and replaced by the current time at
> the moment of the read() funct is executed
> {code:java}
> // step to reproduce
> df = spark.createDataFrame(data=[("NOW",1), ("TEST", 2)], schema = ["col1",
> "id"])
> df.write.partitionBy("col1").parquet("test/test.parquet")
> >>> /home/test/test.parquet/col1=NOW
> df_loaded = spark.read.option(
> "basePath",
> "test/test.parquet",
> ).parquet("test/test.parquet/col1=*")
> >>>
> +---+--------------------------+
> |id |col1 |
> +---+--------------------------+
> |2 |TEST |
> |1 |2021-04-18 14:36:46.532273|
> +---+--------------------------+{code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
