[
https://issues.apache.org/jira/browse/SPARK-35123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17330391#comment-17330391
]
Max Gekk commented on SPARK-35123:
----------------------------------
The PR [https://github.com/apache/spark/pull/31549] should fix this particular
case. [~salticidae] Can you reproduce the issue on the master?
> read partitioned parquet: my_col=NOW replaced by <current_date> on read()
> -------------------------------------------------------------------------
>
> Key: SPARK-35123
> URL: https://issues.apache.org/jira/browse/SPARK-35123
> Project: Spark
> Issue Type: Bug
> Components: PySpark
> Affects Versions: 3.1.1
> Reporter: Killian
> Priority: Major
>
> When reading parquet file partitioned with a column containing the value
> "NOW", The value is interpreted as now() and replaced by the current time at
> the moment of the read() funct is executed
> {code:java}
> // step to reproduce
> df = spark.createDataFrame(data=[("NOW",1), ("TEST", 2)], schema = ["col1",
> "id"])
> df.write.partitionBy("col1").parquet("test/test.parquet")
> >>> /home/test/test.parquet/col1=NOW
> df_loaded = spark.read.option(
> "basePath",
> "test/test.parquet",
> ).parquet("test/test.parquet/col1=*")
> >>>
> +---+--------------------------+
> |id |col1 |
> +---+--------------------------+
> |2 |TEST |
> |1 |2021-04-18 14:36:46.532273|
> +---+--------------------------+{code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
