[
https://issues.apache.org/jira/browse/SPARK-20144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16650492#comment-16650492
]
Daniel Darabos commented on SPARK-20144:
----------------------------------------
Thanks Victor! I've expanded the test with a case where reordering is allowed,
and I've added some explanatory comments in the test.
[~dongjoon], what do you think? Should I try to foster more discussion? Or what
could be a next step?
> spark.read.parquet no long maintains ordering of the data
> ---------------------------------------------------------
>
> Key: SPARK-20144
> URL: https://issues.apache.org/jira/browse/SPARK-20144
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 2.0.2
> Reporter: Li Jin
> Priority: Major
>
> Hi, We are trying to upgrade Spark from 1.6.3 to 2.0.2. One issue we found is
> when we read parquet files in 2.0.2, the ordering of rows in the resulting
> dataframe is not the same as the ordering of rows in the dataframe that the
> parquet file was reproduced with.
> This is because FileSourceStrategy.scala combines the parquet files into
> fewer partitions and also reordered them. This breaks our workflows because
> they assume the ordering of the data.
> Is this considered a bug? Also FileSourceStrategy and FileSourceScanExec
> changed quite a bit from 2.0.2 to 2.1, so not sure if this is an issue with
> 2.1.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
