sc-abhisheksoni commented on issue #20393: [SPARK-23207][SQL] Shuffle+Repartition on a DataFrame could lead to incorrect answers URL: https://github.com/apache/spark/pull/20393#issuecomment-518447174 I encountered this bug in my code on databricks: 20-50 Node cluster was using - Spark 2.4.3, Scala 2.11 I had a Dataframe with records for Unique IDs. Once the data was ready to write to BLOB storage, I repartitioned the data to 2 partitions and wrote to storage. On reading the records back from storage the number of total records remained the same but the number of Unique Ids reduced. On looking at the data the repartitioning introduced duplicate records in the data that was written to BLOB. As descriibed above the problem was non deterministic, and sometimes we got correct number of unique records while other times some data was duplicated. Once I removed Repartitioning I have not encountered this issue again.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
