MIkhail Osckin created ARROW-1660: ------------------------------------- Summary: pandas field values are messed up across rows Key: ARROW-1660 URL: https://issues.apache.org/jira/browse/ARROW-1660 Project: Apache Arrow Issue Type: Bug Components: Python Affects Versions: 0.7.1 Environment: 4.4.0-72-generic #93-Ubuntu SMP x86_64, python3 Reporter: MIkhail Osckin
I have the following scala case class to store sparse matrix data to read it later using python case class CooVector( id: Int, row_ids: Seq[Int], rowsIdx: Seq[Int], colIdx: Seq[Int], data: Seq[Double]) I save the dataset of this type to multiple parquet files using spark and then read it using pyarrow.parquet and convert the result to pandas dataset. The problem i have is that some values end up in wrong rows, for example, row_ids might end up in wrong cooVector row. I have no idea what the reason is but might be it is related to the fact that the fields are of variable sizes. -- This message was sent by Atlassian JIRA (v6.4.14#64029)