[jira] [Commented] (ARROW-1660) pandas field values are messed up across rows
[
https://issues.apache.org/jira/browse/ARROW-1660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16219063#comment-16219063
]
Wes McKinney commented on ARROW-1660:
-
I think it might be related to splicing together files. I'll write some tests
and then close this issue; if you are able to reproduce in the future please
let us know
> pandas field values are messed up across rows
> -
>
> Key: ARROW-1660
> URL: https://issues.apache.org/jira/browse/ARROW-1660
> Project: Apache Arrow
> Issue Type: Bug
> Components: Python
>Affects Versions: 0.7.1
> Environment: 4.4.0-72-generic #93-Ubuntu SMP x86_64, python3
>Reporter: MIkhail Osckin
> Fix For: 0.8.0
>
>
> I have the following scala case class to store sparse matrix data to read it
> later using python
> {code:java}
> case class CooVector(
> id: Int,
> row_ids: Seq[Int],
> rowsIdx: Seq[Int],
> colIdx: Seq[Int],
> data: Seq[Double])
> {code}
> I save the dataset of this type to multiple parquet files using spark and
> then read it using pyarrow.parquet and convert the result to pandas dataset.
> The problem i have is that some values end up in wrong rows, for example,
> row_ids might end up in wrong cooVector row. I have no idea what the reason
> is but might be it is related to the fact that the fields are of variable
> sizes. And everything is correct if i read it using spark. Also i checked
> to_pydict method and the result is correct, so seems like the problem
> somewhere in to_pandas method.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (ARROW-1660) pandas field values are messed up across rows
[
https://issues.apache.org/jira/browse/ARROW-1660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16214063#comment-16214063
]
MIkhail Osckin commented on ARROW-1660:
---
I can't get the same environment i had this issue with and i didn't save the
parquet dataset, and so i failed at trying to reproduce it. I tend to think
that this issue exists (and of course i might be wrong), but maybe it happens
only in some rare cases. Right now to_pydict & to_pandas give me the same
results.
> pandas field values are messed up across rows
> -
>
> Key: ARROW-1660
> URL: https://issues.apache.org/jira/browse/ARROW-1660
> Project: Apache Arrow
> Issue Type: Bug
> Components: Python
>Affects Versions: 0.7.1
> Environment: 4.4.0-72-generic #93-Ubuntu SMP x86_64, python3
>Reporter: MIkhail Osckin
> Fix For: 0.8.0
>
>
> I have the following scala case class to store sparse matrix data to read it
> later using python
> {code:java}
> case class CooVector(
> id: Int,
> row_ids: Seq[Int],
> rowsIdx: Seq[Int],
> colIdx: Seq[Int],
> data: Seq[Double])
> {code}
> I save the dataset of this type to multiple parquet files using spark and
> then read it using pyarrow.parquet and convert the result to pandas dataset.
> The problem i have is that some values end up in wrong rows, for example,
> row_ids might end up in wrong cooVector row. I have no idea what the reason
> is but might be it is related to the fact that the fields are of variable
> sizes. And everything is correct if i read it using spark. Also i checked
> to_pydict method and the result is correct, so seems like the problem
> somewhere in to_pandas method.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (ARROW-1660) pandas field values are messed up across rows
[
https://issues.apache.org/jira/browse/ARROW-1660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16209597#comment-16209597
]
Wes McKinney commented on ARROW-1660:
-
Marked for 0.8.0
> pandas field values are messed up across rows
> -
>
> Key: ARROW-1660
> URL: https://issues.apache.org/jira/browse/ARROW-1660
> Project: Apache Arrow
> Issue Type: Bug
> Components: Python
>Affects Versions: 0.7.1
> Environment: 4.4.0-72-generic #93-Ubuntu SMP x86_64, python3
>Reporter: MIkhail Osckin
> Fix For: 0.8.0
>
>
> I have the following scala case class to store sparse matrix data to read it
> later using python
> {code:java}
> case class CooVector(
> id: Int,
> row_ids: Seq[Int],
> rowsIdx: Seq[Int],
> colIdx: Seq[Int],
> data: Seq[Double])
> {code}
> I save the dataset of this type to multiple parquet files using spark and
> then read it using pyarrow.parquet and convert the result to pandas dataset.
> The problem i have is that some values end up in wrong rows, for example,
> row_ids might end up in wrong cooVector row. I have no idea what the reason
> is but might be it is related to the fact that the fields are of variable
> sizes. And everything is correct if i read it using spark. Also i checked
> to_pydict method and the result is correct, so seems like the problem
> somewhere in to_pandas method.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (ARROW-1660) pandas field values are messed up across rows
[
https://issues.apache.org/jira/browse/ARROW-1660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16206191#comment-16206191
]
MIkhail Osckin commented on ARROW-1660:
---
i will try to provide a working example this week
> pandas field values are messed up across rows
> -
>
> Key: ARROW-1660
> URL: https://issues.apache.org/jira/browse/ARROW-1660
> Project: Apache Arrow
> Issue Type: Bug
> Components: Python
>Affects Versions: 0.7.1
> Environment: 4.4.0-72-generic #93-Ubuntu SMP x86_64, python3
>Reporter: MIkhail Osckin
>
> I have the following scala case class to store sparse matrix data to read it
> later using python
> {code:java}
> case class CooVector(
> id: Int,
> row_ids: Seq[Int],
> rowsIdx: Seq[Int],
> colIdx: Seq[Int],
> data: Seq[Double])
> {code}
> I save the dataset of this type to multiple parquet files using spark and
> then read it using pyarrow.parquet and convert the result to pandas dataset.
> The problem i have is that some values end up in wrong rows, for example,
> row_ids might end up in wrong cooVector row. I have no idea what the reason
> is but might be it is related to the fact that the fields are of variable
> sizes. And everything is correct if i read it using spark. Also i checked
> to_pydict method and the result is correct, so seems like the problem
> somewhere in to_pandas method.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (ARROW-1660) pandas field values are messed up across rows
[
https://issues.apache.org/jira/browse/ARROW-1660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16201441#comment-16201441
]
Wes McKinney commented on ARROW-1660:
-
Would it be possible to provide a reproducible example so that we can debug?
> pandas field values are messed up across rows
> -
>
> Key: ARROW-1660
> URL: https://issues.apache.org/jira/browse/ARROW-1660
> Project: Apache Arrow
> Issue Type: Bug
> Components: Python
>Affects Versions: 0.7.1
> Environment: 4.4.0-72-generic #93-Ubuntu SMP x86_64, python3
>Reporter: MIkhail Osckin
>
> I have the following scala case class to store sparse matrix data to read it
> later using python
> {code:java}
> case class CooVector(
> id: Int,
> row_ids: Seq[Int],
> rowsIdx: Seq[Int],
> colIdx: Seq[Int],
> data: Seq[Double])
> {code}
> I save the dataset of this type to multiple parquet files using spark and
> then read it using pyarrow.parquet and convert the result to pandas dataset.
> The problem i have is that some values end up in wrong rows, for example,
> row_ids might end up in wrong cooVector row. I have no idea what the reason
> is but might be it is related to the fact that the fields are of variable
> sizes. And everything is correct if i read it using spark. Also i checked
> to_pydict method and the result is correct, so seems like the problem
> somewhere in to_pandas method.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
