[
https://issues.apache.org/jira/browse/ARROW-16157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17541072#comment-17541072
]
Nicola Crane commented on ARROW-16157:
--------------------------------------
Figured it out [~egillax]!
Two important things to consider here:
1. Row order is not guaranteed when reading in datasets
2. Datasets are not read into memory until you call {{{}collect(){}}},
{{{}pull(){}}}, or similar.
In your first example, you read in both columns at once and so the values in
each row are properly matched. However, in your second example, you read in
the data 1 column at a time (so technically reading it in on 2 separate
occasions, but with different columns chosen), and so on the occasions where
you're getting values > 1 when you run {{{}unique(newSparse@x){}}}, those will
be where the values have been read in in different orders.
This isn't a bug, though obviously not idea in this case here.
If you are using a single file to store the data instead of datasets, you could
use {{write_feather()}} and {{read_feather();}} in this case, order *is*
guaranteed, and you won't have the same problem.
> [R] Inconsistent behavior for arrow datasets vs working in memory
> -----------------------------------------------------------------
>
> Key: ARROW-16157
> URL: https://issues.apache.org/jira/browse/ARROW-16157
> Project: Apache Arrow
> Issue Type: Bug
> Affects Versions: 7.0.0
> Environment: Ubuntu 21.10
> R 4.1.3.
> Arrow 7.0.0
> Reporter: Egill Axfjord Fridgeirsson
> Assignee: Nicola Crane
> Priority: Major
>
> When I generate a sparse matrix using indices from an arrow dataset I get
> inconsistent behavior, sometimes there are duplicated indexes resulting in a
> matrix with values more than one at some places. When loading the dataset
> first in memory everything works as expected and all the values are one
> Repro
> {code:java}
> library(Matrix)
> library(dplyr)
> library(arrow)
> sparseMatrix <- Matrix::rsparsematrix(1e5,1e3, 0.05, repr="T")
> dF <- data.frame(i=sparseMatrix@i + 1, j=sparseMatrix@j + 1)
> arrow::write_dataset(dF, path='./data/feather', format='feather')
> arrowDataset <- arrow::open_dataset('./data/feather', format='feather')
> # run the below a few times, and at some time the output is more than just #
> 1 for unique(newSparse@x), indicating there are
> # duplicate indices for the sparse matrix (then it adds the values there)
> newSparse <- Matrix::sparseMatrix(i = arrowDataset %>% pull(i) ,
> j = arrowDataset %>% pull(j),
> x = 1)
> unique(newSparse@x) # here is the bug, @x is the slot for values
> arrowInMemory <- arrowDataset %>% collect()
> # after loading in memory the output is never more than 1 no matter how
> # often I run it
> newSparse <- Matrix::sparseMatrix(i = arrowInMemory %>% pull(i) ,
> j = arrowInMemory %>% pull(j),
> x = 1)
> unique(newSparse@x){code}
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
