mytarmail opened a new issue, #37103:
URL: https://github.com/apache/arrow/issues/37103
### Describe the usage question you have. Please include as many useful
details as possible.
imagine that I have a huge matrix (small example)
```
dat <- matrix(1:100,ncol = 10)
colnames(dat) <- paste0("X",1:ncol(dat))
dat
X1 X2 X3 X4 X5 X6 X7 X8 X9 X10
[1,] 1 11 21 31 41 51 61 71 81 91
[2,] 2 12 22 32 42 52 62 72 82 92
[3,] 3 13 23 33 43 53 63 73 83 93
[4,] 4 14 24 34 44 54 64 74 84 94
[5,] 5 15 25 35 45 55 65 75 85 95
[6,] 6 16 26 36 46 56 66 76 86 96
[7,] 7 17 27 37 47 57 67 77 87 97
```
this matrix is divided into `columns` and written in several files
```
path <- paste0(getwd(),"/example/")
dir.create(path = path)
colum_id <- matrix(1:10,ncol = 2,byrow = T)
for(i in 1:nrow(colum_id)) write.csv(dat[,colum_id[i,]], file =
paste0(path,i,".csv"), row.names = F)
```
files in a folder
```
dir(path)
[1] "1.csv" "2.csv" "3.csv" "4.csv" "5.csv"
```
..
```
paths <- paste0(path,dir(path))
lapply(paths , \(Path) head(read.csv(Path),2) )
[[1]]
X1 X2
1 1 11
2 2 12
[[2]]
X3 X4
1 21 31
2 22 32
[[3]]
X5 X6
1 41 51
2 42 52
[[4]]
X7 X8
1 61 71
2 62 72
[[5]]
X9 X10
1 81 91
2 82 92
```
The question is how can I read this matrix (out of memory) in the arrow so
that I can access to any element of the full matrix.
All the examples that I saw with the use of the arrow were with reading
files separated by lines but not by columns.
### Component(s)
R
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
