tomjemmett opened a new issue, #41146:
URL: https://github.com/apache/arrow/issues/41146
### Describe the bug, including details regarding any error messages,
version, and platform.
When using R, `open_dataset()` does not correctly parse partitions if the
`sources` argument is a character vector of paths to individual files. However,
it does work fine in pyarrow.
My particular use case is I have a folder structure like
`x/y/{a,b,c}.parquet`, where the a, b, c files are all different types of
files. That is, I only want to load a.parquet files together, b.parquet files
together etc. (Arguably, the better structure for arrow here would be to create
a, b, c folders before the partitions...)
If I produce the list of files, I can load them in fine with `open_dataset`,
but the partitions are not loaded.
As a minimal example:
``` r
library(palmerpenguins)
library(dplyr)
library(arrow)
data(penguins)
td <- withr::local_tempdir("penguins")
penguins |>
group_by(species) |>
arrow::write_dataset(
format = "parquet",
td,
hive_style = TRUE
)
# this works fine, by passing in the folder
open_dataset(td, partitioning = c("species")) |>
count(species) |>
collect()
# does not work
td |>
fs::dir_ls(recurse = TRUE, glob = "*.parquet") |>
open_dataset(
partitioning = "species"
) |>
count(species) |>
collect()
# as per docs, fails
td |>
fs::dir_ls(recurse = TRUE, glob = "*.parquet") |>
open_dataset(
partitioning = "species",
factory_options = list(
partition_base_dir = td
)
) |>
count(species) |>
collect()
# does not work
with_dir(
td,
{
fs::dir_ls(".", recurse = TRUE, glob = "*.parquet") |>
open_dataset(
partitioning = "species"
) |>
count(species) |>
collect()
}
)
```
This works fine in python though
``` python
import pyarrow as pa
import pyarrow.dataset as ds
import os
path = "penguins"
files = [
f"{path}/{i}/{j}"
for i in os.listdir(path)
for j in os.listdir(f"{path}/{i}")
]
(
ds.dataset(
files,
partition_base_dir = path,
partitioning = ds.partitioning(
pa.schema([
("species", pa.string())
])
)
)
.to_table()
.to_pandas()
.value_counts("species")
)
```
### Component(s)
R
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
