[
https://issues.apache.org/jira/browse/ARROW-18006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17616557#comment-17616557
]
Neal Richardson commented on ARROW-18006:
-----------------------------------------
In this case, you may want to specify the schema for the partition columns so
that it doesn't try to infer them:
{code}
> open_dataset(f)
FileSystemDataset with 3 Parquet files
x: int32
See $metadata for additional Schema metadata
> open_dataset(f, partitioning = hive_partition(x = utf8()))
FileSystemDataset with 3 Parquet files
x: string
See $metadata for additional Schema metadata
{code}
> [R] integer coercion on parition variables
> ------------------------------------------
>
> Key: ARROW-18006
> URL: https://issues.apache.org/jira/browse/ARROW-18006
> Project: Apache Arrow
> Issue Type: Bug
> Components: R
> Affects Versions: 9.0.0
> Reporter: Carl Boettiger
> Priority: Major
>
> If we partition on a character-string column that is "integer-like", arrow
> assumes the resulting partition variable is an integer. This is particularly
> surprising because in most other cases, making a column a partition variable
> coerces it to a character (makes sense, path names are strings), but in this
> case we have the opposite coercion where a string becomes an integer.
> Consider this reprex:
> {code:java}
> library(arrow)f <- tempfile()
> df <- data.frame(x = c("00", "06", "18"))
> df |> write_dataset(f, partitioning = "x")
> open_dataset(f) # x is now an integer{code}
> {code:java}
> ## fails
> open_dataset(f) |> filter(x == "00")
> {code}
> changing this behavior could break existing code I think, so maybe this
> shouldn't be considered a bug, but I think it merits discussion.
> (For instance, we often see large datasets, e.g. from NOAA csv etc, that use
> zero-padded integers in path names to indicate file-specific metadata. One
> side effect of the type-coercion is that it also strips the zero-padding)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
