fmichonneau opened a new issue, #35071: URL: https://github.com/apache/arrow/issues/35071
### Describe the enhancement requested When trying to reproduce the content of this [blog post](https://voltrondata.com/resources/scaling-down-python-libraries-compress-analyze-pums-dataset) using R I found that it was difficult to manipulate the schema and do logical operations on it. I think that the main issue is that the schema is its own data type and that it's difficult to have a good intuition on how to work with it without going deep into the internals of the R6 class. I'm wondering if it would make sense to return the schema as a list or a tibble/data.frame to make it easier to interact with. It would give us name update for free for instance but it would require additional validations to ensure that the users don't make the object invalid with their modifications. I'm also not familiar enough with the internals to know if it's possible/a good idea. I'm happy to elaborate more or provide feedback. Here are a couple of examples of things that I found unintuitive/difficult: ```r ## First let's get one of the PUMS table to work with library(httr2) library(arrow) ## Get some data (ACS PUMS 5-year 2020 Housing data for Alaska) out_dir <- tempdir() out_zip <- file.path(out_dir, "csv_hak.zip") request( "https://www2.census.gov/programs-surveys/acs/data/pums/2020/5-Year/csv_hak.zip" ) %>% req_perform(path = out_file) unzip(out_file, files = "psam_h02.csv", exdir = out_dir) hak <- open_dataset(file.path(out_dir, "psam_h02.csv"), format = "csv") ``` ## Getting the schema ```r ## Getting the schema sch_hak <- hak$schema ``` Not a big deal, but I'm wondering whether it would make sense to provide a function to extract the schema. Maybe `schema()` needs to be a method that can be used to specify a schema by hand or extract the schema from an existing object? Something like `sch_hak <- schema(hak)` ## Changing the type for a given column ```r sch_hak[["SERIALNO"]] <- Field$create("SERIALNO", string()) ``` This feels contrived and could maybe be simplified by using a dedicated function (maybe re-using the `update()` method but it might be too different from the use in modeling, so maybe `schema_update()`): ```r sch_hak <- schema_update(sch_hak, "SERIALNO", string()) ## or using position number sch_hak <- schema_update(sch_hak, 2, string() ## or maybe simply (this doesn't error but doesn't seem to have any effect): sch_hak[["SERIALNO"]] <- string() ``` ## Renaming a field ```r names(sch_hak)[2] # works names(sch_hak)[2] <- "serialno" # doesn't work ``` ## Doing logical operations on the data types It requires to use the `$ToString()` method which feels contrived: ```r sch_hak[["SERIALNO"]]$type$ToString() == "string" ````` ### Component(s) R -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
