fmichonneau opened a new issue, #35071:
URL: https://github.com/apache/arrow/issues/35071

   ### Describe the enhancement requested
   
   When trying to reproduce the content of this [blog 
post](https://voltrondata.com/resources/scaling-down-python-libraries-compress-analyze-pums-dataset)
 using R I found that it was difficult to manipulate the schema and do logical 
operations on it. I think that the main issue is that the schema is its own 
data type and that it's difficult to have a good intuition on how to work with 
it without going deep into the internals of the R6 class. I'm wondering if it 
would make sense to return the schema as a list or a tibble/data.frame to make 
it easier to interact with. It would give us name update for free for instance 
but it would require additional validations to ensure that the users don't make 
the object invalid with their modifications. I'm also not familiar enough with 
the internals to know if it's possible/a good idea. I'm happy to elaborate more 
or provide feedback.
   
   Here are a couple of examples of things that I found unintuitive/difficult:
   
   ```r
   ## First let's get one of the PUMS table to work with
   library(httr2)
   library(arrow)
   
   ## Get some data (ACS PUMS 5-year 2020 Housing data for Alaska)
   out_dir <- tempdir()
   out_zip <- file.path(out_dir, "csv_hak.zip")
   
   request(
     
"https://www2.census.gov/programs-surveys/acs/data/pums/2020/5-Year/csv_hak.zip";
   ) %>%
     req_perform(path = out_file)
   
   unzip(out_file, files = "psam_h02.csv", exdir = out_dir)
   
   hak <- open_dataset(file.path(out_dir, "psam_h02.csv"), format = "csv")
   ```
   
   ## Getting the schema
   
   ```r
   ## Getting the schema
   sch_hak <- hak$schema
   ```
   
   Not a big deal, but I'm wondering whether it would make sense to provide a 
function to extract the schema. Maybe `schema()` needs to be a method that can 
be used to specify a schema by hand or extract the schema from an
   existing object? Something like `sch_hak <- schema(hak)`
   
   ## Changing the type for a given column
   
   ```r
   sch_hak[["SERIALNO"]] <- Field$create("SERIALNO", string())
   ```
   
   This feels contrived and could maybe be simplified by using a dedicated 
function (maybe re-using the `update()` method but it might be too different 
from the use in modeling, so maybe `schema_update()`):
   
   ```r
   sch_hak <- schema_update(sch_hak, "SERIALNO", string())
   ## or using position number
   sch_hak <- schema_update(sch_hak, 2, string()
   ## or maybe simply (this doesn't error but doesn't seem to have any effect):
   sch_hak[["SERIALNO"]] <- string()
   ```
   
   ## Renaming a field
   
   ```r
   names(sch_hak)[2]                # works
   names(sch_hak)[2]  <- "serialno" # doesn't work
   ```
   
   
   ## Doing logical operations on the data types
   
   It requires to use the `$ToString()` method which feels contrived:
   
   ```r
   sch_hak[["SERIALNO"]]$type$ToString() ==  "string"
   `````
   
   ### Component(s)
   
   R


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to