GabsPalomo opened a new issue, #34578:
URL: https://github.com/apache/arrow/issues/34578

   ### Describe the usage question you have. Please include as many useful 
details as  possible.
   
   
   I have a [zip folder with an R project in Google 
Drive](https://drive.google.com/file/d/1JWIw_j0CUKRe-WV4boCPhsxW6hrR8GRZ/view?usp=share_link)
 and it contains a script and data folder with 2 .csv.gz files. In the project 
I have a reproducible example with the code I'm trying to run but getting an 
error. 
   
   Essentially, I have around 300 .csv.gz files organized in different folders 
and I am trying to use arrow to work with them.  
   I can open the dataset just fine using open_dataset() but then when I go and 
collect it I run into this error: ! Invalid: straddling object straddles two 
block boundaries (try to increase block size?)
   
   Can anyone help me or guide me? I'm new to arrow so it may be that I am 
defining the schema wrong, or something else. I just don't know how to fix it. 
   
   ```
   library(arrow)
   library(dplyr)
   
   # Arrow 
   data_path <- "./data"
   files <- list.files(data_path, full.names = TRUE)
   files
   length(files)
   head(file.info(files), 5)
   
   .schema <- schema(
     placekey= utf8(),
     parent_placekey=utf8(),
     safegraph_brand_ids=utf8(),  
     location_name= utf8(),
     brands=utf8(),
     store_id=utf8(),
     top_category=utf8(),
     sub_category=utf8(),
     naics_code= int64(),
     latitude= float32(),
     longitude= float32(),
     street_address=utf8(),
     city=utf8(),
     region=utf8(),
     postal_code=int32(),
     open_hours=utf8(),
     category_tags=utf8(),
     opened_on=date32(),
     closed_on=date32(),
     tracking_closed_since=date32(),
     websites=utf8(),
     geometry_type=utf8(),
     polygon_wkt=utf8(),
     polygon_class=utf8(),
     enclosed=bool(),
     phone_number=float64(),
     is_synthetic=bool(),
     includes_parking_lot=bool(),
     iso_country_code=utf8(),
     wkt_area_sq_meters=uint32(),
     date_range_start=timestamp("s"), 
     date_range_end=timestamp("s"),
     raw_visit_counts= int64(),
     raw_visitor_counts= int64(),
     visits_by_day= utf8(),
     poi_cbg=float64(),
     visitor_home_cbgs=utf8(),
     visitor_home_aggregation=utf8(),
     visitor_daytime_cbgs=utf8(),
     visitor_country_of_origin=utf8(),
     distance_from_home= int64(),
     median_dwell=float16(),
     bucketed_dwell_times=utf8(),
     related_same_day_brand=utf8(),
     related_same_month_brand=utf8(),
     popularity_by_hour=utf8(),
     popularity_by_day=utf8(),
     device_type=utf8(),
     normalized_visits_by_state_scaling=float32(),
     normalized_visits_by_region_naics_visits=float32(),
     normalized_visits_by_region_naics_visitors=float32(),
     normalized_visits_by_total_visits=float32(),
     normalized_visits_by_total_visitors=float32()
   )
   
   dat <- open_dataset(data_path, format = 'csv', skip=1, schema = .schema)
   dat
   
   dat %>%
     select(region, raw_visit_counts) %>% 
     group_by(region) %>% 
     summarise(mean_visitors = mean(raw_visit_counts, na.rm=TRUE)) %>%
     collect() ->tst
   tst
   
   Error in `compute.arrow_dplyr_query()`:
   ! Invalid: straddling object straddles two block boundaries (try to increase 
block size?)
   Run `rlang::last_error()` to see where the error occurred.
   ```
   
   ### Component(s)
   
   R


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to