GabsPalomo opened a new issue, #34578: URL: https://github.com/apache/arrow/issues/34578
### Describe the usage question you have. Please include as many useful details as possible. I have a [zip folder with an R project in Google Drive](https://drive.google.com/file/d/1JWIw_j0CUKRe-WV4boCPhsxW6hrR8GRZ/view?usp=share_link) and it contains a script and data folder with 2 .csv.gz files. In the project I have a reproducible example with the code I'm trying to run but getting an error. Essentially, I have around 300 .csv.gz files organized in different folders and I am trying to use arrow to work with them. I can open the dataset just fine using open_dataset() but then when I go and collect it I run into this error: ! Invalid: straddling object straddles two block boundaries (try to increase block size?) Can anyone help me or guide me? I'm new to arrow so it may be that I am defining the schema wrong, or something else. I just don't know how to fix it. ``` library(arrow) library(dplyr) # Arrow data_path <- "./data" files <- list.files(data_path, full.names = TRUE) files length(files) head(file.info(files), 5) .schema <- schema( placekey= utf8(), parent_placekey=utf8(), safegraph_brand_ids=utf8(), location_name= utf8(), brands=utf8(), store_id=utf8(), top_category=utf8(), sub_category=utf8(), naics_code= int64(), latitude= float32(), longitude= float32(), street_address=utf8(), city=utf8(), region=utf8(), postal_code=int32(), open_hours=utf8(), category_tags=utf8(), opened_on=date32(), closed_on=date32(), tracking_closed_since=date32(), websites=utf8(), geometry_type=utf8(), polygon_wkt=utf8(), polygon_class=utf8(), enclosed=bool(), phone_number=float64(), is_synthetic=bool(), includes_parking_lot=bool(), iso_country_code=utf8(), wkt_area_sq_meters=uint32(), date_range_start=timestamp("s"), date_range_end=timestamp("s"), raw_visit_counts= int64(), raw_visitor_counts= int64(), visits_by_day= utf8(), poi_cbg=float64(), visitor_home_cbgs=utf8(), visitor_home_aggregation=utf8(), visitor_daytime_cbgs=utf8(), visitor_country_of_origin=utf8(), distance_from_home= int64(), median_dwell=float16(), bucketed_dwell_times=utf8(), related_same_day_brand=utf8(), related_same_month_brand=utf8(), popularity_by_hour=utf8(), popularity_by_day=utf8(), device_type=utf8(), normalized_visits_by_state_scaling=float32(), normalized_visits_by_region_naics_visits=float32(), normalized_visits_by_region_naics_visitors=float32(), normalized_visits_by_total_visits=float32(), normalized_visits_by_total_visitors=float32() ) dat <- open_dataset(data_path, format = 'csv', skip=1, schema = .schema) dat dat %>% select(region, raw_visit_counts) %>% group_by(region) %>% summarise(mean_visitors = mean(raw_visit_counts, na.rm=TRUE)) %>% collect() ->tst tst Error in `compute.arrow_dplyr_query()`: ! Invalid: straddling object straddles two block boundaries (try to increase block size?) Run `rlang::last_error()` to see where the error occurred. ``` ### Component(s) R -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
