prantogg opened a new pull request, #46: URL: https://github.com/apache/sedona-spatialbench/pull/46
This PR replaces the old Zone table generation pipeline with a new DataFusion-based implementation, making DataFusion the engine of choice for end-to-end Zone table generation. Key changes: - Removed ZoneGenerator and ZoneGeneratorIterator: generation is no longer handled by the custom generator/multithreading framework. Relies on DataFusion’s execution engine for parallelism instead. - Uses DataFusion’s Arrow → Parquet writer to control row group sizes and output partitions. Row group sizing is derived from hard-coded dataset stats per scale factor (SF) so that --parquet-row-group-size is honored more accurately. - respects `--parts` / `--part` CLI options when splitting the Zone table into multiple Parquet files. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
