GitHub user jiayuasu added a comment to the discussion: Storing GeoParquet with
`zstd` compression
Yes, Sedona supports `zstd` (and any Parquet codec) for GeoParquet writes. The
GeoParquet writer reuses Spark's standard Parquet compression handling, so you
set it the same way you would for a plain Parquet write.
**Per-write, via the `compression` option:**
```python
df.write.format("geoparquet") \
.option("compression", "zstd") \
.mode("overwrite") \
.save("path/to/output")
```
**Or globally on the session:**
```python
config = SedonaContext.builder() \
.config("spark.sql.parquet.compression.codec", "zstd") \
.getOrCreate()
```
Supported codec values are `none`/`uncompressed`, `snappy` (the default),
`gzip`, `lzo`, `brotli`, `lz4`, and `zstd`. `zstd` is bundled in the Parquet
library that ships with Spark 3.3+, so no extra/native libraries are needed.
I checked this locally on a small point dataset (5,000 rows). Writing with the
default vs. `zstd`:
| Write | Output file | Codec in footer | Size |
|---|---|---|---|
| default | `…-c000.snappy.parquet` | `SNAPPY` | 146,931 B |
| `compression=zstd` | `…-c000.zstd.parquet` | `ZSTD` | 42,486 B |
The output filename gets a `.zstd.parquet` suffix, the codec shows as `ZSTD` in
the Parquet footer, the file keeps its GeoParquet `geo` metadata, and it reads
back through `sedona.read.format("geoparquet")` fine.
The option just isn't called out in the GeoParquet docs — worth adding a note
there.
GitHub link:
https://github.com/apache/sedona/discussions/3109#discussioncomment-17666620
----
This is an automatically sent email for [email protected].
To unsubscribe, please send an email to: [email protected]
