BruAPAHE opened a new issue, #1951:
URL: https://github.com/apache/sedona/issues/1951
I encountered a runtime failure when using RS_FromNetCDF(content, 'v') on a
valid NetCDF4 file stored in S3.
The error is:
```Exception occurred while evaluating expression RS_FromNetCDF - inputs:
[[B@xxxx, v], cause: Unknown filter type=32015```
This happens because the file uses an HDF5 compression filter (likely BLOSC,
filter ID 32015), which is not supported by the version of netcdf-java (4.6.11)
used in netcdfAll.
Steps to reproduce:
Load .nc file containing BLOSC/ZSTD compression into Spark via
binaryFile.
Try to evaluate RS_FromNetCDF(content, 'variable').
A workaround is to re-encode the file using nccopy or xarray to remove
unsupported filters, but this limitation is undocumented and leads to runtime
crashes.
Expected:
An informative exception or warning.
Ideally: support for filter 32015
Environment:
Sedona: 1.7.1
Spark: 3.5.0
File: NetCDF4/HDF5 with ZSTD filter
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
