paleolimbot opened a new issue, #1220: URL: https://github.com/apache/sedona-db/issues/1220
Some initial attempts at implementing a spatialbench Raster run have resulted in immediate OOMs for sf1, which is expected because we use the default batch size (8192) and materializing 8000 rasters is almost always a bad idea. When we have an operation that will allocate a new raster for every row (e.g., ensure loaded or rasterize), we need to adaptively shrink the batch size based on some heuristic of number of bytes that will be allocated. I think this will be a physical optimizer rule inserted before a projection or filter, and in its initial form it could be crude (just make the batches defensively smaller when a projection contains a raster operation). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
