marioloko commented on issue #1604: URL: https://github.com/apache/arrow-rs/issues/1604#issuecomment-1295807844
It seems that the estimation of lz4 `uncompress_size` can cause overflow for small compress size. Any compress size smaller than 10 will overflow and though it will panic. So now that we pass the `uncompressed_size` to the `decompress` method, I see too options now: 1. To change predictions formula to return 255 for any compressed size smaller than 10. 2. To only allow lz4_raw if `uncompressed_size` is provided, and to return an error saying 'LZ4_RAW without known `uncompressed_size` is unsupported'. I would go with the second one, as even if the overflow error is only for small compression sizes, if the compressed size is 1G it will reserve ~250GB which is too much. So I would avoid prediction. What do you think? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
