liukun4515 commented on issue #2387: URL: https://github.com/apache/arrow-rs/issues/2387#issuecomment-1224117152
> > Can you tell me where we handle the overflow? I can't get your thoughts > > In all the places where we are currently doing validation? > > > But as far as Decimal data type, If I cast the Decimal(8,0) array) to Decimal(12,0) array`, each element can't be overflow or out of the range for the new precision, we don't need to do validation and improve the performance. > > Only if you can assert that the data in the `Decimal(8,0)` was in bounds of the precision of 8 😁 That's what is so wildly inconsistent at the moment, if you don't consistently validate, you can't elide as you don't know if validation has actually occurred We can do the strict validation, but just skip the point where we make sure it doesn't need the validation. If we follow the strict validation, we can make sure all the element in the `DecimalArray(8,0)` are in the bounds of the precision 8, so we don't need do the validation when casting from `decimal(8,0)` to `decimal(12,0). That is what I am doing!! I only find three scenes that we don't need to do validation when reading decimal data from parquet file (the schema is got from the parquet file metadata. The data/metadata in the parquet are matched); casting decimal array from the small range to the big range; take operation. @tustvold -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@arrow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
