parthchandra commented on PR #756: URL: https://github.com/apache/datafusion-comet/pull/756#issuecomment-2278848115
> @parthchandra I think the idea of getting all buffers at once may help for > > 1. Decimal128 cases I thought about that but the cost of creating BigInteger overshadowed the cost of getting the buffer from the vector. For high precision decimals, I feel we should focus on getting everything done in native. > 2. `Platform.getLong` is still expensive so calling `ByteBuffer.wrap(bytes).getLong(8)` might be cheaper I tried this. It's slower than your current implementation. ByteBuffer creation itself turned out to be expensive and I believe, ByteBuffer.getLong also ends up calling unsafe so I suspect the implementation might be shared. I also tried to convert the decimal bytes to long in native and calling native directly in CometVector. The `getLongDecimal` implementation is the same speed as that so `Platform.getLong` is as fast as we will be able to get for individual values. We could just pass the buffer address to a native method, vectorize the conversion, and return an array/vector of longs which is likely to be even faster, but I skipped that since the `getLongDecimal` implementation is already as fast as the jvm version and does not use any additional memory. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For additional commands, e-mail: github-h...@datafusion.apache.org
