abhioncbr commented on PR #13823: URL: https://github.com/apache/pinot/pull/13823#issuecomment-2308604701
> Thanks for digging into the problem and finding the root cause! I think `PrimitiveArrays` are introduced to reduce memory usage, so I'd prefer to keep that and handle the new types within Pinot to take advantage of the optimization. Can you check if we can fix it by enhancing `AvroSchemaUtil.processArraySchema()` to handle the primitive arrays? I tried your suggestion, but all my trials resulted in the same failures regarding double arrays. Also, I see other users facing similar implementation problems. Here are two filed issues for reference, - https://issues.apache.org/jira/browse/AVRO-4039 - https://issues.apache.org/jira/browse/AVRO-4036 IMO, we should, for now, use the custom [AvroCustomGenericData.java](https://github.com/apache/pinot/pull/13823/files#diff-5eb61add8fdbab8eb47fd54d819a1be152ab9f5c9cea457ea428b79c3e66fdd8) if we want to upgrade the dependency for `1.12.0` or wait for some minor version releases of the avro. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
