gszadovszky commented on PR #3393: URL: https://github.com/apache/parquet-java/pull/3393#issuecomment-4727412037
Thank you, @wgtmac. One more thing. This PR also changes how `NaN` values are encoded in dictionaries. I'm not sure if it is tightly related to IEEE-754, and is a breaking change. Before, we have lost the original `NaN` value during the dictionary encoding, and encoded every `NaN`s into the same bit pattern (e.g. `0x7ff8000000000000L` for doubles). With this change, we preserve the actual bit pattern, whatever it means. We can say this is a bug fix, but maybe, we should discuss this behavioral change with the community first. If we move to that direction, we also need to fix the dictionary filter. We use a boxed `HashSet` there, so even if the dictionary itself preserves the original bit patterns, the filter itself does not. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
