Kasra-G opened a new pull request, #12329: URL: https://github.com/apache/arrow-rs/pull/12329
# Which issue does this PR close? - Closes #12320 # Rationale for this change This fixes a data corruption bug when decoding surrogate pairs in json strings when you rewrite the existing code to not be a one liner. it becomes this: ```rs let high_bits = (high - 0xD800) as u32; // top 10 bits let low_bits = (low - 0xDC00) as u32; // bottom 10 bits let n = 0x1_0000 | (high_bits << 10) | low_bits;; ``` however [the RFC](https://www.rfc-editor.org/info/rfc2781/#section-2.2) mentions (step 5) that we need to add 0x10000, not OR it. ```rs let high_bits = (high - 0xD800) as u32; // top 10 bits let low_bits = (low - 0xDC00) as u32; // bottom 10 bits let n = 0x1_0000 + (high_bits << 10) | low_bits;; ``` the pr just makes the decoding compliant with the rfc # What changes are included in this PR? fixing the surrogate pair decoding calculation and adding some tests to check a few surrogate pairs. its not that extensive but it covers a few boundary values I think are worth. the test fails without the fix but succeeds with it. # Are these changes tested? yes. i added the test as mentioned above and then ran cargo test # Are there any significant user-facing changes? just a fix for data corruption of these unicode characters. I guess its possible someone might have depended on this incorrect behavior but I think it makes more sense to fix the issue. # AI usage disclosure I used ai to help find the rfc text and write a few surrogate pairs for the test. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
