Kasra-G opened a new pull request, #12329:
URL: https://github.com/apache/arrow-rs/pull/12329

   # Which issue does this PR close?
   
   - Closes #12320
   
   # Rationale for this change
   
   This fixes a data corruption bug when decoding surrogate pairs in json 
strings
   
   
   when you rewrite the existing code to not be a one liner. it becomes this:
   
   ```rs
   let high_bits = (high - 0xD800) as u32; // top 10 bits
   let low_bits = (low - 0xDC00) as u32;   // bottom 10 bits
   let n = 0x1_0000 | (high_bits << 10) | low_bits;;
   ```
   
   however [the RFC](https://www.rfc-editor.org/info/rfc2781/#section-2.2) 
mentions (step 5) that we need to add 0x10000, not OR it.
   
   ```rs
   let high_bits = (high - 0xD800) as u32; // top 10 bits
   let low_bits = (low - 0xDC00) as u32;   // bottom 10 bits
   let n = 0x1_0000 + (high_bits << 10) | low_bits;;
   ```
   
   the pr just makes the decoding compliant with the rfc
   
   # What changes are included in this PR?
   
   fixing the surrogate pair decoding calculation and adding some tests to 
check a few surrogate pairs. its not that extensive but it covers a few 
boundary values I think are worth. the test fails without the fix but succeeds 
with it.
   
   # Are these changes tested?
   
   yes. i added the test as mentioned above and then ran cargo test
   
   # Are there any significant user-facing changes?
   
   just a fix for data corruption of these unicode characters. I guess its 
possible someone might have depended on this incorrect behavior but I think it 
makes more sense to fix the issue.
   
   # AI usage disclosure
   
   I used ai to help find the rfc text and write a few surrogate pairs for the 
test.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to