Kasra-G opened a new issue, #12320:
URL: https://github.com/apache/arrow-rs/issues/12320

   ### Describe the bug
   
   Hi. Not sure if right repo to report this, but encountering some data 
corruption with surrogate pairs. Can't quite understand the pattern, but some 
surrogate pairs seem to be stored incorrectly
   
   
   Thanks
   
   
   
   
   
   
   
   ### To Reproduce
   
   1. Make an arrow-json `ReaderBuilder` with a schema that has one `Utf8` 
field, `s`.
   2. Parse this json line:
      ```json
      {"s": "\ud840\udc00"}
      ```
   3. Read back `s`
   
   I was expecting `𠀀` (U+20000), UTF-8 bytes `F0 A0 80 80`. 
   
   However, got back `𐀀` (U+10000), UTF-8 bytes `F0 90 80 80`.
   
   ### Expected behavior
   
   I expect arrow json to not silently change my unicode
   
   ### Additional context
   
   Version: arrow-json 59.3.2
   
   I'm not able to test latest arrow json to see if bug still there, maybe 
someone with Claude can check that
   
   I checked a few more and wasn't quite able to determine a pattern.
   
   | Input | Expected | Actual |
   |---|---|---|
   | `"\ud800\udc00"` | U+10000 | U+10000 |
   | `"\ud801\udc37"` | U+10437 | U+10437 |
   | `"\ud83d\ude00"` | U+1F600 | U+1F600 |
   | `"\ud83d\udca9"` | U+1F4A9 | U+1F4A9 |
   | `"\ud840\udc00"` | U+20000 | U+10000 |
   | `"\ud87e\udc00"` | U+2F800 | U+1F800 |
   | `"\udb40\udc01"` | U+E0001 | U+D0001 |
   | `"\udbff\udfff"` | U+10FFFF | U+FFFFF |
   
   It seemed everything above U+1FFFF was wrong but below it seemed to be fine
   
   ### Willingness to contribute
   
   I cannot contribute a fix for this bug at this time


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to