Hello there again,
I think I fell again on a weird niche issue.
I try to consume some json data generated by another program, before sending
them to another program after processing (mainly filtering). At the beginning
every thing was fine… until I discover some missing data. By digging into it, I
discovered some glitches in the strings I was processing. As I’m not (yet 😉) a
scheme expert, I didn’t really know how to debug that, but I tried with the
bytevector module. I discovered that everything is fine for ascii chars, but
once one of the parsed string contains any non-ascii chars, problems begins.
Not only complicated utf8 stuff like emojis. But also accentuated letters…
I write a simple script bellow to test what happen. Everything is
self-contained: the script generates a very simple json file, then try to parse
it either directly (through with-input-from-file) or from a process call (by
calling cat on it). As you will see the direct parsing works, but not the
process output parsing. So my guess would be that it is more a process-port
than a json module issue… but as far as I can tell, both port report utf8 as
encoding.
If someone has an idea of what can be the issue there?
Here is the result of the script on my machine:
test-data1: "🤫 ascii — éè"
test-data2: "🤫 ascii — éè"
test-data3: "🤫 ascii — éè"
test-data1 == test-data2: #t
test-data1 == test-data3: #f
test-data1 string->utf8: #u8(240 159 164 171 32 97 115 99 105 105
32 226 128 148 32 195 169 195 168)
test-data3 string->utf8: #u8(240 159 164 171 159 164 171 32 97 115 99 105 105
32 226 128 148 128 148 32 195 169 169 195 168 168)
(as you can see, some bytes (159 164 171) seems to be duplicated in test-data3
before the first space (32) and again later around the –, plus the final double
168. I don’t know from where they come.
As the script is a bit long, I say thank you in advance now,
Have a good day,
Étienne
Here is the script:
(import (scheme base)
(chicken format)
(chicken process)
json)
;; Test string with utf8 chars
(define test-data1 "🤫 ascii — éè")
;; Create json file
(with-output-to-file "test.json"
(lambda ()
(fprintf (current-output-port)
"{\"test\": \"~A\"}"
test-data1)))
;; Reading it directly as a file
(define test-data2
(cdr
(vector-ref
(with-input-from-file "test.json" json-read)
0)))
;; Reading it from a process output
(define test-data3
(let* ((proc (process "cat" '("test.json")))
(stdout (process-output-port proc)))
(close-output-port (process-input-port proc))
(let ((output (json-read stdout)))
(close-input-port stdout)
(receive (process-wait proc))
(cdr (vector-ref output 0)))))
(printf "test-data1: ~S~N" test-data1)
(printf "test-data2: ~S~N" test-data2)
(printf "test-data3: ~S~N" test-data3)
(printf "test-data1 == test-data2: ~S~N" (equal? test-data1 test-data2)) ;; #t
(printf "test-data1 == test-data3: ~S~N" (equal? test-data1 test-data3)) ;; #f
<-- error
(printf "test-data1 string->utf8: ~S~N" (string->utf8 test-data1))
(printf "test-data3 string->utf8: ~S~N" (string->utf8 test-data3))
