Github user CHOIJAEHONG1 commented on the pull request:
https://github.com/apache/spark/pull/7494#issuecomment-134832178
@sun-rui
I call `writeBin` with `useBytes=TRUE`, which is default to FALSE, refering
to the below.
https://stat.ethz.ch/R-manual/R-devel/library/base/html/writeLines.html
```
useBytes is for expert use. Normally (when false) character strings with
marked encodings are converted to the current encoding before being passed to
the connection (which might do further re-encoding). useBytes = TRUE suppresses
the re-encoding of marked strings so they are passed byte-by-byte to the
connection: this can be useful when strings have already been re-encoded by
e.g. iconv. (It is invoked automatically for strings with marked encoding
"bytes".)
```
It seems to work. And calling `markUtf8` is not necessary in the testcase
when I set the locale to UTF-8.
@shivaram
Do you agree on using `rawToChar`? It appears to Okay to use here.
```
writeString <- function(con, value) {
utfVal <- enc2utf8(value)
writeInt(con, as.integer(nchar(utfVal, type = "bytes") + 1))
writeBin(utfVal, con, endian = "big", useBytes=TRUE)
}
```
```
readString <- function(con) {
stringLen <- readInt(con)
raw <- readBin(con, raw(), stringLen, endian = "big")
string <- rawToChar(raw)
Encoding(string) <- "UTF-8"
string
}
```
```
test_that("collect() support Unicode characters", {
markUtf8 <- function(s) {
Encoding(s) <- "UTF-8"
s
}
lines <- c("{\"name\":\"ìë
íì¸ì\"}",
"{\"name\":\"æ¨å¥½\", \"age\":30}",
"{\"name\":\"ããã«ã¡ã¯\", \"age\":19}",
"{\"name\":\"Xin chà o\"}")
jsonPath <- tempfile(pattern="sparkr-test", fileext=".tmp")
writeLines(lines, jsonPath)
df <- read.df(sqlContext, jsonPath, "json")
rdf <- collect(df)
expect_true(is.data.frame(rdf))
expect_equal(rdf$name[1], markUtf8("ìë
íì¸ì"))
expect_equal(rdf$name[2], markUtf8("æ¨å¥½"))
expect_equal(rdf$name[3], markUtf8("ããã«ã¡ã¯"))
expect_equal(rdf$name[4], markUtf8("Xin chà o"))
df1 <- createDataFrame(sqlContext, rdf)
expect_equal(collect(where(df1, df1$name == markUtf8("æ¨å¥½")))$name,
markUtf8("æ¨å¥½"))
})
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
