[GitHub] spark pull request: [SPARK-8951][SparkR] support Unicode character...

Sun, 09 Aug 2015 04:59:08 -0700 

Github user CHOIJAEHONG1 commented on the pull request:

    https://github.com/apache/spark/pull/7494#issuecomment-129171042
  
    Okay, using `iconv` works nicely. But, It still needs to modify `context.R` 
to get rid of the leading UTF-8 indicating bit in a string before sending to 
the Spark though.
    However, Do we need to consider using  `Sys.setlocale()` to set the locale 
setting back to that of before the testcase because of the side effect? or Is 
there any other ways to handle it?
    
    deserialize.R
    ```
    readString <- function(con) {
      stringLen <- readInt(con)
      raw <- readBin(con, raw(), stringLen, endian = "big")
      iconv(list(raw), to="UTF-8")
    }
    ```
    
    context.R
    ```
    124   sliceLen <- ceiling(length(coll) / numSlices)
    125   slices <- split(coll, rep(1:(numSlices + 1), each = 
sliceLen)[1:length(coll)])
    126
    127   # Remove the leading UTF-8 indicating bit
    128   removeUtf8EncodingBit <- function(s) {
    129     Encoding(s) <- "bytes"
    130     s
    131   }
    132   slices_ <- rapply(slices, function(x) ifelse(is.character(x), 
removeUtf8EncodingBit(x), x), how="list")
    133
    134   # Serialize each slice: obtain a list of raws, or a list of lists 
(slices) of
    135   # 2-tuples of raws
    136   serializedSlices <- lapply(slices_, serialize, connection = NULL)
    ```
    
    ```
    test_that("collect() support Unicode characters", {
      locale <- Sys.getlocale()
      Sys.setlocale("LC_ALL", "en_US.UTF-8")
    
      lines <- c("{\"name\":\"안녕하세요\"}",
                 "{\"name\":\"您好\", \"age\":30}",
                 "{\"name\":\"こんにちは\", \"age\":19}",
                 "{\"name\":\"Xin chào\"}")
      jsonPath <- tempfile(pattern="sparkr-test", fileext=".tmp")
      writeLines(lines, jsonPath)
    
      df <- read.df(sqlContext, jsonPath, "json")
      rdf <- collect(df)
      expect_true(is.data.frame(rdf))
      expect_equal(rdf$name[1], "안녕하세요")
      expect_equal(rdf$name[2], "您好")
      expect_equal(rdf$name[3], "こんにちは")
      expect_equal(rdf$name[4], "Xin chào")
    
      df1 <- createDataFrame(sqlContext, rdf)
      expect_equal(collect(where(df1, df1$name == "您好"))$name, "您好")
      Sys.setlocale("LC_ALL", locale)
    })
    ```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to