[jira] [Commented] (SPARK-23410) Unable to read jsons in charset different from UTF-8

[ https://issues.apache.org/jira/browse/SPARK-23410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16702714#comment-16702714 ] xuqianjin commented on SPARK-23410: --- [~maxgekk] Thank you very much. I'll get started on this as soon

[jira] [Commented] (SPARK-23410) Unable to read jsons in charset different from UTF-8

[ https://issues.apache.org/jira/browse/SPARK-23410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16701640#comment-16701640 ] Maxim Gekk commented on SPARK-23410: Yes, you can. You can find more info there

[jira] [Commented] (SPARK-23410) Unable to read jsons in charset different from UTF-8

[ https://issues.apache.org/jira/browse/SPARK-23410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16701321#comment-16701321 ] xuqianjin commented on SPARK-23410: --- hi [~maxgekk] [~hyukjin.kwon] Thank you very much. Can I just

[jira] [Commented] (SPARK-23410) Unable to read jsons in charset different from UTF-8

[ https://issues.apache.org/jira/browse/SPARK-23410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16700799#comment-16700799 ] Maxim Gekk commented on SPARK-23410: > Even if lineSeps is set, it is still necessary to identify

[jira] [Commented] (SPARK-23410) Unable to read jsons in charset different from UTF-8

[ https://issues.apache.org/jira/browse/SPARK-23410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16700027#comment-16700027 ] Hyukjin Kwon commented on SPARK-23410: -- I know BOM is only the beginning of the file .. just asked

[jira] [Commented] (SPARK-23410) Unable to read jsons in charset different from UTF-8

[ https://issues.apache.org/jira/browse/SPARK-23410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16699803#comment-16699803 ] xuqianjin commented on SPARK-23410: --- hi [~maxgekk] [~hyukjin.kwon] I think there are two things to

[jira] [Commented] (SPARK-23410) Unable to read jsons in charset different from UTF-8

[ https://issues.apache.org/jira/browse/SPARK-23410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16699333#comment-16699333 ] Maxim Gekk commented on SPARK-23410: > Every line has the BOM? BOM can be only at the beginning of

[jira] [Commented] (SPARK-23410) Unable to read jsons in charset different from UTF-8

[ https://issues.apache.org/jira/browse/SPARK-23410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16699272#comment-16699272 ] Hyukjin Kwon commented on SPARK-23410: -- There look no discussion made about it in that project. I

[jira] [Commented] (SPARK-23410) Unable to read jsons in charset different from UTF-8

[ https://issues.apache.org/jira/browse/SPARK-23410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16696638#comment-16696638 ] xuqianjin commented on SPARK-23410: --- hi [~hyukjin.kwon] At present, most isuses of flink are SQL Table

[jira] [Commented] (SPARK-23410) Unable to read jsons in charset different from UTF-8

[ https://issues.apache.org/jira/browse/SPARK-23410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16696593#comment-16696593 ] Hyukjin Kwon commented on SPARK-23410: -- That's not even merged yet. > Unable to read jsons in

[jira] [Commented] (SPARK-23410) Unable to read jsons in charset different from UTF-8

[ https://issues.apache.org/jira/browse/SPARK-23410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16696542#comment-16696542 ] xuqianjin commented on SPARK-23410: --- hi [~hyukjin.kwon]  this the PR

[jira] [Commented] (SPARK-23410) Unable to read jsons in charset different from UTF-8

[ https://issues.apache.org/jira/browse/SPARK-23410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16695565#comment-16695565 ] Hyukjin Kwon commented on SPARK-23410: -- [~x1q1j1], can you point me out the flink pr? > Unable to

[jira] [Commented] (SPARK-23410) Unable to read jsons in charset different from UTF-8

[ https://issues.apache.org/jira/browse/SPARK-23410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16695539#comment-16695539 ] xuqianjin commented on SPARK-23410: --- [~maxgekk] I want to support utf-16 and utf-32 with BOMs because

[jira] [Commented] (SPARK-23410) Unable to read jsons in charset different from UTF-8

[ https://issues.apache.org/jira/browse/SPARK-23410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16690616#comment-16690616 ] Maxim Gekk commented on SPARK-23410: [~x1q1j1] Encoding different from UTF-8 (except UTF-16 and

[jira] [Commented] (SPARK-23410) Unable to read jsons in charset different from UTF-8

[ https://issues.apache.org/jira/browse/SPARK-23410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16690601#comment-16690601 ] xuqianjin commented on SPARK-23410: --- I want to ask if this bug is still being fixed, I want to try to

[jira] [Commented] (SPARK-23410) Unable to read jsons in charset different from UTF-8

[ https://issues.apache.org/jira/browse/SPARK-23410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16370580#comment-16370580 ] Dongjoon Hyun commented on SPARK-23410: --- I removed the target version, 2.3.0, from here. > Unable

[jira] [Commented] (SPARK-23410) Unable to read jsons in charset different from UTF-8

[ https://issues.apache.org/jira/browse/SPARK-23410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16366547#comment-16366547 ] Maxim Gekk commented on SPARK-23410: [~sameerag] It is not blocker anymore. I unset the blocker flag.

[jira] [Commented] (SPARK-23410) Unable to read jsons in charset different from UTF-8

[ https://issues.apache.org/jira/browse/SPARK-23410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16366537#comment-16366537 ] Sameer Agarwal commented on SPARK-23410: [~maxgekk] [~smilegator] any ETA on this? As

[jira] [Commented] (SPARK-23410) Unable to read jsons in charset different from UTF-8

[ https://issues.apache.org/jira/browse/SPARK-23410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16365666#comment-16365666 ] Hyukjin Kwon commented on SPARK-23410: -- It's reverted in https://github.com/apache/spark/pull/20614

[jira] [Commented] (SPARK-23410) Unable to read jsons in charset different from UTF-8

[ https://issues.apache.org/jira/browse/SPARK-23410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16364954#comment-16364954 ] Xiao Li commented on SPARK-23410: - This is a regression we need to resolve in Spark 2.3. [~maxgekk]

[jira] [Commented] (SPARK-23410) Unable to read jsons in charset different from UTF-8

[ https://issues.apache.org/jira/browse/SPARK-23410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16364929#comment-16364929 ] Bruce Robbins commented on SPARK-23410: --- bq. I am working on a fix, just in case Oh, OK, this one

[jira] [Commented] (SPARK-23410) Unable to read jsons in charset different from UTF-8

[ https://issues.apache.org/jira/browse/SPARK-23410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16364916#comment-16364916 ] Bruce Robbins commented on SPARK-23410: --- On Spark 2.2.1, I got the same result as you. But with

[jira] [Commented] (SPARK-23410) Unable to read jsons in charset different from UTF-8

[ https://issues.apache.org/jira/browse/SPARK-23410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16364889#comment-16364889 ] Maxim Gekk commented on SPARK-23410: I am working on a fix, just in case > Unable to read jsons in

[jira] [Commented] (SPARK-23410) Unable to read jsons in charset different from UTF-8

[ https://issues.apache.org/jira/browse/SPARK-23410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16364875#comment-16364875 ] Maxim Gekk commented on SPARK-23410: I attached the file on which I tested on 2.2.1: {code:scala}

[jira] [Commented] (SPARK-23410) Unable to read jsons in charset different from UTF-8

[ https://issues.apache.org/jira/browse/SPARK-23410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16364866#comment-16364866 ] Bruce Robbins commented on SPARK-23410: --- [~maxgekk] My simple test input of [{"field1": 10,

[jira] [Commented] (SPARK-23410) Unable to read jsons in charset different from UTF-8

[ https://issues.apache.org/jira/browse/SPARK-23410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16364849#comment-16364849 ] Maxim Gekk commented on SPARK-23410: [~bersprockets] does your json contain BOM in the first 2 bytes?

[jira] [Commented] (SPARK-23410) Unable to read jsons in charset different from UTF-8

[ https://issues.apache.org/jira/browse/SPARK-23410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16364788#comment-16364788 ] Bruce Robbins commented on SPARK-23410: --- I am probably misunderstanding the issue, but I couldn't