[GitHub] spark issue #17693: [SPARK-20314][SQL] Inconsistent error handling in JSON p...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/17693 I like the idea but I am not sure of `DROPMALFORMED` mode though. If we use an expression with the mode enabled, whole record (not only the column but all columns) will be dropped in some json expressions, probably not a generator expressions (did I understand correctly?). I think we don't explicitly support parse modes in both `from_json`/`to_json` - https://github.com/apache/spark/blob/465818389aab1217c9de5c685cfaee3ffaec91bb/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/jsonExpressions.scala#L551 It sets `FAILFAST` but resembles `PERMISSIVE` mode up to my knowledge. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17693: [SPARK-20314][SQL] Inconsistent error handling in JSON p...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/17693 @liancheng Good suggestion! Just like what we did for `from_json/to_json`. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17693: [SPARK-20314][SQL] Inconsistent error handling in JSON p...
Github user liancheng commented on the issue: https://github.com/apache/spark/pull/17693 Not suggesting doing it in this PR but maybe adding a SQL option to let the users choose the error handling strategy of all the JSON functions probably makes more sense here? The Spark JSON data source allows users to choose a parsing mode among: - `PERMISSIVE`: replacing malformed records with nulls, - `DROPMALFORMED`: drop malformed records, and - `FAILFAST`: report an error and abort once a malformed record is found. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17693: [SPARK-20314][SQL] Inconsistent error handling in JSON p...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/17693 Could you add a test case in `JsonExpressionsSuite`? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17693: [SPARK-20314][SQL] Inconsistent error handling in JSON p...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/17693 @ewasserman Could you fix the title to point out the JIRA `SPARK-16548`? It seems `SPARK-20314` duplicates that. In terms of behaviour, I see @marmbrus and @srowen in that JIRA. Please let me cc. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17693: [SPARK-20314][SQL] Inconsistent error handling in JSON p...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17693 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
