[GitHub] spark issue #15274: [SPARK-17699] Support for parsing JSON string columns
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/15274 Based on the comment @marmbrus in a JIRA, we prefer to using our DDL format. For example, like what we did for CREATE TABLE, we can specify the schema using `a int, b string` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15274: [SPARK-17699] Support for parsing JSON string columns
Github user Sazpaimon commented on the issue: https://github.com/apache/spark/pull/15274 @gatorsmile Alternatively, one can use do what brickhouse's `from_json` Hive UDF does ( https://gist.github.com/jeromebanks/8855408#file-gistfile1-sql ) (For the record, I actually need this in SQL) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15274: [SPARK-17699] Support for parsing JSON string columns
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/15274 Actually, to specify the schema in SQL language, maybe we can use a JSON string. A little bit odd. Let us see whether users need it in SQL --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15274: [SPARK-17699] Support for parsing JSON string columns
Github user yhuai commented on the issue: https://github.com/apache/spark/pull/15274 @DanielMe oh, I see. `get_json_object` will not parse json array. You need to have a UDF to do that for Spark 1.6. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15274: [SPARK-17699] Support for parsing JSON string columns
Github user DanielMe commented on the issue: https://github.com/apache/spark/pull/15274 @yhuai thanks! My impression was that `get_json_object` does not convert json arrays to `ArrayType`s, maybe I misunderstood the way it's supposed to be used though. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15274: [SPARK-17699] Support for parsing JSON string columns
Github user yhuai commented on the issue: https://github.com/apache/spark/pull/15274 @DanielMe The best options for 1.6 are `get_json_object ` and `json_tuple` (their docs can be found at https://spark.apache.org/docs/1.6.0/api/scala/index.html#org.apache.spark.sql.functions$). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15274: [SPARK-17699] Support for parsing JSON string columns
Github user DanielMe commented on the issue: https://github.com/apache/spark/pull/15274 Is there any workaround I can use to achieve a similar effect in 1.6? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15274: [SPARK-17699] Support for parsing JSON string columns
Github user marmbrus commented on the issue: https://github.com/apache/spark/pull/15274 @HyukjinKwon absolutely. I actually changed the name from `json_parser` to `from_json` in anticipation of adding `to_json` :) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15274: [SPARK-17699] Support for parsing JSON string columns
Github user yhuai commented on the issue: https://github.com/apache/spark/pull/15274 LGTM. Merging to master. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15274: [SPARK-17699] Support for parsing JSON string columns
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/15274 @marmbrus I just wonder if adding `to_json` make senses (although maybe it should be done in another PR). Just curious. I am just imaging the case to write out dataframes by some data sources not supporting nested structured types. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15274: [SPARK-17699] Support for parsing JSON string columns
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15274 **[Test build #66052 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66052/consoleFull)** for PR 15274 at commit [`360b97b`](https://github.com/apache/spark/commit/360b97b7ebd9a0062b1bd65b1a536815420be1ed). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15274: [SPARK-17699] Support for parsing JSON string columns
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15274 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66052/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15274: [SPARK-17699] Support for parsing JSON string columns
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15274 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15274: [SPARK-17699] Support for parsing JSON string columns
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15274 **[Test build #66052 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66052/consoleFull)** for PR 15274 at commit [`360b97b`](https://github.com/apache/spark/commit/360b97b7ebd9a0062b1bd65b1a536815420be1ed). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15274: [SPARK-17699] Support for parsing JSON string columns
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15274 **[Test build #66048 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66048/consoleFull)** for PR 15274 at commit [`983def2`](https://github.com/apache/spark/commit/983def2c5fd61a4680f7b245729026290e6414ad). * This patch **fails Scala style tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15274: [SPARK-17699] Support for parsing JSON string columns
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15274 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66048/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15274: [SPARK-17699] Support for parsing JSON string columns
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15274 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15274: [SPARK-17699] Support for parsing JSON string columns
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15274 **[Test build #66048 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66048/consoleFull)** for PR 15274 at commit [`983def2`](https://github.com/apache/spark/commit/983def2c5fd61a4680f7b245729026290e6414ad). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15274: [SPARK-17699] Support for parsing JSON string columns
Github user marmbrus commented on the issue: https://github.com/apache/spark/pull/15274 Emailed the list. Seems like a popular feature so far :) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15274: [SPARK-17699] Support for parsing JSON string columns
Github user rxin commented on the issue: https://github.com/apache/spark/pull/15274 Might want to send a dev list email to solicit feedback on the API? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15274: [SPARK-17699] Support for parsing JSON string columns
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15274 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66016/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15274: [SPARK-17699] Support for parsing JSON string columns
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15274 **[Test build #66016 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66016/consoleFull)** for PR 15274 at commit [`62f56a7`](https://github.com/apache/spark/commit/62f56a7e4529b35f58a229097b012bc984fd458f). * This patch **fails Python style tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15274: [SPARK-17699] Support for parsing JSON string columns
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15274 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15274: [SPARK-17699] Support for parsing JSON string columns
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15274 **[Test build #66016 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66016/consoleFull)** for PR 15274 at commit [`62f56a7`](https://github.com/apache/spark/commit/62f56a7e4529b35f58a229097b012bc984fd458f). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
