[ https://issues.apache.org/jira/browse/SPARK-13953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hyukjin Kwon updated SPARK-13953: --------------------------------- Description: It would be great if we maybe set {{spark.sql.columnNameOfCorruptRecord}} via {{option()}} not as a global option. Wenchen said, https://github.com/apache/spark/pull/11756#issuecomment-197644079 {quote} This is a good point, if we only use this config while dealing with json data... {quote} Reynold said, https://github.com/apache/spark/pull/11756#issuecomment-197645874 {quote} ...I think it should be an option on each read, not a global option. Most global options don't make a lot of sense as global options. {quote} {quote} But since we had it, i'd say we should keep it to avoid breaking compatibility. We can have the per-read option override the global option. {quote} was: It would be great if we maybe make {{spark.sql.columnNameOfCorruptRecord}} as an option. Wenchen said, https://github.com/apache/spark/pull/11756#issuecomment-197644079 {quote} This is a good point, if we only use this config while dealing with json data... {quote} Reynold said, https://github.com/apache/spark/pull/11756#issuecomment-197645874 {quote} ...I think it should be an option on each read, not a global option. Most global options don't make a lot of sense as global options. {quote} {quote} But since we had it, i'd say we should keep it to avoid breaking compatibility. We can have the per-read option override the global option. {quote} > Support for specifying the field name for corrupted record at JSON datasource. > ------------------------------------------------------------------------------ > > Key: SPARK-13953 > URL: https://issues.apache.org/jira/browse/SPARK-13953 > Project: Spark > Issue Type: Improvement > Components: SQL > Affects Versions: 2.0.0 > Reporter: Hyukjin Kwon > Priority: Minor > > It would be great if we maybe set {{spark.sql.columnNameOfCorruptRecord}} > via {{option()}} not as a global option. > Wenchen said, > https://github.com/apache/spark/pull/11756#issuecomment-197644079 > {quote} > This is a good point, if we only use this config while dealing with json > data... > {quote} > Reynold said, > https://github.com/apache/spark/pull/11756#issuecomment-197645874 > {quote} > ...I think it should be an option on each read, not a global option. Most > global options don't make a lot of sense as global options. > {quote} > {quote} > But since we had it, i'd say we should keep it to avoid breaking > compatibility. We can have the per-read option override the global option. > {quote} -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org