HyukjinKwon commented on issue #20125: [SPARK-17967][SQL] Support for array as an option in SQL parser URL: https://github.com/apache/spark/pull/20125#issuecomment-449186094 1.. Any behavior change in the parser? I believe there's no behaviour changes since option clause itself does not support `[` and ` ]` tokens: ``` CREATE TEMPORARY TABLE tableA USING csv OPTIONS (nullValue [2012, 1.1, 'null'], ...) ``` Currently, option value takes, string`, integer, decimals, and bools. I believe it's not ambiguous or it doesn't introduce a behaviour change in our parser. 2.. What is the corresponding interface for DataFrameReader and DataFrameWriter APIs? I wsa thinking about the interfaces as below: Scala - `Seq[String]` ``` spark.read.format("csv") .option("nullValue", Seq("2012", "Tesla", "null")) ... ``` Java - `String[]` ``` spark.read().format("csv") .option("nullValue", new String[]{"", "null", "NA"}) ... ``` Previous PR includes that APIs https://github.com/apache/spark/pull/16611/files One concern is that: ``` OPTIONS (nullValue "[2012, 1.1, 'null']", ...) ``` ``` option("[2012, 1.1, 'null']") ``` could work in the same way .. which is a bit ugly.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
