[
https://issues.apache.org/jira/browse/SPARK-12669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15103903#comment-15103903
]
Mohit Jaggi commented on SPARK-12669:
-------------------------------------
how about the names above for a start? i think we can use typesafe config (or
an alternative). in the scala API, something like
....withFormatingOptions(...).withNumberParsingOptions(...).withLineExceptionOptions(....)
and in the sql API something like
csv.format.escapeCharacter="\\", csv.realNumberParsing.nan="NaN, Double.NaN" etc
havnt' seen the latest code but i remember a flat namespace in spark-csv. also,
i don't remember a "filler value" for lines with fewer than expected fields.
maybe it was added iater.
i am happy to write the code for this once we have agreement on specifics of
the API.
> Organize options for default values
> -----------------------------------
>
> Key: SPARK-12669
> URL: https://issues.apache.org/jira/browse/SPARK-12669
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
> Affects Versions: 2.0.0
> Reporter: Hossein Falaki
>
> CSV data source in SparkSQL should be able to differentiate empty string,
> null, NaN, “N/A” (maybe data type dependent).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
