[
https://issues.apache.org/jira/browse/ARROW-13028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17360401#comment-17360401
]
Nate Clark commented on ARROW-13028:
------------------------------------
I was already working on this and discovered something which might be a problem
with my idea. It looks like any numeric value can be parsed as a float even if
the precision of the string is too much for a float. Because of this the float
parse will succeed even if it is better to parse it as a double.
Is this by design or accident?
> [C++] CSV add convert option to attempt 32bit number inferences
> ---------------------------------------------------------------
>
> Key: ARROW-13028
> URL: https://issues.apache.org/jira/browse/ARROW-13028
> Project: Apache Arrow
> Issue Type: Improvement
> Components: C++
> Reporter: Nate Clark
> Assignee: Nate Clark
> Priority: Major
>
> When types are being inferred by CSV the numbers are always 64 bit. For large
> data sets it could be better to use 32 bit types to save over all memory. To
> do this it would be useful to add an option to ConvertOptions to try 32 bit
> numbers before 64 bit. By default this option would be disabled.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
