[
https://issues.apache.org/jira/browse/ARROW-10115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17250151#comment-17250151
]
Joris Van den Bossche commented on ARROW-10115:
-----------------------------------------------
For reference, it doesn't seem that pandas can handle this case.
By default, empty strings are considered missing values:
{code:python}
In [1]: s = """a,b,c
...: 1,"string",2
...: ,"",4"""
In [2]: pd.read_csv(StringIO(s))
Out[2]:
a b c
0 1.0 string 2
1 NaN NaN 4
{code}
You can specify a quoting keyword, but that only has influence in this case
when indicating there is no quoting (meaning that the quotes become part of the
data values):
{code}
In [3]: import csv
In [4]: pd.read_csv(StringIO(s), quoting=csv.QUOTE_NONE)
Out[4]:
a b c
0 1.0 "string" 2
1 NaN "" 4
{code}
So this actually "achieves" the goal of not interpreting the quoted empty
string as missing value, but now you would need to manually strip all quotes
after reading to get rid of the quotes again.
There is also the option to specify which values to recognize as missing, so we
can override the default of interpreting an empty string. But, then also
"empty" slots for non-string columns get interpreted as an empty string, making
that full column string (eg in this case the first numeric column gets
interpreted wrongly):
{code}
In [5]: pd.read_csv(StringIO(s), na_values=[], keep_default_na=False)
Out[5]:
a b c
0 1 string 2
1 4
In [6]: pd.read_csv(StringIO(s), na_values=[], keep_default_na=False).values
Out[6]:
array([['1', 'string', 2],
['', '', 4]], dtype=object)
{code}
So the goal of specifying _"don't interpret quoted empty string as missing
value, but still interpret any other empty string as such"_ cannot be directly
achieved with pandas.
It does sound potentially useful to be able to do if you have such a file, but
no idea how common this would be.
> [C++] CSV empty quoted string is treated as NULL
> ------------------------------------------------
>
> Key: ARROW-10115
> URL: https://issues.apache.org/jira/browse/ARROW-10115
> Project: Apache Arrow
> Issue Type: Improvement
> Components: C++
> Affects Versions: 1.0.1
> Reporter: Maciej
> Assignee: Antoine Pitrou
> Priority: Major
>
> When parsing my CSV I have set
> {color:#267f99}ConvertOptions{color}::s{color:#001080}trings_can_be_null{color}
> to true.
> Now as I have values:
> {code:java}
> 1234,"",345
> {code}
> the string value which is an empty string is treated as NULL.
> I've checked default valeus of
> {color:#267f99}ConvertOptions{color}::{color:#001080}null_values{color} and
> there is empty string considered null, but here we have empty quoted string
> which shouldn't be treated as NULL in my opinion. Similar behavior we have in
> Postgresql, empty quoted string is not treated as a NULL:
> https://www.postgresql.org/docs/12/sql-copy.html
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
