[
https://issues.apache.org/jira/browse/ARROW-9522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Athanassios Hatzis updated ARROW-9522:
--------------------------------------
Description:
The following piece of code is logically correct according to documentation of
read_csv()
{code:python}
from pyarrow.csv import read_csv, ParseOptions, ConvertOptions, ReadOptions
file_location = 'spc_catalog.tsv'
sep = '\t'
nulls = ['\\N']
skip_rows = 1
select = 'sid, pid, price, quantity, inspection, check'.split(', ')
as_names = 'sid, pid, price, quantity, inspection, check'.split(', ')
as_types = 'uint32, uint32, float32, uint32, timestamp[ms], bool'.split(', ')
as_columns = dict(zip(as_names, as_types))
read_options = ReadOptions(skip_rows=skip_rows,
autogenerate_column_names=False, use_threads=True, column_names=as_names)
parse_options = ParseOptions(delimiter=sep)
convert_options = ConvertOptions(include_columns=select,
column_types=as_columns, null_values=nulls, strings_can_be_null=True)
read_csv(file_location, read_options, parse_options,
convert_options).to_pandas()
{code}
Nevertheless it produces an error
{code:python}
File "<ipython-input-2-fa6f85add265>", line 17, in <module>
read_csv(file_location, read_options, parse_options,
convert_options).to_pandas()
File "pyarrow/_csv.pyx", line 617, in pyarrow._csv.read_csv
File "pyarrow/error.pxi", line 123, in
pyarrow.lib.pyarrow_internal_check_status
File "pyarrow/error.pxi", line 85, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: CSV parse error: Expected 6 columns, got 7
{code}
The error is caused by the parameter {color:#ff0000}as_names{color} passed to
the argument {color:#ff0000}column_names{color} in
{color:#ff0000}ReadOptions(){color} method. If we set this to:
{code:python}
as_names = 'sid, pid, price, quantity, inspection, check, OMIT'.split(', ')
"""
sid pid price quantity inspection check
0 1081 991 36.099998 300.0 2014-12-20 True
1 1081 992 42.299999 400.0 2014-12-20 True
2 1081 993 15.300000 200.0 2014-03-03 False
3 1081 994 20.500000 100.0 2014-03-03 False
4 1081 995 20.500000 NaN NaT False
5 1081 996 124.230003 NaN NaT False
6 1081 997 124.230003 NaN NaT False
7 1081 998 11.700000 400.0 2014-09-10 True
8 1081 999 75.199997 NaN NaT False
9 1082 991 16.500000 200.0 2014-09-10 True
10 1082 997 0.550000 100.0 2014-09-10 True
11 1082 998 7.950000 200.0 2014-03-03 False
12 1083 998 12.500000 NaN NaT False
13 1083 999 1.000000 NaN NaT False
14 1084 994 57.299999 100.0 2014-12-20 True
15 1084 995 22.200001 NaN NaT False
16 1084 998 48.599998 200.0 2014-12-20 True
"""{code}
the code runs without errors but it is not logical to expect the user to pass
OMIT or any other string for columns that are not included
({color:#ff0000}include_columns{color}) in those to read from the flat file.
Notice that a similar parameter {color:#ff0000}column_types {color}is passed in
the right way, i.e. type for only those columns included.
Issue related
https://issues.apache.org/jira/browse/ARROW-7628
was:
The following piece of code is logically correct according to documentation of
read_csv()
{code:python}
from pyarrow.csv import read_csv, ParseOptions, ConvertOptions, ReadOptions
file_location = 'spc_catalog.tsv'
sep = '\t'
nulls = ['\\N']
skip_rows = 1
select = 'sid, pid, price, quantity, inspection, check'.split(', ')
as_names = 'sid, pid, price, quantity, inspection, check'.split(', ')
as_types = 'uint32, uint32, float32, uint32, timestamp[ms], bool'.split(', ')
as_columns = dict(zip(as_names, as_types))
read_options = ReadOptions(skip_rows=skip_rows,
autogenerate_column_names=False, use_threads=True, column_names=as_names)
parse_options = ParseOptions(delimiter=sep)
convert_options = ConvertOptions(include_columns=select,
column_types=as_columns, null_values=nulls, strings_can_be_null=True)
read_csv(file_location, read_options, parse_options,
convert_options).to_pandas()
{code}
Nevertheless it produces an error
{code:python}
File "<ipython-input-2-fa6f85add265>", line 17, in <module>
read_csv(file_location, read_options, parse_options,
convert_options).to_pandas()
File "pyarrow/_csv.pyx", line 617, in pyarrow._csv.read_csv
File "pyarrow/error.pxi", line 123, in
pyarrow.lib.pyarrow_internal_check_status
File "pyarrow/error.pxi", line 85, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: CSV parse error: Expected 6 columns, got 7
{code}
The error is caused by the parameter {color:#ff0000}as_names{color} passed to
the argument {color:#ff0000}column_names{color} in
{color:#ff0000}ReadOptions(){color} method. If we set this to:
{code:python}
as_names = 'sid, pid, price, quantity, inspection, check, OMIT'.split(', ')
"""
sid pid price quantity inspection check
0 1081 991 36.099998 300.0 2014-12-20 True
1 1081 992 42.299999 400.0 2014-12-20 True
2 1081 993 15.300000 200.0 2014-03-03 False
3 1081 994 20.500000 100.0 2014-03-03 False
4 1081 995 20.500000 NaN NaT False
5 1081 996 124.230003 NaN NaT False
6 1081 997 124.230003 NaN NaT False
7 1081 998 11.700000 400.0 2014-09-10 True
8 1081 999 75.199997 NaN NaT False
9 1082 991 16.500000 200.0 2014-09-10 True
10 1082 997 0.550000 100.0 2014-09-10 True
11 1082 998 7.950000 200.0 2014-03-03 False
12 1083 998 12.500000 NaN NaT False
13 1083 999 1.000000 NaN NaT False
14 1084 994 57.299999 100.0 2014-12-20 True
15 1084 995 22.200001 NaN NaT False
16 1084 998 48.599998 200.0 2014-12-20 True
"""{code}
the code runs without errors but it is not logical to expect the user to pass
OMIT or any other string for columns that are not included
({color:#ff0000}include_columns{color}) in those to read from the flat file.
Notice that a similar parameter {color:#ff0000}column_types {color}is passed in
the right way, i.e. type for only those columns included.
*Jira could not attach the file as there was a missing token. Please try
attaching the file again. I am linking it from the previous issue*
[^spc_catalog.tsv]
Issue related
https://issues.apache.org/jira/browse/ARROW-7628
> [Python] read_csv() case of user specified column_names AND include_columns
> ---------------------------------------------------------------------------
>
> Key: ARROW-9522
> URL: https://issues.apache.org/jira/browse/ARROW-9522
> Project: Apache Arrow
> Issue Type: Bug
> Components: Python
> Affects Versions: 0.17.1
> Reporter: Athanassios Hatzis
> Priority: Trivial
> Attachments: spc_catalog.tsv
>
>
> The following piece of code is logically correct according to documentation
> of read_csv()
> {code:python}
> from pyarrow.csv import read_csv, ParseOptions, ConvertOptions, ReadOptions
> file_location = 'spc_catalog.tsv'
> sep = '\t'
> nulls = ['\\N']
> skip_rows = 1
> select = 'sid, pid, price, quantity, inspection, check'.split(', ')
> as_names = 'sid, pid, price, quantity, inspection, check'.split(', ')
> as_types = 'uint32, uint32, float32, uint32, timestamp[ms], bool'.split(', ')
> as_columns = dict(zip(as_names, as_types))
> read_options = ReadOptions(skip_rows=skip_rows,
> autogenerate_column_names=False, use_threads=True, column_names=as_names)
> parse_options = ParseOptions(delimiter=sep)
> convert_options = ConvertOptions(include_columns=select,
> column_types=as_columns, null_values=nulls, strings_can_be_null=True)
> read_csv(file_location, read_options, parse_options,
> convert_options).to_pandas()
> {code}
> Nevertheless it produces an error
> {code:python}
> File "<ipython-input-2-fa6f85add265>", line 17, in <module>
> read_csv(file_location, read_options, parse_options,
> convert_options).to_pandas()
> File "pyarrow/_csv.pyx", line 617, in pyarrow._csv.read_csv
> File "pyarrow/error.pxi", line 123, in
> pyarrow.lib.pyarrow_internal_check_status
> File "pyarrow/error.pxi", line 85, in pyarrow.lib.check_status
> pyarrow.lib.ArrowInvalid: CSV parse error: Expected 6 columns, got 7
> {code}
> The error is caused by the parameter {color:#ff0000}as_names{color} passed to
> the argument {color:#ff0000}column_names{color} in
> {color:#ff0000}ReadOptions(){color} method. If we set this to:
> {code:python}
> as_names = 'sid, pid, price, quantity, inspection, check, OMIT'.split(', ')
> """
> sid pid price quantity inspection check
> 0 1081 991 36.099998 300.0 2014-12-20 True
> 1 1081 992 42.299999 400.0 2014-12-20 True
> 2 1081 993 15.300000 200.0 2014-03-03 False
> 3 1081 994 20.500000 100.0 2014-03-03 False
> 4 1081 995 20.500000 NaN NaT False
> 5 1081 996 124.230003 NaN NaT False
> 6 1081 997 124.230003 NaN NaT False
> 7 1081 998 11.700000 400.0 2014-09-10 True
> 8 1081 999 75.199997 NaN NaT False
> 9 1082 991 16.500000 200.0 2014-09-10 True
> 10 1082 997 0.550000 100.0 2014-09-10 True
> 11 1082 998 7.950000 200.0 2014-03-03 False
> 12 1083 998 12.500000 NaN NaT False
> 13 1083 999 1.000000 NaN NaT False
> 14 1084 994 57.299999 100.0 2014-12-20 True
> 15 1084 995 22.200001 NaN NaT False
> 16 1084 998 48.599998 200.0 2014-12-20 True
> """{code}
> the code runs without errors but it is not logical to expect the user to pass
> OMIT or any other string for columns that are not included
> ({color:#ff0000}include_columns{color}) in those to read from the flat file.
> Notice that a similar parameter {color:#ff0000}column_types {color}is passed
> in the right way, i.e. type for only those columns included.
> Issue related
> https://issues.apache.org/jira/browse/ARROW-7628
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
