[ https://issues.apache.org/jira/browse/ARROW-5811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16887183#comment-16887183 ]
Neal Richardson commented on ARROW-5811: ---------------------------------------- In principle, a user could parse the header row of the CSV separately to identify the column names, then use that to define {{column_types}} mapping each name to string type. So are we just talking about how to facilitate that, whether/how to internalize that logic and expose it as a simple argument? Or is there something else? If {{column_types}} didn't have to be a map, maybe that would help. Perhaps it could also accept an array of length equal to the number of columns, or a single value, in which case it would recycle that type for every column. > [C++] CSV reader: Ability to not infer column types. > ---------------------------------------------------- > > Key: ARROW-5811 > URL: https://issues.apache.org/jira/browse/ARROW-5811 > Project: Apache Arrow > Issue Type: Improvement > Components: C++, Python > Affects Versions: 0.13.0 > Environment: Ubuntu Xenial > Reporter: Bogdan Klichuk > Priority: Minor > Labels: csv, csvparser, pyarrow > Fix For: 1.0.0 > > > I'm trying to read CSV as is. All columns as strings. I don't know the schema > of these CSVs and they will vary as they are provided by user. > Right now i'm using pandas.read_csv(dtype=str) which works great, but since > final destination of these CSVs are parquet files it seems like much more > efficient to use pyarrow.csv.read_csv in future, as soon as this becomes > available :) > I tried things like > `pyarrow.csv.read_csv(convert_types=ConvertOptions(columns_types=defaultdict(lambda: > 'string')))` but it doesn't work. > Maybe I just didnt' find something that already exists? :) -- This message was sent by Atlassian JIRA (v7.6.14#76016)