MurrayData opened a new issue, #38842:
URL: https://github.com/apache/arrow/issues/38842
### Describe the enhancement requested
I have a problem with a regaulr data source in that it often contains
embedded commas, but fields are not quoted.
I have a workaround for this, using string manipulation to correct it, then
io.BytesIO to pass the corrected data to pyarrow.csv.read_csv
````
tables = []
schema = pa.schema({c:'str' for c in columns})
print(schema)
convert_options = pa.csv.ConvertOptions(column_types=schema)
nfixed = 0
for f in files:
lines = ','.join(columns)+'\n'
with open(f) as infile:
for line in infile:
if line.count(',') > 15:
index = -1
for i in range(5):
index = line.find(',', index+1)
nfixed += 1
#print(f'Fixed {line} {line.count(",")} {index}
{line[index]}')
line = line[:index] + ' ' + line[index+1:]
lines += line
print('File', f)
table_file = io.BytesIO(bytes(lines, encoding='utf8'))
table = pa.csv.read_csv(table_file, convert_options=convert_options)
print(table.shape)
tables += [table]
print(f'Fixed: {nfixed}')
````
It would be good if invalid_row_handler is allowed to return a tuple
containing the fixed version of the row to include in the table, for example:
````
def row_handler(row):
new_row = row[:4]
new_row[5] = f'{row[4]} {row[5]}'.strip()
new_row[6:] = row[7:]
return new_row
````
### Component(s)
Python
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
