MurrayData opened a new issue, #38842: URL: https://github.com/apache/arrow/issues/38842
### Describe the enhancement requested I have a problem with a regaulr data source in that it often contains embedded commas, but fields are not quoted. I have a workaround for this, using string manipulation to correct it, then io.BytesIO to pass the corrected data to pyarrow.csv.read_csv ```` tables = [] schema = pa.schema({c:'str' for c in columns}) print(schema) convert_options = pa.csv.ConvertOptions(column_types=schema) nfixed = 0 for f in files: lines = ','.join(columns)+'\n' with open(f) as infile: for line in infile: if line.count(',') > 15: index = -1 for i in range(5): index = line.find(',', index+1) nfixed += 1 #print(f'Fixed {line} {line.count(",")} {index} {line[index]}') line = line[:index] + ' ' + line[index+1:] lines += line print('File', f) table_file = io.BytesIO(bytes(lines, encoding='utf8')) table = pa.csv.read_csv(table_file, convert_options=convert_options) print(table.shape) tables += [table] print(f'Fixed: {nfixed}') ```` It would be good if invalid_row_handler is allowed to return a tuple containing the fixed version of the row to include in the table, for example: ```` def row_handler(row): new_row = row[:4] new_row[5] = f'{row[4]} {row[5]}'.strip() new_row[6:] = row[7:] return new_row ```` ### Component(s) Python -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@arrow.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org