codewarnab commented on issue #41081: URL: https://github.com/apache/beam/issues/41081#issuecomment-6761496102
I'd like to work on this. I read yaml_errors.py and how ReadFromPubSub takes error_handling in yaml_io.py. I also saw that the YAML ReadFromCsv and ReadFromJson map directly to apache_beam.io.ReadFromCsv / ReadFromJson, which parse each file chunk with pandas. Plan: 1. Add an error_handling parameter to ReadFromCsv and ReadFromJson in YAML. 2. Send each bad record to the error output with the raw record, the source file path, and the error message. 3. Add tests for malformed CSV rows, malformed JSON lines, and quoted multiline CSV fields. One design question: pandas fails the whole chunk when one row is bad. Is it OK to add a record-by-record read path in yaml_io.py that runs only when error_handling is set? Or do you prefer the change inside the pandas-based reader? .take-issue -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
