jk-shah opened a new issue, #41081:
URL: https://github.com/apache/beam/issues/41081
### What would you like to happen?
Beam YAML supports declarative `error_handling: {output: errors}`
dead-letter routing on `MapToFields`, `Filter`, `ReadFromPubSub`,
`ReadFromKafka`, `ReadFromSpanner`, and `WriteToBigQuery`, but **not** on
`ReadFromCsv` or `ReadFromJson`
(`sdks/python/apache_beam/yaml/standard_io.yaml:174-224`, backed by
`apache_beam.io.ReadFromCsv` / `ReadFromJson`).
In real-world file ingestion, CSV and JSONL files are the #1 place malformed
rows appear (corrupt numeric cells, truncated lines, unexpected column counts,
invalid UTF-8, or malformed JSON objects). Today, a single corrupt line in one
shard of `gs://bucket/raw/*.csv` fails the entire pipeline with no way in Beam
YAML to route bad records to a dead-letter sink while preserving RFC-4180
quoted multiline fields and per-shard headers.
Supporting `error_handling: {output: errors}` on `ReadFromCsv` and
`ReadFromJson`—emitting structured error rows containing the offending raw
record/line, source file path, and parse/coercion error message—would bring
file sources to full parity with Beam YAML's streaming and database connectors.
### Issue Priority
Priority: 2 (default / most feature requests should be filed as P2)
### Issue Components
- [x] Component: Python SDK
- [ ] Component: Java SDK
- [ ] Component: Go SDK
- [ ] Component: Typescript SDK
- [ ] Component: IO connector
- [ ] Component: Beam YAML
- [ ] Component: Beam examples
- [ ] Component: Beam playground
- [ ] Component: Beam katas
- [ ] Component: Website
- [ ] Component: Infrastructure
- [ ] Component: Spark Runner
- [ ] Component: Flink Runner
- [ ] Component: Prism Runner
- [ ] Component: Hazelcast Jet Runner
- [ ] Component: Google Cloud Dataflow Runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
