anishmehta24 opened a new issue, #40459:
URL: https://github.com/apache/beam/issues/40459
### What happened?
When `escapechar` is set, `ReadFromText` skips a delimiter that comes right
after an escaped delimiter, so two records come back as one. Reproduces on
2.77.0 and master.
```python
import apache_beam as beam
with open('in.txt', 'wb') as f:
f.write(b'a\\\n\nb\n')
with beam.Pipeline() as p:
(p
| beam.io.ReadFromText('in.txt', escapechar=b'\\')
| beam.Map(lambda x: print(repr(x))))
```
Actual: `'a\\\n\nb'` (one record)
Expected: `'a\\\n'` and `'b'`
The same thing happens with a custom delimiter: `b'a\\;;b;'` read with
`delimiter=b';', escapechar=b'\\'` gives `'a\\;;b'` instead of `'a\\;'` and
`'b'`.
In `_TextSource._find_separator_bounds`, after skipping an escaped delimiter
the search resumes at `next_delim + delimiter_len + 1`, one byte too far, so a
delimiter starting at the next byte is never seen. Because of this, reading a
file in split bundles can also give a different number of records than reading
it in one piece.
### Issue Priority
Priority: 2 (default / most bugs should be filed as P2)
### Issue Components
- [x] Component: Python SDK
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
