[
https://issues.apache.org/jira/browse/BEAM-2802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16143701#comment-16143701
]
Etienne Chauchot commented on BEAM-2802:
----------------------------------------
- text files could have no new line character at all (one line text file) but
only the custom delimiters
- Adding a prior parse and replace (custom delimiter to \n for example)
treatment will be counter productive on big files.
For your first question: the delimiter could be n bytes containing no {{\n}}
or, we have a use case in which the record delimiter is {{\002\n}} and the file
contains also {{\n}} that must be inside the output record.
For your second question: if the text file contains both {{\r\n}} and the
delimiter, the output records should be single Strings that internally contain
{{\r\n}}.
> TextIO should allow specifying a custom delimiter
> -------------------------------------------------
>
> Key: BEAM-2802
> URL: https://issues.apache.org/jira/browse/BEAM-2802
> Project: Beam
> Issue Type: New Feature
> Components: sdk-java-extensions
> Reporter: Etienne Chauchot
> Assignee: Etienne Chauchot
> Priority: Minor
>
> Currently TextIO use {{\r}} {{\n}} or {{\r\n}} or a mix of the two to split a
> text file into PCollection elements. It might happen that a record is spread
> across more than one line. In that case we should be able to specify a custom
> record delimiter to be used in place of the default ones.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
