Ramin Gharib created FLINK-41360:
------------------------------------
Summary: 'raw.line-delimiter' leaves a U+FEFF on each value in
UTF-16, does not unescape '\n' and splits on an empty value
Key: FLINK-41360
URL: https://issues.apache.org/jira/browse/FLINK-41360
Project: Flink
Issue Type: Bug
Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile), Table
SQL / Runtime
Affects Versions: 2.4.0
Reporter: Ramin Gharib
The 'raw.line-delimiter' option from FLINK-39401 has three problems with a
STRING column. Each one was reproduced through {{RawFormatFactory}} on master
at c59810c04f8.
h3. 1. UTF-16 leaves a U+FEFF on every value
The serializer encodes the value and the delimiter separately with
'raw.charset'. Java's UTF-16 encoder starts every result with a byte order
mark, so the delimiter gets one too. On read, only the leading byte order mark
is consumed. The second one decodes as U+FEFF and stays at the end of the value.
{noformat}
'raw.charset' = 'UTF-16', 'raw.line-delimiter' = '|'
write "a" -> FE FF 00 61 FE FF 00 7C
read back -> one row: "a" followed by U+FEFF
expected -> one row: "a"
{noformat}
The docs say that the same delimiter on both sides is round-trip compatible,
which does not hold here. Charsets whose encoder writes no byte order mark,
such as UTF-8 and UTF-16BE, are not affected.
h3. 2. Escape sequences in the value are not processed
The option description says the value supports Java escape sequences, and the
docs list a backslash-n newline as a common value. A Flink SQL string literal
does not process backslash escapes, so {{'raw.line-delimiter' = '\n'}} reaches
the format as the two characters backslash and n. The raw format uses the value
as it is, so a newline-delimited message is not split, and no error is raised.
{noformat}
'raw.line-delimiter' = '\n'
message -> 61 0A 62 0A
read -> one row: "a<LF>b<LF>"
expected -> two rows: "a" and "b"
{noformat}
The CSV format unescapes its delimiter options with
{{StringEscapeUtils.unescapeJava}}. The tests added in FLINK-39401 put a real
newline character into the options map, so they never see the value that SQL
produces.
h3. 3. An empty delimiter splits on read but adds nothing on write
{noformat}
'raw.line-delimiter' = ''
write "abc" -> 61 62 63, nothing appended
read back -> three rows: "a", "b" and "c"
expected -> a ValidationException, or one row: "abc"
{noformat}
The decoded split compiles an empty pattern, which splits between every
character.
h3. Proposal
* Encode the delimiter without a byte order mark, or encode the value and the
delimiter in one call, so that UTF-16 writes a single byte order mark.
* Unescape the value with {{StringEscapeUtils.unescapeJava}}, as the CSV format
does. This changes the meaning of a delimiter that contains a backslash, so it
needs a release note. The alternative is to correct the option description and
the docs.
* Reject an empty 'raw.line-delimiter' with a ValidationException in
{{RawFormatFactory}}.
* Add tests that create the format through the factory with UTF-16, and with
the option value that SQL produces for '\n'.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
