[
https://issues.apache.org/jira/browse/FLINK-3921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tatu Saloranta updated FLINK-3921:
----------------------------------
Priority: Trivial (was: Minor)
Description:
Class `flink.types.parser.StringParser` has javadocs indicating that contents
are expected to be Ascii, similar to `StringValueParser`. That makes sense, but
when constructing actual instance, no encoding is specified; on line 66 f.ex:
this.result = new String(bytes, startPos+1, i - startPos - 2);
which leads to using whatever default platform encoding is. If contents really
are always Ascii (would not count on that as parser is used from CSV reader),
not a big deal, but it can lead to the usual Latin-1-VS-UTF-8 issues.
So I think that encoding should be explicitly specified, whatever is to be
used: javadocs claim ascii, so could be "us-ascii", but could well be UTF-8 or
even ISO-8859-1.
was:
Looking at default `TypeSerializer` instances I noticed what looks like a minor
flaw, unless I am missing something.
Whereas all other array serializers indicate that type is not immutable (since
in Java, arrays are not immutable), `StringArraySerializer` has:
```
@Override
public boolean isImmutableType() {
return true;
}
```
and I think it should instead return `false`. I could create a PR, but seems
like a small enough thing that issue report makes more sense.
I tried looking for deps to see if there's a test for this, but couldn't find
one; otherwise could submit a test fix.
> StringParser not specifying encoding to use
> -------------------------------------------
>
> Key: FLINK-3921
> URL: https://issues.apache.org/jira/browse/FLINK-3921
> Project: Flink
> Issue Type: Improvement
> Components: Core
> Affects Versions: 1.0.3
> Reporter: Tatu Saloranta
> Priority: Trivial
>
> Class `flink.types.parser.StringParser` has javadocs indicating that contents
> are expected to be Ascii, similar to `StringValueParser`. That makes sense,
> but when constructing actual instance, no encoding is specified; on line 66
> f.ex:
> this.result = new String(bytes, startPos+1, i - startPos - 2);
> which leads to using whatever default platform encoding is. If contents
> really are always Ascii (would not count on that as parser is used from CSV
> reader), not a big deal, but it can lead to the usual Latin-1-VS-UTF-8 issues.
> So I think that encoding should be explicitly specified, whatever is to be
> used: javadocs claim ascii, so could be "us-ascii", but could well be UTF-8
> or even ISO-8859-1.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
