ASF GitHub Bot commented on FLINK-3921:

Github user fhueske commented on the issue:

    Hi @greghogan, thanks for working on this. The changes look good. 
    However, I think we need to propagate the configured charset also to the 
`DelimitedInputFormat`. `DelimitedInputFormat` splits the file content into 
lines based on a row delimiter which is always encoded as `UTF-8`. Each line is 
subsequently parsed by the logic of the `CsvInputFormat`.

> StringParser not specifying encoding to use
> -------------------------------------------
>                 Key: FLINK-3921
>                 URL: https://issues.apache.org/jira/browse/FLINK-3921
>             Project: Flink
>          Issue Type: Improvement
>          Components: Core
>    Affects Versions: 1.0.3
>            Reporter: Tatu Saloranta
>            Assignee: Rekha Joshi
>            Priority: Trivial
> Class `flink.types.parser.StringParser` has javadocs indicating that contents 
> are expected to be Ascii, similar to `StringValueParser`. That makes sense, 
> but when constructing actual instance, no encoding is specified; on line 66 
> f.ex:
>    this.result = new String(bytes, startPos+1, i - startPos - 2);
> which leads to using whatever default platform encoding is. If contents 
> really are always Ascii (would not count on that as parser is used from CSV 
> reader), not a big deal, but it can lead to the usual Latin-1-VS-UTF-8 issues.
> So I think that encoding should be explicitly specified, whatever is to be 
> used: javadocs claim ascii, so could be "us-ascii", but could well be UTF-8 
> or even ISO-8859-1.

This message was sent by Atlassian JIRA

Reply via email to