ASF GitHub Bot commented on FLINK-3921:
Github user greghogan commented on the issue:
Apologies for the long delay. I'd like to attempt to summarize this ticket
and pull request to validate my understanding.
Previously StringParser was using the system encoding and
`GenericCsvInputFormat` was using UTF-8 for the delimiter and an overloadable
UTF-8 for the comment prefix.
StringParser's quoteCharacter remains a `byte` with no encoding.
Now GenericCsvInputFormat can be configured with a charset which is used
for the delimiter, comment prefix, and field parsers (only used in
Should `setCommentPrefix(String commentPrefix, Charset charset)` and
`setCommentPrefix(String commentPrefix, String charsetName)` be removed from
`GenericCsvInputFormat`? Would different encodings be used on the same file?
Allow the user to set the character encoding in `CsvReader` which would be
applied in `CsvReader.configureInputFormat`?
Are the new tests checking the encoding? The test strings are using using
characters common to UTF-8 and ASCII. We could instead use one of the UTF-16
> StringParser not specifying encoding to use
> Key: FLINK-3921
> URL: https://issues.apache.org/jira/browse/FLINK-3921
> Project: Flink
> Issue Type: Improvement
> Components: Core
> Affects Versions: 1.0.3
> Reporter: Tatu Saloranta
> Assignee: Rekha Joshi
> Priority: Trivial
> Class `flink.types.parser.StringParser` has javadocs indicating that contents
> are expected to be Ascii, similar to `StringValueParser`. That makes sense,
> but when constructing actual instance, no encoding is specified; on line 66
> this.result = new String(bytes, startPos+1, i - startPos - 2);
> which leads to using whatever default platform encoding is. If contents
> really are always Ascii (would not count on that as parser is used from CSV
> reader), not a big deal, but it can lead to the usual Latin-1-VS-UTF-8 issues.
> So I think that encoding should be explicitly specified, whatever is to be
> used: javadocs claim ascii, so could be "us-ascii", but could well be UTF-8
> or even ISO-8859-1.
This message was sent by Atlassian JIRA