XuQianJin-Stars commented on a change in pull request #6710: [FLINK-10134]
UTF-16 support for TextInputFormat bug fixed
URL: https://github.com/apache/flink/pull/6710#discussion_r223541560
##########
File path:
flink-core/src/main/java/org/apache/flink/api/common/io/FileInputFormat.java
##########
@@ -601,41 +602,44 @@ public LocatableInputSplitAssigner
getInputSplitAssigner(FileInputSplit[] splits
if (unsplittable) {
int splitNum = 0;
for (final FileStatus file : files) {
+ String bomCharsetName = getBomCharset(file);
Review comment:
Well, I first take the logic of checking the BOM in FileInputFormat to
DelimitedInputFormat. I want to use a Map to cache the BOM encoding of the
file, using the filename as the key and the BOM encoding as the value. If the
value exists in the Map, the corresponding value is read, and if the Map does
not exist, the BOM encoding of the file is read.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
