fhueske commented on a change in pull request #6710: [FLINK-10134] UTF-16
support for TextInputFormat bug fixed
URL: https://github.com/apache/flink/pull/6710#discussion_r221900610
##########
File path:
flink-core/src/main/java/org/apache/flink/api/common/io/FileInputFormat.java
##########
@@ -862,35 +865,87 @@ public void close() throws IOException {
stream = null;
}
}
-
+
/**
* Override this method to supports multiple paths.
* When this method will be removed, all FileInputFormats have to
support multiple paths.
*
* @return True if the FileInputFormat supports multiple paths, false
otherwise.
- *
* @deprecated Will be removed for Flink 2.0.
*/
@Deprecated
public boolean supportsMultiPaths() {
return false;
}
+ @Override
public String toString() {
return getFilePaths() == null || getFilePaths().length == 0 ?
"File Input (unknown file)" :
- "File Input (" + Arrays.toString(this.getFilePaths())
+ ')';
+ "File Input (" + Arrays.toString(this.getFilePaths()) +
')';
+ }
+
+ /**
+ * Get file bom encoding
+ *
+ * @param fs
+ * @return
+ */
+ public String getBomCharset(FileStatus fs) {
+ FSDataInputStream inStream = null;
+ String charset, testFileSystem = "TestFileSystem";
+ byte[] bom = new byte[4];
+ byte[] bytes = new byte[]{(byte) 0x00, (byte) 0xFE, (byte)
0xFF, (byte) 0xEF, (byte) 0xBB, (byte) 0xBF};
+ try {
Review comment:
Hmm, `FileInputFormat` is charset agnostics, which makes sense. Maybe we
should move this logic to `DelimitedInputFormat` instead?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
