sollhui commented on code in PR #67412:
URL: https://github.com/apache/doris/pull/67412#discussion_r4332626257
##########
fe/fe-core/src/main/java/org/apache/doris/tablefunction/ExternalFileTableValuedFunction.java:
##########
@@ -553,14 +553,24 @@ private PFetchTableSchemaRequest
getFetchTableStructureRequest() throws TExcepti
.setFileScanRange(ByteString.copyFrom(new
TSerializer().serialize(fileScanRange))).build();
}
- private boolean isFileContentEmpty(TBrokerFileStatus fileStatus) {
+ boolean isFileContentEmpty(TBrokerFileStatus fileStatus) {
if (fileStatus.isIsDir() || fileStatus.size == 0) {
return true;
}
if (Util.isCsvFormat(fileFormatProperties.getFileFormatType())
|| fileFormatProperties.getFileFormatType() ==
TFileFormatType.FORMAT_JSON) {
int magicNumberBytes = 0;
- switch (fileFormatProperties.getCompressionType()) {
+ TFileCompressType compressType =
fileFormatProperties.getCompressionType();
+ if (compressType == TFileCompressType.UNKNOWN) {
+ TFileCompressType inferredCompressType =
Util.getOrInferCompressType(
+ compressType, fileStatus.getPath());
+ // Preserve the established reader error for auto-inferred
empty LZO and other formats.
+ // This path-based empty-file shortcut is currently supported
only for gzip.
+ if (inferredCompressType == TFileCompressType.GZ) {
Review Comment:
[P2] Common empty gzip files are still selected for schema inference
The inferred `GZ` type now reaches the existing `size <= 20` shortcut, but
an empty gzip stream is not necessarily 20 bytes. For example,
`gzip.compress(b'', compresslevel=0)` produces a valid 23-byte stream, and
locally `gzip -c a_empty.csv` on an empty file produces a 32-byte stream
because it includes the original filename. Both decompress to zero bytes.
If one of these empty files is the first match in a CSV/JSON glob,
`isFileContentEmpty()` still returns `false`, so
`getFetchTableStructureRequest()` selects it and stops looking for a schema
source. For `csv_with_names`, BE then returns `The first line is empty, can not
parse column names`; FE propagates the error without trying the later valid
file. This leaves the reported failure unresolved for common gzip producers.
The two added empty regression fixtures are both exactly 20 bytes, so they
do not cover this case. Could we keep the size check as a fast path and
continue to the next candidate when schema inference confirms an empty
decompressed file? Please also cover an empty gzip with a filename header and a
level-0 empty stream. Increasing the size threshold alone could incorrectly
skip small nonempty files.
Validation: reproduced the compressed sizes and verified zero-byte
decompressed payloads locally; traced the FE selection and BE CSV error paths.
No Doris cluster regression run was performed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
