github-actions[bot] commented on code in PR #68027:
URL: https://github.com/apache/doris/pull/68027#discussion_r4021808546
##########
fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveTextProperties.java:
##########
@@ -160,20 +161,21 @@ private static void extractTextSerDeProps(Map<String,
String> sdParams,
}
private static void extractCsvSerDeProps(Map<String, String> params,
- Map<String, String> result) {
+ Map<String, String> tableParams, Map<String, String> result) {
+ // Trino stores CSV settings in table parameters. Honor Hive's
table-over-SerDe precedence
+ // so valid CSV files are not silently split with the default
delimiter and quote characters.
result.put(ScanNodePropertyKeys.TEXT_COLUMN_SEPARATOR,
- getParamOrDefault(params, SEPARATOR_CHAR, ","));
- result.put(ScanNodePropertyKeys.TEXT_LINE_DELIMITER,
getLineDelimiter(params));
- String quoteChar = getParamOrDefault(params, QUOTE_CHAR, "\"");
+ getCsvCharacter(params, tableParams, SEPARATOR_CHAR, ','));
+ // OpenCSVSerde does not use table-level line.delim to frame records.
Preserve the existing
+ // SerDe-only override; applying a table property here can merge
otherwise valid newline records.
+ String lineDelimiter = params == null ? null : params.get(LINE_DELIM);
+ result.put(ScanNodePropertyKeys.TEXT_LINE_DELIMITER,
+ lineDelimiter == null ? DEFAULT_LINE_DELIM : lineDelimiter);
+ String quoteChar = getCsvCharacter(params, tableParams, QUOTE_CHAR,
'"');
result.put(ScanNodePropertyKeys.TEXT_ENCLOSE, quoteChar);
- // #65501: BE strips the wrapping quotes only when the enclose char is
exactly the double-quote '"'.
- // The connector owns this CSV serde semantics, so decide here and
pass an explicit flag; the generic
- // PluginDrivenScanNode then just applies it instead of trimming for
any enclose char. Compare the
- // first byte, matching how the node sets enclose
(enclose.getBytes()[0]) and BE's getEnclose() == '"'.
- boolean trimDoubleQuotes = !quoteChar.isEmpty() &&
quoteChar.getBytes()[0] == (byte) '"';
- result.put(ScanNodePropertyKeys.TEXT_TRIM_DOUBLE_QUOTES,
String.valueOf(trimDoubleQuotes));
- String escapeChar = getParamOrDefault(params, ESCAPE_CHAR, "\\");
- result.put(ScanNodePropertyKeys.TEXT_ESCAPE, escapeChar);
+ // BE's extra double-quote trimming is valid only for the effective
double-quote enclosure.
+ result.put(ScanNodePropertyKeys.TEXT_TRIM_DOUBLE_QUOTES,
String.valueOf("\"".equals(quoteChar)));
+ result.put(ScanNodePropertyKeys.TEXT_ESCAPE, getCsvCharacter(params,
tableParams, ESCAPE_CHAR, '\\'));
result.put(ScanNodePropertyKeys.TEXT_NULL_FORMAT, "");
Review Comment:
[P1] Preserve Hive's explicit default-escape sentinel
Hive does not pass an explicit raw `escapeChar='"'` through to OpenCSV.
`OpenCSVSerde.newReader` recognizes that writer-default value and selects the
reader constructor whose parser escape is backslash. Here it reaches
`TEXT_ESCAPE` as a double quote, so both BE readers use quote as the escape
character; a valid file with a backslash-escaped embedded quote can therefore
return a different value from Hive. Please map this raw sentinel to the
reader-effective backslash before emitting the scan property, and cover the
explicit table/SerDe value through the Thrift and Hive-read paths. This is
separate from the earlier first-character/byte-representation thread.
##########
fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveTextProperties.java:
##########
@@ -160,20 +161,21 @@ private static void extractTextSerDeProps(Map<String,
String> sdParams,
}
private static void extractCsvSerDeProps(Map<String, String> params,
- Map<String, String> result) {
+ Map<String, String> tableParams, Map<String, String> result) {
+ // Trino stores CSV settings in table parameters. Honor Hive's
table-over-SerDe precedence
+ // so valid CSV files are not silently split with the default
delimiter and quote characters.
result.put(ScanNodePropertyKeys.TEXT_COLUMN_SEPARATOR,
- getParamOrDefault(params, SEPARATOR_CHAR, ","));
- result.put(ScanNodePropertyKeys.TEXT_LINE_DELIMITER,
getLineDelimiter(params));
- String quoteChar = getParamOrDefault(params, QUOTE_CHAR, "\"");
+ getCsvCharacter(params, tableParams, SEPARATOR_CHAR, ','));
+ // OpenCSVSerde does not use table-level line.delim to frame records.
Preserve the existing
Review Comment:
[P2] Validate the resolved characters as one OpenCSV configuration
Hive's OpenCSV parser rejects a NUL separator and any equal non-NUL pair
among separator, quote, and escape, but these values are validated
independently here. For example, table-level `separatorChar='|'` plus
`quoteChar='|'` now reaches the BE and is parsed ambiguously even though Hive
rejects the same table while constructing its reader. Please validate the three
parser-effective characters together after applying the default-escape
sentinel, and add conflict/NUL cases while keeping a raw double-quote
quote/escape pair valid. This is distinct from the prior empty/non-ASCII
validation thread.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
