Gabriel39 commented on code in PR #68027:
URL: https://github.com/apache/doris/pull/68027#discussion_r4024734734
##########
fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveTextProperties.java:
##########
@@ -160,19 +161,35 @@ private static void extractTextSerDeProps(Map<String,
String> sdParams,
}
private static void extractCsvSerDeProps(Map<String, String> params,
- Map<String, String> result) {
- result.put(ScanNodePropertyKeys.TEXT_COLUMN_SEPARATOR,
- getParamOrDefault(params, SEPARATOR_CHAR, ","));
- result.put(ScanNodePropertyKeys.TEXT_LINE_DELIMITER,
getLineDelimiter(params));
- String quoteChar = getParamOrDefault(params, QUOTE_CHAR, "\"");
+ Map<String, String> tableParams, Map<String, String> result) {
+ // Trino stores CSV settings in table parameters. Honor Hive's
table-over-SerDe precedence
+ // so valid CSV files are not silently split with the default
delimiter and quote characters.
+ String separator = getCsvCharacter(params, tableParams,
SEPARATOR_CHAR, ',');
+ String quoteChar = getCsvCharacter(params, tableParams, QUOTE_CHAR,
'"');
Review Comment:
Addressed in a10147747e. Both scanners now select a shared Hive OpenCSV
parser with the full field-state transitions, including leading Unicode
whitespace and embedded quotes. The 606-record corpus is generated by the
actual Hive 3.1.3 SerDe and exercised through both BE readers.
##########
fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveTextProperties.java:
##########
@@ -160,19 +161,35 @@ private static void extractTextSerDeProps(Map<String,
String> sdParams,
}
private static void extractCsvSerDeProps(Map<String, String> params,
- Map<String, String> result) {
- result.put(ScanNodePropertyKeys.TEXT_COLUMN_SEPARATOR,
- getParamOrDefault(params, SEPARATOR_CHAR, ","));
- result.put(ScanNodePropertyKeys.TEXT_LINE_DELIMITER,
getLineDelimiter(params));
- String quoteChar = getParamOrDefault(params, QUOTE_CHAR, "\"");
+ Map<String, String> tableParams, Map<String, String> result) {
+ // Trino stores CSV settings in table parameters. Honor Hive's
table-over-SerDe precedence
+ // so valid CSV files are not silently split with the default
delimiter and quote characters.
+ String separator = getCsvCharacter(params, tableParams,
SEPARATOR_CHAR, ',');
+ String quoteChar = getCsvCharacter(params, tableParams, QUOTE_CHAR,
'"');
+ String escapeChar = getCsvCharacter(params, tableParams, ESCAPE_CHAR,
'"');
Review Comment:
Addressed in a10147747e. Doubled quotes are decoded independently of whether
the escape character is NUL. Fields are decoded during OpenCSV parsing and
materialized without a second generic CSV unescape pass. Disabled-escape cases
are covered by the Hive-generated corpus in both scanners.
##########
fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveTextProperties.java:
##########
@@ -160,19 +161,35 @@ private static void extractTextSerDeProps(Map<String,
String> sdParams,
}
private static void extractCsvSerDeProps(Map<String, String> params,
- Map<String, String> result) {
- result.put(ScanNodePropertyKeys.TEXT_COLUMN_SEPARATOR,
- getParamOrDefault(params, SEPARATOR_CHAR, ","));
- result.put(ScanNodePropertyKeys.TEXT_LINE_DELIMITER,
getLineDelimiter(params));
- String quoteChar = getParamOrDefault(params, QUOTE_CHAR, "\"");
+ Map<String, String> tableParams, Map<String, String> result) {
+ // Trino stores CSV settings in table parameters. Honor Hive's
table-over-SerDe precedence
+ // so valid CSV files are not silently split with the default
delimiter and quote characters.
+ String separator = getCsvCharacter(params, tableParams,
SEPARATOR_CHAR, ',');
+ String quoteChar = getCsvCharacter(params, tableParams, QUOTE_CHAR,
'"');
+ String escapeChar = getCsvCharacter(params, tableParams, ESCAPE_CHAR,
'"');
+ // Hive treats the writer-default double quote as a sentinel:
newReader selects the constructor
+ // whose effective escape is backslash. Resolve it before validating
the parser's character tuple.
+ if ("\"".equals(escapeChar)) {
+ escapeChar = "\\";
+ }
+ if ("\0".equals(separator)) {
+ throw new DorisConnectorException("Invalid OpenCSVSerde property
'separatorChar': must not be NUL");
+ }
+ // OpenCSV requires distinct active characters; NUL disables
quote/escape and may be shared by both.
Review Comment:
Addressed in a10147747e. The Hive parser now follows OpenCSV's
inField/inQuotes escape rules even when quoting is disabled, including removal
of leading escapes. Binary and disabled-character records are compared with the
actual Hive SerDe and tested through both BE readers.
##########
fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveTextProperties.java:
##########
@@ -160,19 +161,35 @@ private static void extractTextSerDeProps(Map<String,
String> sdParams,
}
private static void extractCsvSerDeProps(Map<String, String> params,
- Map<String, String> result) {
- result.put(ScanNodePropertyKeys.TEXT_COLUMN_SEPARATOR,
- getParamOrDefault(params, SEPARATOR_CHAR, ","));
- result.put(ScanNodePropertyKeys.TEXT_LINE_DELIMITER,
getLineDelimiter(params));
- String quoteChar = getParamOrDefault(params, QUOTE_CHAR, "\"");
+ Map<String, String> tableParams, Map<String, String> result) {
+ // Trino stores CSV settings in table parameters. Honor Hive's
table-over-SerDe precedence
+ // so valid CSV files are not silently split with the default
delimiter and quote characters.
+ String separator = getCsvCharacter(params, tableParams,
SEPARATOR_CHAR, ',');
+ String quoteChar = getCsvCharacter(params, tableParams, QUOTE_CHAR,
'"');
Review Comment:
Addressed in a10147747e. Hive CSV uses Hadoop-style physical LF/CRLF/CR
records instead of quote-dependent line framing. An unmatched quote drops the
pending field while preserving completed fields from that record. Both readers
are tested for physical boundaries, counts, projections, split offsets, and
empty records; the external regression explicitly selects both scanner modes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
