Doris-Breakwater commented on issue #66419: URL: https://github.com/apache/doris/issues/66419#issuecomment-5175212022
Breakwater-GitHub-Analysis-Slot: slot_f9d715303e76 ### Initial triage **Assessment: confirmed FE-side MySQL LOAD bridge bug; high confidence.** This is not a CSV reader or BE storage issue. The report currently has no labels and no earlier comments. I checked both Doris 4.0.2 tags (`4.0.2-rc01` and `4.0.2-rc02`), the refreshed `branch-4.0`, and current `master` (`6d7992f533697f597a64e14ec2fc355464ba21be`, 2026-08-04). The relevant code shape is still present in all of them. ### Verified primary root cause The initial SQL grammar does accept a backtick-quoted identifier in the MySQL `LOAD DATA` column list. The failure occurs because this statement is translated into an internal Stream Load request and the identifier is serialized a second time: 1. `mysqlDataDesc` parses the list through `identifierList`, and `quotedIdentifier` accepts `BACKQUOTED_IDENTIFIER`. 2. The Nereids [`PostProcessor`](https://github.com/apache/doris/blob/30d2df045941c55c57ce7cc67314d06216b1a9de/fe/fe-core/src/main/java/org/apache/doris/nereids/parser/PostProcessor.java#L44-L50) deliberately removes the outer backticks and normalizes the token to an identifier. Therefore `` `execute` `` becomes the plain string `execute` in `MysqlDataDescription`. 3. [`MysqlLoadManager.getColumns(MysqlDataDescription)`](https://github.com/apache/doris/blob/30d2df045941c55c57ce7cc67314d06216b1a9de/fe/fe-core/src/main/java/org/apache/doris/load/loadv2/MysqlLoadManager.java#L368-L382) joins those normalized names without re-quoting them and sends the result as the internal Stream Load `columns` header. The header is consequently `id,execute,name`, not ``id,`execute`,name``. 4. [`NereidsStreamLoadTask`](https://github.com/apache/doris/blob/30d2df045941c55c57ce7cc67314d06216b1a9de/fe/fe-core/src/main/java/org/apache/doris/nereids/load/NereidsStreamLoadTask.java#L523-L539) reparses that header through `NereidsLoadUtils.parseExpressionSeq()`, which constructs `SELECT id,execute,name`. At this second parse, `EXECUTE` is a reserved token, producing the reported `SyntaxParseException`. This also explains the apparent inconsistency with direct Stream Load: a direct header that still contains backticks reaches the second parser intact and succeeds. The valid scope of this bug is **backtick-quoted identifiers**. Double quotes are string-literal syntax in this grammar, square brackets are not Doris identifier quoting, and an unquoted reserved word is expected to fail. ### Separate verified issue: `PROPERTIES ("columns"=...)` is silently ignored The reported `PROPERTIES` behavior is not a functioning alternative column-mapping path in 4.0.2: - The inner `mysqlDataDesc` property map accepts `"columns"`, but `MysqlLoadManager` only forwards selected properties such as `max_filter_ratio`, `strict_mode`, `timeout`, and `timezone`. - The internal `columns` header is generated exclusively from `desc.getColumns()`, i.e. the SQL-level column list. If that list is absent, no `columns` header is sent and Stream Load falls back to the table's visible base-schema order. - If both forms are supplied, the SQL list becomes the real header while `PROPERTIES ("columns"=...)` remains ignored. Thus `(id,c2,name)` creates a temporary input field `c2`; without an actual mapping to `execute`, that value is discarded and `execute` receives its default/NULL. This matches the reported result. The hidden-column test is consistent with the same mechanism: without a real `columns` header, default mapping uses visible base columns and does not add `__DORIS_DELETE_SIGN__`; the fourth CSV field can therefore be filtered as redundant/mismatched input. Because the example explicitly uses `max_filter_ratio=1.0`, completion with all rows filtered is allowed. The exact row-filter reason still requires the Stream Load response/ErrorURL. There is also a syntax correction: Doris 4.0.2 does have a mapping clause, but its grammar is `SET (<mapping>, ...)`, with mandatory parentheses after `SET`. The example `SET id=c1, ...` is therefore rejected as written. It is not a complete workaround here, however, because the current MySQL-load serializer also removes backticks from mapping SQL via ``replaceAll("`", "")``, so a reserved mapping target can fail at the same second parse. ### Recommended maintainer actions 1. Fix serialization at the MySQL LOAD-to-Stream Load boundary, rather than changing the global `PostProcessor` normalization. When generating the internal `columns` header, safely backtick and escape every field identifier (including embedded backticks), and preserve safe quoting for mapping targets. The other `getColumns(...)` overloads should be audited for the same lossy serialization. 2. Add an end-to-end MySQL `LOAD DATA LOCAL INFILE` regression covering: - a backtick-quoted reserved column; - that column together with `__DORIS_DELETE_SIGN__`; - a quoted mapping target using the supported `SET (...)` grammar; - an identifier containing an escaped backtick. 3. Decide the contract for `PROPERTIES ("columns"=...)`: either implement forwarding with explicit conflict validation, or reject it as unsupported. Supplying both a SQL list and a property-based list should not silently choose one. 4. Short-term workaround: use direct Stream Load with a backtick-preserving `columns` header, including `__DORIS_DELETE_SIGN__` when required. ### Additional information requested These items are not needed to establish the primary root cause, but are needed to close the ancillary hidden-column/filtering claim precisely: - The exact Doris build commit (the relevant code is identical in both 4.0.2 RC tags, but the package-to-commit mapping is not stated). - Exact contents of `/tmp/test_sign.csv`. - Full `SHOW CREATE TABLE test_execute` output, including the Unique Key MOW properties. - The complete internal Stream Load result for the zero-row case, especially `NumberTotalRows`, `NumberLoadedRows`, `NumberFilteredRows`, `Message`, and `ErrorURL`, plus the referenced error rows if available. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
