luchunliang opened a new pull request, #12159:
URL: https://github.com/apache/inlong/pull/12159
Fixes #12158
### Motivation
Description
When a source data field is named after a SQL reserved keyword (e.g. from,
select, where), the Transform SDK fails to look up the field correctly.
JSQLParser requires such identifiers to be quoted with backticks (e.g. `from`
as from_source), but ColumnParser previously stored the raw output of
expr.toString(), which still contained the surrounding backticks. As a result,
the SDK ended up looking up `from` (with backticks) instead of the actual field
name from, and the lookup returned null.
Background
The InLong Transform SDK relies on JSQLParser to parse user-provided
transformation SQL. If a field name collides with a SQL reserved keyword, it
must be quoted with backticks:
sql
复制
SELECT `from` AS from_source, itemid AS itemid FROM source
Without the backticks, JSQLParser raises:
net.sf.jsqlparser.JSQLParserException: Encountered unexpected token: "," ","
at line 1, column 116.
Even when backticks are used, ColumnParser previously stored the raw string
`from` (including the backticks) as the field name. That caused
sourceData.getField(rowIndex, "from") to return null, because the real field
name in the source data is from.
Steps to Reproduce
Configure a source that contains a field named from (a SQL reserved keyword).
Use the following transform SQL:
sql
复制
SELECT `from` AS from_source, itemid AS itemid FROM source
Run the transform.
Expected Behavior
The surrounding backticks should be stripped during parsing, and the SDK
should use the unquoted field name (from) to look up the value from the source
data.
Actual Behavior (before the fix)
ColumnParser stored the field name as `from` (with backticks), so the field
lookup always returned null.
Proposed Fix
Strip the surrounding backticks in ColumnParser before storing the field
name.
File:
inlong-sdk/transform-sdk/src/main/java/org/apache/inlong/sdk/transform/process/parser/ColumnParser.java
java
复制
public ColumnParser(Column expr) {
this.fieldName = this.formatFieldName(expr.toString());
}
public ColumnParser(Function expr) {
this.fieldName = this.formatFieldName(expr.toString());
}
private String formatFieldName(String rawFieldName) {
String tmpFieldName = rawFieldName;
if (tmpFieldName.startsWith("`")) {
tmpFieldName = tmpFieldName.substring(1);
}
if (tmpFieldName.endsWith("`")) {
tmpFieldName = tmpFieldName.substring(0, tmpFieldName.length() - 1);
}
return tmpFieldName;
}
The same backtick-stripping logic is also applied to select-item aliases in
TransformProcessor:
File:
inlong-sdk/transform-sdk/src/main/java/org/apache/inlong/sdk/transform/process/TransformProcessor.java
java
复制
fieldName = exprItem.getAlias().getName();
// Strip surrounding backticks if present
if (fieldName != null && fieldName.length() >= 2
&& fieldName.startsWith("`") && fieldName.endsWith("`")) {
fieldName = fieldName.substring(1, fieldName.length() - 1);
}
Affected Components
inlong-sdk/transform-sdk
ColumnParser.java — added formatFieldName() to strip backticks
TransformProcessor.java — added backtick stripping for select-item aliases
Test Coverage
Updated a test case in TestCsv2KvProcessor.java to cover a backtick-quoted
column name:
java
复制
// In TestCsv2KvProcessor.java
+ ",`from` as random_seq"
Notes
formatFieldName() currently only handles backticks (`). Other quoting styles
such as double quotes ("from") or square brackets ([from]) are not stripped.
This can be addressed in a follow-up improvement if needed.
### Modifications
<!--Describe the modifications you've done.-->
### Verifying this change
*(Please pick either of the following options)*
- [ ] This change is a trivial rework/code cleanup without any test coverage.
- [ ] This change is already covered by existing tests, such as:
*(please describe tests)*
- [ ] This change added tests and can be verified as follows:
*(example:)*
- *Added integration tests for end-to-end deployment with large payloads
(10MB)*
- *Extended integration test for recovery after broker failure*
### Documentation
- Does this pull request introduce a new feature? (yes / no)
- If yes, how is the feature documented? (not applicable / docs / JavaDocs
/ not documented)
- If a feature is not applicable for documentation, explain why?
- If a feature is not documented yet in this PR, please create a follow-up
issue for adding the documentation
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
