This is an automated email from the ASF dual-hosted git repository. github-merge-queue[bot] pushed a commit to branch gh-readonly-queue/dev/pr-12251-310913d494661cef133e5e9c593cafdac9bb41bd in repository https://gitbox.apache.org/repos/asf/seatunnel.git
commit e96669493d5f6993e444952aba4aa86019831ee5 Author: Goutam Adwant <[email protected]> AuthorDate: Sat Nov 7 06:41:43 2026 +0000 [Fix][Transform-V2] Honor JsonPath conversion error policies (#12251) Signed-off-by: Goutam Adwant <[email protected]> --- .../introduction/concepts/incompatible-changes.md | 23 + docs/en/transforms/jsonpath.md | 57 +- .../introduction/concepts/incompatible-changes.md | 16 + docs/zh/transforms/jsonpath.md | 46 +- .../json_path_with_error_handle_way.conf | 18 +- .../exception/JsonPathTransformErrorCode.java | 2 + .../transform/jsonpath/JsonPathTransform.java | 152 ++++- .../transform/JsonPathErrorHandlingTest.java | 742 +++++++++++++++++++++ 8 files changed, 1035 insertions(+), 21 deletions(-) diff --git a/docs/en/introduction/concepts/incompatible-changes.md b/docs/en/introduction/concepts/incompatible-changes.md index 7fea5d0f77f..afadaa2dc02 100644 --- a/docs/en/introduction/concepts/incompatible-changes.md +++ b/docs/en/introduction/concepts/incompatible-changes.md @@ -323,6 +323,29 @@ The Zeta SQL ARRAY function now declares TINYINT elements as `ARRAY<TINYINT>`, m (`existingConfigMap`) and restrict the member port (5801) with a `NetworkPolicy`. Restart the pods after upgrading so the new configuration is applied. +### JsonPath Transform + +- Recognized destination-type conversion failures now honor the existing column + and row error policies. Column `SKIP` yields a null field; column `SKIP_ROW` or + row `SKIP` (without a column override) drops the row. Explicit column `FAIL` + still takes precedence; the default remains `FAIL`. +- Under `FAIL`, recognized conversion errors now surface as + `ErrorDataTransformException` with `JSONPATH_ERROR_CODE-07`, rather than the raw + converter exception or its `COMMON-*` code. Source records, extracted values + and raw causes are intentionally omitted from these diagnostics and skip logs. + Update alerts/runbooks keyed on the old exception text or code. Unsupported + conversions and unexpected failures retain their existing behavior; this does + not add `ROUTE_TO_TABLE` support or change checkpoint formats. +- Path-reading error policies are unchanged. `ErrorDataTransformException` now + attaches a sanitized `JsonPathException` cause, rather than the original + exception, because its message or stack trace may contain source data or a + private path literal. Previously there was no cause. Diagnostics retain only + safe field identifiers and the original exception type; they omit source JSON, + configured paths and raw exception text. The cause chain remains available, + but the original cause subtype and stack detail are intentionally lost. Update + stack-trace parsers and alerts that assumed no nested cause or relied on the + previous message text. + ### Replace Transform Schema Refresh - **Behavior change**: Replace resolves `replace_fields` against the refreshed input schema after schema changes and checkpoint restoration. Previously cached positions could replace an unrelated column or leave the configured field unchanged after column positions shifted. diff --git a/docs/en/transforms/jsonpath.md b/docs/en/transforms/jsonpath.md index 2621408c812..5d668befe2a 100644 --- a/docs/en/transforms/jsonpath.md +++ b/docs/en/transforms/jsonpath.md @@ -25,6 +25,11 @@ This option is used to specify the processing method when an error occurs in the - SKIP: When `SKIP` is selected, data format error will skip this row data. - ROUTE_TO_TABLE: not implemented by the JsonPath transform yet. The value can be configured, but its actual behavior is identical to `FAIL`: rows that fail to parse make the job fail and are not routed to an error table. +This applies to path-reading errors and recognized data-conversion failures when no +column policy is configured. `ROUTE_TO_TABLE` is not supported by JsonPath: it does +not route errors and fails when selected for a failing value. See +[Configure error data handle way](#configure-error-data-handle-way) for precedence and examples. + ### columns [array] #### option @@ -75,6 +80,11 @@ This option is used to specify the processing method when an error occurs in the - SKIP: When `SKIP` is selected, data format error will skip this column data. - SKIP_ROW: When `SKIP_ROW` is selected, data format error will skip this row data. +These policies also apply to recognized data-conversion failures, such as an invalid +date or number. Column `SKIP` sets the destination field to `null`, not the whole row. +`ROUTE_TO_TABLE` is not supported. See +[Configure error data handle way](#configure-error-data-handle-way) for policy precedence and limits. + ## Read Json Example The data read from source is a table like this json: @@ -243,7 +253,52 @@ Then the data result table `fake1` will like this You can configure `row_error_handle_way` and `column_error_handle_way` to handle abnormal data. Both are optional. -`row_error_handle_way` is used to handle all data anomalies in the row data, while `column_error_handle_way` is used to handle data anomalies in a column. It has a higher priority than `row_error_handle_way`. +The policies apply to path-reading errors and recognized value-conversion failures +(numbers, dates/times, binary data and their nested conversions). Column `SKIP` +sets the failed destination field to `null` and continues processing other columns; +column `SKIP_ROW` discards the row. Without a column policy, row `FAIL` or `SKIP` +applies. An explicit column policy takes precedence, so column `FAIL` still fails +when row `SKIP` is configured. `ROUTE_TO_TABLE` does not route records in this +transform and propagates the failure when selected for the failing value. + +Unsupported conversions, unexpected programming/configuration failures and fatal +JVM errors fail the task instead of being skipped. Recognized conversion failures +use `JSONPATH_ERROR_CODE-07` with safe source/destination field identifiers, destination SQL +type and the generic `data conversion failure` category, not a typed cause category. +Invalid `float_vector` shapes and non-numeric vector elements currently use the +shared converter's unsupported-type error and are not skippable by these policies. +Conversion diagnostics and skip logs omit source +records, extracted values, path expressions and original exceptions, which can +contain private data, including literals in configured paths. Path-reading +diagnostics and skip logs include only safe source/destination field identifiers +and the exception type; field identifiers outside `[A-Za-z_][A-Za-z0-9_]{0,63}` +are shown as `<redacted>`. They omit the source JSON, configured path and raw +exception message. The zero-based `column_index` identifies the flattened output +field position, including fields expanded from an array-valued `columns` entry, +even when its name is redacted. For `FAIL`, +`ErrorDataTransformException` retains a sanitized +`JsonPathException` cause, not the original exception: preserving its message or +stack trace could re-expose input data or a private path literal. This keeps the +cause chain present but sacrifices the original cause subtype and stack detail. +Invalid configured paths also fail the task with a sanitized exception, even +under `SKIP`. Unexpected failures still propagate. + +### Skip an invalid converted value + +For input `{"amount":"invalid","description":"retained"}` in `json_data`, this +configuration produces a null `amount` and preserves `description`: + +```hocon +transform { + JsonPath { + row_error_handle_way = FAIL + columns = [ + { src_field = "json_data", path = "$.amount", dest_field = "amount", dest_type = "int", column_error_handle_way = SKIP }, + { src_field = "json_data", path = "$.description", dest_field = "description", dest_type = "string" } + ] + } +} +``` ### Skip error data rows diff --git a/docs/zh/introduction/concepts/incompatible-changes.md b/docs/zh/introduction/concepts/incompatible-changes.md index d205cac061a..26aadd2b650 100644 --- a/docs/zh/introduction/concepts/incompatible-changes.md +++ b/docs/zh/introduction/concepts/incompatible-changes.md @@ -266,6 +266,22 @@ Zeta SQL ARRAY 函数现在将 TINYINT 元素声明为 `ARRAY<TINYINT>`,与实 并使用 `schema_save_mode = "ERROR_WHEN_SCHEMA_NOT_EXIST"` 保留该表结构。不要对手动定义的目标表使用 `RECREATE_SCHEMA`。任意指定索引前缀长度可能拒绝前缀相同但完整值不同的源主键,因此无法保持原有主键语义。 +### JsonPath 转换 + +- 已识别的目标类型转换失败现在遵循现有的列级和行级错误策略。列级 `SKIP` 产生 null + 字段;列级 `SKIP_ROW` 或未被列级策略覆盖的行级 `SKIP` 丢弃整行。显式列级 `FAIL` + 仍优先,默认策略仍为 `FAIL`。 +- 在 `FAIL` 策略下,已识别的转换错误现在以 `ErrorDataTransformException` 和 + `JSONPATH_ERROR_CODE-07` 报告,不再直接报告原始转换器异常或其 `COMMON-*` 错误码。 + 这些诊断及跳过日志会有意省略源记录、提取值和原始异常原因。请更新依赖旧异常文本或错误码 + 的告警和运维手册。不支持的转换和意外失败保留现有行为;本变更不增加 + `ROUTE_TO_TABLE` 支持,也不改变检查点格式。 +- 路径读取错误的处理策略不变。`ErrorDataTransformException` 现在附加经过净化的 + `JsonPathException` cause,而不是原始异常,因为原始消息或堆栈可能包含源数据或私有路径字面量; + 此前该异常没有 cause。诊断仅保留安全的字段标识符和原始异常类型,不包含源 JSON、配置路径或 + 原始异常文本。cause 链仍存在,但不保留原始 cause 的子类型和堆栈细节。请更新原先假设不存在 + 嵌套 cause 或依赖旧异常文本的堆栈解析器和告警。 + ### Replace 转换的模式刷新 - **行为变更**:模式变更或检查点恢复后,Replace 会根据刷新的输入模式重新解析 `replace_fields`。此前,列位置变化后,缓存的位置可能导致替换无关列或漏掉配置的字段。 diff --git a/docs/zh/transforms/jsonpath.md b/docs/zh/transforms/jsonpath.md index 0d4c317d7d8..7b667e91764 100644 --- a/docs/zh/transforms/jsonpath.md +++ b/docs/zh/transforms/jsonpath.md @@ -25,6 +25,10 @@ JsonPath 转换插件支持使用 JSONPath 选择数据。 - SKIP:选择`SKIP`时,数据格式错误会跳过该行数据。 - ROUTE_TO_TABLE:JsonPath 转换尚未实现该处理方式。该值目前可以配置,但实际行为与 `FAIL` 完全相同:解析失败的行会直接使作业失败,不会被路由到错误表。 +未配置列级策略时,此选项适用于路径读取错误和已识别的数据转换失败。 +JsonPath 不支持 `ROUTE_TO_TABLE`:不会路由错误,为失败值选择此策略时会抛出异常。 +策略优先级和示例见[配置异常数据处理策略](#配置异常数据处理策略)。 + ### columns [array] #### 属性 @@ -75,6 +79,10 @@ JsonPath 转换插件支持使用 JSONPath 选择数据。 - SKIP:选择`SKIP`时,数据格式错误会跳过此列数据。 - SKIP_ROW:选择`SKIP_ROW`时,数据格式错误会跳过此行数据。 +这些策略也适用于已识别的数据转换失败,例如无效的日期或数字。 +列级 `SKIP` 将目标字段设为 `null`,不会丢弃整行。不支持 `ROUTE_TO_TABLE`。 +策略优先级和限制见[配置异常数据处理策略](#配置异常数据处理策略)。 + ## 读取 JSON 示例 从源读取的数据是像这样的 JSON @@ -243,7 +251,43 @@ transform { 您可以配置 `row_error_handle_way` 与 `column_error_handle_way` 来处理异常数据,两者都是非必填项。 -`row_error_handle_way` 配置对行数据内所有数据异常进行处理,`column_error_handle_way` 配置对某列数据异常进行处理,优先级高于 `row_error_handle_way`。 +这些策略适用于路径读取错误和已识别的值转换失败(数字、日期/时间、二进制数据及其嵌套转换)。 +列级 `SKIP` 将失败的目标字段设为 `null` 并继续处理其他列;列级 `SKIP_ROW` 丢弃整行。 +未配置列级策略时,使用行级 `FAIL` 或 `SKIP`。显式列级策略优先,因此即使行级策略为 +`SKIP`,列级 `FAIL` 仍会失败。此转换不支持 `ROUTE_TO_TABLE`,为失败值选择该策略时 +会传播异常,而不是路由记录。 + +不支持的转换、意外的程序/配置错误和致命 JVM 错误会使任务失败,不会被跳过。 +已识别的转换失败使用 `JSONPATH_ERROR_CODE-07`,诊断包含安全的源/目标字段标识符、目标 SQL 类型 +及通用的 `data conversion failure` 类别,而不是按原因类型细分的类别。 +无效的 `float_vector` 结构及非数值向量元素目前使用共享转换器的不支持类型错误,不能由这些策略跳过。 +转换异常和跳过日志不包含源记录、提取值、路径表达式或可能携带隐私数据的原始异常; +配置路径中的字面量也可能包含隐私数据。路径读取错误诊断及跳过日志仅包含安全的源/目标字段标识符 +和异常类型;不符合 `[A-Za-z_][A-Za-z0-9_]{0,63}` 的字段标识符显示为 `<redacted>`。 +从零开始的 `column_index` 表示展开后的输出字段位置,包括由同一 `columns` 配置项中的数组展开的字段, +即使字段名被隐藏也能用于定位。 +源 JSON、配置路径和原始异常消息均不会包含在这些诊断中。`FAIL` 时, +`ErrorDataTransformException` 附加经过净化的 `JsonPathException` cause,而不是原始异常: +保留原始消息或堆栈可能重新暴露输入数据或私有路径字面量。这样保留了 cause 链, +但不保留原始 cause 的子类型和堆栈细节。无效的配置路径即使在 `SKIP` 下也会以经过净化的 +异常使任务失败。意外失败仍会传播。 + +### 跳过无法转换的值 + +当 `json_data` 中的输入为 `{"amount":"invalid","description":"retained"}` 时, +以下配置将 `amount` 设为 null,并保留 `description`: + +```hocon +transform { + JsonPath { + row_error_handle_way = FAIL + columns = [ + { src_field = "json_data", path = "$.amount", dest_field = "amount", dest_type = "int", column_error_handle_way = SKIP }, + { src_field = "json_data", path = "$.description", dest_field = "description", dest_type = "string" } + ] + } +} +``` ### 跳过异常数据行 diff --git a/seatunnel-e2e/seatunnel-transforms-v2-e2e/seatunnel-transforms-v2-e2e-part-2/src/test/resources/json_path_transform/json_path_with_error_handle_way.conf b/seatunnel-e2e/seatunnel-transforms-v2-e2e/seatunnel-transforms-v2-e2e-part-2/src/test/resources/json_path_transform/json_path_with_error_handle_way.conf index 8c39caf377f..2bb0bceb97b 100644 --- a/seatunnel-e2e/seatunnel-transforms-v2-e2e/seatunnel-transforms-v2-e2e-part-2/src/test/resources/json_path_transform/json_path_with_error_handle_way.conf +++ b/seatunnel-e2e/seatunnel-transforms-v2-e2e/seatunnel-transforms-v2-e2e-part-2/src/test/resources/json_path_transform/json_path_with_error_handle_way.conf @@ -34,11 +34,11 @@ source { rows = [ { kind = INSERT - fields = [1, "{\"f1\": \"v1\"}"] + fields = [1, "{\"f1\": \"v1\", \"amount\": \"invalid\"}"] }, { kind = INSERT - fields = [2, "{\"f1\": \"v1\", \"f2\": \"v2\"}"] + fields = [2, "{\"f1\": \"v1\", \"f2\": \"v2\", \"amount\": \"invalid\"}"] } ] } @@ -59,6 +59,13 @@ transform { path = "$.f2" dest_field = "data_f2" column_error_handle_way = SKIP + }, + { + src_field = "data" + path = "$.amount" + dest_field = "amount" + dest_type = "int" + column_error_handle_way = SKIP } ] } @@ -97,8 +104,13 @@ sink { field_name = data_f1 field_type = "string" field_value = [{equals_to = "v1"}] + }, + { + field_name = amount + field_type = "int" + field_value = [{rule_type = NULL}] } ] } } -} \ No newline at end of file +} diff --git a/seatunnel-transforms-v2/src/main/java/org/apache/seatunnel/transform/exception/JsonPathTransformErrorCode.java b/seatunnel-transforms-v2/src/main/java/org/apache/seatunnel/transform/exception/JsonPathTransformErrorCode.java index 52b75a1ed02..4fcf7796199 100644 --- a/seatunnel-transforms-v2/src/main/java/org/apache/seatunnel/transform/exception/JsonPathTransformErrorCode.java +++ b/seatunnel-transforms-v2/src/main/java/org/apache/seatunnel/transform/exception/JsonPathTransformErrorCode.java @@ -33,6 +33,8 @@ public enum JsonPathTransformErrorCode implements SeaTunnelErrorCode { "JSONPATH_ERROR_CODE-06", "JsonPathTransform dest_type must not empty"), SRC_FIELD_NOT_FOUND( "JSONPATH_ERROR_CODE-02", "JsonPathTransform src_field not found in source"), + JSON_PATH_CONVERSION_ERROR( + "JSONPATH_ERROR_CODE-07", "JsonPathTransform field conversion failed"), ; private final String code; private final String description; diff --git a/seatunnel-transforms-v2/src/main/java/org/apache/seatunnel/transform/jsonpath/JsonPathTransform.java b/seatunnel-transforms-v2/src/main/java/org/apache/seatunnel/transform/jsonpath/JsonPathTransform.java index 9ef4f9d2664..643c6063a42 100644 --- a/seatunnel-transforms-v2/src/main/java/org/apache/seatunnel/transform/jsonpath/JsonPathTransform.java +++ b/seatunnel-transforms-v2/src/main/java/org/apache/seatunnel/transform/jsonpath/JsonPathTransform.java @@ -16,7 +16,9 @@ */ package org.apache.seatunnel.transform.jsonpath; +import org.apache.seatunnel.shade.com.fasterxml.jackson.core.JsonProcessingException; import org.apache.seatunnel.shade.com.fasterxml.jackson.databind.JsonNode; +import org.apache.seatunnel.shade.org.apache.commons.lang3.exception.ExceptionUtils; import org.apache.seatunnel.api.table.catalog.CatalogTable; import org.apache.seatunnel.api.table.catalog.Column; @@ -25,6 +27,9 @@ import org.apache.seatunnel.api.table.type.SeaTunnelRow; import org.apache.seatunnel.api.table.type.SeaTunnelRowAccessor; import org.apache.seatunnel.api.table.type.SeaTunnelRowType; import org.apache.seatunnel.common.exception.CommonError; +import org.apache.seatunnel.common.exception.CommonErrorCode; +import org.apache.seatunnel.common.exception.SeaTunnelErrorCode; +import org.apache.seatunnel.common.exception.SeaTunnelRuntimeException; import org.apache.seatunnel.common.utils.JsonUtils; import org.apache.seatunnel.format.json.JsonToRowConverters; import org.apache.seatunnel.transform.common.MultipleFieldOutputTransform; @@ -35,19 +40,24 @@ import com.jayway.jsonpath.JsonPath; import com.jayway.jsonpath.JsonPathException; import lombok.extern.slf4j.Slf4j; +import java.time.DateTimeException; import java.util.Arrays; import java.util.HashSet; import java.util.List; import java.util.Map; import java.util.Set; import java.util.concurrent.ConcurrentHashMap; +import java.util.regex.Pattern; import static org.apache.seatunnel.transform.exception.JsonPathTransformErrorCode.JSON_PATH_COMPILE_ERROR; +import static org.apache.seatunnel.transform.exception.JsonPathTransformErrorCode.JSON_PATH_CONVERSION_ERROR; @Slf4j public class JsonPathTransform extends MultipleFieldOutputTransform { public static final String PLUGIN_NAME = "JsonPath"; + private static final Pattern SAFE_FIELD_IDENTIFIER = + Pattern.compile("[A-Za-z_][A-Za-z0-9_]{0,63}"); private static final Map<String, JsonPath> JSON_PATH_CACHE = new ConcurrentHashMap<>(); private final JsonPathTransformConfig config; private final SeaTunnelRowType seaTunnelRowType; @@ -119,6 +129,7 @@ public class JsonPathTransform extends MultipleFieldOutputTransform { doTransform( seaTunnelRowType.getFieldType(pos), inputRow.getField(pos), + i, fieldConfig, converters[i]); } @@ -128,13 +139,20 @@ public class JsonPathTransform extends MultipleFieldOutputTransform { private Object doTransform( SeaTunnelDataType<?> inputDataType, Object value, + int columnIndex, ColumnConfig columnConfig, JsonToRowConverters.JsonToObjectConverter converter) { if (value == null) { return null; } - JSON_PATH_CACHE.computeIfAbsent(columnConfig.getPath(), JsonPath::compile); + try { + JSON_PATH_CACHE.computeIfAbsent(columnConfig.getPath(), JsonPath::compile); + } catch (JsonPathException e) { + // Invalid configuration is task-failing, regardless of the data error policy. + throw new JsonPathException(pathFailureMessage(columnIndex, columnConfig, e)); + } String jsonString = ""; + JsonNode jsonNode; try { switch (inputDataType.getSqlType()) { case STRING: @@ -158,25 +176,127 @@ public class JsonPathTransform extends MultipleFieldOutputTransform { columnConfig.getSrcField()); } Object result = JSON_PATH_CACHE.get(columnConfig.getPath()).read(jsonString); - JsonNode jsonNode = JsonUtils.toJsonNode(result); - return converter.convert(jsonNode, columnConfig.getDestField()); + jsonNode = JsonUtils.toJsonNode(result); } catch (JsonPathException e) { - if (columnConfig.errorHandleWay() != null - && columnConfig.errorHandleWay().allowSkip()) { + return handleError(columnIndex, columnConfig, JSON_PATH_COMPILE_ERROR, e); + } + try { + return converter.convert(jsonNode, columnConfig.getDestField()); + } catch (RuntimeException e) { + // Nested row converters can wrap Errors; these are not skippable data failures. + List<Throwable> causes = ExceptionUtils.getThrowableList(e); + for (Throwable cause : causes) { + if (cause instanceof Error) { + throw (Error) cause; + } + } + if (!isDataConversionFailure(causes)) { + throw e; + } + return handleConversionError(columnIndex, columnConfig); + } + } + + /** + * Accepts only numeric, date/time and wrapped JSON data failures. Every cause must be + * recognized; unknown failures, unsupported conversions and cyclic chains fail closed. + */ + private static boolean isDataConversionFailure(List<Throwable> causes) { + boolean jsonOperation = false; + for (Throwable cause : causes) { + if (cause instanceof SeaTunnelRuntimeException) { + SeaTunnelErrorCode code = + ((SeaTunnelRuntimeException) cause).getSeaTunnelErrorCode(); + if (code == CommonErrorCode.JSON_OPERATION_FAILED && cause.getCause() != null) { + // This wrapper also carries programming failures; inspect the rest of the + // chain. + jsonOperation = true; + continue; + } + if (code == CommonErrorCode.FORMAT_DATE_ERROR + || code == CommonErrorCode.FORMAT_DATETIME_ERROR) { + continue; + } + } + if (cause instanceof NumberFormatException + || cause instanceof DateTimeException + || (jsonOperation && cause instanceof JsonProcessingException)) { + continue; + } + return false; + } + // A cyclic cause chain has no recognized terminal data failure. + return causes.get(causes.size() - 1).getCause() == null; + } + + /** + * Applies column SKIP locally and delegates other policies to the row handler. Diagnostics + * identify a generic conversion failure, never values, paths or original exceptions. + */ + private Object handleConversionError(int columnIndex, ColumnConfig columnConfig) { + if (columnConfig.errorHandleWay() != null && columnConfig.errorHandleWay().allowSkip()) { + if (log.isDebugEnabled()) { log.debug( - "JsonPath transform error, ignore error, config: {}, value: {}", - columnConfig, - jsonString, - e); - return null; + "Skipping column: {}", conversionFailureMessage(columnIndex, columnConfig)); } - throw new ErrorDataTransformException( - columnConfig.errorHandleWay(), - JSON_PATH_COMPILE_ERROR, - String.format( - "JsonPath transform error, config: %s, value: %s, error: %s", - columnConfig, jsonString, e.getMessage())); + return null; } + throw new ErrorDataTransformException( + columnConfig.errorHandleWay(), + JSON_PATH_CONVERSION_ERROR, + conversionFailureMessage(columnIndex, columnConfig)); + } + + private static String conversionFailureMessage(int columnIndex, ColumnConfig columnConfig) { + return String.format( + "JsonPath data conversion failure, column_index=%d, src_field=%s, dest_field=%s, dest_type=%s", + columnIndex, + safeFieldIdentifier(columnConfig.getSrcField()), + safeFieldIdentifier(columnConfig.getDestField()), + columnConfig.getDestType().getSqlType()); + } + + private static String safeFieldIdentifier(String field) { + return field != null && SAFE_FIELD_IDENTIFIER.matcher(field).matches() + ? field + : "<redacted>"; + } + + /** + * Applies an explicit column policy first. Otherwise the exception leaves policy resolution to + * the row-level handler in AbstractSeaTunnelTransform. + */ + private Object handleError( + int columnIndex, + ColumnConfig columnConfig, + SeaTunnelErrorCode errorCode, + RuntimeException cause) { + if (columnConfig.errorHandleWay() != null && columnConfig.errorHandleWay().allowSkip()) { + if (log.isDebugEnabled()) { + log.debug( + "JsonPath transform error, ignore error, {}", + pathFailureMessage(columnIndex, columnConfig, cause)); + } + return null; + } + ErrorDataTransformException error = + new ErrorDataTransformException( + columnConfig.errorHandleWay(), + errorCode, + pathFailureMessage(columnIndex, columnConfig, cause)); + // The original JsonPathException may contain the source or configured path in its message. + error.initCause(new JsonPathException("Original path-reading details omitted")); + throw error; + } + + private static String pathFailureMessage( + int columnIndex, ColumnConfig columnConfig, RuntimeException cause) { + return String.format( + "JsonPath path-reading failure, column_index=%d, src_field=%s, dest_field=%s, cause_type=%s", + columnIndex, + safeFieldIdentifier(columnConfig.getSrcField()), + safeFieldIdentifier(columnConfig.getDestField()), + cause.getClass().getSimpleName()); } @Override diff --git a/seatunnel-transforms-v2/src/test/java/org/apache/seatunnel/transform/JsonPathErrorHandlingTest.java b/seatunnel-transforms-v2/src/test/java/org/apache/seatunnel/transform/JsonPathErrorHandlingTest.java new file mode 100644 index 00000000000..e4f9b88c694 --- /dev/null +++ b/seatunnel-transforms-v2/src/test/java/org/apache/seatunnel/transform/JsonPathErrorHandlingTest.java @@ -0,0 +1,742 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.seatunnel.transform; + +import org.apache.seatunnel.shade.com.fasterxml.jackson.databind.node.JsonNodeFactory; +import org.apache.seatunnel.shade.com.fasterxml.jackson.databind.node.ObjectNode; +import org.apache.seatunnel.shade.com.fasterxml.jackson.databind.node.TextNode; + +import org.apache.seatunnel.api.configuration.ReadonlyConfig; +import org.apache.seatunnel.api.table.catalog.CatalogTable; +import org.apache.seatunnel.api.table.catalog.CatalogTableUtil; +import org.apache.seatunnel.api.table.catalog.PhysicalColumn; +import org.apache.seatunnel.api.table.type.BasicType; +import org.apache.seatunnel.api.table.type.RowKind; +import org.apache.seatunnel.api.table.type.SeaTunnelDataType; +import org.apache.seatunnel.api.table.type.SeaTunnelRow; +import org.apache.seatunnel.api.table.type.SeaTunnelRowType; +import org.apache.seatunnel.common.exception.CommonError; +import org.apache.seatunnel.common.exception.CommonErrorCodeDeprecated; +import org.apache.seatunnel.common.utils.ExceptionUtils; +import org.apache.seatunnel.format.json.JsonToRowConverters; +import org.apache.seatunnel.format.json.exception.SeaTunnelJsonFormatException; +import org.apache.seatunnel.transform.common.AbstractSeaTunnelTransform; +import org.apache.seatunnel.transform.common.ErrorHandleWay; +import org.apache.seatunnel.transform.exception.ErrorDataTransformException; +import org.apache.seatunnel.transform.exception.JsonPathTransformErrorCode; +import org.apache.seatunnel.transform.jsonpath.ColumnConfig; +import org.apache.seatunnel.transform.jsonpath.JsonPathTransform; +import org.apache.seatunnel.transform.jsonpath.JsonPathTransformConfig; + +import org.apache.logging.log4j.Level; +import org.apache.logging.log4j.LogManager; +import org.apache.logging.log4j.core.LogEvent; +import org.apache.logging.log4j.core.Logger; +import org.apache.logging.log4j.core.appender.AbstractAppender; +import org.apache.logging.log4j.core.config.Property; + +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.params.ParameterizedTest; +import org.junit.jupiter.params.provider.ValueSource; + +import com.jayway.jsonpath.JsonPathException; + +import java.io.IOException; +import java.io.PrintWriter; +import java.io.StringWriter; +import java.lang.reflect.Field; +import java.time.LocalDate; +import java.util.ArrayList; +import java.util.Arrays; +import java.util.HashMap; +import java.util.List; +import java.util.Map; + +class JsonPathErrorHandlingTest { + + private static final String PRIVATE_VALUE = "SYNTHETIC_PRIVATE_VALUE_9778"; + private static final String PRIVATE_RECORD = "SYNTHETIC_PRIVATE_RECORD_9778"; + private static final String PRIVATE_SUPPRESSED = "SYNTHETIC_PRIVATE_SUPPRESSED_9778"; + + @ParameterizedTest + @ValueSource( + strings = { + "int", + "date", + "time", + "timestamp", + "decimal(10,2)", + "bytes", + "array<int>", + "map<string,int>", + "row" + }) + void testConversionErrorPolicies(String type) { + String invalidJson = "{\"amount\":\"!invalid!\",\"description\":\"retained\"}"; + if (type.startsWith("array")) { + invalidJson = "{\"amount\":[\"!invalid!\"],\"description\":\"retained\"}"; + } else if (type.startsWith("map") || type.startsWith("row")) { + invalidJson = "{\"amount\":{\"amount\":\"!invalid!\"},\"description\":\"retained\"}"; + } + SeaTunnelRow input = new SeaTunnelRow(new Object[] {invalidJson}); + input.setRowKind(RowKind.UPDATE_AFTER); + input.setTableId("orders"); + JsonPathTransform skipColumn = + createTransform(type, ErrorHandleWay.SKIP, ErrorHandleWay.FAIL); + SeaTunnelRow output = skipColumn.map(input); + Assertions.assertNotNull(output); + Assertions.assertNull(output.getField(1)); + Assertions.assertEquals("retained", output.getField(2)); + Assertions.assertEquals(invalidJson, input.getField(0)); + Assertions.assertEquals(input.getRowKind(), output.getRowKind()); + Assertions.assertEquals(input.getTableId(), output.getTableId()); + + Assertions.assertNull( + createTransform(type, ErrorHandleWay.SKIP_ROW, ErrorHandleWay.FAIL).map(input)); + Assertions.assertNull(createTransform(type, null, ErrorHandleWay.SKIP).map(input)); + + JsonPathTransform failColumn = + createTransform(type, ErrorHandleWay.FAIL, ErrorHandleWay.SKIP); + ErrorDataTransformException failure = + Assertions.assertThrows( + ErrorDataTransformException.class, () -> failColumn.map(input)); + Assertions.assertEquals(ErrorHandleWay.FAIL, failure.getErrorHandleWay()); + Assertions.assertEquals( + JsonPathTransformErrorCode.JSON_PATH_CONVERSION_ERROR, + failure.getSeaTunnelErrorCode()); + Assertions.assertNull(failure.getCause()); + + JsonPathTransform defaultPolicy = createTransform(type, null, null); + Assertions.assertThrows(ErrorDataTransformException.class, () -> defaultPolicy.map(input)); + } + + @Test + void testDateConversionAfterValidRow() { + JsonPathTransform transform = createTransform("date", ErrorHandleWay.SKIP, null); + Assertions.assertEquals( + LocalDate.of(2026, 9, 9), transform.map(row("\"2026-09-09\"")).getField(1)); + Assertions.assertNull(transform.map(row("\"invalid\"")).getField(1)); + Assertions.assertEquals( + LocalDate.of(2026, 9, 10), transform.map(row("\"2026-09-10\"")).getField(1)); + } + + @Test + void testNullAndValidValues() { + JsonPathTransform transform = createTransform("int", null, null); + Assertions.assertEquals(42, transform.map(row("42")).getField(1)); + Assertions.assertNull(transform.map(row("null")).getField(1)); + SeaTunnelRow nullSource = transform.map(new SeaTunnelRow(new Object[] {null})); + Assertions.assertNull(nullSource.getField(1)); + Assertions.assertNull(nullSource.getField(2)); + } + + @Test + void testMissingPathAndInvalidJsonPolicies() { + JsonPathTransform skipColumn = createTransform("int", ErrorHandleWay.SKIP, null); + Assertions.assertNull(skipColumn.map(new SeaTunnelRow(new Object[] {"{}"})).getField(1)); + JsonPathTransform skipRow = createTransform("int", null, ErrorHandleWay.SKIP); + Assertions.assertNull(skipRow.map(new SeaTunnelRow(new Object[] {"{invalid"}))); + ErrorDataTransformException failure = + Assertions.assertThrows( + ErrorDataTransformException.class, + () -> + createTransform("int", null, null) + .map(new SeaTunnelRow(new Object[] {"{}"}))); + Assertions.assertTrue(failure.getMessage().contains("src_field=content")); + Assertions.assertTrue(failure.getMessage().contains("dest_field=amount")); + Assertions.assertFalse(failure.getMessage().contains("{}")); + } + + @Test + void testPathFailureDiagnosticsOmitShortAndLongPayloads() { + String longJson = largePrivateJson(); + String shortJson = "{\"private\":\"" + PRIVATE_RECORD + "\"}"; + for (String source : + Arrays.asList(shortJson, longJson, longJson.substring(0, longJson.length() - 1))) { + JsonPathTransform transform = createTransform("int", null, null); + ErrorDataTransformException failure = + Assertions.assertThrows( + ErrorDataTransformException.class, + () -> transform.map(new SeaTunnelRow(new Object[] {source}))); + + assertSafePathFailure(failure, source); + Assertions.assertNull( + createTransform("int", ErrorHandleWay.SKIP, null) + .map(new SeaTunnelRow(new Object[] {source})) + .getField(1)); + } + } + + @Test + void testPrivatePathLiteralIsAbsentFromFailureAndCause() { + for (String path : + Arrays.asList("$['" + PRIVATE_VALUE + "']", "$['" + largePrivateValue() + "']")) { + ErrorDataTransformException failure = + Assertions.assertThrows( + ErrorDataTransformException.class, + () -> + createTransform("int", null, null, path) + .map(new SeaTunnelRow(new Object[] {"{}"}))); + assertSafePathFailure(failure, "{}"); + } + } + + @Test + void testInvalidPrivatePathIsSanitizedAndNotSkipped() { + String path = "$['" + PRIVATE_VALUE; + JsonPathTransform transform = + createTransform("int", ErrorHandleWay.SKIP, ErrorHandleWay.SKIP, path); + JsonPathException failure = + Assertions.assertThrows( + JsonPathException.class, + () -> transform.map(new SeaTunnelRow(new Object[] {"{}"}))); + String trace = renderedTrace(failure); + Assertions.assertFalse(trace.contains(PRIVATE_VALUE)); + Assertions.assertTrue(trace.contains("src_field=content")); + Assertions.assertNull(failure.getCause()); + } + + @Test + void testUnsafeFieldIdentifierIsRedacted() { + JsonPathTransform transform = + createTransform("int", null, null, "$.amount", "private/" + PRIVATE_VALUE); + ErrorDataTransformException failure = + Assertions.assertThrows( + ErrorDataTransformException.class, + () -> transform.map(new SeaTunnelRow(new Object[] {"{}"}))); + Assertions.assertTrue(failure.getMessage().contains("dest_field=<redacted>")); + Assertions.assertTrue(failure.getMessage().contains("column_index=0")); + Assertions.assertFalse(renderedTrace(failure).contains(PRIVATE_VALUE)); + } + + @Test + void testNumericSensitiveFieldIdentifierIsRedacted() { + JsonPathTransform transform = createTransform("int", null, null, "$.amount", "123-45-6789"); + ErrorDataTransformException failure = + Assertions.assertThrows( + ErrorDataTransformException.class, + () -> transform.map(new SeaTunnelRow(new Object[] {"{}"}))); + Assertions.assertTrue(failure.getMessage().contains("dest_field=<redacted>")); + Assertions.assertTrue(failure.getMessage().contains("column_index=0")); + Assertions.assertFalse(renderedTrace(failure).contains("123-45-6789")); + } + + @ParameterizedTest + @ValueSource(strings = {"user-id", "user.name", "customer name", "客户编号"}) + void testRedactedFieldIdentifierHasColumnIndex(String destField) { + JsonPathTransform transform = createTransform("int", null, null, "$.amount", destField); + ErrorDataTransformException failure = + Assertions.assertThrows( + ErrorDataTransformException.class, + () -> transform.map(new SeaTunnelRow(new Object[] {"{}"}))); + Assertions.assertTrue(failure.getMessage().contains("dest_field=<redacted>")); + Assertions.assertTrue(failure.getMessage().contains("column_index=0")); + } + + @Test + void testBatchExtractionReportsFlattenedColumnIndex() { + Map<String, Object> column = new HashMap<>(); + column.put("src_field", "content"); + column.put("path", Arrays.asList("$.first", "$.second")); + column.put("dest_field", Arrays.asList("first", "user-id")); + column.put("dest_type", Arrays.asList("string", "int")); + Map<String, Object> options = new HashMap<>(); + options.put("columns", Arrays.asList(column)); + CatalogTable table = + CatalogTableUtil.getCatalogTable( + "orders", + new SeaTunnelRowType( + new String[] {"content"}, + new SeaTunnelDataType[] {BasicType.STRING_TYPE})); + JsonPathTransform transform = + new JsonPathTransform( + JsonPathTransformConfig.of(ReadonlyConfig.fromMap(options), table), table); + + ErrorDataTransformException failure = + Assertions.assertThrows( + ErrorDataTransformException.class, + () -> + transform.map( + new SeaTunnelRow( + new Object[] { + "{\"first\":\"ok\",\"second\":\"invalid\"}" + }))); + Assertions.assertTrue(failure.getMessage().contains("column_index=1")); + Assertions.assertTrue(failure.getMessage().contains("dest_field=<redacted>")); + } + + private static void assertSafePathFailure(ErrorDataTransformException failure, String source) { + Assertions.assertTrue(failure.getMessage().contains("src_field=content")); + Assertions.assertTrue(failure.getMessage().contains("dest_field=amount")); + Assertions.assertTrue(failure.getMessage().contains("cause_type=")); + Assertions.assertEquals(JsonPathException.class, failure.getCause().getClass()); + Assertions.assertNull(failure.getCause().getCause()); + Assertions.assertEquals(0, failure.getCause().getSuppressed().length); + String trace = renderedTrace(failure); + Assertions.assertFalse(trace.contains(source)); + Assertions.assertFalse(trace.contains(PRIVATE_VALUE)); + Assertions.assertFalse(trace.contains(PRIVATE_RECORD)); + } + + private static String renderedTrace(Throwable failure) { + StringWriter writer = new StringWriter(); + failure.printStackTrace(new PrintWriter(writer)); + return writer.toString(); + } + + private static String largePrivateJson() { + return "{\"private\":\"" + largePrivateValue() + "\"}"; + } + + private static String largePrivateValue() { + StringBuilder longValueBuilder = new StringBuilder(); + for (int i = 0; i < 40; i++) { + longValueBuilder.append(PRIVATE_RECORD); + } + return longValueBuilder.toString(); + } + + @ParameterizedTest + @ValueSource(strings = {"int", "date", "bytes", "row"}) + void testConversionFailureDoesNotExposeSourceOrExtractedValue(String type) { + String value = "\"" + PRIVATE_VALUE + "\""; + if (type.startsWith("row")) { + value = "{\"amount\":" + value + "}"; + } + SeaTunnelRow input = + new SeaTunnelRow( + new Object[] { + "{\"amount\":" + value + ",\"private\":\"" + PRIVATE_RECORD + "\"}" + }); + JsonPathTransform transform = createTransform(type, null, null); + ErrorDataTransformException failure = + Assertions.assertThrows( + ErrorDataTransformException.class, () -> transform.map(input)); + + assertNoPayload(failure); + Assertions.assertTrue(failure.getMessage().contains("src_field=content")); + Assertions.assertTrue(failure.getMessage().contains("dest_field=amount")); + Assertions.assertTrue(failure.getMessage().contains("dest_type=")); + } + + @Test + void testConfiguredPathLiteralIsNotExposedByConversionFailure() { + JsonPathTransform transform = + createTransform("int", null, null, "$['" + PRIVATE_VALUE + "']"); + SeaTunnelRow input = + new SeaTunnelRow(new Object[] {"{\"" + PRIVATE_VALUE + "\":\"invalid\"}"}); + ErrorDataTransformException failure = + Assertions.assertThrows( + ErrorDataTransformException.class, () -> transform.map(input)); + assertNoPayload(failure); + Assertions.assertTrue(failure.getMessage().contains("JsonPath data conversion failure")); + } + + @Test + void testKnownConversionFailureDropsCausesAndSuppressedData() throws Exception { + RuntimeException cause = + CommonError.jsonOperationError( + "Common", PRIVATE_RECORD, new NumberFormatException(PRIVATE_VALUE)); + cause.addSuppressed(new IllegalStateException(PRIVATE_SUPPRESSED)); + JsonPathTransform transform = createTransform("int", null, null); + setConverter( + transform, + (node, field) -> { + throw cause; + }); + assertNoPayload( + Assertions.assertThrows( + ErrorDataTransformException.class, () -> transform.map(row("42")))); + } + + @ParameterizedTest + @ValueSource(strings = {"SKIP", "SKIP_ROW", "ROW_SKIP"}) + void testDateErrorWrappersInspectTheirCauses(String policy) throws Exception { + for (RuntimeException dateError : + new RuntimeException[] { + CommonError.formatDateError(PRIVATE_VALUE, "amount"), + CommonError.formatDateTimeError(PRIVATE_VALUE, "amount") + }) { + JsonPathTransform transform = createSkippingTransform(policy); + RuntimeException wrapper = + CommonError.jsonOperationError("Common", PRIVATE_RECORD, dateError); + setConverter( + transform, + (node, field) -> { + throw wrapper; + }); + SeaTunnelRow output = transform.map(row("42")); + if ("SKIP".equals(policy)) { + Assertions.assertNull(output.getField(1)); + } else { + Assertions.assertNull(output); + } + dateError.initCause(new NullPointerException(PRIVATE_VALUE)); + Assertions.assertSame( + wrapper, + Assertions.assertThrows( + RuntimeException.class, () -> transform.map(row("42")))); + } + } + + @Test + void testSkippedErrorDebugMessagesDoNotExposeFullPayload() { + Logger logger = (Logger) LogManager.getLogger(JsonPathTransform.class); + Level oldLevel = logger.getLevel(); + List<LogEvent> events = new ArrayList<>(); + AbstractAppender appender = + new AbstractAppender( + "jsonpath-conversion-test", null, null, true, Property.EMPTY_ARRAY) { + @Override + public void append(LogEvent event) { + events.add(event.toImmutable()); + } + }; + appender.start(); + logger.addAppender(appender); + logger.setLevel(Level.DEBUG); + try { + JsonPathTransform transform = createTransform("int", ErrorHandleWay.SKIP, null); + Assertions.assertNull(transform.map(row("\"" + PRIVATE_VALUE + "\"")).getField(1)); + Assertions.assertEquals(1, events.size()); + Assertions.assertTrue( + events.get(0) + .getMessage() + .getFormattedMessage() + .contains("data conversion failure")); + Assertions.assertFalse( + events.get(0).getMessage().getFormattedMessage().contains(PRIVATE_VALUE)); + Assertions.assertNull(events.get(0).getThrown()); + + for (String json : + Arrays.asList("{\"private\":\"" + PRIVATE_RECORD + "\"}", largePrivateJson())) { + JsonPathTransform pathTransform = createTransform("int", ErrorHandleWay.SKIP, null); + Assertions.assertNull( + pathTransform.map(new SeaTunnelRow(new Object[] {json})).getField(1)); + } + JsonPathTransform privatePath = + createTransform("int", ErrorHandleWay.SKIP, null, "$['" + PRIVATE_VALUE + "']"); + Assertions.assertNull( + privatePath.map(new SeaTunnelRow(new Object[] {"{}"})).getField(1)); + Assertions.assertEquals(7, events.size()); + for (int i = 1; i < events.size(); i++) { + String message = events.get(i).getMessage().getFormattedMessage(); + Assertions.assertFalse(message.contains(PRIVATE_RECORD)); + Assertions.assertFalse(message.contains(PRIVATE_VALUE)); + Assertions.assertFalse(message.contains("{}")); + Assertions.assertTrue(message.contains("src_field=content")); + Assertions.assertNull(events.get(i).getThrown()); + } + } finally { + logger.removeAppender(appender); + logger.setLevel(oldLevel); + appender.stop(); + } + } + + @Test + void testSkippedRowDebugTraceOmitsPayloadAndPath() { + Logger logger = (Logger) LogManager.getLogger(AbstractSeaTunnelTransform.class); + Level oldLevel = logger.getLevel(); + boolean oldAdditive = logger.isAdditive(); + List<LogEvent> events = new ArrayList<>(); + AbstractAppender appender = + new AbstractAppender( + "jsonpath-row-skip-test", null, null, true, Property.EMPTY_ARRAY) { + @Override + public void append(LogEvent event) { + events.add(event.toImmutable()); + } + }; + appender.start(); + logger.addAppender(appender); + logger.setLevel(Level.DEBUG); + logger.setAdditive(false); + try { + for (String source : + Arrays.asList("{\"private\":\"" + PRIVATE_RECORD + "\"}", largePrivateJson())) { + JsonPathTransform transform = + createTransform( + "int", null, ErrorHandleWay.SKIP, "$['" + PRIVATE_VALUE + "']"); + Assertions.assertNull(transform.map(new SeaTunnelRow(new Object[] {source}))); + } + Assertions.assertEquals(2, events.size()); + for (LogEvent event : events) { + Assertions.assertNotNull(event.getThrown()); + String trace = renderedTrace(event.getThrown()); + Assertions.assertFalse(trace.contains(PRIVATE_RECORD)); + Assertions.assertFalse(trace.contains(PRIVATE_VALUE)); + Assertions.assertTrue(trace.contains("src_field=content")); + } + } finally { + logger.removeAppender(appender); + logger.setLevel(oldLevel); + logger.setAdditive(oldAdditive); + appender.stop(); + } + } + + @ParameterizedTest + @ValueSource(strings = {"SKIP", "SKIP_ROW", "ROW_SKIP"}) + void testProgrammingAndUnsupportedFailuresAreNotSkipped(String policy) throws Exception { + for (RuntimeException cause : + new RuntimeException[] { + new NullPointerException(PRIVATE_VALUE), + new UnsupportedOperationException(PRIVATE_VALUE), + new IllegalStateException(PRIVATE_VALUE), + new IllegalArgumentException(PRIVATE_VALUE), + CommonError.jsonOperationError("Common", PRIVATE_RECORD), + CommonError.jsonOperationError( + "Common", PRIVATE_RECORD, new IOException(PRIVATE_VALUE)), + new SeaTunnelJsonFormatException( + CommonErrorCodeDeprecated.UNSUPPORTED_DATA_TYPE, PRIVATE_VALUE), + new IllegalStateException( + PRIVATE_VALUE, new NumberFormatException(PRIVATE_VALUE)) + }) { + for (RuntimeException failure : + new RuntimeException[] { + cause, CommonError.jsonOperationError("Common", PRIVATE_RECORD, cause) + }) { + JsonPathTransform transform = createSkippingTransform(policy); + setConverter( + transform, + (node, field) -> { + throw failure; + }); + Assertions.assertSame( + failure, + Assertions.assertThrows( + RuntimeException.class, () -> transform.map(row("42")))); + } + } + } + + @ParameterizedTest + @ValueSource(strings = {"SKIP", "SKIP_ROW", "ROW_SKIP"}) + void testRealUnsupportedVectorConversionIsNotSkipped(String policy) { + JsonPathTransform transform = + createTransform( + "float_vector", + "ROW_SKIP".equals(policy) ? null : ErrorHandleWay.valueOf(policy), + ErrorHandleWay.SKIP); + SeaTunnelJsonFormatException failure = + Assertions.assertThrows( + SeaTunnelJsonFormatException.class, + () -> transform.map(row("\"invalid\""))); + Assertions.assertEquals( + CommonErrorCodeDeprecated.UNSUPPORTED_DATA_TYPE, failure.getSeaTunnelErrorCode()); + } + + @Test + void testRouteToTableRemainsUnsupportedForDataErrors() { + for (String value : new String[] {"{\"amount\":\"invalid\"}", "{}"}) { + SeaTunnelRow input = new SeaTunnelRow(new Object[] {value}); + JsonPathTransform rowRoute = + createTransform("int", null, ErrorHandleWay.ROUTE_TO_TABLE); + JsonPathTransform columnRoute = + createTransform("int", ErrorHandleWay.ROUTE_TO_TABLE, ErrorHandleWay.SKIP); + Assertions.assertThrows(ErrorDataTransformException.class, () -> rowRoute.map(input)); + Assertions.assertThrows( + ErrorDataTransformException.class, () -> columnRoute.map(input)); + } + } + + private static void assertNoPayload(RuntimeException failure) { + String trace = ExceptionUtils.getMessage(failure); + Assertions.assertFalse(trace.contains(PRIVATE_VALUE)); + Assertions.assertFalse(trace.contains(PRIVATE_RECORD)); + Assertions.assertFalse(trace.contains(PRIVATE_SUPPRESSED)); + Assertions.assertNull(failure.getCause()); + Assertions.assertEquals(0, failure.getSuppressed().length); + } + + @Test + void testSourceFailureIsNotSkipped() { + IllegalStateException sourceFailure = new IllegalStateException("Source value unavailable"); + Object sourceValue = + new Object() { + @Override + public String toString() { + throw sourceFailure; + } + }; + JsonPathTransform transform = + createTransform("int", ErrorHandleWay.SKIP, ErrorHandleWay.SKIP); + Assertions.assertSame( + sourceFailure, + Assertions.assertThrows( + IllegalStateException.class, + () -> transform.map(new SeaTunnelRow(new Object[] {sourceValue})))); + } + + @ParameterizedTest + @ValueSource(strings = {"SKIP", "SKIP_ROW", "ROW_SKIP"}) + void testWrappedErrorsAreNotSkipped(String policy) throws Exception { + for (Error fatal : + new Error[] { + new OutOfMemoryError("synthetic failure"), + new StackOverflowError("synthetic failure"), + new LinkageError("synthetic failure") + }) { + JsonPathTransform transform = createSkippingTransform(policy); + ObjectNode object = JsonNodeFactory.instance.objectNode(); + object.set( + "amount", + new TextNode("42") { + @Override + public String asText() { + throw fatal; + } + }); + JsonToRowConverters.JsonToObjectConverter rowConverter = + new JsonToRowConverters(false, false) + .createRowConverter( + new SeaTunnelRowType( + new String[] {"amount"}, + new SeaTunnelDataType[] {BasicType.INT_TYPE})); + // Exercise the real row converter's Throwable wrapper without exhausting resources. + setConverter(transform, (node, field) -> rowConverter.convert(object, field)); + Assertions.assertSame( + fatal, Assertions.assertThrows(Error.class, () -> transform.map(row("42")))); + } + } + + @ParameterizedTest + @ValueSource(strings = {"SKIP", "SKIP_ROW", "ROW_SKIP"}) + void testWrappedDataErrorsStillUsePolicy(String policy) { + JsonPathTransform transform = + createTransform( + "row", + "ROW_SKIP".equals(policy) ? null : ErrorHandleWay.valueOf(policy), + ErrorHandleWay.SKIP); + SeaTunnelRow output = transform.map(row("{\"amount\":\"invalid\"}")); + if ("SKIP".equals(policy)) { + Assertions.assertNotNull(output); + Assertions.assertNull(output.getField(1)); + Assertions.assertEquals("retained", output.getField(2)); + } else { + Assertions.assertNull(output); + } + } + + @Test + void testCyclicConversionCauseDoesNotLoop() throws Exception { + RuntimeException outer = new IllegalArgumentException("outer"); + RuntimeException inner = new IllegalArgumentException("inner", outer); + outer.initCause(inner); + JsonPathTransform transform = createSkippingTransform("SKIP"); + setConverter( + transform, + (node, field) -> { + throw outer; + }); + Assertions.assertSame( + outer, + Assertions.assertThrows(RuntimeException.class, () -> transform.map(row("42")))); + } + + private static JsonPathTransform createSkippingTransform(String policy) { + return createTransform( + "int", + "ROW_SKIP".equals(policy) ? null : ErrorHandleWay.valueOf(policy), + ErrorHandleWay.SKIP); + } + + private static void setConverter( + JsonPathTransform transform, JsonToRowConverters.JsonToObjectConverter converter) + throws Exception { + // Controlled failures cannot be produced from ordinary JSON without provoking real JVM + // bugs. + // Keep injection local to this test instead of exposing a production API for tests. + Field field = JsonPathTransform.class.getDeclaredField("converters"); + field.setAccessible(true); + ((JsonToRowConverters.JsonToObjectConverter[]) field.get(transform))[0] = converter; + } + + private static SeaTunnelRow row(String value) { + return new SeaTunnelRow( + new Object[] {"{\"amount\":" + value + ",\"description\":\"retained\"}"}); + } + + private static JsonPathTransform createTransform( + String type, ErrorHandleWay columnPolicy, ErrorHandleWay rowPolicy) { + return createTransform(type, columnPolicy, rowPolicy, "$.amount"); + } + + private static JsonPathTransform createTransform( + String type, ErrorHandleWay columnPolicy, ErrorHandleWay rowPolicy, String path) { + return createTransform(type, columnPolicy, rowPolicy, path, "amount"); + } + + private static JsonPathTransform createTransform( + String type, + ErrorHandleWay columnPolicy, + ErrorHandleWay rowPolicy, + String path, + String destField) { + Map<String, Object> column = new HashMap<>(); + column.put("src_field", "content"); + column.put("path", path); + column.put("dest_field", destField); + column.put("dest_type", "row".equals(type) ? "string" : type); + if (columnPolicy != null) { + column.put("column_error_handle_way", columnPolicy.name()); + } + Map<String, Object> description = new HashMap<>(); + description.put("src_field", "content"); + description.put("path", "$.description"); + description.put("dest_field", "description"); + description.put("column_error_handle_way", "SKIP"); + Map<String, Object> options = new HashMap<>(); + options.put("columns", Arrays.asList(column, description)); + if (rowPolicy != null) { + options.put("row_error_handle_way", rowPolicy.name()); + } + CatalogTable table = + CatalogTableUtil.getCatalogTable( + "orders", + new SeaTunnelRowType( + new String[] {"content"}, + new SeaTunnelDataType[] {BasicType.STRING_TYPE})); + JsonPathTransformConfig config = + JsonPathTransformConfig.of(ReadonlyConfig.fromMap(options), table); + if ("row".equals(type)) { + // The config parser does not accept row<...> strings; use the public typed config. + config.getColumnConfigs() + .set( + 0, + new ColumnConfig( + "$.amount", + "content", + "amount", + PhysicalColumn.of( + "amount", + new SeaTunnelRowType( + new String[] {"amount"}, + new SeaTunnelDataType[] {BasicType.INT_TYPE}), + (Long) null, + true, + null, + null), + columnPolicy)); + } + JsonPathTransform transform = new JsonPathTransform(config, table); + transform.getProducedCatalogTable(); + return transform; + } +}
