zhangshenghang opened a new issue, #12714: URL: https://github.com/apache/seatunnel/issues/12714
### Search before asking - [x] I had searched in the [issues](https://github.com/apache/seatunnel/issues?q=is%3Aissue+label%3A%22bug%22) and found no similar issues. ### What happened `StringFunction.md5` only recognizes the primitive `byte[]` as binary input: ```java // seatunnel-transforms-v2/.../StringFunction.java public static String md5(List<Object> args) { ... if (arg instanceof byte[]) { return Hashing.md5().hashBytes((byte[]) arg).toString(); } return Hashing.md5().hashString(arg.toString(), StandardCharsets.UTF_8).toString(); } ``` SeaTunnel array columns are **boxed** arrays at runtime: `ArrayType.BYTE_ARRAY_TYPE` is `Byte[].class` (seatunnel-api `ArrayType.java`). A `Byte[]` never matches `instanceof byte[]`, so `MD5()` applied to an `ARRAY<BYTE>` column silently falls into the `toString()` branch and hashes the JVM identity string of the array object. Running the exact code path with the same library confirms: ``` string actually hashed = "[Ljava.lang.Byte;@58644d46" MD5 result run 1 (same contents) = cceb6b971ad8573f23c3e97df9bc4ebe MD5 result run 2 (same contents) = e3dcfa442d03d0edc3b15b39b5144cb4 md5('hello') for reference = 5d41402abc4b2a76b9719d911017c592 ``` Two consequences: 1. The returned "MD5" has no relation to the data. 2. It is **nondeterministic** — the same input value produces a different hash on every run (identity hash code), which silently breaks any dedup/change-detection logic built on top of it. Nothing rejects the query up front: `ZetaSQLType` infers `STRING` for `MD5(...)` without validating argument types, and `ZetaSQLFunction` passes the raw column value into the function. `ARRAY<BYTE>` columns are produced by several connectors' type mappings (e.g. ClickHouse, Cassandra, Doris, Elasticsearch), so `SELECT MD5(arr_col)` is reachable in normal jobs. Note that the AES functions added in the same release explicitly reject array inputs for exactly this reason — `CryptoFunction.rejectNonScalar` ("rejects array ... inputs that would otherwise be encrypted as garbage"). MD5 should apply the same guard (or handle boxed byte arrays explicitly); silently hashing the identity string is the worst possible behavior. ### SeaTunnel Version dev (commit 0278a6a74); MD5 was added in #12564 ### SeaTunnel Config ```conf env { job.mode = "BATCH" } source { # any source producing an ARRAY<BYTE> column, e.g. ClickHouse `Array(Int8)` } transform { Sql { source_table_name = "src" result_table_name = "out" query = "SELECT MD5(arr_col) AS h FROM src" } } sink { Console {} } ``` ### Running Command ```shell $SEATUNNEL_HOME/bin/seatunnel.sh --config ./md5.conf -e run ``` ### Error Exception No exception — which is the problem. The transform silently emits a 32-character hex string that (a) is not the MD5 of the data and (b) changes between runs for identical input. ### Zeta or Flink or Spark Version Zeta ### Java or Scala Version Java 17 ### Are you willing to submit PR? - [x] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
