chovy-3012 opened a new issue, #12563: URL: https://github.com/apache/seatunnel/issues/12563
### Search before asking - [x] I had searched in the [features](https://github.com/apache/seatunnel/issues?q=is%3Aissue+label%3A%22Feature%22) and found no similar feature requirement. The closest is #10723, which proposed exposing "a function similar to MD5 in the transformer" but was closed without the function being added. ### Description The Zeta SQL transform (`seatunnel-transforms-v2`) currently has no `MD5` built-in. A basic digest function is needed in everyday ETL / data-sync pipelines for: - **Stable fingerprints / dedup keys** — a deterministic, content-addressed key from one or more columns to detect duplicates and changes across sources. - **Sensitive-data masking / tokenization** — replace a sensitive identifier with a fixed-length, non-reversible token while keeping joinability on the digest. - **Partition / bucket key derivation** — turn arbitrary strings into a uniformly distributed sharding key. Hive, MySQL, Spark SQL, and Flink SQL all ship `MD5` as a built-in; its absence is a recurring friction point for users migrating onto SeaTunnel. ### Usage Scenario Function signature (Hive-compatible): `MD5(string) → string` — 32-char lowercase hex, same as Hive's `md5(string|binary)`. ```sql -- HBase rowkey de-hotspotting: salt the rowkey with the first 8 hex chars of MD5 SELECT SUBSTRING(MD5(user_id), 1, 8) || '_' || user_id AS rowkey, user_id, ... ``` ```sql -- stable fingerprint for dedup / change-data identification across sources SELECT MD5(CONCAT(order_id, '|', CAST(updated_at AS STRING))) AS fingerprint, ... ``` ```sql -- sensitive-data masking: replace an id with a fixed-length, non-reversible token SELECT MD5(id_card) AS id_token, ... ``` In Hive these are written `md5(...)`, `substr(md5(...), 1, 8)`, etc.; the SeaTunnel equivalent differs only in keyword casing, so existing pipelines port with a trivial `md5` → `MD5` rewrite. ### Related issues - #10723 — requested "a function similar to MD5 in the transformer" (closed; this issue tracks adding the function itself). ### Are you willing to submit a PR? - [x] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/code-of-conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
