zhang-jianq commented on PR #13420:
URL: https://github.com/apache/gluten/pull/13420#issuecomment-6847413480
> Thanks for your PR. Could you check whether the following comments make
sense?
>
> * Nit: could these two entries go next to the SHA-2 family entries (`SHA2`
/ `SHA224` ... `SHA512`), so the hash functions stay together?
> * Non-blocking: Flink's `MD5` / `SHA1` accept the STRING family, which
includes binary types, but velox only registers `varchar`. So `MD5(bytes_col)`
would now fail at runtime with a missing signature instead of at planning.
Could we reject non-character input in `isSuitable` for a clearer error, or add
varbinary signatures in a follow-up (same as the SHA-2 family)?
> * Nit: `MD5('')` / `MD5('abc')` are likely constant-folded by the planner,
so `testMd5Constant` / `testSha1Constant` may not exercise the velox functions.
That leaves the empty-string case untested natively. Similarly, the bare
`MD5(n)` rejection comes from Flink's validator. Fine as regression checks, but
the comments could say so.
> * Nit: could `drop table if exists printT` go in a `finally`, so a failed
assertion doesn't break the next test's `CREATE TABLE printT`?
@philo-he Thanks for review. All four make sense. Here is what I changed:
1. Placement — done: `MD5` / `SHA1` now sit right after `SHA512` and before
`count_char`, so the hash family is contiguous.
2. Binary input — confirmed by an end-to-end run. Flink native handles it:
`MD5(X'6162')` / `SHA1(X'6162')` succeed and hash the raw bytes (same digests
as `'ab'`). Gluten does not: the job fails at planning with
`java.lang.RuntimeException: Unsupported logical type: BYTES NOT NULL`, before
any velox signature resolution — `LogicalTypeConverter` has no `VarBinaryType`
mapping. This is a pre-existing, function-agnostic limitation: the [SHA-2
family](https://github.com/apache/gluten/pull/13484) has exactly the same gap
(velox registers `Varchar` only there too). Full support needs two follow-ups:
map `VarBinaryType` in `LogicalTypeConverter`, and register `(Varbinary) ->
Varchar` md5/sha1 signatures in velox (plus the SHA-2 family). An `isSuitable`
rejection would only make the message clearer, and does not help a BYTES column
(type conversion already fails before the converter runs), so I left the
converter unchanged. The follow-up for binary support is tracked by [GLUTEN-1363
5](https://github.com/apache/gluten/issues/13635).
3. Constant folding — done: dropped `testMd5Constant` / `testSha1Constant`.
MD5/SHA1 correctness is now column-driven (`testMd5` / `testSha1`) with rows
for the empty string (placed in a non-trailing column so it is not dropped),
`test`, `abc`, UTF-8, a long input and NULL, asserted against pinned digests.
The non-string tests now state that the bare `MD5(n)` / `SHA1(n)` rejection
comes from Flink's validator at planning, not from our converter.
4. `finally` — done: `drop table if exists printT` now runs in a `finally`
block in both error-path tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]