jiangxt2 opened a new pull request, #58036: URL: https://github.com/apache/spark/pull/58036
### What changes were proposed in this pull request? This PR revisits the earlier SPARK-57926 implementation in [PR #57328](https://github.com/apache/spark/pull/57328) and adds four native Spark SQL functions for IPv4 text and numeric conversion: - `inet_aton(string) -> long`: converts IPv4 text to an unsigned 32-bit value represented as `LONG`. - `inet_ntoa(long) -> string`: converts an unsigned 32-bit value represented as `LONG` to canonical four-part IPv4 text. - `try_inet_aton(string) -> long`: the non-throwing variant of `inet_aton`. - `try_inet_ntoa(long) -> string`: the non-throwing variant of `inet_ntoa`. The functions are implemented as Catalyst-native expressions with interpreted and whole-stage codegen paths. The `try_*` functions use Spark's existing non-throwing expression pattern and do not duplicate the IPv4 parser. The functions are registered and exposed through SQL, the Scala API, PySpark Classic, PySpark Connect, and the generic Spark Connect function-expression path. No new Connect protobuf field is required. The IPv4 contract is explicit: - One to four dot-separated parts are accepted. - Each part must contain decimal digits and have a value in `[0, 255]`. - Leading zeroes are interpreted as decimal digits. - Short forms map as `a -> 0.0.0.a`, `a.b -> a.0.0.b`, and `a.b.c -> a.b.0.c`. - Empty parts, whitespace, signs, non-ASCII characters, more than four parts, and out-of-range parts are rejected. - `inet_ntoa` accepts values in `[0, 4294967295]` and always returns canonical four-part text. - `inet_aton` and `inet_ntoa` follow Spark's ANSI behavior; the `try_*` variants always return `NULL` for invalid content or out-of-range values. The documented argument types are `STRING` and `LONG`, while Spark's existing implicit input-type coercion remains applicable. For example, `inet_aton(1.5D)` is converted through `STRING` and returns `16777221`, while `inet_ntoa(1.5D)` is converted through `BIGINT` and returns `0.0.0.1`. Cast failures retain Spark's `CAST_INVALID_INPUT` precedence rather than being reported as IPv4 range errors. ### Why are the changes needed? Spark SQL currently has no built-in IPv4 conversion functions. Users need UDFs or application-side conversion for common network log analysis, security auditing, IP range filtering, and geolocation workloads. These operations are scalar expressions used in projections, filters, conditional expressions, and joins. Native functions keep the operation visible to analysis and whole-stage codegen, provide Spark-native ANSI and `try_*` behavior, and make the same capability available across SQL, Scala, PySpark, and Spark Connect without requiring users to maintain UDFs. The function names follow established SQL conventions, while the behavior is defined by the Spark contract above. ### Does this PR introduce _any_ user-facing change? Yes. Spark SQL and the corresponding Scala, PySpark Classic, and PySpark Connect APIs gain four new IPv4 conversion functions. For example: ```sql SELECT inet_aton('192.168.1.1'); -- 3232235777 SELECT inet_aton('127.1'); -- 2130706433 SELECT inet_ntoa(3232235777); -- 192.168.1.1 SELECT try_inet_aton('not_an_ip'); -- NULL ``` Invalid IPv4 text returns `NULL` from `inet_aton` when ANSI mode is disabled and raises a structured error when ANSI mode is enabled. Values outside the valid IPv4 range behave analogously for `inet_ntoa`. The `try_*` functions return `NULL` regardless of ANSI mode. The new error classes distinguish invalid IPv4 text (`INVALID_IPV4_ADDRESS`, SQLSTATE `22P02`) from numeric range violations (`INVALID_IPV4_LONG`, SQLSTATE `22003`). ### How was this patch tested? - `IpExpressionsSuite`: 30 tests covering valid and invalid addresses, short forms, decimal leading zeroes, boundaries, non-ASCII input, ANSI behavior, implicit casts, foldability, interpreted evaluation, and whole-stage codegen. - `SQLQueryTestSuite`: four generated and validated SQL analyzer/result golden tests covering ANSI and non-ANSI behavior, `NULL` handling, cast precedence, ordinary and `try_*` functions, and boundaries. The SQL golden files were generated with Spark's official tooling. - `ExpressionsSchemaSuite`: one test passed after regenerating `sql-expression-schema.md` with the official generator. - Spark Connect plan and explain goldens: four plan-generation tests and four parsed-plan tests passed, with the JSON, protobuf, and explain assets generated by the official test tools. - Hive-profile build: `./build/sbt -no-share -batch -Phive package` passed. - PySpark Classic: IPv4 execution and API-parity tests passed. - PySpark Connect: real IPv4 execution and API-parity tests passed. - PySpark doctests: both `pyspark.sql.functions.builtin` and `pyspark.sql.connect.functions.builtin` doctest modules passed with absolute Python interpreter paths. - Static validation: affected scalafmt and scalastyle checks, targeted Ruff checks, Python compilation, JSON validation, source line-length checks, non-ASCII checks, and `git diff --check` passed. License validation passed for project files; RAT's only report was the worktree-only `.git` pointer. - `spark-pr-precheck.py --skip-scalafmt` completed all applicable checks. Its only remaining report is the known `check_string_consistency` false positive when one change set adds multiple valid function names. The full repository-wide SQL suite was not run; the validation above covers the affected Catalyst, SQL, Connect, PySpark, documentation, and generated-asset paths. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Codex and Claude AI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
