Subodh-17 opened a new pull request, #390:
URL: https://github.com/apache/datafusion-spark/pull/390
<!--
SPDX-License-Identifier: Apache-2.0
SPDX-FileCopyrightText: See the NOTICE file distributed with this work for
additional information regarding copyright ownership
SPDX-FileContributor: Licensed to the Apache Software Foundation (ASF) under
one or more contributor license agreements
-->
## Which issue does this PR close?
- Closes #365.
## Rationale for this change
In Apache Spark, `parse_url` delegates component extraction to
`java.net.URI`, which tokenizes the authority component without enforcing
WHATWG 16-bit integer bounds (0 <= port <= 65535). Consequently, Spark
successfully extracts requested components (such as `HOST`, `AUTHORITY`,
`PATH`, `QUERY`, `REF`, and `PROTOCOL`) for URLs containing ports above 65535
(e.g. `parse_url('http://spark.apache.org:99999/path', 'HOST')` ->
`'spark.apache.org'`).
Previously, `datafusion-spark` relied exclusively on Rust's
`url::Url::parse`, which fails with `ParseError::InvalidPort` for any port
above 65535, causing `parse_url` to return `NULL` in non-ANSI mode and raise an
`InvalidPort` execution error in ANSI mode.
## What changes are included in this PR?
- Adds a targeted port-normalization fallback in `ParseUrl::parse` activated
strictly upon `Err(ParseError::InvalidPort)`:
- Isolates the authority component while safely respecting userinfo
(`user:pass@`), bracketed IPv6 addresses (`[::1]:99999`), and delimiters (`/`,
`?`, `#`).
- Rejects authorities containing whitespace, maintaining parity with JVM
`URISyntaxException`.
- Replicates Spark/JVM signed 32-bit port semantics: ports <= 2147483647
extract host; ports > 2147483647 or non-numeric ports (`:abc`, `:-80`) return
`NULL` for host while preserving raw authority, path, query, and ref.
- Substitutes a valid placeholder port (`:80`) in a sanitized URL string
for WHATWG component extraction, but injects the verbatim unmutated authority
string when `part == "AUTHORITY"`.
- Preserves empty-path semantics (`""` vs `"/"`) by checking `Self::path`
against the original URL.
- Valid URLs (0 <= port <= 65535, default ports, schemeless) continue to
evaluate through the standard fast path with zero overhead.
## Are these changes tested?
- Added unit tests in `crates/spark/src/function/url/parse_url.rs`:
- `test_parse_url_port_overflow_and_spark_parity`: tests standard ports
(8080, 80), boundary ports (65535, 65536), overflow ports (99999, 1000000),
32-bit overflow ports (9999999999), empty ports, no ports, and non-numeric
ports (`:abc`).
- `test_parse_url_structures_with_overflow_port`: tests IPv4, bracketed
IPv6, userinfo, encoded path characters, empty paths, and query-only paths with
port 99999.
- `test_parse_url_overflow_port_eval_modes`: tests legacy, try, and ANSI
evaluation modes.
- Updated `crates/sqllogictest/test_files/url/parse_url.slt`:
- Removed `# xfail spark: ... (#365)` comments on lines 339 and 364 per
runner guidelines.
- Updated expected result to `'spark.apache.org'` for both legacy and ANSI
modes.
- Added SLT test queries for `AUTHORITY`, `PATH`, `QUERY`, `REF`,
`PROTOCOL`, and bracketed IPv6 with port 99999.
- Verified expected outputs against live JVM `java.net.URI` execution and
PySpark 4.2.0.
## Are there any user-facing changes?
Yes. `parse_url` no longer rejects URLs containing ports above 65535,
matching Apache Spark behavior.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]