Subodh-17 opened a new pull request, #390:
URL: https://github.com/apache/datafusion-spark/pull/390

   <!--
   SPDX-License-Identifier: Apache-2.0
   SPDX-FileCopyrightText: See the NOTICE file distributed with this work for 
additional information regarding copyright ownership
   SPDX-FileContributor: Licensed to the Apache Software Foundation (ASF) under 
one or more contributor license agreements
   -->
   
   ## Which issue does this PR close?
   
   - Closes #365.
   
   ## Rationale for this change
   
   In Apache Spark, `parse_url` delegates component extraction to 
`java.net.URI`, which tokenizes the authority component without enforcing 
WHATWG 16-bit integer bounds (0 <= port <= 65535). Consequently, Spark 
successfully extracts requested components (such as `HOST`, `AUTHORITY`, 
`PATH`, `QUERY`, `REF`, and `PROTOCOL`) for URLs containing ports above 65535 
(e.g. `parse_url('http://spark.apache.org:99999/path', 'HOST')` -> 
`'spark.apache.org'`).
   
   Previously, `datafusion-spark` relied exclusively on Rust's 
`url::Url::parse`, which fails with `ParseError::InvalidPort` for any port 
above 65535, causing `parse_url` to return `NULL` in non-ANSI mode and raise an 
`InvalidPort` execution error in ANSI mode.
   
   ## What changes are included in this PR?
   
   - Adds a targeted port-normalization fallback in `ParseUrl::parse` activated 
strictly upon `Err(ParseError::InvalidPort)`:
     - Isolates the authority component while safely respecting userinfo 
(`user:pass@`), bracketed IPv6 addresses (`[::1]:99999`), and delimiters (`/`, 
`?`, `#`).
     - Rejects authorities containing whitespace, maintaining parity with JVM 
`URISyntaxException`.
     - Replicates Spark/JVM signed 32-bit port semantics: ports <= 2147483647 
extract host; ports > 2147483647 or non-numeric ports (`:abc`, `:-80`) return 
`NULL` for host while preserving raw authority, path, query, and ref.
     - Substitutes a valid placeholder port (`:80`) in a sanitized URL string 
for WHATWG component extraction, but injects the verbatim unmutated authority 
string when `part == "AUTHORITY"`.
     - Preserves empty-path semantics (`""` vs `"/"`) by checking `Self::path` 
against the original URL.
   - Valid URLs (0 <= port <= 65535, default ports, schemeless) continue to 
evaluate through the standard fast path with zero overhead.
   
   ## Are these changes tested?
   
   - Added unit tests in `crates/spark/src/function/url/parse_url.rs`:
     - `test_parse_url_port_overflow_and_spark_parity`: tests standard ports 
(8080, 80), boundary ports (65535, 65536), overflow ports (99999, 1000000), 
32-bit overflow ports (9999999999), empty ports, no ports, and non-numeric 
ports (`:abc`).
     - `test_parse_url_structures_with_overflow_port`: tests IPv4, bracketed 
IPv6, userinfo, encoded path characters, empty paths, and query-only paths with 
port 99999.
     - `test_parse_url_overflow_port_eval_modes`: tests legacy, try, and ANSI 
evaluation modes.
   - Updated `crates/sqllogictest/test_files/url/parse_url.slt`:
     - Removed `# xfail spark: ... (#365)` comments on lines 339 and 364 per 
runner guidelines.
     - Updated expected result to `'spark.apache.org'` for both legacy and ANSI 
modes.
     - Added SLT test queries for `AUTHORITY`, `PATH`, `QUERY`, `REF`, 
`PROTOCOL`, and bracketed IPv6 with port 99999.
   - Verified expected outputs against live JVM `java.net.URI` execution and 
PySpark 4.2.0.
   
   ## Are there any user-facing changes?
   
   Yes. `parse_url` no longer rejects URLs containing ports above 65535, 
matching Apache Spark behavior.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to