[
https://issues.apache.org/jira/browse/SPARK-60661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
DB Tsai updated SPARK-60661:
----------------------------
Fix Version/s: connect-rust-4.3.0
Affects Version/s: connect-rust-4.3.0
Description:
Current scope after the master rebase
In pandas row conversion and other generic callers of py_to_value, exact
built-in int and float values still pass through generic Decimal and
numeric-detection work. Dispatch these exact types directly while retaining the
existing fallback for other objects.
The remaining scope is the pandas row path and other py_to_value callers. The
rewritten local createDataFrame inference and conversion path already
dispatches built-in numbers directly, so this issue does not restore or
optimize that older local-input path.
Preserve behavior for bools, subclasses, Decimal, custom numeric protocols,
overflowing integers, nested values, and special floating-point values,
including NaN, infinities, and signed zero. Keep current local-input inference
and error behavior intact.
Regression tests should use object-backed pandas inputs where needed to
exercise the remaining generic path. Rebenchmark the current patch against
current master before claiming CPU, wall-time, or memory improvements; the
historical local tuple and Row results do not validate the rebased scope.
Live-Spark parity and current CI remain part of validation.
Proposed patch: https://github.com/apache/spark-connect-rust/pull/111
Historical pre-rebase description and validation
The text below records the original proposal and its historical measurements
and validation. It does not validate or measure the rebased patch. The current
scope above takes precedence.
Numeric Python values in spark-connect-rust currently pass through generic
bool/Decimal detection in py_to_value. Avoid this overhead for exact built-in
int and float values by using cast_exact before the generic path. Integers that
fit i64 convert directly to Value::Long; exact floats convert to Value::Double.
Booleans, subclasses, custom conversion protocols, and overflowing integers
retain the existing fallback behavior. No behavior or API change is intended.
Add regression coverage for values and types, float bit preservation (including
NaN payloads and signed zero), subclasses, Decimal, conversion protocols, and
nested inputs. Integer overflow coercion remains a separate existing PySpark
parity gap.
Original-patch confirmation medians versus c36c846 on macOS arm64 / Python 3.12
showed CPU reductions of 74.39% for 10,000 x 32 tuples, 41.92% for 10,000 x 3
tuples, and 37.80% for numeric arrays. These measurements include conversion
and IPC serialization. The cast_exact cleanup has not been rebenchmarked; full
CPU/memory gates remain unmet because of known SPARK-60607 failures and
inconclusive regression bounds.
The PR reports 289 offline Python tests passing with an unchanged 108-class API
snapshot, plus 410 Rust unit/golden/sharing checks. Full live
Spark/official-suite parity remains pending.
Pull request: https://github.com/apache/spark-connect-rust/pull/111
Frozen benchmark harness:
https://github.com/dbtsai/pyspark-connect-benchmark/tree/23bdf9046815ae615c921b2e6bab827eed00aaae
was:
Numeric Python values in spark-connect-rust currently pass through generic
bool/Decimal detection in py_to_value. Avoid this overhead for exact built-in
int and float values by using cast_exact before the generic path. Integers that
fit i64 convert directly to Value::Long; exact floats convert to Value::Double.
Booleans, subclasses, custom conversion protocols, and overflowing integers
retain the existing fallback behavior. No behavior or API change is intended.
Add regression coverage for values and types, float bit preservation (including
NaN payloads and signed zero), subclasses, Decimal, conversion protocols, and
nested inputs. Integer overflow coercion remains a separate existing PySpark
parity gap.
Original-patch confirmation medians versus c36c846 on macOS arm64 / Python 3.12
showed CPU reductions of 74.39% for 10,000 x 32 tuples, 41.92% for 10,000 x 3
tuples, and 37.80% for numeric arrays. These measurements include conversion
and IPC serialization. The cast_exact cleanup has not been rebenchmarked; full
CPU/memory gates remain unmet because of known SPARK-60607 failures and
inconclusive regression bounds.
The PR reports 289 offline Python tests passing with an unchanged 108-class API
snapshot, plus 410 Rust unit/golden/sharing checks. Full live
Spark/official-suite parity remains pending.
Pull request: https://github.com/apache/spark-connect-rust/pull/111
Frozen benchmark harness:
https://github.com/dbtsai/pyspark-connect-benchmark/tree/23bdf9046815ae615c921b2e6bab827eed00aaae
Summary: Avoid generic detection for exact built-in numbers in
Spark Connect Rust (was: Avoid generic type detection for exact built-in
Python numbers in spark-connect-rust)
> Avoid generic detection for exact built-in numbers in Spark Connect Rust
> ------------------------------------------------------------------------
>
> Key: SPARK-60661
> URL: https://issues.apache.org/jira/browse/SPARK-60661
> Project: Spark
> Issue Type: Improvement
> Components: Connect, PySpark
> Affects Versions: connect-rust-4.2.0, connect-rust-4.3.0
> Reporter: DB Tsai
> Priority: Major
> Labels: pull-request-available
> Fix For: connect-rust-4.3.0
>
>
> Current scope after the master rebase
> In pandas row conversion and other generic callers of py_to_value, exact
> built-in int and float values still pass through generic Decimal and
> numeric-detection work. Dispatch these exact types directly while retaining
> the existing fallback for other objects.
> The remaining scope is the pandas row path and other py_to_value callers. The
> rewritten local createDataFrame inference and conversion path already
> dispatches built-in numbers directly, so this issue does not restore or
> optimize that older local-input path.
> Preserve behavior for bools, subclasses, Decimal, custom numeric protocols,
> overflowing integers, nested values, and special floating-point values,
> including NaN, infinities, and signed zero. Keep current local-input
> inference and error behavior intact.
> Regression tests should use object-backed pandas inputs where needed to
> exercise the remaining generic path. Rebenchmark the current patch against
> current master before claiming CPU, wall-time, or memory improvements; the
> historical local tuple and Row results do not validate the rebased scope.
> Live-Spark parity and current CI remain part of validation.
> Proposed patch: https://github.com/apache/spark-connect-rust/pull/111
> Historical pre-rebase description and validation
> The text below records the original proposal and its historical measurements
> and validation. It does not validate or measure the rebased patch. The
> current scope above takes precedence.
> Numeric Python values in spark-connect-rust currently pass through generic
> bool/Decimal detection in py_to_value. Avoid this overhead for exact built-in
> int and float values by using cast_exact before the generic path. Integers
> that fit i64 convert directly to Value::Long; exact floats convert to
> Value::Double. Booleans, subclasses, custom conversion protocols, and
> overflowing integers retain the existing fallback behavior. No behavior or
> API change is intended.
> Add regression coverage for values and types, float bit preservation
> (including NaN payloads and signed zero), subclasses, Decimal, conversion
> protocols, and nested inputs. Integer overflow coercion remains a separate
> existing PySpark parity gap.
> Original-patch confirmation medians versus c36c846 on macOS arm64 / Python
> 3.12 showed CPU reductions of 74.39% for 10,000 x 32 tuples, 41.92% for
> 10,000 x 3 tuples, and 37.80% for numeric arrays. These measurements include
> conversion and IPC serialization. The cast_exact cleanup has not been
> rebenchmarked; full CPU/memory gates remain unmet because of known
> SPARK-60607 failures and inconclusive regression bounds.
> The PR reports 289 offline Python tests passing with an unchanged 108-class
> API snapshot, plus 410 Rust unit/golden/sharing checks. Full live
> Spark/official-suite parity remains pending.
> Pull request: https://github.com/apache/spark-connect-rust/pull/111
> Frozen benchmark harness:
> https://github.com/dbtsai/pyspark-connect-benchmark/tree/23bdf9046815ae615c921b2e6bab827eed00aaae
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
