viirya commented on PR #57104: URL: https://github.com/apache/spark/pull/57104#issuecomment-4928769189
Closing this PR. Rationale: During review of the sibling PRs, concerns were raised about the maintainability of routing data values through NumPy (behavior across the supported arrow/numpy version matrix, and keeping the "no type coercion" invariant guarded by tests over time). While the conversions here are exactness-tested — the object-dtype conversion for strings/binary can only produce `str`/`bytes`/`None`, and nullable numerics are filled and restored from the validity bitmap rather than going through a float representation — this is also precisely the layer that apache/arrow#50327 implements properly in C, planned for PyArrow 25.0.1 per the Arrow dev-list discussion. With the version gate added in SPARK-58019 (#57099), Spark automatically uses PyArrow's native conversion once the installed PyArrow contains that fix — so the leaf-level speedup reaches users through a PyArrow upgrade rather than through NumPy-based code maintained here. The remaining Spark-side PRs are scoped to what Arrow cannot do for Spark: SPARK-58019/SPARK-58024 (bulk offsets/validity slicing; no data values through NumPy) and SPARK-58050 (bulk result assembly; no NumPy at all). For the record: this PR gave flat string 196ms→20ms and int64 99ms→28ms per 1M rows, and its removal lowers the struct/map gains in SPARK-58024 from 5.2x/6.8x to 2.0x/2.7x until the PyArrow fix ships. We accept that interim gap in exchange for the smaller maintenance surface. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
