HyukjinKwon opened a new pull request, #256: URL: https://github.com/apache/spark-connect-rust/pull/256
### What changes were proposed in this pull request? - `pyspark.sql.functions.__all__` is now official PySpark 4.3.0's static list of 562 names, in its order, so `from pyspark.sql.functions import *` no longer exports `chr`, `random`, `uuid`, `countDistinct`, `bitwiseNOT`, `shiftLeft`, `shiftRight`, `shiftRightUnsigned`, `toDegrees`, `toRadians`, `sumDistinct`, `approxCountDistinct` and the avro and protobuf functions; those that official has stay attributes. The public attributes of the module are official's: `from_avro`, `to_avro`, `from_protobuf` and `to_protobuf` move to `pyspark.sql.avro.functions` and `pyspark.sql.protobuf.functions`, with official's signatures and argument checks (`from_avro_with_options` and `to_avro_with_schema` fold into `options=` and `jsonFormatSchema=`), and `descFilePath` is read as in official. - `lit()`, and Column operations with a value, type a NumPy scalar by its dtype, as official's `_from_numpy_type` does: `int8`/`int16`/`int32`/`int64` as tinyint/smallint/int/bigint, `float32` as float, `float64` as double, `bool_` as boolean; unsigned ints, `float16`, `longdouble` and complex numbers raise `UNSUPPORTED_DATA_TYPE` (`crates/pyspark-rs/src/functions.rs`). ### Why are the changes needed? [SPARK-61065](https://issues.apache.org/jira/browse/SPARK-61065): the star import, common in notebooks, shadowed builtins: | Code | Current master | This change and PySpark 4.3.0 | | --- | --- | --- | | `from pyspark.sql.functions import *; chr(65)` | `PySparkTypeError [NOT_EXPECTED_TYPE]` | `'A'` | | `import random; from pyspark.sql.functions import *; random.randint(1, 1)` | `AttributeError` | `1` | | `hasattr(F, "from_avro")` | `True` | `False` | | `spark.range(1).select(F.lit(np.int64(5))).schema` | `struct<5:int>` | `struct<5:bigint>` | | `F.lit(np.float32(1.5))` | double | float | | `F.lit(np.uint8(5))` | int | `UNSUPPORTED_DATA_TYPE` `uint8` | ### Does this PR introduce _any_ user-facing change? Yes. The star import and the module's attributes are official's, the avro and protobuf functions are on their official modules, and NumPy scalars are typed as in PySpark. ### How was this patch tested? - New offline tests in `python/tests/test_dropin_offline.py` for the exports, the avro and protobuf modules, and the literal types of each NumPy scalar (from the plan), and a live test in `python/tests/test_dropin_e2e.py`; `test_functions_exports_fresh_process` now compares with official's list. - The offline suite (961 passed, 8 skipped); about 40 `lit` cases match official's types, column names and errors. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Opus 5.5 This pull request and its description were written by Isaac. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
