This is an automated email from the ASF dual-hosted git repository.

HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git


The following commit(s) were added to refs/heads/master by this push:
     new 198d042  [MINOR][DOCS] Remove the type inference of createDataFrame 
from the known differences
198d042 is described below

commit 198d042b77dcfb7c9d66887262e3fcdb8c1760d1
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Sat Nov 7 05:42:33 2026 +0900

    [MINOR][DOCS] Remove the type inference of createDataFrame from the known 
differences
    
    ### What changes were proposed in this pull request?
    
    This PR removes the type inference of `createDataFrame` across rows from 
`dev/parity/known-differences.md`. The entry was tracked by SPARK-60908, which 
made the inference match PySpark Connect 4.3.0.
    
    ### Why are the changes needed?
    
    The entry described the behavior before SPARK-60908. On a local Spark 4.3.0 
server, `createDataFrame` now infers, and fails for, the cases that the entry 
listed as official PySpark 4.3.0 does:
    - other atomic types merge into a string type
    - nested values null in every row raise `CANNOT_DETERMINE_TYPE`
    - other conflicts raise `CANNOT_MERGE_TYPE`
    - a `Decimal` infers `decimal(38,18)`
    
    ### Does this PR introduce _any_ user-facing change?
    
    No. It changes a maintainer note.
    
    ### How was this patch tested?
    
    On a local Spark 4.3.0 server, 7 cases of the inference give the same 
schemas and errors as official PySpark 4.3.0.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Opus 5.5
    
    This pull request and its description were written by Isaac.
    
    Closes #230 from HyukjinKwon/minor-known-differences.
    
    Authored-by: Hyukjin Kwon <[email protected]>
    Signed-off-by: Hyukjin Kwon <[email protected]>
---
 dev/parity/known-differences.md | 6 ------
 1 file changed, 6 deletions(-)

diff --git a/dev/parity/known-differences.md b/dev/parity/known-differences.md
index a07fada..4f34958 100644
--- a/dev/parity/known-differences.md
+++ b/dev/parity/known-differences.md
@@ -11,10 +11,4 @@ difference is intentional or tracked for a fix.
   fewer fields than columns it silently drops the extra columns, and with more 
fields it
   raises PyArrow's `ArrowInvalid`. This client raises `AXIS_LENGTH_MISMATCH` 
for both
   rather than drop data.
-- **Type inference across rows** (tracked by SPARK-60908). PySpark infers each 
row's type
-  and merges them: other atomic types merge into a string type, whose values 
they become;
-  nested values null in every row raise `CANNOT_DETERMINE_TYPE`; other 
conflicts raise
-  `CANNOT_MERGE_TYPE` or `CANNOT_INFER_TYPE_FOR_FIELD`; and a `Decimal` value 
infers
-  `decimal(38,18)`. This client takes each top-level field's type from its 
first non-null
-  value, and a nested type from the first non-null element or map value.
 - **NumPy ndarray input** (tracked by SPARK-60772). Not accepted yet.


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to