This is an automated email from the ASF dual-hosted git repository.

HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git


The following commit(s) were added to refs/heads/master by this push:
     new bdb076f  [SPARK-61133][CONNECT][PYTHON] Add the private helpers of 
pyspark.sql.types, such as _infer_schema, _merge_type and _make_type_verifier, 
in the Python client as PySpark has them
bdb076f is described below

commit bdb076f3ba680318772346a4849883f72bdc5353
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Sun Nov 8 07:55:13 2026 +0900

    [SPARK-61133][CONNECT][PYTHON] Add the private helpers of 
pyspark.sql.types, such as _infer_schema, _merge_type and _make_type_verifier, 
in the Python client as PySpark has them
    
    ### What changes were proposed in this pull request?
    
    [SPARK-61133](https://issues.apache.org/jira/browse/SPARK-61133): 
`pyspark.sql.types` of the client gains the private helpers of PySpark's, as of 
Apache Spark v4.3.0:
    - the inference, merging, conversion and verification of the types of 
Python objects: `_infer_type`, `_infer_schema`, `_merge_type`, `_has_nulltype`, 
`_need_converter`, `_create_converter`, `_make_type_verifier`, 
`_create_row_inbound_converter` and `_int_size_to_type`
    - their tables: `_type_mappings`, `_array_type_mappings`, 
`_array_signed_int_typecode_ctype_mappings`, 
`_array_unsigned_int_typecode_ctype_mappings`, `_acceptable_types`, 
`_atomic_types`, `_complex_types`, `_all_complex_types`, `_all_mappable_types` 
and `JVM_INT_MAX`
    
    They are PySpark's code, since the client's types share the classes and the 
attributes of PySpark's. There is one change: the client's `Row` is not a 
tuple. So wherever PySpark takes a tuple of a `Row`, these helpers take the 
client's `Row` too, in five places:
    - `_infer_schema`
    - the converter of a struct in `_create_converter`
    - the acceptable types of a struct and of a variant
    - the verifier of a struct
    
    The client's own `createDataFrame` still infers and converts in Rust, as 
before.
    
    ### Why are the changes needed?
    
    PySpark Connect's `session.py` imports `_infer_schema`, `_has_nulltype` and 
`_merge_type`. Apache Spark's `test_types.py` imports `_infer_type`, 
`_make_type_verifier`, `_merge_type` and the typecode tables, as do some 
libraries. On this client, `from pyspark.sql.types import _merge_type` raised 
`ImportError`, so Apache Spark's `test_parity_types.py` could not even be 
collected.
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes. These helpers can be imported from `pyspark.sql.types`, and behave as 
PySpark's.
    
    ### How was this patch tested?
    
    - New offline tests check:
      - the helpers and their tables
      - the verifiers of each type, with the success and failure cases of 
Apache Spark's `DataTypeVerificationTests`, also of the client's `Row`, and 
their errors with the names of the fields
      - the inference of each type of value, also nested and of the options, 
the merging of types, and the converters
    
      They pass with official PySpark 4.3.0 and fail without this change. With 
them, the coverage of the offline suites stays at 83%.
    - Apache Spark's `test_parity_types.py` now runs on this client against a 
local server. 61 tests pass and 21 are skipped. Of the 9 that fail, one fails 
with official PySpark too. The other 8 are gaps of the types that the file 
could not show before, now filed:
      - SPARK-61134: the fields of the interval types, and the arguments of the 
types without parameters
      - SPARK-61135: the JSON of `GeometryType` and `GeographyType`
      - SPARK-61136: the decimals of a negative scale
      - SPARK-61137: the `StructField` objects of a `StructType`
      - SPARK-61138: `str` of a `VariantVal`, and rows that are `VariantVal`s
    - The lib tests of both crates (590 passed), the build of all tests, also 
with `wasm-udf`, the offline Python suites (2198 passed, 8 skipped), and the 
live drop-in and Arrow suites (181 passed) and live cases (10 passed) against a 
local Spark 4.3.0 server. Clippy reports no diagnostics in the changed lines.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Opus 5.5
    
    This pull request and its description were written by Isaac.
    
    Closes #326 from HyukjinKwon/SPARK-61133-types-helpers.
    
    Authored-by: Hyukjin Kwon <[email protected]>
    Signed-off-by: Hyukjin Kwon <[email protected]>
---
 python/pyspark/sql/types.py         | 989 ++++++++++++++++++++++++++++++++++++
 python/tests/test_dropin_offline.py | 210 ++++++++
 2 files changed, 1199 insertions(+)

diff --git a/python/pyspark/sql/types.py b/python/pyspark/sql/types.py
index cced81b..75b4b7b 100644
--- a/python/pyspark/sql/types.py
+++ b/python/pyspark/sql/types.py
@@ -747,3 +747,992 @@ def _from_internal(dataType, obj):
         return _create_row(dataType.names, values)
     else:
         return obj
+
+
+# The private helpers of PySpark's pyspark.sql.types, as of Apache Spark 
v4.3.0: the inference,
+# merging, conversion and verification of the types of Python objects, and 
their tables, which
+# PySpark Connect's modules, Apache Spark's tests and libraries import. They 
are PySpark's, as
+# the client's types share the classes and the attributes of PySpark's; but 
the client's Row is
+# not a tuple, so where PySpark takes a tuple of a Row, they take the client's 
Row too.
+import ctypes  # noqa: E402
+import datetime  # noqa: E402
+import decimal  # noqa: E402
+import sys  # noqa: E402
+from array import array  # noqa: E402
+from collections.abc import Iterable  # noqa: E402
+from functools import reduce  # noqa: E402
+from typing import Any, Callable, Dict, List, Optional, Tuple, Type, Union, 
overload  # noqa: E402
+
+from pyspark.errors import PySparkTypeError, PySparkValueError  # noqa: E402
+from pyspark.util import JVM_INT_MAX  # noqa: E402, F401
+
+_atomic_types: List[Type[DataType]] = [
+    StringType,
+    CharType,
+    VarcharType,
+    BinaryType,
+    BooleanType,
+    DecimalType,
+    FloatType,
+    DoubleType,
+    ByteType,
+    ShortType,
+    IntegerType,
+    LongType,
+    DateType,
+    TimestampType,
+    TimestampNTZType,
+    NullType,
+    VariantType,
+    YearMonthIntervalType,
+    DayTimeIntervalType,
+]
+
+_complex_types: List[Type[Union[ArrayType, MapType, StructType]]] = [
+    ArrayType,
+    MapType,
+    StructType,
+]
+_all_complex_types: Dict[str, Type[Union[ArrayType, MapType, StructType]]] = {
+    "array": ArrayType,
+    "map": MapType,
+    "struct": StructType,
+}
+
+# Datatypes that can be directly parsed by mapping a json string without regex.
+# This dict should be only used in json parsing.
+# Note that:
+# 1, CharType and VarcharType are not listed here, since they need regex;
+# 2, DecimalType can be parsed by both mapping ('decimal') and regex 
('decimal(10, 2)');
+# 3, CalendarIntervalType is not an atomic type, but can be mapped by 
'interval';
+_all_mappable_types: Dict[str, Type[DataType]] = {
+    "string": StringType,
+    "binary": BinaryType,
+    "boolean": BooleanType,
+    "decimal": DecimalType,
+    "float": FloatType,
+    "double": DoubleType,
+    "byte": ByteType,
+    "short": ShortType,
+    "integer": IntegerType,
+    "long": LongType,
+    "date": DateType,
+    "timestamp": TimestampType,
+    "timestamp_ntz": TimestampNTZType,
+    "void": NullType,
+    "variant": VariantType,
+    "interval": CalendarIntervalType,
+}
+
+_type_mappings = {
+    type(None): NullType,
+    Geometry: GeometryType,
+    Geography: GeographyType,
+    bool: BooleanType,
+    int: LongType,
+    float: DoubleType,
+    str: StringType,
+    bytearray: BinaryType,
+    decimal.Decimal: DecimalType,
+    datetime.date: DateType,
+    datetime.datetime: TimestampType,  # can be TimestampNTZType
+    datetime.time: TimeType,
+    datetime.timedelta: DayTimeIntervalType,
+    bytes: BinaryType,
+}
+
+# Mapping Python array types to Spark SQL DataType
+# We should be careful here. The size of these types in python depends on C
+# implementation. We need to make sure that this conversion does not lose any
+# precision. Also, JVM only support signed types, when converting unsigned 
types,
+# keep in mind that it require 1 more bit when stored as signed types.
+#
+# Reference for C integer size, see:
+# ISO/IEC 9899:201x specification, chapter 5.2.4.2.1 Sizes of integer types 
<limits.h>.
+# Reference for python array typecode, see:
+# https://docs.python.org/2/library/array.html
+# https://docs.python.org/3.6/library/array.html
+# Reference for JVM's supported integral types:
+# http://docs.oracle.com/javase/specs/jvms/se8/html/jvms-2.html#jvms-2.3.1
+
+_array_signed_int_typecode_ctype_mappings = {
+    "b": ctypes.c_byte,
+    "h": ctypes.c_short,
+    "i": ctypes.c_int,
+    "l": ctypes.c_long,
+}
+
+_array_unsigned_int_typecode_ctype_mappings = {
+    "B": ctypes.c_ubyte,
+    "H": ctypes.c_ushort,
+    "I": ctypes.c_uint,
+    "L": ctypes.c_ulong,
+}
+
+
+def _int_size_to_type(
+    size: int,
+) -> Optional[Union[Type[ByteType], Type[ShortType], Type[IntegerType], 
Type[LongType]]]:
+    """
+    Return the Catalyst datatype from the size of integers.
+    """
+    if size <= 8:
+        return ByteType
+    elif size <= 16:
+        return ShortType
+    elif size <= 32:
+        return IntegerType
+    elif size <= 64:
+        return LongType
+    else:
+        return None
+
+
+# The list of all supported array typecodes, is stored here
+_array_type_mappings: Dict[str, Type[DataType]] = {
+    # Warning: Actual properties for float and double in C is not specified in 
C.
+    # On almost every system supported by both python and JVM, they are IEEE 
754
+    # single-precision binary floating-point format and IEEE 754 
double-precision
+    # binary floating-point format. And we do assume the same thing here for 
now.
+    "f": FloatType,
+    "d": DoubleType,
+}
+
+# compute array typecode mappings for signed integer types
+for _typecode in _array_signed_int_typecode_ctype_mappings:
+    size = ctypes.sizeof(_array_signed_int_typecode_ctype_mappings[_typecode]) 
* 8
+    dt = _int_size_to_type(size)
+    if dt is not None:
+        _array_type_mappings[_typecode] = dt
+
+# compute array typecode mappings for unsigned integer types
+for _typecode in _array_unsigned_int_typecode_ctype_mappings:
+    # JVM does not have unsigned types, so use signed types that is at least 1
+    # bit larger to store
+    size = 
ctypes.sizeof(_array_unsigned_int_typecode_ctype_mappings[_typecode]) * 8 + 1
+    dt = _int_size_to_type(size)
+    if dt is not None:
+        _array_type_mappings[_typecode] = dt
+
+# Type code 'u' in Python's array is deprecated since version 3.3, and will be
+# removed in version 4.0. See: https://docs.python.org/3/library/array.html
+if sys.version_info[0] < 4:
+    _array_type_mappings["u"] = StringType
+
+
+def _infer_type(
+    obj: Any,
+    infer_dict_as_struct: bool = False,
+    infer_array_from_first_element: bool = False,
+    infer_map_from_first_pair: bool = False,
+    prefer_timestamp_ntz: bool = False,
+) -> DataType:
+    """Infer the DataType from obj"""
+    if obj is None:
+        return NullType()
+
+    if hasattr(obj, "__UDT__"):
+        return obj.__UDT__
+
+    dataType = _type_mappings.get(type(obj))
+    if dataType is GeographyType:
+        assert isinstance(obj, Geography)
+        return GeographyType(obj.getSrid())
+    if dataType is GeometryType:
+        assert isinstance(obj, Geometry)
+        return GeometryType(obj.getSrid())
+    if dataType is DecimalType:
+        # the precision and scale of `obj` may be different from row to row.
+        return DecimalType(38, 18)
+    if dataType is TimestampType and prefer_timestamp_ntz and obj.tzinfo is 
None:
+        return TimestampNTZType()
+    if dataType is DayTimeIntervalType:
+        return DayTimeIntervalType()
+    if dataType is YearMonthIntervalType:
+        return YearMonthIntervalType()
+    if dataType is CalendarIntervalType:
+        return CalendarIntervalType()
+    elif dataType is not None:
+        return dataType()
+
+    if isinstance(obj, dict):
+        if infer_dict_as_struct:
+            struct = StructType()
+            for key, value in obj.items():
+                if key is not None and value is not None:
+                    struct.add(
+                        key,
+                        _infer_type(
+                            value,
+                            infer_dict_as_struct,
+                            infer_array_from_first_element,
+                            infer_map_from_first_pair,
+                            prefer_timestamp_ntz,
+                        ),
+                        True,
+                    )
+            return struct
+        elif infer_map_from_first_pair:
+            for key, value in obj.items():
+                if key is not None and value is not None:
+                    return MapType(
+                        _infer_type(
+                            key,
+                            infer_dict_as_struct,
+                            infer_array_from_first_element,
+                            infer_map_from_first_pair,
+                            prefer_timestamp_ntz,
+                        ),
+                        _infer_type(
+                            value,
+                            infer_dict_as_struct,
+                            infer_array_from_first_element,
+                            infer_map_from_first_pair,
+                            prefer_timestamp_ntz,
+                        ),
+                        True,
+                    )
+            return MapType(NullType(), NullType(), True)
+        else:
+            key_type: DataType = NullType()
+            value_type: DataType = NullType()
+            for key, value in obj.items():
+                if key is not None:
+                    key_type = _merge_type(
+                        key_type,
+                        _infer_type(
+                            key,
+                            infer_dict_as_struct,
+                            infer_array_from_first_element,
+                            infer_map_from_first_pair,
+                            prefer_timestamp_ntz,
+                        ),
+                    )
+                if value is not None:
+                    value_type = _merge_type(
+                        value_type,
+                        _infer_type(
+                            value,
+                            infer_dict_as_struct,
+                            infer_array_from_first_element,
+                            infer_map_from_first_pair,
+                            prefer_timestamp_ntz,
+                        ),
+                    )
+
+            return MapType(key_type, value_type, True)
+    elif isinstance(obj, list):
+        if len(obj) > 0:
+            if infer_array_from_first_element:
+                return ArrayType(
+                    _infer_type(
+                        obj[0],
+                        infer_dict_as_struct,
+                        infer_array_from_first_element,
+                        prefer_timestamp_ntz,
+                    ),
+                    True,
+                )
+            else:
+                return ArrayType(
+                    reduce(
+                        _merge_type,
+                        (
+                            _infer_type(
+                                v,
+                                infer_dict_as_struct,
+                                infer_array_from_first_element,
+                                prefer_timestamp_ntz,
+                            )
+                            for v in obj
+                        ),
+                    ),
+                    True,
+                )
+        return ArrayType(NullType(), True)
+    elif isinstance(obj, array):
+        if obj.typecode in _array_type_mappings:
+            return ArrayType(_array_type_mappings[obj.typecode](), False)
+        else:
+            raise PySparkTypeError(
+                errorClass="UNSUPPORTED_DATA_TYPE",
+                messageParameters={"data_type": f"array({obj.typecode})"},
+            )
+    elif isinstance(obj, VariantVal):
+        return VariantType()
+    else:
+        try:
+            return _infer_schema(
+                obj,
+                infer_dict_as_struct=infer_dict_as_struct,
+                infer_array_from_first_element=infer_array_from_first_element,
+                prefer_timestamp_ntz=prefer_timestamp_ntz,
+            )
+        except TypeError:
+            raise PySparkTypeError(
+                errorClass="UNSUPPORTED_DATA_TYPE",
+                messageParameters={"data_type": type(obj).__name__},
+            )
+
+
+def _infer_schema(
+    row: Any,
+    names: Optional[List[str]] = None,
+    infer_dict_as_struct: bool = False,
+    infer_array_from_first_element: bool = False,
+    infer_map_from_first_pair: bool = False,
+    prefer_timestamp_ntz: bool = False,
+) -> StructType:
+    """Infer the schema from dict/namedtuple/object"""
+    items: Iterable[Tuple[str, Any]]
+    if isinstance(row, dict):
+        items = sorted(row.items())
+
+    elif isinstance(row, (tuple, list, Row)):
+        if hasattr(row, "__fields__"):  # Row
+            items = zip(row.__fields__, tuple(row))
+        elif hasattr(row, "_fields"):  # namedtuple
+            items = zip(row._fields, tuple(row))
+        else:
+            if names is None:
+                names = ["_%d" % i for i in range(1, len(row) + 1)]
+            elif len(names) < len(row):
+                names.extend("_%d" % i for i in range(len(names) + 1, len(row) 
+ 1))
+            items = zip(names, row)
+
+    elif hasattr(row, "__dict__"):  # object
+        items = sorted(row.__dict__.items())
+
+    else:
+        raise PySparkTypeError(
+            errorClass="CANNOT_INFER_SCHEMA_FOR_TYPE",
+            messageParameters={"data_type": type(row).__name__},
+        )
+
+    fields = []
+    for k, v in items:
+        try:
+            fields.append(
+                StructField(
+                    k,
+                    _infer_type(
+                        v,
+                        infer_dict_as_struct,
+                        infer_array_from_first_element,
+                        infer_map_from_first_pair,
+                        prefer_timestamp_ntz,
+                    ),
+                    True,
+                )
+            )
+        except TypeError:
+            raise PySparkTypeError(
+                errorClass="CANNOT_INFER_TYPE_FOR_FIELD",
+                messageParameters={"field_name": k},
+            )
+    return StructType(fields)
+
+
+def _has_nulltype(dt: DataType) -> bool:
+    """Return whether there is a NullType in `dt` or not"""
+    if isinstance(dt, StructType):
+        return any(_has_nulltype(f.dataType) for f in dt.fields)
+    elif isinstance(dt, ArrayType):
+        return _has_nulltype((dt.elementType))
+    elif isinstance(dt, MapType):
+        return _has_nulltype(dt.keyType) or _has_nulltype(dt.valueType)
+    else:
+        return isinstance(dt, NullType)
+
+
+def _merge_type(a: StructType, b: StructType, name: Optional[str] = None) -> 
StructType: ...
+
+
+@overload
+def _merge_type(a: ArrayType, b: ArrayType, name: Optional[str] = None) -> 
ArrayType: ...
+
+
+@overload
+def _merge_type(a: MapType, b: MapType, name: Optional[str] = None) -> 
MapType: ...
+
+
+@overload
+def _merge_type(a: DataType, b: DataType, name: Optional[str] = None) -> 
DataType: ...
+
+
+def _merge_type(
+    a: Union[StructType, ArrayType, MapType, DataType],
+    b: Union[StructType, ArrayType, MapType, DataType],
+    name: Optional[str] = None,
+) -> Union[StructType, ArrayType, MapType, DataType]:
+    if name is None:
+
+        def new_msg(msg: str) -> str:
+            return msg
+
+        def new_name(n: str) -> str:
+            return "field %s" % n
+
+    else:
+
+        def new_msg(msg: str) -> str:
+            return "%s: %s" % (name, msg)
+
+        def new_name(n: str) -> str:
+            return "field %s in %s" % (n, name)
+
+    if isinstance(a, NullType):
+        return b
+    elif isinstance(b, NullType):
+        return a
+    elif isinstance(a, TimestampType) and isinstance(b, TimestampNTZType):
+        return a
+    elif isinstance(a, TimestampNTZType) and isinstance(b, TimestampType):
+        return b
+    elif isinstance(a, GeometryType) and isinstance(b, GeometryType) and 
a.srid != b.srid:
+        return GeometryType("ANY")
+    elif isinstance(a, GeographyType) and isinstance(b, GeographyType) and 
a.srid != b.srid:
+        return GeographyType("ANY")
+    elif isinstance(a, AtomicType) and isinstance(b, StringType):
+        return b
+    elif isinstance(a, StringType) and isinstance(b, AtomicType):
+        return a
+    elif type(a) is not type(b):
+        # TODO: type cast (such as int -> long)
+        raise PySparkTypeError(
+            errorClass="CANNOT_MERGE_TYPE",
+            messageParameters={"data_type1": type(a).__name__, "data_type2": 
type(b).__name__},
+        )
+
+    # same type
+    if isinstance(a, StructType):
+        nfs = dict((f.name, f.dataType) for f in cast(StructType, b).fields)
+        fields = [
+            StructField(
+                f.name, _merge_type(f.dataType, nfs.get(f.name, NullType()), 
name=new_name(f.name))
+            )
+            for f in a.fields
+        ]
+        names = set([f.name for f in fields])
+        for n in nfs:
+            if n not in names:
+                fields.append(StructField(n, nfs[n]))
+        return StructType(fields)
+
+    elif isinstance(a, ArrayType):
+        return ArrayType(
+            _merge_type(
+                a.elementType, cast(ArrayType, b).elementType, name="element 
in array %s" % name
+            ),
+            True,
+        )
+
+    elif isinstance(a, MapType):
+        return MapType(
+            _merge_type(a.keyType, cast(MapType, b).keyType, name="key of map 
%s" % name),
+            _merge_type(a.valueType, cast(MapType, b).valueType, name="value 
of map %s" % name),
+            True,
+        )
+    else:
+        return a
+
+
+def _need_converter(dataType: DataType) -> bool:
+    if isinstance(dataType, StructType):
+        return True
+    elif isinstance(dataType, ArrayType):
+        return _need_converter(dataType.elementType)
+    elif isinstance(dataType, MapType):
+        return _need_converter(dataType.keyType) or 
_need_converter(dataType.valueType)
+    elif isinstance(dataType, NullType):
+        return True
+    elif isinstance(dataType, StringType):
+        # Coercion to StringType is allowed, e.g. dict -> str
+        return True
+    else:
+        return False
+
+
+def _create_converter(dataType: DataType) -> Callable:
+    """Create a converter to drop the names of fields in obj"""
+    if not _need_converter(dataType):
+        return lambda x: x
+
+    if isinstance(dataType, ArrayType):
+        conv = _create_converter(dataType.elementType)
+        return lambda row: [conv(v) for v in row] if row is not None else None
+
+    elif isinstance(dataType, MapType):
+        kconv = _create_converter(dataType.keyType)
+        vconv = _create_converter(dataType.valueType)
+        return lambda row: (
+            dict((kconv(k), vconv(v)) for k, v in row.items()) if row is not 
None else None
+        )
+
+    elif isinstance(dataType, NullType):
+        return lambda x: None
+
+    elif isinstance(dataType, StringType):
+
+        def convert_string(value: Any) -> Any:
+            if value is None:
+                return None
+            else:
+                if isinstance(value, bool):
+                    # To match the Classic behavior
+                    return str(value).lower()
+                else:
+                    # Coercion to StringType is allowed, e.g. dict -> str
+                    return str(value)
+
+        return convert_string
+
+    elif not isinstance(dataType, StructType):
+        return lambda x: x
+
+    # dataType must be StructType
+    names = [f.name for f in dataType.fields]
+    converters = [_create_converter(f.dataType) for f in dataType.fields]
+    convert_fields = any(_need_converter(f.dataType) for f in dataType.fields)
+
+    def convert_struct(obj: Any) -> Optional[Tuple]:
+        if obj is None:
+            return None
+
+        if isinstance(obj, (tuple, list, Row)):
+            if convert_fields:
+                return tuple(conv(v) for v, conv in zip(obj, converters))
+            else:
+                return tuple(obj)
+
+        if isinstance(obj, dict):
+            d = obj
+        elif hasattr(obj, "__dict__"):  # object
+            d = obj.__dict__
+        else:
+            raise PySparkTypeError(
+                errorClass="UNSUPPORTED_DATA_TYPE",
+                messageParameters={"data_type": type(obj).__name__},
+            )
+
+        if convert_fields:
+            return tuple([conv(d.get(name)) for name, conv in zip(names, 
converters)])
+        else:
+            return tuple([d.get(name) for name in names])
+
+    return convert_struct
+
+
+_acceptable_types = {
+    BooleanType: (bool,),
+    ByteType: (int,),
+    ShortType: (int,),
+    IntegerType: (int,),
+    LongType: (int,),
+    FloatType: (float,),
+    DoubleType: (float,),
+    DecimalType: (decimal.Decimal,),
+    StringType: (str,),
+    CharType: (str,),
+    VarcharType: (str,),
+    BinaryType: (bytearray, bytes),
+    DateType: (datetime.date, datetime.datetime),
+    TimeType: (datetime.time,),
+    TimestampType: (datetime.datetime,),
+    TimestampNTZType: (datetime.datetime,),
+    DayTimeIntervalType: (datetime.timedelta,),
+    ArrayType: (list, tuple, array),
+    MapType: (dict,),
+    StructType: (tuple, list, dict, Row),
+    GeometryType: (Geometry,),
+    GeographyType: (Geography,),
+    VariantType: (
+        bool,
+        int,
+        float,
+        decimal.Decimal,
+        str,
+        bytearray,
+        bytes,
+        datetime.date,
+        datetime.datetime,
+        datetime.timedelta,
+        tuple,
+        Row,
+        list,
+        dict,
+        array,
+    ),
+}
+
+
+def _make_type_verifier(
+    dataType: DataType,
+    nullable: bool = True,
+    name: Optional[str] = None,
+) -> Callable:
+    """
+    Make a verifier that checks the type of obj against dataType and raises a 
TypeError if they do
+    not match.
+
+    This verifier also checks the value of obj against datatype and raises a 
ValueError if it's not
+    within the allowed range, e.g. using 128 as ByteType will overflow. Note 
that, Python float is
+    not checked, so it will become infinity when cast to Java float, if it 
overflows.
+
+    Examples
+    --------
+    >>> _make_type_verifier(StructType([]))(None)
+    >>> _make_type_verifier(StringType())("")
+    >>> _make_type_verifier(LongType())(0)
+    >>> _make_type_verifier(LongType())(1 << 64) # doctest: 
+IGNORE_EXCEPTION_DETAIL
+    Traceback (most recent call last):
+        ...
+    pyspark.errors.exceptions.base.PySparkValueError:...
+    >>> _make_type_verifier(ArrayType(ShortType()))(list(range(3)))
+    >>> _make_type_verifier(ArrayType(StringType()))(set()) # doctest: 
+IGNORE_EXCEPTION_DETAIL
+    Traceback (most recent call last):
+        ...
+    pyspark.errors.exceptions.base.PySparkTypeError:...
+    >>> _make_type_verifier(MapType(StringType(), IntegerType()))({})
+    >>> _make_type_verifier(StructType([]))(())
+    >>> _make_type_verifier(StructType([]))([])
+    >>> _make_type_verifier(StructType([]))([1]) # doctest: 
+IGNORE_EXCEPTION_DETAIL
+    Traceback (most recent call last):
+        ...
+    pyspark.errors.exceptions.base.PySparkValueError:...
+    >>> # Check if numeric values are within the allowed range.
+    >>> _make_type_verifier(ByteType())(12)
+    >>> _make_type_verifier(ByteType())(1234) # doctest: 
+IGNORE_EXCEPTION_DETAIL
+    Traceback (most recent call last):
+        ...
+    pyspark.errors.exceptions.base.PySparkValueError:...
+    >>> _make_type_verifier(ByteType(), False)(None) # doctest: 
+IGNORE_EXCEPTION_DETAIL
+    Traceback (most recent call last):
+        ...
+    pyspark.errors.exceptions.base.PySparkValueError:...
+    >>> _make_type_verifier(
+    ...     ArrayType(ShortType(), False))([1, None]) # doctest: 
+IGNORE_EXCEPTION_DETAIL
+    Traceback (most recent call last):
+        ...
+    pyspark.errors.exceptions.base.PySparkValueError:...
+    >>> _make_type_verifier(  # doctest: +IGNORE_EXCEPTION_DETAIL
+    ...     MapType(StringType(), IntegerType())
+    ...     )({None: 1})
+    Traceback (most recent call last):
+        ...
+    pyspark.errors.exceptions.base.PySparkValueError:...
+    >>> schema = StructType().add("a", IntegerType()).add("b", StringType(), 
False)
+    >>> _make_type_verifier(schema)((1, None)) # doctest: 
+IGNORE_EXCEPTION_DETAIL
+    Traceback (most recent call last):
+        ...
+    pyspark.errors.exceptions.base.PySparkValueError:...
+    """
+
+    if name is None:
+
+        def new_msg(msg: str) -> str:
+            return msg
+
+        def new_name(n: str) -> str:
+            return "field %s" % n
+
+    else:
+
+        def new_msg(msg: str) -> str:
+            return "%s: %s" % (name, msg)
+
+        def new_name(n: str) -> str:
+            return "field %s in %s" % (n, name)
+
+    def verify_nullability(obj: Any) -> bool:
+        if obj is None or (isinstance(obj, decimal.Decimal) and obj.is_nan()):
+            # Spark's DecimalType doesn't support NaN,
+            # casting DoubleType NaN to DecimalType will return Null.
+            if nullable:
+                return True
+            else:
+                if name is not None:
+                    raise PySparkValueError(
+                        errorClass="FIELD_NOT_NULLABLE_WITH_NAME",
+                        messageParameters={
+                            "field_name": str(name),
+                        },
+                    )
+                raise PySparkValueError(
+                    errorClass="FIELD_NOT_NULLABLE",
+                    messageParameters={},
+                )
+        else:
+            return False
+
+    _type = type(dataType)
+
+    def assert_acceptable_types(obj: Any) -> None:
+        assert _type in _acceptable_types, new_msg(
+            "unknown datatype: %s for object %r" % (dataType, obj)
+        )
+
+    def verify_acceptable_types(obj: Any) -> None:
+        # subclass of them can not be fromInternal in JVM
+        if type(obj) not in _acceptable_types[_type]:
+            if name is not None:
+                raise PySparkTypeError(
+                    errorClass="FIELD_DATA_TYPE_UNACCEPTABLE_WITH_NAME",
+                    messageParameters={
+                        "field_name": str(name),
+                        "data_type": str(dataType),
+                        "obj": repr(obj),
+                        "obj_type": str(type(obj)),
+                    },
+                )
+            raise PySparkTypeError(
+                errorClass="FIELD_DATA_TYPE_UNACCEPTABLE",
+                messageParameters={
+                    "data_type": str(dataType),
+                    "obj": repr(obj),
+                    "obj_type": str(type(obj)),
+                },
+            )
+
+    if isinstance(dataType, (StringType, CharType, VarcharType)):
+        # StringType, CharType and VarcharType can work with any types
+        def verify_value(obj: Any) -> None:
+            pass
+
+    elif isinstance(dataType, UserDefinedType):
+        verifier = _make_type_verifier(dataType.sqlType(), name=name)
+
+        def verify_udf(obj: Any) -> None:
+            if not (hasattr(obj, "__UDT__") and obj.__UDT__ == dataType):
+                if name is not None:
+                    raise PySparkValueError(
+                        errorClass="FIELD_TYPE_MISMATCH_WITH_NAME",
+                        messageParameters={
+                            "field_name": str(name),
+                            "obj": str(obj),
+                            "data_type": str(dataType),
+                        },
+                    )
+                raise PySparkValueError(
+                    errorClass="FIELD_TYPE_MISMATCH",
+                    messageParameters={
+                        "obj": str(obj),
+                        "data_type": str(dataType),
+                    },
+                )
+            verifier(dataType.toInternal(obj))
+
+        verify_value = verify_udf
+
+    elif isinstance(dataType, ByteType):
+
+        def verify_byte(obj: Any) -> None:
+            assert_acceptable_types(obj)
+            verify_acceptable_types(obj)
+            lower_bound = -128
+            upper_bound = 127
+            if obj < lower_bound or obj > upper_bound:
+                raise PySparkValueError(
+                    errorClass="VALUE_OUT_OF_BOUNDS",
+                    messageParameters={
+                        "arg_name": "obj",
+                        "lower_bound": str(lower_bound),
+                        "upper_bound": str(upper_bound),
+                        "actual": str(obj),
+                    },
+                )
+
+        verify_value = verify_byte
+
+    elif isinstance(dataType, ShortType):
+
+        def verify_short(obj: Any) -> None:
+            assert_acceptable_types(obj)
+            verify_acceptable_types(obj)
+            lower_bound = -32768
+            upper_bound = 32767
+            if obj < lower_bound or obj > upper_bound:
+                raise PySparkValueError(
+                    errorClass="VALUE_OUT_OF_BOUNDS",
+                    messageParameters={
+                        "arg_name": "obj",
+                        "lower_bound": str(lower_bound),
+                        "upper_bound": str(upper_bound),
+                        "actual": str(obj),
+                    },
+                )
+
+        verify_value = verify_short
+
+    elif isinstance(dataType, IntegerType):
+
+        def verify_integer(obj: Any) -> None:
+            assert_acceptable_types(obj)
+            verify_acceptable_types(obj)
+            lower_bound = -2147483648
+            upper_bound = 2147483647
+            if obj < lower_bound or obj > upper_bound:
+                raise PySparkValueError(
+                    errorClass="VALUE_OUT_OF_BOUNDS",
+                    messageParameters={
+                        "arg_name": "obj",
+                        "lower_bound": str(lower_bound),
+                        "upper_bound": str(upper_bound),
+                        "actual": str(obj),
+                    },
+                )
+
+        verify_value = verify_integer
+
+    elif isinstance(dataType, LongType):
+
+        def verify_long(obj: Any) -> None:
+            assert_acceptable_types(obj)
+            verify_acceptable_types(obj)
+            lower_bound = -9223372036854775808
+            upper_bound = 9223372036854775807
+            if obj < lower_bound or obj > upper_bound:
+                raise PySparkValueError(
+                    errorClass="VALUE_OUT_OF_BOUNDS",
+                    messageParameters={
+                        "arg_name": "obj",
+                        "lower_bound": str(lower_bound),
+                        "upper_bound": str(upper_bound),
+                        "actual": str(obj),
+                    },
+                )
+
+        verify_value = verify_long
+
+    elif isinstance(dataType, ArrayType):
+        element_verifier = _make_type_verifier(
+            dataType.elementType, dataType.containsNull, name="element in 
array %s" % name
+        )
+
+        def verify_array(obj: Any) -> None:
+            assert_acceptable_types(obj)
+            verify_acceptable_types(obj)
+            for i in obj:
+                element_verifier(i)
+
+        verify_value = verify_array
+
+    elif isinstance(dataType, MapType):
+        key_verifier = _make_type_verifier(dataType.keyType, False, name="key 
of map %s" % name)
+        value_verifier = _make_type_verifier(
+            dataType.valueType, dataType.valueContainsNull, name="value of map 
%s" % name
+        )
+
+        def verify_map(obj: Any) -> None:
+            assert_acceptable_types(obj)
+            verify_acceptable_types(obj)
+            for k, v in obj.items():
+                key_verifier(k)
+                value_verifier(v)
+
+        verify_value = verify_map
+
+    elif isinstance(dataType, StructType):
+        verifiers = []
+        for f in dataType.fields:
+            verifier = _make_type_verifier(f.dataType, f.nullable, 
name=new_name(f.name))
+            verifiers.append((f.name, verifier))
+
+        def verify_struct(obj: Any) -> None:
+            assert_acceptable_types(obj)
+
+            if isinstance(obj, dict):
+                for f, verifier in verifiers:
+                    verifier(obj.get(f))
+            elif isinstance(obj, (tuple, list, Row)):
+                if len(obj) != len(verifiers):
+                    if name is not None:
+                        raise PySparkValueError(
+                            
errorClass="FIELD_STRUCT_LENGTH_MISMATCH_WITH_NAME",
+                            messageParameters={
+                                "field_name": str(name),
+                                "object_length": str(len(obj)),
+                                "field_length": str(len(verifiers)),
+                            },
+                        )
+                    raise PySparkValueError(
+                        errorClass="FIELD_STRUCT_LENGTH_MISMATCH",
+                        messageParameters={
+                            "object_length": str(len(obj)),
+                            "field_length": str(len(verifiers)),
+                        },
+                    )
+                for v, (_, verifier) in zip(obj, verifiers):
+                    verifier(v)
+            elif hasattr(obj, "__dict__"):
+                d = obj.__dict__
+                for f, verifier in verifiers:
+                    verifier(d.get(f))
+            else:
+                if name is not None:
+                    raise PySparkTypeError(
+                        errorClass="FIELD_DATA_TYPE_UNACCEPTABLE_WITH_NAME",
+                        messageParameters={
+                            "field_name": str(name),
+                            "data_type": str(dataType),
+                            "obj": repr(obj),
+                            "obj_type": str(type(obj)),
+                        },
+                    )
+                raise PySparkTypeError(
+                    errorClass="FIELD_DATA_TYPE_UNACCEPTABLE",
+                    messageParameters={
+                        "data_type": str(dataType),
+                        "obj": repr(obj),
+                        "obj_type": str(type(obj)),
+                    },
+                )
+
+        verify_value = verify_struct
+
+    elif isinstance(dataType, VariantType):
+
+        def verify_variant(obj: Any) -> None:
+            # The variant data type can take in any type.
+            pass
+
+        verify_value = verify_variant
+
+    elif isinstance(dataType, GeometryType):
+
+        def verify_geometry(obj: Any) -> None:
+            assert_acceptable_types(obj)
+            verify_acceptable_types(obj)
+            assert isinstance(obj, Geometry)
+
+        verify_value = verify_geometry
+
+    elif isinstance(dataType, GeographyType):
+
+        def verify_geography(obj: Any) -> None:
+            assert_acceptable_types(obj)
+            verify_acceptable_types(obj)
+            assert isinstance(obj, Geography)
+
+        verify_value = verify_geography
+
+    else:
+
+        def verify_default(obj: Any) -> None:
+            assert_acceptable_types(obj)
+            verify_acceptable_types(obj)
+
+        verify_value = verify_default
+
+    def verify(obj: Any) -> None:
+        if not verify_nullability(obj):
+            verify_value(obj)
+
+    return verify
+
+
+# This is used to unpickle a Row from JVM
+def _create_row_inbound_converter(dataType: DataType) -> Callable:
+    return lambda *a: dataType.fromInternal(a)
+
+
diff --git a/python/tests/test_dropin_offline.py 
b/python/tests/test_dropin_offline.py
index 128059b..553048a 100644
--- a/python/tests/test_dropin_offline.py
+++ b/python/tests/test_dropin_offline.py
@@ -206,6 +206,216 @@ def test_spatial_and_collated_json_forms_parse():
         assert want.json() == json.dumps(j)
 
 
+def test_private_type_helpers_are_those_of_pyspark():
+    # As PySpark's pyspark.sql.types: the inference, merging, conversion and 
verification of the
+    # types of Python objects, and their tables. They take the client's Row as 
PySpark's take
+    # theirs, a tuple.
+    from array import array
+
+    from pyspark.errors import PySparkTypeError, PySparkValueError
+    from pyspark.sql.types import (
+        JVM_INT_MAX,
+        _acceptable_types,
+        _array_type_mappings,
+        _create_converter,
+        _has_nulltype,
+        _infer_schema,
+        _infer_type,
+        _make_type_verifier,
+        _merge_type,
+    )
+
+    assert JVM_INT_MAX == 2**31 - 1
+    assert _array_type_mappings["d"] is T.DoubleType and 
_array_type_mappings["i"] is T.IntegerType
+    assert T.StructType in _acceptable_types and dict in 
_acceptable_types[T.StructType]
+    assert _infer_type(array("d", [1.0])) == T.ArrayType(T.DoubleType(), False)
+    assert _infer_type({"k": [1]}) == T.MapType(T.StringType(), 
T.ArrayType(T.LongType()))
+    schema = T.StructType([T.StructField("a", T.LongType()), 
T.StructField("b", T.StringType())])
+    assert _infer_schema(Row(a=1, b="x")) == schema
+    assert _infer_schema(Row(1, "x"), names=["a", "b"]) == schema
+    assert _has_nulltype(_infer_schema({"a": None}))
+    merged = _merge_type(schema, T.StructType([T.StructField("c", 
T.DoubleType())]))
+    assert merged.names == ["a", "b", "c"]
+    with pytest.raises(PySparkTypeError) as error:
+        _merge_type(T.LongType(), T.DoubleType())
+    assert error.value.getCondition() == "CANNOT_MERGE_TYPE"
+
+    verify = _make_type_verifier(T.StructType([T.StructField("a", 
T.IntegerType(), False)]))
+    for value in (Row(a=1), (1,), {"a": 1}):
+        verify(value)
+    with pytest.raises(PySparkValueError) as error:
+        verify(Row(a=None))
+    assert error.value.getCondition() == "FIELD_NOT_NULLABLE_WITH_NAME"
+    with pytest.raises(PySparkValueError) as error:
+        verify(Row(a=1, b=2))
+    assert error.value.getCondition() == "FIELD_STRUCT_LENGTH_MISMATCH"
+    struct = T.StructType([T.StructField("m", T.MapType(T.StringType(), 
T.IntegerType()))])
+    assert _create_converter(struct)(Row(m={"k": 1})) == ({"k": 1},)
+
+
+
+
+def _type_verifier_cases():
+    # The cases of Apache Spark's 
DataTypeVerificationTests.test_verify_type_not_nullable: the
+    # values that a type takes, and those it rejects, of the error of each.
+    import array
+    import datetime
+    import decimal
+
+    from pyspark.testing.objects import ExamplePoint, ExamplePointUDT, 
PythonOnlyUDT
+
+    schema = T.StructType(
+        [T.StructField("s", T.StringType(), False), T.StructField("i", 
T.IntegerType(), True)]
+    )
+
+    class MyObj:
+        def __init__(self, **kwargs):
+            for k, v in kwargs.items():
+                setattr(self, k, v)
+
+    success = [
+        ("", T.StringType()), (1, T.StringType()), ([], T.StringType()), ({}, 
T.StringType()),
+        ("", T.StringType("UTF8_LCASE")), (1.0, T.CharType(10)), ({}, 
T.VarcharType(10)),
+        (ExamplePoint(1.0, 2.0), ExamplePointUDT()), (True, T.BooleanType()),
+        (-(2**7), T.ByteType()), (2**7 - 1, T.ByteType()), (-(2**15), 
T.ShortType()),
+        (2**15 - 1, T.ShortType()), (-(2**31), T.IntegerType()), (2**31 - 1, 
T.IntegerType()),
+        (-(2**63), T.LongType()), (2**63 - 1, T.LongType()), (1.0, 
T.FloatType()),
+        (1.0, T.DoubleType()), (decimal.Decimal("1.0"), T.DecimalType()),
+        (bytearray([1, 2]), T.BinaryType()), (datetime.date(2000, 1, 2), 
T.DateType()),
+        (datetime.time(1, 0, 0), T.TimeType()), (datetime.datetime(2000, 1, 
2), T.DateType()),
+        (datetime.datetime(2000, 1, 2, 3, 4), T.TimestampType()),
+        ([], T.ArrayType(T.IntegerType())), (["1", None], 
T.ArrayType(T.StringType())),
+        ((1, 2), T.ArrayType(T.IntegerType())), (array.array("h", [1]), 
T.ArrayType(T.IntegerType())),
+        ({"a": 1}, T.MapType(T.StringType(), T.IntegerType())),
+        ({"a": None}, T.MapType(T.StringType(), T.IntegerType(), True)),
+        ({"s": "a", "i": None}, schema), ({"s": "a", "f": 1.0}, schema), 
(Row(s="a", i=1), schema),
+        (Row(s="a", i=None), schema), (["a", 1], schema), (("a", None), 
schema),
+        (MyObj(s="a", i=1), schema), (MyObj(s="a"), schema),
+    ]
+    failure = [
+        (None, T.StringType(), ValueError), (None, T.CharType(10), ValueError),
+        (None, T.VarcharType(10), ValueError), (ExamplePoint(1.0, 2.0), 
PythonOnlyUDT(), ValueError),
+        (1, T.BooleanType(), TypeError), ("True", T.BooleanType(), TypeError),
+        (-(2**7) - 1, T.ByteType(), ValueError), (2**7, T.ByteType(), 
ValueError),
+        ("1", T.ByteType(), TypeError), (1.0, T.ByteType(), TypeError),
+        (2**15, T.ShortType(), ValueError), (2**31, T.IntegerType(), 
ValueError),
+        (1, T.FloatType(), TypeError), (1, T.DoubleType(), TypeError),
+        (1.0, T.DecimalType(), TypeError), ("1.0", T.DecimalType(), TypeError),
+        (1, T.BinaryType(), TypeError), ("2000-01-02", T.DateType(), 
TypeError),
+        ("23:59:59", T.TimeType(), TypeError), (946811040, T.TimestampType(), 
TypeError),
+        (["1", None], T.ArrayType(T.StringType(), False), ValueError),
+        ([1, "2"], T.ArrayType(T.IntegerType()), TypeError),
+        ({"a": 1}, T.MapType(T.IntegerType(), T.IntegerType()), TypeError),
+        ({"a": "1"}, T.MapType(T.StringType(), T.IntegerType()), TypeError),
+        ({"a": None}, T.MapType(T.StringType(), T.IntegerType(), False), 
ValueError),
+        ({"s": "a", "i": "1"}, schema, TypeError), (Row(s="a"), schema, 
ValueError),
+        (Row(s="a", i="1"), schema, TypeError), (["a"], schema, ValueError),
+        (["a", "1"], schema, TypeError), (MyObj(s="a", i="1"), schema, 
TypeError),
+        (MyObj(s=None, i="1"), schema, ValueError),
+    ]
+    return success, failure
+
+
+def test_type_verifiers_take_and_reject_the_values_of_pyspark():
+    # As Apache Spark's DataTypeVerificationTests: _make_type_verifier takes 
the values of each
+    # type, of the client's Row too, and raises the errors of PySpark for the 
others, with the
+    # names of the fields.
+    from pyspark.errors import PySparkTypeError, PySparkValueError
+    from pyspark.sql.types import _make_type_verifier
+
+    success, failure = _type_verifier_cases()
+    for obj, data_type in success:
+        _make_type_verifier(data_type, nullable=False)(obj)
+    for obj, data_type, error in failure:
+        with pytest.raises(error):
+            _make_type_verifier(data_type, nullable=False)(obj)
+    for data_type in (T.IntegerType(), T.StringType(), T.StructType([])):
+        _make_type_verifier(data_type, nullable=True)(None)
+    with pytest.raises(PySparkValueError) as e:
+        _make_type_verifier(T.StringType(), nullable=False, 
name="test_name")(None)
+    assert e.value.getCondition() == "FIELD_NOT_NULLABLE_WITH_NAME"
+    schema = T.StructType([T.StructField("a", T.StructType([T.StructField("b", 
T.IntegerType())]))])
+    with pytest.raises(PySparkTypeError) as e:
+        _make_type_verifier(schema)([["data"]])
+    assert e.value.getCondition() == "FIELD_DATA_TYPE_UNACCEPTABLE_WITH_NAME"
+    assert e.value.getMessageParameters()["field_name"] == "field b in field a"
+
+
+def 
test_types_of_python_objects_are_inferred_merged_and_converted_as_pyspark():
+    # As PySpark's _infer_type, _infer_schema, _merge_type and 
_create_converter, of the
+    # values of each type, also nested, of the client's Row too.
+    import datetime
+    import decimal
+
+    from pyspark.errors import PySparkTypeError
+    from pyspark.sql.types import (
+        _create_converter,
+        _infer_schema,
+        _infer_type,
+        _merge_type,
+        _need_converter,
+    )
+
+    values = [
+        (None, T.NullType()), (True, T.BooleanType()), (1, T.LongType()), 
(1.5, T.DoubleType()),
+        ("s", T.StringType()), (b"b", T.BinaryType()),
+        (decimal.Decimal("1.50"), T.DecimalType(38, 18)), (datetime.date(2000, 
1, 1), T.DateType()),
+        (datetime.datetime(2000, 1, 1), T.TimestampType()), (datetime.time(1, 
2), T.TimeType()),
+        (datetime.timedelta(1), T.DayTimeIntervalType()),
+    ]
+    for value, data_type in values:
+        assert _infer_type(value) == data_type
+    assert _infer_type([1, None]) == T.ArrayType(T.LongType())
+    assert _infer_type([None, 1], infer_array_from_first_element=True) == 
T.ArrayType(
+        T.NullType()
+    )
+    assert _infer_type({"a": {"b": 1}}, infer_dict_as_struct=True) == 
T.StructType(
+        [T.StructField("a", T.StructType([T.StructField("b", T.LongType())]))]
+    )
+    assert _infer_type(datetime.datetime(2000, 1, 1), 
prefer_timestamp_ntz=True) == (
+        T.TimestampNTZType()
+    )
+    assert _infer_type(Row(a=[Row(b=1)])) == T.StructType(
+        [T.StructField("a", T.ArrayType(T.StructType([T.StructField("b", 
T.LongType())])))]
+    )
+
+    class Obj:
+        def __init__(self):
+            self.b, self.a = "x", 1
+
+    assert _infer_schema(Obj()).names == ["a", "b"]
+    assert _infer_schema((1, 2), names=["x"]).names == ["x", "_2"]
+    with pytest.raises(PySparkTypeError) as e:
+        _infer_schema(1)
+    assert e.value.getCondition() == "CANNOT_INFER_SCHEMA_FOR_TYPE"
+
+    a = T.StructType([T.StructField("f", T.ArrayType(T.NullType()))])
+    b = T.StructType([T.StructField("f", T.ArrayType(T.LongType()))])
+    assert _merge_type(a, b) == b and _merge_type(T.NullType(), T.LongType()) 
== T.LongType()
+    assert _merge_type(
+        T.MapType(T.StringType(), T.NullType()), T.MapType(T.StringType(), 
T.LongType())
+    ) == T.MapType(T.StringType(), T.LongType())
+    strings = T.StructType([T.StructField("f", T.ArrayType(T.StringType()))])
+    assert _merge_type(b, strings) == strings
+    with pytest.raises(PySparkTypeError) as e:
+        _merge_type(b, T.StructType([T.StructField("f", 
T.ArrayType(T.DateType()))]))
+    assert e.value.getCondition() == "CANNOT_MERGE_TYPE"
+
+    struct = T.StructType(
+        [T.StructField("m", T.MapType(T.StringType(), T.StringType())),
+         T.StructField("s", T.StructType([T.StructField("x", T.LongType())]))]
+    )
+    assert _need_converter(struct) and not _need_converter(T.LongType())
+    convert = _create_converter(struct)
+    assert convert(Row(m={"k": "v"}, s=Row(x=1))) == ({"k": "v"}, (1,))
+    assert convert({"m": None, "s": {"x": 2}}) == (None, (2,))
+    with pytest.raises(PySparkTypeError) as e:
+        convert(1)
+    assert e.value.getCondition() == "UNSUPPORTED_DATA_TYPE"
+
+
+
+
 def test_unparsed_datatype_pickle_roundtrip():
     # The core's JSON form of an unparsed type is the UnparsedDataType of
     # pyspark.sql.connect.types, which pickles of its DDL string, as its JSON 
raises.


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to