[
https://issues.apache.org/jira/browse/SPARK-61080?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon updated SPARK-61080:
---------------------------------
Description:
In PySpark Connect, the data types have the attributes of their parameters,
{{typeName}} is a classmethod, and the types without parameters are singletons:
* {{StringType("UTF8_LCASE").collation}}, {{CharType(3).length}},
{{VarcharType(4).length}}, {{TimeType(3).precision}}, the {{startField}} and
{{endField}} of {{DayTimeIntervalType}} and {{YearMonthIntervalType}}, and
{{DecimalType(10, 2).hasPrecisionInfo}}
* {{IntegerType.typeName()}} is "integer", of the name of the class
({{cls.__name__[:-4].lower()}}), also for subclasses of {{DataType}}, and
"void", "timestamp_ntz" and "interval" for {{NullType}}, {{TimestampNTZType}}
and {{CalendarIntervalType}}
* {{StructField.typeName()}} raises the {{PySparkTypeError}}
{{INVALID_TYPENAME_CALL}}
* {{IntegerType() is IntegerType()}}, as for the other types of
{{DataTypeSingleton}}: {{NullType}}, the integral types, {{FloatType}},
{{DoubleType}}, {{BinaryType}}, {{BooleanType}}, {{DateType}},
{{TimestampType}}, {{TimestampNTZType}} and {{CalendarIntervalType}}
The Rust-backed Python client lacks these attributes ({{AttributeError}}). Its
{{typeName}} is an instance method, so {{IntegerType.typeName()}} raises a
{{TypeError}}, and {{CalendarIntervalType().typeName()}} is "calendarinterval".
{{StructField.typeName()}} raises a plain {{TypeError}}, and every call of a
type class returns a new object. Add them as PySpark does.
was:
In PySpark Connect, the data types have the attributes of their parameters, and
{{typeName}} is a classmethod:
* {{StringType("UTF8_LCASE").collation}}, {{CharType(3).length}},
{{VarcharType(4).length}}, {{TimeType(3).precision}}, the {{startField}} and
{{endField}} of {{DayTimeIntervalType}} and {{YearMonthIntervalType}}, and
{{DecimalType(10, 2).hasPrecisionInfo}}
* {{IntegerType.typeName()}} is "integer", of the name of the class
({{cls.__name__[:-4].lower()}}), also for subclasses, and "void",
"timestamp_ntz" and "interval" for {{NullType}}, {{TimestampNTZType}} and
{{CalendarIntervalType}}
* {{StructField.typeName()}} raises the {{PySparkTypeError}}
{{INVALID_TYPENAME_CALL}}
The Rust-backed Python client lacks these attributes ({{AttributeError}}),
{{typeName}} is an instance method, so {{IntegerType.typeName()}} raises a
{{TypeError}}, and {{StructField.typeName()}} raises a plain {{TypeError}}. Add
them as PySpark does.
Summary: Expose the attributes of the data types, typeName as a
classmethod, and the singleton types in the Python client as PySpark does
(was: Expose the attributes of the data types, and typeName as a classmethod,
in the Python client as PySpark does)
> Expose the attributes of the data types, typeName as a classmethod, and the
> singleton types in the Python client as PySpark does
> --------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-61080
> URL: https://issues.apache.org/jira/browse/SPARK-61080
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: Hyukjin Kwon
> Priority: Major
> Fix For: connect-rust-4.3.0
>
>
> In PySpark Connect, the data types have the attributes of their parameters,
> {{typeName}} is a classmethod, and the types without parameters are
> singletons:
> * {{StringType("UTF8_LCASE").collation}}, {{CharType(3).length}},
> {{VarcharType(4).length}}, {{TimeType(3).precision}}, the {{startField}} and
> {{endField}} of {{DayTimeIntervalType}} and {{YearMonthIntervalType}}, and
> {{DecimalType(10, 2).hasPrecisionInfo}}
> * {{IntegerType.typeName()}} is "integer", of the name of the class
> ({{cls.__name__[:-4].lower()}}), also for subclasses of {{DataType}}, and
> "void", "timestamp_ntz" and "interval" for {{NullType}}, {{TimestampNTZType}}
> and {{CalendarIntervalType}}
> * {{StructField.typeName()}} raises the {{PySparkTypeError}}
> {{INVALID_TYPENAME_CALL}}
> * {{IntegerType() is IntegerType()}}, as for the other types of
> {{DataTypeSingleton}}: {{NullType}}, the integral types, {{FloatType}},
> {{DoubleType}}, {{BinaryType}}, {{BooleanType}}, {{DateType}},
> {{TimestampType}}, {{TimestampNTZType}} and {{CalendarIntervalType}}
> The Rust-backed Python client lacks these attributes ({{AttributeError}}).
> Its {{typeName}} is an instance method, so {{IntegerType.typeName()}} raises
> a {{TypeError}}, and {{CalendarIntervalType().typeName()}} is
> "calendarinterval". {{StructField.typeName()}} raises a plain {{TypeError}},
> and every call of a type class returns a new object. Add them as PySpark does.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
