magpierre commented on PR #80:
URL: https://github.com/apache/spark-connect-go/pull/80#issuecomment-2436276503
> Would be good to add some additional tests for more recursive behavior:
>
> ```
> >>> spark.sql("select map(1, 2), map(\"a\", 2)").printSchema()
> root
> |-- map(1, 2): map (nullable = false)
> | |-- key: integer
> | |-- value: integer (valueContainsNull = false)
> |-- map(a, 2): map (nullable = false)
> | |-- key: string
> | |-- value: integer (valueContainsNull = false)
>
> >>> spark.sql("select map(1, 2), map(\"a\", map(1, 2))").printSchema()
> root
> |-- map(1, 2): map (nullable = false)
> | |-- key: integer
> | |-- value: integer (valueContainsNull = false)
> |-- map(a, map(1, 2)): map (nullable = false)
> | |-- key: string
> | |-- value: map (valueContainsNull = false)
> | | |-- key: integer
> | | |-- value: integer (valueContainsNull = false)
> ```
I could add recursive tests but then I first need to have recursive schema
capabilities in PrintSchema which is currently missing since the types Map, and
Array is missing types, and StructType are not embeddable in a StructField. For
instance the implemented StructType is not following the interface for DataType
and it needs to in order to be embedded in a StructField (if I understand the
code correctly). So there is additional work that needs to be done on the
DataType side before I can extend printSchema to also include hierarchical
capabilities.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
