[
https://issues.apache.org/jira/browse/SPARK-11010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14983192#comment-14983192
]
Katherine Prevost commented on SPARK-11010:
-------------------------------------------
In addition to the difficulties with UDTs meant to support 3rd-party types,
there's also currently a problem if you wish to provide UDTs on home-grown
types without requiring Spark as a dependency for that type. Annotating a class
with SQLUserDefinedType causes that class to require SQLUserDefinedType to be
available at load time. If the underlying class doesn't know or care about
Spark, this is rather painful and not at all what you want. You can still use
the UDT without the annotation—but it requires all use of the UDT to be done
via explicitly defining your schemas.
Any solution that solves the 3rd party or extensions problem will probably also
solve the load-time dependency problem.
> Fixes and enhancements addressing UDTs' api and several usability concerns
> --------------------------------------------------------------------------
>
> Key: SPARK-11010
> URL: https://issues.apache.org/jira/browse/SPARK-11010
> Project: Spark
> Issue Type: Epic
> Components: SQL
> Affects Versions: 1.5.0, 1.5.1
> Reporter: John Muller
> Labels: UDT
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> Catalyst SQL types allow for UserDefinedTypes, but do not allow for easy
> extension of 3rd party types or extensions to built-in types like DecimalType
> or StringType (private classes).
> Additionally, the API can infer much more of what's needed from the type
> parameter than it currently does.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
