[
https://issues.apache.org/jira/browse/SPARK-55441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-55441:
-----------------------------------
Labels: pull-request-available (was: )
> Types Framework - Phase 1c - Client Integration
> -----------------------------------------------
>
> Key: SPARK-55441
> URL: https://issues.apache.org/jira/browse/SPARK-55441
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
> Affects Versions: 4.2.0
> Reporter: David Milicevic
> Priority: Major
> Labels: pull-request-available
>
> *Summary:*
> Add client-side Ops interfaces and integrate with Spark Connect, Arrow, JDBC,
> Python, and Thrift
> *Description:*
> Extend the framework with client-facing operations and integrate with all
> client communication paths.
> *What this includes:*
> * New optional interfaces: {{ProtoTypeOps}} (Spark Connect proto
> serialization), {{ClientTypeOps}} (JDBC, Arrow, Python, Thrift)
> * Optional interfaces use \{{TypeOps(dt).collect { case p: ProtoTypeOps => p
> }}} for granular support checking
> * Extend TimeTypeOps with client operation implementations
> * Integration in ~18 files: {{{}DataTypeProtoConverter{}}},
> {{{}LiteralValueProtoConverter{}}}, {{{}ArrowUtils{}}}, {{{}ArrowWriter{}}},
> {{{}ArrowSerializer{}}}, {{{}ArrowVectorReader{}}},
> {{{}ArrowDeserializer{}}}, {{{}JdbcTypeUtils{}}},
> {{{}SparkConnectResultSet{}}}, {{{}SparkExecuteStatementOperation{}}},
> {{{}HiveResult{}}}, {{{}EvaluatePython{}}}, {{{}DeserializerBuildHelper{}}},
> {{{}SerializerBuildHelper{}}}, {{{}udaf.scala{}}}, and others
> *Note:*
> Client integration has many non-obvious patterns beyond {{case _: TimeType
> =>}} - Arrow uses {{{}TimeNanoVector{}}}, proto uses
> {{{}.hasTime(){}}}/{{{}.getTime(){}}}, Thrift uses
> {{{}TTypeId.STRING_TYPE{}}}. The framework consolidates all of these behind a
> single Ops interface.
> *Design doc:*
> Linked in the parent work item.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
