[
https://issues.apache.org/jira/browse/SPARK-55441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Milicevic updated SPARK-55441:
------------------------------------
Summary: Types Framework - Phase 1b - Client Integration (was: Phase 1b -
Client Integration)
> Types Framework - Phase 1b - Client Integration
> -----------------------------------------------
>
> Key: SPARK-55441
> URL: https://issues.apache.org/jira/browse/SPARK-55441
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
> Affects Versions: 4.2.0
> Reporter: David Milicevic
> Priority: Major
>
> *Summary:*
> Add client-side Ops interfaces and integrate with Spark Connect, Arrow, JDBC,
> Python, and Thrift
> *Description:*
> Extend the framework with client-facing operations and integrate with all
> client communication paths.
> *What this includes:*
> * New interfaces: {{ProtoTypeOps}} (Spark Connect proto serialization),
> {{ClientTypeOps}} (JDBC, Arrow, Python, Thrift)
> * Extend TimeType Ops with client operation implementations
> * Integration in ~13 files: {{{}DataTypeProtoConverter{}}},
> {{{}LiteralValueProtoConverter{}}}, {{{}ArrowUtils{}}}, {{{}ArrowWriter{}}},
> {{{}ArrowSerializer{}}}, {{{}ArrowVectorReader{}}}, {{{}JdbcTypeUtils{}}},
> {{{}SparkExecuteStatementOperation{}}}, {{{}HiveResult{}}},
> {{{}EvaluatePython{}}}, {{{}DeserializerBuildHelper{}}},
> {{{}SerializerBuildHelper{}}}, {{udaf.scala}}
> *Note:*
> Client integration has many non-obvious patterns beyond {{case _: TimeType
> =>}} - Arrow uses {{{}TimeNanoVector{}}}, proto uses
> {{{}.hasTime(){}}}/{{{}.getTime(){}}}, Thrift uses
> {{{}TTypeId.STRING_TYPE{}}}. The framework consolidates all of these behind a
> single Ops interface.
> *Design doc:*
> Linked in the parent work item.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
