[
https://issues.apache.org/jira/browse/SPARK-57738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jubin Soni updated SPARK-57738:
-------------------------------
Parent: SPARK-56822
Issue Type: Sub-task (was: Bug)
> ArrowVectorReader guard no longer rejects unsupported nanosecond timestamp
> types over Spark Connect
> ---------------------------------------------------------------------------------------------------
>
> Key: SPARK-57738
> URL: https://issues.apache.org/jira/browse/SPARK-57738
> Project: Spark
> Issue Type: Sub-task
> Components: Connect
> Affects Versions: 4.3.0
> Reporter: Jubin Soni
> Priority: Minor
> Labels: pull-request-available
>
> *What is the issue?*
> {{ArrowVectorReader.applyDefault}} contains a guard that is supposed to
> reject any attempt to read an unsupported target type over Spark Connect:
> {{if (!UpCastRule.canUpCast(vectorDataType, targetDataType)) \{
> throw new RuntimeException(
> s"Reading '$targetDataType' values from a ${vector.getClass} instance is
> not supported.")
> }}}
> SPARK-57303 updated {{UpCastRule.canUpCast}} to return {{true}} for lossless
> widening within the timestamp family (e.g. {{{}TimestampType ->
> TimestampLTZNanosType(p){}}}). As a side effect this guard now silently
> passes for nanosecond timestamp targets, even though no Arrow vector encoding
> or reader implementation exists for them. Execution then falls through the
> {{vector match}} block to the generic catch-all:
> {{case _ => throw new RuntimeException("Unsupported Vector Type: " +
> vector.getClass)}}
> This is a confusing, misleading error that gives no indication the problem is
> the target type rather than the vector type. The SPARK-57303 commit message
> explicitly acknowledges this regression: _"ArrowVectorReader's {{canUpCast}}
> guard no longer fails fast on a micro-vector/nanos-target mismatch; whenever
> nanos-over-Connect is implemented, that PR should add the reader and
> re-examine this guard."_
> ----
> *How to reproduce*
> Call {{ArrowVectorReader.apply}} with a {{TimestampLTZNanosType}} or
> {{TimestampNTZNanosType}} target against a micro-precision timestamp vector
> (the vector type a Connect server sends for any LTZ timestamp column):
> {{import org.apache.arrow.memory.RootAllocator
> import org.apache.spark.sql.connect.client.arrow.ArrowVectorReader
> import org.apache.spark.sql.types.\{TimestampLTZNanosType, TimestampType}
> import org.apache.spark.sql.util.ArrowUtils
> val allocator = new RootAllocator()
> val vector = ArrowUtils.toArrowField("ts", TimestampType, nullable = true,
> "UTC")
> .createVector(allocator)
> ArrowVectorReader(TimestampLTZNanosType(9), vector, "UTC")}}
> ----
> *Actual behavior*
> {{java.lang.RuntimeException: Unsupported Vector Type: class
> org.apache.arrow.vector.TimeStampMicroTZVector}}
> ----
> *Expected behavior*
> {{java.lang.RuntimeException: Reading 'timestamp_ltz(9)' values over Spark
> Connect is not yet supported.}}
> ----
> *Proposed fix*
> Add an explicit rejection on {{AnyTimestampNanoType}} in
> {{ArrowVectorReader.applyDefault}} between the {{canUpCast}} guard and the
> {{vector match}} block, in
> {{{}sql/connect/common/src/main/scala/org/apache/spark/sql/connect/client/arrow/ArrowVectorReader.scala{}}}:
> {{if (targetDataType.isInstanceOf[AnyTimestampNanoType]) \{
> throw new RuntimeException(
> s"Reading '$targetDataType' values over Spark Connect is not yet
> supported.")
> }}}
> This guard should be removed when a proper Connect reader for nanosecond
> timestamp types is implemented.
> ----
> ----
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
