Gengliang Wang created SPARK-60753:
--------------------------------------
Summary: Read the columns of native data sources from their Arrow
buffers, and their decimals as longs
Key: SPARK-60753
URL: https://issues.apache.org/jira/browse/SPARK-60753
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 5.0.0
Reporter: Gengliang Wang
Native data sources (SPARK-60101) return Arrow batches, which
NativePartitionReader imports into the same Arrow vectors and Spark reads
through ArrowColumnVector. Each value goes through Arrow Java, which by default
checks on each access the bounds and the reference count of the buffer and
whether the value is null, and a decimal of at most 18 digits becomes a
BigDecimal-backed Decimal, which Spark then adds as BigDecimals.
Read the top-level columns of the reader at the addresses of the buffers of
each batch instead, and return the decimals that fit their precision in a long
as long-backed Decimals, as Spark's own column vectors do.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
