[
https://issues.apache.org/jira/browse/SPARK-60753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-60753:
-----------------------------------
Labels: pull-request-available (was: )
> Read the columns of native data sources from their Arrow buffers, and their
> decimals as longs
> ---------------------------------------------------------------------------------------------
>
> Key: SPARK-60753
> URL: https://issues.apache.org/jira/browse/SPARK-60753
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 5.0.0
> Reporter: Gengliang Wang
> Priority: Major
> Labels: pull-request-available
>
> Native data sources (SPARK-60101) return Arrow batches, which
> NativePartitionReader imports into the same Arrow vectors and Spark reads
> through ArrowColumnVector. Each value goes through Arrow Java, which by
> default checks on each access the bounds and the reference count of the
> buffer and whether the value is null, and a decimal of at most 18 digits
> becomes a BigDecimal-backed Decimal, which Spark then adds as BigDecimals.
> Read the top-level columns of the reader at the addresses of the buffers of
> each batch instead, and return the decimals that fit their precision in a
> long as long-backed Decimals, as Spark's own column vectors do.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
