Gengliang Wang created SPARK-61904:
--------------------------------------
Summary: Account for the memory of native data sources in the
memory manager of Spark
Key: SPARK-61904
URL: https://issues.apache.org/jira/browse/SPARK-61904
Project: Spark
Issue Type: Sub-task
Components: SQL
Affects Versions: 5.0.0
Reporter: Gengliang Wang
Spark does not manage the memory of native data sources. The memory that a
library allocates and the Arrow buffers that it exports, which Spark reads
without copying them, are not accounted by the memory manager of Spark, nor
limited by {{spark.memory.offHeap.size}}. They come out of the memory overhead
of the driver and the executors, and on YARN or Kubernetes an executor that
uses more memory than it requested is killed.
Proposal: let a library acquire and release the memory of a task from Spark,
for example with functions appended to {{SparkDsTaskContext}} within version 1
of the interface ({{spark_datasource.h}}), so that Spark accounts for it and a
library can react when Spark has no memory left for it. Then expose it in the
Rust SDK (SPARK-61282).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]