Gengliang Wang created SPARK-61904:
--------------------------------------

             Summary: Account for the memory of native data sources in the 
memory manager of Spark
                 Key: SPARK-61904
                 URL: https://issues.apache.org/jira/browse/SPARK-61904
             Project: Spark
          Issue Type: Sub-task
          Components: SQL
    Affects Versions: 5.0.0
            Reporter: Gengliang Wang


Spark does not manage the memory of native data sources. The memory that a 
library allocates and the Arrow buffers that it exports, which Spark reads 
without copying them, are not accounted by the memory manager of Spark, nor 
limited by {{spark.memory.offHeap.size}}. They come out of the memory overhead 
of the driver and the executors, and on YARN or Kubernetes an executor that 
uses more memory than it requested is killed.

Proposal: let a library acquire and release the memory of a task from Spark, 
for example with functions appended to {{SparkDsTaskContext}} within version 1 
of the interface ({{spark_datasource.h}}), so that Spark accounts for it and a 
library can react when Spark has no memory left for it. Then expose it in the 
Rust SDK (SPARK-61282).




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to