WonYong-Jang opened a new issue, #58310:
URL: https://github.com/apache/spark/issues/58310
## Problem
When using `spark-submit --files` with
`spark.kubernetes.file.upload.path=s3a://...`, executor pods fail to access the
uploaded files in Kubernetes, while the same approach works in YARN.
```
spark-submit --files test.csv \
--conf spark.kubernetes.file.upload.path=s3a://bucket/uploads \
app.py
```
## Root Cause
YARN (works):
- Files uploaded to HDFS staging directory
- `SPARK_YARN_STAGING_DIR` environment variable set to shared HDFS path
- All executors access files from shared HDFS
K8S (broken):
- Files upload to S3: s3a://bucket/spark-upload-<UUID>/test.csv
- Only driver pod downloads file to /tmp/spark-<uuid>/
- SPARK_YARN_STAGING_DIR NOT set in K8s
- Executor pods have no access mechanism
- Files not distributed to executor working directories
## Reproduction
```
from os import getenv
from pyspark.sql import SparkSession
spark = SparkSession.builder.appName("test").getOrCreate()
staging_dir = getenv("SPARK_YARN_STAGING_DIR")
if staging_dir:
# YARN: SPARK_YARN_STAGING_DIR is set
# All executors can access shared HDFS path
path = f"{staging_dir}/test.csv"
df = spark.read.csv(path, header=True) # Success
else:
# K8s: SPARK_YARN_STAGING_DIR is NOT set
# File not accessible from executors
print("Cannot access file in K8s") # Failure
```
## Runtime Investigation
Attempted to discover S3 path at runtime:
```
s3_path = spark.sparkContext.getConf().get("spark.files", "<none>")
print("spark.files:", s3_path)
# Expected: s3a://bucket/spark-upload-UUID/test.csv
# Actual: file:/tmp/spark-fff175ac-.../test.csv
# Conclusion: S3 path is lost, replaced with local path
# that doesn't exist on executor pods
```
## Workaround(Not Ideal)
```
from os import getenv
from pyspark.files import SparkFiles
def _read_source(file_name):
staging_dir = getenv("SPARK_YARN_STAGING_DIR")
if staging_dir:
# YARN: Use shared HDFS path
return f"{staging_dir}/{file_name}"
# K8s: Read in driver, distribute via RDD (inefficient workaround)
with open(SparkFiles.get(file_name)) as f:
return spark.sparkContext.parallelize(f.read().splitlines())
```
Problems with this workaround:
- Driver memory limited (OOM for large files)
## Environment
- Spark: 3.4.4
- K8S: v1.35.3
- Storage: S3
## Proposed Solutions
This issue can be fixed by implementing one of the following approaches:
Solution 1(Recommended): Provide K8s equivalent of SPARK_YARN_STAGING_DIR
- Introduce new environment variable(e.g., SPARK_K8S_FILES_PATH)
- Set to S3 staging directory path that driver already knows
- All executors can access the S3 path via this environment variable
Solution 2: Distribute files to executor working directories
- Add init-container to executor pods (similar to driver pods)
- Init-container downloads files from S3 to /tmp/spark-<uuid>/
- All executors can access local files in their working directory.
## Contribution
I am interested in contributing a fix for this issue.
I welcome discussion and feedback on the proposed approach before proceeding
with a PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
