Hi all.
I read source code at spark/python/pyspark/sql/connect/session.py at master
· apache/spark (github.com) and the comment for the "stop" method is described
as follows:
def stop(self) -> None:
# Stopping the session will only close the connection to the current
session (and
# the life cycle of the session is maintained by the server),
# whereas the regular PySpark session immediately terminates the Spark
Context
# itself, meaning that stopping all Spark sessions.
# It is controversial to follow the existing the regular Spark
session's behavior
# specifically in Spark Connect the Spark Connect server is designed for
# multi-tenancy - the remote client side cannot just stop the server
and stop
# other remote clients being used from other users.
So, that's how it was designed.
eabour
From: eab...@163.com
Date: 2023-10-20 15:56
To: user @spark
Subject: spark.stop() cannot stop spark connect session
Hi,
my code:
from pyspark.sql import SparkSession
spark = SparkSession.builder.remote("sc://172.29.190.147").getOrCreate()
import pandas as pd
# 创建pandas dataframe
pdf = pd.DataFrame({
"name": ["Alice", "Bob", "Charlie"],
"age": [25, 30, 35],
"gender": ["F", "M", "M"]
})
# 将pandas dataframe转换为spark dataframe
sdf = spark.createDataFrame(pdf)
# 显示spark dataframe
sdf.show()
spark.stop()
After stop, execute sdf.show() throw
pyspark.errors.exceptions.connect.SparkConnectException: [NO_ACTIVE_SESSION] No
active Spark session found. Please create a new Spark session before running
the code. Visit the Spark web UI at http://172.29.190.147:4040/connect/ to
check if the current session is still running and has not been stopped yet.
1 session(s) are online, running 0 Request(s)
Session Statistics (1)
1 Pages. Jump to. Showitems in a page.Go
Page:
1
User
Session ID
Start Time ▾
Finish Time
Duration
Total Execute
29f05cde-8f8b-418d-95c0-8dbbbfb556d22023/10/20 15:30:0414 minutes 49 seconds2
eabour
