HyukjinKwon opened a new pull request #34893:
URL: https://github.com/apache/spark/pull/34893
### What changes were proposed in this pull request?
This PR proposes to use an existing active Spark session instead of
`SparkSession.getOrCreate` in pandas API on Spark.
### Why are the changes needed?
Because it shows warnings for configurations not taking effect as below:
Otherwise, it attempts to create a new session, and shows warnings as below:
```python
>>> ps.range(10)
21/12/14 16:12:58 WARN SparkSession: Using an existing SparkSession; the
static sql configurations will not take effect.
21/12/14 16:12:58 WARN SparkSession: Using an existing SparkSession; some
spark core configurations may not take effect.
21/12/14 16:12:58 WARN SparkSession: Using an existing SparkSession; the
static sql configurations will not take effect.
21/12/14 16:12:58 WARN SparkSession: Using an existing SparkSession; some
spark core configurations may not take effect.
21/12/14 16:12:58 WARN SparkSession: Using an existing SparkSession; the
static sql configurations will not take effect.
21/12/14 16:12:58 WARN SparkSession: Using an existing SparkSession; some
spark core configurations may not take effect.
21/12/14 16:12:58 WARN SparkSession: Using an existing SparkSession; the
static sql configurations will not take effect.
21/12/14 16:12:58 WARN SparkSession: Using an existing SparkSession; some
spark core configurations may not take effect.
...
id
0 0
1 1
2 2
3 3
4 4
5 5
6 6
7 7
8 8
```
### Does this PR introduce _any_ user-facing change?
Yes, after this PR, it will explicitly uses active Spark session, and does
not show such warnings:
```python
>>> import pyspark.pandas as ps
>>> ps.range(10)
...
id
0 0
1 1
2 2
3 3
```
### How was this patch tested?
Manually tested as below:
```python
import pyspark.pandas as ps
ps.range(10)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
