ueshin opened a new pull request, #53021:
URL: https://github.com/apache/spark/pull/53021
### What changes were proposed in this pull request?
Changes the way to access logs to TVF instead of system view.
```sql
SELECT * FROM python_worker_logs()
```
```py
spark.tvf.python_worker_logs()
```
Also blocks the TVF when the python worker logging is disabled.
```py
>>> spark.conf.get('spark.sql.pyspark.worker.logging.enabled')
'false'
>>> spark.tvf.python_worker_logs().show()
Traceback (most recent call last):
...
pyspark.errors.exceptions.captured.AnalysisException: [FEATURE_NOT_ENABLED]
The feature Python Worker Logging is not enabled. Consider setting the config
spark.sql.pyspark.worker.logging.enabled to true to enable this capability.
SQLSTATE: 56038
```
### Why are the changes needed?
There may be namespace conflicts with the other system tables/views, etc..
For example, the variables in SQL has the same namespace
`system.session.varname`, which may potentially cause an issue.
### Does this PR introduce _any_ user-facing change?
Yes, the way to access python worker logs will be changed.
### How was this patch tested?
Modified the related tests.
### Was this patch authored or co-authored using generative AI tooling?
No.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
