itholic opened a new pull request #34550:
URL: https://github.com/apache/spark/pull/34550
### What changes were proposed in this pull request?
This PR proposes to add an option `display.log_advice` to determine whether
or not issue the advice message for pandas API on Spark.
```python
psdf = ps.DataFrame({"A": [1, 2, 3]})
# Turn on the advice log
with ps.option_context("display.log_advice", True):
psdf.to_pandas()
.../spark/python/pyspark/pandas/utils.py:970: UserWarning: `to_pandas` loads
all data into the driver's memory. It should only be used if the resulting
pandas DataFrame is expected to be small.
warnings.warn(message, UserWarning)
A
0 1
1 2
2 3
# Turn off the advice log
with ps.option_context("display.log_advice", False):
psdf.to_pandas()
A
0 1
1 2
2 3
```
### Why are the changes needed?
Sometimes the messages are too verbose to display, so someone might not need
to see the advice log.
So we'd better give an option to turn on/off the advice messages.
### Does this PR introduce _any_ user-facing change?
Users can use new option, `display.log_advice`.
### How was this patch tested?
It's tested in `tests/test_config.py` by automatically adding to the
existing config list.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
