zhengruifeng commented on PR #57754:
URL: https://github.com/apache/spark/pull/57754#issuecomment-5176863470
Here is a simple end-to-end PySpark ML example for manually checking the UI.
Start a Connect server from the built Spark checkout:
```bash
./sbin/start-connect-server.sh --master local[2]
```
In another terminal, start a Connect PySpark shell:
```bash
./bin/pyspark --remote sc://localhost:15002
```
Then run:
```python
from pyspark.ml.classification import LogisticRegression
from pyspark.ml.linalg import Vectors
training = spark.createDataFrame(
[
(1.0, Vectors.dense([0.0, 1.1, 0.1])),
(0.0, Vectors.dense([2.0, 1.0, -1.0])),
(0.0, Vectors.dense([2.0, 1.3, 1.0])),
(1.0, Vectors.dense([0.0, 1.2, -0.5])),
],
["label", "features"],
)
model = LogisticRegression(maxIter=5, regParam=0.01).fit(training)
model.transform(training).select("label", "prediction", "probability").show(
truncate=False
)
```
Keep the PySpark shell open and visit `http://localhost:4040/connect/` (or
the Spark UI URL printed by the server). Before `fit`, the session's **ML
Cache** cell should say `Not used`. After `fit`, reload the page; it should
show one in-memory object and nonzero in-memory/total cache usage.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
