NoahKusaba commented on code in PR #2478:
URL:
https://github.com/apache/datafusion-ballista/pull/2478#discussion_r4077757380
##########
python/README.md:
##########
@@ -62,10 +62,17 @@ df : DataFrame = ctx.sql("select * from t limit 5")
df.show()
```
+Session and Ballista settings are set with the `cluster_config` argument:
+
+```python
+ctx = BallistaSessionContext(
+ "df://localhost:50050",
+ cluster_config={"datafusion.execution.target_partitions": "256"},
+)
+```
+
Known limitations and inefficiencies of the current approach:
-- The client's `SessionConfig` is not propagated to Ballista.
Review Comment:
You're right, thanks. `create_ballista_data_frame` starts from a fresh
`new_with_ballista()` config, so `config=` never leaves the local context. I
restored the limitation with your wording, pointing at `cluster_config`.
##########
docs/source/user-guide/tuning-guide.md:
##########
@@ -256,8 +256,8 @@ let session_config = SessionConfig::new_with_ballista()
When AQE is enabled, the scheduler builds the stage DAG incrementally. As each
shuffle stage completes, the planner re-optimizes the remaining plan and emits
-the next set of runnable stages. Two adaptive optimizations are currently
-implemented:
+the next set of runnable stages. The following adaptive optimizations are
+currently implemented:
- **Join reordering.** Uses runtime row counts from completed stages so the
Review Comment:
Good call. Both bullets were describing the same AQE rule in
`join_selection.rs`: it swaps the inputs so the smaller side builds (comparing
byte sizes, falling back to row counts) and decides whether to broadcast. I
merged "Join reordering" and "Broadcast join selection" into one **Join
selection** bullet to match the architecture guide. It also fixes the "row
counts" claim. #2479 was making that same correction, so I've removed it there
to avoid a conflict between the two PRs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
