NoahKusaba commented on code in PR #2480:
URL:
https://github.com/apache/datafusion-ballista/pull/2480#discussion_r4094049439
##########
examples/README.md:
##########
@@ -27,308 +27,80 @@ The standalone example is the easiest to get started with.
Ballista supports a s
and executor are started in-process.
```bash
-cargo run --example standalone_sql --features="ballista/standalone"
+cargo run --example standalone_sql
```
-### Source code for standalone SQL example
-
-```rust
-use ballista::{
- extension::SessionConfigExt,
- prelude::*
-};
-use datafusion::{
- execution::{options::ParquetReadOptions, SessionStateBuilder},
- prelude::{SessionConfig, SessionContext},
-};
-
-#[tokio::main]
-async fn main() -> Result<()> {
- let config = SessionConfig::new_with_ballista()
- .with_target_partitions(1)
- .with_ballista_standalone_parallelism(2);
-
- let state = SessionStateBuilder::new()
- .with_config(config)
- .with_default_features()
- .build();
-
- let ctx = SessionContext::standalone_with_state(state).await?;
-
- let test_data = test_util::examples_test_data();
-
- // register parquet file with the execution context
- ctx.register_parquet(
- "test",
- &format!("{test_data}/alltypes_plain.parquet"),
- ParquetReadOptions::default(),
- )
- .await?;
-
- let df = ctx.sql("select count(1) from test").await?;
-
- df.show().await?;
- Ok(())
-}
-
-```
+Source:
[`examples/examples/standalone-sql.rs`](https://github.com/apache/datafusion-ballista/blob/main/examples/examples/standalone-sql.rs)
```bash
-cargo run --example standalone-substrait
--features="ballista/standalone","substrait"
+cargo run --example standalone-substrait
--features="ballista-scheduler/substrait"
Review Comment:
Yes, unfortunately. `required-features` only stops Cargo from building the
target without those features; it doesn't turn them on. Without the flag you
get:
```
error: target `standalone-substrait` in package `ballista-examples` requires
the features: `ballista-scheduler/substrait`
Consider enabling them by passing, e.g.,
`--features="ballista-scheduler/substrait"`
```
The standalone examples are the exception, since `ballista/standalone` is on
by default, which is why I dropped the flag from those. It's a fair question
though, so I've added a line under the table explaining it.
##########
examples/README.md:
##########
@@ -27,308 +27,80 @@ The standalone example is the easiest to get started with.
Ballista supports a s
and executor are started in-process.
```bash
-cargo run --example standalone_sql --features="ballista/standalone"
+cargo run --example standalone_sql
```
-### Source code for standalone SQL example
-
-```rust
-use ballista::{
- extension::SessionConfigExt,
- prelude::*
-};
-use datafusion::{
- execution::{options::ParquetReadOptions, SessionStateBuilder},
- prelude::{SessionConfig, SessionContext},
-};
-
-#[tokio::main]
-async fn main() -> Result<()> {
- let config = SessionConfig::new_with_ballista()
- .with_target_partitions(1)
- .with_ballista_standalone_parallelism(2);
-
- let state = SessionStateBuilder::new()
- .with_config(config)
- .with_default_features()
- .build();
-
- let ctx = SessionContext::standalone_with_state(state).await?;
-
- let test_data = test_util::examples_test_data();
-
- // register parquet file with the execution context
- ctx.register_parquet(
- "test",
- &format!("{test_data}/alltypes_plain.parquet"),
- ParquetReadOptions::default(),
- )
- .await?;
-
- let df = ctx.sql("select count(1) from test").await?;
-
- df.show().await?;
- Ok(())
-}
-
-```
+Source:
[`examples/examples/standalone-sql.rs`](https://github.com/apache/datafusion-ballista/blob/main/examples/examples/standalone-sql.rs)
```bash
-cargo run --example standalone-substrait
--features="ballista/standalone","substrait"
+cargo run --example standalone-substrait
--features="ballista-scheduler/substrait"
```
-### Source code for standalone Substrait example
-
-```rust
-use std::sync::Arc;
-
-use ballista::datafusion::common::Result;
-use ballista::extension::{Extension, SubstraitExec};
-use ballista_core::extension::SessionConfigExt;
-use ballista_examples::test_util;
-use datafusion::catalog::MemoryCatalogProviderList;
-use datafusion::execution::SessionStateBuilder;
-use datafusion::prelude::{SessionConfig, SessionContext};
-use datafusion_substrait::serializer::serialize_bytes;
-use futures::StreamExt;
-
-#[tokio::main]
-async fn main() -> Result<()> {
- let catalog_list = Arc::new(MemoryCatalogProviderList::new());
-
- let config = SessionConfig::new_with_ballista();
- let state = SessionStateBuilder::new()
- .with_config(config)
- .with_default_features()
- .with_catalog_list(catalog_list.clone())
- .build();
- let ctx = SessionContext::new_with_state(state.clone());
-
- // Use any frontend to serialize a Substrait plan
- let test_data = test_util::examples_test_data();
- let ddl_plan_bytes = serialize_bytes(&format!(
- "CREATE EXTERNAL TABLE IF NOT EXISTS another_data \
- STORED AS PARQUET \
- LOCATION '{}/alltypes_plain.parquet'",
- test_data
- ), &ctx).await?;
- let select_plan_bytes = serialize_bytes(
- "SELECT id, string_col FROM another_data",
- &ctx)
- .await?;
-
- let session_id = ctx.session_id();
- let scheduler_url = Extension::setup_standalone(Some(&state)).await?;
- let client = SubstraitSchedulerClient::new(scheduler_url,
session_id.to_string()).await?;
-
- client.execute_query(ddl_plan_bytes).await?;
- let mut stream = client.execute_query(select_plan_bytes).await?;
-
- let mut batch_count = 0;
- let mut total_rows = 0;
- while let Some(batch_result) = stream.next().await {
- let batch = batch_result?;
- batch_count += 1;
- total_rows += batch.num_rows();
-
- println!("Batch {}: {} rows", batch_count, batch.num_rows());
- println!("{:?}", batch);
- }
- println!("---------");
- println!("Query executed successfully!");
- println!("Total batches: {}, Total rows: {}", batch_count, total_rows);
-
- Ok(())
-}
-```
+Source:
[`examples/examples/standalone-substrait.rs`](https://github.com/apache/datafusion-ballista/blob/main/examples/examples/standalone-substrait.rs)
## Distributed Examples
For background information on the Ballista architecture, refer to
the [Ballista README](../ballista/client/README.md).
-### Start a standalone cluster
-
-From the root of the project, build release binaries.
-
-```bash
-cargo build --release
-```
-
-Start a Ballista scheduler process in a new terminal session.
-
-```bash
-RUST_LOG=info ./target/release/ballista-scheduler
-```
+### Start a cluster
-Start one or more Ballista executor processes in new terminal sessions. When
starting more than one
-executor, a unique port number must be specified for each executor.
-
-```bash
-RUST_LOG=info ./target/release/ballista-executor -c 2 -p 50051
-RUST_LOG=info ./target/release/ballista-executor -c 2 -p 50052
-```
+The distributed examples need a running scheduler and at least one executor.
Follow
+[Ballista
Quickstart](https://datafusion.apache.org/ballista/user-guide/deployment/quick-start.html)
+to build the binaries and start them, then come back here.
### Running the examples
-The examples can be run using the `cargo run --bin` syntax.
+The examples can be run using the `cargo run --example` syntax, from the
`examples` directory.
### Distributed SQL Example
```bash
cargo run --release --example remote-sql
```
-#### Source code for distributed SQL example
-
-```rust
-use ballista::{extension::SessionConfigExt, prelude::*};
-use datafusion::{
- execution::SessionStateBuilder,
- prelude::{CsvReadOptions, SessionConfig, SessionContext},
-};
-
-/// This example demonstrates executing a simple query against an Arrow data
source (CSV) and
-/// fetching results, using SQL
-#[tokio::main]
-async fn main() -> Result<()> {
- let config = SessionConfig::new_with_ballista()
- .with_target_partitions(4)
- .with_ballista_job_name("Remote SQL Example");
-
- let state = SessionStateBuilder::new()
- .with_config(config)
- .with_default_features()
- .build();
-
- let ctx = SessionContext::remote_with_state("df://localhost:50050",
state).await?;
-
- let test_data = test_util::examples_test_data();
-
- ctx.register_csv(
- "test",
- &format!("{test_data}/aggregate_test_100.csv"),
- CsvReadOptions::new(),
- )
- .await?;
-
- let df = ctx
- .sql(
- "SELECT c1, MIN(c12), MAX(c12) \
- FROM test \
- WHERE c11 > 0.1 AND c11 < 0.9 \
- GROUP BY c1",
- )
- .await?;
-
- df.show().await?;
-
- Ok(())
-}
-```
+Source:
[`examples/examples/remote-sql.rs`](https://github.com/apache/datafusion-ballista/blob/main/examples/examples/remote-sql.rs)
### Distributed DataFrame Example
```bash
cargo run --release --example remote-dataframe
```
-#### Source code for distributed DataFrame example
-
-```rust
-use ballista::{extension::SessionConfigExt, prelude::*};
-use datafusion::{
- execution::SessionStateBuilder,
- prelude::{col, lit, ParquetReadOptions, SessionConfig, SessionContext},
-};
-
-/// This example demonstrates executing a simple query against an Arrow data
source (Parquet) and
-/// fetching results, using the DataFrame trait
-#[tokio::main]
-async fn main() -> Result<()> {
- let config = SessionConfig::new_with_ballista().with_target_partitions(4);
-
- let state = SessionStateBuilder::new()
- .with_config(config)
- .with_default_features()
- .build();
-
- let ctx = SessionContext::remote_with_state("df://localhost:50050",
state).await?;
-
- let test_data = test_util::examples_test_data();
- let filename = format!("{test_data}/alltypes_plain.parquet");
-
- let df = ctx
- .read_parquet(filename, ParquetReadOptions::default())
- .await?
- .select_columns(&["id", "bool_col", "timestamp_col"])?
- .filter(col("id").gt(lit(1)))?;
-
- df.show().await?;
-
- Ok(())
-}
-```
+Source:
[`examples/examples/remote-dataframe.rs`](https://github.com/apache/datafusion-ballista/blob/main/examples/examples/remote-dataframe.rs)
### Distributed datafusion-spark example
+The scheduler and executor binaries must be built with the `spark-compat`
feature enabled for this
+example.
-The release binaries must be built with the spark-compat feature enabled for
this example.
```bash
cargo build --release --features spark-compat
```
```bash
-cargo run --release --example remote-spark-functions --features="spark-compat"
+cargo run --release --example remote-spark-functions
--features="ballista-core/spark-compat"
```
-#### Source code for distributed spark functions example
-
-```rust
-use ballista::datafusion::{
- common::Result,
- execution::SessionStateBuilder,
- prelude::{CsvReadOptions, SessionConfig, SessionContext},
-};
-use ballista::prelude::*;
-use ballista_examples::test_util;
-
-#[tokio::main]
-async fn main() -> Result<()> {
- let config = SessionConfig::new_with_ballista()
- .with_target_partitions(4)
- .with_ballista_job_name("Remote Datafusion Spark Example");
-
- let state = SessionStateBuilder::new()
- .with_config(config)
- .with_default_features()
- .build();
+Source:
[`examples/examples/remote-spark-functions.rs`](https://github.com/apache/datafusion-ballista/blob/main/examples/examples/remote-spark-functions.rs)
- let ctx = SessionContext::remote_with_state("df://localhost:50050",
state).await?;
+## All examples
- let test_data = test_util::examples_test_data();
+Every example in
[`examples/examples/`](https://github.com/apache/datafusion-ballista/tree/main/examples/examples),
+including those not walked through above. Run each from this directory.
- ctx.register_csv(
- "test",
- &format!("{test_data}/aggregate_test_100.csv"),
- CsvReadOptions::new(),
- )
- .await?;
+| Example | Needs
| Command
|
+| ------------------------------ |
----------------------------------------------------------------------------------
|
------------------------------------------------------------------------------------
|
+| `standalone-sql.rs` | nothing, runs in-process
| `cargo run --example standalone_sql`
|
+| `standalone-broadcast-join.rs` | nothing, runs in-process
| `cargo run --example
standalone_broadcast_join` |
+| `standalone-substrait.rs` | nothing, runs in-process
| `cargo run --example
standalone-substrait --features="ballista-scheduler/substrait"` |
Review Comment:
Same answer as above: it's needed here too, as is `--features=tls` for
`mtls-cluster` and the `spark-compat` one for `remote-spark-functions`.
##########
examples/README.md:
##########
@@ -27,308 +27,80 @@ The standalone example is the easiest to get started with.
Ballista supports a s
and executor are started in-process.
```bash
-cargo run --example standalone_sql --features="ballista/standalone"
+cargo run --example standalone_sql
```
-### Source code for standalone SQL example
-
-```rust
-use ballista::{
- extension::SessionConfigExt,
- prelude::*
-};
-use datafusion::{
- execution::{options::ParquetReadOptions, SessionStateBuilder},
- prelude::{SessionConfig, SessionContext},
-};
-
-#[tokio::main]
-async fn main() -> Result<()> {
- let config = SessionConfig::new_with_ballista()
- .with_target_partitions(1)
- .with_ballista_standalone_parallelism(2);
-
- let state = SessionStateBuilder::new()
- .with_config(config)
- .with_default_features()
- .build();
-
- let ctx = SessionContext::standalone_with_state(state).await?;
-
- let test_data = test_util::examples_test_data();
-
- // register parquet file with the execution context
- ctx.register_parquet(
- "test",
- &format!("{test_data}/alltypes_plain.parquet"),
- ParquetReadOptions::default(),
- )
- .await?;
-
- let df = ctx.sql("select count(1) from test").await?;
-
- df.show().await?;
- Ok(())
-}
-
-```
+Source:
[`examples/examples/standalone-sql.rs`](https://github.com/apache/datafusion-ballista/blob/main/examples/examples/standalone-sql.rs)
```bash
-cargo run --example standalone-substrait
--features="ballista/standalone","substrait"
+cargo run --example standalone-substrait
--features="ballista-scheduler/substrait"
```
-### Source code for standalone Substrait example
-
-```rust
-use std::sync::Arc;
-
-use ballista::datafusion::common::Result;
-use ballista::extension::{Extension, SubstraitExec};
-use ballista_core::extension::SessionConfigExt;
-use ballista_examples::test_util;
-use datafusion::catalog::MemoryCatalogProviderList;
-use datafusion::execution::SessionStateBuilder;
-use datafusion::prelude::{SessionConfig, SessionContext};
-use datafusion_substrait::serializer::serialize_bytes;
-use futures::StreamExt;
-
-#[tokio::main]
-async fn main() -> Result<()> {
- let catalog_list = Arc::new(MemoryCatalogProviderList::new());
-
- let config = SessionConfig::new_with_ballista();
- let state = SessionStateBuilder::new()
- .with_config(config)
- .with_default_features()
- .with_catalog_list(catalog_list.clone())
- .build();
- let ctx = SessionContext::new_with_state(state.clone());
-
- // Use any frontend to serialize a Substrait plan
- let test_data = test_util::examples_test_data();
- let ddl_plan_bytes = serialize_bytes(&format!(
- "CREATE EXTERNAL TABLE IF NOT EXISTS another_data \
- STORED AS PARQUET \
- LOCATION '{}/alltypes_plain.parquet'",
- test_data
- ), &ctx).await?;
- let select_plan_bytes = serialize_bytes(
- "SELECT id, string_col FROM another_data",
- &ctx)
- .await?;
-
- let session_id = ctx.session_id();
- let scheduler_url = Extension::setup_standalone(Some(&state)).await?;
- let client = SubstraitSchedulerClient::new(scheduler_url,
session_id.to_string()).await?;
-
- client.execute_query(ddl_plan_bytes).await?;
- let mut stream = client.execute_query(select_plan_bytes).await?;
-
- let mut batch_count = 0;
- let mut total_rows = 0;
- while let Some(batch_result) = stream.next().await {
- let batch = batch_result?;
- batch_count += 1;
- total_rows += batch.num_rows();
-
- println!("Batch {}: {} rows", batch_count, batch.num_rows());
- println!("{:?}", batch);
- }
- println!("---------");
- println!("Query executed successfully!");
- println!("Total batches: {}, Total rows: {}", batch_count, total_rows);
-
- Ok(())
-}
-```
+Source:
[`examples/examples/standalone-substrait.rs`](https://github.com/apache/datafusion-ballista/blob/main/examples/examples/standalone-substrait.rs)
## Distributed Examples
For background information on the Ballista architecture, refer to
the [Ballista README](../ballista/client/README.md).
-### Start a standalone cluster
-
-From the root of the project, build release binaries.
-
-```bash
-cargo build --release
-```
-
-Start a Ballista scheduler process in a new terminal session.
-
-```bash
-RUST_LOG=info ./target/release/ballista-scheduler
-```
+### Start a cluster
-Start one or more Ballista executor processes in new terminal sessions. When
starting more than one
-executor, a unique port number must be specified for each executor.
-
-```bash
-RUST_LOG=info ./target/release/ballista-executor -c 2 -p 50051
-RUST_LOG=info ./target/release/ballista-executor -c 2 -p 50052
-```
+The distributed examples need a running scheduler and at least one executor.
Follow
+[Ballista
Quickstart](https://datafusion.apache.org/ballista/user-guide/deployment/quick-start.html)
+to build the binaries and start them, then come back here.
### Running the examples
-The examples can be run using the `cargo run --bin` syntax.
+The examples can be run using the `cargo run --example` syntax, from the
`examples` directory.
### Distributed SQL Example
```bash
cargo run --release --example remote-sql
```
-#### Source code for distributed SQL example
-
-```rust
-use ballista::{extension::SessionConfigExt, prelude::*};
-use datafusion::{
- execution::SessionStateBuilder,
- prelude::{CsvReadOptions, SessionConfig, SessionContext},
-};
-
-/// This example demonstrates executing a simple query against an Arrow data
source (CSV) and
-/// fetching results, using SQL
-#[tokio::main]
-async fn main() -> Result<()> {
- let config = SessionConfig::new_with_ballista()
- .with_target_partitions(4)
- .with_ballista_job_name("Remote SQL Example");
-
- let state = SessionStateBuilder::new()
- .with_config(config)
- .with_default_features()
- .build();
-
- let ctx = SessionContext::remote_with_state("df://localhost:50050",
state).await?;
-
- let test_data = test_util::examples_test_data();
-
- ctx.register_csv(
- "test",
- &format!("{test_data}/aggregate_test_100.csv"),
- CsvReadOptions::new(),
- )
- .await?;
-
- let df = ctx
- .sql(
- "SELECT c1, MIN(c12), MAX(c12) \
- FROM test \
- WHERE c11 > 0.1 AND c11 < 0.9 \
- GROUP BY c1",
- )
- .await?;
-
- df.show().await?;
-
- Ok(())
-}
-```
+Source:
[`examples/examples/remote-sql.rs`](https://github.com/apache/datafusion-ballista/blob/main/examples/examples/remote-sql.rs)
### Distributed DataFrame Example
```bash
cargo run --release --example remote-dataframe
```
-#### Source code for distributed DataFrame example
-
-```rust
-use ballista::{extension::SessionConfigExt, prelude::*};
-use datafusion::{
- execution::SessionStateBuilder,
- prelude::{col, lit, ParquetReadOptions, SessionConfig, SessionContext},
-};
-
-/// This example demonstrates executing a simple query against an Arrow data
source (Parquet) and
-/// fetching results, using the DataFrame trait
-#[tokio::main]
-async fn main() -> Result<()> {
- let config = SessionConfig::new_with_ballista().with_target_partitions(4);
-
- let state = SessionStateBuilder::new()
- .with_config(config)
- .with_default_features()
- .build();
-
- let ctx = SessionContext::remote_with_state("df://localhost:50050",
state).await?;
-
- let test_data = test_util::examples_test_data();
- let filename = format!("{test_data}/alltypes_plain.parquet");
-
- let df = ctx
- .read_parquet(filename, ParquetReadOptions::default())
- .await?
- .select_columns(&["id", "bool_col", "timestamp_col"])?
- .filter(col("id").gt(lit(1)))?;
-
- df.show().await?;
-
- Ok(())
-}
-```
+Source:
[`examples/examples/remote-dataframe.rs`](https://github.com/apache/datafusion-ballista/blob/main/examples/examples/remote-dataframe.rs)
### Distributed datafusion-spark example
+The scheduler and executor binaries must be built with the `spark-compat`
feature enabled for this
+example.
-The release binaries must be built with the spark-compat feature enabled for
this example.
```bash
cargo build --release --features spark-compat
```
```bash
-cargo run --release --example remote-spark-functions --features="spark-compat"
+cargo run --release --example remote-spark-functions
--features="ballista-core/spark-compat"
```
-#### Source code for distributed spark functions example
-
-```rust
-use ballista::datafusion::{
- common::Result,
- execution::SessionStateBuilder,
- prelude::{CsvReadOptions, SessionConfig, SessionContext},
-};
-use ballista::prelude::*;
-use ballista_examples::test_util;
-
-#[tokio::main]
-async fn main() -> Result<()> {
- let config = SessionConfig::new_with_ballista()
- .with_target_partitions(4)
- .with_ballista_job_name("Remote Datafusion Spark Example");
-
- let state = SessionStateBuilder::new()
- .with_config(config)
- .with_default_features()
- .build();
+Source:
[`examples/examples/remote-spark-functions.rs`](https://github.com/apache/datafusion-ballista/blob/main/examples/examples/remote-spark-functions.rs)
- let ctx = SessionContext::remote_with_state("df://localhost:50050",
state).await?;
+## All examples
- let test_data = test_util::examples_test_data();
+Every example in
[`examples/examples/`](https://github.com/apache/datafusion-ballista/tree/main/examples/examples),
+including those not walked through above. Run each from this directory.
- ctx.register_csv(
- "test",
- &format!("{test_data}/aggregate_test_100.csv"),
- CsvReadOptions::new(),
- )
- .await?;
+| Example | Needs
| Command
|
Review Comment:
I don't think it would help here. In `docs/` it only sits above the
generated config tables, where `update_config_docs.rs` emits it so prettier and
the generator don't reformat each other's output. This table is hand-written
and already aligned the way prettier would, and `examples/README.md` isn't in
the prettier CI scope (`{ballista,docs}/**/*.md`, `README.md`,
`CONTRIBUTING.md`) anyway.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
