This is an automated email from the ASF dual-hosted git repository.
HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git
The following commit(s) were added to refs/heads/master by this push:
new 42e207e [SPARK-61007][CONNECT][PYTHON] Support the subquery
expressions of scalar, exists, isin and asTable as PySpark does
42e207e is described below
commit 42e207e5fd63890fe67d0c4450c0d2035a62834c
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Fri Nov 6 23:34:37 2026 +0900
[SPARK-61007][CONNECT][PYTHON] Support the subquery expressions of scalar,
exists, isin and asTable as PySpark does
### What changes were proposed in this pull request?
[SPARK-61007](https://issues.apache.org/jira/browse/SPARK-61007): the
Python client now supports PySpark's subquery expressions.
- `DataFrame.scalar()` and `DataFrame.exists()` return the Columns of a
scalar and an EXISTS subquery of the DataFrame.
- `Column.isin(df)` returns that of an IN subquery, of the fields of a
`struct(...)` for one.
- A UDTF call takes the `TableArg` of `DataFrame.asTable()`, partitioned
and ordered as given, as an argument that passes the rows of the DataFrame.
These are `SubqueryExpression`s of the plan of the DataFrame, by its plan
ID. As PySpark's `LogicalPlan._with_relations` does, a relation whose
expressions have subqueries is sent in a `WithRelations` with the plans that
they refer to:
```
with_relations [plan ID of the relation]
root: the relation [its root plan ID]
references: the plan of each subquery [its plan ID]
```
Each relation collects the subqueries of its own expressions as it is
serialized, so this holds for every operator, and for nested subqueries, which
are sent in the relations that have them. The plan IDs stay stable, as for any
plan.
As PySpark's `TableArg`, `partitionBy()`, `orderBy()` and
`withSinglePartition()` in the wrong order raise `IllegalArgumentException`.
In the core, `DataFrame::scalar()` and `DataFrame::exists()` return the
Columns too, as PySpark's do, and `Column::isin_subquery` is the IN subquery.
This is based on #209, whose commits are the first ones here.
### Why are the changes needed?
`scalar()` ran the query and returned its first value, and `exists()`
returned a bool, so the subqueries of PySpark failed. For example,
`t1.where(t2.where(col("t2.a") == col("t1.a").outer()).exists())` raised
`UNRESOLVED_COLUMN`. `isin(df)` raised `UNSUPPORTED_DATA_TYPE`, and so did a
UDTF call with a `TableArg`.
### Does this PR introduce _any_ user-facing change?
Yes. `scalar()`, `exists()`, `isin(df)` and `asTable()` work as in PySpark
4.3.0. In the Rust API, `DataFrame::scalar()` and `DataFrame::exists()` now
return Columns of the subqueries.
### How was this patch tested?
- A new live test runs scalar, correlated and IN subqueries, an IN subquery
of a struct, and a UDTF call with the rows of a single, ordered partition and
with a named table argument. It also checks the errors of the `TableArg`
builders. It passes with official PySpark 4.3.0, and fails without this change.
- A new offline test checks, against a mock Connect server, the
`WithRelations` of a relation with subqueries, its plan IDs, and the subqueries
and table argument that it sends. It fails without this change.
- New lib tests check the relations of subqueries, nested ones included,
the stability of their plan IDs, and their rendering. A new e2e test checks the
rows of subqueries from Rust. The tests of the `TableArg` builders and of
`scalar()` and `exists()` now expect PySpark's.
- The lib tests and every integration test, e2e included, against a local
Spark 4.3.0 server (669 passed, and 125 of the transport crate), the build of
all tests with `wasm-udf`, the offline Python suites (1162 passed, 8 skipped),
and the live drop-in and Arrow suites (89 passed) and live cases (10 passed).
Clippy reports no diagnostics in the changed lines.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
This pull request and its description were written by Isaac.
Closes #210 from HyukjinKwon/SPARK-61007-subqueries.
Authored-by: Hyukjin Kwon <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
---
crates/pyspark-rs/src/column.rs | 11 ++
crates/pyspark-rs/src/dataframe.rs | 17 +-
crates/pyspark-rs/src/functions.rs | 20 ++-
crates/spark-connect/src/column.rs | 17 ++
crates/spark-connect/src/dataframe.rs | 189 ++++++++++++++++++++--
crates/spark-connect/src/expression.rs | 171 ++++++++++++++++++++
crates/spark-connect/src/group.rs | 10 +-
crates/spark-connect/src/plan.rs | 106 +++++++++++-
crates/spark-connect/src/table_arg.rs | 223 +++++++++++---------------
crates/spark-connect/tests/e2e_dataframe.rs | 86 +++++++++-
crates/spark-connect/tests/e2e_new_methods.rs | 20 ++-
crates/spark-connect/tests/e2e_ops_breadth.rs | 2 +-
python/tests/test_dropin_e2e.py | 39 +++++
python/tests/test_dropin_offline.py | 61 +++++++
14 files changed, 795 insertions(+), 177 deletions(-)
diff --git a/crates/pyspark-rs/src/column.rs b/crates/pyspark-rs/src/column.rs
index d60c0d6..e78334a 100644
--- a/crates/pyspark-rs/src/column.rs
+++ b/crates/pyspark-rs/src/column.rs
@@ -631,6 +631,17 @@ impl PyColumn {
}
#[pyo3(signature = (*cols))]
fn isin(&self, py: Python<'_>, cols: Vec<Py<PyAny>>) -> PyResult<PyColumn>
{
+ // As pyspark's, a single DataFrame makes an IN subquery of it.
+ if let [only] = cols.as_slice() {
+ if let Ok(df) = only
+ .bind(py)
+ .extract::<PyRef<'_, crate::dataframe::PyDataFrame>>()
+ {
+ return Ok(PyColumn::new(
+ self.column.clone().isin_subquery(&df.dataframe),
+ ));
+ }
+ }
// pyspark's Column.isin unpacks a single list/tuple/set argument:
// col.isin([1, 2]) == col.isin(1, 2). pandas-on-Spark's Series.isin
relies on
// this (it passes a single list of lit(..) columns).
diff --git a/crates/pyspark-rs/src/dataframe.rs
b/crates/pyspark-rs/src/dataframe.rs
index 1bac91a..de0dfc6 100644
--- a/crates/pyspark-rs/src/dataframe.rs
+++ b/crates/pyspark-rs/src/dataframe.rs
@@ -10,7 +10,7 @@ use crate::column::PyColumn;
use crate::errors::ResultExt;
use crate::functions::to_column;
use crate::group::PyGroupedData;
-use crate::row::{value_to_py, FieldConverters, PyRow};
+use crate::row::{FieldConverters, PyRow};
use crate::streaming::PyDataStreamWriter;
use crate::types::PyDataType;
@@ -1116,17 +1116,14 @@ impl PyDataFrame {
))
}
- /// Whether the (scalar-subquery) DataFrame would return at least one row.
- fn exists(&self, py: Python<'_>) -> PyResult<bool> {
- py.detach(|| self.dataframe.exists()).to_pyerr()
+ /// The EXISTS subquery of this DataFrame, as a Column. Mirrors
`DataFrame.exists()`.
+ fn exists(&self) -> PyColumn {
+ PyColumn::new(self.dataframe.exists())
}
- /// Return the single scalar value produced by this (scalar-subquery)
DataFrame.
- fn scalar(&self, py: Python<'_>) -> PyResult<Py<PyAny>> {
- match py.detach(|| self.dataframe.scalar()).to_pyerr()? {
- Some(v) => Ok(value_to_py(py, &v)?.unbind()),
- None => Ok(py.None()),
- }
+ /// The scalar subquery of this DataFrame, as a Column. Mirrors
`DataFrame.scalar()`.
+ fn scalar(&self) -> PyColumn {
+ PyColumn::new(self.dataframe.scalar())
}
/// Return a `TableArg` wrapping this DataFrame, for use as a table-valued
function
diff --git a/crates/pyspark-rs/src/functions.rs
b/crates/pyspark-rs/src/functions.rs
index 2952a07..763812c 100644
--- a/crates/pyspark-rs/src/functions.rs
+++ b/crates/pyspark-rs/src/functions.rs
@@ -44,7 +44,7 @@ fn pyfunc_make_udf(
let return_data_type = crate::types::py_to_data_type(return_type)?;
// The positional arguments, then the named ones, as PySpark's UDF call.
- let arg_exprs: Vec<Expression> = udf_arguments(args, named_args)?
+ let arg_exprs: Vec<Expression> = udf_arguments(args, named_args, false)?
.into_iter()
.map(|c| c.expression().clone())
.collect();
@@ -63,17 +63,27 @@ fn pyfunc_make_udf(
/// The arguments of a UDF or UDTF call: the positional ones, then a
`NamedArgumentExpression`
/// of each keyword argument, as PySpark's `UserDefinedFunction` and
`UserDefinedTableFunction`
-/// build them.
+/// build them. As PySpark's, a UDTF call also takes a `TableArg`, which
passes the rows of
+/// its DataFrame.
fn udf_arguments(
args: Vec<Bound<'_, PyAny>>,
named_args: Vec<(String, Bound<'_, PyAny>)>,
+ table_args: bool,
) -> PyResult<Vec<Column>> {
+ let argument = |arg: &Bound<'_, PyAny>| -> PyResult<Column> {
+ if table_args {
+ if let Ok(table_arg) = arg.extract::<PyRef<'_,
crate::tablearg::PyTableArg>>() {
+ return Ok(table_arg.inner.to_column());
+ }
+ }
+ to_column(arg)
+ };
let mut columns = Vec::with_capacity(args.len() + named_args.len());
for arg in args {
- columns.push(to_column(&arg)?);
+ columns.push(argument(&arg)?);
}
for (key, value) in named_args {
- let value = to_column(&value)?.expression().clone();
+ let value = argument(&value)?.expression().clone();
columns.push(Column::new(
ExpressionNode::NamedArgument(Box::new(NamedArgument::new(key,
value))).into(),
));
@@ -106,7 +116,7 @@ fn pyfunc_make_udtf(
deterministic: bool,
named_args: Vec<(String, Bound<'_, PyAny>)>,
) -> PyResult<PyDataFrame> {
- let col_args = udf_arguments(args, named_args)?;
+ let col_args = udf_arguments(args, named_args, true)?;
// Connect leaves DDL parsing and validation to the server. In particular,
// do not use py_to_data_type's eager DDL parser for string return types.
let rt = return_type
diff --git a/crates/spark-connect/src/column.rs
b/crates/spark-connect/src/column.rs
index ab736ce..6729036 100644
--- a/crates/spark-connect/src/column.rs
+++ b/crates/spark-connect/src/column.rs
@@ -231,6 +231,23 @@ impl Column {
}
}
+ /// The IN subquery of whether this column is in the rows of `df`, of one
column; of a
+ /// `struct(...)` of columns, whether they are in the rows of `df` of as
many columns.
+ /// Mirrors `Column.isin(df)`.
+ pub fn isin_subquery(self, df: &crate::dataframe::DataFrame) -> Column {
+ let values = match self.expr.node() {
+ ExpressionNode::UnresolvedFunction(function) if function.name ==
"struct" => {
+ function.args.clone()
+ }
+ _ => vec![self.expr],
+ };
+ let mut subquery = df.subquery(crate::expression::SubqueryType::In);
+ subquery.in_subquery_values = values;
+ Column {
+ expr: ExpressionNode::Subquery(Box::new(subquery)).into(),
+ }
+ }
+
/// Mirrors `pyspark.sql.column.Column.withField`: add or replace a field
in
/// a struct-typed column.
pub fn with_field(self, field_name: &str, value: Column) -> Column {
diff --git a/crates/spark-connect/src/dataframe.rs
b/crates/spark-connect/src/dataframe.rs
index 1fd7d8e..ebd9385 100644
--- a/crates/spark-connect/src/dataframe.rs
+++ b/crates/spark-connect/src/dataframe.rs
@@ -1925,19 +1925,38 @@ impl DataFrame {
Self::new_with_schema(self.session.clone(), plan, Some(schema))
}
- /// Check if the DataFrame exists (is not empty).
- pub fn exists(&self) -> Result<bool> {
- self.limit(1).count().map(|c| c > 0)
+ /// A subquery of `subquery_type` of this DataFrame's plan, which refers
to the plan by
+ /// its plan ID in the session.
+ pub(crate) fn subquery(
+ &self,
+ subquery_type: crate::expression::SubqueryType,
+ ) -> crate::expression::SubqueryExpression {
+ crate::expression::SubqueryExpression::new(
+ Arc::clone(&self.plan),
+ self.session.plan_ids().id_of(&self.plan),
+ subquery_type,
+ )
+ }
+
+ /// The scalar subquery of this DataFrame, which returns one row of one
column, as a
+ /// Column of that value. Mirrors `DataFrame.scalar()`. A column of the
outer query is
+ /// referred to as `col(...).outer()`.
+ pub fn scalar(&self) -> Column {
+ let subquery = self.subquery(crate::expression::SubqueryType::Scalar);
+ Column::new(ExpressionNode::Subquery(Box::new(subquery)).into())
+ }
+
+ /// The EXISTS subquery of this DataFrame, as a Column of whether it
returns a row.
+ /// Mirrors `DataFrame.exists()`.
+ pub fn exists(&self) -> Column {
+ let subquery = self.subquery(crate::expression::SubqueryType::Exists);
+ Column::new(ExpressionNode::Subquery(Box::new(subquery)).into())
}
- /// Get a scalar value from a single-row, single-column result.
- pub fn scalar(&self) -> Result<Option<Value>> {
+ /// The first value of the first row, if any.
+ pub(crate) fn first_value(&self) -> Result<Option<Value>> {
let rows = self.limit(1).collect()?;
- if rows.is_empty() {
- return Ok(None);
- }
- let row = &rows[0];
- Ok(row.get(0).cloned())
+ Ok(rows.first().and_then(|row| row.get(0).cloned()))
}
/// Transpose the DataFrame: swap rows and columns (server-side `Transpose`
@@ -4498,3 +4517,153 @@ mod plan_id_column_tests {
);
}
}
+
+#[cfg(test)]
+mod subquery_tests {
+ use super::*;
+ use crate::column::col;
+ use crate::functions::max;
+ use crate::session::SparkSession;
+ use proto::relation::RelType;
+
+ fn session() -> SparkSession {
+ SparkSession::builder()
+ .remote("sc://localhost:15002")
+ .get_or_create()
+ .expect("failed to build session")
+ }
+
+ fn plan_id(relation: &proto::Relation) -> Option<i64> {
+ relation.common.as_ref().and_then(|common| common.plan_id)
+ }
+
+ /// The root and the references of a `WithRelations`.
+ fn with_relations(relation: &proto::Relation) -> (&proto::Relation,
&[proto::Relation]) {
+ let Some(RelType::WithRelations(with)) = &relation.rel_type else {
+ panic!("expected WithRelations: {relation:?}");
+ };
+ (with.root.as_deref().expect("root"), &with.references)
+ }
+
+ fn subquery(expr: &proto::Expression) -> &proto::SubqueryExpression {
+ let Some(proto::expression::ExprType::SubqueryExpression(subquery)) =
&expr.expr_type
+ else {
+ panic!("expected a SubqueryExpression: {expr:?}");
+ };
+ subquery
+ }
+
+ /// As PySpark's `_with_relations`: a relation with subqueries is a
`WithRelations` of
+ /// its own plan ID, whose root has another, and whose references are the
plans of the
+ /// subqueries, with the plan IDs that the subqueries refer to.
+ #[test]
+ fn subqueries_are_sent_with_the_plans_that_they_refer_to() {
+ let spark = session();
+ let inner = spark.range(3).unwrap().select(vec![max(col("id"))]);
+ let df = spark
+ .range(2)
+ .unwrap()
+ .select(vec![col("id"), inner.scalar().alias("m")]);
+
+ let relation = df.relation();
+ let (root, references) = with_relations(&relation);
+ assert_eq!(plan_id(&relation), Some(spark.plan_ids().id_of(&df.plan)));
+ assert!(plan_id(root).is_some() && plan_id(root) !=
plan_id(&relation));
+ let Some(RelType::Project(project)) = &root.rel_type else {
+ panic!("expected Project: {root:?}");
+ };
+ let Some(proto::expression::ExprType::Alias(alias)) =
&project.expressions[1].expr_type
+ else {
+ panic!("expected an alias");
+ };
+ let scalar = subquery(alias.expr.as_deref().unwrap());
+ assert_eq!(
+ scalar.subquery_type,
+ proto::subquery_expression::SubqueryType::Scalar as i32
+ );
+ assert_eq!(references.len(), 1);
+ assert_eq!(plan_id(&references[0]), Some(scalar.plan_id));
+ assert_eq!(references[0], inner.relation());
+ // The plan IDs are stable, as PySpark's.
+ assert_eq!(df.relation(), relation);
+ // Without plan IDs, the relation is sent the same way.
+ let Some(RelType::WithRelations(with)) = df.plan.to_proto().rel_type
else {
+ panic!("expected WithRelations");
+ };
+ assert_eq!(with.references.len(), 1);
+ // A plan without subqueries is sent as it is.
+ assert!(matches!(
+ inner.relation().rel_type,
+ Some(RelType::Project(_))
+ ));
+ }
+
+ /// The relations below a relation with subqueries, and the plans that
they refer to,
+ /// send their own subqueries in their own `WithRelations`.
+ #[test]
+ fn nested_subqueries_are_sent_in_their_own_relations() {
+ let spark = session();
+ let t1 = spark.range(4).unwrap();
+ let t2 = spark.range(2).unwrap();
+ let smallest = t2.select(vec![crate::functions::min(col("id"))]);
+ let filtered = t2.filter(col("id").gt(smallest.scalar()));
+ let df = t1
+ .filter(col("id").isin_subquery(&filtered))
+ .filter(t2.exists());
+
+ let relation = df.relation();
+ let (root, references) = with_relations(&relation);
+ let Some(RelType::Filter(outer)) = &root.rel_type else {
+ panic!("expected Filter: {root:?}");
+ };
+ let exists = subquery(outer.condition.as_ref().unwrap());
+ assert_eq!(references.len(), 1);
+ assert_eq!(plan_id(&references[0]), Some(exists.plan_id));
+
+ // The inner filter, with the IN subquery of the filtered plan, which
in turn has the
+ // scalar subquery of the smallest.
+ let (inner_root, inner_references) =
with_relations(outer.input.as_deref().unwrap());
+ let Some(RelType::Filter(inner)) = &inner_root.rel_type else {
+ panic!("expected Filter: {inner_root:?}");
+ };
+ let in_subquery = subquery(inner.condition.as_ref().unwrap());
+ assert_eq!(
+ in_subquery.subquery_type,
+ proto::subquery_expression::SubqueryType::In as i32
+ );
+ assert_eq!(
+ in_subquery.in_subquery_values,
+ vec![col("id").expression().to_proto()]
+ );
+ assert_eq!(inner_references.len(), 1);
+ assert_eq!(plan_id(&inner_references[0]), Some(in_subquery.plan_id));
+ let (filtered_root, filtered_references) =
with_relations(&inner_references[0]);
+ assert!(matches!(filtered_root.rel_type, Some(RelType::Filter(_))));
+ assert_eq!(
+ filtered_references,
+ std::slice::from_ref(&smallest.relation())
+ );
+ }
+
+ /// As PySpark's `Column.isin(df)`, the values of an IN subquery of a
struct are its
+ /// fields; and the subqueries render as PySpark's, with the plan ID of
their plan.
+ #[test]
+ fn in_subqueries_of_structs_compare_their_fields() {
+ let spark = session();
+ let df = spark.range(1).unwrap();
+ let id = spark.plan_ids().id_of(&df.plan);
+ let fields = crate::functions::r#struct(vec![col("a"),
col("b")]).isin_subquery(&df);
+ assert_eq!(
+ fields.expression().render(),
+ format!("SubqueryExpression(plan=<plan {id}>, type=in, values=[a,
b])")
+ );
+ assert_eq!(
+ df.scalar().expression().render(),
+ format!("SubqueryExpression(plan=<plan {id}>, type=scalar)")
+ );
+ assert_eq!(
+ df.exists().expression().render(),
+ format!("SubqueryExpression(plan=<plan {id}>, type=exists)")
+ );
+ }
+}
diff --git a/crates/spark-connect/src/expression.rs
b/crates/spark-connect/src/expression.rs
index 4731f87..e5f6365 100644
--- a/crates/spark-connect/src/expression.rs
+++ b/crates/spark-connect/src/expression.rs
@@ -7,6 +7,7 @@ use std::sync::Arc;
use spark_connect_proto as proto;
+use crate::plan::LogicalPlan;
use crate::types::DataType;
use crate::udf::CommonInlineUserDefinedFunctionExpression;
@@ -95,6 +96,174 @@ pub enum ExpressionNode {
/// `pyspark.sql.connect.expressions.NamedArgumentExpression` - a named
argument of a
/// function call, such as `a => 1` of a Python UDF called with `a=lit(1)`.
NamedArgument(Box<NamedArgument>),
+ /// `pyspark.sql.connect.expressions.SubqueryExpression` - a scalar,
EXISTS or IN
+ /// subquery of a DataFrame, or a table argument of a table-valued
function.
+ Subquery(Box<SubqueryExpression>),
+}
+
+/// The type of a [`SubqueryExpression`]. Mirrors the `subquery_type` of
+/// `pyspark.sql.connect.expressions.SubqueryExpression`.
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub enum SubqueryType {
+ /// A scalar subquery, of `DataFrame.scalar()`.
+ Scalar,
+ /// An EXISTS subquery, of `DataFrame.exists()`.
+ Exists,
+ /// A table argument, of `DataFrame.asTable()`.
+ TableArg,
+ /// An IN subquery, of `Column.isin(df)`.
+ In,
+}
+
+impl SubqueryType {
+ fn name(self) -> &'static str {
+ match self {
+ SubqueryType::Scalar => "scalar",
+ SubqueryType::Exists => "exists",
+ SubqueryType::TableArg => "table_arg",
+ SubqueryType::In => "in",
+ }
+ }
+}
+
+/// A subquery of the plan of a DataFrame. Mirrors
+/// `pyspark.sql.connect.expressions.SubqueryExpression`: it refers to the
plan by its plan
+/// ID, and the relation that contains it sends the plan in a `WithRelations`,
as PySpark's
+/// `LogicalPlan._with_relations` does.
+#[derive(Debug, Clone)]
+pub struct SubqueryExpression {
+ /// The plan of the subquery.
+ pub(crate) plan: Arc<LogicalPlan>,
+ /// The plan ID of `plan` in the session of its DataFrame.
+ pub(crate) plan_id: i64,
+ /// The type of the subquery.
+ pub subquery_type: SubqueryType,
+ /// The partitioning of a table argument.
+ pub partition_spec: Vec<Expression>,
+ /// The ordering of a table argument, within its partitions.
+ pub order_spec: Vec<Expression>,
+ /// Whether a table argument is a single partition, when set.
+ pub with_single_partition: Option<bool>,
+ /// The values of an IN subquery.
+ pub in_subquery_values: Vec<Expression>,
+}
+
+impl PartialEq for SubqueryExpression {
+ fn eq(&self, other: &Self) -> bool {
+ // Plans have no equality: the subqueries of the same plan are equal.
+ Arc::ptr_eq(&self.plan, &other.plan)
+ && self.plan_id == other.plan_id
+ && self.subquery_type == other.subquery_type
+ && self.partition_spec == other.partition_spec
+ && self.order_spec == other.order_spec
+ && self.with_single_partition == other.with_single_partition
+ && self.in_subquery_values == other.in_subquery_values
+ }
+}
+
+impl SubqueryExpression {
+ /// A subquery of `subquery_type` of `plan`, whose plan ID is `plan_id`.
+ pub(crate) fn new(plan: Arc<LogicalPlan>, plan_id: i64, subquery_type:
SubqueryType) -> Self {
+ SubqueryExpression {
+ plan,
+ plan_id,
+ subquery_type,
+ partition_spec: Vec::new(),
+ order_spec: Vec::new(),
+ with_single_partition: None,
+ in_subquery_values: Vec::new(),
+ }
+ }
+
+ /// As `SubqueryExpression.__repr__`, but with the plan ID of the plan,
which PySpark
+ /// shows as the address of its plan object.
+ fn render(&self) -> String {
+ let mut parts = vec![
+ format!("plan=<plan {}>", self.plan_id),
+ format!("type={}", self.subquery_type.name()),
+ ];
+ let list = |exprs: &[Expression]| {
+ let rendered: Vec<String> =
exprs.iter().map(Expression::render).collect();
+ format!("[{}]", rendered.join(", "))
+ };
+ match self.subquery_type {
+ SubqueryType::TableArg => {
+ if !self.partition_spec.is_empty() {
+ parts.push(format!("partition_spec={}",
list(&self.partition_spec)));
+ }
+ if !self.order_spec.is_empty() {
+ parts.push(format!("order_spec={}",
list(&self.order_spec)));
+ }
+ if let Some(single) = self.with_single_partition {
+ let single = if single { "True" } else { "False" };
+ parts.push(format!("with_single_partition={single}"));
+ }
+ }
+ SubqueryType::In => parts.push(format!("values={}",
list(&self.in_subquery_values))),
+ SubqueryType::Scalar | SubqueryType::Exists => {}
+ }
+ format!("SubqueryExpression({})", parts.join(", "))
+ }
+
+ /// As `SubqueryExpression.to_plan`. The relation being serialized sends
`plan` in its
+ /// `WithRelations`.
+ fn to_proto(&self) -> proto::Expression {
+ use proto::subquery_expression::SubqueryType as ProtoType;
+
+ crate::plan::refer_to(&self.plan);
+ let subquery_type = match self.subquery_type {
+ SubqueryType::Scalar => ProtoType::Scalar,
+ SubqueryType::Exists => ProtoType::Exists,
+ SubqueryType::TableArg => ProtoType::TableArg,
+ SubqueryType::In => ProtoType::In,
+ } as i32;
+ // As PySpark, which sets the options of a table argument only when
one is given.
+ let table_arg_options = (self.subquery_type == SubqueryType::TableArg
+ && (!self.partition_spec.is_empty()
+ || !self.order_spec.is_empty()
+ || self.with_single_partition.is_some()))
+ .then(|| proto::subquery_expression::TableArgOptions {
+ partition_spec: self
+ .partition_spec
+ .iter()
+ .map(Expression::to_proto)
+ .collect(),
+ order_spec: self.order_spec.iter().map(sort_order_proto).collect(),
+ with_single_partition: self.with_single_partition,
+ });
+ let in_subquery_values = match self.subquery_type {
+ SubqueryType::In => self
+ .in_subquery_values
+ .iter()
+ .map(Expression::to_proto)
+ .collect(),
+ _ => Vec::new(),
+ };
+ let subquery = proto::SubqueryExpression {
+ plan_id: self.plan_id,
+ subquery_type,
+ table_arg_options,
+ in_subquery_values,
+ };
+ proto::Expression {
+ expr_type:
Some(proto::expression::ExprType::SubqueryExpression(subquery)),
+ ..Default::default()
+ }
+ }
+}
+
+/// The sort order of an ordering expression of a table argument: an explicit
sort order
+/// as it is, and another expression ascending, nulls first, as PySpark's
`_sort_col`.
+fn sort_order_proto(expr: &Expression) -> proto::expression::SortOrder {
+ let wrapped = match expr.node() {
+ ExpressionNode::SortOrder(_) => expr.to_proto(),
+ _ => SortOrder::asc_nulls_first(expr.clone()).to_proto(),
+ };
+ match wrapped.expr_type {
+ Some(proto::expression::ExprType::SortOrder(sort)) => *sort,
+ // SortOrder::to_proto always yields a SortOrder expr_type.
+ _ => unreachable!("expected a SortOrder expression"),
+ }
}
/// A named argument of a function call. Mirrors
@@ -221,6 +390,7 @@ impl Expression {
}
// As `NamedArgumentExpression.__repr__`.
ExpressionNode::NamedArgument(arg) => format!("{} => {}", arg.key,
arg.value.render()),
+ ExpressionNode::Subquery(subquery) => subquery.render(),
}
}
@@ -292,6 +462,7 @@ impl Expression {
)),
..Default::default()
},
+ ExpressionNode::Subquery(subquery) => subquery.to_proto(),
}
}
}
diff --git a/crates/spark-connect/src/group.rs
b/crates/spark-connect/src/group.rs
index 33d5f0a..f4619fb 100644
--- a/crates/spark-connect/src/group.rs
+++ b/crates/spark-connect/src/group.rs
@@ -421,7 +421,10 @@ impl StatFunctions {
col2: col2.to_string(),
};
let df = DataFrame::new(self.dataframe.session.clone(), plan);
- Ok(df.scalar()?.and_then(|v| v.as_f64()).unwrap_or(f64::NAN))
+ Ok(df
+ .first_value()?
+ .and_then(|v| v.as_f64())
+ .unwrap_or(f64::NAN))
}
/// Compute covariance between two columns.
@@ -434,7 +437,10 @@ impl StatFunctions {
col2: col2.to_string(),
};
let df = DataFrame::new(self.dataframe.session.clone(), plan);
- Ok(df.scalar()?.and_then(|v| v.as_f64()).unwrap_or(f64::NAN))
+ Ok(df
+ .first_value()?
+ .and_then(|v| v.as_f64())
+ .unwrap_or(f64::NAN))
}
/// Sample by values in a column.
diff --git a/crates/spark-connect/src/plan.rs b/crates/spark-connect/src/plan.rs
index 0b11008..fbc33a7 100644
--- a/crates/spark-connect/src/plan.rs
+++ b/crates/spark-connect/src/plan.rs
@@ -472,7 +472,18 @@ impl PlanIds {
/// The plan ID of `node`, assigned the first time it is serialized.
pub(crate) fn id_of(&self, node: &Arc<LogicalPlan>) -> i64 {
- let key = Arc::as_ptr(node) as usize;
+ self.id_of_key(node, Arc::as_ptr(node) as usize)
+ }
+
+ /// The plan ID of the relation of `node` within the `WithRelations` that
sends it with the
+ /// plans of its subqueries, as PySpark's `_root_plan_id`: `id_of(node)`
is that of the
+ /// `WithRelations`, as PySpark's `_plan_id`. A node is aligned, so its
address with the
+ /// low bit set is the key of no other node.
+ pub(crate) fn root_id_of(&self, node: &Arc<LogicalPlan>) -> i64 {
+ self.id_of_key(node, Arc::as_ptr(node) as usize | 1)
+ }
+
+ fn id_of_key(&self, node: &Arc<LogicalPlan>, key: usize) -> i64 {
let mut nodes = self.nodes.lock().unwrap_or_else(|e| e.into_inner());
// The entry's `Weak` keeps its allocation from being freed, so a live
node at this
// address is the node the entry was made for.
@@ -503,17 +514,100 @@ pub(crate) fn relation_with_ids(node: &Arc<LogicalPlan>,
ids: &PlanIds) -> proto
/// Serialize a child node, with its stable plan ID when `ids` is given.
fn relation_of(node: &Arc<LogicalPlan>, ids: Option<&PlanIds>) ->
proto::Relation {
- let mut relation = node.to_proto_with(ids);
- if let Some(ids) = ids {
- relation.common.get_or_insert_with(Default::default).plan_id =
Some(ids.id_of(node));
+ relation_with_references(node, Some(node), ids)
+}
+
+thread_local! {
+ /// For each relation being serialized, innermost last, the plans that the
subquery
+ /// expressions of its own expressions refer to.
+ static SUBQUERY_REFERENCES: std::cell::RefCell<Vec<Vec<Arc<LogicalPlan>>>>
=
+ const { std::cell::RefCell::new(Vec::new()) };
+}
+
+/// Record that a subquery expression being serialized refers to `plan`, which
the relation
+/// that contains the expression sends in its `WithRelations`.
+pub(crate) fn refer_to(plan: &Arc<LogicalPlan>) {
+ SUBQUERY_REFERENCES.with(|frames| {
+ if let Some(frame) = frames.borrow_mut().last_mut() {
+ frame.push(Arc::clone(plan));
+ }
+ });
+}
+
+/// The relation of `node`, and the plans that the subqueries of its own
expressions refer
+/// to; its children collect those of theirs as they are serialized.
+fn relation_and_references(
+ node: &LogicalPlan,
+ ids: Option<&PlanIds>,
+) -> (proto::Relation, Vec<Arc<LogicalPlan>>) {
+ struct Frame;
+ impl Drop for Frame {
+ fn drop(&mut self) {
+ SUBQUERY_REFERENCES.with(|frames| {
+ frames.borrow_mut().pop();
+ });
+ }
+ }
+ SUBQUERY_REFERENCES.with(|frames| frames.borrow_mut().push(Vec::new()));
+ let _frame = Frame;
+ let relation = node.to_proto_with(ids);
+ let references = SUBQUERY_REFERENCES.with(|frames| {
+ frames
+ .borrow_mut()
+ .last_mut()
+ .map(std::mem::take)
+ .unwrap_or_default()
+ });
+ (relation, references)
+}
+
+/// The relation of `node`, with the stable plan ID of `shared` when `ids` is
given. As
+/// PySpark's `LogicalPlan._with_relations`, a node whose expressions have
subqueries is sent
+/// in a `WithRelations` with the plans that they refer to:
+///
+/// ```text
+/// with_relations [plan ID of the node]
+/// root: the relation of the node [its root plan ID]
+/// references: the relation of each plan [its plan ID]
+/// ```
+fn relation_with_references(
+ node: &LogicalPlan,
+ shared: Option<&Arc<LogicalPlan>>,
+ ids: Option<&PlanIds>,
+) -> proto::Relation {
+ let (mut relation, references) = relation_and_references(node, ids);
+ let plan_ids = ids.zip(shared);
+ if references.is_empty() {
+ if let Some((ids, shared)) = plan_ids {
+ relation.common.get_or_insert_with(Default::default).plan_id =
Some(ids.id_of(shared));
+ }
+ return relation;
+ }
+ if let Some((ids, shared)) = plan_ids {
+ relation.common.get_or_insert_with(Default::default).plan_id =
Some(ids.root_id_of(shared));
+ }
+ let with_relations = proto::WithRelations {
+ root: Some(Box::new(relation)),
+ references: references
+ .iter()
+ .map(|reference| relation_of(reference, ids))
+ .collect(),
+ };
+ proto::Relation {
+ common: Some(proto::RelationCommon {
+ plan_id: plan_ids.map(|(ids, shared)| ids.id_of(shared)),
+ ..Default::default()
+ }),
+ rel_type: Some(proto::relation::RelType::WithRelations(Box::new(
+ with_relations,
+ ))),
}
- relation
}
impl LogicalPlan {
/// Convert this logical plan to a `spark.connect.Relation` protobuf,
without plan IDs.
pub fn to_proto(&self) -> proto::Relation {
- self.to_proto_with(None)
+ relation_with_references(self, None, None)
}
fn to_proto_with(&self, plan_ids: Option<&PlanIds>) -> proto::Relation {
diff --git a/crates/spark-connect/src/table_arg.rs
b/crates/spark-connect/src/table_arg.rs
index 4404d51..3f15f7b 100644
--- a/crates/spark-connect/src/table_arg.rs
+++ b/crates/spark-connect/src/table_arg.rs
@@ -2,57 +2,55 @@
//!
//! Provides table argument support for table-valued functions with
partitioning and ordering.
-use std::sync::Arc;
-
use crate::column::Column;
use crate::dataframe::DataFrame;
-use crate::expression::{Expression, ExpressionNode, SortOrder};
+use crate::expression::{ExpressionNode, SubqueryExpression, SubqueryType};
+#[cfg(test)]
use crate::plan::LogicalPlan;
-use spark_connect_core::error::{Result, SparkError};
-use spark_connect_proto as proto;
+use spark_connect_core::error::{Result, SparkError, SparkErrorKind};
+
+/// The `IllegalArgumentException` of `message`, as PySpark's `TableArg`
raises for a
+/// partitioning or ordering in the wrong order.
+fn illegal_argument(message: &str) -> SparkError {
+ let mut error = SparkError::value_msg(message);
+ error.kind = SparkErrorKind::IllegalArgument;
+ error
+}
/// Represents a table argument for a table-valued function.
///
-/// Supports partitioning, ordering, and single-partition modes.
-/// Mirrors `pyspark.sql.connect.table_arg.TableArg`.
-#[allow(dead_code)]
+/// Supports partitioning, ordering, and single-partition modes. Mirrors
+/// `pyspark.sql.connect.table_arg.TableArg`: a `SubqueryExpression` of type
`TABLE_ARG` of the
+/// plan of the DataFrame, which a UDTF call takes as an argument.
#[derive(Clone)]
pub struct TableArg {
- /// The underlying DataFrame/plan
- plan: Arc<LogicalPlan>,
- /// Partition expressions
- partition_spec: Vec<Expression>,
- /// Order expressions
- order_spec: Vec<Expression>,
- /// Whether to use a single partition
- with_single_partition: bool,
+ /// The subquery of the plan, with its partition and order specs.
+ subquery: SubqueryExpression,
}
impl TableArg {
/// Create a new TableArg from a DataFrame.
pub fn new(df: DataFrame) -> Self {
TableArg {
- plan: df.plan,
- partition_spec: Vec::new(),
- order_spec: Vec::new(),
- with_single_partition: false,
+ subquery: df.subquery(SubqueryType::TableArg),
}
}
- /// Create a new TableArg from a LogicalPlan.
- #[allow(dead_code)]
- pub(crate) fn from_plan(plan: LogicalPlan) -> Self {
+ /// Create a new TableArg of a LogicalPlan whose plan ID is `plan_id`.
+ #[cfg(test)]
+ pub(crate) fn from_plan(plan: LogicalPlan, plan_id: i64) -> Self {
TableArg {
- plan: Arc::new(plan),
- partition_spec: Vec::new(),
- order_spec: Vec::new(),
- with_single_partition: false,
+ subquery: SubqueryExpression::new(
+ std::sync::Arc::new(plan),
+ plan_id,
+ SubqueryType::TableArg,
+ ),
}
}
/// Check if partitioning has been applied.
fn is_partitioned(&self) -> bool {
- !self.partition_spec.is_empty() || self.with_single_partition
+ !self.subquery.partition_spec.is_empty() || self.is_single_partition()
}
/// Add partitioning to this table argument.
@@ -72,14 +70,16 @@ impl TableArg {
cols: impl IntoIterator<Item = C>,
) -> Result<Self> {
if self.is_partitioned() {
- return Err(SparkError::value(
- "ILLEGAL_ARGUMENT",
- &[("msg", "Cannot call partitionBy() after partitionBy() or
withSinglePartition() has been called.")],
+ return Err(illegal_argument(
+ "Cannot call partitionBy() after partitionBy() or
withSinglePartition() has \
+ been called.",
));
}
for col in cols {
- self.partition_spec.push(col.into().expression().clone());
+ self.subquery
+ .partition_spec
+ .push(col.into().expression().clone());
}
Ok(self)
@@ -102,17 +102,15 @@ impl TableArg {
/// ```
pub fn order_by<C: Into<Column>>(mut self, cols: impl IntoIterator<Item =
C>) -> Result<Self> {
if !self.is_partitioned() {
- return Err(SparkError::value(
- "ILLEGAL_ARGUMENT",
- &[(
- "msg",
- "Please call partitionBy() or withSinglePartition() before
orderBy().",
- )],
+ return Err(illegal_argument(
+ "Please call partitionBy() or withSinglePartition() before
orderBy().",
));
}
for col in cols {
- self.order_spec.push(col.into().expression().clone());
+ self.subquery
+ .order_spec
+ .push(col.into().expression().clone());
}
Ok(self)
@@ -128,87 +126,37 @@ impl TableArg {
/// ```
pub fn with_single_partition(mut self) -> Result<Self> {
if self.is_partitioned() {
- return Err(SparkError::value(
- "ILLEGAL_ARGUMENT",
- &[("msg", "Cannot call withSinglePartition() after
partitionBy() or withSinglePartition() has been called.")],
+ return Err(illegal_argument(
+ "Cannot call withSinglePartition() after partitionBy() or
withSinglePartition() \
+ has been called.",
));
}
- self.with_single_partition = true;
+ self.subquery.with_single_partition = Some(true);
Ok(self)
}
/// Get the underlying plan.
- #[allow(dead_code)]
+ #[cfg(test)]
pub(crate) fn plan(&self) -> &LogicalPlan {
- &self.plan
- }
-
- /// Get partition specification.
- #[allow(dead_code)]
- pub(crate) fn partition_spec(&self) -> &[Expression] {
- &self.partition_spec
- }
-
- /// Get order specification.
- #[allow(dead_code)]
- pub(crate) fn order_spec(&self) -> &[Expression] {
- &self.order_spec
+ &self.subquery.plan
}
/// Check if single partition mode is enabled.
- #[allow(dead_code)]
pub(crate) fn is_single_partition(&self) -> bool {
- self.with_single_partition
+ self.subquery.with_single_partition == Some(true)
}
- /// Serialize this table argument to a proto `Expression` - a
- /// `SubqueryExpression` of type `TABLE_ARG` carrying the partition/order
- /// specs and single-partition flag.
- ///
- /// Mirrors `SubqueryExpression.to_plan` in the reference PySpark client:
- /// `plan_id` is the plan-id assigned to the underlying subquery relation,
- /// each partition expression is emitted as-is, each order expression is
- /// emitted as a `SortOrder` (an explicit sort direction is preserved;
- /// otherwise it defaults to ascending, nulls-first), and
- /// `with_single_partition` is only set when true (left unset otherwise, as
- /// the reference leaves it `None`).
- #[allow(dead_code)]
- pub(crate) fn to_proto(&self, plan_id: i64) -> proto::Expression {
- let mut options =
proto::subquery_expression::TableArgOptions::default();
-
- options.partition_spec = self.partition_spec.iter().map(|e|
e.to_proto()).collect();
-
- options.order_spec = self
- .order_spec
- .iter()
- .map(|e| {
- // Preserve an explicit SortOrder; otherwise default to
ascending,
- // nulls-first (matching the reference's `_sort_col`).
- let wrapped = match e.node() {
- ExpressionNode::SortOrder(_) => e.to_proto(),
- _ => SortOrder::asc_nulls_first(e.clone()).to_proto(),
- };
- match wrapped.expr_type {
- Some(proto::expression::ExprType::SortOrder(s)) => *s,
- // SortOrder::to_proto always yields a SortOrder expr_type.
- _ => unreachable!("expected a SortOrder expression"),
- }
- })
- .collect();
-
- if self.with_single_partition {
- options.with_single_partition = Some(true);
- }
-
- let mut subquery = proto::SubqueryExpression::default();
- subquery.plan_id = plan_id;
- subquery.subquery_type =
proto::subquery_expression::SubqueryType::TableArg as i32;
- subquery.table_arg_options = Some(options);
+ /// This table argument as a Column, the argument of a UDTF call that
passes the rows of
+ /// the DataFrame. As the `_subquery_expr` of a PySpark `TableArg`.
+ pub fn to_column(&self) -> Column {
+
Column::new(ExpressionNode::Subquery(Box::new(self.subquery.clone())).into())
+ }
+}
- let mut expr = proto::Expression::default();
- expr.expr_type =
Some(proto::expression::ExprType::SubqueryExpression(subquery));
- expr
+impl From<TableArg> for Column {
+ fn from(table_arg: TableArg) -> Column {
+ table_arg.to_column()
}
}
@@ -216,6 +164,8 @@ impl TableArg {
mod tests {
use super::*;
use crate::column::col;
+ use crate::expression::Expression;
+ use spark_connect_proto as proto;
fn base_plan() -> LogicalPlan {
LogicalPlan::Range {
@@ -227,7 +177,21 @@ mod tests {
}
fn arg() -> TableArg {
- TableArg::from_plan(base_plan())
+ TableArg::from_plan(base_plan(), 7)
+ }
+
+ fn spec(t: &TableArg) -> (&[Expression], &[Expression]) {
+ (&t.subquery.partition_spec, &t.subquery.order_spec)
+ }
+
+ /// The `SubqueryExpression` of the table argument.
+ fn subquery_proto(t: &TableArg) -> proto::SubqueryExpression {
+ let Some(proto::expression::ExprType::SubqueryExpression(subquery)) =
+ t.to_column().expression().to_proto().expr_type
+ else {
+ panic!("expected a SubqueryExpression");
+ };
+ subquery
}
#[test]
@@ -235,16 +199,20 @@ mod tests {
let t = arg();
assert!(!t.is_partitioned());
assert!(!t.is_single_partition());
- assert!(t.partition_spec().is_empty());
- assert!(t.order_spec().is_empty());
+ assert!(spec(&t).0.is_empty());
+ assert!(spec(&t).1.is_empty());
assert!(matches!(t.plan(), LogicalPlan::Range { .. }));
+ // As PySpark, which sets no options for a table argument without any.
+ let subquery = subquery_proto(&t);
+ assert_eq!(subquery.plan_id, 7);
+ assert_eq!(subquery.table_arg_options, None);
}
#[test]
fn partition_by_records_expressions_and_marks_partitioned() {
let t = arg().partition_by(vec![col("a"), col("b")]).unwrap();
assert!(t.is_partitioned());
- assert_eq!(t.partition_spec().len(), 2);
+ assert_eq!(spec(&t).0.len(), 2);
}
#[test]
@@ -261,7 +229,7 @@ mod tests {
.unwrap()
.order_by(vec![col("b"), col("c")])
.unwrap();
- assert_eq!(t.order_spec().len(), 2);
+ assert_eq!(spec(&t).1.len(), 2);
}
#[test]
@@ -271,18 +239,18 @@ mod tests {
.unwrap()
.order_by(vec![col("b")])
.unwrap();
- assert_eq!(t.order_spec().len(), 1);
+ assert_eq!(spec(&t).1.len(), 1);
}
- /// Assert a `Result<TableArg>` is an `ILLEGAL_ARGUMENT` error. (Written
as a
- /// helper because `unwrap_err` would require `TableArg: Debug`.)
+ /// Assert a `Result<TableArg>` is an `IllegalArgumentException`, as
PySpark raises.
+ /// (Written as a helper because `unwrap_err` would require `TableArg:
Debug`.)
fn assert_illegal(r: Result<TableArg>) {
match r {
- Ok(_) => panic!("expected an ILLEGAL_ARGUMENT error, got Ok"),
- Err(e) => assert!(
- format!("{e:?}").contains("ILLEGAL_ARGUMENT"),
- "unexpected error: {e:?}"
- ),
+ Ok(_) => panic!("expected an IllegalArgumentException, got Ok"),
+ Err(e) => {
+ assert_eq!(e.kind, SparkErrorKind::IllegalArgument, "{e:?}");
+ assert!(e.message.contains("() "), "unexpected error: {e:?}");
+ }
}
}
@@ -311,18 +279,12 @@ mod tests {
#[test]
fn to_proto_emits_table_arg_subquery_with_specs() {
- use proto::expression::ExprType;
-
let t = arg()
.partition_by(vec![col("a")])
.unwrap()
.order_by(vec![col("b")])
.unwrap();
- let expr = t.to_proto(7);
-
- let Some(ExprType::SubqueryExpression(subq)) = expr.expr_type else {
- panic!("expected a SubqueryExpression");
- };
+ let subq = subquery_proto(&t);
assert_eq!(subq.plan_id, 7);
assert_eq!(
subq.subquery_type,
@@ -336,6 +298,11 @@ mod tests {
assert_eq!(opts.order_spec[0].null_ordering, 1);
// partitionBy path leaves the single-partition flag unset (None).
assert_eq!(opts.with_single_partition, None);
+ assert_eq!(
+ t.to_column().expression().render(),
+ "SubqueryExpression(plan=<plan 7>, type=table_arg,
partition_spec=[a], \
+ order_spec=[b])"
+ );
}
#[test]
@@ -345,11 +312,7 @@ mod tests {
.unwrap()
.order_by(vec![col("b").desc_nulls_last()])
.unwrap();
- let expr = t.to_proto(3);
-
- let Some(proto::expression::ExprType::SubqueryExpression(subq)) =
expr.expr_type else {
- panic!("expected a SubqueryExpression");
- };
+ let subq = subquery_proto(&t);
let opts = subq.table_arg_options.expect("table_arg_options");
assert_eq!(opts.with_single_partition, Some(true));
assert!(opts.partition_spec.is_empty());
diff --git a/crates/spark-connect/tests/e2e_dataframe.rs
b/crates/spark-connect/tests/e2e_dataframe.rs
index e783a50..78ed569 100644
--- a/crates/spark-connect/tests/e2e_dataframe.rs
+++ b/crates/spark-connect/tests/e2e_dataframe.rs
@@ -100,13 +100,16 @@ fn dataframe_metadata_json_semantics() {
assert!(json
.iter()
.all(|j| j.starts_with('{') && j.contains("id") &&
j.contains("name")));
- // exists / scalar
- assert!(df.exists().unwrap());
+ // exists / scalar: the EXISTS and scalar subqueries of a DataFrame, as
PySpark's.
let one = s.sql("SELECT 42 AS v").unwrap();
- assert_eq!(
- one.scalar().unwrap(),
- Some(spark_connect::row::Value::Integer(42))
- );
+ let row = &s
+ .range(1)
+ .unwrap()
+ .select(vec![df.exists(), one.scalar()])
+ .collect()
+ .unwrap()[0];
+ assert_eq!(row.get(0), Some(&spark_connect::row::Value::Bool(true)));
+ assert_eq!(row.get(1), Some(&spark_connect::row::Value::Integer(42)));
// semantics (server-backed AnalyzePlan): two identical plans are equal
assert!(df.same_semantics(&df4(&s)).unwrap());
let _ = df.semantic_hash().unwrap();
@@ -239,3 +242,74 @@ fn day_time_interval_input_roundtrips() {
collected
);
}
+
+/// The rows of `df`, as (i32, i32) pairs, sorted.
+fn int_pairs(df: &DataFrame) -> Vec<(i32, i32)> {
+ use spark_connect::row::Value;
+ let mut pairs: Vec<(i32, i32)> = df
+ .collect()
+ .unwrap()
+ .iter()
+ .map(|row| match (row.get(0), row.get(1)) {
+ (Some(Value::Integer(a)), Some(Value::Integer(b))) => (*a, *b),
+ other => panic!("unexpected row {other:?}"),
+ })
+ .collect();
+ pairs.sort();
+ pairs
+}
+
+#[test]
+fn subqueries_return_the_rows_of_pyspark() {
+ // As PySpark's DataFrame.scalar(), DataFrame.exists() and
Column.isin(df): scalar,
+ // correlated EXISTS and IN subqueries, also nested and in other operators.
+ if !should_run() {
+ return;
+ }
+ let s = session();
+ let t1 = s
+ .sql("SELECT * FROM VALUES (1, 2), (3, 4) AS t1(a, b)")
+ .unwrap()
+ .alias("t1");
+ let t2 = s
+ .sql("SELECT * FROM VALUES (1, 5), (5, 6), (3, 7) AS t2(a, c)")
+ .unwrap()
+ .alias("t2");
+
+ let largest = t2.select(vec![f::max(col("c"))]).scalar();
+ let with_largest = t1.select(vec![col("a"), largest.clone().alias("m")]);
+ assert_eq!(int_pairs(&with_largest), vec![(1, 7), (3, 7)]);
+ // A DataFrame with a subquery is the base, and the subquery, of others.
+ assert_eq!(
+ int_pairs(&with_largest.filter(col("m").gt(col("a")))),
+ vec![(1, 7), (3, 7)]
+ );
+ assert_eq!(
+
int_pairs(&t2.filter(col("a").isin_subquery(&with_largest.select(vec![col("a")])))),
+ vec![(1, 5), (3, 7)]
+ );
+
+ let correlated = t2.filter(col("t2.a").eq(col("t1.a")));
+ let largest_of_a = correlated.select(vec![f::max(col("c"))]).scalar();
+ assert_eq!(
+ int_pairs(&t1.select(vec![col("a"), largest_of_a.alias("m")])),
+ vec![(1, 5), (3, 7)]
+ );
+ assert_eq!(
+ int_pairs(&t1.filter(correlated.exists())),
+ vec![(1, 2), (3, 4)]
+ );
+ assert_eq!(int_pairs(&t1.filter(correlated.exists().not())), vec![]);
+
+ let smallest = t2.select(vec![f::min(col("c"))]).scalar();
+ let above_smallest =
t2.filter(col("c").gt(smallest)).select(vec![col("a")]);
+ assert_eq!(
+ int_pairs(&t1.filter(col("a").isin_subquery(&above_smallest))),
+ vec![(3, 4)]
+ );
+ let pairs = s.sql("SELECT 1 AS x, 2 AS y").unwrap();
+ assert_eq!(
+ int_pairs(&t1.filter(f::r#struct(vec![col("a"),
col("b")]).isin_subquery(&pairs))),
+ vec![(1, 2)]
+ );
+}
diff --git a/crates/spark-connect/tests/e2e_new_methods.rs
b/crates/spark-connect/tests/e2e_new_methods.rs
index 2314ab2..125fa51 100644
--- a/crates/spark-connect/tests/e2e_new_methods.rs
+++ b/crates/spark-connect/tests/e2e_new_methods.rs
@@ -142,14 +142,20 @@ fn exists_scalar_transpose_dedup_astable() {
}
let spark = session();
- // scalar()/exists() on a single-value subquery-shaped DataFrame.
+ // scalar()/exists() are the subqueries of a DataFrame, as PySpark's.
let one = spark.sql("SELECT 1 AS x").expect("sql one");
- assert!(one.exists().expect("exists"));
- let _v = spark
- .sql("SELECT 42 AS x")
- .expect("sql")
- .scalar()
- .expect("scalar");
+ let answer = spark.sql("SELECT 42 AS x").expect("sql");
+ let rows = spark
+ .range(1)
+ .expect("range")
+ .select(vec![one.exists().alias("e"), answer.scalar().alias("v")])
+ .collect()
+ .expect("subqueries");
+ assert_eq!(rows[0].get(0), Some(&spark_connect::row::Value::Bool(true)));
+ assert_eq!(
+ rows[0].get(1),
+ Some(&spark_connect::row::Value::Integer(42))
+ );
// dropDuplicates on a subset.
let df = numeric(&spark);
diff --git a/crates/spark-connect/tests/e2e_ops_breadth.rs
b/crates/spark-connect/tests/e2e_ops_breadth.rs
index d3da8c0..d95453b 100644
--- a/crates/spark-connect/tests/e2e_ops_breadth.rs
+++ b/crates/spark-connect/tests/e2e_ops_breadth.rs
@@ -252,7 +252,7 @@ fn sources_and_accessors_execute() {
let _ = df.head().unwrap();
let _ = df.take(2).unwrap();
let _ = df.to_json().unwrap();
- let _ = df.select(vec![col("id")]).limit(1).scalar();
+ let _ = df.select(vec![df.select(vec![col("id")]).limit(1).scalar()]);
let _ = df.semantic_hash();
let _ = df.same_semantics(&df);
let _ = df.to_local_iterator(false);
diff --git a/python/tests/test_dropin_e2e.py b/python/tests/test_dropin_e2e.py
index bdca686..b19fc9c 100644
--- a/python/tests/test_dropin_e2e.py
+++ b/python/tests/test_dropin_e2e.py
@@ -575,6 +575,45 @@ def test_reader_schema_keeps_field_metadata(session,
tmp_path, ddl):
assert stream.schema.fields[0].metadata == {"comment": "the key"}
+def test_subqueries_and_table_arguments_return_the_rows_of_pyspark(session):
+ # As PySpark's DataFrame.scalar(), DataFrame.exists(), Column.isin(df) and
asTable():
+ # scalar, correlated and IN subqueries, and the rows of a DataFrame as a
UDTF argument.
+ from pyspark.errors import IllegalArgumentException
+
+ t1 = session.createDataFrame([(1, 2), (3, 4)], "a int, b int").alias("t1")
+ t2 = session.createDataFrame([(1, 5), (5, 6), (3, 7)], "a int, c
int").alias("t2")
+ largest = t2.select(F.max("c")).scalar()
+ assert sorted(t1.select("a", largest.alias("m")).collect()) == [Row(a=1,
m=7), Row(a=3, m=7)]
+ correlated = t2.where(F.col("t2.a") == F.col("t1.a").outer())
+ assert sorted(t1.select("a",
correlated.select(F.max("c")).scalar().alias("m")).collect()) == [
+ Row(a=1, m=5), Row(a=3, m=7)
+ ]
+ assert t1.where(~correlated.exists()).collect() == []
+ above = t2.where(F.col("c") > t2.select(F.min("c")).scalar()).select("a")
+ assert t1.where(F.col("a").isin(above)).collect() == [Row(a=3, b=4)]
+ pairs = session.createDataFrame([(1, 2)], "x int, y int")
+ assert t1.where(F.struct("a", "b").isin(pairs)).collect() == [Row(a=1,
b=2)]
+
+ @F.udtf(returnType="id bigint, n int")
+ class Numbered:
+ def __init__(self):
+ self.n = 0
+
+ def eval(self, row):
+ self.n += 1
+ yield row["id"], self.n
+
+ table =
session.range(4).asTable().withSinglePartition().orderBy(F.col("id").desc())
+ assert sorted(Numbered(table).collect()) == [
+ Row(id=0, n=4), Row(id=1, n=3), Row(id=2, n=2), Row(id=3, n=1)
+ ]
+ assert sorted(Numbered(row=session.range(2).asTable()).collect()) == [
+ Row(id=0, n=1), Row(id=1, n=1)
+ ]
+ with pytest.raises(IllegalArgumentException, match="before orderBy"):
+ session.range(1).asTable().orderBy("id")
+
+
def test_grouped_map_functions_take_iterators_by_their_type_hints(session):
from typing import Iterator, Tuple
diff --git a/python/tests/test_dropin_offline.py
b/python/tests/test_dropin_offline.py
index 16798f0..9120a93 100644
--- a/python/tests/test_dropin_offline.py
+++ b/python/tests/test_dropin_offline.py
@@ -886,6 +886,67 @@ def
test_udf_and_udtf_calls_pass_named_arguments_as_pyspark(mock_connect):
assert parse_message(first[2][0])[1] == [b"x"] and key == "b" and
set(value) == {1}
+def
test_subqueries_are_sent_with_the_plans_they_refer_to_as_pyspark(mock_connect):
+ # As PySpark's DataFrame.scalar(), DataFrame.exists() and Column.isin(df):
Columns of
+ # SubqueryExpressions of the plan of the DataFrame, sent in a
WithRelations of the plans
+ # that they refer to; and asTable() is a table argument that a UDTF call
takes.
+ from pyspark.sql import Column, functions as F
+
+ spark, service = mock_connect
+
+ def relation():
+ # ExecutePlanRequest.plan=3, Plan.root=1.
+ return
parse_message(parse_message(parse_message(service.execute_requests[-1])[3][0])[1][0])
+
+ def plan_id(rel):
+ # Relation.common=1, RelationCommon.plan_id=2.
+ return parse_message(rel[1][0])[2][0]
+
+ def with_relations(rel):
+ # Relation.with_relations=41: root=1, references=2.
+ with_rel = parse_message(rel[41][0])
+ return parse_message(with_rel[1][0]), [parse_message(r) for r in
with_rel[2]]
+
+ def subquery(expression):
+ # Expression.alias=6 (expr=1); .subquery_expression=21: plan_id=1,
subquery_type=2,
+ # table_arg_options=3, in_subquery_values=4.
+ expression = parse_message(expression)
+ if 6 in expression:
+ expression = parse_message(parse_message(expression[6][0])[1][0])
+ return parse_message(expression[21][0])
+
+ inner = spark.range(3)
+ scalar = inner.scalar()
+ assert isinstance(scalar, Column) and isinstance(inner.exists(), Column)
+ assert "type=scalar" in repr(scalar) and "type=exists" in
repr(inner.exists())
+ spark.range(1).select(
+ scalar.alias("s"), inner.exists().alias("e"),
F.col("id").isin(inner).alias("i")
+ ).collect()
+ rel = relation()
+ root, references = with_relations(rel)
+ assert plan_id(root) != plan_id(rel)
+ # Relation.project=3, Project.expressions=3.
+ subqueries = [subquery(e) for e in parse_message(root[3][0])[3]]
+ assert [s[2][0] for s in subqueries] == [1, 2, 4] # SCALAR, EXISTS, IN
+ assert {s[1][0] for s in subqueries} == {plan_id(r) for r in references}
+ assert len(references) == 3 and all(3 not in s for s in subqueries)
+ assert parse_message(subqueries[2][4][0])[2] # IN of unresolved_attribute
"id"
+
+ @F.udtf(returnType="id bigint")
+ class Rows:
+ def eval(self, row):
+ yield row["id"],
+
+ table =
spark.range(2).asTable().partitionBy("id").orderBy(F.col("id").desc())
+ Rows(table).collect()
+ root, references = with_relations(relation())
+ # Relation.common_inline_user_defined_table_function=38, its arguments=3.
+ argument = subquery(parse_message(root[38][0])[3][0])
+ assert argument[2] == [3] and argument[1][0] == plan_id(references[0]) #
TABLE_ARG
+ options = parse_message(argument[3][0])
+ assert len(options[1]) == 1 and len(options[2]) == 1 and 3 not in options
+
+
def test_udtfs_are_created_as_pyspark(mock_connect):
# As PySpark's _create_py_udtf and _create_pyarrow_udtf: a UDTF is not
deterministic
# unless asDeterministic() marks it so, in place; arrow_udtf() makes a
PyArrow-native
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]