This is an automated email from the ASF dual-hosted git repository.
HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git
The following commit(s) were added to refs/heads/master by this push:
new 9f264b21 [SPARK-61180][CONNECT][PYTHON] Add DataFrame._joinAsOf, the
as-of join of pandas-on-Spark's merge_asof, in the Rust and Python clients as
PySpark does
9f264b21 is described below
commit 9f264b21ecc721b65b2e97e5b0f8f631e2405cd0
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Mon Nov 9 06:20:55 2026 +0900
[SPARK-61180][CONNECT][PYTHON] Add DataFrame._joinAsOf, the as-of join of
pandas-on-Spark's merge_asof, in the Rust and Python clients as PySpark does
### What changes were proposed in this pull request?
[SPARK-61180](https://issues.apache.org/jira/browse/SPARK-61180): PySpark
Connect's `DataFrame._joinAsOf(other, leftAsOfColumn, rightAsOfColumn, on=None,
how=None, *, tolerance=None, allowExactMatches=True, direction="backward")`
builds the Spark Connect `AsOfJoin` relation. pandas-on-Spark's `merge_asof`
calls it.
- Rust core: `DataFrame::as_of_join(&right, AsOfJoinOptions)` with a new
`AsOfJoinOptions` struct (`AsOfJoinOptions::new(left_as_of, right_as_of)` sets
the PySpark defaults: inner, exact matches allowed, `"backward"`). The plan is
a new `LogicalPlan::AsOfJoin`, serialized to `AsOfJoin` with `left_as_of`,
`right_as_of`, `join_expr` or `using_columns`, `join_type`, `tolerance`,
`allow_exact_matches` and `direction`. A DataFrame of another session fails
with `SESSION_NOT_SAME`.
- PyO3: `DataFrame._joinAsOf` with PySpark's signature. `how` defaults to
inner and must be a string. `tolerance` must be a Column. `on` is a name, a
Column, or a list of names or Columns, as in `plan.AsOfJoin`. A string as-of
column is a column of its DataFrame, as `df._col(name)`. `join_type` and
`direction` are sent as they are, so the server checks them as it does for
PySpark.
### Why are the changes needed?
`ps.merge_asof` failed, because `_joinAsOf` did not exist and the name
resolved through `DataFrame.__getattr__` to a Column:
```
>>> ps.merge_asof(ps.DataFrame(L), ps.DataFrame(R), on="t")
TypeError: 'pyspark.sql.connect.column.Column' object is not callable
```
PySpark returns `[[1, 1.0, 'a', nan, None], [5, 5.0, 'b', 4.0, 'y']]` for L
= `[(1, 1.0, "a"), (5, 5.0, "b")]` ("t int, x double, k string") and R = `[(2,
2.0, "x"), (4, 4.0, "y")]` ("t int, y double, k string").
### Does this PR introduce _any_ user-facing change?
Yes. `DataFrame._joinAsOf` and `pandas.merge_asof` (pandas-on-Spark) work
on the Rust client as they do on PySpark Connect. Before, they raised the
`TypeError` above.
### How was this patch tested?
- `test_join_as_of_sends_the_as_of_join_as_pyspark` (offline,
`python/tests/test_dropin_offline.py`): checks the fields of the sent
`AsOfJoin` relation for a list `on` with `how`, `tolerance`,
`allowExactMatches=False` and `direction`, for the defaults, and for a Column
`on`.
- `test_join_as_of_and_merge_asof_give_the_rows_of_pyspark` (e2e,
`python/tests/test_dropin_e2e.py`): the rows of `_joinAsOf` for each direction,
`allowExactMatches`, tolerance, `on` as a list, and the inner default, and the
rows of `ps.merge_asof` with `on`, `by`, `direction`, `tolerance`, and
`left_on`/`right_on`/`left_by`/`right_by`.
- `as_of_join_sends_every_field_of_the_as_of_join`,
`as_of_join_defaults_and_sends_a_condition_as_join_expr` and
`as_of_join_rejects_a_dataframe_of_another_session` (Rust unit tests,
`crates/spark-connect/src/dataframe.rs`).
- The offline test fails without this change, with `TypeError:
'pyspark.sql.connect.column.Column' object is not callable`. The same cases
give the same schemas, rows, and error messages on official PySpark 4.3.0
against the same server. The explain text matches apart from plan IDs.
- The lib tests of both crates (611 passed), the build of all tests, also
with `wasm-udf`, the offline Python suites (2228 passed, 8 skipped), and the
live drop-in and Arrow suites (211 passed) and live cases (10 passed) against a
local Spark 4.3.0 server. Clippy reports no diagnostics in the changed lines.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
This pull request and its description were written by Isaac.
Closes #376 from HyukjinKwon/SPARK-61180-join-as-of.
Authored-by: Hyukjin Kwon <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
---
crates/pyspark-rs/src/dataframe.rs | 135 ++++++++++++++++++++++++++++++-
crates/spark-connect/src/dataframe.rs | 148 +++++++++++++++++++++++++++++++++-
crates/spark-connect/src/plan.rs | 40 +++++++++
python/tests/test_dropin_e2e.py | 60 ++++++++++++++
python/tests/test_dropin_offline.py | 38 +++++++++
5 files changed, 419 insertions(+), 2 deletions(-)
diff --git a/crates/pyspark-rs/src/dataframe.rs
b/crates/pyspark-rs/src/dataframe.rs
index 670766ed..4b4d9d0f 100644
--- a/crates/pyspark-rs/src/dataframe.rs
+++ b/crates/pyspark-rs/src/dataframe.rs
@@ -2,7 +2,7 @@
use pyo3::prelude::*;
use pyo3::types::{PyDict, PyInt, PyList, PyTuple};
-use spark_connect::dataframe::{DataFrame, LocalRowIterator};
+use spark_connect::dataframe::{AsOfJoinOptions, DataFrame, LocalRowIterator};
use spark_connect::plan::JoinType;
use spark_connect::udf::{eval_type, CommonInlineUserDefinedFunctionExpression,
PythonUDFPayload};
@@ -526,6 +526,83 @@ fn no_to_plan(obj: &Bound<'_, PyAny>) -> PyErr {
))
}
+/// The `TypeError` of protobuf for a name that is not a string, which the
plan of PySpark's
+/// `_joinAsOf` raises for a name in `on` or an as-of column that is not a
string.
+fn proto_string_type_error(value: &Bound<'_, PyAny>) -> PyErr {
+ let repr = value
+ .repr()
+ .map(|repr| repr.to_string())
+ .unwrap_or_default();
+ let typ = value
+ .get_type()
+ .repr()
+ .map(|typ| typ.to_string())
+ .unwrap_or_default();
+ pyo3::exceptions::PyTypeError::new_err(format!(
+ "{repr} has type {typ}, but expected one of: (<class 'bytes'>, <class
'str'>)"
+ ))
+}
+
+/// The join condition of `_joinAsOf` for its `on`, as PySpark's `AsOfJoin`
plan sends it: a
+/// name or a list of names (`using_columns`), or a Column or a list of
Columns, which are
+/// joined on their conjunction (`join_expr`).
+fn as_of_join_condition(
+ on: &Bound<'_, PyAny>,
+) -> PyResult<(Option<spark_connect::column::Column>, Vec<String>)> {
+ if let Ok(column) = on.extract::<PyRef<PyColumn>>() {
+ return Ok((Some(column.column.clone()), Vec::new()));
+ }
+ if on.is_instance_of::<pyo3::types::PyString>() {
+ return Ok((None, vec![on.extract::<String>()?]));
+ }
+ // As PySpark's `_collect_references`, which iterates `on` first, and then
its `to_plan`.
+ if !on.is_instance_of::<PyList>() {
+ on.try_iter()?;
+ return Err(no_to_plan(on));
+ }
+ let list = on.cast::<PyList>()?;
+ // An empty list is no condition at all.
+ let Some(first) = list.iter().next() else {
+ return Ok((None, Vec::new()));
+ };
+ if first.is_instance_of::<pyo3::types::PyString>() {
+ let names = list
+ .iter()
+ .map(|item| {
+ item.extract::<String>()
+ .map_err(|_| proto_string_type_error(&item))
+ })
+ .collect::<PyResult<Vec<_>>>()?;
+ return Ok((None, names));
+ }
+ let mut condition: Option<spark_connect::column::Column> = None;
+ for item in list.iter() {
+ let Ok(col) = item.extract::<PyRef<PyColumn>>() else {
+ return Err(no_to_plan(&item));
+ };
+ condition = Some(match condition {
+ None => col.column.clone(),
+ Some(left) => left.and(col.column.clone()),
+ });
+ }
+ Ok((condition, Vec::new()))
+}
+
+/// The as-of column of `_joinAsOf` of `df`: a Column as it is, or the column
of `df` that
+/// has this name, as PySpark's `df._col(name)`.
+fn as_of_column(
+ df: &DataFrame,
+ column: &Bound<'_, PyAny>,
+) -> PyResult<spark_connect::column::Column> {
+ if let Ok(column) = column.extract::<PyRef<PyColumn>>() {
+ return Ok(column.column.clone());
+ }
+ let name = column
+ .extract::<String>()
+ .map_err(|_| proto_string_type_error(column))?;
+ Ok(df.col(&name))
+}
+
/// Resolve a filter/where condition: a Column, or a SQL string parsed via
`expr`.
fn cond_column(condition: &Bound<'_, PyAny>) ->
PyResult<spark_connect::column::Column> {
if let Ok(c) = condition.extract::<PyRef<PyColumn>>() {
@@ -941,6 +1018,62 @@ impl PyDataFrame {
Err(no_to_plan(&on))
}
+ /// `DataFrame._joinAsOf(other, leftAsOfColumn, rightAsOfColumn, on=None,
how=None, *,
+ /// tolerance=None, allowExactMatches=True, direction="backward")`, the
as-of join of
+ /// pandas-on-Spark's `merge_asof`, as PySpark's. The as-of columns are
Columns, or names of
+ /// their DataFrame. `on` is a name, a Column condition, or a list of
either. `how` (inner by
+ /// default) and `direction` are sent as they are, and the server checks
them.
+ #[pyo3(name = "_joinAsOf", signature = (other, leftAsOfColumn,
rightAsOfColumn, on=None, how=None, *, tolerance=None, allowExactMatches=true,
direction="backward"))]
+ #[allow(non_snake_case, clippy::too_many_arguments)]
+ fn join_as_of(
+ &self,
+ other: &Bound<'_, PyAny>,
+ leftAsOfColumn: &Bound<'_, PyAny>,
+ rightAsOfColumn: &Bound<'_, PyAny>,
+ on: Option<Bound<'_, PyAny>>,
+ how: Option<Bound<'_, PyAny>>,
+ tolerance: Option<Bound<'_, PyAny>>,
+ allowExactMatches: bool,
+ direction: &str,
+ ) -> PyResult<PyDataFrame> {
+ // As PySpark's `_check_same_session`, first.
+ let other = self.same_session(other)?;
+ let join_type = match how.filter(|how| !how.is_none()) {
+ None => "inner".to_string(),
+ Some(how) => how.extract::<String>().map_err(|_| {
+ pyo3::exceptions::PyAssertionError::new_err("how should be a
string")
+ })?,
+ };
+ let tolerance = match tolerance.filter(|tolerance|
!tolerance.is_none()) {
+ None => None,
+ Some(tolerance) => {
+ let column =
tolerance.extract::<PyRef<PyColumn>>().map_err(|_| {
+ pyo3::exceptions::PyAssertionError::new_err("tolerance
should be Column")
+ })?;
+ Some(column.column.clone())
+ }
+ };
+ let (on, using_columns) = match on.filter(|on| !on.is_none()) {
+ None => (None, Vec::new()),
+ Some(on) => as_of_join_condition(&on)?,
+ };
+ let options = AsOfJoinOptions {
+ left_as_of: as_of_column(&self.dataframe, leftAsOfColumn)?,
+ right_as_of: as_of_column(&other.dataframe, rightAsOfColumn)?,
+ on,
+ using_columns,
+ join_type,
+ tolerance,
+ allow_exact_matches: allowExactMatches,
+ direction: direction.to_string(),
+ };
+ let df = self
+ .dataframe
+ .as_of_join(&other.dataframe, options)
+ .to_pyerr()?;
+ Ok(PyDataFrame::new(df))
+ }
+
/// Cross join.
fn crossJoin(&self, other: &Bound<'_, PyAny>) -> PyResult<PyDataFrame> {
let other = self.same_session(other)?;
diff --git a/crates/spark-connect/src/dataframe.rs
b/crates/spark-connect/src/dataframe.rs
index 6e6cb56f..32b4c9c9 100644
--- a/crates/spark-connect/src/dataframe.rs
+++ b/crates/spark-connect/src/dataframe.rs
@@ -5,7 +5,7 @@
use std::collections::HashMap;
use std::sync::{Arc, Mutex};
-use spark_connect_core::error::{Result, SparkError};
+use spark_connect_core::error::{Result, SparkError, SparkErrorKind};
use spark_connect_core::runtime::{block_on, get_runtime};
use spark_connect_proto as proto;
@@ -41,6 +41,47 @@ pub(crate) type ExecutionSlot =
Arc<Mutex<Option<ExecutionInfo>>>;
/// Iterator over rows from a DataFrame, yielded lazily as the server streams
results.
///
+/// The options of an as-of join, as `DataFrame._joinAsOf` takes them.
+///
+/// `on` and `using_columns` are the join condition: a condition, or names
that both sides
+/// share. PySpark sets at most one of them. The defaults are those of
PySpark: an inner join,
+/// exact matches allowed, and the `"backward"` direction.
+#[derive(Debug, Clone)]
+pub struct AsOfJoinOptions {
+ /// The as-of column of this DataFrame.
+ pub left_as_of: Column,
+ /// The as-of column of the other DataFrame.
+ pub right_as_of: Column,
+ /// The join condition, sent as `join_expr`.
+ pub on: Option<Column>,
+ /// Column names that both sides share, sent as `using_columns`.
+ pub using_columns: Vec<String>,
+ /// The join type, such as `"inner"` or `"left"`, sent as it is.
+ pub join_type: String,
+ /// The as-of tolerance: the largest distance a match may have.
+ pub tolerance: Option<Column>,
+ /// Whether a row may match a row with the same as-of value.
+ pub allow_exact_matches: bool,
+ /// `"backward"`, `"forward"` or `"nearest"`, sent as it is.
+ pub direction: String,
+}
+
+impl AsOfJoinOptions {
+ /// Options joining on `left_as_of` and `right_as_of`, with the defaults
of PySpark.
+ pub fn new(left_as_of: Column, right_as_of: Column) -> Self {
+ Self {
+ left_as_of,
+ right_as_of,
+ on: None,
+ using_columns: Vec::new(),
+ join_type: "inner".to_string(),
+ tolerance: None,
+ allow_exact_matches: true,
+ direction: "backward".to_string(),
+ }
+ }
+}
+
/// Returned by `DataFrame::to_local_iterator()`, this iterator consumes the
ExecutePlan
/// response stream incrementally and yields Row objects without buffering the
entire result.
pub struct LocalRowIterator {
@@ -532,6 +573,35 @@ impl DataFrame {
DataFrame::new(self.session.clone(), plan)
}
+ /// As-of join with another DataFrame: each row of this DataFrame is
matched with the
+ /// nearest row of `right` on the as-of columns, as `DataFrame._joinAsOf`
does.
+ ///
+ /// Errors with `SESSION_NOT_SAME` when `right` is of another session. The
join type and
+ /// the direction are sent as they are; the server checks them.
+ pub fn as_of_join(&self, right: &DataFrame, options: AsOfJoinOptions) ->
Result<DataFrame> {
+ // As PySpark's `_check_same_session`.
+ if self.session.client().session_id() !=
right.session.client().session_id() {
+ return Err(SparkError::classed(
+ SparkErrorKind::SessionNotSame,
+ "SESSION_NOT_SAME",
+ &[],
+ ));
+ }
+ let plan = LogicalPlan::AsOfJoin {
+ left: Arc::clone(&self.plan),
+ right: Arc::clone(&right.plan),
+ left_as_of: options.left_as_of,
+ right_as_of: options.right_as_of,
+ on: options.on,
+ using_columns: options.using_columns,
+ join_type: options.join_type,
+ tolerance: options.tolerance,
+ allow_exact_matches: options.allow_exact_matches,
+ direction: options.direction,
+ };
+ Ok(DataFrame::new(self.session.clone(), plan))
+ }
+
/// Cross join.
pub fn cross_join(&self, right: &DataFrame) -> DataFrame {
self.join(right, None, JoinType::Cross)
@@ -4312,6 +4382,82 @@ mod plan_construction_tests {
assert_eq!(joined.plan.to_proto(), expected);
}
+ /// As PySpark's `AsOfJoin` plan: the as-of columns, the names, the join
type, the tolerance,
+ /// whether exact matches are allowed, and the direction are each sent.
+ #[test]
+ fn as_of_join_sends_every_field_of_the_as_of_join() {
+ use crate::functions::col;
+ use proto::relation::RelType;
+
+ let spark = session();
+ let left = spark.range(3).unwrap();
+ let right = spark.range(4).unwrap();
+ let options = AsOfJoinOptions {
+ using_columns: vec!["id".to_string()],
+ join_type: "left".to_string(),
+ tolerance: Some(crate::column::lit(1)),
+ allow_exact_matches: false,
+ direction: "forward".to_string(),
+ ..AsOfJoinOptions::new(col("id"), col("id"))
+ };
+ let joined = left.as_of_join(&right, options).unwrap();
+ let relation = joined.plan.to_proto();
+ let Some(RelType::AsOfJoin(as_of)) = &relation.rel_type else {
+ panic!("expected AsOfJoin: {relation:?}");
+ };
+ assert_eq!(as_of.left.as_deref(), Some(&left.plan.to_proto()));
+ assert_eq!(as_of.right.as_deref(), Some(&right.plan.to_proto()));
+ assert_eq!(as_of.left_as_of, Some(col("id").to_proto()));
+ assert_eq!(as_of.right_as_of, Some(col("id").to_proto()));
+ assert_eq!(as_of.join_expr, None);
+ assert_eq!(as_of.using_columns, vec!["id".to_string()]);
+ assert_eq!(as_of.join_type, "left");
+ assert_eq!(as_of.tolerance, Some(crate::column::lit(1).to_proto()));
+ assert!(!as_of.allow_exact_matches);
+ assert_eq!(as_of.direction, "forward");
+ }
+
+ /// The defaults of PySpark's `_joinAsOf` (inner, exact matches,
backward), and a condition
+ /// sent as `join_expr` instead of names.
+ #[test]
+ fn as_of_join_defaults_and_sends_a_condition_as_join_expr() {
+ let spark = session();
+ let left = spark.range(3).unwrap();
+ let right = spark.range(4).unwrap();
+ let condition = left.col("id").eq(right.col("id"));
+ let options = AsOfJoinOptions {
+ on: Some(condition.clone()),
+ ..AsOfJoinOptions::new(left.col("id"), right.col("id"))
+ };
+ let joined = left.as_of_join(&right, options).unwrap();
+ let relation = joined.plan.to_proto();
+ let Some(proto::relation::RelType::AsOfJoin(as_of)) =
&relation.rel_type else {
+ panic!("expected AsOfJoin: {relation:?}");
+ };
+ assert_eq!(as_of.join_expr, Some(condition.to_proto()));
+ assert!(as_of.using_columns.is_empty());
+ assert_eq!(as_of.join_type, "inner");
+ assert_eq!(as_of.tolerance, None);
+ assert!(as_of.allow_exact_matches);
+ assert_eq!(as_of.direction, "backward");
+ }
+
+ /// As PySpark's `_check_same_session`: a DataFrame of another session is
not joined.
+ #[test]
+ fn as_of_join_rejects_a_dataframe_of_another_session() {
+ use crate::functions::col;
+
+ let spark = session();
+ let left = spark.range(3).unwrap();
+ let right = spark.new_session().range(3).unwrap();
+ let options = AsOfJoinOptions::new(col("id"), col("id"));
+ let error = left
+ .as_of_join(&right, options)
+ .err()
+ .expect("a session mismatch");
+ assert_eq!(error.get_error_class().as_deref(),
Some("SESSION_NOT_SAME"));
+ }
+
#[test]
fn observed_metrics_set_the_observations_of_their_relations() {
use crate::observation::Observation;
diff --git a/crates/spark-connect/src/plan.rs b/crates/spark-connect/src/plan.rs
index 82cb00a8..5d398205 100644
--- a/crates/spark-connect/src/plan.rs
+++ b/crates/spark-connect/src/plan.rs
@@ -442,6 +442,19 @@ pub enum LogicalPlan {
other_grouping_expressions: Vec<Expression>,
func: CommonInlineUserDefinedFunctionExpression,
},
+ /// AsOfJoin: `DataFrame._joinAsOf`, which pandas-on-Spark's `merge_asof`
calls.
+ AsOfJoin {
+ left: Arc<LogicalPlan>,
+ right: Arc<LogicalPlan>,
+ left_as_of: Column,
+ right_as_of: Column,
+ on: Option<Column>,
+ using_columns: Vec<String>,
+ join_type: String,
+ tolerance: Option<Column>,
+ allow_exact_matches: bool,
+ direction: String,
+ },
/// NearestByJoin: `DataFrame.nearestByJoin`.
NearestByJoin {
left: Arc<LogicalPlan>,
@@ -821,6 +834,33 @@ impl LogicalPlan {
relation.rel_type =
Some(proto::relation::RelType::Join(Box::new(join)));
}
+ LogicalPlan::AsOfJoin {
+ left,
+ right,
+ left_as_of,
+ right_as_of,
+ on,
+ using_columns,
+ join_type,
+ tolerance,
+ allow_exact_matches,
+ direction,
+ } => {
+ let aj = proto::AsOfJoin {
+ left: Some(Box::new(relation_of(left, plan_ids))),
+ right: Some(Box::new(relation_of(right, plan_ids))),
+ left_as_of: Some(left_as_of.to_proto()),
+ right_as_of: Some(right_as_of.to_proto()),
+ join_expr: on.as_ref().map(|condition|
condition.to_proto()),
+ using_columns: using_columns.clone(),
+ join_type: join_type.clone(),
+ tolerance: tolerance.as_ref().map(|tolerance|
tolerance.to_proto()),
+ allow_exact_matches: *allow_exact_matches,
+ direction: direction.clone(),
+ };
+ relation.rel_type =
Some(proto::relation::RelType::AsOfJoin(Box::new(aj)));
+ }
+
LogicalPlan::LateralJoin {
left,
right,
diff --git a/python/tests/test_dropin_e2e.py b/python/tests/test_dropin_e2e.py
index 3cf32aa6..0c3a8894 100644
--- a/python/tests/test_dropin_e2e.py
+++ b/python/tests/test_dropin_e2e.py
@@ -306,6 +306,66 @@ def
test_renames_are_applied_in_their_order_as_pyspark(session):
+def test_join_as_of_and_merge_asof_give_the_rows_of_pyspark(session):
+ # As PySpark: _joinAsOf matches each row of the left DataFrame with the
nearest row of the
+ # right one on the as-of column, as the direction, exact matches and
tolerance say, and
+ # pandas-on-Spark's merge_asof, which calls it, gives the same rows.
+ import math
+
+ pytest.importorskip("pandas")
+ import pyspark.pandas as ps
+ from pyspark.sql import functions as F
+
+ def rows(values):
+ return sorted(
+ tuple("nan" if isinstance(v, float) and math.isnan(v) else v for v
in row)
+ for row in values
+ )
+
+ left = session.createDataFrame(
+ [(1, 1.0, "a"), (3, 3.0, "c"), (5, 5.0, "b")], "t int, x double, k
string")
+ right = session.createDataFrame(
+ [(2, 2.0, "x"), (3, 3.5, "c"), (4, 4.0, "y")], "t int, y double, k
string")
+ joined = rows(left._joinAsOf(right, "t", "t", how="left").collect())
+ assert joined == [
+ (1, 1.0, "a", None, None, None), (3, 3.0, "c", 3, 3.5, "c"), (5, 5.0,
"b", 4, 4.0, "y")]
+ assert rows(left._joinAsOf(right, "t", "t", how="left",
allowExactMatches=False).collect()) == [
+ (1, 1.0, "a", None, None, None), (3, 3.0, "c", 2, 2.0, "x"), (5, 5.0,
"b", 4, 4.0, "y")]
+ assert rows(left._joinAsOf(right, "t", "t", how="left",
direction="forward").collect()) == [
+ (1, 1.0, "a", 2, 2.0, "x"), (3, 3.0, "c", 3, 3.5, "c"), (5, 5.0, "b",
None, None, None)]
+ assert rows(left._joinAsOf(right, "t", "t", how="left",
direction="nearest").collect()) == [
+ (1, 1.0, "a", 2, 2.0, "x"), (3, 3.0, "c", 3, 3.5, "c"), (5, 5.0, "b",
4, 4.0, "y")]
+ assert rows(left._joinAsOf(
+ right, "t", "t", how="left", tolerance=F.lit(1)).collect()) == [
+ (1, 1.0, "a", None, None, None), (3, 3.0, "c", 3, 3.5, "c"), (5, 5.0,
"b", 4, 4.0, "y")]
+ assert rows(left._joinAsOf(right, "t", "t", on=["k"],
how="left").collect()) == [
+ (1, 1.0, "a", None, None, None),
+ (3, 3.0, "c", 3, 3.5, "c"),
+ (5, 5.0, "b", None, None, None),
+ ]
+ assert rows(left._joinAsOf(right, "t", "t").collect()) == [
+ (3, 3.0, "c", 3, 3.5, "c"), (5, 5.0, "b", 4, 4.0, "y")]
+
+ # pandas-on-Spark's ANSI check is off for the merges, as in the tests of
PySpark.
+ key = "spark.sql.ansi.enabled"
+ session.conf.set(key, "false")
+ try:
+ def merged(**kwargs):
+ frame = ps.merge_asof(ps.DataFrame(left), ps.DataFrame(right),
**kwargs)
+ return rows(frame.to_pandas().values.tolist())
+
+ assert merged(on="t") == [
+ (1, 1.0, "a", "nan", None), (3, 3.0, "c", 3.5, "c"), (5, 5.0, "b",
4.0, "y")]
+ assert merged(on="t", by="k", direction="forward") == [
+ (1, 1.0, "a", "nan"), (3, 3.0, "c", 3.5), (5, 5.0, "b", "nan")]
+ assert merged(on="t", tolerance=1, allow_exact_matches=False) == [
+ (1, 1.0, "a", "nan", None), (3, 3.0, "c", "nan", None), (5, 5.0,
"b", "nan", None)]
+ assert merged(left_on="t", right_on="t", left_by="k", right_by="k") ==
[
+ (1, 1.0, "a", "nan"), (3, 3.0, "c", 3.5), (5, 5.0, "b", "nan")]
+ finally:
+ session.conf.unset(key)
+
+
def test_join_with_an_unsupported_join_type_raises_as_pyspark(session):
# As PySpark: the join type is checked when the join is built, which
raises the
# AnalysisException of UNSUPPORTED_JOIN_TYPE, naming the type as it is
normalized.
diff --git a/python/tests/test_dropin_offline.py
b/python/tests/test_dropin_offline.py
index 34f7a1f1..252cd1ae 100644
--- a/python/tests/test_dropin_offline.py
+++ b/python/tests/test_dropin_offline.py
@@ -7397,6 +7397,44 @@ if __name__ == "__main__":
assert result.stdout.strip().splitlines()[-1] == str(["col"] * 6)
+def test_join_as_of_sends_the_as_of_join_as_pyspark(mock_connect):
+ # As PySpark's plan.AsOfJoin: Relation.as_of_join=39, with left=1,
right=2, left_as_of=3,
+ # right_as_of=4, join_expr=5, using_columns=6, join_type=7, tolerance=8,
+ # allow_exact_matches=9 and direction=10. The as-of columns are the
columns of their
+ # DataFrames, and `on` is a condition, or names when it is a name or a
list of names.
+ from pyspark.sql import functions as F
+
+ spark, service = mock_connect
+ left, right = spark.range(2), spark.range(3)
+
+ def sent_as_of_join(joined):
+ joined.collect()
+ # ExecutePlanRequest.plan=3, Plan.root=1, Relation.as_of_join=39.
+ plan =
parse_message(parse_message(service.execute_requests[-1])[3][0])[1][0]
+ return parse_message(parse_message(plan)[39][0])
+
+ def attribute(expression):
+ # Expression.unresolved_attribute=2,
UnresolvedAttribute.unparsed_identifier=1.
+ return parse_message(parse_message(expression)[2][0])[1][0].decode()
+
+ join = sent_as_of_join(left._joinAsOf(
+ right, "id", "id", on=["id"], how="left", tolerance=F.lit(1),
+ allowExactMatches=False, direction="forward"))
+ assert attribute(join[3][0]) == "id" and attribute(join[4][0]) == "id"
+ assert 1 in join and 2 in join and 5 not in join and join[6] == [b"id"]
+ assert join[7] == [b"left"] and 8 in join
+ assert join.get(9, [0]) == [0] and join[10] == [b"forward"]
+
+ join = sent_as_of_join(left._joinAsOf(right, "id", "id"))
+ assert 5 not in join and 6 not in join and 8 not in join
+ assert join[7] == [b"inner"] and join[9] == [1] and join[10] ==
[b"backward"]
+
+ join = sent_as_of_join(left._joinAsOf(
+ right, left.id, right.id, on=left.id == right.id, how="left"))
+ assert attribute(join[3][0]) == "id" and 5 in join and 6 not in join
+ assert join[7] == [b"left"] and join[10] == [b"backward"]
+
+
def
test_grouping_sets_group_by_their_grouping_columns_as_pyspark(mock_connect):
# As PySpark's, groupingSets(groupingSets, *cols) groups by the sets with
the grouping
# columns `cols`: Relation.aggregate=9, Aggregate.group_type=2
(GROUPING_SETS=5),
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]