This is an automated email from the ASF dual-hosted git repository.

HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git


The following commit(s) were added to refs/heads/master by this push:
     new 9f264b21 [SPARK-61180][CONNECT][PYTHON] Add DataFrame._joinAsOf, the 
as-of join of pandas-on-Spark's merge_asof, in the Rust and Python clients as 
PySpark does
9f264b21 is described below

commit 9f264b21ecc721b65b2e97e5b0f8f631e2405cd0
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Mon Nov 9 06:20:55 2026 +0900

    [SPARK-61180][CONNECT][PYTHON] Add DataFrame._joinAsOf, the as-of join of 
pandas-on-Spark's merge_asof, in the Rust and Python clients as PySpark does
    
    ### What changes were proposed in this pull request?
    
    [SPARK-61180](https://issues.apache.org/jira/browse/SPARK-61180): PySpark 
Connect's `DataFrame._joinAsOf(other, leftAsOfColumn, rightAsOfColumn, on=None, 
how=None, *, tolerance=None, allowExactMatches=True, direction="backward")` 
builds the Spark Connect `AsOfJoin` relation. pandas-on-Spark's `merge_asof` 
calls it.
    
    - Rust core: `DataFrame::as_of_join(&right, AsOfJoinOptions)` with a new 
`AsOfJoinOptions` struct (`AsOfJoinOptions::new(left_as_of, right_as_of)` sets 
the PySpark defaults: inner, exact matches allowed, `"backward"`). The plan is 
a new `LogicalPlan::AsOfJoin`, serialized to `AsOfJoin` with `left_as_of`, 
`right_as_of`, `join_expr` or `using_columns`, `join_type`, `tolerance`, 
`allow_exact_matches` and `direction`. A DataFrame of another session fails 
with `SESSION_NOT_SAME`.
    - PyO3: `DataFrame._joinAsOf` with PySpark's signature. `how` defaults to 
inner and must be a string. `tolerance` must be a Column. `on` is a name, a 
Column, or a list of names or Columns, as in `plan.AsOfJoin`. A string as-of 
column is a column of its DataFrame, as `df._col(name)`. `join_type` and 
`direction` are sent as they are, so the server checks them as it does for 
PySpark.
    
    ### Why are the changes needed?
    
    `ps.merge_asof` failed, because `_joinAsOf` did not exist and the name 
resolved through `DataFrame.__getattr__` to a Column:
    
    ```
    >>> ps.merge_asof(ps.DataFrame(L), ps.DataFrame(R), on="t")
    TypeError: 'pyspark.sql.connect.column.Column' object is not callable
    ```
    
    PySpark returns `[[1, 1.0, 'a', nan, None], [5, 5.0, 'b', 4.0, 'y']]` for L 
= `[(1, 1.0, "a"), (5, 5.0, "b")]` ("t int, x double, k string") and R = `[(2, 
2.0, "x"), (4, 4.0, "y")]` ("t int, y double, k string").
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes. `DataFrame._joinAsOf` and `pandas.merge_asof` (pandas-on-Spark) work 
on the Rust client as they do on PySpark Connect. Before, they raised the 
`TypeError` above.
    
    ### How was this patch tested?
    
    - `test_join_as_of_sends_the_as_of_join_as_pyspark` (offline, 
`python/tests/test_dropin_offline.py`): checks the fields of the sent 
`AsOfJoin` relation for a list `on` with `how`, `tolerance`, 
`allowExactMatches=False` and `direction`, for the defaults, and for a Column 
`on`.
    - `test_join_as_of_and_merge_asof_give_the_rows_of_pyspark` (e2e, 
`python/tests/test_dropin_e2e.py`): the rows of `_joinAsOf` for each direction, 
`allowExactMatches`, tolerance, `on` as a list, and the inner default, and the 
rows of `ps.merge_asof` with `on`, `by`, `direction`, `tolerance`, and 
`left_on`/`right_on`/`left_by`/`right_by`.
    - `as_of_join_sends_every_field_of_the_as_of_join`, 
`as_of_join_defaults_and_sends_a_condition_as_join_expr` and 
`as_of_join_rejects_a_dataframe_of_another_session` (Rust unit tests, 
`crates/spark-connect/src/dataframe.rs`).
    - The offline test fails without this change, with `TypeError: 
'pyspark.sql.connect.column.Column' object is not callable`. The same cases 
give the same schemas, rows, and error messages on official PySpark 4.3.0 
against the same server. The explain text matches apart from plan IDs.
    - The lib tests of both crates (611 passed), the build of all tests, also 
with `wasm-udf`, the offline Python suites (2228 passed, 8 skipped), and the 
live drop-in and Arrow suites (211 passed) and live cases (10 passed) against a 
local Spark 4.3.0 server. Clippy reports no diagnostics in the changed lines.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Opus 5.5
    
    This pull request and its description were written by Isaac.
    
    Closes #376 from HyukjinKwon/SPARK-61180-join-as-of.
    
    Authored-by: Hyukjin Kwon <[email protected]>
    Signed-off-by: Hyukjin Kwon <[email protected]>
---
 crates/pyspark-rs/src/dataframe.rs    | 135 ++++++++++++++++++++++++++++++-
 crates/spark-connect/src/dataframe.rs | 148 +++++++++++++++++++++++++++++++++-
 crates/spark-connect/src/plan.rs      |  40 +++++++++
 python/tests/test_dropin_e2e.py       |  60 ++++++++++++++
 python/tests/test_dropin_offline.py   |  38 +++++++++
 5 files changed, 419 insertions(+), 2 deletions(-)

diff --git a/crates/pyspark-rs/src/dataframe.rs 
b/crates/pyspark-rs/src/dataframe.rs
index 670766ed..4b4d9d0f 100644
--- a/crates/pyspark-rs/src/dataframe.rs
+++ b/crates/pyspark-rs/src/dataframe.rs
@@ -2,7 +2,7 @@
 
 use pyo3::prelude::*;
 use pyo3::types::{PyDict, PyInt, PyList, PyTuple};
-use spark_connect::dataframe::{DataFrame, LocalRowIterator};
+use spark_connect::dataframe::{AsOfJoinOptions, DataFrame, LocalRowIterator};
 use spark_connect::plan::JoinType;
 use spark_connect::udf::{eval_type, CommonInlineUserDefinedFunctionExpression, 
PythonUDFPayload};
 
@@ -526,6 +526,83 @@ fn no_to_plan(obj: &Bound<'_, PyAny>) -> PyErr {
     ))
 }
 
+/// The `TypeError` of protobuf for a name that is not a string, which the 
plan of PySpark's
+/// `_joinAsOf` raises for a name in `on` or an as-of column that is not a 
string.
+fn proto_string_type_error(value: &Bound<'_, PyAny>) -> PyErr {
+    let repr = value
+        .repr()
+        .map(|repr| repr.to_string())
+        .unwrap_or_default();
+    let typ = value
+        .get_type()
+        .repr()
+        .map(|typ| typ.to_string())
+        .unwrap_or_default();
+    pyo3::exceptions::PyTypeError::new_err(format!(
+        "{repr} has type {typ}, but expected one of: (<class 'bytes'>, <class 
'str'>)"
+    ))
+}
+
+/// The join condition of `_joinAsOf` for its `on`, as PySpark's `AsOfJoin` 
plan sends it: a
+/// name or a list of names (`using_columns`), or a Column or a list of 
Columns, which are
+/// joined on their conjunction (`join_expr`).
+fn as_of_join_condition(
+    on: &Bound<'_, PyAny>,
+) -> PyResult<(Option<spark_connect::column::Column>, Vec<String>)> {
+    if let Ok(column) = on.extract::<PyRef<PyColumn>>() {
+        return Ok((Some(column.column.clone()), Vec::new()));
+    }
+    if on.is_instance_of::<pyo3::types::PyString>() {
+        return Ok((None, vec![on.extract::<String>()?]));
+    }
+    // As PySpark's `_collect_references`, which iterates `on` first, and then 
its `to_plan`.
+    if !on.is_instance_of::<PyList>() {
+        on.try_iter()?;
+        return Err(no_to_plan(on));
+    }
+    let list = on.cast::<PyList>()?;
+    // An empty list is no condition at all.
+    let Some(first) = list.iter().next() else {
+        return Ok((None, Vec::new()));
+    };
+    if first.is_instance_of::<pyo3::types::PyString>() {
+        let names = list
+            .iter()
+            .map(|item| {
+                item.extract::<String>()
+                    .map_err(|_| proto_string_type_error(&item))
+            })
+            .collect::<PyResult<Vec<_>>>()?;
+        return Ok((None, names));
+    }
+    let mut condition: Option<spark_connect::column::Column> = None;
+    for item in list.iter() {
+        let Ok(col) = item.extract::<PyRef<PyColumn>>() else {
+            return Err(no_to_plan(&item));
+        };
+        condition = Some(match condition {
+            None => col.column.clone(),
+            Some(left) => left.and(col.column.clone()),
+        });
+    }
+    Ok((condition, Vec::new()))
+}
+
+/// The as-of column of `_joinAsOf` of `df`: a Column as it is, or the column 
of `df` that
+/// has this name, as PySpark's `df._col(name)`.
+fn as_of_column(
+    df: &DataFrame,
+    column: &Bound<'_, PyAny>,
+) -> PyResult<spark_connect::column::Column> {
+    if let Ok(column) = column.extract::<PyRef<PyColumn>>() {
+        return Ok(column.column.clone());
+    }
+    let name = column
+        .extract::<String>()
+        .map_err(|_| proto_string_type_error(column))?;
+    Ok(df.col(&name))
+}
+
 /// Resolve a filter/where condition: a Column, or a SQL string parsed via 
`expr`.
 fn cond_column(condition: &Bound<'_, PyAny>) -> 
PyResult<spark_connect::column::Column> {
     if let Ok(c) = condition.extract::<PyRef<PyColumn>>() {
@@ -941,6 +1018,62 @@ impl PyDataFrame {
         Err(no_to_plan(&on))
     }
 
+    /// `DataFrame._joinAsOf(other, leftAsOfColumn, rightAsOfColumn, on=None, 
how=None, *,
+    /// tolerance=None, allowExactMatches=True, direction="backward")`, the 
as-of join of
+    /// pandas-on-Spark's `merge_asof`, as PySpark's. The as-of columns are 
Columns, or names of
+    /// their DataFrame. `on` is a name, a Column condition, or a list of 
either. `how` (inner by
+    /// default) and `direction` are sent as they are, and the server checks 
them.
+    #[pyo3(name = "_joinAsOf", signature = (other, leftAsOfColumn, 
rightAsOfColumn, on=None, how=None, *, tolerance=None, allowExactMatches=true, 
direction="backward"))]
+    #[allow(non_snake_case, clippy::too_many_arguments)]
+    fn join_as_of(
+        &self,
+        other: &Bound<'_, PyAny>,
+        leftAsOfColumn: &Bound<'_, PyAny>,
+        rightAsOfColumn: &Bound<'_, PyAny>,
+        on: Option<Bound<'_, PyAny>>,
+        how: Option<Bound<'_, PyAny>>,
+        tolerance: Option<Bound<'_, PyAny>>,
+        allowExactMatches: bool,
+        direction: &str,
+    ) -> PyResult<PyDataFrame> {
+        // As PySpark's `_check_same_session`, first.
+        let other = self.same_session(other)?;
+        let join_type = match how.filter(|how| !how.is_none()) {
+            None => "inner".to_string(),
+            Some(how) => how.extract::<String>().map_err(|_| {
+                pyo3::exceptions::PyAssertionError::new_err("how should be a 
string")
+            })?,
+        };
+        let tolerance = match tolerance.filter(|tolerance| 
!tolerance.is_none()) {
+            None => None,
+            Some(tolerance) => {
+                let column = 
tolerance.extract::<PyRef<PyColumn>>().map_err(|_| {
+                    pyo3::exceptions::PyAssertionError::new_err("tolerance 
should be Column")
+                })?;
+                Some(column.column.clone())
+            }
+        };
+        let (on, using_columns) = match on.filter(|on| !on.is_none()) {
+            None => (None, Vec::new()),
+            Some(on) => as_of_join_condition(&on)?,
+        };
+        let options = AsOfJoinOptions {
+            left_as_of: as_of_column(&self.dataframe, leftAsOfColumn)?,
+            right_as_of: as_of_column(&other.dataframe, rightAsOfColumn)?,
+            on,
+            using_columns,
+            join_type,
+            tolerance,
+            allow_exact_matches: allowExactMatches,
+            direction: direction.to_string(),
+        };
+        let df = self
+            .dataframe
+            .as_of_join(&other.dataframe, options)
+            .to_pyerr()?;
+        Ok(PyDataFrame::new(df))
+    }
+
     /// Cross join.
     fn crossJoin(&self, other: &Bound<'_, PyAny>) -> PyResult<PyDataFrame> {
         let other = self.same_session(other)?;
diff --git a/crates/spark-connect/src/dataframe.rs 
b/crates/spark-connect/src/dataframe.rs
index 6e6cb56f..32b4c9c9 100644
--- a/crates/spark-connect/src/dataframe.rs
+++ b/crates/spark-connect/src/dataframe.rs
@@ -5,7 +5,7 @@
 use std::collections::HashMap;
 use std::sync::{Arc, Mutex};
 
-use spark_connect_core::error::{Result, SparkError};
+use spark_connect_core::error::{Result, SparkError, SparkErrorKind};
 use spark_connect_core::runtime::{block_on, get_runtime};
 use spark_connect_proto as proto;
 
@@ -41,6 +41,47 @@ pub(crate) type ExecutionSlot = 
Arc<Mutex<Option<ExecutionInfo>>>;
 
 /// Iterator over rows from a DataFrame, yielded lazily as the server streams 
results.
 ///
+/// The options of an as-of join, as `DataFrame._joinAsOf` takes them.
+///
+/// `on` and `using_columns` are the join condition: a condition, or names 
that both sides
+/// share. PySpark sets at most one of them. The defaults are those of 
PySpark: an inner join,
+/// exact matches allowed, and the `"backward"` direction.
+#[derive(Debug, Clone)]
+pub struct AsOfJoinOptions {
+    /// The as-of column of this DataFrame.
+    pub left_as_of: Column,
+    /// The as-of column of the other DataFrame.
+    pub right_as_of: Column,
+    /// The join condition, sent as `join_expr`.
+    pub on: Option<Column>,
+    /// Column names that both sides share, sent as `using_columns`.
+    pub using_columns: Vec<String>,
+    /// The join type, such as `"inner"` or `"left"`, sent as it is.
+    pub join_type: String,
+    /// The as-of tolerance: the largest distance a match may have.
+    pub tolerance: Option<Column>,
+    /// Whether a row may match a row with the same as-of value.
+    pub allow_exact_matches: bool,
+    /// `"backward"`, `"forward"` or `"nearest"`, sent as it is.
+    pub direction: String,
+}
+
+impl AsOfJoinOptions {
+    /// Options joining on `left_as_of` and `right_as_of`, with the defaults 
of PySpark.
+    pub fn new(left_as_of: Column, right_as_of: Column) -> Self {
+        Self {
+            left_as_of,
+            right_as_of,
+            on: None,
+            using_columns: Vec::new(),
+            join_type: "inner".to_string(),
+            tolerance: None,
+            allow_exact_matches: true,
+            direction: "backward".to_string(),
+        }
+    }
+}
+
 /// Returned by `DataFrame::to_local_iterator()`, this iterator consumes the 
ExecutePlan
 /// response stream incrementally and yields Row objects without buffering the 
entire result.
 pub struct LocalRowIterator {
@@ -532,6 +573,35 @@ impl DataFrame {
         DataFrame::new(self.session.clone(), plan)
     }
 
+    /// As-of join with another DataFrame: each row of this DataFrame is 
matched with the
+    /// nearest row of `right` on the as-of columns, as `DataFrame._joinAsOf` 
does.
+    ///
+    /// Errors with `SESSION_NOT_SAME` when `right` is of another session. The 
join type and
+    /// the direction are sent as they are; the server checks them.
+    pub fn as_of_join(&self, right: &DataFrame, options: AsOfJoinOptions) -> 
Result<DataFrame> {
+        // As PySpark's `_check_same_session`.
+        if self.session.client().session_id() != 
right.session.client().session_id() {
+            return Err(SparkError::classed(
+                SparkErrorKind::SessionNotSame,
+                "SESSION_NOT_SAME",
+                &[],
+            ));
+        }
+        let plan = LogicalPlan::AsOfJoin {
+            left: Arc::clone(&self.plan),
+            right: Arc::clone(&right.plan),
+            left_as_of: options.left_as_of,
+            right_as_of: options.right_as_of,
+            on: options.on,
+            using_columns: options.using_columns,
+            join_type: options.join_type,
+            tolerance: options.tolerance,
+            allow_exact_matches: options.allow_exact_matches,
+            direction: options.direction,
+        };
+        Ok(DataFrame::new(self.session.clone(), plan))
+    }
+
     /// Cross join.
     pub fn cross_join(&self, right: &DataFrame) -> DataFrame {
         self.join(right, None, JoinType::Cross)
@@ -4312,6 +4382,82 @@ mod plan_construction_tests {
         assert_eq!(joined.plan.to_proto(), expected);
     }
 
+    /// As PySpark's `AsOfJoin` plan: the as-of columns, the names, the join 
type, the tolerance,
+    /// whether exact matches are allowed, and the direction are each sent.
+    #[test]
+    fn as_of_join_sends_every_field_of_the_as_of_join() {
+        use crate::functions::col;
+        use proto::relation::RelType;
+
+        let spark = session();
+        let left = spark.range(3).unwrap();
+        let right = spark.range(4).unwrap();
+        let options = AsOfJoinOptions {
+            using_columns: vec!["id".to_string()],
+            join_type: "left".to_string(),
+            tolerance: Some(crate::column::lit(1)),
+            allow_exact_matches: false,
+            direction: "forward".to_string(),
+            ..AsOfJoinOptions::new(col("id"), col("id"))
+        };
+        let joined = left.as_of_join(&right, options).unwrap();
+        let relation = joined.plan.to_proto();
+        let Some(RelType::AsOfJoin(as_of)) = &relation.rel_type else {
+            panic!("expected AsOfJoin: {relation:?}");
+        };
+        assert_eq!(as_of.left.as_deref(), Some(&left.plan.to_proto()));
+        assert_eq!(as_of.right.as_deref(), Some(&right.plan.to_proto()));
+        assert_eq!(as_of.left_as_of, Some(col("id").to_proto()));
+        assert_eq!(as_of.right_as_of, Some(col("id").to_proto()));
+        assert_eq!(as_of.join_expr, None);
+        assert_eq!(as_of.using_columns, vec!["id".to_string()]);
+        assert_eq!(as_of.join_type, "left");
+        assert_eq!(as_of.tolerance, Some(crate::column::lit(1).to_proto()));
+        assert!(!as_of.allow_exact_matches);
+        assert_eq!(as_of.direction, "forward");
+    }
+
+    /// The defaults of PySpark's `_joinAsOf` (inner, exact matches, 
backward), and a condition
+    /// sent as `join_expr` instead of names.
+    #[test]
+    fn as_of_join_defaults_and_sends_a_condition_as_join_expr() {
+        let spark = session();
+        let left = spark.range(3).unwrap();
+        let right = spark.range(4).unwrap();
+        let condition = left.col("id").eq(right.col("id"));
+        let options = AsOfJoinOptions {
+            on: Some(condition.clone()),
+            ..AsOfJoinOptions::new(left.col("id"), right.col("id"))
+        };
+        let joined = left.as_of_join(&right, options).unwrap();
+        let relation = joined.plan.to_proto();
+        let Some(proto::relation::RelType::AsOfJoin(as_of)) = 
&relation.rel_type else {
+            panic!("expected AsOfJoin: {relation:?}");
+        };
+        assert_eq!(as_of.join_expr, Some(condition.to_proto()));
+        assert!(as_of.using_columns.is_empty());
+        assert_eq!(as_of.join_type, "inner");
+        assert_eq!(as_of.tolerance, None);
+        assert!(as_of.allow_exact_matches);
+        assert_eq!(as_of.direction, "backward");
+    }
+
+    /// As PySpark's `_check_same_session`: a DataFrame of another session is 
not joined.
+    #[test]
+    fn as_of_join_rejects_a_dataframe_of_another_session() {
+        use crate::functions::col;
+
+        let spark = session();
+        let left = spark.range(3).unwrap();
+        let right = spark.new_session().range(3).unwrap();
+        let options = AsOfJoinOptions::new(col("id"), col("id"));
+        let error = left
+            .as_of_join(&right, options)
+            .err()
+            .expect("a session mismatch");
+        assert_eq!(error.get_error_class().as_deref(), 
Some("SESSION_NOT_SAME"));
+    }
+
     #[test]
     fn observed_metrics_set_the_observations_of_their_relations() {
         use crate::observation::Observation;
diff --git a/crates/spark-connect/src/plan.rs b/crates/spark-connect/src/plan.rs
index 82cb00a8..5d398205 100644
--- a/crates/spark-connect/src/plan.rs
+++ b/crates/spark-connect/src/plan.rs
@@ -442,6 +442,19 @@ pub enum LogicalPlan {
         other_grouping_expressions: Vec<Expression>,
         func: CommonInlineUserDefinedFunctionExpression,
     },
+    /// AsOfJoin: `DataFrame._joinAsOf`, which pandas-on-Spark's `merge_asof` 
calls.
+    AsOfJoin {
+        left: Arc<LogicalPlan>,
+        right: Arc<LogicalPlan>,
+        left_as_of: Column,
+        right_as_of: Column,
+        on: Option<Column>,
+        using_columns: Vec<String>,
+        join_type: String,
+        tolerance: Option<Column>,
+        allow_exact_matches: bool,
+        direction: String,
+    },
     /// NearestByJoin: `DataFrame.nearestByJoin`.
     NearestByJoin {
         left: Arc<LogicalPlan>,
@@ -821,6 +834,33 @@ impl LogicalPlan {
                 relation.rel_type = 
Some(proto::relation::RelType::Join(Box::new(join)));
             }
 
+            LogicalPlan::AsOfJoin {
+                left,
+                right,
+                left_as_of,
+                right_as_of,
+                on,
+                using_columns,
+                join_type,
+                tolerance,
+                allow_exact_matches,
+                direction,
+            } => {
+                let aj = proto::AsOfJoin {
+                    left: Some(Box::new(relation_of(left, plan_ids))),
+                    right: Some(Box::new(relation_of(right, plan_ids))),
+                    left_as_of: Some(left_as_of.to_proto()),
+                    right_as_of: Some(right_as_of.to_proto()),
+                    join_expr: on.as_ref().map(|condition| 
condition.to_proto()),
+                    using_columns: using_columns.clone(),
+                    join_type: join_type.clone(),
+                    tolerance: tolerance.as_ref().map(|tolerance| 
tolerance.to_proto()),
+                    allow_exact_matches: *allow_exact_matches,
+                    direction: direction.clone(),
+                };
+                relation.rel_type = 
Some(proto::relation::RelType::AsOfJoin(Box::new(aj)));
+            }
+
             LogicalPlan::LateralJoin {
                 left,
                 right,
diff --git a/python/tests/test_dropin_e2e.py b/python/tests/test_dropin_e2e.py
index 3cf32aa6..0c3a8894 100644
--- a/python/tests/test_dropin_e2e.py
+++ b/python/tests/test_dropin_e2e.py
@@ -306,6 +306,66 @@ def 
test_renames_are_applied_in_their_order_as_pyspark(session):
 
 
 
+def test_join_as_of_and_merge_asof_give_the_rows_of_pyspark(session):
+    # As PySpark: _joinAsOf matches each row of the left DataFrame with the 
nearest row of the
+    # right one on the as-of column, as the direction, exact matches and 
tolerance say, and
+    # pandas-on-Spark's merge_asof, which calls it, gives the same rows.
+    import math
+
+    pytest.importorskip("pandas")
+    import pyspark.pandas as ps
+    from pyspark.sql import functions as F
+
+    def rows(values):
+        return sorted(
+            tuple("nan" if isinstance(v, float) and math.isnan(v) else v for v 
in row)
+            for row in values
+        )
+
+    left = session.createDataFrame(
+        [(1, 1.0, "a"), (3, 3.0, "c"), (5, 5.0, "b")], "t int, x double, k 
string")
+    right = session.createDataFrame(
+        [(2, 2.0, "x"), (3, 3.5, "c"), (4, 4.0, "y")], "t int, y double, k 
string")
+    joined = rows(left._joinAsOf(right, "t", "t", how="left").collect())
+    assert joined == [
+        (1, 1.0, "a", None, None, None), (3, 3.0, "c", 3, 3.5, "c"), (5, 5.0, 
"b", 4, 4.0, "y")]
+    assert rows(left._joinAsOf(right, "t", "t", how="left", 
allowExactMatches=False).collect()) == [
+        (1, 1.0, "a", None, None, None), (3, 3.0, "c", 2, 2.0, "x"), (5, 5.0, 
"b", 4, 4.0, "y")]
+    assert rows(left._joinAsOf(right, "t", "t", how="left", 
direction="forward").collect()) == [
+        (1, 1.0, "a", 2, 2.0, "x"), (3, 3.0, "c", 3, 3.5, "c"), (5, 5.0, "b", 
None, None, None)]
+    assert rows(left._joinAsOf(right, "t", "t", how="left", 
direction="nearest").collect()) == [
+        (1, 1.0, "a", 2, 2.0, "x"), (3, 3.0, "c", 3, 3.5, "c"), (5, 5.0, "b", 
4, 4.0, "y")]
+    assert rows(left._joinAsOf(
+        right, "t", "t", how="left", tolerance=F.lit(1)).collect()) == [
+        (1, 1.0, "a", None, None, None), (3, 3.0, "c", 3, 3.5, "c"), (5, 5.0, 
"b", 4, 4.0, "y")]
+    assert rows(left._joinAsOf(right, "t", "t", on=["k"], 
how="left").collect()) == [
+        (1, 1.0, "a", None, None, None),
+        (3, 3.0, "c", 3, 3.5, "c"),
+        (5, 5.0, "b", None, None, None),
+    ]
+    assert rows(left._joinAsOf(right, "t", "t").collect()) == [
+        (3, 3.0, "c", 3, 3.5, "c"), (5, 5.0, "b", 4, 4.0, "y")]
+
+    # pandas-on-Spark's ANSI check is off for the merges, as in the tests of 
PySpark.
+    key = "spark.sql.ansi.enabled"
+    session.conf.set(key, "false")
+    try:
+        def merged(**kwargs):
+            frame = ps.merge_asof(ps.DataFrame(left), ps.DataFrame(right), 
**kwargs)
+            return rows(frame.to_pandas().values.tolist())
+
+        assert merged(on="t") == [
+            (1, 1.0, "a", "nan", None), (3, 3.0, "c", 3.5, "c"), (5, 5.0, "b", 
4.0, "y")]
+        assert merged(on="t", by="k", direction="forward") == [
+            (1, 1.0, "a", "nan"), (3, 3.0, "c", 3.5), (5, 5.0, "b", "nan")]
+        assert merged(on="t", tolerance=1, allow_exact_matches=False) == [
+            (1, 1.0, "a", "nan", None), (3, 3.0, "c", "nan", None), (5, 5.0, 
"b", "nan", None)]
+        assert merged(left_on="t", right_on="t", left_by="k", right_by="k") == 
[
+            (1, 1.0, "a", "nan"), (3, 3.0, "c", 3.5), (5, 5.0, "b", "nan")]
+    finally:
+        session.conf.unset(key)
+
+
 def test_join_with_an_unsupported_join_type_raises_as_pyspark(session):
     # As PySpark: the join type is checked when the join is built, which 
raises the
     # AnalysisException of UNSUPPORTED_JOIN_TYPE, naming the type as it is 
normalized.
diff --git a/python/tests/test_dropin_offline.py 
b/python/tests/test_dropin_offline.py
index 34f7a1f1..252cd1ae 100644
--- a/python/tests/test_dropin_offline.py
+++ b/python/tests/test_dropin_offline.py
@@ -7397,6 +7397,44 @@ if __name__ == "__main__":
     assert result.stdout.strip().splitlines()[-1] == str(["col"] * 6)
 
 
+def test_join_as_of_sends_the_as_of_join_as_pyspark(mock_connect):
+    # As PySpark's plan.AsOfJoin: Relation.as_of_join=39, with left=1, 
right=2, left_as_of=3,
+    # right_as_of=4, join_expr=5, using_columns=6, join_type=7, tolerance=8,
+    # allow_exact_matches=9 and direction=10. The as-of columns are the 
columns of their
+    # DataFrames, and `on` is a condition, or names when it is a name or a 
list of names.
+    from pyspark.sql import functions as F
+
+    spark, service = mock_connect
+    left, right = spark.range(2), spark.range(3)
+
+    def sent_as_of_join(joined):
+        joined.collect()
+        # ExecutePlanRequest.plan=3, Plan.root=1, Relation.as_of_join=39.
+        plan = 
parse_message(parse_message(service.execute_requests[-1])[3][0])[1][0]
+        return parse_message(parse_message(plan)[39][0])
+
+    def attribute(expression):
+        # Expression.unresolved_attribute=2, 
UnresolvedAttribute.unparsed_identifier=1.
+        return parse_message(parse_message(expression)[2][0])[1][0].decode()
+
+    join = sent_as_of_join(left._joinAsOf(
+        right, "id", "id", on=["id"], how="left", tolerance=F.lit(1),
+        allowExactMatches=False, direction="forward"))
+    assert attribute(join[3][0]) == "id" and attribute(join[4][0]) == "id"
+    assert 1 in join and 2 in join and 5 not in join and join[6] == [b"id"]
+    assert join[7] == [b"left"] and 8 in join
+    assert join.get(9, [0]) == [0] and join[10] == [b"forward"]
+
+    join = sent_as_of_join(left._joinAsOf(right, "id", "id"))
+    assert 5 not in join and 6 not in join and 8 not in join
+    assert join[7] == [b"inner"] and join[9] == [1] and join[10] == 
[b"backward"]
+
+    join = sent_as_of_join(left._joinAsOf(
+        right, left.id, right.id, on=left.id == right.id, how="left"))
+    assert attribute(join[3][0]) == "id" and 5 in join and 6 not in join
+    assert join[7] == [b"left"] and join[10] == [b"backward"]
+
+
 def 
test_grouping_sets_group_by_their_grouping_columns_as_pyspark(mock_connect):
     # As PySpark's, groupingSets(groupingSets, *cols) groups by the sets with 
the grouping
     # columns `cols`: Relation.aggregate=9, Aggregate.group_type=2 
(GROUPING_SETS=5),


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to