[
https://issues.apache.org/jira/browse/SPARK-51847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-51847:
-----------------------------------
Labels: pull-request-available (was: )
> Extend PySpark testing framework util functions with basic data tests
> ---------------------------------------------------------------------
>
> Key: SPARK-51847
> URL: https://issues.apache.org/jira/browse/SPARK-51847
> Project: Spark
> Issue Type: New Feature
> Components: PySpark, Tests
> Affects Versions: 4.0.0
> Reporter: Stan Lochtenberg
> Priority: Major
> Labels: pull-request-available
>
> *Background*
> The PySpark testing framework currently provides utilities like
> assertDataFrameEqual and assertSchemaEqual for testing DataFrame operations.
> However, it lacks utilities for common data quality and integrity tests that
> are essential for data validation in ETL pipelines and data applications.
> *Proposal*
> Extend the PySpark testing framework with four new utility functions that
> enable developers to perform common data quality tests:
> # assertColumnUnique: Verifies that specified column(s) contain only unique
> values.
> # assertColumnNonNull: Checks that specified column(s) do not contain null
> values.
> # assertColumnValuesInSet: Ensures all values in specified column(s) are
> within a given set of accepted values.
> # assertReferentialIntegrity: Validates that all non-null values in a source
> column exist in a target column (similar to foreign key constraints).
> *Benefits*
> * Simplifies data validation in PySpark applications and tests
> * Reduces boilerplate code for common data quality checks
> * Provides consistent error reporting for data quality issues
> * Enables testing patterns similar to those in popular data testing
> frameworks like in dbt
> * Improves developer productivity when writing data quality tests
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
