[
https://issues.apache.org/jira/browse/SPARK-56704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Boyang Jerry Peng updated SPARK-56704:
--------------------------------------
Description:
Scala's {{StreamTest}} provides a declarative, action-based testing paradigm
used by 38+ test suites. PySpark previously had no equivalent: streaming tests
were ad-hoc {{writeStream}} + {{processAllAvailable}} + {{time.sleep}} loops.
This framework brings the same paradigm to Python with an idiomatic API
({{{}unittest.TestCase{}}} base, dataclass-style action objects, full type
hints).
https://github.com/jerrypeng/spark/blob/fbcf4074e280e89834628e818b70e688bc64764e/python/pyspark/testing/streaming/DESIGN.md
> Native StreamTest framework for PySpark
> ---------------------------------------
>
> Key: SPARK-56704
> URL: https://issues.apache.org/jira/browse/SPARK-56704
> Project: Spark
> Issue Type: Epic
> Components: PySpark, Structured Streaming
> Affects Versions: 4.2.0
> Reporter: Boyang Jerry Peng
> Priority: Major
>
> Scala's {{StreamTest}} provides a declarative, action-based testing paradigm
> used by 38+ test suites. PySpark previously had no equivalent: streaming
> tests were ad-hoc {{writeStream}} + {{processAllAvailable}} + {{time.sleep}}
> loops. This framework brings the same paradigm to Python with an idiomatic
> API ({{{}unittest.TestCase{}}} base, dataclass-style action objects, full
> type hints).
>
> https://github.com/jerrypeng/spark/blob/fbcf4074e280e89834628e818b70e688bc64764e/python/pyspark/testing/streaming/DESIGN.md
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
