Hyukjin Kwon created SPARK-60732:
------------------------------------
Summary: Support admission control for streaming reads of native
data sources
Key: SPARK-60732
URL: https://issues.apache.org/jira/browse/SPARK-60732
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 5.0.0
Reporter: Hyukjin Kwon
A follow-up to SPARK-60101 (https://github.com/apache/spark/pull/59424) and
SPARK-60723. Native data sources cannot limit the data of a micro-batch of a
streaming read, such as a maximum number of rows for each micro-batch, which
the guide lists as a limitation of version 1 of spark_datasource.h. Add
admission control within version 1: a capability SPARK_DS_ADMISSION_CONTROL, a
function default_read_limit appended to SparkDsLibrary, and the read limit and,
for Trigger.AvailableNow, the end of the run appended to
SparkDsLatestOffsetInfo.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
