[ 
https://issues.apache.org/jira/browse/SPARK-62035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ASF GitHub Bot updated SPARK-62035:
-----------------------------------
    Labels: pull-request-available  (was: )

> Add spark.sql.execution.maxScanBytes to cancel queries that read too much data
> ------------------------------------------------------------------------------
>
>                 Key: SPARK-62035
>                 URL: https://issues.apache.org/jira/browse/SPARK-62035
>             Project: Spark
>          Issue Type: New Feature
>          Components: SQL
>    Affects Versions: 5.0.0
>            Reporter: Gengliang Wang
>            Priority: Major
>              Labels: pull-request-available
>
> A query with a missing filter can read terabytes of data before anyone 
> notices. A timeout does not help: it fires only after the compute has been 
> spent, and the query returns nothing.
> This adds an optional setting, off by default:
> spark.sql.execution.maxScanBytes: the most bytes a query may read from its 
> data sources.
> A query that reads more is cancelled and fails with 
> SCAN_BYTES_LIMIT_EXCEEDED. This is similar to maximum_bytes_billed in 
> BigQuery and query_max_scan_physical_bytes in Trino.
> The limit is a best-effort guard rather than a hard cap:
> * Spark learns how much the tasks of a query have read from the executor 
> heartbeats and when a task ends, so a query can read more than the limit 
> before it is cancelled.
> * Only data sources that report the bytes they read are limited. File-based 
> sources report the bytes they read from storage, while sources such as JDBC 
> report none.
> * Reading cached data counts the size of the cached blocks.
> * Streaming queries are not limited.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to