This is an automated email from the ASF dual-hosted git repository.
bowenliang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/kyuubi.git
The following commit(s) were added to refs/heads/master by this push:
new fb65a12936 [KYUUBI #6756] [REST] Check max file size of uploaded
resource and extra resources in batch creation
fb65a12936 is described below
commit fb65a129364ab9f8e8607c7ed2a7614b923fb812
Author: Bowen Liang <[email protected]>
AuthorDate: Mon Oct 21 16:04:33 2024 +0800
[KYUUBI #6756] [REST] Check max file size of uploaded resource and extra
resources in batch creation
# :mag: Description
## Issue References ๐
This pull request fixes #
## Describe Your Solution ๐ง
Check the uploaded resource files when creating batch via REST API
- add config `kyuubi.batch.resource.file.max.size` for resource file's max
size in bytes
- add config `kyuubi.batch.extra.resource.file.max.size` for each extra
resource file's max size in bytes
## Types of changes :bookmark:
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
## Test Plan ๐งช
#### Behavior Without This Pull Request :coffin:
#### Behavior With This Pull Request :tada:
#### Related Unit Tests
---
# Checklist ๐
- [ ] This patch was not authored or co-authored using [Generative
Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes #6756 from bowenliang123/resource-maxsize.
Closes #6756
5c409c425 [Bowen Liang] nit
4b16bcfc4 [Bowen Liang] nit
743920d25 [Bowen Liang] check resource file size max size
Authored-by: Bowen Liang <[email protected]>
Signed-off-by: Bowen Liang <[email protected]>
---
docs/configuration/settings.md | 14 ++++++++------
.../scala/org/apache/kyuubi/config/KyuubiConf.scala | 18 ++++++++++++++++++
.../apache/kyuubi/server/api/v1/BatchesResource.scala | 16 ++++++++++++++--
3 files changed, 40 insertions(+), 8 deletions(-)
diff --git a/docs/configuration/settings.md b/docs/configuration/settings.md
index 40caf9be99..d1d8cc08af 100644
--- a/docs/configuration/settings.md
+++ b/docs/configuration/settings.md
@@ -77,12 +77,14 @@ You can configure the Kyuubi properties in
`$KYUUBI_HOME/conf/kyuubi-defaults.co
### Batch
-| Key | Default |
Meaning
[...]
-|---------------------------------------------|---------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
[...]
-| kyuubi.batch.application.check.interval | PT5S | The interval to
check batch job application information.
[...]
-| kyuubi.batch.application.starvation.timeout | PT3M | Threshold above
which to warn batch application may be starved.
[...]
-| kyuubi.batch.conf.ignore.list || A comma-separated
list of ignored keys for batch conf. If the batch conf contains any of them,
the key and the corresponding value will be removed silently during batch job
submission. Note that this rule is for server-side protection defined via
administrators to prevent some essential configs from tampering. You can also
pre-define some config for batch job submission with the prefix:
kyuubi.batchConf.[batchType]. For example, y [...]
-| kyuubi.batch.session.idle.timeout | PT6H | Batch session idle
timeout, it will be closed when it's not accessed for this duration
[...]
+| Key | Default |
Meaning
[...]
+|---------------------------------------------|---------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
[...]
+| kyuubi.batch.application.check.interval | PT5S | The interval to
check batch job application information.
[...]
+| kyuubi.batch.application.starvation.timeout | PT3M | Threshold above
which to warn batch application may be starved.
[...]
+| kyuubi.batch.conf.ignore.list || A comma-separated
list of ignored keys for batch conf. If the batch conf contains any of them,
the key and the corresponding value will be removed silently during batch job
submission. Note that this rule is for server-side protection defined via
administrators to prevent some essential configs from tampering. You can also
pre-define some config for batch job submission with the prefix:
kyuubi.batchConf.[batchType]. For example, y [...]
+| kyuubi.batch.extra.resource.file.max.size | 0 | The maximum size in
bytes of each uploaded extra resource file when creating batch. 0 or negative
value means no limit.
[...]
+| kyuubi.batch.resource.file.max.size | 0 | The maximum size in
bytes of the uploaded resource file when creating batch. 0 or negative value
means no limit.
[...]
+| kyuubi.batch.session.idle.timeout | PT6H | Batch session idle
timeout, it will be closed when it's not accessed for this duration
[...]
### Credentials
diff --git
a/kyuubi-common/src/main/scala/org/apache/kyuubi/config/KyuubiConf.scala
b/kyuubi-common/src/main/scala/org/apache/kyuubi/config/KyuubiConf.scala
index b51b852b02..901703924e 100644
--- a/kyuubi-common/src/main/scala/org/apache/kyuubi/config/KyuubiConf.scala
+++ b/kyuubi-common/src/main/scala/org/apache/kyuubi/config/KyuubiConf.scala
@@ -1876,6 +1876,24 @@ object KyuubiConf {
.booleanConf
.createWithDefault(true)
+ val BATCH_RESOURCE_FILE_MAX_SIZE: ConfigEntry[Long] =
+ buildConf("kyuubi.batch.resource.file.max.size")
+ .doc("The maximum size in bytes of the uploaded resource file" +
+ " when creating batch. 0 or negative value means no limit.")
+ .version("1.10.0")
+ .serverOnly
+ .longConf
+ .createWithDefault(0)
+
+ val BATCH_EXTRA_RESOURCE_FILE_MAX_SIZE: ConfigEntry[Long] =
+ buildConf("kyuubi.batch.extra.resource.file.max.size")
+ .doc("The maximum size in bytes of each uploaded extra resource file" +
+ " when creating batch. 0 or negative value means no limit.")
+ .version("1.10.0")
+ .serverOnly
+ .longConf
+ .createWithDefault(0)
+
val BATCH_SUBMITTER_ENABLED: ConfigEntry[Boolean] =
buildConf("kyuubi.batch.submitter.enabled")
.internal
diff --git
a/kyuubi-server/src/main/scala/org/apache/kyuubi/server/api/v1/BatchesResource.scala
b/kyuubi-server/src/main/scala/org/apache/kyuubi/server/api/v1/BatchesResource.scala
index f778bcb0b7..e3e981abdc 100644
---
a/kyuubi-server/src/main/scala/org/apache/kyuubi/server/api/v1/BatchesResource.scala
+++
b/kyuubi-server/src/main/scala/org/apache/kyuubi/server/api/v1/BatchesResource.scala
@@ -18,7 +18,7 @@
package org.apache.kyuubi.server.api.v1
import java.io.InputStream
-import java.nio.file.{Path => JPath}
+import java.nio.file.{Files, Path => JPath}
import java.util
import java.util.{Collections, Locale, UUID}
import java.util.concurrent.ConcurrentHashMap
@@ -42,7 +42,7 @@ import org.apache.kyuubi.client.exception.KyuubiRestException
import org.apache.kyuubi.client.util.BatchUtils._
import org.apache.kyuubi.config.KyuubiConf._
import org.apache.kyuubi.config.KyuubiReservedKeys._
-import org.apache.kyuubi.engine.{ApplicationInfo, ApplicationManagerInfo,
ApplicationState, KillResponse, KyuubiApplicationManager}
+import org.apache.kyuubi.engine._
import org.apache.kyuubi.operation.{BatchJobSubmission, FetchOrientation,
OperationState}
import org.apache.kyuubi.server.api.ApiRequestContext
import org.apache.kyuubi.server.api.v1.BatchesResource._
@@ -65,6 +65,8 @@ private[v1] class BatchesResource extends ApiRequestContext
with Logging {
fe.getConf.get(BATCH_INTERNAL_REST_CLIENT_REQUEST_ATTEMPT_WAIT).toInt
private lazy val internalSecurityEnabled =
fe.getConf.get(ENGINE_SECURITY_ENABLED)
+ private lazy val resourceFileMaxSize =
fe.getConf.get(BATCH_RESOURCE_FILE_MAX_SIZE)
+ private lazy val extraResourceFileMaxSize =
fe.getConf.get(BATCH_EXTRA_RESOURCE_FILE_MAX_SIZE)
private def batchV2Enabled(reqConf: Map[String, String]): Boolean = {
fe.getConf.get(BATCH_SUBMITTER_ENABLED) &&
@@ -585,6 +587,10 @@ private[v1] class BatchesResource extends
ApiRequestContext with Logging {
uploadFileFolderPath: JPath): Unit = {
try {
val tempFile = Utils.writeToTempFile(inputStream, uploadFileFolderPath,
fileName)
+ if (resourceFileMaxSize > 0 && Files.size(tempFile.toPath) >
resourceFileMaxSize) {
+ throw new RuntimeException(
+ s"Resource file $fileName exceeds the maximum size limit
$resourceFileMaxSize bytes")
+ }
fe.sessionManager.tempFileService.addPathToExpiration(tempFile.toPath)
request.setResource(tempFile.getPath)
} catch {
@@ -621,6 +627,12 @@ private[v1] class BatchesResource extends
ApiRequestContext with Logging {
filePart.getValueAs(classOf[InputStream]),
uploadFileFolderPath,
fileName)
+ if (extraResourceFileMaxSize > 0
+ && Files.size(tempFile.toPath) > extraResourceFileMaxSize) {
+ throw new RuntimeException(
+ s"Extra resource file $fileName exceeds the maximum size
limit " +
+ s"$extraResourceFileMaxSize bytes")
+ }
fe.sessionManager.tempFileService.addPathToExpiration(tempFile.toPath)
tempFile.getPath
} catch {
