kfaraz commented on code in PR #16512:
URL: https://github.com/apache/druid/pull/16512#discussion_r1622559161
##########
indexing-service/src/main/java/org/apache/druid/indexing/overlord/TaskQueue.java:
##########
@@ -1018,4 +1023,34 @@ List<Task> getTasks()
giant.unlock();
}
}
+
+ private void validateTaskPayload(Task task)
+ {
+ try {
+ String payload = passwordRedactingMapper.writeValueAsString(task);
+ if (config.getMaxTaskPayloadSize() != null &&
config.getMaxTaskPayloadSize().getBytesInInt() < payload.length()) {
+ throw InvalidInput.exception(
+ "Task[%s] has payload of size[%d] but max allowed size is
[%d]. " +
+ "Reduce the size of the task payload or increase
'druid.indexer.queue.maxTaskPayloadSize'.",
+ task.getId(), payload.length(), config.getMaxTaskPayloadSize()
+ );
+ } else if (payload.length() > TASK_SIZE_WARNING_THRESHOLD) {
+ log.warn(
+ "Task[%s] of datasource[%s] has payload size[%d] larger than the
recommended maximum[%d]. " +
+ "Large task payloads may cause stability issues in the
Overlord and may fail while persisting to the metadata store." +
Review Comment:
Typo: missing space
```suggestion
"Large task payloads may cause stability issues in the
Overlord and may fail while persisting to the metadata store. " +
```
##########
docs/configuration/index.md:
##########
@@ -1125,6 +1125,7 @@ These Overlord static configurations can be defined in
the `overlord/runtime.pro
|`druid.indexer.queue.startDelay`|Sleep this long before starting Overlord
queue management. This can be useful to give a cluster time to re-orient itself
(for example, after a widespread network issue).|`PT1M`|
|`druid.indexer.queue.restartDelay`|Sleep this long when Overlord queue
management throws an exception before trying again.|`PT30S`|
|`druid.indexer.queue.storageSyncRate`|Sync Overlord state this often with an
underlying task persistence mechanism.|`PT1M`|
+|`druid.indexer.queue.maxTaskPayloadSize`|Maximum size in bytes of a single
task payload.|`64 MiB`|
Review Comment:
```suggestion
|`druid.indexer.queue.maxTaskPayloadSize`|Maximum size in bytes of a single
task payload. The Overlord rejects tasks that have a payload greater than this
size.|none (accept all task payload sizes)|
```
##########
indexing-service/src/main/java/org/apache/druid/indexing/overlord/TaskQueue.java:
##########
@@ -1018,4 +1023,34 @@ List<Task> getTasks()
giant.unlock();
}
}
+
+ private void validateTaskPayload(Task task)
+ {
+ try {
+ String payload = passwordRedactingMapper.writeValueAsString(task);
+ if (config.getMaxTaskPayloadSize() != null &&
config.getMaxTaskPayloadSize().getBytesInInt() < payload.length()) {
+ throw InvalidInput.exception(
+ "Task[%s] has payload of size[%d] but max allowed size is
[%d]. " +
+ "Reduce the size of the task payload or increase
'druid.indexer.queue.maxTaskPayloadSize'.",
+ task.getId(), payload.length(), config.getMaxTaskPayloadSize()
+ );
+ } else if (payload.length() > TASK_SIZE_WARNING_THRESHOLD) {
+ log.warn(
+ "Task[%s] of datasource[%s] has payload size[%d] larger than the
recommended maximum[%d]. " +
+ "Large task payloads may cause stability issues in the
Overlord and may fail while persisting to the metadata store." +
+ "These large payloads will be rejected by the overlord in the
future.",
Review Comment:
```suggestion
"Such tasks may be rejected by the Overlord in future Druid
versions.",
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
