RiccardoSale opened a new issue, #20492:
URL: https://github.com/apache/druid/issues/20492
### Affected Version
33.0.0 (verified) and current master branch.
### Description
A task whose ID is between 218 and 255 bytes long can never start on a
MiddleManager or Indexer. The worker fails to persist the task with `File name
too long`, even though the ID itself is a valid file name.
`WorkerTaskManager` persists assigned and completed tasks with
`FileUtils.writeAtomically`, using the task ID as the file name.
`writeAtomically` first writes to a temporary file named `.<fileName>.<uuid>`,
which adds 38 characters. For IDs longer than 217 bytes, the temporary name
goes over the 255-byte file name limit of common filesystems such as ext4 and
XFS.
The worker logs:
```
Error while trying to persist assigned task[<taskId>]
java.nio.file.FileSystemException:
<baseTaskDir>/workerTaskManagerTmp/.<taskId>.<uuid>: File name too long
```
The Overlord then waits the full
`druid.indexer.runner.taskAssignmentTimeout` (default `PT5M`) before failing
the task with `Task assignment timed out on worker [...]`. A pending-task
runner thread is blocked for that whole time.
This is easy to hit with auto-kill, because coordinator-issued kill task IDs
include the datasource name, a random suffix, the interval and a timestamp:
```
coordinator-issued_kill_<dataSource>_<random>_<intervalStart>_<intervalEnd>_<createdTime>
```
These IDs add 108 characters to the datasource name, so any datasource name
longer than 109 characters produces kill tasks that can never run. The
Coordinator issues new ones every kill period, and each one ties up an
assignment thread for the full timeout. In our case the backlog grew until the
task queue reached `druid.indexer.queue.maxSize`, and new ingestion tasks were
rejected. Ingestion task IDs for the same datasource are shorter (no interval),
so they are not affected.
#### Configurations in use
```properties
druid.coordinator.kill.on=true
```
#### Steps to reproduce
Submit any task with an ID between 218 and 255 characters, for example:
```json
{"type": "noop", "id": "<220 x characters>"}
```
The task never starts, and the MiddleManager or Indexer log shows the error
above.
#### Proposed fix
In `FileUtils.writeAtomically`, truncate the target file name inside the
temporary file name so the temporary name never goes over 255 bytes. The UUID
already keeps the name unique. Task IDs and all other file names stay
unchanged. I have a patch with unit tests ready and will open a PR.
As an optional follow-up, kill task IDs could leave out the interval, which
would make them 50 characters shorter. The interval is already stored in the
task payload.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
