FrankChen021 opened a new issue #11298: URL: https://github.com/apache/druid/issues/11298
This problem was first reported by @clintropolis when verifying the 0.21.1-RC1. On Linux platform, tasks fail due to failure of creating task log dir. However, on macOS, tasks works well. ``` middlemanager | 2021-05-18T21:20:39,239 INFO [forking-task-runner-0] org.apache.druid.indexing.overlord.ForkingTaskRunner - Exception caught during execution middlemanager | org.apache.druid.java.util.common.IOE: Unable to create task log dir[/opt/data/indexing-logs] middlemanager | at org.apache.druid.indexing.common.tasklogs.FileTaskLogs.pushTaskLog(FileTaskLogs.java:59) ~[druid-indexing-service-0.21.1.jar:0.21.1] middlemanager | at org.apache.druid.indexing.overlord.ForkingTaskRunner$1.call(ForkingTaskRunner.java:386) [druid-indexing-service-0.21.1.jar:0.21.1] middlemanager | at org.apache.druid.indexing.overlord.ForkingTaskRunner$1.call(ForkingTaskRunner.java:137) [druid-indexing-service-0.21.1.jar:0.21.1] middlemanager | at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_275] middlemanager | at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)[?:1.8.0_275] middlemanager | at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)[?:1.8.0_275] middlemanager | at java.lang.Thread.run(Thread.java:748)[?:1.8.0_275] middlemanager | 2021-05-18T21:20:39,247 INFO [forking-task-runner-0] org.apache.druid.indexing.overlord.ForkingTaskRunner - Removing task directory: var/druid/task/index_parallel_wikipedia_gmkopeph_2021-05-18T21:20:10.899Z ``` In the example docker-compose.yml, `/opt/data` is mounted to a host directory `distribution\docker\storage`, if the host directory is belong to root, the permission of `/opt/data` inside container is also root. While druid processes inside contains are running as a normal user, they have no permission to create sub directories under `/opt/data`. There's no way for us to control the permission of the directory on host OS in docker-compose, so either we leave this problem for users to setup the permission of the host directories or uses a named volume to avoid the problem. - If we choose the first, we have to update the docker.md to tell users how to do that. - If we choose the latter, docker will create a dir automatically on the host OS and setup permission correctly, but that directory is located at `/var/lib/docker/volumes/` by default. Since this dir usually has smaller disk size, putting druid data and task logs under that directory might consume that directory quickly. I don't know how many people use this example docker-compose for long term use, but we might need to state this in the doc to let users know how to change the default directory to a directory they want to keep data. BTW, @clintropolis What's your initial thinking about using host directory in the docker-compose.yml since I see you added these volumes according to the code history ? cc @jihoonson @clintropolis @xvrl -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
