[GitHub] [hadoop] ZanderXu commented on pull request #3806: HDFS-16386.Reduce DataNode load when FsDatasetAsyncDiskService is working.
ZanderXu commented on PR #3806: URL: https://github.com/apache/hadoop/pull/3806#issuecomment-1136727763 Thanks, and i will create a new PR to do it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] ZanderXu commented on pull request #3806: HDFS-16386.Reduce DataNode load when FsDatasetAsyncDiskService is working.
ZanderXu commented on PR #3806: URL: https://github.com/apache/hadoop/pull/3806#issuecomment-1136695695 Thanks @jianghuazhu for you comment. - I have a question, if the queue is unbounded, will the number of active thread in the ThreadPool be greater than the number of core thread? - I think that we need to support the ability to dynamically adjust the number of core threads, so that we can adjust it in time for different load to archive the best result. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] ZanderXu commented on pull request #3806: HDFS-16386.Reduce DataNode load when FsDatasetAsyncDiskService is working.
ZanderXu commented on PR #3806: URL: https://github.com/apache/hadoop/pull/3806#issuecomment-1135792522 `ThreadPoolExecutor executor = new ThreadPoolExecutor( CORE_THREADS_PER_VOLUME, maxNumThreadsPerVolume, THREADS_KEEP_ALIVE_SECONDS, TimeUnit.SECONDS, new LinkedBlockingQueue(), threadFactory);` The ThreadPoolExecutor used the unbounded LinkedBlockingQueue, so the actual thread number should be less than or equal to the corePoolSize. When NN needs one DN to delete a large number of blocks, this DN will create a large number of ReplicaFileDeleteTask, and stored all ReplicaFileDeleteTasks in the LinkedBlockingQueue of the ThreadPoolExecutor, resulting in increased memory or even OOM. Feel free to correct me if there are mistakes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] ZanderXu commented on pull request #3806: HDFS-16386.Reduce DataNode load when FsDatasetAsyncDiskService is working.
ZanderXu commented on PR #3806: URL: https://github.com/apache/hadoop/pull/3806#issuecomment-1135779919 @jianghuazhu I'm so sorry to discuss this issue again. Setting smaller MAX THREAD can reduce memory usage? [HDFS-16386](https://issues.apache.org/jira/browse/HDFS-16386) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
