[
https://issues.apache.org/jira/browse/HADOOP-11684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15473544#comment-15473544
]
Thomas Demoor commented on HADOOP-11684:
----------------------------------------
[[email protected]] you should not make the queue (tasks) long. Indeed the
consumed memory is (threads + tasks + 1) * partitionSize) (the 1 is due to the
buffer itself )
The point of this ticket was to introduce the BlockingThreadPoolExecutorService
that, in contrast to the default Executorservice, does not error out when the
queue is full but blocks ALL clients from making progress (= using additional
memory), not even in the calling thread (thus different from CallerRunsPolicy).
Threads = number of parallel busy (part)uploads
Tasks = waiting uploads (already in memory)
The intended usage is setting the number of threads and tasks to small values
(threads=3, (waiting)tasks= 1). Additional work coming in from the application
will be blocked due to the special executorservice.
> S3a to use thread pool that blocks clients
> ------------------------------------------
>
> Key: HADOOP-11684
> URL: https://issues.apache.org/jira/browse/HADOOP-11684
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 2.7.0
> Reporter: Thomas Demoor
> Assignee: Thomas Demoor
> Fix For: 3.0.0-alpha1
>
> Attachments: HADOOP-11684-001.patch, HADOOP-11684-002.patch,
> HADOOP-11684-003.patch, HADOOP-11684-004.patch, HADOOP-11684-005.patch,
> HADOOP-11684-006.patch
>
>
> Currently, if fs.s3a.max.total.tasks are queued and another (part)upload
> wants to start, a RejectedExecutionException is thrown.
> We should use a threadpool that blocks clients, nicely throtthling them,
> rather than throwing an exception. F.i. something similar to
> https://github.com/apache/incubator-s4/blob/master/subprojects/s4-comm/src/main/java/org/apache/s4/comm/staging/BlockingThreadPoolExecutorService.java
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
