> On Nov. 19, 2019, 6:53 p.m., Slim Bouguerra wrote: > > ql/src/java/org/apache/hadoop/hive/ql/hooks/HiveProtoLoggingHook.java > > Lines 279 (patched) > > <https://reviews.apache.org/r/71784/diff/1/?file=2173907#file2173907line280> > > > > i am still not sure how this is going to work? > > the original code was dropping events when the queue is full that is > > the case where you see the `RejectedExecutionException` > > Attila Magyar wrote: > RejectedExecutionException was never thrown with the original code > because of the unbounded queue. The queue continued to be larger. In the heap > dump there 17000 elements in the queue totally and it takes about 2.5g space. > > Slim Bouguerra wrote: > Very strange, > As per the java doc it says ` Executor uses finite bounds for both > maximum threads and work queue capacity, and is saturated` > > https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/ThreadPoolExecutor.html > > ``` > New tasks submitted in method execute(Runnable) will be rejected when the > Executor has been shut down, and also when the Executor uses finite bounds > for both maximum threads and work queue capacity, and is saturated. In either > case, the execute method invokes the > RejectedExecutionHandler.rejectedExecution(Runnable, ThreadPoolExecutor) > method of its RejectedExecutionHandler. Four predefined handler policies are > provided: > In the default ThreadPoolExecutor.AbortPolicy, the handler throws a > runtime RejectedExecutionException upon rejection. > In ThreadPoolExecutor.CallerRunsPolicy, the thread that invokes execute > itself runs the task. This provides a simple feedback control mechanism that > will slow down the rate that new tasks are submitted. > In ThreadPoolExecutor.DiscardPolicy, a task that cannot be executed is > simply dropped. > In ThreadPoolExecutor.DiscardOldestPolicy, if the executor is not shut > down, the task at the head of the work queue is dropped, and then execution > is retried (which can fail again, causing this to be repeated.) > It is possible to define and use other kinds of RejectedExecutionHandler > classes. Doing so requires some care especially when policies are designed to > work only under particular capacity or queuing policies. > Hook methods > ```
You are correct i did run some tests and i do not see any rejection. Thanks - Slim ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/71784/#review218679 ----------------------------------------------------------- On Nov. 19, 2019, 3:43 p.m., Attila Magyar wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/71784/ > ----------------------------------------------------------- > > (Updated Nov. 19, 2019, 3:43 p.m.) > > > Review request for hive, Laszlo Bodor, Harish Jaiprakash, Mustafa Iman, and > Panos Garefalakis. > > > Bugs: HIVE-22514 > https://issues.apache.org/jira/browse/HIVE-22514 > > > Repository: hive-git > > > Description > ------- > > HiveProtoLoggingHook uses a ScheduledThreadPoolExecutor to submit writer > tasks and to periodically handle rollover. The builtin > ScheduledThreadPoolExecutor uses a unbounded queue which cannot be replaced > from the outside. If log events are generated at a very fast rate this queue > can grow large. > > Since ScheduledThreadPoolExecutor does not support changing the default > unbounded queue to a bounded one, the queue capacity is checked manually by > the patch. > > > Diffs > ----- > > common/src/java/org/apache/hadoop/hive/conf/HiveConf.java a7687d59004 > ql/src/java/org/apache/hadoop/hive/ql/hooks/HiveProtoLoggingHook.java > 8eab54859bf > ql/src/test/org/apache/hadoop/hive/ql/hooks/TestHiveProtoLoggingHook.java > 450a0b544d6 > > > Diff: https://reviews.apache.org/r/71784/diff/1/ > > > Testing > ------- > > unittest > > > Thanks, > > Attila Magyar > >
