[
https://issues.apache.org/jira/browse/TEZ-2450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14547618#comment-14547618
]
Siddharth Seth commented on TEZ-2450:
-------------------------------------
Minor stuff
- Default value for TEZ_RUNTIME_SHUFFLE_USE_ASYNC_HTTP in TezConfiguration
- ShuffleUtils.getHttpConnection - Avoid reading from config on each request.
- TestPipelinedShuffle -
conf.setBoolean(TezRuntimeConfiguration.TEZ_RUNTIME_SHUFFLE_USE_ASYNC_HTTP,
true); and later set to true again in the test. Think all tests may end up
running with the AsyncClient
More important
- CompletionHandler - this throws exceptions from the various callbacks. In
case of errors (Connection Validation), this probably needs to inform the
ShuffleManager / ShuffleScheduler - like the Fetcher does. Not sure how the
library handles exceptions thrown by the callback.
- response = httpAsyncClient.executeRequest(request, new
CompletionHandler()).get(); - Will this only return after all data has been
fetched, or will it return right after the Connection establishment ? Will this
need to go to disk / memory based on the response size - using the
onPartReceived callback ?
Given the async nature and callbacks when data is received - I'm guessing the
requirement for N fetcher threads probably goes away in the future, to be
replaced by thread management done in the library itself ?
> support async http clients in ordered & unordered inputs
> --------------------------------------------------------
>
> Key: TEZ-2450
> URL: https://issues.apache.org/jira/browse/TEZ-2450
> Project: Apache Tez
> Issue Type: Improvement
> Reporter: Rajesh Balamohan
> Assignee: Rajesh Balamohan
> Attachments: TEZ-2450.1.patch, TEZ-2450.WIP.patch
>
>
> It will be helpful to switch between JDK & other async http impls. For LLAP
> scenarios, it would be useful to make http clients interruptible which is
> supported in async libraries.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
