[
https://issues.apache.org/jira/browse/TEZ-2450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rajesh Balamohan updated TEZ-2450:
----------------------------------
Attachment: TEZ-2450.3.patch
>> HttpConnnection - connection, input, connectionSucceeded volatile. In
>> cleanup, connection and input should be set to null after they're cleaned up.
- Fixed
>> HttpConnectionParams - Ideally should be final fields with a constructor
>> used to setup the params in the helper
- Fixed
>> AsyncHttpConnection
>> final fields where possible, if they're not unset
- Fixed
>> initClient may not be safe. A partially constructed connection may be
>> returned if there's thread contention
- httpAsyncClient is already marked as volatile. Is there an issue?
>> //.setMaximumConnectionsTotal, //.setMaximumConnectionsPerHost commented out
- Both defaults to -1 by library. If set to numFetchers (or 1), it could throw
exceptions for lots of jobs and there is no config added for this. Instead,
setting this to getKeepAliveMaxConnections() which defaults to 20 (which is
applicable for HttpUrlConnection as well. i.e, "maximum number of connections
per destination to be kept alive at any given time"). In case of exception,
this parameter needs to be set to higher value.
>> Fetcher - catch(InterruptedException)
- Fixed. Added check for shutdown flag.
>> FetcherOrderedGrouped. asyncHttp can be final
- Fixed.
>> ShuffleUtils - some commented out code which can be removed
- removed commented out line.
>> TestHttpConnection
- Added timeouts
>> NOT_USED_URL - should this be localhost instead ? Not sure what the current
>> url implies
- Used this non-routable private ip
(http://en.wikipedia.org/wiki/Private_network) for simulating connection
timeout. If localhost is used, it would end up return "connection refused"
immediately. Used this with a random port so that it would not be able to
connect.
- Removed testHttpConnection as it was just trying to check if interrupting the
task would work or not. It is redundant test, so removed it altogether.
>> testAsyncHttpConnectionInterrupt - Replace the sleep(500) with a
>> future.get() and verify an InterruptException ?
- Calling get() after cancel() (to interrupt worker) would throw
CancellationException. Intention was to check if connect() call can be
interrupted.
>> TestPipelinedShuffle - setupTezCluster enables async by default - which
>> means all tests run with this ?
- Was added by mistake (for quick testing). Removed it.
- Removed last conf.set
> support async http clients in ordered & unordered inputs
> --------------------------------------------------------
>
> Key: TEZ-2450
> URL: https://issues.apache.org/jira/browse/TEZ-2450
> Project: Apache Tez
> Issue Type: Improvement
> Reporter: Rajesh Balamohan
> Assignee: Rajesh Balamohan
> Attachments: TEZ-2450.1.patch, TEZ-2450.2.WIP.patch,
> TEZ-2450.2.patch, TEZ-2450.3.patch, TEZ-2450.WIP.patch
>
>
> It will be helpful to switch between JDK & other async http impls. For LLAP
> scenarios, it would be useful to make http clients interruptible which is
> supported in async libraries.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
