[
https://issues.apache.org/jira/browse/HDFS-10796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
James Clampffer updated HDFS-10796:
-----------------------------------
Attachment: HDFS-10796.HDFS-8707.000.patch
Patch up:
-IoService now inherits from std::enabled_shared_from_this and has a
"MakeShared" factory. The FileSystem and test shims now use this.
-IoService owns its worker threads, will default to hardware concurrency.
FileSystem will use Options::io_threads_which by default is -1 to indicate
hardware concurrency.
-IoService has a PostTask method so async operations that aren't IO don't need
the underlying asio::io_service
-lots more logging
As a result of defaulting to hardware concurrency all tests can now run with
more than one worker thread (used to be exclusively single threaded). It looks
like this is exposing some bugs that have been hiding elsewhere so for now
multithreaded IoServices are disabled by default. On some platforms tests work
well with multithreaded mode and on others, including the docker image, some
fail. These don't look like new bugs so if possible I'd like to file a second
jira to sort them out, the tools/examples have had issues with multiple
io_service threads for a while now.
> libhdfs++: rationalize ioservice interactions
> ---------------------------------------------
>
> Key: HDFS-10796
> URL: https://issues.apache.org/jira/browse/HDFS-10796
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs-client
> Reporter: Bob Hansen
> Assignee: James Clampffer
> Attachments: HDFS-10796.HDFS-8707.000.patch
>
>
> Firstly, we should be pulling the number of threads from options.io_threads
> (which should default to std::thread::hardware_concurrency()). The library
> should pass all tests always with io_threads set to 1 or to <a very high
> number>
> Secondly, we should have _a_ constructor where the consumer doesn't need to
> manage the IOService explicitly, and the FileSystemImpl should create its own
> internally.
> Since the FileSystem is defined as being for a particular user/identity,
> there is a valid use case for the consumer to be constructing many FileSystem
> instances to represent many authenticated users in the same process, but want
> to share resources (notably have a single io_service shared amongst them
> all). In this case, the consumer would want to own the IOService and pass
> the same instance to multiple FileSystem instances.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
