[
https://issues.apache.org/jira/browse/ARROW-2458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16444621#comment-16444621
]
ASF GitHub Bot commented on ARROW-2458:
---------------------------------------
robertnishihara commented on issue #1893: ARROW-2458: [Plasma] Use one thread
pool per PlasmaClient
URL: https://github.com/apache/arrow/pull/1893#issuecomment-382843906
Thanks! The point of the hash is to have a simple way to check if two
objects are identical. In particular, given that objects can be evicted from
the object store, if an object is recreated in the object store after eviction,
you can compare hashes to see if the two objects differ.
Note that the probably more important place where we use multiple threads is
https://github.com/apache/arrow/blob/b8754eba4683e7300a751f60a2fc1eef152cea1d/cpp/src/arrow/util/memory.h#L34
This already seems to be thread safe (since the thread creation is
encapsulated in the `parallel_memcopy` function). Though it has the same
potential issue that you're raising here.
If we use a thread pool implementation then we should run some benchmarks
before merging, because the performance requirements are pretty demanding here.
We want low latency for small objects but also to be able to write to the
object store at near the memory-bandwidth (ideally at least 20GB/s, though
currently it seems a bit slower).
I'm still in favor of merging this PR since it fixes a segfault and has a
very a simple design. If the thread pool approach ends up working out that'd be
great since it would be nice to support applications with hundreds of plasma
clients (which is something we've been trying recently).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
> [Plasma] PlasmaClient uses global variable
> ------------------------------------------
>
> Key: ARROW-2458
> URL: https://issues.apache.org/jira/browse/ARROW-2458
> Project: Apache Arrow
> Issue Type: Improvement
> Components: Plasma (C++)
> Affects Versions: 0.9.0
> Reporter: Philipp Moritz
> Assignee: Philipp Moritz
> Priority: Major
> Labels: pull-request-available
>
> The threadpool threadpool_ that PlasmaClient is using is global at the
> moment. This prevents us from using multiple PlasmaClients in the same
> process (one per thread).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
