[
https://issues.apache.org/jira/browse/LUCENE-7745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15932714#comment-15932714
]
Uwe Schindler edited comment on LUCENE-7745 at 3/20/17 2:24 PM:
----------------------------------------------------------------
Hi,
in General, including CUDA into Lucene may be a good idea, but I see no real
possibility to do this inside Lucene Core or any other module. My idea would be
to add some abstraction to the relevant parts of Lucene and make it easier to
"plug in" different implementations. Then this code could also be hosted
outside Lucene (if Licenses is a problem) anywhere on Github.
We still should have the following in our head: Mike's example looks "simple"
as a quick test if we see gains, but making the whole thing ready for commit or
bundling in any project in/outside Lucene is a whole different story. Currently
BooleanScorer calls a lot of classes, e.g. the BM25 similarity or TF-IDF to do
the calculation that could possibly be parallelized. But for moving all this to
CUDA, you have to add "plugin points" all there and change APIs completely. It
is also hard to test, because none of our Jenkins servers has a GPU! Also for
0/8/15 users of Lucene, this could be a huge problem, if we add native stuff
into Lucene that they may never use. Because of that it MUST BE SEPARATED from
Lucene core. Completely...
IMHO, I'd create a full new search engine like CLucene in C code if I would
solely focus on GPU parallelization. The current iterator based approaches are
not easy to transform or plug into CUDA...
For the GSoc project, we should make sure to the GSoc student that this is just
a project to "explore" GPU acceleration: if it brings any performance - I doubt
that, because the call overhead between Java and CUDA is way too high - in
contrast to Postgres where all in plain C/C++. The results would then be used
to plan and investigate ways how to include that into Lucene as "plugin points"
(e.g., as SPI modules).
was (Author: thetaphi):
Hi,
in General, including CUDA into Lucene may be a good idea, but I see no real
possibility to do this inside Lucene Core or any other module. My idea would be
to add some abstraction to the relevant parts of Lucene and make it easier to
"plug in" different implementations. Then this code could also be hosted
outside Lucene (if Licenses is a problem) anywhere on Github.
We still should have the following in our head: Mike's example looks "simple"
as a quick test if we see gains, but making the whole thing ready for commit or
bundling in any project in/outside Lucene is a whole different story. Currently
BooleanScorer calls a lot of classes, e.g. the BM25 similarity or TF-IDF to do
the calculation that could possibly be parallelized. But for moving all this to
CUDA, you have to add "plugin points" all there and change APIs completely. It
is also hard to test, because none of our Jenkins servers has a GPU! Also for
uses of Lucene, this could be a huge problem, if we add native stuff into
Lucene that they may never use. Because of that it MUST BE SEPARATED from
Lucene core. Completely...
IMHO, I'd create a full new search engine like CLucene in C code if I would
solely focus on GPU parallelization. The current iterator based approaches are
not easy to transform or plug into CUDA...
For the GSoc project, we should make sure to the GSoc student that this is just
a project to "explore" GPU acceleration: if it brings any performance - I doubt
that, because the call overhead between Java and CUDA is way too high - in
contrast to Postgres where all in plain C/C++. The results would then be used
to plan and investigate ways how to include that into Lucene as "plugin points"
(e.g., as SPI modules).
> Explore GPU acceleration
> ------------------------
>
> Key: LUCENE-7745
> URL: https://issues.apache.org/jira/browse/LUCENE-7745
> Project: Lucene - Core
> Issue Type: Improvement
> Reporter: Ishan Chattopadhyaya
> Labels: gsoc2017, mentor
>
> There are parts of Lucene that can potentially be speeded up if computations
> were to be offloaded from CPU to the GPU(s). With commodity GPUs having as
> high as 12GB of high bandwidth RAM, we might be able to leverage GPUs to
> speed parts of Lucene (indexing, search).
> First that comes to mind is spatial filtering, which is traditionally known
> to be a good candidate for GPU based speedup (esp. when complex polygons are
> involved). In the past, Mike McCandless has mentioned that "both initial
> indexing and merging are CPU/IO intensive, but they are very amenable to
> soaking up the hardware's concurrency."
> I'm opening this issue as an exploratory task, suitable for a GSoC project. I
> volunteer to mentor any GSoC student willing to work on this this summer.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
