[
https://issues.apache.org/jira/browse/SOLR-11838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16321567#comment-16321567
]
Gus Heck commented on SOLR-11838:
---------------------------------
One interesting thing to think about is that ND4J and thus DL4J makes extensive
use of off heap memory, and require native drivers for GPU utilization. This
will probably mean that folks might want to add GPU nodes to their existing
cluster and define a collection that resides on only those nodes, then the ML
expressions can target collections that have GPU enabled nodes only... drawing
data from the existing regular nodes... (I *think* that should be possible
anyway :) ).
Other wide ranging thoughts I've had... [~cpoerschke], let me know if I'm way
too far afield, I can go get my own jira(s)... or this could become a parent of
several jiras...
* Pre-trained models loaded into blob store, with a name that can be used to
retrieve them and hydrate them with
ModelSerializer.restoreMultiLayerNetwork(InputStream) where it can serve as the
guts of a generic predict() expression that can act as a tuple transformer
(i.e. categorizing each tuple, etc) - including loading keras models/transfer
learning etc (something DL4J should be able to do for us).
* an expression that accepts the hyperparameters/dimensions of a layer that can
be combined with other such expressions to create an untrained network.
* expressions for partitioning the data into test/train (or K-Folds) and
iterating the training a model (I believe I've seen jiras go by that sound like
something of the sort exits in streaming expressions already, I know
[~joel.bernstein] talked about efficient sampling in his LSR talk)
* writing the trained model back to the blob store automatically on each epoch
on a rolling basis (keeping last N copies) to enable early stopping, or
selection of best model after K-folds.
* a solrj implementation of DataSetIterator that can use a query to specify a
set of data to be used for training which then is streamed down, and
potentially cached locally or re-streamed for training iteration.
In all cases the intent would be that Solr provides Data, environment and
infrastructure and all ML heavy lifting would be DL4J based, and I would hope
reusable in a LTR context at the very least by drawing trained models from
blobstore. I notice LTR has a a model store, is that backed by the blob store,
or are there 2 places to store content now? (sorry haven't had any real
opportunity to use LTR yet)
Anyway, that stuff's all been rattling around my head trying to get out, hope
it's not too much for this ticket.
> explore supporting Deeplearning4j NeuralNetwork models in contrib/ltr
> ---------------------------------------------------------------------
>
> Key: SOLR-11838
> URL: https://issues.apache.org/jira/browse/SOLR-11838
> Project: Solr
> Issue Type: New Feature
> Components: contrib - LTR
> Reporter: Christine Poerschke
> Attachments: SOLR-11838.patch
>
>
> [~yuyano] wrote in SOLR-11597:
> bq. ... If we think to apply this to more complex neural networks in the
> future, we will need to support layers ...
> [~malcorn_redhat] wrote in SOLR-11597:
> bq. ... In my opinion, if this is a route Solr eventually wants to go, I
> think a better strategy would be to just add a dependency on
> [Deeplearning4j|https://deeplearning4j.org/] ...
> Creating this ticket for the idea to be explored further (if anyone is
> interested in exploring it), complimentary to and independent of the
> SOLR-11597 RankNet related effort.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
