[
https://issues.apache.org/jira/browse/LUCENE-1336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12616323#action_12616323
]
Jason Rutherglen commented on LUCENE-1336:
------------------------------------------
The dynamic classloading mechanism as it is currently implemented is flawed
because it only takes into account the serialized classes and not the classes
loaded by the serialized classes.
To address this, the next release will do the class serialization differently.
A byte code class library such as ASM will be used to lookup all dependencies
for a "top level object" on the client. A "top level object" in this context
is the parameter object classes of a remote method such as
Searchable.search(Query query) where Query would be the "top level object".
Each object's class will have a jar file written with all class dependencies.
This jar file will be automatically generated. The server on deserialization
will load the jar file the first time it sees the "top level object" class.
This process described should be fine for Lucene given the number of classes is
usually quite small. For Ocean I plan on using the process to allow search
related tasks to be submitted that interact directly with the core server APIs.
This will allow things like highlighters, facet code, and other search side
tasks to be submitted to the server without the server already having the
libraries required to execute the task. This also solves the problem of
application changes to search code that otherwise would require provisioning of
new configuration files, or application logic to the server. A good example of
this is interaction with IndexReader methods not directly available via
Searchable. One could write a task that interacts with IndexReader and returns
the result to the client without manually installing a new library on the
server to handle the desired logic. This type of system is probably obviously
useful for Lucene on a large number of servers where advanced dynamic query and
other advanced logic is desired. Use cases that come to mind are SpanQueries,
Payload queries. Payloads in particular because they require code level
updates to Analyzers and Queries, and may need to be changed often.
> Distributed Lucene using Hadoop RPC based RMI with dynamic classloading
> -----------------------------------------------------------------------
>
> Key: LUCENE-1336
> URL: https://issues.apache.org/jira/browse/LUCENE-1336
> Project: Lucene - Java
> Issue Type: New Feature
> Components: contrib/*
> Affects Versions: 2.3.1
> Reporter: Jason Rutherglen
> Priority: Minor
> Attachments: lucene-1336.patch, lucene-1336.patch, lucene-1336.patch
>
>
> Hadoop RPC based RMI system for use with Lucene Searchable. Keeps the
> application logic on the client side with removing the need to deploy
> application logic to the Lucene servers. Removes the need to provision new
> code to potentially hundreds of servers for every application logic change.
> The use case is any deployment requiring Lucene on many servers. This system
> provides the added advantage of allowing custom Query and Filter classes (or
> other classes) to be defined on for example a development machine and
> executed on the server without deploying the custom classes to the servers
> first. This can save a lot of time and effort in provisioning, restarting
> processes. In the future this patch will include an IndexWriterService
> interface which will enable document indexing. This will allow subclasses of
> Analyzer to be dynamically loaded onto a server as documents are added by the
> client.
> Hadoop RPC is more scalable than Sun's RMI implementation because it uses non
> blocking sockets. Hadoop RPC is also far easier to understand and customize
> if needed as it is embodied in 2 main class files
> org.apache.hadoop.ipc.Client and org.apache.hadoop.ipc.Server.
> Features include automatic dynamic classloading. The dynamic classloading
> enables newly compiled client classes inheriting core objects such as Query
> or Filter to be used to query the server without first deploying the code to
> the server.
> Using RMI dynamic classloading is not used in practice because it is hard to
> setup, requiring placing the new code in jar files on a web server on the
> client. Then requires custom system properties to be setup as well as Java
> security manager configuration.
> The dynamic classloading in Hadoop RMI for Lucene uses RMI to load the
> classes. Custom serialization and deserialization manages the classes and
> the class versions on the server and client side. New class files are
> automatically detected and loaded using ClassLoader.getResourceAsStream and
> so this system does not require creating a JAR file. The use of the same
> networking system used for the remote method invocation is used for the
> loading classes over the network. This removes the necessity of a separate
> web server dedicated to the task and makes deployment a few lines of code.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
