[GitHub] [lucene-solr] dsmiley commented on a change in pull request #2066: SOLR-14975: Optimize CoreContainer.getAllCoreNames and getLoadedCoreNames.
dsmiley commented on a change in pull request #2066:
URL: https://github.com/apache/lucene-solr/pull/2066#discussion_r519263690
##
File path: solr/core/src/java/org/apache/solr/core/SolrCores.java
##
@@ -198,35 +188,91 @@ protected SolrCore putCore(CoreDescriptor cd, SolrCore
core) {
*
* Put another way, this will not return any names of cores that are lazily
loaded but have not been called for yet
* or are transient and either not loaded or have been swapped out.
- *
- * @return List of currently loaded cores.
+ *
+ * @return A unsorted collection.
*/
- Set getLoadedCoreNames() {
-Set set;
-
+ Collection getLoadedCoreNames() {
synchronized (modifyLock) {
- set = new TreeSet<>(cores.keySet());
- if (getTransientCacheHandler() != null) {
-set.addAll(getTransientCacheHandler().getLoadedCoreNames());
- }
+ TransientSolrCoreCache transientCoreCache = getTransientCacheHandler();
Review comment:
@ErickErickson IMO a bad config should stop Solr in its tracks. Why
allow this? I'd rather disallow it and thus we'd *always* have either the
default impl or some custom impl, but _something_.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] madrob commented on a change in pull request #2065: SOLR-14977 : ContainerPlugins should be configurable
madrob commented on a change in pull request #2065:
URL: https://github.com/apache/lucene-solr/pull/2065#discussion_r519255446
##
File path: solr/core/src/java/org/apache/solr/api/ConfigurablePlugin.java
##
@@ -0,0 +1,31 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.solr.api;
+
+/**Implement this interface if your plugin needs to accept some configuration
+ *
+ * @param the configuration Object type
+ */
+public interface ConfigurablePlugin {
Review comment:
Does T need to extend `PluginMeta` as you do in the unit test or no?
##
File path: solr/core/src/java/org/apache/solr/api/ContainerPluginsRegistry.java
##
@@ -372,9 +396,27 @@ public void init() throws Exception {
holders.add(new ApiHolder((AnnotatedApi) api));
}
}
+
+@SuppressWarnings("rawtypes")
Review comment:
At the very least, don't suppress raw types here.
##
File path: solr/core/src/java/org/apache/solr/api/ContainerPluginsRegistry.java
##
@@ -372,9 +396,27 @@ public void init() throws Exception {
holders.add(new ApiHolder((AnnotatedApi) api));
}
}
+
+@SuppressWarnings("rawtypes")
+private Class getConfigObj(Object o) {
+ Type[] interfaces = o.getClass().getGenericInterfaces();
Review comment:
This is super fragile and fails if we have `Widget implements
ConfigurablePlugin` and then `NewerWidget extends Widget`
##
File path: solr/core/src/java/org/apache/solr/api/ContainerPluginsRegistry.java
##
@@ -69,7 +72,8 @@
public class ContainerPluginsRegistry implements ClusterPropertiesListener,
MapWriter, Closeable {
private static final Logger log =
LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
- private final ObjectMapper mapper =
SolrJacksonAnnotationInspector.createObjectMapper();
+ private static final ObjectMapper mapper =
SolrJacksonAnnotationInspector.createObjectMapper()
+ .configure(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES, false);
Review comment:
Is this a security risk? Why do we need this?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14985) Slow indexing and search performance when using HttpClusterStateProvider
[ https://issues.apache.org/jira/browse/SOLR-14985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17227873#comment-17227873 ] Noble Paul commented on SOLR-14985: --- We need to cache the data longer. There is no reason why we should not cache the data longer. (both alias and state.json) > Slow indexing and search performance when using HttpClusterStateProvider > > > Key: SOLR-14985 > URL: https://issues.apache.org/jira/browse/SOLR-14985 > Project: Solr > Issue Type: Bug > Security Level: Public(Default Security Level. Issues are Public) > Components: SolrJ >Reporter: Shalin Shekhar Mangar >Priority: Major > > HttpClusterStateProvider fetches and caches Aliases and Live Nodes for 5 > seconds. > The BaseSolrCloudClient caches DocCollection for 60 seconds but only if the > DocCollection is not lazy and all collections returned by > HttpClusterStateProvider are not lazy which means they are never cached. > The BaseSolrCloudClient has a method for resolving aliases which fetches > DocCollection for each input collection. This is an HTTP call with no caching > when using HttpClusterStateProvider. This resolveAliases method is called > twice for each update. > So overall, at least 3 HTTP calls are made to fetch cluster state for each > update request when using HttpClusterStateProvider. There may be more if > aliases are involved or if more than one collection is specified in the > request. Similar problems exist on the query path as well. > Due to these reasons, using HttpClusterStateProvider causes horrible > latencies and throughput for update and search requests. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] bruno-roustant commented on a change in pull request #2066: SOLR-14975: Optimize CoreContainer.getAllCoreNames and getLoadedCoreNames.
bruno-roustant commented on a change in pull request #2066:
URL: https://github.com/apache/lucene-solr/pull/2066#discussion_r519227113
##
File path: solr/core/src/java/org/apache/solr/cloud/ReplicateFromLeader.java
##
@@ -69,7 +71,10 @@ public void startReplication(boolean switchTransactionLog) {
if (cc.isShutDown()) {
return;
} else {
- throw new SolrException(SolrException.ErrorCode.SERVER_ERROR,
"SolrCore not found:" + coreName + " in " + cc.getLoadedCoreNames());
+ Collection loadedCoreNames = cc.getLoadedCoreNames();
+ throw new SolrException(SolrException.ErrorCode.SERVER_ERROR,
"SolrCore not found:" + coreName + " in "
Review comment:
Ok. I'll share in a new method CloudUtil.getLoadedCoreNamesAsString().
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] bruno-roustant commented on a change in pull request #2066: SOLR-14975: Optimize CoreContainer.getAllCoreNames and getLoadedCoreNames.
bruno-roustant commented on a change in pull request #2066:
URL: https://github.com/apache/lucene-solr/pull/2066#discussion_r519224133
##
File path: solr/core/src/java/org/apache/solr/core/CoreContainer.java
##
@@ -1568,33 +1568,49 @@ private void resetIndexDirectory(CoreDescriptor dcore,
ConfigSet coreConfig) {
}
/**
- * @return a Collection of registered SolrCores
+ * Gets the permanent (non-transient) cores that are currently loaded.
+ *
+ * @return An unsorted collection.
*/
public Collection getCores() {
Review comment:
Very good point!
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] bruno-roustant commented on a change in pull request #2066: SOLR-14975: Optimize CoreContainer.getAllCoreNames and getLoadedCoreNames.
bruno-roustant commented on a change in pull request #2066:
URL: https://github.com/apache/lucene-solr/pull/2066#discussion_r519224315
##
File path: solr/core/src/java/org/apache/solr/core/SolrCores.java
##
@@ -198,35 +188,91 @@ protected SolrCore putCore(CoreDescriptor cd, SolrCore
core) {
*
* Put another way, this will not return any names of cores that are lazily
loaded but have not been called for yet
* or are transient and either not loaded or have been swapped out.
- *
- * @return List of currently loaded cores.
+ *
+ * @return A unsorted collection.
*/
- Set getLoadedCoreNames() {
-Set set;
-
+ Collection getLoadedCoreNames() {
synchronized (modifyLock) {
- set = new TreeSet<>(cores.keySet());
- if (getTransientCacheHandler() != null) {
-set.addAll(getTransientCacheHandler().getLoadedCoreNames());
- }
+ TransientSolrCoreCache transientCoreCache = getTransientCacheHandler();
+ Set transientCoreNames = transientCoreCache == null ?
Collections.emptySet() : transientCoreCache.getLoadedCoreNames();
+ return distinctSetsUnion(cores.keySet(), transientCoreNames);
}
-return set;
}
+
/**
- * Gets a list of all cores, loaded and unloaded
+ * Gets a collection of all cores names, loaded and unloaded.
+ * For efficiency, prefer to check {@link #getCoreDescriptor(String)} !=
null instead of {@link #getAllCoreNames()}.contains(String)
*
- * @return all cores names, whether loaded or unloaded, transient or
permanent.
+ * @return A unsorted collection.
*/
public Collection getAllCoreNames() {
-Set set;
synchronized (modifyLock) {
- set = new TreeSet<>(cores.keySet());
- if (getTransientCacheHandler() != null) {
-set.addAll(getTransientCacheHandler().getAllCoreNames());
- }
- set.addAll(residentDesciptors.keySet());
+ TransientSolrCoreCache transientCoreCache = getTransientCacheHandler();
+ Set transientCoreNames = transientCoreCache == null ?
Collections.emptySet() : transientCoreCache.getAllCoreNames();
+ return distinctSetsUnion(residentDescriptors.keySet(),
transientCoreNames);
+}
+ }
+
+ /**
+ * Makes the union of two distinct sets.
+ */
+ private static Collection distinctSetsUnion(Set set1, Set set2)
{
+assert areSetsDistinct(set1, set2);
Review comment:
Yes it must be a snapshot (like in getCores()). I'll add the comment.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] bruno-roustant commented on a change in pull request #2066: SOLR-14975: Optimize CoreContainer.getAllCoreNames and getLoadedCoreNames.
bruno-roustant commented on a change in pull request #2066:
URL: https://github.com/apache/lucene-solr/pull/2066#discussion_r519225448
##
File path: solr/core/src/java/org/apache/solr/core/SolrCores.java
##
@@ -198,35 +188,91 @@ protected SolrCore putCore(CoreDescriptor cd, SolrCore
core) {
*
* Put another way, this will not return any names of cores that are lazily
loaded but have not been called for yet
* or are transient and either not loaded or have been swapped out.
- *
- * @return List of currently loaded cores.
+ *
+ * @return A unsorted collection.
*/
- Set getLoadedCoreNames() {
-Set set;
-
+ Collection getLoadedCoreNames() {
synchronized (modifyLock) {
- set = new TreeSet<>(cores.keySet());
- if (getTransientCacheHandler() != null) {
-set.addAll(getTransientCacheHandler().getLoadedCoreNames());
- }
+ TransientSolrCoreCache transientCoreCache = getTransientCacheHandler();
Review comment:
There is a comment in TransientSolrCoreCacheFactory.newInstance() line
60 that says a bad config should not prevent the CoreContainer from loading. It
logs an continues.
Then SolrCores.getTransientCacheHandler() checks transientCoreCache null. If
so it logs an error and returns null.
Before this first PR I tried to replace the null by a NOOP
TransientSolrCoreCache singleton. But I realize now that I could use an
instance with the parent container. Let's do it, it'll save a lot of null
checks.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Created] (SOLR-14989) SolrJ deleteByQuery does not work
Dr. Daniel Georg Kirschner created SOLR-14989:
-
Summary: SolrJ deleteByQuery does not work
Key: SOLR-14989
URL: https://issues.apache.org/jira/browse/SOLR-14989
Project: Solr
Issue Type: Bug
Security Level: Public (Default Security Level. Issues are Public)
Components: SolrJ
Affects Versions: 8.6.3
Reporter: Dr. Daniel Georg Kirschner
Solr 8.6.3 standalone accessed with SolrJ via HTTPS. Querys/post documents,
etc. works fine.
Code:
solrClient.deleteByQuery("some_core", '*:*')
throws:
Exception in thread "main" org.apache.solr.client.solrj.SolrServerException:
IOException occurred when talking to server at:
https://localhost:8983/solrException in thread "main"
org.apache.solr.client.solrj.SolrServerException: IOException occurred when
talking to server at: https://localhost:8983/solr at
org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:695)
at
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:266)
at
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:248)
at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:214) at
org.apache.solr.client.solrj.SolrClient.deleteByQuery(SolrClient.java:940) at
de.vw.solr.crawl.Solaritas.resetCore(Solaritas.groovy:89) at
de.vw.solr.crawl.Solaritas.main(Solaritas.groovy:205)Caused by:
org.apache.http.client.ClientProtocolException at
org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:187)
at
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
at
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
at
org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:571)
... 6 moreCaused by: org.apache.http.client.NonRepeatableRequestException:
Cannot retry request with a non-repeatable request entity. at
org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:225)
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at
org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at
org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
... 9 more
Problem seems to be known:
[https://stackoverflow.com/questions/35825280/solrj-deletebyquery-is-not-working]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (SOLR-8673) o.a.s.search.facet classes not public/extendable
[
https://issues.apache.org/jira/browse/SOLR-8673?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mikhail Khludnev updated SOLR-8673:
---
Status: Patch Available (was: Open)
> o.a.s.search.facet classes not public/extendable
>
>
> Key: SOLR-8673
> URL: https://issues.apache.org/jira/browse/SOLR-8673
> Project: Solr
> Issue Type: Improvement
> Components: Facet Module
>Affects Versions: 5.4.1
>Reporter: Markus Jelsma
>Priority: Major
> Fix For: 6.2, 7.0
>
> Attachments: SOLR-8673.patch, SOLR-8673.patch
>
>
> It is not easy to create a custom JSON facet function. A simple function
> based on AvgAgg quickly results in the following compilation failures:
> {code}
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-compiler-plugin:3.3:compile (default-compile)
> on project openindex-solr: Compilation failure: Compilation failure:
> [ERROR]
> /home/markus/projects/openindex/solr/trunk/src/main/java/i.o.s.search/facet/CustomAvgAgg.java:[22,36]
> org.apache.solr.search.facet.FacetContext is not public in
> org.apache.solr.search.facet; cannot be accessed from outside package
> [ERROR]
> /home/markus/projects/openindex/solr/trunk/src/main/java/i.o.s.search/facet/CustomAvgAgg.java:[23,36]
> org.apache.solr.search.facet.FacetDoubleMerger is not public in
> org.apache.solr.search.facet; cannot be accessed from outside package
> [ERROR]
> /home/markus/projects/openindex/solr/trunk/src/main/java/i.o.s.search/facet/CustomAvgAgg.java:[40,32]
> cannot find symbol
> [ERROR] symbol: class FacetContext
> [ERROR] location: class i.o.s.search.facet.CustomAvgAgg
> [ERROR]
> /home/markus/projects/openindex/solr/trunk/src/main/java/i.o.s.search/facet/CustomAvgAgg.java:[49,39]
> cannot find symbol
> [ERROR] symbol: class FacetDoubleMerger
> [ERROR] location: class i.o.s.search.facet.CustomAvgAgg
> [ERROR]
> /home/markus/projects/openindex/solr/trunk/src/main/java/i.o.s.search/facet/CustomAvgAgg.java:[54,43]
> cannot find symbol
> [ERROR] symbol: class Context
> [ERROR] location: class i.o.s.search.facet.CustomAvgAgg.Merger
> [ERROR]
> /home/markus/projects/openindex/solr/trunk/src/main/java/i.o.s.search/facet/CustomAvgAgg.java:[41,16]
> cannot find symbol
> [ERROR] symbol: class AvgSlotAcc
> [ERROR] location: class i.o.s.search.facet.CustomAvgAgg
> [ERROR]
> /home/markus/projects/openindex/solr/trunk/src/main/java/i.o.s.search/facet/CustomAvgAgg.java:[46,12]
> incompatible types: i.o.s.search.facet.CustomAvgAgg.Merger cannot be
> converted to org.apache.solr.search.facet.FacetMerger
> [ERROR]
> /home/markus/projects/openindex/solr/trunk/src/main/java/i.o.s.search/facet/CustomAvgAgg.java:[53,5]
> method does not override or implement a method from a supertype
> [ERROR]
> /home/markus/projects/openindex/solr/trunk/src/main/java/i.o.s.search/facet/CustomAvgAgg.java:[60,5]
> method does not override or implement a method from a supertype
> {code}
> It seems lots of classes are tucked away in FacetModule, which we can't reach
> from outside.
> Originates from this thread:
> http://mail-archives.apache.org/mod_mbox/lucene-solr-user/201602.mbox/%3ccab_8yd9ldbg_0zxm_h1igkfm6bqeypd5ilyy7tty8cztscv...@mail.gmail.com%3E
> ( also available at
> https://lists.apache.org/thread.html/9fddcad3136ec908ce1c57881f8d3069e5d153f08b71f80f3e18d995%401455019826%40%3Csolr-user.lucene.apache.org%3E
> )
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-9956) Solr java.lang.ArrayIndexOutOfBoundsException when indexed large amount of documents
[
https://issues.apache.org/jira/browse/SOLR-9956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17227826#comment-17227826
]
Erick Erickson commented on SOLR-9956:
--
The resolution is "unresolved", so no. At this point it's highly doubtful that
any new 6X releases will be made, we should probably close this.
> Solr java.lang.ArrayIndexOutOfBoundsException when indexed large amount of
> documents
>
>
> Key: SOLR-9956
> URL: https://issues.apache.org/jira/browse/SOLR-9956
> Project: Solr
> Issue Type: Bug
> Components: SolrCloud
>Affects Versions: 6.2.1, 6.3
> Environment: Ubuntu 14.04.4 LTS
>Reporter: Zhu JiaJun
>Priority: Critical
> Labels: query, solr, stats
> Attachments: solr.log
>
>
> I'm using solr 6.3.0. I indexed a big amount of docuements into one solr
> collection with one shard, it's 60G in the disk, which has around 2506889
> documents.
> I frequently get the ArrayIndexOutOfBoundsException when I send a simple
> stats request, for example:
> http://localhost:8983/solr/staging-update/select?start=0=0=2.2=*:*=true=6=json=asp_community_facet=asp_group_facet
> The solr log capture following exception as well as in the response like
> below:
> {code}
> {
> "responseHeader": {
> "zkConnected": true,
> "status": 500,
> "QTime": 11,
> "params": {
> "q": "*:*",
> "stats": "true",
> "start": "0",
> "timeAllowed": "6",
> "rows": "0",
> "version": "2.2",
> "wt": "json",
> "stats.field": [
> "asp_community_facet",
> "asp_group_facet"
> ]
> }
> },
> "response": {
> "numFound": 2506211,
> "start": 0,
> "docs": [ ]
> },
> "error": {
> "msg": "28",
> "trace": "java.lang.ArrayIndexOutOfBoundsException: 28\n\tat
> org.apache.solr.request.DocValuesStats.accumMulti(DocValuesStats.java:213)\n\tat
>
> org.apache.solr.request.DocValuesStats.getCounts(DocValuesStats.java:135)\n\tat
>
> org.apache.solr.handler.component.StatsField.computeLocalStatsValues(StatsField.java:424)\n\tat
>
> org.apache.solr.handler.component.StatsComponent.process(StatsComponent.java:58)\n\tat
>
> org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:295)\n\tat
>
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:153)\n\tat
> org.apache.solr.core.SolrCore.execute(SolrCore.java:2213)\n\tat
> org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:654)\n\tat
> org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:460)\n\tat
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:303)\n\tat
>
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:254)\n\tat
>
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)\n\tat
>
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)\n\tat
>
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)\n\tat
>
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)\n\tat
>
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)\n\tat
>
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1160)\n\tat
>
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)\n\tat
>
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)\n\tat
>
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1092)\n\tat
>
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)\n\tat
>
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)\n\tat
>
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)\n\tat
>
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)\n\tat
> org.eclipse.jetty.server.Server.handle(Server.java:518)\n\tat
> org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:308)\n\tat
> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:244)\n\tat
>
> org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)\n\tat
> org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)\n\tat
> org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)\n\tat
>
> org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:246)\n\tat
>
>
[GitHub] [lucene-solr] alessandrobenedetti commented on a change in pull request #1571: SOLR-14560: Interleaving for Learning To Rank
alessandrobenedetti commented on a change in pull request #1571:
URL: https://github.com/apache/lucene-solr/pull/1571#discussion_r519162680
##
File path:
solr/contrib/ltr/src/java/org/apache/solr/ltr/response/transform/LTRFeatureLoggerTransformerFactory.java
##
@@ -210,50 +216,59 @@ public void setContext(ResultContext context) {
}
// Setup LTRScoringQuery
- scoringQuery = SolrQueryRequestContextUtils.getScoringQuery(req);
- docsWereNotReranked = (scoringQuery == null);
- String featureStoreName =
SolrQueryRequestContextUtils.getFvStoreName(req);
- if (docsWereNotReranked || (featureStoreName != null &&
(!featureStoreName.equals(scoringQuery.getScoringModel().getFeatureStoreName()
{
-// if store is set in the transformer we should overwrite the logger
-
-final ManagedFeatureStore fr =
ManagedFeatureStore.getManagedFeatureStore(req.getCore());
-
-final FeatureStore store = fr.getFeatureStore(featureStoreName);
-featureStoreName = store.getName(); // if featureStoreName was null
before this gets actual name
-
-try {
- final LoggingModel lm = new LoggingModel(loggingModelName,
- featureStoreName, store.getFeatures());
-
- scoringQuery = new LTRScoringQuery(lm,
- LTRQParserPlugin.extractEFIParams(localparams),
- true,
- threadManager); // request feature weights to be created for all
features
-
-}catch (final Exception e) {
- throw new SolrException(SolrException.ErrorCode.BAD_REQUEST,
- "retrieving the feature store "+featureStoreName, e);
-}
- }
+ rerankingQueries = SolrQueryRequestContextUtils.getScoringQueries(req);
- if (scoringQuery.getOriginalQuery() == null) {
-scoringQuery.setOriginalQuery(context.getQuery());
+ docsWereNotReranked = (rerankingQueries == null ||
rerankingQueries.length == 0);
+ if (docsWereNotReranked) {
+rerankingQueries = new LTRScoringQuery[]{null};
}
- if (scoringQuery.getFeatureLogger() == null){
-scoringQuery.setFeatureLogger(
SolrQueryRequestContextUtils.getFeatureLogger(req) );
- }
- scoringQuery.setRequest(req);
-
- featureLogger = scoringQuery.getFeatureLogger();
+ modelWeights = new LTRScoringQuery.ModelWeight[rerankingQueries.length];
+ String featureStoreName =
SolrQueryRequestContextUtils.getFvStoreName(req);
+ for (int i = 0; i < rerankingQueries.length; i++) {
+LTRScoringQuery scoringQuery = rerankingQueries[i];
+if ((scoringQuery == null || !(scoringQuery instanceof
OriginalRankingLTRScoringQuery)) && (docsWereNotReranked || (featureStoreName
!= null &&
!featureStoreName.equals(scoringQuery.getScoringModel().getFeatureStoreName()
{
Review comment:
Spent a lot of time on this, I will continue on Monday, some
considerations so far:
- in interleaving the logging model scoring query keeps also the
interleaving picks (currently as a subclass of the scoring query), so we would
need 2 logging models, by the way, I had various problems due to class cast
exceptions and array around, so I will think a bit more about the subclass
implementation to understand if it's actually an headache or full benefit,
- I think the solution will be to implement only one model, 2 different
scoringQuery (that contain the picks) and change the feature vector cache to
key on: private static int fvCacheKey(LTRScoringQuery scoringQuery, int docid) {
return scoringQuery.getScoringModel().hashCode() + (31 * docid);
}
I keep you posted on monday
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8982) Make NativeUnixDirectory pure java now that direct IO is possible
[
https://issues.apache.org/jira/browse/LUCENE-8982?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17227753#comment-17227753
]
Dawid Weiss commented on LUCENE-8982:
-
Hi Zach. I don't think we've ported this particular build fragment from ant -
thank you for contributing it. It can be a separate module but I think it
shouldn't be forced upon everyone on default convention tasks (assemble, check)
- I didn't check if it's currently the case. Native builds require extra setup
and tools that some platforms may lack (Windows, Mac). If it's the case that
these tasks run by default is then we'll need to add either an explicit
property (-Pbuild.native=true?) or a dedicated task (native) and make the build
conditional on whether it is explicitly defined.
I can take a look if you have a problem with this, let me know.
> Make NativeUnixDirectory pure java now that direct IO is possible
> -
>
> Key: LUCENE-8982
> URL: https://issues.apache.org/jira/browse/LUCENE-8982
> Project: Lucene - Core
> Issue Type: Improvement
> Components: modules/misc
>Reporter: Michael McCandless
>Priority: Major
> Time Spent: 1h
> Remaining Estimate: 0h
>
> {{NativeUnixDirectory}} is a {{Directory}} implementation that uses direct IO
> to write newly merged segments. Direct IO bypasses the kernel's buffer cache
> and write cache, making merge writes "invisible" to the kernel, though the
> reads for merging the N segments are still going through the kernel.
> But today, {{NativeUnixDirectory}} uses a small JNI wrapper to access the
> {{O_DIRECT}} flag to {{open}} ... since JDK9 we can now pass that flag in
> pure java code, so we should now fix {{NativeUnixDirectory}} to not use JNI
> anymore.
> We should also run some more realistic benchmarks seeing if this option
> really helps nodes that are doing concurrent indexing (merging) and searching.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
