[jira] [Commented] (LUCENE-8920) Reduce size of FSTs due to use of direct-addressing encoding
[ https://issues.apache.org/jira/browse/LUCENE-8920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16973993#comment-16973993 ] Adrien Grand commented on LUCENE-8920: -- This gives a nice bump on the PKLookup task http://people.apache.org/~mikemccand/lucenebench/PKLookup.html. Almost on par with the previous direct addressing patch. > Reduce size of FSTs due to use of direct-addressing encoding > - > > Key: LUCENE-8920 > URL: https://issues.apache.org/jira/browse/LUCENE-8920 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Michael Sokolov >Priority: Minor > Attachments: TestTermsDictRamBytesUsed.java > > Time Spent: 4.5h > Remaining Estimate: 0h > > Some data can lead to worst-case ~4x RAM usage due to this optimization. > Several ideas were suggested to combat this on the mailing list: > bq. I think we can improve thesituation here by tracking, per-FST instance, > the size increase we're seeing while building (or perhaps do a preliminary > pass before building) in order to decide whether to apply the encoding. > bq. we could also make the encoding a bit more efficient. For instance I > noticed that arc metadata is pretty large in some cases (in the 10-20 bytes) > which make gaps very costly. Associating each label with a dense id and > having an intermediate lookup, ie. lookup label -> id and then id->arc offset > instead of doing label->arc directly could save a lot of space in some cases? > Also it seems that we are repeating the label in the arc metadata when > array-with-gaps is used, even though it shouldn't be necessary since the > label is implicit from the address? -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Comment Edited] (SOLR-13930) Fix failing TestKoreanTokenizer test
[ https://issues.apache.org/jira/browse/SOLR-13930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16973953#comment-16973953 ] Pinkesh Sharma edited comment on SOLR-13930 at 11/14/19 6:34 AM: - I was looking into this and I build the specific project and the build was successful, can you help me the with steps to reproduce this. was (Author: pinkeshsharm...@gmail.com): I was looking into this and I build the specific project and the build was successful, can you help me with steps to reproduce this. > Fix failing TestKoreanTokenizer test > > > Key: SOLR-13930 > URL: https://issues.apache.org/jira/browse/SOLR-13930 > Project: Solr > Issue Type: Task > Security Level: Public(Default Security Level. Issues are Public) > Environment: This fails with: > java.lang.RuntimeException: Cannot find userdict.txt in test classpath! > userdict.txt gets copied when I test on the trunk branch to (at least I think > this is the corresponding one): > ./lucene/build/analysis/nori/*classes*/test/org/apache/lucene/analysis/ko/userdict.txt > So my presumption is that the ant build takes care of this and somehow the > classpath is set to include it. > This is on a clean checkout of the current gradle_8 branch, _without_ trying > to do anything with Gradle. >Reporter: Erick Erickson >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-13930) Fix failing TestKoreanTokenizer test
[ https://issues.apache.org/jira/browse/SOLR-13930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16973953#comment-16973953 ] Pinkesh Sharma commented on SOLR-13930: --- I was looking into this and I build the specific project and the build was successful, can you help me with steps to reproduce this. > Fix failing TestKoreanTokenizer test > > > Key: SOLR-13930 > URL: https://issues.apache.org/jira/browse/SOLR-13930 > Project: Solr > Issue Type: Task > Security Level: Public(Default Security Level. Issues are Public) > Environment: This fails with: > java.lang.RuntimeException: Cannot find userdict.txt in test classpath! > userdict.txt gets copied when I test on the trunk branch to (at least I think > this is the corresponding one): > ./lucene/build/analysis/nori/*classes*/test/org/apache/lucene/analysis/ko/userdict.txt > So my presumption is that the ant build takes care of this and somehow the > classpath is set to include it. > This is on a clean checkout of the current gradle_8 branch, _without_ trying > to do anything with Gradle. >Reporter: Erick Erickson >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-13890) Add postfilter support to {!terms} queries
[
https://issues.apache.org/jira/browse/SOLR-13890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16973952#comment-16973952
]
Mikhail Khludnev commented on SOLR-13890:
-
FYI, Automaton uses inverted index, it works per segment. If number of terms is
small it builds per segment conjunction, thus if it bypasses filtercache it
will be even lazy, otherwise it eagerly builds per-segment docset.
> Add postfilter support to {!terms} queries
> --
>
> Key: SOLR-13890
> URL: https://issues.apache.org/jira/browse/SOLR-13890
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
> Components: query parsers
>Affects Versions: master (9.0)
>Reporter: Jason Gerlowski
>Assignee: Jason Gerlowski
>Priority: Major
> Attachments: SOLR-13890.patch, SOLR-13890.patch
>
>
> There are some use-cases where it'd be nice if the "terms" qparser created a
> query that could be run as a postfilter. Particularly, when users are
> checking for hundreds or thousands of terms, a postfilter implementation can
> be more performant than the standard processing.
> WIth this issue, I'd like to propose a post-filter implementation for the
> {{docValuesTermsFilter}} "method". Postfilter creation can use a
> SortedSetDocValues object to populate a DV bitset with the "terms" being
> checked for. Each document run through the post-filter can look at their
> doc-values for the field in question and check them efficiently against the
> constructed bitset.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] noblepaul commented on a change in pull request #994: SOLR-13662: Package Manager (CLI)
noblepaul commented on a change in pull request #994: SOLR-13662: Package
Manager (CLI)
URL: https://github.com/apache/lucene-solr/pull/994#discussion_r346117479
##
File path:
solr/core/src/java/org/apache/solr/packagemanager/PackageManager.java
##
@@ -0,0 +1,415 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.solr.packagemanager;
+
+import static org.apache.solr.packagemanager.PackageUtils.getMapper;
+
+import java.io.Closeable;
+import java.io.IOException;
+import java.lang.invoke.MethodHandles;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Scanner;
+
+import org.apache.solr.client.solrj.impl.HttpSolrClient;
+import org.apache.solr.common.SolrException;
+import org.apache.solr.common.SolrException.ErrorCode;
+import org.apache.solr.common.cloud.SolrZkClient;
+import org.apache.solr.packagemanager.SolrPackage.Command;
+import org.apache.solr.packagemanager.SolrPackage.Manifest;
+import org.apache.solr.packagemanager.SolrPackage.Plugin;
+import org.apache.solr.pkg.PackagePluginHolder;
+import org.apache.solr.util.SolrCLI;
+import org.apache.zookeeper.KeeperException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.base.Strings;
+import com.jayway.jsonpath.JsonPath;
+import com.jayway.jsonpath.PathNotFoundException;
+
+/**
+ * Handles most of the management of packages that are already installed in
Solr.
+ */
+public class PackageManager implements Closeable {
+
+ final String solrBaseUrl;
+ final HttpSolrClient solrClient;
+ final SolrZkClient zkClient;
+
+ private Map> packages = null;
+
+ private static final Logger log =
LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+
+ public PackageManager(HttpSolrClient solrClient, String solrBaseUrl, String
zkHost) {
+this.solrBaseUrl = solrBaseUrl;
+this.solrClient = solrClient;
+this.zkClient = new SolrZkClient(zkHost, 3);
+log.info("Done initializing a zkClient instance...");
+ }
+
+ @Override
+ public void close() throws IOException {
+if (zkClient != null) {
+ zkClient.close();
+}
+ }
+
+ public List fetchInstalledPackageInstances() throws
SolrException {
+log.info("Getting packages from packages.json...");
+List ret = new ArrayList();
+packages = new HashMap>();
+try {
+ Map packagesZnodeMap = null;
+
+ if (zkClient.exists("/packages.json", true) == true) {
+packagesZnodeMap = (Map)getMapper().readValue(
+new String(zkClient.getData("/packages.json", null, null, true),
"UTF-8"), Map.class).get("packages");
+for (Object packageName: packagesZnodeMap.keySet()) {
+ List pkg = (List)packagesZnodeMap.get(packageName);
+ for (Map pkgVersion: (List)pkg) {
+Manifest manifest = PackageUtils.fetchManifest(solrClient,
solrBaseUrl, pkgVersion.get("manifest").toString(),
pkgVersion.get("manifestSHA512").toString());
+List solrplugins = manifest.plugins;
+SolrPackageInstance pkgInstance = new
SolrPackageInstance(packageName.toString(), null,
+pkgVersion.get("version").toString(), manifest, solrplugins,
manifest.parameterDefaults);
+List list =
packages.containsKey(packageName)? packages.get(packageName): new
ArrayList();
+list.add(pkgInstance);
+packages.put(packageName.toString(), list);
+ret.add(pkgInstance);
+ }
+}
+ }
+} catch (Exception e) {
+ throw new SolrException(ErrorCode.BAD_REQUEST, e);
+}
+log.info("Got packages: "+ret);
+return ret;
+ }
+
+ public Map getPackagesDeployed(String
collection) {
+String paramsJson =
PackageUtils.getJsonStringFromUrl(solrClient.getHttpClient(),
solrBaseUrl+"/api/collections/"+collection+"/config/params/PKG_VERSIONS?omitHeader=true");
Review comment:
We can simply use the following and avoid dependency on `JsonPath`
` NavigableObject result = (NavigableObject)
[GitHub] [lucene-solr] noblepaul commented on a change in pull request #994: SOLR-13662: Package Manager (CLI)
noblepaul commented on a change in pull request #994: SOLR-13662: Package
Manager (CLI)
URL: https://github.com/apache/lucene-solr/pull/994#discussion_r346117479
##
File path:
solr/core/src/java/org/apache/solr/packagemanager/PackageManager.java
##
@@ -0,0 +1,415 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.solr.packagemanager;
+
+import static org.apache.solr.packagemanager.PackageUtils.getMapper;
+
+import java.io.Closeable;
+import java.io.IOException;
+import java.lang.invoke.MethodHandles;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Scanner;
+
+import org.apache.solr.client.solrj.impl.HttpSolrClient;
+import org.apache.solr.common.SolrException;
+import org.apache.solr.common.SolrException.ErrorCode;
+import org.apache.solr.common.cloud.SolrZkClient;
+import org.apache.solr.packagemanager.SolrPackage.Command;
+import org.apache.solr.packagemanager.SolrPackage.Manifest;
+import org.apache.solr.packagemanager.SolrPackage.Plugin;
+import org.apache.solr.pkg.PackagePluginHolder;
+import org.apache.solr.util.SolrCLI;
+import org.apache.zookeeper.KeeperException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.base.Strings;
+import com.jayway.jsonpath.JsonPath;
+import com.jayway.jsonpath.PathNotFoundException;
+
+/**
+ * Handles most of the management of packages that are already installed in
Solr.
+ */
+public class PackageManager implements Closeable {
+
+ final String solrBaseUrl;
+ final HttpSolrClient solrClient;
+ final SolrZkClient zkClient;
+
+ private Map> packages = null;
+
+ private static final Logger log =
LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+
+ public PackageManager(HttpSolrClient solrClient, String solrBaseUrl, String
zkHost) {
+this.solrBaseUrl = solrBaseUrl;
+this.solrClient = solrClient;
+this.zkClient = new SolrZkClient(zkHost, 3);
+log.info("Done initializing a zkClient instance...");
+ }
+
+ @Override
+ public void close() throws IOException {
+if (zkClient != null) {
+ zkClient.close();
+}
+ }
+
+ public List fetchInstalledPackageInstances() throws
SolrException {
+log.info("Getting packages from packages.json...");
+List ret = new ArrayList();
+packages = new HashMap>();
+try {
+ Map packagesZnodeMap = null;
+
+ if (zkClient.exists("/packages.json", true) == true) {
+packagesZnodeMap = (Map)getMapper().readValue(
+new String(zkClient.getData("/packages.json", null, null, true),
"UTF-8"), Map.class).get("packages");
+for (Object packageName: packagesZnodeMap.keySet()) {
+ List pkg = (List)packagesZnodeMap.get(packageName);
+ for (Map pkgVersion: (List)pkg) {
+Manifest manifest = PackageUtils.fetchManifest(solrClient,
solrBaseUrl, pkgVersion.get("manifest").toString(),
pkgVersion.get("manifestSHA512").toString());
+List solrplugins = manifest.plugins;
+SolrPackageInstance pkgInstance = new
SolrPackageInstance(packageName.toString(), null,
+pkgVersion.get("version").toString(), manifest, solrplugins,
manifest.parameterDefaults);
+List list =
packages.containsKey(packageName)? packages.get(packageName): new
ArrayList();
+list.add(pkgInstance);
+packages.put(packageName.toString(), list);
+ret.add(pkgInstance);
+ }
+}
+ }
+} catch (Exception e) {
+ throw new SolrException(ErrorCode.BAD_REQUEST, e);
+}
+log.info("Got packages: "+ret);
+return ret;
+ }
+
+ public Map getPackagesDeployed(String
collection) {
+String paramsJson =
PackageUtils.getJsonStringFromUrl(solrClient.getHttpClient(),
solrBaseUrl+"/api/collections/"+collection+"/config/params/PKG_VERSIONS?omitHeader=true");
Review comment:
We can simply use the following and avoid dependency on `JsonPath`
` NavigableObject result = (NavigableObject)
[GitHub] [lucene-solr] noblepaul commented on a change in pull request #994: SOLR-13662: Package Manager (CLI)
noblepaul commented on a change in pull request #994: SOLR-13662: Package
Manager (CLI)
URL: https://github.com/apache/lucene-solr/pull/994#discussion_r346110733
##
File path:
solr/core/src/test-files/solr/question-answer-repository/repository.json
##
@@ -0,0 +1,57 @@
+[
+ {
+"name": "question-answer",
+"description": "A natural language question answering plugin",
+"versions": [
+ {
+"version": "1.0.0",
+"date": "2019-01-01",
+"artifacts": [
+ {
+"url": "question-answer-request-handler-1.0.jar",
+"sig":
"C9UWKkucmY3UNzqn0VLneVMe9kCbJjw7Urc76vGenoRwp32xvNn5ZIGZ7G34xZP7cVjqn/ltDlLWBZ/C3eAtuw=="
+ }
+],
+"manifest": {
+ "min-solr-version": "8.0",
+ "max-solr-version": "9.99",
+ "plugins": [
+{
+ "name": "request-handler",
+ "setup-command": {
+"path": "/api/collections/${collection}/config",
Review comment:
do you limit the calls to `/api/collections/${collection}/config*`
`/api/collections/${collection}/schema*` ?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9042) Refactor TopGroups.merge tests
[
https://issues.apache.org/jira/browse/LUCENE-9042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16973912#comment-16973912
]
Lucene/Solr QA commented on LUCENE-9042:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
16s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 17s{color}
| {color:red} lucene_grouping generated 4 new + 108 unchanged - 0 fixed = 112
total (was 108) {color} |
| {color:green}+1{color} | {color:green} Release audit (RAT) {color} |
{color:green} 0m 17s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} Check forbidden APIs {color} | {color:red}
0m 17s{color} | {color:red} Check forbidden APIs check-forbidden-apis failed
{color} |
| {color:red}-1{color} | {color:red} Validate source patterns {color} |
{color:red} 0m 17s{color} | {color:red} Check forbidden APIs
check-forbidden-apis failed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
23s{color} | {color:green} grouping in the patch passed. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 1m 55s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | LUCENE-9042 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12985735/LUCENE-9042.patch |
| Optional Tests | compile javac unit ratsources checkforbiddenapis
validatesourcepatterns |
| uname | Linux lucene1-us-west 4.15.0-54-generic #58-Ubuntu SMP Mon Jun 24
10:55:24 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | ant |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-LUCENE-Build/sourcedir/dev-tools/test-patch/lucene-solr-yetus-personality.sh
|
| git revision | master / 068b6babac3 |
| ant | version: Apache Ant(TM) version 1.10.5 compiled on March 28 2019 |
| Default Java | LTS |
| javac |
https://builds.apache.org/job/PreCommit-LUCENE-Build/228/artifact/out/diff-compile-javac-lucene_grouping.txt
|
| Check forbidden APIs |
https://builds.apache.org/job/PreCommit-LUCENE-Build/228/artifact/out/patch-check-forbidden-apis-lucene.txt
|
| Validate source patterns |
https://builds.apache.org/job/PreCommit-LUCENE-Build/228/artifact/out/patch-check-forbidden-apis-lucene.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-LUCENE-Build/228/testReport/ |
| modules | C: lucene/grouping U: lucene/grouping |
| Console output |
https://builds.apache.org/job/PreCommit-LUCENE-Build/228/console |
| Powered by | Apache Yetus 0.7.0 http://yetus.apache.org |
This message was automatically generated.
> Refactor TopGroups.merge tests
> --
>
> Key: LUCENE-9042
> URL: https://issues.apache.org/jira/browse/LUCENE-9042
> Project: Lucene - Core
> Issue Type: Improvement
>Reporter: Diego Ceccarelli
>Priority: Minor
> Attachments: LUCENE-9042.patch
>
>
> This task proposes a refactoring of the test coverage for the
> {{TopGroups.merge}} method implemented in LUCENE-9010. For now it will cover
> only 3 main cases.
> 1. Merging to empty TopGroups
> 2. Merging a TopGroups with scores and a TopGroups without scores (currently
> broken because of LUCENE-8996 bug)
> 3. Merging two TopGroups with scores.
> I'm planning to increase the coverage testing also invalid inputs but I would
> do that in a separate PR to keep the code readable.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-13452) Update the lucene-solr build from Ivy+Ant+Maven (shadow build) to Gradle.
[ https://issues.apache.org/jira/browse/SOLR-13452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16973883#comment-16973883 ] ASF subversion and git services commented on SOLR-13452: Commit 40db8743e2e65728eee1819a1242ecbb809f4c3d in lucene-solr's branch refs/heads/jira/SOLR-13452_gradle_8 from Erick Erickson [ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=40db874 ] SOLR-13452: Update the lucene-solr build from Ivy+Ant+Maven (shadow build) to Gradle. Updated dependencies and created parallel license directories (Lucene and Solr) to handle 'jar-checksum' and 'jarChecksum' thrashing > Update the lucene-solr build from Ivy+Ant+Maven (shadow build) to Gradle. > - > > Key: SOLR-13452 > URL: https://issues.apache.org/jira/browse/SOLR-13452 > Project: Solr > Issue Type: Improvement > Components: Build >Reporter: Mark Miller >Assignee: Mark Miller >Priority: Major > Fix For: master (9.0) > > Attachments: gradle-build.pdf > > Time Spent: 0.5h > Remaining Estimate: 0h > > I took some things from the great work that Dat did in > [https://github.com/apache/lucene-solr/tree/jira/gradle] and took the ball a > little further. > > When working with gradle in sub modules directly, I recommend > [https://github.com/dougborg/gdub] > This gradle branch uses the following plugin for version locking, version > configuration and version consistency across modules: > [https://github.com/palantir/gradle-consistent-versions] > > https://github.com/apache/lucene-solr/tree/jira/SOLR-13452_gradle_8 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Created] (SOLR-13930) Fix failing TestKoreanTokenizer test
Erick Erickson created SOLR-13930: - Summary: Fix failing TestKoreanTokenizer test Key: SOLR-13930 URL: https://issues.apache.org/jira/browse/SOLR-13930 Project: Solr Issue Type: Task Security Level: Public (Default Security Level. Issues are Public) Environment: This fails with: java.lang.RuntimeException: Cannot find userdict.txt in test classpath! userdict.txt gets copied when I test on the trunk branch to (at least I think this is the corresponding one): ./lucene/build/analysis/nori/*classes*/test/org/apache/lucene/analysis/ko/userdict.txt So my presumption is that the ant build takes care of this and somehow the classpath is set to include it. This is on a clean checkout of the current gradle_8 branch, _without_ trying to do anything with Gradle. Reporter: Erick Erickson -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Created] (SOLR-13928) Insure that Gradle build flags files not part of Git like "Ant precommit"
Erick Erickson created SOLR-13928: - Summary: Insure that Gradle build flags files not part of Git like "Ant precommit" Key: SOLR-13928 URL: https://issues.apache.org/jira/browse/SOLR-13928 Project: Solr Issue Type: Task Security Level: Public (Default Security Level. Issues are Public) Reporter: Erick Erickson I have a bunch of files in my source tree that don't get flagged by "gradlew build -x test" which runs _some_ of the precommit-like tests. I can't find a similar task in the gradle build. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] chatman commented on a change in pull request #994: SOLR-13662: Package Manager (CLI)
chatman commented on a change in pull request #994: SOLR-13662: Package Manager
(CLI)
URL: https://github.com/apache/lucene-solr/pull/994#discussion_r346099127
##
File path: solr/bin/solr
##
@@ -764,6 +764,56 @@ function get_info() {
return $CODE
} # end get_info
+function run_package() {
+ runningSolrUrl=""
+
+ numSolrs=`find "$SOLR_PID_DIR" -name "solr-*.pid" -type f | wc -l | tr -d '
'`
+ if [ "$numSolrs" != "0" ]; then
+#echo -e "\nFound $numSolrs Solr nodes: "
+while read PIDF
+ do
+ID=`cat "$PIDF"`
+port=`jetty_port "$ID"`
+if [ "$port" != "" ]; then
+ #echo -e "\nSolr process $ID running on port $port"
+ runningSolrUrl="$SOLR_URL_SCHEME://$SOLR_TOOL_HOST:$port/solr"
+ break
+ CODE=$?
+ echo ""
+else
+ echo -e "\nSolr process $ID from $PIDF not found."
+ CODE=1
+fi
+done < <(find "$SOLR_PID_DIR" -name "solr-*.pid" -type f)
+ else
+# no pid files but check using ps just to be sure
+numSolrs=`ps auxww | grep start\.jar | grep solr\.solr\.home | grep -v
grep | wc -l | sed -e 's/^[ \t]*//'`
Review comment:
Added a script that doesn't have the magic of auto-supplying the -solrUrl
parameter (based on a running Solr instance). User would have to supply that
himself/herself while using the package manager. I'll open a separate JIRA to
remove this annoyance that I can't tackle myself at the moment.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] chatman edited a comment on issue #994: SOLR-13662: Package Manager (CLI)
chatman edited a comment on issue #994: SOLR-13662: Package Manager (CLI) URL: https://github.com/apache/lucene-solr/pull/994#issuecomment-553687639 > Cannot find RefGuide docs in this PR? Better to include it now than after merge to master. I'll add it in a separate PR so that it is easier for Cassandra to review. > Question on signatures. Can we have a pure Java way of signing packages or do we force use of openssl? Wasn't aware of any such way. I actually want to use GPG (since Apache commiters all use trusted GPG keys). I just didn't feel like tackling the bouncy castle dependency at the moment. Maybe, we can add that later. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9018) Separator for ConcatenateGraphFilterFactory
[
https://issues.apache.org/jira/browse/LUCENE-9018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16973872#comment-16973872
]
Lucene/Solr QA commented on LUCENE-9018:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
3s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} Release audit (RAT) {color} |
{color:green} 1m 24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} Check forbidden APIs {color} |
{color:green} 1m 24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} Validate source patterns {color} |
{color:green} 1m 24s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m
39s{color} | {color:green} common in the patch passed. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 13m 7s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | LUCENE-9018 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12985615/LUCENE-9018.patch |
| Optional Tests | compile javac unit ratsources checkforbiddenapis
validatesourcepatterns |
| uname | Linux lucene2-us-west.apache.org 4.4.0-112-generic #135-Ubuntu SMP
Fri Jan 19 11:48:36 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | ant |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-LUCENE-Build/sourcedir/dev-tools/test-patch/lucene-solr-yetus-personality.sh
|
| git revision | master / 068b6ba |
| ant | version: Apache Ant(TM) version 1.9.6 compiled on July 20 2018 |
| Default Java | LTS |
| Test Results |
https://builds.apache.org/job/PreCommit-LUCENE-Build/227/testReport/ |
| modules | C: lucene/analysis/common U: lucene/analysis/common |
| Console output |
https://builds.apache.org/job/PreCommit-LUCENE-Build/227/console |
| Powered by | Apache Yetus 0.7.0 http://yetus.apache.org |
This message was automatically generated.
> Separator for ConcatenateGraphFilterFactory
> ---
>
> Key: LUCENE-9018
> URL: https://issues.apache.org/jira/browse/LUCENE-9018
> Project: Lucene - Core
> Issue Type: Improvement
> Components: modules/analysis
>Reporter: Stanislav Mikulchik
>Assignee: David Smiley
>Priority: Minor
> Attachments: LUCENE-9018.patch, LUCENE-9018.patch, LUCENE-9018.patch
>
>
> I would like to have an option to choose a separator to use for token
> concatenation. Currently ConcatenateGraphFilterFactory can use only "\u001F"
> symbol.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] chatman commented on issue #994: SOLR-13662: Package Manager (CLI)
chatman commented on issue #994: SOLR-13662: Package Manager (CLI) URL: https://github.com/apache/lucene-solr/pull/994#issuecomment-553687639 > Cannot find RefGuide docs in this PR? Better to include it now than after merge to master. I'll add it in a separate PR so that it is easier for Cassandra to review. > Question on signatures. Can we have a pure Java way of signing packages or do we force use of openssl? Wasn't aware of any such way. I actually want to use GPG (since Apache commiters all use trusted GPG keys). I just didn't feel like tackling the bouncy castle dependency at the moment. Maybe, we can add that later. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] chatman commented on a change in pull request #994: SOLR-13662: Package Manager (CLI)
chatman commented on a change in pull request #994: SOLR-13662: Package Manager
(CLI)
URL: https://github.com/apache/lucene-solr/pull/994#discussion_r346092779
##
File path:
solr/core/src/test-files/solr/question-answer-repository/repository.json
##
@@ -0,0 +1,57 @@
+[
+ {
+"name": "question-answer",
+"description": "A natural language question answering plugin",
+"versions": [
+ {
+"version": "1.0.0",
+"date": "2019-01-01",
+"artifacts": [
+ {
+"url": "question-answer-request-handler-1.0.jar",
+"sig":
"C9UWKkucmY3UNzqn0VLneVMe9kCbJjw7Urc76vGenoRwp32xvNn5ZIGZ7G34xZP7cVjqn/ltDlLWBZ/C3eAtuw=="
+ }
+],
+"manifest": {
+ "min-solr-version": "8.0",
+ "max-solr-version": "9.99",
+ "plugins": [
+{
+ "name": "request-handler",
+ "setup-command": {
+"path": "/api/collections/${collection}/config",
Review comment:
Package manager (CLI) needs write access to ZK for "add-repo", since it
needs to publish the trusted public key. A user must only "add-repo" for a
trusted publisher. Only then should the user expect non-malicious packages.
Once a package is already in the system, it doesn't matter if the deploy
step (which is a CLI action) executes a HTTP command or does something
malicious from inside the jar file.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] chatman commented on a change in pull request #994: SOLR-13662: Package Manager (CLI)
chatman commented on a change in pull request #994: SOLR-13662: Package Manager
(CLI)
URL: https://github.com/apache/lucene-solr/pull/994#discussion_r346092020
##
File path: solr/core/src/java/org/apache/solr/packagemanager/SolrPackage.java
##
@@ -0,0 +1,143 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.solr.packagemanager;
+
+
+import java.util.Date;
+import java.util.List;
+import java.util.Map;
+
+import org.apache.solr.common.annotation.JsonProperty;
+import org.apache.solr.common.util.ReflectMapWriter;
+
+/**
+ * Describes a package (along with all released versions) as it appears in a
repository.
+ */
+public class SolrPackage implements Comparable, ReflectMapWriter {
+
+ @JsonProperty("name")
+ public String name;
+
+ @JsonProperty("description")
+ public String description;
+
+ @JsonProperty("versions")
+ public List versions;
+
+ @JsonProperty("repository")
+ private String repository;
+
+ @Override
+ public String toString() {
+return jsonStr();
+ }
+
+ public static class SolrPackageRelease implements ReflectMapWriter {
+@JsonProperty("version")
+public String version;
+
+@JsonProperty("date")
+public Date date;
+
+@JsonProperty("artifacts")
+public List artifacts;
+
+@JsonProperty("manifest")
Review comment:
This represents a version / release. The manifest is inside (either
explicitly specified in the repository.json or contained within the jar).
Here's an example:
https://github.com/apache/lucene-solr/pull/994/files#diff-ab1e50fd5605ee1a39764390b9d4faeeR6-R42
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] chatman commented on a change in pull request #994: SOLR-13662: Package Manager (CLI)
chatman commented on a change in pull request #994: SOLR-13662: Package Manager
(CLI)
URL: https://github.com/apache/lucene-solr/pull/994#discussion_r346091340
##
File path:
solr/core/src/java/org/apache/solr/packagemanager/RepositoryManager.java
##
@@ -0,0 +1,328 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.solr.packagemanager;
+
+import static org.apache.solr.packagemanager.PackageUtils.getMapper;
+
+import java.io.IOException;
+import java.io.UnsupportedEncodingException;
+import java.lang.invoke.MethodHandles;
+import java.net.MalformedURLException;
+import java.net.URL;
+import java.nio.ByteBuffer;
+import java.nio.file.Path;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+
+import org.apache.commons.io.FileUtils;
+import org.apache.commons.io.IOUtils;
+import org.apache.lucene.util.Version;
+import org.apache.solr.client.solrj.SolrRequest;
+import org.apache.solr.client.solrj.SolrServerException;
+import org.apache.solr.client.solrj.impl.HttpSolrClient;
+import org.apache.solr.client.solrj.request.V2Request;
+import org.apache.solr.client.solrj.request.beans.Package;
+import org.apache.solr.client.solrj.response.V2Response;
+import org.apache.solr.common.SolrException;
+import org.apache.solr.common.SolrException.ErrorCode;
+import org.apache.solr.common.cloud.SolrZkClient;
+import org.apache.solr.core.BlobRepository;
+import org.apache.solr.packagemanager.SolrPackage.Artifact;
+import org.apache.solr.packagemanager.SolrPackage.SolrPackageRelease;
+import org.apache.solr.pkg.PackageAPI;
+import org.apache.zookeeper.CreateMode;
+import org.apache.zookeeper.KeeperException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+/**
+ * Handles most of the management of repositories and packages present in
external repositories.
+ */
+public class RepositoryManager {
+
+ private static final Logger log =
LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+ final private PackageManager packageManager;
+
+ public static final String systemVersion = Version.LATEST.toString();
+
+ final HttpSolrClient solrClient;
+
+ public RepositoryManager(HttpSolrClient solrClient, PackageManager
packageManager) {
+this.packageManager = packageManager;
+this.solrClient = solrClient;
+ }
+
+ public List getPackages() {
+List list = new ArrayList<>(getPackagesMap().values());
+Collections.sort(list);
+return list;
+ }
+
+ /**
+ * Get a map of package name to {@link SolrPackage} objects
+ */
+ public Map getPackagesMap() {
+Map packagesMap = new HashMap<>();
+for (PackageRepository repository: getRepositories()) {
+ packagesMap.putAll(repository.getPackages());
+}
+
+return packagesMap;
+ }
+
+ /**
+ * List of added repositories
+ */
+ public List getRepositories() {
+// TODO: Instead of fetching again and again, we should look for caching
this
+PackageRepository items[];
+try {
+ items =
getMapper().readValue(getRepositoriesJson(packageManager.zkClient),
DefaultPackageRepository[].class);
+} catch (IOException | KeeperException | InterruptedException e) {
+ throw new SolrException(ErrorCode.SERVER_ERROR, e);
+}
+List repositories = Arrays.asList(items);
+
+for (PackageRepository updateRepository: repositories) {
+ updateRepository.refresh();
+}
+
+return repositories;
+ }
+
+ /**
+ * Add a repository to Solr
+ */
+ public void addRepository(String name, String uri) throws KeeperException,
InterruptedException, MalformedURLException, IOException {
+String existingRepositoriesJson =
getRepositoriesJson(packageManager.zkClient);
+log.info(existingRepositoriesJson);
+
+List repos = getMapper().readValue(existingRepositoriesJson, List.class);
+repos.add(new DefaultPackageRepository(name, uri));
+if (packageManager.zkClient.exists("/repositories.json", true) == false) {
+ packageManager.zkClient.create("/repositories.json",
getMapper().writeValueAsString(repos).getBytes("UTF-8"), CreateMode.PERSISTENT,
[GitHub] [lucene-solr] millerjeff0 commented on issue #1009: SOLR-13926
millerjeff0 commented on issue #1009: SOLR-13926 URL: https://github.com/apache/lucene-solr/pull/1009#issuecomment-553645188 Yeah I’ll take a shot at method docs next, don’t merge this yet I wanted to get the class doc out for review first. > On Nov 13, 2019, at 2:56 PM, David Smiley wrote: > > > Cool. > > In addition, I find the method contracts to the methods on DocRouter and it's subclasses highly unspecified (no javadocs or don't specify this). In particular, it's not evident when a id/shardKey/routeKey param is assumed to be the prefix, or assumed to be the prefix plus exclamation, or prefix+!+theRest or some combinations only or whatever. It makes reasoning about what a method should do difficult. > > — > You are receiving this because you authored the thread. > Reply to this email directly, view it on GitHub, or unsubscribe. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] dsmiley commented on issue #1009: SOLR-13926
dsmiley commented on issue #1009: SOLR-13926 URL: https://github.com/apache/lucene-solr/pull/1009#issuecomment-553642504 Cool. In addition, I find the method contracts to the methods on DocRouter and it's subclasses highly unspecified (no javadocs or don't specify this). In particular, it's not evident when a id/shardKey/routeKey param is assumed to be the prefix, or assumed to be the prefix plus exclamation, or prefix+!+theRest or some combinations only or whatever. It makes reasoning about what a method should do difficult. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8920) Reduce size of FSTs due to use of direct-addressing encoding
[ https://issues.apache.org/jira/browse/LUCENE-8920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16973729#comment-16973729 ] ASF subversion and git services commented on LUCENE-8920: - Commit 068b6babac38998f10a52b1776abca6218d970e1 in lucene-solr's branch refs/heads/master from Bruno Roustant [ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=068b6ba ] LUCENE-8920: Reduce the memory used by direct addressing of arcs (#980) > Reduce size of FSTs due to use of direct-addressing encoding > - > > Key: LUCENE-8920 > URL: https://issues.apache.org/jira/browse/LUCENE-8920 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Michael Sokolov >Priority: Minor > Attachments: TestTermsDictRamBytesUsed.java > > Time Spent: 4.5h > Remaining Estimate: 0h > > Some data can lead to worst-case ~4x RAM usage due to this optimization. > Several ideas were suggested to combat this on the mailing list: > bq. I think we can improve thesituation here by tracking, per-FST instance, > the size increase we're seeing while building (or perhaps do a preliminary > pass before building) in order to decide whether to apply the encoding. > bq. we could also make the encoding a bit more efficient. For instance I > noticed that arc metadata is pretty large in some cases (in the 10-20 bytes) > which make gaps very costly. Associating each label with a dense id and > having an intermediate lookup, ie. lookup label -> id and then id->arc offset > instead of doing label->arc directly could save a lot of space in some cases? > Also it seems that we are repeating the label in the arc metadata when > array-with-gaps is used, even though it shouldn't be necessary since the > label is implicit from the address? -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] jpountz merged pull request #980: LUCENE-8920: Reduce the memory used by direct addressing of arcs
jpountz merged pull request #980: LUCENE-8920: Reduce the memory used by direct addressing of arcs URL: https://github.com/apache/lucene-solr/pull/980 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Created] (LUCENE-9047) Directory APIs should be little endian
Adrien Grand created LUCENE-9047: Summary: Directory APIs should be little endian Key: LUCENE-9047 URL: https://issues.apache.org/jira/browse/LUCENE-9047 Project: Lucene - Core Issue Type: Improvement Reporter: Adrien Grand We started discussing this on LUCENE-9027. It's a shame that we need to keep reversing the order of bytes all the time because our APIs are big endian while the vast majority of architectures are little endian. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9029) Deprecate SloppyMath toRadians/toDegrees in favor of Java Math
[ https://issues.apache.org/jira/browse/LUCENE-9029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16973709#comment-16973709 ] ASF subversion and git services commented on LUCENE-9029: - Commit c1ac14645409b6652c0c48858c2522c49c941be1 in lucene-solr's branch refs/heads/master from Adrien Grand [ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=c1ac146 ] LUCENE-9029: Deprecate SloppyMath toRadians/toDegrees in favor of Java Math > Deprecate SloppyMath toRadians/toDegrees in favor of Java Math > -- > > Key: LUCENE-9029 > URL: https://issues.apache.org/jira/browse/LUCENE-9029 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Jack Conradson >Priority: Trivial > Attachments: LUCENE-9029.patch, LUCENE-9029.patch > > > This change follows a TODO left in SloppyMath to remove toRadians/toDegrees > since from Java 9 forward Math toRadians/toDegrees is now identical. Since > these methods/constants are public, deprecation messages are added to each > one. Internally, in Lucene, all instances of the SloppyMath versions are > replaced with the standard Java Math versions. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-13909) Everything about CheckBackupStatus is broken
[
https://issues.apache.org/jira/browse/SOLR-13909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16973708#comment-16973708
]
Chris M. Hostetter commented on SOLR-13909:
---
Sigh this is more complex/worse then i originally realized. We have some
test code that tries to mix "named backups" with "unnamed" backups (either
using {{command=backup}} w/o a name param, or using things like {{commit}} ) ... and when dealing with the replication
handler's status info it's basically impossible to tell *which* unnamed backup
the most recent success relates to.
> Everything about CheckBackupStatus is broken
>
>
> Key: SOLR-13909
> URL: https://issues.apache.org/jira/browse/SOLR-13909
> Project: Solr
> Issue Type: Test
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Chris M. Hostetter
>Assignee: Chris M. Hostetter
>Priority: Major
>
> While working on SOLR-13872 I tried to take advantage of the existing
> {{CheckBackupStatus}} helper class and discovered that just about every
> aspect of this class is broken and needs fixed:
> * doesn't use SolrClient, pulls out it's URL to do a bare HTTP request
> * hardcoded assumption of xml - but doesn't parse it just tries to match
> regexes against it
> * almost every usage of this class follows the same broken "loop" pattern
> that garuntees the test will sleep more then it needs to even after
> {{CheckBackupStatus}} thinks the backup is a success...
> {code:java}
> CheckBackupStatus checkBackupStatus = new CheckBackupStatus(...);
> while (!checkBackupStatus.success) {
> checkBackupStatus.fetchStatus();
> Thread.sleep(1000);
> }
> {code}
> * the 3 arg constructor is broken both in design and in implementation:
> ** it appears to be useful for checking that a _new_ backup has succeeded
> after a {{lastBackupTimestamp}} from some previously successful check
> ** in reality it only ever reports {{success}} if it's status check
> indicates the most recent backup has the exact {{.equals()}} time stamp as
> {{lastBackupTimestamp}}
> ** *AND THESE TIMESTAMPS ONLY HAVE MINUTE PRECISION*
> ** As far as i can tell, the only the tests using the 3 arg version ever
> pass is because of the broken loop pattern:
> *** they ask for the status so quick, it's either already done (during the
> same wall clock minute) or it's not done yet and they re-read the "old"
> status (with the old matching timestamp)
> *** either way, the test then sleeps for a second giving the "new" backup
> enough time to actually finish
> ** AFAICT if the System clock ticks over to a new minute in between these
> sleep calls, the test is garunteed to loop forever!
>
> Everything about this class needs to die and be replaced with something
> better.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9027) SIMD-based decoding of postings lists
[
https://issues.apache.org/jira/browse/LUCENE-9027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16973707#comment-16973707
]
Yonik Seeley commented on LUCENE-9027:
--
Yep, going little-endian is the right choice. big-endian has been dead for a
while, and is very unlikely to be revived.
> SIMD-based decoding of postings lists
> -
>
> Key: LUCENE-9027
> URL: https://issues.apache.org/jira/browse/LUCENE-9027
> Project: Lucene - Core
> Issue Type: Improvement
>Reporter: Adrien Grand
>Priority: Minor
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> [~rcmuir] has been mentioning the idea for quite some time that we might be
> able to write the decoding logic in such a way that Java would use SIMD
> instructions. More recently [~paul.masurel] wrote a [blog
> post|https://fulmicoton.com/posts/bitpacking/] that raises the point that
> Lucene could still do decode multiple ints at once in a single instruction by
> packing two ints in a long and we've had some discussions about what we could
> try in Lucene to speed up the decoding of postings. This made me want to look
> a bit deeper at what we could do.
> Our current decoding logic reads data in a byte[] and decodes packed integers
> from it. Unfortunately it doesn't make use of SIMD instructions and looks
> like
> [this|https://github.com/jpountz/decode-128-ints-benchmark/blob/master/src/main/java/jpountz/NaiveByteDecoder.java].
> I confirmed by looking at the generated assembly that if I take an array of
> integers and shift them all by the same number of bits then Java will use
> SIMD instructions to shift multiple integers at once. This led me to writing
> this
> [implementation|https://github.com/jpountz/decode-128-ints-benchmark/blob/master/src/main/java/jpountz/SimpleSIMDDecoder.java]
> that tries as much as possible to shift long sequences of ints by the same
> number of bits to speed up decoding. It is indeed faster than the current
> logic we have, up to about 2x faster for some numbers of bits per value.
> Currently the best
> [implementation|https://github.com/jpountz/decode-128-ints-benchmark/blob/master/src/main/java/jpountz/SIMDDecoder.java]
> I've been able to come up with combines the above idea with the idea that
> Paul mentioned in his blog that consists of emulating SIMD from Java by
> packing multiple integers into a long: 2 ints, 4 shorts or 8 bytes. It is a
> bit harder to read but gives another speedup on top of the above
> implementation.
> I have a [JMH
> benchmark|https://github.com/jpountz/decode-128-ints-benchmark/] available in
> case someone would like to play with this and maybe even come up with an even
> faster implementation. It is 2-2.5x faster than our current implementation
> for most numbers of bits per value. I'm copying results here:
> {noformat}
> * `readLongs` just reads 2*bitsPerValue longs from the ByteBuffer, it serves
> as
>a baseline.
> * `decodeNaiveFromBytes` reads a byte[] and decodes from it. This is what the
>current Lucene codec does.
> * `decodeNaiveFromLongs` decodes from longs on the fly.
> * `decodeSimpleSIMD` is a simple implementation that relies on how Java
>recognizes some patterns and uses SIMD instructions.
> * `decodeSIMD` is a more complex implementation that both relies on the C2
>compiler to generate SIMD instructions and encodes 8 bytes, 4 shorts or
>2 ints in a long in order to decompress multiple values at once.
> Benchmark (bitsPerValue) (byteOrder)
> Mode Cnt Score Error Units
> PackedIntsDecodeBenchmark.decodeNaiveFromBytes 1 LE
> thrpt5 12.912 ± 0.393 ops/us
> PackedIntsDecodeBenchmark.decodeNaiveFromBytes 1 BE
> thrpt5 12.862 ± 0.395 ops/us
> PackedIntsDecodeBenchmark.decodeNaiveFromBytes 2 LE
> thrpt5 13.040 ± 1.162 ops/us
> PackedIntsDecodeBenchmark.decodeNaiveFromBytes 2 BE
> thrpt5 13.027 ± 0.270 ops/us
> PackedIntsDecodeBenchmark.decodeNaiveFromBytes 3 LE
> thrpt5 12.409 ± 0.637 ops/us
> PackedIntsDecodeBenchmark.decodeNaiveFromBytes 3 BE
> thrpt5 12.268 ± 0.947 ops/us
> PackedIntsDecodeBenchmark.decodeNaiveFromBytes 4 LE
> thrpt5 14.177 ± 2.263 ops/us
> PackedIntsDecodeBenchmark.decodeNaiveFromBytes 4 BE
> thrpt5 11.457 ± 0.150 ops/us
> PackedIntsDecodeBenchmark.decodeNaiveFromBytes 5 LE
> thrpt5 10.988 ± 1.179 ops/us
> PackedIntsDecodeBenchmark.decodeNaiveFromBytes 5 BE
> thrpt5 11.226 ± 0.088 ops/us
> PackedIntsDecodeBenchmark.decodeNaiveFromBytes 6
[jira] [Commented] (LUCENE-9027) SIMD-based decoding of postings lists
[
https://issues.apache.org/jira/browse/LUCENE-9027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16973691#comment-16973691
]
Adrien Grand commented on LUCENE-9027:
--
Following a suggestion by [~rcmuir] I specialized the patch for little-endian
architectures. Big-endian architectures will get a slowdown due to the need to
do a Long#reverseBytes on all read longs, but this probably doesn't matter much
and helps simplify DataInput.
> SIMD-based decoding of postings lists
> -
>
> Key: LUCENE-9027
> URL: https://issues.apache.org/jira/browse/LUCENE-9027
> Project: Lucene - Core
> Issue Type: Improvement
>Reporter: Adrien Grand
>Priority: Minor
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> [~rcmuir] has been mentioning the idea for quite some time that we might be
> able to write the decoding logic in such a way that Java would use SIMD
> instructions. More recently [~paul.masurel] wrote a [blog
> post|https://fulmicoton.com/posts/bitpacking/] that raises the point that
> Lucene could still do decode multiple ints at once in a single instruction by
> packing two ints in a long and we've had some discussions about what we could
> try in Lucene to speed up the decoding of postings. This made me want to look
> a bit deeper at what we could do.
> Our current decoding logic reads data in a byte[] and decodes packed integers
> from it. Unfortunately it doesn't make use of SIMD instructions and looks
> like
> [this|https://github.com/jpountz/decode-128-ints-benchmark/blob/master/src/main/java/jpountz/NaiveByteDecoder.java].
> I confirmed by looking at the generated assembly that if I take an array of
> integers and shift them all by the same number of bits then Java will use
> SIMD instructions to shift multiple integers at once. This led me to writing
> this
> [implementation|https://github.com/jpountz/decode-128-ints-benchmark/blob/master/src/main/java/jpountz/SimpleSIMDDecoder.java]
> that tries as much as possible to shift long sequences of ints by the same
> number of bits to speed up decoding. It is indeed faster than the current
> logic we have, up to about 2x faster for some numbers of bits per value.
> Currently the best
> [implementation|https://github.com/jpountz/decode-128-ints-benchmark/blob/master/src/main/java/jpountz/SIMDDecoder.java]
> I've been able to come up with combines the above idea with the idea that
> Paul mentioned in his blog that consists of emulating SIMD from Java by
> packing multiple integers into a long: 2 ints, 4 shorts or 8 bytes. It is a
> bit harder to read but gives another speedup on top of the above
> implementation.
> I have a [JMH
> benchmark|https://github.com/jpountz/decode-128-ints-benchmark/] available in
> case someone would like to play with this and maybe even come up with an even
> faster implementation. It is 2-2.5x faster than our current implementation
> for most numbers of bits per value. I'm copying results here:
> {noformat}
> * `readLongs` just reads 2*bitsPerValue longs from the ByteBuffer, it serves
> as
>a baseline.
> * `decodeNaiveFromBytes` reads a byte[] and decodes from it. This is what the
>current Lucene codec does.
> * `decodeNaiveFromLongs` decodes from longs on the fly.
> * `decodeSimpleSIMD` is a simple implementation that relies on how Java
>recognizes some patterns and uses SIMD instructions.
> * `decodeSIMD` is a more complex implementation that both relies on the C2
>compiler to generate SIMD instructions and encodes 8 bytes, 4 shorts or
>2 ints in a long in order to decompress multiple values at once.
> Benchmark (bitsPerValue) (byteOrder)
> Mode Cnt Score Error Units
> PackedIntsDecodeBenchmark.decodeNaiveFromBytes 1 LE
> thrpt5 12.912 ± 0.393 ops/us
> PackedIntsDecodeBenchmark.decodeNaiveFromBytes 1 BE
> thrpt5 12.862 ± 0.395 ops/us
> PackedIntsDecodeBenchmark.decodeNaiveFromBytes 2 LE
> thrpt5 13.040 ± 1.162 ops/us
> PackedIntsDecodeBenchmark.decodeNaiveFromBytes 2 BE
> thrpt5 13.027 ± 0.270 ops/us
> PackedIntsDecodeBenchmark.decodeNaiveFromBytes 3 LE
> thrpt5 12.409 ± 0.637 ops/us
> PackedIntsDecodeBenchmark.decodeNaiveFromBytes 3 BE
> thrpt5 12.268 ± 0.947 ops/us
> PackedIntsDecodeBenchmark.decodeNaiveFromBytes 4 LE
> thrpt5 14.177 ± 2.263 ops/us
> PackedIntsDecodeBenchmark.decodeNaiveFromBytes 4 BE
> thrpt5 11.457 ± 0.150 ops/us
> PackedIntsDecodeBenchmark.decodeNaiveFromBytes 5 LE
> thrpt5 10.988 ± 1.179 ops/us
>
[jira] [Commented] (LUCENE-9020) Find a way to publish Solr RefGuide without checking into git
[ https://issues.apache.org/jira/browse/LUCENE-9020?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16973684#comment-16973684 ] Jan Høydahl commented on LUCENE-9020: - [~uschindler]? > Find a way to publish Solr RefGuide without checking into git > - > > Key: LUCENE-9020 > URL: https://issues.apache.org/jira/browse/LUCENE-9020 > Project: Lucene - Core > Issue Type: Sub-task >Reporter: Jan Høydahl >Priority: Major > > Currently we check in all versions of RefGuide (hundreds of small html files) > into svn to publish as part of the site. With new site we should find a > smoother way to do this. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9015) Configure branches, auto build and auto stage/publish
[ https://issues.apache.org/jira/browse/LUCENE-9015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16973683#comment-16973683 ] Jan Høydahl commented on LUCENE-9015: - Ok, INFRA had a few bugs in the builedbot setup, they are on the case now and hopefully the site will eventually build :) Looks like we are one of the first projects to actually stress their env with real-life MarkDown and structures.. > Configure branches, auto build and auto stage/publish > - > > Key: LUCENE-9015 > URL: https://issues.apache.org/jira/browse/LUCENE-9015 > Project: Lucene - Core > Issue Type: Sub-task >Reporter: Jan Høydahl >Assignee: Jan Høydahl >Priority: Major > Time Spent: 20m > Remaining Estimate: 0h > > Commit to master should build and publish the staging site > Find a simple way to trigger publishing of main site from staging -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] epugh opened a new pull request #1010: Update documentation to fix SOLR-13927
epugh opened a new pull request #1010: Update documentation to fix SOLR-13927 URL: https://github.com/apache/lucene-solr/pull/1010 # Description Fix api v2 sample urls to be solrcloud instead of standalone. # Solution text change. # Tests n/a # Checklist Please review the following and check all that apply: - [ X] I have reviewed the guidelines for [How to Contribute](https://wiki.apache.org/solr/HowToContribute) and my code conforms to the standards described there to the best of my ability. - [ X] I have created a Jira issue and added the issue ID to my pull request title. - [ X] I am authorized to contribute this code to the ASF and have removed any code I do not have a license to distribute. - [ X] I have given Solr maintainers [access](https://help.github.com/en/articles/allowing-changes-to-a-pull-request-branch-created-from-a-fork) to contribute to my PR branch. (optional but recommended) - [ X] I have developed this patch against the `master` branch. - [ ] I have run `ant precommit` and the appropriate test suite. - [ ] I have added tests for my changes. - [ X] I have added documentation for the [Ref Guide](https://github.com/apache/lucene-solr/tree/master/solr/solr-ref-guide) (for Solr changes only). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Created] (SOLR-13927) Schema API v2 Sample URLS refers to single node not solrcloud mode..
David Eric Pugh created SOLR-13927:
--
Summary: Schema API v2 Sample URLS refers to single node not
solrcloud mode..
Key: SOLR-13927
URL: https://issues.apache.org/jira/browse/SOLR-13927
Project: Solr
Issue Type: Improvement
Security Level: Public (Default Security Level. Issues are Public)
Components: documentation, Schema and Analysis
Affects Versions: 8.3
Reporter: David Eric Pugh
Fix For: 8.4
The v2 sample URLS on https://lucene.apache.org/solr/guide/8_1/schema-api.html
reference modifying `/api/cores/{corename}/schema`, but if you are in
SolrCloud, you would send the request to
`/api/collections/{collectionname}/schema`.
We should reference the solrcloud mode in the sample docs, not standalone solr,
as that is probably more common usecase for modifying schema.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] millerjeff0 commented on issue #1009: SOLR-13926
millerjeff0 commented on issue #1009: SOLR-13926 URL: https://github.com/apache/lucene-solr/pull/1009#issuecomment-553582740 @yonik Can you take a look This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] millerjeff0 opened a new pull request #1009: SOLR-13926
millerjeff0 opened a new pull request #1009: SOLR-13926 URL: https://github.com/apache/lucene-solr/pull/1009 # Description Trying to improve documentation around mystical classes # Solution javadoc # Tests Committers eyes # Checklist Please review the following and check all that apply: - [ x] I have reviewed the guidelines for [How to Contribute](https://wiki.apache.org/solr/HowToContribute) and my code conforms to the standards described there to the best of my ability. - [x ] I have created a Jira issue and added the issue ID to my pull request title. - [x ] I am authorized to contribute this code to the ASF and have removed any code I do not have a license to distribute. - [ ] I have given Solr maintainers [access](https://help.github.com/en/articles/allowing-changes-to-a-pull-request-branch-created-from-a-fork) to contribute to my PR branch. (optional but recommended) - [x ] I have developed this patch against the `master` branch. - [x ] I have run `ant precommit` and the appropriate test suite. - [Z ] I have added tests for my changes. - [X ] I have added documentation for the [Ref Guide](https://github.com/apache/lucene-solr/tree/master/solr/solr-ref-guide) (for Solr changes only). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] jpountz commented on issue #973: LUCENE-9027: Use SIMD instructions to decode postings.
jpountz commented on issue #973: LUCENE-9027: Use SIMD instructions to decode postings. URL: https://github.com/apache/lucene-solr/pull/973#issuecomment-553579095 @mikemccand I changed ByteBufferIndexInput to create the LongBuffer views in order to hopefully address your concern. I also removed write support from the Lucene50 postings format and moved it to backward-codecs. I couldn't do it for Completion50 because it would require adding a dependency to suggest to backward-codecs so I left it in the suggest module. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9015) Configure branches, auto build and auto stage/publish
[
https://issues.apache.org/jira/browse/LUCENE-9015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16973642#comment-16973642
]
Jan Høydahl commented on LUCENE-9015:
-
{quote}[Daniel Gruno|https://app.slack.com/team/UBY0LUX0D][5:46
PM|https://the-asf.slack.com/archives/CBX4TSBQ8/p1573663619487600]
welp, lucene-site build works now, yay
{quote}
However, only the images were put in the new branch:
[https://github.com/apache/lucene-site/tree/asf-staging]
Will followup...
> Configure branches, auto build and auto stage/publish
> -
>
> Key: LUCENE-9015
> URL: https://issues.apache.org/jira/browse/LUCENE-9015
> Project: Lucene - Core
> Issue Type: Sub-task
>Reporter: Jan Høydahl
>Assignee: Jan Høydahl
>Priority: Major
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Commit to master should build and publish the staging site
> Find a simple way to trigger publishing of main site from staging
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Created] (SOLR-13926) Document CompositeIDRouter.java
Jeff Miller created SOLR-13926: -- Summary: Document CompositeIDRouter.java Key: SOLR-13926 URL: https://issues.apache.org/jira/browse/SOLR-13926 Project: Solr Issue Type: Improvement Security Level: Public (Default Security Level. Issues are Public) Components: SolrCloud Affects Versions: 8.3 Reporter: Jeff Miller Fix For: master (9.0) CompositeIDRouter documentation is lacking, I would like to take a shot at updating the java doc so potential uses can be understood by client. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-13921) Processing UpdateRequest with delegation token throws NullPointerException
[
https://issues.apache.org/jira/browse/SOLR-13921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16973577#comment-16973577
]
Kevin Risden commented on SOLR-13921:
-
Thanks [~warper]
> Processing UpdateRequest with delegation token throws NullPointerException
> --
>
> Key: SOLR-13921
> URL: https://issues.apache.org/jira/browse/SOLR-13921
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
> Components: SolrJ
>Affects Versions: 7.4, 7.5, 7.6, 7.7, 7.7.1, 7.7.2, 8.0, 8.1, 8.2, 8.3
>Reporter: Istvan Farkas
>Assignee: Kevin Risden
>Priority: Minor
> Fix For: 8.4
>
> Attachments: SOLR-13921.patch
>
>
> When sending UpdateRequests with delegation tokens to Solr using SolrJ, the
> createMethod of DelegationTokenHttpSolrClient will throw a
> NullPointerException:
>
> {code:java}
> [junit4] ERROR 3.41s |
> TestSolrCloudWithDelegationTokens.testDelegationTokenSolrClientWithUpdateRequests
> <<<
>[junit4]> Throwable #1: java.lang.NullPointerException
>[junit4]> at
> __randomizedtesting.SeedInfo.seed([B9AE8E4E0CDF1B3D:DBA0B722C813061D]:0)
>[junit4]> at
> org.apache.solr.client.solrj.impl.DelegationTokenHttpSolrClient.createMethod(DelegationTokenHttpSolrClient.java:93)
>[junit4]> at
> org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:258)
>[junit4]> at
> org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:249)
>[junit4]> at
> org.apache.solr.cloud.TestSolrCloudWithDelegationTokens.doSolrRequest(TestSolrCloudWithDelegationTokens.java:246)
>[junit4]> at
> org.apache.solr.cloud.TestSolrCloudWithDelegationTokens.testDelegationTokenSolrClientWithUpdateRequests(TestSolrCloudWithDelegationTokens.java:477)
>[junit4]> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>[junit4]> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>[junit4]> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>[junit4]> at
> java.base/java.lang.reflect.Method.invoke(Method.java:566)
>[junit4]> at java.base/java.lang.Thread.run(Thread.java:834)
> {code}
> This happens to all SolrJ clients including the Spark Crunch indexer which
> use Delegation Tokens and do not specify commit / optimize commands.
> The cause seems to be a missing null check before dereferencing 'params'. The
> intention of the code seems to be verifying if the delegation token is passed
> a parameter of the SolrRequest (which is not supported), however the check
> fails with NPE if the request has no params at all. For update requests which
> do commit or optimize, the setCommand method initializes the params so no
> NPE is thrown.
> {code}
> @Override
> protected HttpRequestBase createMethod(final SolrRequest request, String
> collection) throws IOException, SolrServerException {
> SolrParams params = request.getParams();
> if (params.getParams(DELEGATION_TOKEN_PARAM) != null) {
> throw new IllegalArgumentException(DELEGATION_TOKEN_PARAM + " parameter
> not supported");
> }
> return super.createMethod(request, collection);
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-13921) Processing UpdateRequest with delegation token throws NullPointerException
[
https://issues.apache.org/jira/browse/SOLR-13921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16973574#comment-16973574
]
ASF subversion and git services commented on SOLR-13921:
Commit a1777b540bd7eee2838c0ef324135eec26095450 in lucene-solr's branch
refs/heads/branch_8x from Istvan Farkas
[ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=a1777b5 ]
SOLR-13921: Processing UpdateRequest with delegation token throws
NullPointerException
Signed-off-by: Kevin Risden
> Processing UpdateRequest with delegation token throws NullPointerException
> --
>
> Key: SOLR-13921
> URL: https://issues.apache.org/jira/browse/SOLR-13921
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
> Components: SolrJ
>Affects Versions: 7.4, 7.5, 7.6, 7.7, 7.7.1, 7.7.2, 8.0, 8.1, 8.2, 8.3
>Reporter: Istvan Farkas
>Assignee: Kevin Risden
>Priority: Minor
> Fix For: 8.4
>
> Attachments: SOLR-13921.patch
>
>
> When sending UpdateRequests with delegation tokens to Solr using SolrJ, the
> createMethod of DelegationTokenHttpSolrClient will throw a
> NullPointerException:
>
> {code:java}
> [junit4] ERROR 3.41s |
> TestSolrCloudWithDelegationTokens.testDelegationTokenSolrClientWithUpdateRequests
> <<<
>[junit4]> Throwable #1: java.lang.NullPointerException
>[junit4]> at
> __randomizedtesting.SeedInfo.seed([B9AE8E4E0CDF1B3D:DBA0B722C813061D]:0)
>[junit4]> at
> org.apache.solr.client.solrj.impl.DelegationTokenHttpSolrClient.createMethod(DelegationTokenHttpSolrClient.java:93)
>[junit4]> at
> org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:258)
>[junit4]> at
> org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:249)
>[junit4]> at
> org.apache.solr.cloud.TestSolrCloudWithDelegationTokens.doSolrRequest(TestSolrCloudWithDelegationTokens.java:246)
>[junit4]> at
> org.apache.solr.cloud.TestSolrCloudWithDelegationTokens.testDelegationTokenSolrClientWithUpdateRequests(TestSolrCloudWithDelegationTokens.java:477)
>[junit4]> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>[junit4]> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>[junit4]> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>[junit4]> at
> java.base/java.lang.reflect.Method.invoke(Method.java:566)
>[junit4]> at java.base/java.lang.Thread.run(Thread.java:834)
> {code}
> This happens to all SolrJ clients including the Spark Crunch indexer which
> use Delegation Tokens and do not specify commit / optimize commands.
> The cause seems to be a missing null check before dereferencing 'params'. The
> intention of the code seems to be verifying if the delegation token is passed
> a parameter of the SolrRequest (which is not supported), however the check
> fails with NPE if the request has no params at all. For update requests which
> do commit or optimize, the setCommand method initializes the params so no
> NPE is thrown.
> {code}
> @Override
> protected HttpRequestBase createMethod(final SolrRequest request, String
> collection) throws IOException, SolrServerException {
> SolrParams params = request.getParams();
> if (params.getParams(DELEGATION_TOKEN_PARAM) != null) {
> throw new IllegalArgumentException(DELEGATION_TOKEN_PARAM + " parameter
> not supported");
> }
> return super.createMethod(request, collection);
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (SOLR-13921) Processing UpdateRequest with delegation token throws NullPointerException
[
https://issues.apache.org/jira/browse/SOLR-13921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kevin Risden updated SOLR-13921:
Resolution: Fixed
Status: Resolved (was: Patch Available)
> Processing UpdateRequest with delegation token throws NullPointerException
> --
>
> Key: SOLR-13921
> URL: https://issues.apache.org/jira/browse/SOLR-13921
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
> Components: SolrJ
>Affects Versions: 7.4, 7.5, 7.6, 7.7, 7.7.1, 7.7.2, 8.0, 8.1, 8.2, 8.3
>Reporter: Istvan Farkas
>Assignee: Kevin Risden
>Priority: Minor
> Fix For: 8.4
>
> Attachments: SOLR-13921.patch
>
>
> When sending UpdateRequests with delegation tokens to Solr using SolrJ, the
> createMethod of DelegationTokenHttpSolrClient will throw a
> NullPointerException:
>
> {code:java}
> [junit4] ERROR 3.41s |
> TestSolrCloudWithDelegationTokens.testDelegationTokenSolrClientWithUpdateRequests
> <<<
>[junit4]> Throwable #1: java.lang.NullPointerException
>[junit4]> at
> __randomizedtesting.SeedInfo.seed([B9AE8E4E0CDF1B3D:DBA0B722C813061D]:0)
>[junit4]> at
> org.apache.solr.client.solrj.impl.DelegationTokenHttpSolrClient.createMethod(DelegationTokenHttpSolrClient.java:93)
>[junit4]> at
> org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:258)
>[junit4]> at
> org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:249)
>[junit4]> at
> org.apache.solr.cloud.TestSolrCloudWithDelegationTokens.doSolrRequest(TestSolrCloudWithDelegationTokens.java:246)
>[junit4]> at
> org.apache.solr.cloud.TestSolrCloudWithDelegationTokens.testDelegationTokenSolrClientWithUpdateRequests(TestSolrCloudWithDelegationTokens.java:477)
>[junit4]> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>[junit4]> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>[junit4]> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>[junit4]> at
> java.base/java.lang.reflect.Method.invoke(Method.java:566)
>[junit4]> at java.base/java.lang.Thread.run(Thread.java:834)
> {code}
> This happens to all SolrJ clients including the Spark Crunch indexer which
> use Delegation Tokens and do not specify commit / optimize commands.
> The cause seems to be a missing null check before dereferencing 'params'. The
> intention of the code seems to be verifying if the delegation token is passed
> a parameter of the SolrRequest (which is not supported), however the check
> fails with NPE if the request has no params at all. For update requests which
> do commit or optimize, the setCommand method initializes the params so no
> NPE is thrown.
> {code}
> @Override
> protected HttpRequestBase createMethod(final SolrRequest request, String
> collection) throws IOException, SolrServerException {
> SolrParams params = request.getParams();
> if (params.getParams(DELEGATION_TOKEN_PARAM) != null) {
> throw new IllegalArgumentException(DELEGATION_TOKEN_PARAM + " parameter
> not supported");
> }
> return super.createMethod(request, collection);
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Resolved] (SOLR-13898) Non-atomic use of SolrCache get / put
[
https://issues.apache.org/jira/browse/SOLR-13898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrzej Bialecki resolved SOLR-13898.
-
Resolution: Fixed
> Non-atomic use of SolrCache get / put
> -
>
> Key: SOLR-13898
> URL: https://issues.apache.org/jira/browse/SOLR-13898
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
>Affects Versions: 8.3
>Reporter: Andrzej Bialecki
>Assignee: Andrzej Bialecki
>Priority: Major
> Fix For: 8.4
>
> Attachments: SOLR-13898.patch, SOLR-13898.patch, SOLR-13898.patch
>
>
> As pointed out by [~ben.manes] in SOLR-13817 Solr code base in many key
> places uses a similar pattern of non-atomic get / put calls to SolrCache-s.
> In multi-threaded environment this leads to cache misses and additional
> unnecessary computations when competing threads both discover a missing
> value, non-atomically compute it and update the cache.
> Some of these places are known performance bottlenecks where efficient
> caching is especially important, such as {{SolrIndexSearcher}},
> {{SolrDocumentFetcher}}, {{UninvertedField}} and join queries .
> I propose to add {{SolrCache.computeIfAbsent(key, mappingFunction)}} that
> will atomically retrieve existing values or compute and update the cache.
> This will require also changing how the {{SolrCache.get(...)}} is used in
> many components.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-13898) Non-atomic use of SolrCache get / put
[
https://issues.apache.org/jira/browse/SOLR-13898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16973538#comment-16973538
]
ASF subversion and git services commented on SOLR-13898:
Commit 32c3255b938c04a9e25076da5f0ac448b6daa66c in lucene-solr's branch
refs/heads/master from Andrzej Bialecki
[ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=32c3255 ]
SOLR-13898: fix a typo.
> Non-atomic use of SolrCache get / put
> -
>
> Key: SOLR-13898
> URL: https://issues.apache.org/jira/browse/SOLR-13898
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
>Affects Versions: 8.3
>Reporter: Andrzej Bialecki
>Assignee: Andrzej Bialecki
>Priority: Major
> Fix For: 8.4
>
> Attachments: SOLR-13898.patch, SOLR-13898.patch, SOLR-13898.patch
>
>
> As pointed out by [~ben.manes] in SOLR-13817 Solr code base in many key
> places uses a similar pattern of non-atomic get / put calls to SolrCache-s.
> In multi-threaded environment this leads to cache misses and additional
> unnecessary computations when competing threads both discover a missing
> value, non-atomically compute it and update the cache.
> Some of these places are known performance bottlenecks where efficient
> caching is especially important, such as {{SolrIndexSearcher}},
> {{SolrDocumentFetcher}}, {{UninvertedField}} and join queries .
> I propose to add {{SolrCache.computeIfAbsent(key, mappingFunction)}} that
> will atomically retrieve existing values or compute and update the cache.
> This will require also changing how the {{SolrCache.get(...)}} is used in
> many components.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-13898) Non-atomic use of SolrCache get / put
[
https://issues.apache.org/jira/browse/SOLR-13898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16973535#comment-16973535
]
ASF subversion and git services commented on SOLR-13898:
Commit 3c06fbcad242c6b47d1499c8c3e1bdc49ec8f1d8 in lucene-solr's branch
refs/heads/branch_8x from Andrzej Bialecki
[ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=3c06fbc ]
SOLR-13898: Non-atomic use of SolrCache get / put.
> Non-atomic use of SolrCache get / put
> -
>
> Key: SOLR-13898
> URL: https://issues.apache.org/jira/browse/SOLR-13898
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
>Affects Versions: 8.3
>Reporter: Andrzej Bialecki
>Assignee: Andrzej Bialecki
>Priority: Major
> Fix For: 8.4
>
> Attachments: SOLR-13898.patch, SOLR-13898.patch, SOLR-13898.patch
>
>
> As pointed out by [~ben.manes] in SOLR-13817 Solr code base in many key
> places uses a similar pattern of non-atomic get / put calls to SolrCache-s.
> In multi-threaded environment this leads to cache misses and additional
> unnecessary computations when competing threads both discover a missing
> value, non-atomically compute it and update the cache.
> Some of these places are known performance bottlenecks where efficient
> caching is especially important, such as {{SolrIndexSearcher}},
> {{SolrDocumentFetcher}}, {{UninvertedField}} and join queries .
> I propose to add {{SolrCache.computeIfAbsent(key, mappingFunction)}} that
> will atomically retrieve existing values or compute and update the cache.
> This will require also changing how the {{SolrCache.get(...)}} is used in
> many components.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (SOLR-13872) Backup can fail to read index files w/NoSuchFileException during merges (SOLR-11616 regression)
[
https://issues.apache.org/jira/browse/SOLR-13872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris M. Hostetter updated SOLR-13872:
--
Fix Version/s: 8.4
master (9.0)
Resolution: Fixed
Status: Resolved (was: Patch Available)
> Backup can fail to read index files w/NoSuchFileException during merges
> (SOLR-11616 regression)
>
>
> Key: SOLR-13872
> URL: https://issues.apache.org/jira/browse/SOLR-13872
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Chris M. Hostetter
>Assignee: Chris M. Hostetter
>Priority: Major
> Fix For: master (9.0), 8.4
>
> Attachments: SOLR-13872.patch, SOLR-13872.patch, SOLR-13872.patch,
> SOLR-13872.patch, index_churn.pl
>
>
> SOLR-11616 purports to fix a bug in Solr's backup functionality that causes
> 'NoSuchFileException' errors when attempting to backup an index while it is
> undergoing indexing (and segment merging)
> Although SOLR-11616 is marked with "Fix Version: 7.2" it's pretty easy to
> demonstrate that this bug still exists on master, branch_8x, and even in 7.2
> - so it seems less like the current problem is a "regression" and more that
> the original fix didn't work.
>
> The crux of the problem seems to be concurrency bugs in if/how a commit is
> "reserved" before attempting to copy the files in that commit to the backup
> location.
> A possible work around discussed in more depth in the comments below is to
> update {{solrconfig.xml}} to explicitly configure the {{SolrDeletionPolicy}}
> with either the {{maxCommitsToKeep}} or {{maxCommitAge}} options to ensure
> the commits are kept around long enough for the backup to be created.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-13872) Backup can fail to read index files w/NoSuchFileException during merges (SOLR-11616 regression)
[
https://issues.apache.org/jira/browse/SOLR-13872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16973517#comment-16973517
]
ASF subversion and git services commented on SOLR-13872:
Commit 8c12979fddd4fe822a300d1b04d49f93b5106916 in lucene-solr's branch
refs/heads/branch_8x from Chris M. Hostetter
[ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=8c12979 ]
SOLR-13872: Fixed Backup failures due to race conditions in saving/reserving
commit points
(cherry picked from commit 30e55e2b6efc55c04761b80c22a106f4a1115722)
> Backup can fail to read index files w/NoSuchFileException during merges
> (SOLR-11616 regression)
>
>
> Key: SOLR-13872
> URL: https://issues.apache.org/jira/browse/SOLR-13872
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Chris M. Hostetter
>Assignee: Chris M. Hostetter
>Priority: Major
> Attachments: SOLR-13872.patch, SOLR-13872.patch, SOLR-13872.patch,
> SOLR-13872.patch, index_churn.pl
>
>
> SOLR-11616 purports to fix a bug in Solr's backup functionality that causes
> 'NoSuchFileException' errors when attempting to backup an index while it is
> undergoing indexing (and segment merging)
> Although SOLR-11616 is marked with "Fix Version: 7.2" it's pretty easy to
> demonstrate that this bug still exists on master, branch_8x, and even in 7.2
> - so it seems less like the current problem is a "regression" and more that
> the original fix didn't work.
>
> The crux of the problem seems to be concurrency bugs in if/how a commit is
> "reserved" before attempting to copy the files in that commit to the backup
> location.
> A possible work around discussed in more depth in the comments below is to
> update {{solrconfig.xml}} to explicitly configure the {{SolrDeletionPolicy}}
> with either the {{maxCommitsToKeep}} or {{maxCommitAge}} options to ensure
> the commits are kept around long enough for the backup to be created.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Created] (LUCENE-9046) Fix wrong example in Javadoc of TermInSetQuery
Namgyu Kim created LUCENE-9046:
--
Summary: Fix wrong example in Javadoc of TermInSetQuery
Key: LUCENE-9046
URL: https://issues.apache.org/jira/browse/LUCENE-9046
Project: Lucene - Core
Issue Type: Improvement
Reporter: Namgyu Kim
Assignee: Namgyu Kim
Fix For: 8.x, master (9.0)
There is a wrong example in Javadoc of TermInSetQuery.
This patch will be merged to the master and 8.x branch.
Before
{code:java}
Query q1 = new TermInSetQuery(new Term("field", "foo"), new Term("field",
"bar"));
BooleanQuery bq = new BooleanQuery();
bq.add(new TermQuery(new Term("field", "foo")), Occur.SHOULD);
bq.add(new TermQuery(new Term("field", "bar")), Occur.SHOULD);
Query q2 = new ConstantScoreQuery(bq);
{code}
After
{code:java}
Query q1 = new TermInSetQuery("field", new BytesRef("foo"), new
BytesRef("bar"));
BooleanQuery bq = new BooleanQuery();
bq.add(new TermQuery(new Term("field", "foo")), Occur.SHOULD);
bq.add(new TermQuery(new Term("field", "bar")), Occur.SHOULD);
Query q2 = new ConstantScoreQuery(bq);
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (SOLR-13921) Processing UpdateRequest with delegation token throws NullPointerException
[
https://issues.apache.org/jira/browse/SOLR-13921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kevin Risden updated SOLR-13921:
Fix Version/s: 8.4
> Processing UpdateRequest with delegation token throws NullPointerException
> --
>
> Key: SOLR-13921
> URL: https://issues.apache.org/jira/browse/SOLR-13921
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
> Components: SolrJ
>Affects Versions: 7.4, 7.5, 7.6, 7.7, 7.7.1, 7.7.2, 8.0, 8.1, 8.2, 8.3
>Reporter: Istvan Farkas
>Assignee: Kevin Risden
>Priority: Minor
> Fix For: 8.4
>
> Attachments: SOLR-13921.patch
>
>
> When sending UpdateRequests with delegation tokens to Solr using SolrJ, the
> createMethod of DelegationTokenHttpSolrClient will throw a
> NullPointerException:
>
> {code:java}
> [junit4] ERROR 3.41s |
> TestSolrCloudWithDelegationTokens.testDelegationTokenSolrClientWithUpdateRequests
> <<<
>[junit4]> Throwable #1: java.lang.NullPointerException
>[junit4]> at
> __randomizedtesting.SeedInfo.seed([B9AE8E4E0CDF1B3D:DBA0B722C813061D]:0)
>[junit4]> at
> org.apache.solr.client.solrj.impl.DelegationTokenHttpSolrClient.createMethod(DelegationTokenHttpSolrClient.java:93)
>[junit4]> at
> org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:258)
>[junit4]> at
> org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:249)
>[junit4]> at
> org.apache.solr.cloud.TestSolrCloudWithDelegationTokens.doSolrRequest(TestSolrCloudWithDelegationTokens.java:246)
>[junit4]> at
> org.apache.solr.cloud.TestSolrCloudWithDelegationTokens.testDelegationTokenSolrClientWithUpdateRequests(TestSolrCloudWithDelegationTokens.java:477)
>[junit4]> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>[junit4]> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>[junit4]> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>[junit4]> at
> java.base/java.lang.reflect.Method.invoke(Method.java:566)
>[junit4]> at java.base/java.lang.Thread.run(Thread.java:834)
> {code}
> This happens to all SolrJ clients including the Spark Crunch indexer which
> use Delegation Tokens and do not specify commit / optimize commands.
> The cause seems to be a missing null check before dereferencing 'params'. The
> intention of the code seems to be verifying if the delegation token is passed
> a parameter of the SolrRequest (which is not supported), however the check
> fails with NPE if the request has no params at all. For update requests which
> do commit or optimize, the setCommand method initializes the params so no
> NPE is thrown.
> {code}
> @Override
> protected HttpRequestBase createMethod(final SolrRequest request, String
> collection) throws IOException, SolrServerException {
> SolrParams params = request.getParams();
> if (params.getParams(DELEGATION_TOKEN_PARAM) != null) {
> throw new IllegalArgumentException(DELEGATION_TOKEN_PARAM + " parameter
> not supported");
> }
> return super.createMethod(request, collection);
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-13921) Processing UpdateRequest with delegation token throws NullPointerException
[
https://issues.apache.org/jira/browse/SOLR-13921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16973483#comment-16973483
]
ASF subversion and git services commented on SOLR-13921:
Commit 21a54c4bc70bff7db69ca993e7b4426167ce26a9 in lucene-solr's branch
refs/heads/master from Istvan Farkas
[ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=21a54c4 ]
SOLR-13921: Processing UpdateRequest with delegation token throws
NullPointerException
Signed-off-by: Kevin Risden
> Processing UpdateRequest with delegation token throws NullPointerException
> --
>
> Key: SOLR-13921
> URL: https://issues.apache.org/jira/browse/SOLR-13921
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
> Components: SolrJ
>Affects Versions: 7.4, 7.5, 7.6, 7.7, 7.7.1, 7.7.2, 8.0, 8.1, 8.2, 8.3
>Reporter: Istvan Farkas
>Assignee: Kevin Risden
>Priority: Minor
> Attachments: SOLR-13921.patch
>
>
> When sending UpdateRequests with delegation tokens to Solr using SolrJ, the
> createMethod of DelegationTokenHttpSolrClient will throw a
> NullPointerException:
>
> {code:java}
> [junit4] ERROR 3.41s |
> TestSolrCloudWithDelegationTokens.testDelegationTokenSolrClientWithUpdateRequests
> <<<
>[junit4]> Throwable #1: java.lang.NullPointerException
>[junit4]> at
> __randomizedtesting.SeedInfo.seed([B9AE8E4E0CDF1B3D:DBA0B722C813061D]:0)
>[junit4]> at
> org.apache.solr.client.solrj.impl.DelegationTokenHttpSolrClient.createMethod(DelegationTokenHttpSolrClient.java:93)
>[junit4]> at
> org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:258)
>[junit4]> at
> org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:249)
>[junit4]> at
> org.apache.solr.cloud.TestSolrCloudWithDelegationTokens.doSolrRequest(TestSolrCloudWithDelegationTokens.java:246)
>[junit4]> at
> org.apache.solr.cloud.TestSolrCloudWithDelegationTokens.testDelegationTokenSolrClientWithUpdateRequests(TestSolrCloudWithDelegationTokens.java:477)
>[junit4]> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>[junit4]> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>[junit4]> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>[junit4]> at
> java.base/java.lang.reflect.Method.invoke(Method.java:566)
>[junit4]> at java.base/java.lang.Thread.run(Thread.java:834)
> {code}
> This happens to all SolrJ clients including the Spark Crunch indexer which
> use Delegation Tokens and do not specify commit / optimize commands.
> The cause seems to be a missing null check before dereferencing 'params'. The
> intention of the code seems to be verifying if the delegation token is passed
> a parameter of the SolrRequest (which is not supported), however the check
> fails with NPE if the request has no params at all. For update requests which
> do commit or optimize, the setCommand method initializes the params so no
> NPE is thrown.
> {code}
> @Override
> protected HttpRequestBase createMethod(final SolrRequest request, String
> collection) throws IOException, SolrServerException {
> SolrParams params = request.getParams();
> if (params.getParams(DELEGATION_TOKEN_PARAM) != null) {
> throw new IllegalArgumentException(DELEGATION_TOKEN_PARAM + " parameter
> not supported");
> }
> return super.createMethod(request, collection);
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (SOLR-13921) Processing UpdateRequest with delegation token throws NullPointerException
[
https://issues.apache.org/jira/browse/SOLR-13921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kevin Risden updated SOLR-13921:
Affects Version/s: 7.5
7.6
7.7
7.7.1
7.7.2
8.0
8.1
8.2
> Processing UpdateRequest with delegation token throws NullPointerException
> --
>
> Key: SOLR-13921
> URL: https://issues.apache.org/jira/browse/SOLR-13921
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
> Components: SolrJ
>Affects Versions: 7.4, 7.5, 7.6, 7.7, 7.7.1, 7.7.2, 8.0, 8.1, 8.2, 8.3
>Reporter: Istvan Farkas
>Assignee: Kevin Risden
>Priority: Minor
> Attachments: SOLR-13921.patch
>
>
> When sending UpdateRequests with delegation tokens to Solr using SolrJ, the
> createMethod of DelegationTokenHttpSolrClient will throw a
> NullPointerException:
>
> {code:java}
> [junit4] ERROR 3.41s |
> TestSolrCloudWithDelegationTokens.testDelegationTokenSolrClientWithUpdateRequests
> <<<
>[junit4]> Throwable #1: java.lang.NullPointerException
>[junit4]> at
> __randomizedtesting.SeedInfo.seed([B9AE8E4E0CDF1B3D:DBA0B722C813061D]:0)
>[junit4]> at
> org.apache.solr.client.solrj.impl.DelegationTokenHttpSolrClient.createMethod(DelegationTokenHttpSolrClient.java:93)
>[junit4]> at
> org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:258)
>[junit4]> at
> org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:249)
>[junit4]> at
> org.apache.solr.cloud.TestSolrCloudWithDelegationTokens.doSolrRequest(TestSolrCloudWithDelegationTokens.java:246)
>[junit4]> at
> org.apache.solr.cloud.TestSolrCloudWithDelegationTokens.testDelegationTokenSolrClientWithUpdateRequests(TestSolrCloudWithDelegationTokens.java:477)
>[junit4]> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>[junit4]> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>[junit4]> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>[junit4]> at
> java.base/java.lang.reflect.Method.invoke(Method.java:566)
>[junit4]> at java.base/java.lang.Thread.run(Thread.java:834)
> {code}
> This happens to all SolrJ clients including the Spark Crunch indexer which
> use Delegation Tokens and do not specify commit / optimize commands.
> The cause seems to be a missing null check before dereferencing 'params'. The
> intention of the code seems to be verifying if the delegation token is passed
> a parameter of the SolrRequest (which is not supported), however the check
> fails with NPE if the request has no params at all. For update requests which
> do commit or optimize, the setCommand method initializes the params so no
> NPE is thrown.
> {code}
> @Override
> protected HttpRequestBase createMethod(final SolrRequest request, String
> collection) throws IOException, SolrServerException {
> SolrParams params = request.getParams();
> if (params.getParams(DELEGATION_TOKEN_PARAM) != null) {
> throw new IllegalArgumentException(DELEGATION_TOKEN_PARAM + " parameter
> not supported");
> }
> return super.createMethod(request, collection);
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] jpountz commented on a change in pull request #973: LUCENE-9027: Use SIMD instructions to decode postings.
jpountz commented on a change in pull request #973: LUCENE-9027: Use SIMD
instructions to decode postings.
URL: https://github.com/apache/lucene-solr/pull/973#discussion_r345847792
##
File path:
lucene/core/src/java/org/apache/lucene/store/ByteBufferIndexInput.java
##
@@ -63,7 +69,16 @@ public static ByteBufferIndexInput newInstance(String
resourceDescription, ByteB
assert chunkSizePower >= 0 && chunkSizePower <= 30;
assert (length >>> chunkSizePower) < Integer.MAX_VALUE;
}
-
+
+ protected void setCurBuf(ByteBuffer curBuf) {
Review comment:
I could also build the LongBuffer view on demand if that works better for
you. Are you more worried about CPU, memory or both?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] jpountz commented on a change in pull request #973: LUCENE-9027: Use SIMD instructions to decode postings.
jpountz commented on a change in pull request #973: LUCENE-9027: Use SIMD
instructions to decode postings.
URL: https://github.com/apache/lucene-solr/pull/973#discussion_r345847390
##
File path:
lucene/core/src/java/org/apache/lucene/store/ByteBufferIndexInput.java
##
@@ -107,6 +122,24 @@ public final void readBytes(byte[] b, int offset, int
len) throws IOException {
}
}
+ @Override
+ public void readLongs(ByteOrder byteOrder, long[] dst, int offset, int
length) throws IOException {
+try {
+ final int position = curBuf.position();
+ guard.getLongs(curLongBufferViews[position & 0x07].position(position >>>
3), dst, offset, length);
Review comment:
For the record, there are 8, not 63 clones. Currently blocks all take
`bitsPerValue * 16 + 1` bytes, so adding padding would waste 7 bytes all the
time. This would be a 42% increase for 1 bpv blocks and 21% for 2 bpv, which
are not that uncommon bits per value for term freqs, especially when exceptions
help lower the number of bits per value.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-13872) Backup can fail to read index files w/NoSuchFileException during merges (SOLR-11616 regression)
[
https://issues.apache.org/jira/browse/SOLR-13872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16973470#comment-16973470
]
ASF subversion and git services commented on SOLR-13872:
Commit 30e55e2b6efc55c04761b80c22a106f4a1115722 in lucene-solr's branch
refs/heads/master from Chris M. Hostetter
[ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=30e55e2 ]
SOLR-13872: Fixed Backup failures due to race conditions in saving/reserving
commit points
> Backup can fail to read index files w/NoSuchFileException during merges
> (SOLR-11616 regression)
>
>
> Key: SOLR-13872
> URL: https://issues.apache.org/jira/browse/SOLR-13872
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Chris M. Hostetter
>Assignee: Chris M. Hostetter
>Priority: Major
> Attachments: SOLR-13872.patch, SOLR-13872.patch, SOLR-13872.patch,
> SOLR-13872.patch, index_churn.pl
>
>
> SOLR-11616 purports to fix a bug in Solr's backup functionality that causes
> 'NoSuchFileException' errors when attempting to backup an index while it is
> undergoing indexing (and segment merging)
> Although SOLR-11616 is marked with "Fix Version: 7.2" it's pretty easy to
> demonstrate that this bug still exists on master, branch_8x, and even in 7.2
> - so it seems less like the current problem is a "regression" and more that
> the original fix didn't work.
>
> The crux of the problem seems to be concurrency bugs in if/how a commit is
> "reserved" before attempting to copy the files in that commit to the backup
> location.
> A possible work around discussed in more depth in the comments below is to
> update {{solrconfig.xml}} to explicitly configure the {{SolrDeletionPolicy}}
> with either the {{maxCommitsToKeep}} or {{maxCommitAge}} options to ensure
> the commits are kept around long enough for the backup to be created.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Resolved] (LUCENE-9030) Solr- and WordnetSynonymParser behaviour differs
[
https://issues.apache.org/jira/browse/LUCENE-9030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alan Woodward resolved LUCENE-9030.
---
Fix Version/s: 8.4
master (9.0)
Resolution: Fixed
> Solr- and WordnetSynonymParser behaviour differs
>
>
> Key: LUCENE-9030
> URL: https://issues.apache.org/jira/browse/LUCENE-9030
> Project: Lucene - Core
> Issue Type: Bug
> Components: modules/analysis
>Affects Versions: 8.2
>Reporter: Christoph Büscher
>Assignee: Alan Woodward
>Priority: Minor
> Fix For: master (9.0), 8.4
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> Equivalent synonyms are showing up with different token types and ordering
> depending on whether the Solr format or the Wordnet format is used. A synonym
> set like
> "woods, wood, forest" in Solr format leads to the following token stream
> (term and type) when analyzing the term "forest":
> "forest"/word, "woods"/SYNONYM, "wood" /SYNONYM
>
> The following set in Wordnet format should give the same output (all terms
> are in the same synset), however all tokens are of type SYNONYM here and the
> original input token "forest" isn't the first one:
> synonyms.txt:
> {code:java}
> s(10001,1,'woods',n,1,0)
> s(10001,2,'wood',n,1,0)
> s(10001,3,'forest',n,1,0){code}
> Token stream (term/type) when an
> woods"/SYNONYM, "wood" /SYNONYM, "forest"/SYNONYM
> I don't think this is intentional and is confusing (especially because the
> "original" input token type gets lost). I saw that the way the synsets are
> added to the SynonymMap in the respective parsers differes and have a PR that
> changes this.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] jpountz commented on a change in pull request #973: LUCENE-9027: Use SIMD instructions to decode postings.
jpountz commented on a change in pull request #973: LUCENE-9027: Use SIMD
instructions to decode postings.
URL: https://github.com/apache/lucene-solr/pull/973#discussion_r345840144
##
File path:
lucene/core/src/java/org/apache/lucene/store/ByteBufferIndexInput.java
##
@@ -107,6 +122,24 @@ public final void readBytes(byte[] b, int offset, int
len) throws IOException {
}
}
+ @Override
+ public void readLongs(ByteOrder byteOrder, long[] dst, int offset, int
length) throws IOException {
+try {
+ final int position = curBuf.position();
+ guard.getLongs(curLongBufferViews[position & 0x07].position(position >>>
3), dst, offset, length);
+ if (byteOrder != ByteOrder.nativeOrder()) {
+for (int i = offset, end = offset + length; i < end; ++i) {
+ dst[i] = Long.reverseBytes(dst[i]);
+}
+ }
+ curBuf.position(position + (length << 3));
+} catch (BufferUnderflowException e) {
+ super.readLongs(byteOrder, dst, offset, length);
Review comment:
This is correct.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9030) Solr- and WordnetSynonymParser behaviour differs
[
https://issues.apache.org/jira/browse/LUCENE-9030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16973453#comment-16973453
]
ASF subversion and git services commented on LUCENE-9030:
-
Commit 3a7b25ba92864b334526d827f3aa5aa6445d9829 in lucene-solr's branch
refs/heads/master from Christoph Büscher
[ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=3a7b25b ]
LUCENE-9030: Fix different Solr- and WordnetSynonymParser behaviour (#981)
This fixes an issue where sets of equivalent synonyms in the Wordnet format are
parsed and added to the SynonymMap in a way that leads to the original input
token not being typed as "word" but as SYNONYM instead. Also the original token
doesn't appear first in the token stream output, which is the case for
equivalent solr formatted synonym files.
Currently the WordnetSynonymParser adds all combinations of input/output pairs
of a synset entry into the synonym map, while the SolrSynonymParser excludes
those where input and output term are the same. This change adds the same
behaviour to WordnetSynonymParser and adds tests that show the two formats are
outputting the same token order and types now.
> Solr- and WordnetSynonymParser behaviour differs
>
>
> Key: LUCENE-9030
> URL: https://issues.apache.org/jira/browse/LUCENE-9030
> Project: Lucene - Core
> Issue Type: Bug
> Components: modules/analysis
>Affects Versions: 8.2
>Reporter: Christoph Büscher
>Assignee: Alan Woodward
>Priority: Minor
> Time Spent: 40m
> Remaining Estimate: 0h
>
> Equivalent synonyms are showing up with different token types and ordering
> depending on whether the Solr format or the Wordnet format is used. A synonym
> set like
> "woods, wood, forest" in Solr format leads to the following token stream
> (term and type) when analyzing the term "forest":
> "forest"/word, "woods"/SYNONYM, "wood" /SYNONYM
>
> The following set in Wordnet format should give the same output (all terms
> are in the same synset), however all tokens are of type SYNONYM here and the
> original input token "forest" isn't the first one:
> synonyms.txt:
> {code:java}
> s(10001,1,'woods',n,1,0)
> s(10001,2,'wood',n,1,0)
> s(10001,3,'forest',n,1,0){code}
> Token stream (term/type) when an
> woods"/SYNONYM, "wood" /SYNONYM, "forest"/SYNONYM
> I don't think this is intentional and is confusing (especially because the
> "original" input token type gets lost). I saw that the way the synsets are
> added to the SynonymMap in the respective parsers differes and have a PR that
> changes this.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] jpountz commented on a change in pull request #973: LUCENE-9027: Use SIMD instructions to decode postings.
jpountz commented on a change in pull request #973: LUCENE-9027: Use SIMD
instructions to decode postings.
URL: https://github.com/apache/lucene-solr/pull/973#discussion_r345832545
##
File path: lucene/core/src/java/org/apache/lucene/codecs/lucene84/PForUtil.java
##
@@ -0,0 +1,126 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.lucene.codecs.lucene84;
+
+import java.io.IOException;
+import java.util.Arrays;
+
+import org.apache.lucene.store.DataInput;
+import org.apache.lucene.store.DataOutput;
+import org.apache.lucene.util.packed.PackedInts;
+
+final class PForUtil {
+
+ private static boolean allEqual(long[] l) {
+for (int i = 1; i < ForUtil.BLOCK_SIZE; ++i) {
+ if (l[i] != l[0]) {
+return false;
+ }
+}
+return true;
+ }
+
+ private final ForUtil forUtil;
+
+ PForUtil(ForUtil forUtil) {
+this.forUtil = forUtil;
+ }
+
+ /**
+ * Encode 128 8-bits integers from {@code data} into {@code out}.
Review comment:
Sorry I've had many iterations, and this comment is outdated, I'll fix.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] romseygeek merged pull request #981: LUCENE-9030: Fix different Solr- and WordnetSynonymParser behaviour
romseygeek merged pull request #981: LUCENE-9030: Fix different Solr- and WordnetSynonymParser behaviour URL: https://github.com/apache/lucene-solr/pull/981 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] jpountz commented on a change in pull request #973: LUCENE-9027: Use SIMD instructions to decode postings.
jpountz commented on a change in pull request #973: LUCENE-9027: Use SIMD
instructions to decode postings.
URL: https://github.com/apache/lucene-solr/pull/973#discussion_r345832291
##
File path: lucene/core/src/java/org/apache/lucene/codecs/lucene84/PForUtil.java
##
@@ -0,0 +1,126 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.lucene.codecs.lucene84;
+
+import java.io.IOException;
+import java.util.Arrays;
+
+import org.apache.lucene.store.DataInput;
+import org.apache.lucene.store.DataOutput;
+import org.apache.lucene.util.packed.PackedInts;
+
+final class PForUtil {
+
+ private static boolean allEqual(long[] l) {
+for (int i = 1; i < ForUtil.BLOCK_SIZE; ++i) {
+ if (l[i] != l[0]) {
+return false;
+ }
+}
+return true;
+ }
+
+ private final ForUtil forUtil;
+
+ PForUtil(ForUtil forUtil) {
+this.forUtil = forUtil;
+ }
+
+ /**
+ * Encode 128 8-bits integers from {@code data} into {@code out}.
+ */
+ void encode(long[] longs, DataOutput out) throws IOException {
+// At most 3 exceptions
+final long[] top4 = new long[4];
+Arrays.fill(top4, -1L);
+for (int i = 0; i < ForUtil.BLOCK_SIZE; ++i) {
+ if (longs[i] > top4[0]) {
+top4[0] = longs[i];
+Arrays.sort(top4);
+ }
+}
+
+final int maxBitsRequired = PackedInts.bitsRequired(top4[3]);
+// We store the patch on a byte, so we can't decrease the number of bits
required by more than 8
+final int patchedBitsRequired =
Math.max(PackedInts.bitsRequired(top4[0]), maxBitsRequired - 8);
+int numExceptions = 0;
+for (int i = 1; i < 4; ++i) {
+ if (top4[i] > (1L << patchedBitsRequired) - 1) {
+numExceptions++;
+ }
+}
+final byte[] exceptions = new byte[numExceptions*2];
+if (numExceptions > 0) {
+ int exceptionCount = 0;
+ for (int i = 0; i < ForUtil.BLOCK_SIZE; ++i) {
+if (longs[i] > (1L << patchedBitsRequired) - 1) {
+ exceptions[exceptionCount*2] = (byte) i;
+ exceptions[exceptionCount*2+1] = (byte) (longs[i] >>>
patchedBitsRequired);
+ longs[i] &= (1L << patchedBitsRequired) - 1;
+ exceptionCount++;
+}
+ }
+ assert exceptionCount == numExceptions : exceptionCount + " " +
numExceptions;
+}
+
+if (allEqual(longs) && maxBitsRequired <= 8) {
Review comment:
All values in `longs` have been masked at this point, so there might be
exceptions. My intuition is that this case is important for rare terms that
might have most of their term freqs equal to 1, but maybe a couple exceptions.
Does it answer your question?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] cbuescher commented on issue #981: LUCENE-9030: Fix different Solr- and WordnetSynonymParser behaviour
cbuescher commented on issue #981: LUCENE-9030: Fix different Solr- and WordnetSynonymParser behaviour URL: https://github.com/apache/lucene-solr/pull/981#issuecomment-553457992 @romseygeek thanks for the review, I updated the CHANGES.txt. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] romseygeek commented on issue #981: LUCENE-9030: Fix different Solr- and WordnetSynonymParser behaviour
romseygeek commented on issue #981: LUCENE-9030: Fix different Solr- and WordnetSynonymParser behaviour URL: https://github.com/apache/lucene-solr/pull/981#issuecomment-553436971 Could you add a CHANGES.txt entry? All looks good other than that, and precommit passes locally for me. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9030) Solr- and WordnetSynonymParser behaviour differs
[
https://issues.apache.org/jira/browse/LUCENE-9030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16973396#comment-16973396
]
Alan Woodward commented on LUCENE-9030:
---
Thanks for opening this fix, [~cbuescher] - I'm just running precommit now and
will merge it in once that check passes.
> Solr- and WordnetSynonymParser behaviour differs
>
>
> Key: LUCENE-9030
> URL: https://issues.apache.org/jira/browse/LUCENE-9030
> Project: Lucene - Core
> Issue Type: Bug
> Components: modules/analysis
>Affects Versions: 8.2
>Reporter: Christoph Büscher
>Assignee: Alan Woodward
>Priority: Minor
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Equivalent synonyms are showing up with different token types and ordering
> depending on whether the Solr format or the Wordnet format is used. A synonym
> set like
> "woods, wood, forest" in Solr format leads to the following token stream
> (term and type) when analyzing the term "forest":
> "forest"/word, "woods"/SYNONYM, "wood" /SYNONYM
>
> The following set in Wordnet format should give the same output (all terms
> are in the same synset), however all tokens are of type SYNONYM here and the
> original input token "forest" isn't the first one:
> synonyms.txt:
> {code:java}
> s(10001,1,'woods',n,1,0)
> s(10001,2,'wood',n,1,0)
> s(10001,3,'forest',n,1,0){code}
> Token stream (term/type) when an
> woods"/SYNONYM, "wood" /SYNONYM, "forest"/SYNONYM
> I don't think this is intentional and is confusing (especially because the
> "original" input token type gets lost). I saw that the way the synsets are
> added to the SynonymMap in the respective parsers differes and have a PR that
> changes this.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Assigned] (LUCENE-9030) Solr- and WordnetSynonymParser behaviour differs
[
https://issues.apache.org/jira/browse/LUCENE-9030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alan Woodward reassigned LUCENE-9030:
-
Assignee: Alan Woodward
> Solr- and WordnetSynonymParser behaviour differs
>
>
> Key: LUCENE-9030
> URL: https://issues.apache.org/jira/browse/LUCENE-9030
> Project: Lucene - Core
> Issue Type: Bug
> Components: modules/analysis
>Affects Versions: 8.2
>Reporter: Christoph Büscher
>Assignee: Alan Woodward
>Priority: Minor
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Equivalent synonyms are showing up with different token types and ordering
> depending on whether the Solr format or the Wordnet format is used. A synonym
> set like
> "woods, wood, forest" in Solr format leads to the following token stream
> (term and type) when analyzing the term "forest":
> "forest"/word, "woods"/SYNONYM, "wood" /SYNONYM
>
> The following set in Wordnet format should give the same output (all terms
> are in the same synset), however all tokens are of type SYNONYM here and the
> original input token "forest" isn't the first one:
> synonyms.txt:
> {code:java}
> s(10001,1,'woods',n,1,0)
> s(10001,2,'wood',n,1,0)
> s(10001,3,'forest',n,1,0){code}
> Token stream (term/type) when an
> woods"/SYNONYM, "wood" /SYNONYM, "forest"/SYNONYM
> I don't think this is intentional and is confusing (especially because the
> "original" input token type gets lost). I saw that the way the synsets are
> added to the SynonymMap in the respective parsers differes and have a PR that
> changes this.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Assigned] (SOLR-13921) Processing UpdateRequest with delegation token throws NullPointerException
[
https://issues.apache.org/jira/browse/SOLR-13921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kevin Risden reassigned SOLR-13921:
---
Assignee: Kevin Risden
> Processing UpdateRequest with delegation token throws NullPointerException
> --
>
> Key: SOLR-13921
> URL: https://issues.apache.org/jira/browse/SOLR-13921
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
> Components: SolrJ
>Affects Versions: 7.4, 8.3
>Reporter: Istvan Farkas
>Assignee: Kevin Risden
>Priority: Minor
> Attachments: SOLR-13921.patch
>
>
> When sending UpdateRequests with delegation tokens to Solr using SolrJ, the
> createMethod of DelegationTokenHttpSolrClient will throw a
> NullPointerException:
>
> {code:java}
> [junit4] ERROR 3.41s |
> TestSolrCloudWithDelegationTokens.testDelegationTokenSolrClientWithUpdateRequests
> <<<
>[junit4]> Throwable #1: java.lang.NullPointerException
>[junit4]> at
> __randomizedtesting.SeedInfo.seed([B9AE8E4E0CDF1B3D:DBA0B722C813061D]:0)
>[junit4]> at
> org.apache.solr.client.solrj.impl.DelegationTokenHttpSolrClient.createMethod(DelegationTokenHttpSolrClient.java:93)
>[junit4]> at
> org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:258)
>[junit4]> at
> org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:249)
>[junit4]> at
> org.apache.solr.cloud.TestSolrCloudWithDelegationTokens.doSolrRequest(TestSolrCloudWithDelegationTokens.java:246)
>[junit4]> at
> org.apache.solr.cloud.TestSolrCloudWithDelegationTokens.testDelegationTokenSolrClientWithUpdateRequests(TestSolrCloudWithDelegationTokens.java:477)
>[junit4]> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>[junit4]> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>[junit4]> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>[junit4]> at
> java.base/java.lang.reflect.Method.invoke(Method.java:566)
>[junit4]> at java.base/java.lang.Thread.run(Thread.java:834)
> {code}
> This happens to all SolrJ clients including the Spark Crunch indexer which
> use Delegation Tokens and do not specify commit / optimize commands.
> The cause seems to be a missing null check before dereferencing 'params'. The
> intention of the code seems to be verifying if the delegation token is passed
> a parameter of the SolrRequest (which is not supported), however the check
> fails with NPE if the request has no params at all. For update requests which
> do commit or optimize, the setCommand method initializes the params so no
> NPE is thrown.
> {code}
> @Override
> protected HttpRequestBase createMethod(final SolrRequest request, String
> collection) throws IOException, SolrServerException {
> SolrParams params = request.getParams();
> if (params.getParams(DELEGATION_TOKEN_PARAM) != null) {
> throw new IllegalArgumentException(DELEGATION_TOKEN_PARAM + " parameter
> not supported");
> }
> return super.createMethod(request, collection);
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-13898) Non-atomic use of SolrCache get / put
[
https://issues.apache.org/jira/browse/SOLR-13898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16973371#comment-16973371
]
ASF subversion and git services commented on SOLR-13898:
Commit 0c3233877b61a8f6a1d9132ea8cac4d83fccb337 in lucene-solr's branch
refs/heads/master from Andrzej Bialecki
[ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=0c32338 ]
SOLR-13898: Non-atomic use of SolrCache get / put.
> Non-atomic use of SolrCache get / put
> -
>
> Key: SOLR-13898
> URL: https://issues.apache.org/jira/browse/SOLR-13898
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
>Affects Versions: 8.3
>Reporter: Andrzej Bialecki
>Assignee: Andrzej Bialecki
>Priority: Major
> Fix For: 8.4
>
> Attachments: SOLR-13898.patch, SOLR-13898.patch, SOLR-13898.patch
>
>
> As pointed out by [~ben.manes] in SOLR-13817 Solr code base in many key
> places uses a similar pattern of non-atomic get / put calls to SolrCache-s.
> In multi-threaded environment this leads to cache misses and additional
> unnecessary computations when competing threads both discover a missing
> value, non-atomically compute it and update the cache.
> Some of these places are known performance bottlenecks where efficient
> caching is especially important, such as {{SolrIndexSearcher}},
> {{SolrDocumentFetcher}}, {{UninvertedField}} and join queries .
> I propose to add {{SolrCache.computeIfAbsent(key, mappingFunction)}} that
> will atomically retrieve existing values or compute and update the cache.
> This will require also changing how the {{SolrCache.get(...)}} is used in
> many components.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] mikemccand commented on a change in pull request #973: LUCENE-9027: Use SIMD instructions to decode postings.
mikemccand commented on a change in pull request #973: LUCENE-9027: Use SIMD
instructions to decode postings.
URL: https://github.com/apache/lucene-solr/pull/973#discussion_r345755022
##
File path:
lucene/core/src/java/org/apache/lucene/store/ByteBufferIndexInput.java
##
@@ -107,6 +122,24 @@ public final void readBytes(byte[] b, int offset, int
len) throws IOException {
}
}
+ @Override
+ public void readLongs(ByteOrder byteOrder, long[] dst, int offset, int
length) throws IOException {
+try {
+ final int position = curBuf.position();
+ guard.getLongs(curLongBufferViews[position & 0x07].position(position >>>
3), dst, offset, length);
+ if (byteOrder != ByteOrder.nativeOrder()) {
+for (int i = offset, end = offset + length; i < end; ++i) {
+ dst[i] = Long.reverseBytes(dst[i]);
+}
+ }
+ curBuf.position(position + (length << 3));
+} catch (BufferUnderflowException e) {
+ super.readLongs(byteOrder, dst, offset, length);
Review comment:
Hmm, how come we don't also do the `setCurBuf` here? It seems like we are
falling back to the slow (`super.readLongs`) implementation always on the first
`readLongs` call, but since that slow version will use `readByte` it will then
initialize the `curLongBufferViews` for subsequent `readLongs` calls?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] mikemccand commented on a change in pull request #973: LUCENE-9027: Use SIMD instructions to decode postings.
mikemccand commented on a change in pull request #973: LUCENE-9027: Use SIMD
instructions to decode postings.
URL: https://github.com/apache/lucene-solr/pull/973#discussion_r345737509
##
File path: lucene/core/src/java/org/apache/lucene/codecs/lucene84/PForUtil.java
##
@@ -0,0 +1,126 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.lucene.codecs.lucene84;
+
+import java.io.IOException;
+import java.util.Arrays;
+
+import org.apache.lucene.store.DataInput;
+import org.apache.lucene.store.DataOutput;
+import org.apache.lucene.util.packed.PackedInts;
+
+final class PForUtil {
+
+ private static boolean allEqual(long[] l) {
+for (int i = 1; i < ForUtil.BLOCK_SIZE; ++i) {
+ if (l[i] != l[0]) {
+return false;
+ }
+}
+return true;
+ }
+
+ private final ForUtil forUtil;
+
+ PForUtil(ForUtil forUtil) {
+this.forUtil = forUtil;
+ }
+
+ /**
+ * Encode 128 8-bits integers from {@code data} into {@code out}.
+ */
+ void encode(long[] longs, DataOutput out) throws IOException {
+// At most 3 exceptions
+final long[] top4 = new long[4];
+Arrays.fill(top4, -1L);
+for (int i = 0; i < ForUtil.BLOCK_SIZE; ++i) {
+ if (longs[i] > top4[0]) {
+top4[0] = longs[i];
+Arrays.sort(top4);
+ }
+}
+
+final int maxBitsRequired = PackedInts.bitsRequired(top4[3]);
+// We store the patch on a byte, so we can't decrease the number of bits
required by more than 8
+final int patchedBitsRequired =
Math.max(PackedInts.bitsRequired(top4[0]), maxBitsRequired - 8);
+int numExceptions = 0;
+for (int i = 1; i < 4; ++i) {
+ if (top4[i] > (1L << patchedBitsRequired) - 1) {
+numExceptions++;
+ }
+}
+final byte[] exceptions = new byte[numExceptions*2];
+if (numExceptions > 0) {
+ int exceptionCount = 0;
+ for (int i = 0; i < ForUtil.BLOCK_SIZE; ++i) {
+if (longs[i] > (1L << patchedBitsRequired) - 1) {
+ exceptions[exceptionCount*2] = (byte) i;
+ exceptions[exceptionCount*2+1] = (byte) (longs[i] >>>
patchedBitsRequired);
+ longs[i] &= (1L << patchedBitsRequired) - 1;
+ exceptionCount++;
+}
+ }
+ assert exceptionCount == numExceptions : exceptionCount + " " +
numExceptions;
+}
+
+if (allEqual(longs) && maxBitsRequired <= 8) {
Review comment:
I think you safely insert `numExceptions == 0` into this if conjunction?
I.e. when all longs are equal, there would be no exceptions, I think?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] mikemccand commented on a change in pull request #973: LUCENE-9027: Use SIMD instructions to decode postings.
mikemccand commented on a change in pull request #973: LUCENE-9027: Use SIMD
instructions to decode postings.
URL: https://github.com/apache/lucene-solr/pull/973#discussion_r345737023
##
File path: lucene/core/src/java/org/apache/lucene/codecs/lucene84/PForUtil.java
##
@@ -0,0 +1,126 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.lucene.codecs.lucene84;
+
+import java.io.IOException;
+import java.util.Arrays;
+
+import org.apache.lucene.store.DataInput;
+import org.apache.lucene.store.DataOutput;
+import org.apache.lucene.util.packed.PackedInts;
+
+final class PForUtil {
+
+ private static boolean allEqual(long[] l) {
+for (int i = 1; i < ForUtil.BLOCK_SIZE; ++i) {
+ if (l[i] != l[0]) {
+return false;
+ }
+}
+return true;
+ }
+
+ private final ForUtil forUtil;
+
+ PForUtil(ForUtil forUtil) {
+this.forUtil = forUtil;
+ }
+
+ /**
+ * Encode 128 8-bits integers from {@code data} into {@code out}.
+ */
+ void encode(long[] longs, DataOutput out) throws IOException {
+// At most 3 exceptions
+final long[] top4 = new long[4];
+Arrays.fill(top4, -1L);
+for (int i = 0; i < ForUtil.BLOCK_SIZE; ++i) {
+ if (longs[i] > top4[0]) {
+top4[0] = longs[i];
+Arrays.sort(top4);
+ }
+}
+
+final int maxBitsRequired = PackedInts.bitsRequired(top4[3]);
+// We store the patch on a byte, so we can't decrease the number of bits
required by more than 8
+final int patchedBitsRequired =
Math.max(PackedInts.bitsRequired(top4[0]), maxBitsRequired - 8);
+int numExceptions = 0;
+for (int i = 1; i < 4; ++i) {
+ if (top4[i] > (1L << patchedBitsRequired) - 1) {
Review comment:
Maybe add a new local variable, `long patchLimit = (1L <<
patchedBitsRequired) - 1;`, used here and below?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] mikemccand commented on a change in pull request #973: LUCENE-9027: Use SIMD instructions to decode postings.
mikemccand commented on a change in pull request #973: LUCENE-9027: Use SIMD
instructions to decode postings.
URL: https://github.com/apache/lucene-solr/pull/973#discussion_r345736527
##
File path: lucene/core/src/java/org/apache/lucene/codecs/lucene84/PForUtil.java
##
@@ -0,0 +1,126 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.lucene.codecs.lucene84;
+
+import java.io.IOException;
+import java.util.Arrays;
+
+import org.apache.lucene.store.DataInput;
+import org.apache.lucene.store.DataOutput;
+import org.apache.lucene.util.packed.PackedInts;
+
+final class PForUtil {
+
+ private static boolean allEqual(long[] l) {
+for (int i = 1; i < ForUtil.BLOCK_SIZE; ++i) {
+ if (l[i] != l[0]) {
+return false;
+ }
+}
+return true;
+ }
+
+ private final ForUtil forUtil;
+
+ PForUtil(ForUtil forUtil) {
+this.forUtil = forUtil;
+ }
+
+ /**
+ * Encode 128 8-bits integers from {@code data} into {@code out}.
Review comment:
Hmm, should be `{@code longs}`? And, are they really 8-bit integers
arriving in `longs`? If so, why is it `long[]`?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] mikemccand commented on a change in pull request #973: LUCENE-9027: Use SIMD instructions to decode postings.
mikemccand commented on a change in pull request #973: LUCENE-9027: Use SIMD
instructions to decode postings.
URL: https://github.com/apache/lucene-solr/pull/973#discussion_r345737876
##
File path: lucene/core/src/java/org/apache/lucene/codecs/lucene84/PForUtil.java
##
@@ -0,0 +1,126 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.lucene.codecs.lucene84;
+
+import java.io.IOException;
+import java.util.Arrays;
+
+import org.apache.lucene.store.DataInput;
+import org.apache.lucene.store.DataOutput;
+import org.apache.lucene.util.packed.PackedInts;
+
+final class PForUtil {
+
+ private static boolean allEqual(long[] l) {
+for (int i = 1; i < ForUtil.BLOCK_SIZE; ++i) {
+ if (l[i] != l[0]) {
+return false;
+ }
+}
+return true;
+ }
+
+ private final ForUtil forUtil;
+
+ PForUtil(ForUtil forUtil) {
+this.forUtil = forUtil;
+ }
+
+ /**
+ * Encode 128 8-bits integers from {@code data} into {@code out}.
+ */
+ void encode(long[] longs, DataOutput out) throws IOException {
+// At most 3 exceptions
+final long[] top4 = new long[4];
+Arrays.fill(top4, -1L);
+for (int i = 0; i < ForUtil.BLOCK_SIZE; ++i) {
+ if (longs[i] > top4[0]) {
+top4[0] = longs[i];
+Arrays.sort(top4);
+ }
+}
+
+final int maxBitsRequired = PackedInts.bitsRequired(top4[3]);
+// We store the patch on a byte, so we can't decrease the number of bits
required by more than 8
+final int patchedBitsRequired =
Math.max(PackedInts.bitsRequired(top4[0]), maxBitsRequired - 8);
+int numExceptions = 0;
+for (int i = 1; i < 4; ++i) {
+ if (top4[i] > (1L << patchedBitsRequired) - 1) {
+numExceptions++;
+ }
+}
+final byte[] exceptions = new byte[numExceptions*2];
+if (numExceptions > 0) {
+ int exceptionCount = 0;
+ for (int i = 0; i < ForUtil.BLOCK_SIZE; ++i) {
+if (longs[i] > (1L << patchedBitsRequired) - 1) {
+ exceptions[exceptionCount*2] = (byte) i;
+ exceptions[exceptionCount*2+1] = (byte) (longs[i] >>>
patchedBitsRequired);
+ longs[i] &= (1L << patchedBitsRequired) - 1;
+ exceptionCount++;
+}
+ }
+ assert exceptionCount == numExceptions : exceptionCount + " " +
numExceptions;
+}
+
+if (allEqual(longs) && maxBitsRequired <= 8) {
+ for (int i = 0; i < numExceptions; ++i) {
Review comment:
Hmm, maybe I'm wrong and confused about this, since you are iterating the
exceptions here :)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (LUCENE-9044) Currently Lucene doesn't have an analyzer for Sinhala. We have built analyzer which consist of language dependent tokenizer, stemming algorithm and list of stop words.
[ https://issues.apache.org/jira/browse/LUCENE-9044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] pavithra kariyawasam updated LUCENE-9044: - Status: Open (was: Patch Available) > Currently Lucene doesn't have an analyzer for Sinhala. We have built analyzer > which consist of language dependent tokenizer, stemming algorithm and list of > stop words. > --- > > Key: LUCENE-9044 > URL: https://issues.apache.org/jira/browse/LUCENE-9044 > Project: Lucene - Core > Issue Type: New Feature > Components: modules/analysis > Environment: Lucene >Reporter: pavithra kariyawasam >Priority: Major > Labels: features > Fix For: 5.5.6 > > Attachments: SinhalaAnalyzer.java, SinhalaStemmer.java, > SinhalaTokenizer.java, stopwords.txt > > > This component is developed based on three main researches. > Lucene did not have component to analyze Sinhala documents. So our intension > is to fill that space with an Analyzer which can analyze Sinhala documents. > Sinhala Analyzer has implemented by performing Sinhala morphological > analysis. Tokenizing the document content precisely, Removing stopwords > accordingly and converting the terms to its base/root form accurately are the > main three functionalities of Sinhala Analyzer. These are built by > considering the grammatical rules in Sinhala -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (LUCENE-9044) Currently Lucene doesn't have an analyzer for Sinhala. We have built analyzer which consist of language dependent tokenizer, stemming algorithm and list of stop words.
[ https://issues.apache.org/jira/browse/LUCENE-9044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] pavithra kariyawasam updated LUCENE-9044: - Status: Patch Available (was: Open) > Currently Lucene doesn't have an analyzer for Sinhala. We have built analyzer > which consist of language dependent tokenizer, stemming algorithm and list of > stop words. > --- > > Key: LUCENE-9044 > URL: https://issues.apache.org/jira/browse/LUCENE-9044 > Project: Lucene - Core > Issue Type: New Feature > Components: modules/analysis > Environment: Lucene >Reporter: pavithra kariyawasam >Priority: Major > Labels: features > Fix For: 5.5.6 > > Attachments: SinhalaAnalyzer.java, SinhalaStemmer.java, > SinhalaTokenizer.java, stopwords.txt > > > This component is developed based on three main researches. > Lucene did not have component to analyze Sinhala documents. So our intension > is to fill that space with an Analyzer which can analyze Sinhala documents. > Sinhala Analyzer has implemented by performing Sinhala morphological > analysis. Tokenizing the document content precisely, Removing stopwords > accordingly and converting the terms to its base/root form accurately are the > main three functionalities of Sinhala Analyzer. These are built by > considering the grammatical rules in Sinhala -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (LUCENE-9043) Currently Lucene doesn't have an analyzer for Sinhala. We have built analyzer which consist of language dependent tokenizer, stemming algorithm and list of stop words.
[ https://issues.apache.org/jira/browse/LUCENE-9043?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] pavithra kariyawasam updated LUCENE-9043: - Status: Patch Available (was: Open) > Currently Lucene doesn't have an analyzer for Sinhala. We have built analyzer > which consist of language dependent tokenizer, stemming algorithm and list of > stop words. > --- > > Key: LUCENE-9043 > URL: https://issues.apache.org/jira/browse/LUCENE-9043 > Project: Lucene - Core > Issue Type: Improvement > Components: modules/analysis >Affects Versions: 8.3 >Reporter: pavithra kariyawasam >Priority: Major > Fix For: 5.5.6 > > Attachments: SinhalaAnalyzer.java, SinhalaStemmer.java, > SinhalaTokenizer.java, stopwords.txt > > > This component is developed based on three main researches. > Sinhala Analyzer, as it word implies it is an enhanced software library to > analyze documents which are written in Sinhala language. Sinhala Analyzer has > implemented by performing Sinhala morphological analysis. Tokenizing the > document content precisely, Removing stopwords accordingly and converting the > terms to its base/root form accurately are the main three functionalities of > Sinhala Analyzer. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (LUCENE-9043) Currently Lucene doesn't have an analyzer for Sinhala. We have built analyzer which consist of language dependent tokenizer, stemming algorithm and list of stop words.
[ https://issues.apache.org/jira/browse/LUCENE-9043?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] pavithra kariyawasam updated LUCENE-9043: - Status: Open (was: Patch Available) > Currently Lucene doesn't have an analyzer for Sinhala. We have built analyzer > which consist of language dependent tokenizer, stemming algorithm and list of > stop words. > --- > > Key: LUCENE-9043 > URL: https://issues.apache.org/jira/browse/LUCENE-9043 > Project: Lucene - Core > Issue Type: Improvement > Components: modules/analysis >Affects Versions: 8.3 >Reporter: pavithra kariyawasam >Priority: Major > Fix For: 5.5.6 > > Attachments: SinhalaAnalyzer.java, SinhalaStemmer.java, > SinhalaTokenizer.java, stopwords.txt > > > This component is developed based on three main researches. > Sinhala Analyzer, as it word implies it is an enhanced software library to > analyze documents which are written in Sinhala language. Sinhala Analyzer has > implemented by performing Sinhala morphological analysis. Tokenizing the > document content precisely, Removing stopwords accordingly and converting the > terms to its base/root form accurately are the main three functionalities of > Sinhala Analyzer. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (LUCENE-9044) Currently Lucene doesn't have an analyzer for Sinhala. We have built analyzer which consist of language dependent tokenizer, stemming algorithm and list of stop words.
[ https://issues.apache.org/jira/browse/LUCENE-9044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] pavithra kariyawasam updated LUCENE-9044: - Review Patch?: (was: Yes) > Currently Lucene doesn't have an analyzer for Sinhala. We have built analyzer > which consist of language dependent tokenizer, stemming algorithm and list of > stop words. > --- > > Key: LUCENE-9044 > URL: https://issues.apache.org/jira/browse/LUCENE-9044 > Project: Lucene - Core > Issue Type: New Feature > Components: modules/analysis > Environment: Lucene >Reporter: pavithra kariyawasam >Priority: Major > Labels: features > Fix For: 5.5.6 > > Attachments: SinhalaAnalyzer.java, SinhalaStemmer.java, > SinhalaTokenizer.java, stopwords.txt > > > This component is developed based on three main researches. > Lucene did not have component to analyze Sinhala documents. So our intension > is to fill that space with an Analyzer which can analyze Sinhala documents. > Sinhala Analyzer has implemented by performing Sinhala morphological > analysis. Tokenizing the document content precisely, Removing stopwords > accordingly and converting the terms to its base/root form accurately are the > main three functionalities of Sinhala Analyzer. These are built by > considering the grammatical rules in Sinhala -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (LUCENE-9044) Currently Lucene doesn't have an analyzer for Sinhala. We have built analyzer which consist of language dependent tokenizer, stemming algorithm and list of stop words.
[ https://issues.apache.org/jira/browse/LUCENE-9044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] pavithra kariyawasam updated LUCENE-9044: - Affects Version/s: (was: 8.3) > Currently Lucene doesn't have an analyzer for Sinhala. We have built analyzer > which consist of language dependent tokenizer, stemming algorithm and list of > stop words. > --- > > Key: LUCENE-9044 > URL: https://issues.apache.org/jira/browse/LUCENE-9044 > Project: Lucene - Core > Issue Type: New Feature > Components: modules/analysis > Environment: Lucene >Reporter: pavithra kariyawasam >Priority: Major > Labels: features > Fix For: 5.5.6 > > Attachments: SinhalaAnalyzer.java, SinhalaStemmer.java, > SinhalaTokenizer.java, stopwords.txt > > > This component is developed based on three main researches. > Lucene did not have component to analyze Sinhala documents. So our intension > is to fill that space with an Analyzer which can analyze Sinhala documents. > Sinhala Analyzer has implemented by performing Sinhala morphological > analysis. Tokenizing the document content precisely, Removing stopwords > accordingly and converting the terms to its base/root form accurately are the > main three functionalities of Sinhala Analyzer. These are built by > considering the grammatical rules in Sinhala -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (LUCENE-9044) Currently Lucene doesn't have an analyzer for Sinhala. We have built analyzer which consist of language dependent tokenizer, stemming algorithm and list of stop words.
[ https://issues.apache.org/jira/browse/LUCENE-9044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] pavithra kariyawasam updated LUCENE-9044: - Issue Type: New Feature (was: Improvement) > Currently Lucene doesn't have an analyzer for Sinhala. We have built analyzer > which consist of language dependent tokenizer, stemming algorithm and list of > stop words. > --- > > Key: LUCENE-9044 > URL: https://issues.apache.org/jira/browse/LUCENE-9044 > Project: Lucene - Core > Issue Type: New Feature > Components: modules/analysis >Affects Versions: 8.3 > Environment: Lucene >Reporter: pavithra kariyawasam >Priority: Major > Labels: features > Fix For: 5.5.6 > > Attachments: SinhalaAnalyzer.java, SinhalaStemmer.java, > SinhalaTokenizer.java, stopwords.txt > > > This component is developed based on three main researches. > Lucene did not have component to analyze Sinhala documents. So our intension > is to fill that space with an Analyzer which can analyze Sinhala documents. > Sinhala Analyzer has implemented by performing Sinhala morphological > analysis. Tokenizing the document content precisely, Removing stopwords > accordingly and converting the terms to its base/root form accurately are the > main three functionalities of Sinhala Analyzer. These are built by > considering the grammatical rules in Sinhala -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9027) SIMD-based decoding of postings lists
[
https://issues.apache.org/jira/browse/LUCENE-9027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16973328#comment-16973328
]
Adrien Grand commented on LUCENE-9027:
--
I guess it depends what kind of analytics, I expect counts to get the best
speedup indeed, since they are only about decoding postings and counting
matches. However other analytics use-cases like facets still need to read
values for a doc-value field, which likely drowns the speedup a bit, like
sorting. We have some benchmarks for Elasticsearch that run nightly at
https://benchmarks.elastic.co but most analytics queries like histograms or
terms facets run on a match_all query, so I'm not expecting to see any speedup.
I've done some more research regarding the prefix sum. Unfortunately, I didn't
manage to get C2 to generate SIMD instructions for the prefix sum (even with
gaps of 2 or 4 instead of 1). The best option I have for now works by hooking
into the decoding logic and summing up longs when they still represent 2 packed
integers (which effectively computes the prefix sum for [0:64) and [64:128) in
parallel and then add the value at index 63 to all values at indices [64:128).
This last step is vectorized by the JVM. See benchmark results at the bottom of
https://github.com/jpountz/decode-128-ints-benchmark if you are interested. I
wish we could do better, but it's still nice that we can both decompress and
compute the prefix sum faster that we can just decompress with the current
codec on average. Having the sum pre-computed also helps simplify some
conditions in the PostingsEnum implementations of nextDoc/advance.
Here is the result of a run on wikibigall with the current pull request. I
included some sorting tasks as well to see how much they benefit from this
change
{noformat}
TaskQPS baseline StdDev QPS patch StdDev
Pct diff
Fuzzy2 99.40 (10.4%) 95.46 (6.9%)
-4.0% ( -19% - 14%)
IntNRQ 966.00 (2.4%) 977.27 (2.1%)
1.2% ( -3% -5%)
Fuzzy1 97.25 (7.8%) 99.66 (9.4%)
2.5% ( -13% - 21%)
OrHighHigh 84.95 (3.7%) 87.40 (4.2%)
2.9% ( -4% - 11%)
Term 1316.61 (3.1%) 1357.55 (4.8%)
3.1% ( -4% - 11%)
HighTermDayOfYearSort 35.89 (6.3%) 37.66 (4.2%)
4.9% ( -5% - 16%)
Phrase 60.22 (2.1%) 63.45 (4.3%)
5.4% ( -1% - 12%)
HighTermMonthSort 63.83 (9.1%) 67.40 (10.7%)
5.6% ( -12% - 27%)
AndHighHigh 27.27 (3.4%) 28.80 (4.0%)
5.6% ( -1% - 13%)
SloppyPhrase1.35 (7.4%)1.43 (9.3%)
6.2% ( -9% - 24%)
AndHighOrMedMed 26.38 (1.7%) 28.17 (2.6%)
6.8% ( 2% - 11%)
IntervalsOrdered 21.99 (2.8%) 23.57 (1.9%)
7.2% ( 2% - 12%)
OrHighMed 39.27 (2.8%) 42.32 (3.0%)
7.8% ( 1% - 13%)
SpanNear9.74 (3.2%) 10.71 (2.0%)
10.0% ( 4% - 15%)
AndHighMed 59.40 (3.1%) 65.44 (3.6%)
10.2% ( 3% - 17%)
Wildcard 127.70 (7.7%) 140.93 (3.3%)
10.4% ( 0% - 23%)
AndMedOrHighHigh 30.65 (1.4%) 34.15 (2.1%)
11.4% ( 7% - 15%)
Prefix3 46.12 (9.9%) 53.01 (10.1%)
14.9% ( -4% - 38%)
{noformat}
I now plan to focus of getting this into a mergeable state.
> SIMD-based decoding of postings lists
> -
>
> Key: LUCENE-9027
> URL: https://issues.apache.org/jira/browse/LUCENE-9027
> Project: Lucene - Core
> Issue Type: Improvement
>Reporter: Adrien Grand
>Priority: Minor
> Time Spent: 10m
> Remaining Estimate: 0h
>
> [~rcmuir] has been mentioning the idea for quite some time that we might be
> able to write the decoding logic in such a way that Java would use SIMD
> instructions. More recently [~paul.masurel] wrote a [blog
> post|https://fulmicoton.com/posts/bitpacking/] that raises the point that
> Lucene could still do decode multiple ints at once in a single instruction by
> packing two ints in a long and we've had some discussions about what we could
> try in Lucene to speed up the decoding of postings. This made me want to look
> a bit deeper at what we could do.
> Our current decoding logic reads data in a byte[] and decodes packed integers
> from it. Unfortunately it doesn't make use of SIMD instructions and looks
> like
>
[jira] [Updated] (LUCENE-9042) Refactor TopGroups.merge tests
[
https://issues.apache.org/jira/browse/LUCENE-9042?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Diego Ceccarelli updated LUCENE-9042:
-
Attachment: (was: LUCENE-9042.patch)
> Refactor TopGroups.merge tests
> --
>
> Key: LUCENE-9042
> URL: https://issues.apache.org/jira/browse/LUCENE-9042
> Project: Lucene - Core
> Issue Type: Improvement
>Reporter: Diego Ceccarelli
>Priority: Minor
> Attachments: LUCENE-9042.patch
>
>
> This task proposes a refactoring of the test coverage for the
> {{TopGroups.merge}} method implemented in LUCENE-9010. For now it will cover
> only 3 main cases.
> 1. Merging to empty TopGroups
> 2. Merging a TopGroups with scores and a TopGroups without scores (currently
> broken because of LUCENE-8996 bug)
> 3. Merging two TopGroups with scores.
> I'm planning to increase the coverage testing also invalid inputs but I would
> do that in a separate PR to keep the code readable.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (LUCENE-9042) Refactor TopGroups.merge tests
[
https://issues.apache.org/jira/browse/LUCENE-9042?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Diego Ceccarelli updated LUCENE-9042:
-
Attachment: LUCENE-9042.patch
> Refactor TopGroups.merge tests
> --
>
> Key: LUCENE-9042
> URL: https://issues.apache.org/jira/browse/LUCENE-9042
> Project: Lucene - Core
> Issue Type: Improvement
>Reporter: Diego Ceccarelli
>Priority: Minor
> Attachments: LUCENE-9042.patch
>
>
> This task proposes a refactoring of the test coverage for the
> {{TopGroups.merge}} method implemented in LUCENE-9010. For now it will cover
> only 3 main cases.
> 1. Merging to empty TopGroups
> 2. Merging a TopGroups with scores and a TopGroups without scores (currently
> broken because of LUCENE-8996 bug)
> 3. Merging two TopGroups with scores.
> I'm planning to increase the coverage testing also invalid inputs but I would
> do that in a separate PR to keep the code readable.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9042) Refactor TopGroups.merge tests
[
https://issues.apache.org/jira/browse/LUCENE-9042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16973309#comment-16973309
]
Diego Ceccarelli commented on LUCENE-9042:
--
patch updated to fix the javac warnings.
> Refactor TopGroups.merge tests
> --
>
> Key: LUCENE-9042
> URL: https://issues.apache.org/jira/browse/LUCENE-9042
> Project: Lucene - Core
> Issue Type: Improvement
>Reporter: Diego Ceccarelli
>Priority: Minor
> Attachments: LUCENE-9042.patch
>
>
> This task proposes a refactoring of the test coverage for the
> {{TopGroups.merge}} method implemented in LUCENE-9010. For now it will cover
> only 3 main cases.
> 1. Merging to empty TopGroups
> 2. Merging a TopGroups with scores and a TopGroups without scores (currently
> broken because of LUCENE-8996 bug)
> 3. Merging two TopGroups with scores.
> I'm planning to increase the coverage testing also invalid inputs but I would
> do that in a separate PR to keep the code readable.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-13898) Non-atomic use of SolrCache get / put
[
https://issues.apache.org/jira/browse/SOLR-13898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16973242#comment-16973242
]
Andrzej Bialecki commented on SOLR-13898:
-
Hmm, I got an interesting test failure caused by the changes in
{{SolrIndexSearcher}}:
{code}
[junit4] 2> 1816820 ERROR (qtp46186557-36639) [n:127.0.0.1:60103_solr
c:org.apache.solr.search.facet.TestCloudJSONFacetJoinDomain_collection s:shard1
r:core_node2
x:org.apache.solr.search.facet.TestCloudJSONFacetJoinDomain_collection_shard1_replica_n1
] o.a.s.s.HttpSolrCall null:java.lang.IllegalStateException: Recursive update
[junit4] 2>at
java.base/java.util.concurrent.ConcurrentHashMap.putVal(ConcurrentHashMap.java:1063)
[junit4] 2>at
java.base/java.util.concurrent.ConcurrentHashMap.put(ConcurrentHashMap.java:1006)
[junit4] 2>at
org.apache.solr.util.ConcurrentLRUCache.putCacheEntry(ConcurrentLRUCache.java:264)
[junit4] 2>at
org.apache.solr.util.ConcurrentLRUCache.put(ConcurrentLRUCache.java:254)
[junit4] 2>at
org.apache.solr.search.FastLRUCache.put(FastLRUCache.java:240)
[junit4] 2>at
org.apache.solr.search.SolrIndexSearcher.getDocSet(SolrIndexSearcher.java:1198)
[junit4] 2>at
org.apache.solr.search.JoinQuery$JoinQueryWeight.getDocSetEnumerate(JoinQParserPlugin.java:442)
[junit4] 2>at
org.apache.solr.search.JoinQuery$JoinQueryWeight.getDocSet(JoinQParserPlugin.java:324)
[junit4] 2>at
org.apache.solr.search.JoinQuery$JoinQueryWeight.scorer(JoinQParserPlugin.java:253)
[junit4] 2>at
org.apache.lucene.search.Weight.bulkScorer(Weight.java:168)
[junit4] 2>at
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:741)
[junit4] 2>at
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:516)
[junit4] 2>at
org.apache.solr.search.DocSetUtil.createDocSetGeneric(DocSetUtil.java:151)
[junit4] 2>at
org.apache.solr.search.DocSetUtil.createDocSet(DocSetUtil.java:140)
[junit4] 2>at
org.apache.solr.search.SolrIndexSearcher.getDocSetNC(SolrIndexSearcher.java:1206)
[junit4] 2>at
org.apache.solr.search.SolrIndexSearcher.lambda$getDocSet$1(SolrIndexSearcher.java:812)
[junit4] 2>at
org.apache.solr.util.ConcurrentLRUCache.lambda$computeIfAbsent$1(ConcurrentLRUCache.java:223)
[junit4] 2>at
java.base/java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1705)
[junit4] 2>at
org.apache.solr.util.ConcurrentLRUCache.computeIfAbsent(ConcurrentLRUCache.java:222)
[junit4] 2>at
org.apache.solr.search.FastLRUCache.computeIfAbsent(FastLRUCache.java:255)
[junit4] 2>at
org.apache.solr.search.SolrIndexSearcher.getDocSet(SolrIndexSearcher.java:810)
[junit4] 2>at
org.apache.solr.search.facet.FacetProcessor.handleJoinField(FacetProcessor.java:244)
[junit4] 2>at
org.apache.solr.search.facet.FacetProcessor.handleDomainChanges(FacetProcessor.java:167)
[junit4] 2>at
org.apache.solr.search.facet.FacetProcessor.process(FacetProcessor.java:69)
[junit4] 2>at
org.apache.solr.search.facet.FacetFieldProcessorByArray.process(FacetFieldProcessorByArray.java:61)
[junit4] 2>at
org.apache.solr.search.facet.FacetRequest.process(FacetRequest.java:416)
[junit4] 2>at
org.apache.solr.search.facet.FacetProcessor.processSubs(FacetProcessor.java:475)
[junit4] 2>at
org.apache.solr.search.facet.FacetFieldProcessor.fillBucketFromSlot(FacetFieldProcessor.java:545)
[junit4] 2>at
org.apache.solr.search.facet.FacetFieldProcessor.findTopSlots(FacetFieldProcessor.java:446)
[junit4] 2>at
org.apache.solr.search.facet.FacetFieldProcessorByArray.calcFacets(FacetFieldProcessorByArray.java:114)
[junit4] 2>at
org.apache.solr.search.facet.FacetFieldProcessorByArray.process(FacetFieldProcessorByArray.java:62)
[junit4] 2>at
org.apache.solr.search.facet.FacetRequest.process(FacetRequest.java:416)
[junit4] 2>at
org.apache.solr.search.facet.FacetProcessor.processSubs(FacetProcessor.java:475)
[junit4] 2>at
org.apache.solr.search.facet.FacetProcessor.fillBucket(FacetProcessor.java:432)
[junit4] 2>at
org.apache.solr.search.facet.FacetQueryProcessor.process(FacetQuery.java:64)
[junit4] 2>at
org.apache.solr.search.facet.FacetRequest.process(FacetRequest.java:416)
[junit4] 2>at
org.apache.solr.search.facet.FacetModule.process(FacetModule.java:147)
[junit4] 2>at
org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:328)
[junit4] 2>at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:197)
{code}
I don't see any
[GitHub] [lucene-site] janhoy commented on issue #4: SOLR-13910 Add security pages news feeds for solr, core, and tlp
janhoy commented on issue #4: SOLR-13910 Add security pages news feeds for solr, core, and tlp URL: https://github.com/apache/lucene-site/pull/4#issuecomment-553320200 I could not see atom/rss enabled either. Should be very easy to do. I wonder if we should duplicate security announcements in the main solr news section or keep them separate? Or if we should just have one list of news, but with a `category` tag `news` or `security` and then select only security category for the /security.html page and the security RSS? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] bruno-roustant opened a new pull request #1007: LUCENE-9045: Do not use TreeMap/TreeSet in BlockTree and PerFieldPostingsFormat
bruno-roustant opened a new pull request #1007: LUCENE-9045: Do not use TreeMap/TreeSet in BlockTree and PerFieldPostingsFormat URL: https://github.com/apache/lucene-solr/pull/1007 The BlockTreeTermsReader.iterator() was detected by a profiler to spend abnormal time in TreeMap.keySet().iterator() iteration. The fix is to use a simple sorted list once the collection of fields is built. Same detection and fix for PerFieldPostingsFormat. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Created] (LUCENE-9045) Do not use TreeMap/TreeSet in BlockTree and PerFieldPostingsFormat
Bruno Roustant created LUCENE-9045: -- Summary: Do not use TreeMap/TreeSet in BlockTree and PerFieldPostingsFormat Key: LUCENE-9045 URL: https://issues.apache.org/jira/browse/LUCENE-9045 Project: Lucene - Core Issue Type: Improvement Reporter: Bruno Roustant TreeMap/TreeSet is a heavy structure designed to dynamically sort keys. It's iterator is much less performant than a list iterator. We should not use it when we don't need the sorting capability once built. And this is the case in BlockTreeTermsReader and PerFieldPostingsFormat. We need a Map and to sort keys at building time. But once built, we don't need to sort anymore, we can use a simple list for iteration efficiency. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9015) Configure branches, auto build and auto stage/publish
[
https://issues.apache.org/jira/browse/LUCENE-9015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16973165#comment-16973165
]
Jan Høydahl commented on LUCENE-9015:
-
Yahoo, we got our first build status email to the
[commits@l.a.o|mailto:commits@l.a.o] mailing list:
{noformat}
The Buildbot has detected a failed build on builder pelican_websites while
building lucene.
Full details are available at:
https://ci2.apache.org/#builders/3/builds/135
Buildbot URL: https://ci2.apache.org/
Worker for this Build: bb_slave9_ubuntu
Build Reason: Triggered pelican auto-build via .asf.yaml by janhoy
Blamelist: asfinfra, comm...@lucene.apache.org
BUILD FAILED: failed '/usr/local/bin/pelican-build.py --sourcetype ...'
(failure)
Sincerely,
-The Buildbot {noformat}
> Configure branches, auto build and auto stage/publish
> -
>
> Key: LUCENE-9015
> URL: https://issues.apache.org/jira/browse/LUCENE-9015
> Project: Lucene - Core
> Issue Type: Sub-task
>Reporter: Jan Høydahl
>Assignee: Jan Høydahl
>Priority: Major
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Commit to master should build and publish the staging site
> Find a simple way to trigger publishing of main site from staging
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] janhoy commented on a change in pull request #994: SOLR-13662: Package Manager (CLI)
janhoy commented on a change in pull request #994: SOLR-13662: Package Manager
(CLI)
URL: https://github.com/apache/lucene-solr/pull/994#discussion_r345644978
##
File path:
solr/core/src/test-files/solr/question-answer-repository/repository.json
##
@@ -0,0 +1,57 @@
+[
+ {
+"name": "question-answer",
+"description": "A natural language question answering plugin",
+"versions": [
+ {
+"version": "1.0.0",
+"date": "2019-01-01",
+"artifacts": [
+ {
+"url": "question-answer-request-handler-1.0.jar",
+"sig":
"C9UWKkucmY3UNzqn0VLneVMe9kCbJjw7Urc76vGenoRwp32xvNn5ZIGZ7G34xZP7cVjqn/ltDlLWBZ/C3eAtuw=="
+ }
+],
+"manifest": {
+ "min-solr-version": "8.0",
+ "max-solr-version": "9.99",
+ "plugins": [
+{
+ "name": "request-handler",
+ "setup-command": {
+"path": "/api/collections/${collection}/config",
Review comment:
Will you allow a package to run any HTTP command against the cluster? What
privilege does the pkg manager run under? Imagine someone posts a malicious
package "superduper-solr-package" somewhere that attempts to run a
setup-command `/api/authentication/disable` :-) or `/api/c/.system/update/foo`
or some other arbitrary command?
That's one reason I think we should turn this whole deploy upside down and
let Solr plugin points fetch info from the plugin instead of the plugin author
hardcoding some literal api commands against the cluster. ´
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] janhoy commented on a change in pull request #994: SOLR-13662: Package Manager (CLI)
janhoy commented on a change in pull request #994: SOLR-13662: Package Manager
(CLI)
URL: https://github.com/apache/lucene-solr/pull/994#discussion_r345639359
##
File path: solr/core/src/java/org/apache/solr/util/PackageTool.java
##
@@ -0,0 +1,255 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.util;
+
+import java.lang.invoke.MethodHandles;
+import java.util.Map;
+
+import org.apache.commons.cli.CommandLine;
+import org.apache.commons.cli.Option;
+import org.apache.commons.cli.OptionBuilder;
+import org.apache.http.impl.client.CloseableHttpClient;
+import org.apache.logging.log4j.Level;
+import org.apache.logging.log4j.core.config.Configurator;
+import org.apache.lucene.util.SuppressForbidden;
+import org.apache.solr.client.solrj.impl.HttpClientUtil;
+import org.apache.solr.client.solrj.impl.HttpSolrClient;
+import org.apache.solr.packagemanager.PackageManager;
+import org.apache.solr.packagemanager.PackageUtils;
+import org.apache.solr.packagemanager.RepositoryManager;
+import org.apache.solr.packagemanager.SolrPackageInstance;
+import org.apache.solr.util.SolrCLI.StatusTool;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+@SuppressForbidden(reason = "Need to use System.out.println() instead of
log4j/slf4j for cleaner output")
+public class PackageTool extends SolrCLI.ToolBase {
+
+ private static final Logger log =
LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+ @SuppressForbidden(reason = "Need to turn off logging, and SLF4J doesn't
seem to provide for a way.")
+ public PackageTool() {
+// Need a logging free, clean output going through to the user.
+Configurator.setRootLevel(Level.OFF);
+ }
+
+ @Override
+ public String getName() {
+return "package";
+ }
+
+ public static String solrUrl = null;
+ public static String solrBaseUrl = null;
+ public PackageManager packageManager;
+ public RepositoryManager repositoryManager;
+
+ @Override
+ protected void runImpl(CommandLine cli) throws Exception {
+try {
+ solrUrl =
cli.getOptionValues("solrUrl")[cli.getOptionValues("solrUrl").length-1];
+ solrBaseUrl = solrUrl.replaceAll("\\/solr$", ""); // strip out ending
"/solr"
+ log.info("Solr url: "+solrUrl+", solr base url: "+solrBaseUrl);
+ String zkHost = getZkHost(cli);
+
+ log.info("ZK: "+zkHost);
+ String cmd = cli.getArgList().size() == 0? "help": cli.getArgs()[0];
+
+ try (HttpSolrClient solrClient = new
HttpSolrClient.Builder(solrBaseUrl).build()) {
+if (cmd != null) {
+ packageManager = new PackageManager(solrClient, solrBaseUrl,
zkHost);
+ try {
+repositoryManager = new RepositoryManager(solrClient,
packageManager);
+
+switch (cmd) {
+ case "add-repo":
+repositoryManager.addRepository(cli.getArgs()[1],
cli.getArgs()[2]);
+break;
+ case "list-installed":
+packageManager.listInstalled();
+break;
+ case "list-available":
+repositoryManager.listAvailable();
+break;
+ case "list-deployed":
+if (cli.hasOption('c')) {
+ String collection = cli.getArgs()[1];
+ Map packages =
packageManager.getPackagesDeployed(collection);
+ PackageUtils.printGreen("Packages deployed on " + collection
+ ":");
+ for (String packageName: packages.keySet()) {
+PackageUtils.printGreen("\t" + packages.get(packageName));
+ }
+} else {
+ String packageName = cli.getArgs()[1];
+ Map deployedCollections =
packageManager.getDeployedCollections(packageName);
+ PackageUtils.printGreen("Collections on which package " +
packageName + " was deployed:");
+ for (String collection: deployedCollections.keySet()) {
+PackageUtils.printGreen("\t" + collection +
"("+packageName+":"+deployedCollections.get(collection)+")");
+ }
+}
+break;
+
[GitHub] [lucene-solr] janhoy commented on a change in pull request #994: SOLR-13662: Package Manager (CLI)
janhoy commented on a change in pull request #994: SOLR-13662: Package Manager
(CLI)
URL: https://github.com/apache/lucene-solr/pull/994#discussion_r345641481
##
File path: solr/core/src/java/org/apache/solr/util/PackageTool.java
##
@@ -0,0 +1,255 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.util;
+
+import java.lang.invoke.MethodHandles;
+import java.util.Map;
+
+import org.apache.commons.cli.CommandLine;
+import org.apache.commons.cli.Option;
+import org.apache.commons.cli.OptionBuilder;
+import org.apache.http.impl.client.CloseableHttpClient;
+import org.apache.logging.log4j.Level;
+import org.apache.logging.log4j.core.config.Configurator;
+import org.apache.lucene.util.SuppressForbidden;
+import org.apache.solr.client.solrj.impl.HttpClientUtil;
+import org.apache.solr.client.solrj.impl.HttpSolrClient;
+import org.apache.solr.packagemanager.PackageManager;
+import org.apache.solr.packagemanager.PackageUtils;
+import org.apache.solr.packagemanager.RepositoryManager;
+import org.apache.solr.packagemanager.SolrPackageInstance;
+import org.apache.solr.util.SolrCLI.StatusTool;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+@SuppressForbidden(reason = "Need to use System.out.println() instead of
log4j/slf4j for cleaner output")
+public class PackageTool extends SolrCLI.ToolBase {
+
+ private static final Logger log =
LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+ @SuppressForbidden(reason = "Need to turn off logging, and SLF4J doesn't
seem to provide for a way.")
+ public PackageTool() {
+// Need a logging free, clean output going through to the user.
+Configurator.setRootLevel(Level.OFF);
+ }
+
+ @Override
+ public String getName() {
+return "package";
+ }
+
+ public static String solrUrl = null;
+ public static String solrBaseUrl = null;
+ public PackageManager packageManager;
+ public RepositoryManager repositoryManager;
+
+ @Override
+ protected void runImpl(CommandLine cli) throws Exception {
+try {
+ solrUrl =
cli.getOptionValues("solrUrl")[cli.getOptionValues("solrUrl").length-1];
+ solrBaseUrl = solrUrl.replaceAll("\\/solr$", ""); // strip out ending
"/solr"
+ log.info("Solr url: "+solrUrl+", solr base url: "+solrBaseUrl);
+ String zkHost = getZkHost(cli);
+
+ log.info("ZK: "+zkHost);
+ String cmd = cli.getArgList().size() == 0? "help": cli.getArgs()[0];
+
+ try (HttpSolrClient solrClient = new
HttpSolrClient.Builder(solrBaseUrl).build()) {
+if (cmd != null) {
+ packageManager = new PackageManager(solrClient, solrBaseUrl,
zkHost);
+ try {
+repositoryManager = new RepositoryManager(solrClient,
packageManager);
+
+switch (cmd) {
+ case "add-repo":
+repositoryManager.addRepository(cli.getArgs()[1],
cli.getArgs()[2]);
+break;
+ case "list-installed":
+packageManager.listInstalled();
+break;
+ case "list-available":
+repositoryManager.listAvailable();
+break;
+ case "list-deployed":
+if (cli.hasOption('c')) {
+ String collection = cli.getArgs()[1];
+ Map packages =
packageManager.getPackagesDeployed(collection);
+ PackageUtils.printGreen("Packages deployed on " + collection
+ ":");
+ for (String packageName: packages.keySet()) {
+PackageUtils.printGreen("\t" + packages.get(packageName));
+ }
+} else {
+ String packageName = cli.getArgs()[1];
+ Map deployedCollections =
packageManager.getDeployedCollections(packageName);
+ PackageUtils.printGreen("Collections on which package " +
packageName + " was deployed:");
+ for (String collection: deployedCollections.keySet()) {
+PackageUtils.printGreen("\t" + collection +
"("+packageName+":"+deployedCollections.get(collection)+")");
+ }
+}
+break;
+
[GitHub] [lucene-solr] janhoy commented on a change in pull request #994: SOLR-13662: Package Manager (CLI)
janhoy commented on a change in pull request #994: SOLR-13662: Package Manager
(CLI)
URL: https://github.com/apache/lucene-solr/pull/994#discussion_r345633566
##
File path:
solr/core/src/java/org/apache/solr/packagemanager/RepositoryManager.java
##
@@ -0,0 +1,328 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.solr.packagemanager;
+
+import static org.apache.solr.packagemanager.PackageUtils.getMapper;
+
+import java.io.IOException;
+import java.io.UnsupportedEncodingException;
+import java.lang.invoke.MethodHandles;
+import java.net.MalformedURLException;
+import java.net.URL;
+import java.nio.ByteBuffer;
+import java.nio.file.Path;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+
+import org.apache.commons.io.FileUtils;
+import org.apache.commons.io.IOUtils;
+import org.apache.lucene.util.Version;
+import org.apache.solr.client.solrj.SolrRequest;
+import org.apache.solr.client.solrj.SolrServerException;
+import org.apache.solr.client.solrj.impl.HttpSolrClient;
+import org.apache.solr.client.solrj.request.V2Request;
+import org.apache.solr.client.solrj.request.beans.Package;
+import org.apache.solr.client.solrj.response.V2Response;
+import org.apache.solr.common.SolrException;
+import org.apache.solr.common.SolrException.ErrorCode;
+import org.apache.solr.common.cloud.SolrZkClient;
+import org.apache.solr.core.BlobRepository;
+import org.apache.solr.packagemanager.SolrPackage.Artifact;
+import org.apache.solr.packagemanager.SolrPackage.SolrPackageRelease;
+import org.apache.solr.pkg.PackageAPI;
+import org.apache.zookeeper.CreateMode;
+import org.apache.zookeeper.KeeperException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+/**
+ * Handles most of the management of repositories and packages present in
external repositories.
+ */
+public class RepositoryManager {
+
+ private static final Logger log =
LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+ final private PackageManager packageManager;
+
+ public static final String systemVersion = Version.LATEST.toString();
+
+ final HttpSolrClient solrClient;
+
+ public RepositoryManager(HttpSolrClient solrClient, PackageManager
packageManager) {
+this.packageManager = packageManager;
+this.solrClient = solrClient;
+ }
+
+ public List getPackages() {
+List list = new ArrayList<>(getPackagesMap().values());
+Collections.sort(list);
+return list;
+ }
+
+ /**
+ * Get a map of package name to {@link SolrPackage} objects
+ */
+ public Map getPackagesMap() {
+Map packagesMap = new HashMap<>();
+for (PackageRepository repository: getRepositories()) {
+ packagesMap.putAll(repository.getPackages());
+}
+
+return packagesMap;
+ }
+
+ /**
+ * List of added repositories
+ */
+ public List getRepositories() {
+// TODO: Instead of fetching again and again, we should look for caching
this
+PackageRepository items[];
+try {
+ items =
getMapper().readValue(getRepositoriesJson(packageManager.zkClient),
DefaultPackageRepository[].class);
+} catch (IOException | KeeperException | InterruptedException e) {
+ throw new SolrException(ErrorCode.SERVER_ERROR, e);
+}
+List repositories = Arrays.asList(items);
+
+for (PackageRepository updateRepository: repositories) {
+ updateRepository.refresh();
+}
+
+return repositories;
+ }
+
+ /**
+ * Add a repository to Solr
+ */
+ public void addRepository(String name, String uri) throws KeeperException,
InterruptedException, MalformedURLException, IOException {
+String existingRepositoriesJson =
getRepositoriesJson(packageManager.zkClient);
+log.info(existingRepositoriesJson);
+
+List repos = getMapper().readValue(existingRepositoriesJson, List.class);
+repos.add(new DefaultPackageRepository(name, uri));
+if (packageManager.zkClient.exists("/repositories.json", true) == false) {
+ packageManager.zkClient.create("/repositories.json",
getMapper().writeValueAsString(repos).getBytes("UTF-8"), CreateMode.PERSISTENT,
[GitHub] [lucene-solr] janhoy commented on a change in pull request #994: SOLR-13662: Package Manager (CLI)
janhoy commented on a change in pull request #994: SOLR-13662: Package Manager
(CLI)
URL: https://github.com/apache/lucene-solr/pull/994#discussion_r345632143
##
File path:
solr/core/src/java/org/apache/solr/packagemanager/RepositoryManager.java
##
@@ -0,0 +1,328 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.solr.packagemanager;
+
+import static org.apache.solr.packagemanager.PackageUtils.getMapper;
+
+import java.io.IOException;
+import java.io.UnsupportedEncodingException;
+import java.lang.invoke.MethodHandles;
+import java.net.MalformedURLException;
+import java.net.URL;
+import java.nio.ByteBuffer;
+import java.nio.file.Path;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+
+import org.apache.commons.io.FileUtils;
+import org.apache.commons.io.IOUtils;
+import org.apache.lucene.util.Version;
+import org.apache.solr.client.solrj.SolrRequest;
+import org.apache.solr.client.solrj.SolrServerException;
+import org.apache.solr.client.solrj.impl.HttpSolrClient;
+import org.apache.solr.client.solrj.request.V2Request;
+import org.apache.solr.client.solrj.request.beans.Package;
+import org.apache.solr.client.solrj.response.V2Response;
+import org.apache.solr.common.SolrException;
+import org.apache.solr.common.SolrException.ErrorCode;
+import org.apache.solr.common.cloud.SolrZkClient;
+import org.apache.solr.core.BlobRepository;
+import org.apache.solr.packagemanager.SolrPackage.Artifact;
+import org.apache.solr.packagemanager.SolrPackage.SolrPackageRelease;
+import org.apache.solr.pkg.PackageAPI;
+import org.apache.zookeeper.CreateMode;
+import org.apache.zookeeper.KeeperException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+/**
+ * Handles most of the management of repositories and packages present in
external repositories.
+ */
+public class RepositoryManager {
+
+ private static final Logger log =
LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+ final private PackageManager packageManager;
+
+ public static final String systemVersion = Version.LATEST.toString();
+
+ final HttpSolrClient solrClient;
+
+ public RepositoryManager(HttpSolrClient solrClient, PackageManager
packageManager) {
+this.packageManager = packageManager;
+this.solrClient = solrClient;
+ }
+
+ public List getPackages() {
+List list = new ArrayList<>(getPackagesMap().values());
+Collections.sort(list);
+return list;
+ }
+
+ /**
+ * Get a map of package name to {@link SolrPackage} objects
+ */
+ public Map getPackagesMap() {
+Map packagesMap = new HashMap<>();
+for (PackageRepository repository: getRepositories()) {
+ packagesMap.putAll(repository.getPackages());
+}
+
+return packagesMap;
+ }
+
+ /**
+ * List of added repositories
+ */
+ public List getRepositories() {
+// TODO: Instead of fetching again and again, we should look for caching
this
+PackageRepository items[];
+try {
+ items =
getMapper().readValue(getRepositoriesJson(packageManager.zkClient),
DefaultPackageRepository[].class);
+} catch (IOException | KeeperException | InterruptedException e) {
+ throw new SolrException(ErrorCode.SERVER_ERROR, e);
+}
+List repositories = Arrays.asList(items);
+
+for (PackageRepository updateRepository: repositories) {
+ updateRepository.refresh();
+}
+
+return repositories;
+ }
+
+ /**
+ * Add a repository to Solr
+ */
+ public void addRepository(String name, String uri) throws KeeperException,
InterruptedException, MalformedURLException, IOException {
+String existingRepositoriesJson =
getRepositoriesJson(packageManager.zkClient);
+log.info(existingRepositoriesJson);
+
+List repos = getMapper().readValue(existingRepositoriesJson, List.class);
+repos.add(new DefaultPackageRepository(name, uri));
+if (packageManager.zkClient.exists("/repositories.json", true) == false) {
+ packageManager.zkClient.create("/repositories.json",
getMapper().writeValueAsString(repos).getBytes("UTF-8"), CreateMode.PERSISTENT,
[GitHub] [lucene-solr] janhoy commented on a change in pull request #994: SOLR-13662: Package Manager (CLI)
janhoy commented on a change in pull request #994: SOLR-13662: Package Manager
(CLI)
URL: https://github.com/apache/lucene-solr/pull/994#discussion_r345634253
##
File path:
solr/core/src/java/org/apache/solr/packagemanager/RepositoryManager.java
##
@@ -0,0 +1,328 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.solr.packagemanager;
+
+import static org.apache.solr.packagemanager.PackageUtils.getMapper;
+
+import java.io.IOException;
+import java.io.UnsupportedEncodingException;
+import java.lang.invoke.MethodHandles;
+import java.net.MalformedURLException;
+import java.net.URL;
+import java.nio.ByteBuffer;
+import java.nio.file.Path;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+
+import org.apache.commons.io.FileUtils;
+import org.apache.commons.io.IOUtils;
+import org.apache.lucene.util.Version;
+import org.apache.solr.client.solrj.SolrRequest;
+import org.apache.solr.client.solrj.SolrServerException;
+import org.apache.solr.client.solrj.impl.HttpSolrClient;
+import org.apache.solr.client.solrj.request.V2Request;
+import org.apache.solr.client.solrj.request.beans.Package;
+import org.apache.solr.client.solrj.response.V2Response;
+import org.apache.solr.common.SolrException;
+import org.apache.solr.common.SolrException.ErrorCode;
+import org.apache.solr.common.cloud.SolrZkClient;
+import org.apache.solr.core.BlobRepository;
+import org.apache.solr.packagemanager.SolrPackage.Artifact;
+import org.apache.solr.packagemanager.SolrPackage.SolrPackageRelease;
+import org.apache.solr.pkg.PackageAPI;
+import org.apache.zookeeper.CreateMode;
+import org.apache.zookeeper.KeeperException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+/**
+ * Handles most of the management of repositories and packages present in
external repositories.
+ */
+public class RepositoryManager {
+
+ private static final Logger log =
LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+ final private PackageManager packageManager;
+
+ public static final String systemVersion = Version.LATEST.toString();
+
+ final HttpSolrClient solrClient;
+
+ public RepositoryManager(HttpSolrClient solrClient, PackageManager
packageManager) {
+this.packageManager = packageManager;
+this.solrClient = solrClient;
+ }
+
+ public List getPackages() {
+List list = new ArrayList<>(getPackagesMap().values());
+Collections.sort(list);
+return list;
+ }
+
+ /**
+ * Get a map of package name to {@link SolrPackage} objects
+ */
+ public Map getPackagesMap() {
+Map packagesMap = new HashMap<>();
+for (PackageRepository repository: getRepositories()) {
+ packagesMap.putAll(repository.getPackages());
+}
+
+return packagesMap;
+ }
+
+ /**
+ * List of added repositories
+ */
+ public List getRepositories() {
+// TODO: Instead of fetching again and again, we should look for caching
this
+PackageRepository items[];
+try {
+ items =
getMapper().readValue(getRepositoriesJson(packageManager.zkClient),
DefaultPackageRepository[].class);
+} catch (IOException | KeeperException | InterruptedException e) {
+ throw new SolrException(ErrorCode.SERVER_ERROR, e);
+}
+List repositories = Arrays.asList(items);
+
+for (PackageRepository updateRepository: repositories) {
+ updateRepository.refresh();
+}
+
+return repositories;
+ }
+
+ /**
+ * Add a repository to Solr
+ */
+ public void addRepository(String name, String uri) throws KeeperException,
InterruptedException, MalformedURLException, IOException {
+String existingRepositoriesJson =
getRepositoriesJson(packageManager.zkClient);
+log.info(existingRepositoriesJson);
+
+List repos = getMapper().readValue(existingRepositoriesJson, List.class);
+repos.add(new DefaultPackageRepository(name, uri));
+if (packageManager.zkClient.exists("/repositories.json", true) == false) {
+ packageManager.zkClient.create("/repositories.json",
getMapper().writeValueAsString(repos).getBytes("UTF-8"), CreateMode.PERSISTENT,
[GitHub] [lucene-solr] janhoy commented on a change in pull request #994: SOLR-13662: Package Manager (CLI)
janhoy commented on a change in pull request #994: SOLR-13662: Package Manager
(CLI)
URL: https://github.com/apache/lucene-solr/pull/994#discussion_r345636779
##
File path: solr/core/src/java/org/apache/solr/packagemanager/SolrPackage.java
##
@@ -0,0 +1,143 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.solr.packagemanager;
+
+
+import java.util.Date;
+import java.util.List;
+import java.util.Map;
+
+import org.apache.solr.common.annotation.JsonProperty;
+import org.apache.solr.common.util.ReflectMapWriter;
+
+/**
+ * Describes a package (along with all released versions) as it appears in a
repository.
+ */
+public class SolrPackage implements Comparable, ReflectMapWriter {
+
+ @JsonProperty("name")
+ public String name;
+
+ @JsonProperty("description")
+ public String description;
+
+ @JsonProperty("versions")
+ public List versions;
+
+ @JsonProperty("repository")
+ private String repository;
+
+ @Override
+ public String toString() {
+return jsonStr();
+ }
+
+ public static class SolrPackageRelease implements ReflectMapWriter {
+@JsonProperty("version")
+public String version;
+
+@JsonProperty("date")
+public Date date;
+
+@JsonProperty("artifacts")
+public List artifacts;
+
+@JsonProperty("manifest")
Review comment:
Won't this class represent the content of `manifest.json`? In that case it
could be misleading to have a property `manifest` inside? Perhaps I
misunderstand?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] janhoy commented on a change in pull request #994: SOLR-13662: Package Manager (CLI)
janhoy commented on a change in pull request #994: SOLR-13662: Package Manager
(CLI)
URL: https://github.com/apache/lucene-solr/pull/994#discussion_r345633116
##
File path:
solr/core/src/java/org/apache/solr/packagemanager/RepositoryManager.java
##
@@ -0,0 +1,328 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.solr.packagemanager;
+
+import static org.apache.solr.packagemanager.PackageUtils.getMapper;
+
+import java.io.IOException;
+import java.io.UnsupportedEncodingException;
+import java.lang.invoke.MethodHandles;
+import java.net.MalformedURLException;
+import java.net.URL;
+import java.nio.ByteBuffer;
+import java.nio.file.Path;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+
+import org.apache.commons.io.FileUtils;
+import org.apache.commons.io.IOUtils;
+import org.apache.lucene.util.Version;
+import org.apache.solr.client.solrj.SolrRequest;
+import org.apache.solr.client.solrj.SolrServerException;
+import org.apache.solr.client.solrj.impl.HttpSolrClient;
+import org.apache.solr.client.solrj.request.V2Request;
+import org.apache.solr.client.solrj.request.beans.Package;
+import org.apache.solr.client.solrj.response.V2Response;
+import org.apache.solr.common.SolrException;
+import org.apache.solr.common.SolrException.ErrorCode;
+import org.apache.solr.common.cloud.SolrZkClient;
+import org.apache.solr.core.BlobRepository;
+import org.apache.solr.packagemanager.SolrPackage.Artifact;
+import org.apache.solr.packagemanager.SolrPackage.SolrPackageRelease;
+import org.apache.solr.pkg.PackageAPI;
+import org.apache.zookeeper.CreateMode;
+import org.apache.zookeeper.KeeperException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+/**
+ * Handles most of the management of repositories and packages present in
external repositories.
+ */
+public class RepositoryManager {
+
+ private static final Logger log =
LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+ final private PackageManager packageManager;
+
+ public static final String systemVersion = Version.LATEST.toString();
+
+ final HttpSolrClient solrClient;
+
+ public RepositoryManager(HttpSolrClient solrClient, PackageManager
packageManager) {
+this.packageManager = packageManager;
+this.solrClient = solrClient;
+ }
+
+ public List getPackages() {
+List list = new ArrayList<>(getPackagesMap().values());
+Collections.sort(list);
+return list;
+ }
+
+ /**
+ * Get a map of package name to {@link SolrPackage} objects
+ */
+ public Map getPackagesMap() {
+Map packagesMap = new HashMap<>();
+for (PackageRepository repository: getRepositories()) {
+ packagesMap.putAll(repository.getPackages());
+}
+
+return packagesMap;
+ }
+
+ /**
+ * List of added repositories
+ */
+ public List getRepositories() {
+// TODO: Instead of fetching again and again, we should look for caching
this
+PackageRepository items[];
+try {
+ items =
getMapper().readValue(getRepositoriesJson(packageManager.zkClient),
DefaultPackageRepository[].class);
+} catch (IOException | KeeperException | InterruptedException e) {
+ throw new SolrException(ErrorCode.SERVER_ERROR, e);
+}
+List repositories = Arrays.asList(items);
+
+for (PackageRepository updateRepository: repositories) {
+ updateRepository.refresh();
+}
+
+return repositories;
+ }
+
+ /**
+ * Add a repository to Solr
+ */
+ public void addRepository(String name, String uri) throws KeeperException,
InterruptedException, MalformedURLException, IOException {
+String existingRepositoriesJson =
getRepositoriesJson(packageManager.zkClient);
+log.info(existingRepositoriesJson);
+
+List repos = getMapper().readValue(existingRepositoriesJson, List.class);
+repos.add(new DefaultPackageRepository(name, uri));
+if (packageManager.zkClient.exists("/repositories.json", true) == false) {
+ packageManager.zkClient.create("/repositories.json",
getMapper().writeValueAsString(repos).getBytes("UTF-8"), CreateMode.PERSISTENT,
[GitHub] [lucene-solr] janhoy commented on a change in pull request #994: SOLR-13662: Package Manager (CLI)
janhoy commented on a change in pull request #994: SOLR-13662: Package Manager
(CLI)
URL: https://github.com/apache/lucene-solr/pull/994#discussion_r345629730
##
File path: solr/core/src/java/org/apache/solr/packagemanager/SolrPackage.java
##
@@ -0,0 +1,143 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.solr.packagemanager;
+
+
+import java.util.Date;
+import java.util.List;
+import java.util.Map;
+
+import org.apache.solr.common.annotation.JsonProperty;
+import org.apache.solr.common.util.ReflectMapWriter;
+
+/**
+ * Describes a package (along with all released versions) as it appears in a
repository.
+ */
+public class SolrPackage implements Comparable, ReflectMapWriter {
+
+ @JsonProperty("name")
+ public String name;
+
+ @JsonProperty("description")
+ public String description;
+
+ @JsonProperty("versions")
+ public List versions;
+
+ @JsonProperty("repository")
+ private String repository;
+
+ @Override
+ public String toString() {
+return jsonStr();
+ }
+
+ public static class SolrPackageRelease implements ReflectMapWriter {
+@JsonProperty("version")
+public String version;
+
+@JsonProperty("date")
+public Date date;
+
+@JsonProperty("artifacts")
+public List artifacts;
+
+@JsonProperty("manifest")
+public Manifest manifest;
+
+@Override
+public String toString() {
+ return jsonStr();
+}
+ }
+
+ public static class Artifact implements ReflectMapWriter {
+@JsonProperty("url")
+public String url;
+
+@JsonProperty("sig")
+public String sig;
+ }
+
+ public static class Manifest implements ReflectMapWriter {
+@JsonProperty("min-solr-version")
Review comment:
`min-solr-version` and `max-solr-version` can be replaced by a SemVer
expression (https://devhints.io/semver) which would allow a simple `8` to say
you are compatible with all 8.x, or `*` to allow all etc.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] janhoy commented on a change in pull request #994: SOLR-13662: Package Manager (CLI)
janhoy commented on a change in pull request #994: SOLR-13662: Package Manager
(CLI)
URL: https://github.com/apache/lucene-solr/pull/994#discussion_r345631027
##
File path:
solr/core/src/java/org/apache/solr/packagemanager/RepositoryManager.java
##
@@ -0,0 +1,328 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.solr.packagemanager;
+
+import static org.apache.solr.packagemanager.PackageUtils.getMapper;
+
+import java.io.IOException;
+import java.io.UnsupportedEncodingException;
+import java.lang.invoke.MethodHandles;
+import java.net.MalformedURLException;
+import java.net.URL;
+import java.nio.ByteBuffer;
+import java.nio.file.Path;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+
+import org.apache.commons.io.FileUtils;
+import org.apache.commons.io.IOUtils;
+import org.apache.lucene.util.Version;
+import org.apache.solr.client.solrj.SolrRequest;
+import org.apache.solr.client.solrj.SolrServerException;
+import org.apache.solr.client.solrj.impl.HttpSolrClient;
+import org.apache.solr.client.solrj.request.V2Request;
+import org.apache.solr.client.solrj.request.beans.Package;
+import org.apache.solr.client.solrj.response.V2Response;
+import org.apache.solr.common.SolrException;
+import org.apache.solr.common.SolrException.ErrorCode;
+import org.apache.solr.common.cloud.SolrZkClient;
+import org.apache.solr.core.BlobRepository;
+import org.apache.solr.packagemanager.SolrPackage.Artifact;
+import org.apache.solr.packagemanager.SolrPackage.SolrPackageRelease;
+import org.apache.solr.pkg.PackageAPI;
+import org.apache.zookeeper.CreateMode;
+import org.apache.zookeeper.KeeperException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+/**
+ * Handles most of the management of repositories and packages present in
external repositories.
+ */
+public class RepositoryManager {
+
+ private static final Logger log =
LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+ final private PackageManager packageManager;
+
+ public static final String systemVersion = Version.LATEST.toString();
+
+ final HttpSolrClient solrClient;
+
+ public RepositoryManager(HttpSolrClient solrClient, PackageManager
packageManager) {
+this.packageManager = packageManager;
+this.solrClient = solrClient;
+ }
+
+ public List getPackages() {
+List list = new ArrayList<>(getPackagesMap().values());
+Collections.sort(list);
+return list;
+ }
+
+ /**
+ * Get a map of package name to {@link SolrPackage} objects
+ */
+ public Map getPackagesMap() {
+Map packagesMap = new HashMap<>();
+for (PackageRepository repository: getRepositories()) {
+ packagesMap.putAll(repository.getPackages());
+}
+
+return packagesMap;
+ }
+
+ /**
+ * List of added repositories
+ */
+ public List getRepositories() {
+// TODO: Instead of fetching again and again, we should look for caching
this
+PackageRepository items[];
+try {
+ items =
getMapper().readValue(getRepositoriesJson(packageManager.zkClient),
DefaultPackageRepository[].class);
+} catch (IOException | KeeperException | InterruptedException e) {
+ throw new SolrException(ErrorCode.SERVER_ERROR, e);
+}
+List repositories = Arrays.asList(items);
+
+for (PackageRepository updateRepository: repositories) {
+ updateRepository.refresh();
+}
+
+return repositories;
+ }
+
+ /**
+ * Add a repository to Solr
+ */
+ public void addRepository(String name, String uri) throws KeeperException,
InterruptedException, MalformedURLException, IOException {
+String existingRepositoriesJson =
getRepositoriesJson(packageManager.zkClient);
+log.info(existingRepositoriesJson);
+
+List repos = getMapper().readValue(existingRepositoriesJson, List.class);
+repos.add(new DefaultPackageRepository(name, uri));
+if (packageManager.zkClient.exists("/repositories.json", true) == false) {
+ packageManager.zkClient.create("/repositories.json",
getMapper().writeValueAsString(repos).getBytes("UTF-8"), CreateMode.PERSISTENT,
[GitHub] [lucene-solr] janhoy commented on a change in pull request #994: SOLR-13662: Package Manager (CLI)
janhoy commented on a change in pull request #994: SOLR-13662: Package Manager
(CLI)
URL: https://github.com/apache/lucene-solr/pull/994#discussion_r345635689
##
File path:
solr/core/src/java/org/apache/solr/packagemanager/RepositoryManager.java
##
@@ -0,0 +1,328 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.solr.packagemanager;
+
+import static org.apache.solr.packagemanager.PackageUtils.getMapper;
+
+import java.io.IOException;
+import java.io.UnsupportedEncodingException;
+import java.lang.invoke.MethodHandles;
+import java.net.MalformedURLException;
+import java.net.URL;
+import java.nio.ByteBuffer;
+import java.nio.file.Path;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+
+import org.apache.commons.io.FileUtils;
+import org.apache.commons.io.IOUtils;
+import org.apache.lucene.util.Version;
+import org.apache.solr.client.solrj.SolrRequest;
+import org.apache.solr.client.solrj.SolrServerException;
+import org.apache.solr.client.solrj.impl.HttpSolrClient;
+import org.apache.solr.client.solrj.request.V2Request;
+import org.apache.solr.client.solrj.request.beans.Package;
+import org.apache.solr.client.solrj.response.V2Response;
+import org.apache.solr.common.SolrException;
+import org.apache.solr.common.SolrException.ErrorCode;
+import org.apache.solr.common.cloud.SolrZkClient;
+import org.apache.solr.core.BlobRepository;
+import org.apache.solr.packagemanager.SolrPackage.Artifact;
+import org.apache.solr.packagemanager.SolrPackage.SolrPackageRelease;
+import org.apache.solr.pkg.PackageAPI;
+import org.apache.zookeeper.CreateMode;
+import org.apache.zookeeper.KeeperException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+/**
+ * Handles most of the management of repositories and packages present in
external repositories.
+ */
+public class RepositoryManager {
+
+ private static final Logger log =
LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+ final private PackageManager packageManager;
+
+ public static final String systemVersion = Version.LATEST.toString();
+
+ final HttpSolrClient solrClient;
+
+ public RepositoryManager(HttpSolrClient solrClient, PackageManager
packageManager) {
+this.packageManager = packageManager;
+this.solrClient = solrClient;
+ }
+
+ public List getPackages() {
+List list = new ArrayList<>(getPackagesMap().values());
+Collections.sort(list);
+return list;
+ }
+
+ /**
+ * Get a map of package name to {@link SolrPackage} objects
+ */
+ public Map getPackagesMap() {
+Map packagesMap = new HashMap<>();
+for (PackageRepository repository: getRepositories()) {
+ packagesMap.putAll(repository.getPackages());
+}
+
+return packagesMap;
+ }
+
+ /**
+ * List of added repositories
+ */
+ public List getRepositories() {
+// TODO: Instead of fetching again and again, we should look for caching
this
+PackageRepository items[];
+try {
+ items =
getMapper().readValue(getRepositoriesJson(packageManager.zkClient),
DefaultPackageRepository[].class);
+} catch (IOException | KeeperException | InterruptedException e) {
+ throw new SolrException(ErrorCode.SERVER_ERROR, e);
+}
+List repositories = Arrays.asList(items);
+
+for (PackageRepository updateRepository: repositories) {
+ updateRepository.refresh();
+}
+
+return repositories;
+ }
+
+ /**
+ * Add a repository to Solr
+ */
+ public void addRepository(String name, String uri) throws KeeperException,
InterruptedException, MalformedURLException, IOException {
+String existingRepositoriesJson =
getRepositoriesJson(packageManager.zkClient);
+log.info(existingRepositoriesJson);
+
+List repos = getMapper().readValue(existingRepositoriesJson, List.class);
+repos.add(new DefaultPackageRepository(name, uri));
+if (packageManager.zkClient.exists("/repositories.json", true) == false) {
+ packageManager.zkClient.create("/repositories.json",
getMapper().writeValueAsString(repos).getBytes("UTF-8"), CreateMode.PERSISTENT,
[jira] [Commented] (SOLR-13911) Support missing() aggregation in JSON facet module
[
https://issues.apache.org/jira/browse/SOLR-13911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16973122#comment-16973122
]
Lucene/Solr QA commented on SOLR-13911:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
5s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} Release audit (RAT) {color} |
{color:green} 1m 5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} Check forbidden APIs {color} |
{color:green} 1m 1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} Validate source patterns {color} |
{color:green} 1m 1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} Validate ref guide {color} |
{color:green} 1m 1s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 47m
13s{color} | {color:green} core in the patch passed. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 51m 51s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | SOLR-13911 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12985641/SOLR-13911.patch |
| Optional Tests | compile javac unit ratsources checkforbiddenapis
validatesourcepatterns validaterefguide |
| uname | Linux lucene1-us-west 4.15.0-54-generic #58-Ubuntu SMP Mon Jun 24
10:55:24 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | ant |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-SOLR-Build/sourcedir/dev-tools/test-patch/lucene-solr-yetus-personality.sh
|
| git revision | master / 550c7296b6f |
| ant | version: Apache Ant(TM) version 1.10.5 compiled on March 28 2019 |
| Default Java | LTS |
| Test Results |
https://builds.apache.org/job/PreCommit-SOLR-Build/596/testReport/ |
| modules | C: solr/core solr/solr-ref-guide U: solr |
| Console output |
https://builds.apache.org/job/PreCommit-SOLR-Build/596/console |
| Powered by | Apache Yetus 0.7.0 http://yetus.apache.org |
This message was automatically generated.
> Support missing() aggregation in JSON facet module
> --
>
> Key: SOLR-13911
> URL: https://issues.apache.org/jira/browse/SOLR-13911
> Project: Solr
> Issue Type: Sub-task
> Components: Facet Module
>Reporter: Munendra S N
>Priority: Major
> Attachments: SOLR-11695.patch, SOLR-13911.patch, SOLR-13911.patch,
> SOLR-13911.patch
>
>
> Add {{missing()}} aggregation in JSON Facet module similar to
> StatsComponent's missing
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9015) Configure branches, auto build and auto stage/publish
[
https://issues.apache.org/jira/browse/LUCENE-9015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16973110#comment-16973110
]
Jan Høydahl commented on LUCENE-9015:
-
Guess you can try notify now, the wiki docs are updated.
{quote}Daniel Gruno 11:15 PM
@Jan Høydahl email should work now
I'll get the cwiki doc updated with how to set it
TL;DR: you add notify: d...@lucene.apache.org or similar to the pelican yaml
section
{quote}
I kicked off a build yesterday but no new branch, so I created an empty branch
manually now and pushed a small README edit to see if that works better.
> Configure branches, auto build and auto stage/publish
> -
>
> Key: LUCENE-9015
> URL: https://issues.apache.org/jira/browse/LUCENE-9015
> Project: Lucene - Core
> Issue Type: Sub-task
>Reporter: Jan Høydahl
>Assignee: Jan Høydahl
>Priority: Major
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Commit to master should build and publish the staging site
> Find a simple way to trigger publishing of main site from staging
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-13821) Package Store
[
https://issues.apache.org/jira/browse/SOLR-13821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16973097#comment-16973097
]
ASF subversion and git services commented on SOLR-13821:
Commit 59cc299c7e1fe307af564f820454164c7462858c in lucene-solr's branch
refs/heads/master from Noble Paul
[ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=59cc299 ]
SOLR-13821: Return the size of the file
> Package Store
> -
>
> Key: SOLR-13821
> URL: https://issues.apache.org/jira/browse/SOLR-13821
> Project: Solr
> Issue Type: Sub-task
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Ishan Chattopadhyaya
>Assignee: Noble Paul
>Priority: Major
> Fix For: 8.4
>
> Time Spent: 2h
> Remaining Estimate: 0h
>
> Package store is a storage managed by Solr that holds the package artifacts.
> This is replicated across nodes.
> Design is here:
> [https://docs.google.com/document/d/15b3m3i3NFDKbhkhX_BN0MgvPGZaBj34TKNF2-UNC3U8/edit?ts=5d86a8ad#]
> The package store is powered by an underlying filestore. This filestore is a
> fully replicated p2p filesystem storage for artifacts.
> The APIs are as follows
> {code:java}
> # add a file
> POST /api/cluster/files/path/to/file.jar
> #retrieve a file
> GET /api/cluster/files/path/to/file.jar
> #list files in the /path/to directory
> GET /api/cluster/files/path/to
> #GET meta info of the jar
> GET /api/cluster/files/path/to/file.jar?meta=true
> {code}
> This store keeps 2 files per file
> # The actual file say {{myplugin.jar}}
> # A metadata file {{.myplugin.jar.json}} in the same directory
> The contenbts of the metadata file is
> {code:json}
> {
> "sha512" : ""
> "sig": {
> "" :""
> }}
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
