Re: [jira] [Commented] (SOLR-11528) ltr unit tests failed
Hello Christine Poerschke
My steps are as follows：
1.ant eclipse
2.import the project
3.clean and build project
4.run the test method as junit under solr\contrib\ltr\src\test-files
It will have the following error {... unknown field 'description' ...}. I
think the reason is that the scheam wich the test method load does not have the
'description' field. the reason is solr/core/src/test-files was also compile to
the same floader. so the scheam was not true yet.
i think we should rename the solr.collection1.conf under
solr/contrib/ltr/src/test-files .beacuse other solr/contrib also Named with
their module name infort of solr.collection1.conf sunch as
langid.solr.collection1.conf and extraction.solr.collection1.conf
380382...@qq.com
From: Christine Poerschke (JIRA)
Date: 2017-10-25 04:02
To: dev
Subject: [jira] [Commented] (SOLR-11528) ltr unit tests failed
[
https://issues.apache.org/jira/browse/SOLR-11528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16217578#comment-16217578
]
Christine Poerschke commented on SOLR-11528:
Hello [~jj380382856] - thanks for opening this ticket and attaching a patch!
bq. If we build the lucene project first and then run the ltr unit test, it
will have the following error: ...
Could you share more details on how you build and run the tests, possibly with
seeds that reproduce the issue, e.g. in the output you might see something like
{code}
NOTE: reproduce with: ant test -Dtestcase=... -Dtests.seed=... ...
{code}
Personally I haven't come across the {{... unknown field 'description' ...}}
error locally and it would be great to reproduce the exact issue before making
changes.
> ltr unit tests failed
> --
>
> Key: SOLR-11528
> URL: https://issues.apache.org/jira/browse/SOLR-11528
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
> Components: contrib - LTR
>Affects Versions: master (8.0)
> Environment: eclipse oxygen
>Reporter: jin jing
> Fix For: master (8.0)
>
> Attachments: SOLR-11528.patch
>
>
> If we build the lucene project first and then run the ltr unit test, it will
> have the following error:
> ERROR: [doc=1] unknown field 'description'
> i found this is beacuse of the under the floader
> solr\contrib\ltr\src\test-files we shoud modify solr\collection1\conf to
> ltr\solr\collection1\conf. and we shoud Specify solrhome in TestRerankBase.
> If you do not modify the path schema may be overwritten
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
Re: RE: run in eclipse error
I recently look at solr and lucene source, I do not know if I can solve this error and submit a patch? 380382...@qq.com From: Uwe Schindler Date: 2017-10-17 15:45 To: java-u...@lucene.apache.org CC: dev@lucene.apache.org Subject: RE: run in eclipse error Hi, this has nothing to do with the Java version. I generally ignore this Eclipse-failure as I only develop in Eclipse, but run from command line. The reason for this behaviour is a problem with Eclipse's resource management/compiler with the way how some classes in Solr (especially facet component) are setup. In general, it is nowadays a no-go to have so called "non-inner" pkg-private classes. These are classes which share the same source code file, but are not nested in the main class. Instead they appear next to each other in the source file. This is a relic from Java 1.0 and should really no longer used! Unfortunately some Solr developers still create such non-nested classes. Whenever I see them I change them to be static inner classes. The problem with the bug caused by this is that Eclipse randomly fails (it depends on the order how it compiles). The problem is that Eclipse (but also other tools) cannot relate the non-inner class file to a source file and therefore cannot figure out when it needs to be recompiled. BTW. The same problem applies to other build system like javac and Ant when it needs to compile. When you change such an inner non-nested inner class, it fails to compile in most cases unless you do "ant clean". The problem is again, that the compiler cannot relate the class files to source code files! We should really fix those classes to be static and inner - or place them in separate source files. I am looking to find a solution to detect this with forbiddenapis or our Source Code Regexes, if anybody has an idea: tell me! Uwe - Uwe Schindler Achterdiek 19, D-28357 Bremen http://www.thetaphi.de eMail: u...@thetaphi.de > -Original Message- > From: 380382...@qq.com [mailto:380382...@qq.com] > Sent: Tuesday, October 17, 2017 4:43 AM > To: java-user <java-u...@lucene.apache.org> > Subject: run in eclipse error > > i am trying to run solr in eclipse. but got the error "The type > FacetDoubleMerger is already defined". i don't know why. Whether it is jdk > version wrong? > Does git master need to use java9 for development? > > > 380382...@qq.com - To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org For additional commands, e-mail: java-user-h...@lucene.apache.org
Re: RE: run in eclipse error
Thank you for your reply, this question confused me for a long time 380382...@qq.com From: Uwe Schindler Date: 2017-10-17 15:45 To: java-u...@lucene.apache.org CC: dev@lucene.apache.org Subject: RE: run in eclipse error Hi, this has nothing to do with the Java version. I generally ignore this Eclipse-failure as I only develop in Eclipse, but run from command line. The reason for this behaviour is a problem with Eclipse's resource management/compiler with the way how some classes in Solr (especially facet component) are setup. In general, it is nowadays a no-go to have so called "non-inner" pkg-private classes. These are classes which share the same source code file, but are not nested in the main class. Instead they appear next to each other in the source file. This is a relic from Java 1.0 and should really no longer used! Unfortunately some Solr developers still create such non-nested classes. Whenever I see them I change them to be static inner classes. The problem with the bug caused by this is that Eclipse randomly fails (it depends on the order how it compiles). The problem is that Eclipse (but also other tools) cannot relate the non-inner class file to a source file and therefore cannot figure out when it needs to be recompiled. BTW. The same problem applies to other build system like javac and Ant when it needs to compile. When you change such an inner non-nested inner class, it fails to compile in most cases unless you do "ant clean". The problem is again, that the compiler cannot relate the class files to source code files! We should really fix those classes to be static and inner - or place them in separate source files. I am looking to find a solution to detect this with forbiddenapis or our Source Code Regexes, if anybody has an idea: tell me! Uwe - Uwe Schindler Achterdiek 19, D-28357 Bremen http://www.thetaphi.de eMail: u...@thetaphi.de > -Original Message- > From: 380382...@qq.com [mailto:380382...@qq.com] > Sent: Tuesday, October 17, 2017 4:43 AM > To: java-user <java-u...@lucene.apache.org> > Subject: run in eclipse error > > i am trying to run solr in eclipse. but got the error "The type > FacetDoubleMerger is already defined". i don't know why. Whether it is jdk > version wrong? > Does git master need to use java9 for development? > > > 380382...@qq.com - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
Re: RE: Question about grouping in distribute mode
Why the implementation does not consider removing duplicate group data?Is there a technical difficulty or any other reason? 380382...@qq.com From: Ian Caldwell Date: 2017-04-07 10:33 To: 'dev@lucene.apache.org' Subject: RE: RE: Question about grouping in distribute mode Yes, It looks that each shard will return the total count of groups in that shard and would give a higher number if the same group is in more than one shard. The other problem is when merging the group data the counts and total count are being held in Integers that can overflow resulting in negative numbers.(solr 5.5.3) Ian NLA From: 380382...@qq.com [mailto:380382...@qq.com] Sent: Friday, 7 April 2017 11:50 AM To: dev <dev@lucene.apache.org> Subject: Re: RE: Question about grouping in distribute mode thank you i think it is only use shard1.groupNumber add shard2.groupNumber。but groupA may also in shar1 and shard2. so the group.ngroup always bigger than the realy number? 380382...@qq.com From: Ian Caldwell Date: 2017-04-07 09:32 To: 'dev@lucene.apache.org' Subject: RE: Re: Question about grouping in distribute mode I think the this happens because the First Pass gets the top nGroups and holds the shards that they came from, then for the second pass it is only searching the shards that contributed to the list instead of searching all shards. So if searching for the top 10 groups a shard may have data from that group but it is ranked 11th (outside the top 10) then this shard is left off the list for the second pass. Searching(for 3 groups) could return From GROUPING_DISTRIBUTED_FIRST shard1: groupA, groupB & groupC (groupD ranked 4th so not returned in the list) shard2: groupA, groupC & groupD After merging, the top groups would be groupA, groupC & groupD From GROUPING_DISTRIBUTED_SECOND Shard1: groupA: doc1, doc3 & doc5 groupC: doc 11, doc13 & doc15 groupD: doc111, doc113 & doc115 Shard2: groupA: doc2, doc4 & doc6 groupC: doc12, doc14 & doc16 groupD: doc112, doc114 & doc116. So you need to do the second pass against all shards for the top docs so that you don’t miss the docs from groupD in shard1. Ian NLA -Original Message- From: Erick Erickson [mailto:erickerick...@gmail.com] Sent: Friday, 7 April 2017 1:16 AM To: dev@lucene.apache.org Subject: Re: Re: Question about grouping in distribute mode from the reference guide: group.ngroups and group.facet require that all documents in each group must be co-located on the same shard in order for accurate counts to be returned. Can't give you a technical reason, but there's no expectation it is supported with composite ID routing. Best, Erick On Thu, Apr 6, 2017 at 2:52 AM, 380382...@qq.com <380382...@qq.com> wrote: > thank for your help > when i use compseId route ,i find the group.ngroup is a wrong number. > I would like to know what implementation mechanism has led to this > happening。why we must use implict route when we want to use the group > correctly > > > 380382...@qq.com > > > From: Diego Ceccarelli (BLOOMBERG/ LONDON) > Date: 2017-04-06 17:16 > To: 380382856 > Subject: Re: Re: Question about grouping in distribute mode Dear > 380382856, I would be happy to help you if you can provide more > informations, do you want to know why grouping implements a specific > route strategy? My point is that usually grouping involves 3 > communications between the federator and the shards, but in case of > ngroup=1 it would be possible to obtain the same result with 2 > communications. > > Can I please ask to post your question on the user solr mailing list > [1]? in this way my answer will be useful to all solr users and people > more expert than me can also answer (or correct me if I say something > wrong :)) > > Have a good day! > Diego > > [1] http://lucene.apache.org/solr/community.html#mailing-lists-irc > > > From: 380382...@qq.com At: 04/06/17 08:38:20 > To: DIEGO CECCARELLI (BLOOMBERG/ LONDON) > Subject: Re: Re: Question about grouping in distribute mode > > hello can you help me? > There is a problem that has been bothering me.why solrcloud use > group.ngroup shoud implements implict route stratege? > 380382...@qq.com > > > From: Diego Ceccarelli (BLOOMBERG/ LONDON) > Date: 2017-03-30 22:09 > To: dev > Subject: Re: Question about grouping in distribute mode Yes, I agree. > And if there are not problems with the logic it would improve the > performance in both the cases.. > > From: dev@lucene.apache.org At: 03/30/17 14:59:31 > To: dev@lucene.apache.org > Subject: Re: Question about grouping in distribute mode > > This is also the case for non-distributed, isn’t it? The lucene-level > FirstPassGroupingCollector doesn’t actually record the docid of
Re: RE: Question about grouping in distribute mode
thank you i think it is only use shard1.groupNumber add shard2.groupNumber。but groupA may also in shar1 and shard2. so the group.ngroup always bigger than the realy number? 380382...@qq.com From: Ian Caldwell Date: 2017-04-07 09:32 To: 'dev@lucene.apache.org' Subject: RE: Re: Question about grouping in distribute mode I think the this happens because the First Pass gets the top nGroups and holds the shards that they came from, then for the second pass it is only searching the shards that contributed to the list instead of searching all shards. So if searching for the top 10 groups a shard may have data from that group but it is ranked 11th (outside the top 10) then this shard is left off the list for the second pass. Searching(for 3 groups) could return From GROUPING_DISTRIBUTED_FIRST shard1: groupA, groupB & groupC (groupD ranked 4th so not returned in the list) shard2: groupA, groupC & groupD After merging, the top groups would be groupA, groupC & groupD From GROUPING_DISTRIBUTED_SECOND Shard1: groupA: doc1, doc3 & doc5 groupC: doc 11, doc13 & doc15 groupD: doc111, doc113 & doc115 Shard2: groupA: doc2, doc4 & doc6 groupC: doc12, doc14 & doc16 groupD: doc112, doc114 & doc116. So you need to do the second pass against all shards for the top docs so that you don’t miss the docs from groupD in shard1. Ian NLA -Original Message- From: Erick Erickson [mailto:erickerick...@gmail.com] Sent: Friday, 7 April 2017 1:16 AM To: dev@lucene.apache.org Subject: Re: Re: Question about grouping in distribute mode from the reference guide: group.ngroups and group.facet require that all documents in each group must be co-located on the same shard in order for accurate counts to be returned. Can't give you a technical reason, but there's no expectation it is supported with composite ID routing. Best, Erick On Thu, Apr 6, 2017 at 2:52 AM, 380382...@qq.com <380382...@qq.com> wrote: > thank for your help > when i use compseId route ,i find the group.ngroup is a wrong number. > I would like to know what implementation mechanism has led to this > happening。why we must use implict route when we want to use the group > correctly > > > 380382...@qq.com > > > From: Diego Ceccarelli (BLOOMBERG/ LONDON) > Date: 2017-04-06 17:16 > To: 380382856 > Subject: Re: Re: Question about grouping in distribute mode Dear > 380382856, I would be happy to help you if you can provide more > informations, do you want to know why grouping implements a specific > route strategy? My point is that usually grouping involves 3 > communications between the federator and the shards, but in case of > ngroup=1 it would be possible to obtain the same result with 2 > communications. > > Can I please ask to post your question on the user solr mailing list > [1]? in this way my answer will be useful to all solr users and people > more expert than me can also answer (or correct me if I say something > wrong :)) > > Have a good day! > Diego > > [1] http://lucene.apache.org/solr/community.html#mailing-lists-irc > > > From: 380382...@qq.com At: 04/06/17 08:38:20 > To: DIEGO CECCARELLI (BLOOMBERG/ LONDON) > Subject: Re: Re: Question about grouping in distribute mode > > hello can you help me? > There is a problem that has been bothering me.why solrcloud use > group.ngroup shoud implements implict route stratege? > 380382...@qq.com > > > From: Diego Ceccarelli (BLOOMBERG/ LONDON) > Date: 2017-03-30 22:09 > To: dev > Subject: Re: Question about grouping in distribute mode Yes, I agree. > And if there are not problems with the logic it would improve the > performance in both the cases.. > > From: dev@lucene.apache.org At: 03/30/17 14:59:31 > To: dev@lucene.apache.org > Subject: Re: Question about grouping in distribute mode > > This is also the case for non-distributed, isn’t it? The lucene-level > FirstPassGroupingCollector doesn’t actually record the docid of the > top doc for each group at the moment, but I don’t think there’s any > reason it couldn’t - it’s stored in the relevant FieldComparator. And > it would be a nice shortcut in GroupingSearch more generally. > > Alan Woodward > www.flax.co.uk > > > On 30 Mar 2017, at 14:26, Diego Ceccarelli > <diego.ceccare...@gmail.com> > wrote: > > Hello, I'm currently working on Solr grouping in order to support > reranking [1]. > I've a working patch for non distribute search, and I'm now working on > the distribute setting. > > Looking at the code of distribute grouping (top-k groups, top-n > documents for each group) search consists in: > > GROUPING_DISTRIBUTED_FIRST > 1. given the grouping query, each shard will return the
Re: Re: Question about grouping in distribute mode
thank for your help when i use compseId route ,i find the group.ngroup is a wrong number. I would like to know what implementation mechanism has led to this happening。why we must use implict route when we want to use the group correctly 380382...@qq.com From: Diego Ceccarelli (BLOOMBERG/ LONDON) Date: 2017-04-06 17:16 To: 380382856 Subject: Re: Re: Question about grouping in distribute mode Dear 380382856, I would be happy to help you if you can provide more informations, do you want to know why grouping implements a specific route strategy? My point is that usually grouping involves 3 communications between the federator and the shards, but in case of ngroup=1 it would be possible to obtain the same result with 2 communications. Can I please ask to post your question on the user solr mailing list [1]? in this way my answer will be useful to all solr users and people more expert than me can also answer (or correct me if I say something wrong :)) Have a good day! Diego [1] http://lucene.apache.org/solr/community.html#mailing-lists-irc From: 380382...@qq.com At: 04/06/17 08:38:20 To: DIEGO CECCARELLI (BLOOMBERG/ LONDON) Subject: Re: Re: Question about grouping in distribute mode hello can you help me? There is a problem that has been bothering me.why solrcloud use group.ngroup shoud implements implict route stratege? 380382...@qq.com From: Diego Ceccarelli (BLOOMBERG/ LONDON) Date: 2017-03-30 22:09 To: dev Subject: Re: Question about grouping in distribute mode Yes, I agree. And if there are not problems with the logic it would improve the performance in both the cases.. From: dev@lucene.apache.org At: 03/30/17 14:59:31 To: dev@lucene.apache.org Subject: Re: Question about grouping in distribute mode This is also the case for non-distributed, isn’t it? The lucene-level FirstPassGroupingCollector doesn’t actually record the docid of the top doc for each group at the moment, but I don’t think there’s any reason it couldn’t - it’s stored in the relevant FieldComparator. And it would be a nice shortcut in GroupingSearch more generally. Alan Woodward www.flax.co.uk On 30 Mar 2017, at 14:26, Diego Ceccarelli <diego.ceccare...@gmail.com> wrote: Hello, I'm currently working on Solr grouping in order to support reranking [1]. I've a working patch for non distribute search, and I'm now working on the distribute setting. Looking at the code of distribute grouping (top-k groups, top-n documents for each group) search consists in: GROUPING_DISTRIBUTED_FIRST 1. given the grouping query, each shard will return the top-k groups 2. federator will merge the top-k groups and will produce the top-k groups for the query GROUPING_DISTRIBUTED_SECOND 1. given the top-k groups each shard will return its top-n documents for each group. 2. federator will then compute top-n documents for each group merging all the shards responses. GET_FIELDS as usual My plan was to change the collector in GROUPING_DISTRIBUTED_SECOND, and return the top documents for each group with a new score given by the function used to rerank (affecting maxScore for each group and then also the order of the groups). Looking at the code then I realized that TopGroups asserts that order of the groups is not changing, and I realized that indeed _ if the ranking function is the same, group order can't change after the first stage _. My question is: if the user is interested only in the top document for each group (i.e., the default: group.limit = 1) do we really need GROUPING_DISTRIBUTED_SECOND, or could we skip it? is there any reason to perform grouping distributed second in this case? or we could just return the top docid together with the topgroups in GROUPING_DISTRIBUTED_FIRST and then go directly to GET_FIELDS? Cheers, Diego [1] https://issues.apache.org/jira/browse/SOLR-8542
Re: RE: [VOTE] Release Lucene/Solr 6.4.2 RC1
i think java 9 have not be release？ 380382...@qq.com From: Uwe Schindler Date: 2017-03-03 15:42 To: dev@lucene.apache.org Subject: RE: [VOTE] Release Lucene/Solr 6.4.2 RC1 Hi, Jenkins complains about a compile error in the 6.4 branch, failed already 3 times (it could be a Java 9 bug, but strangely it only happens in the 6.4 branch): Build: https://jenkins.thetaphi.de/job/Lucene-Solr-6.4-Linux/124/ Java: 64bit/jdk-9-ea+158 -XX:+UseCompressedOops -XX:+UseG1GC All tests passed Build Log: [...truncated 9932 lines...] [javac] Compiling 323 source files to /home/jenkins/workspace/Lucene-Solr-6.4-Linux/solr/build/solr-solrj/classes/java [javac] /home/jenkins/workspace/Lucene-Solr-6.4-Linux/solr/solrj/src/java/org/apache/solr/common/util/NamedList.java:398: error: reference to NamedList is ambiguous [javac] return new NamedList<>( Collections.unmodifiableList(copy.nvPairs)); [javac]^ [javac] both constructor NamedList(Map<String,? extends T>) in NamedList and constructor NamedList(List) in NamedList match [javac] where T is a type-variable: [javac] T extends Object declared in class NamedList [javac] /home/jenkins/workspace/Lucene-Solr-6.4-Linux/solr/solrj/src/java/org/apache/solr/common/util/NamedList.java:398: error: incompatible types: cannot infer type arguments for NamedList<> [javac] return new NamedList<>( Collections.unmodifiableList(copy.nvPairs)); [javac] ^ [javac] reason: cannot infer type-variable(s) T [javac] (argument mismatch; List cannot be converted to Map<String,? extends T>) [javac] where T is a type-variable: [javac] T extends Object declared in class NamedList [javac] Note: Some input files use or override a deprecated API. [javac] Note: Recompile with -Xlint:deprecation for details. [javac] Note: Some input files use unchecked or unsafe operations. [javac] Note: Recompile with -Xlint:unchecked for details. [javac] 2 errors BUILD FAILED /home/jenkins/workspace/Lucene-Solr-6.4-Linux/build.xml:775: The following error occurred while executing this line: /home/jenkins/workspace/Lucene-Solr-6.4-Linux/build.xml:719: The following error occurred while executing this line: /home/jenkins/workspace/Lucene-Solr-6.4-Linux/build.xml:59: The following error occurred while executing this line: /home/jenkins/workspace/Lucene-Solr-6.4-Linux/solr/build.xml:252: The following error occurred while executing this line: /home/jenkins/workspace/Lucene-Solr-6.4-Linux/solr/common-build.xml:536: The following error occurred while executing this line: /home/jenkins/workspace/Lucene-Solr-6.4-Linux/solr/common-build.xml:484: The following error occurred while executing this line: /home/jenkins/workspace/Lucene-Solr-6.4-Linux/solr/common-build.xml:385: The following error occurred while executing this line: /home/jenkins/workspace/Lucene-Solr-6.4-Linux/solr/common-build.xml:405: The following error occurred while executing this line: /home/jenkins/workspace/Lucene-Solr-6.4-Linux/lucene/common-build.xml:501: The following error occurred while executing this line: /home/jenkins/workspace/Lucene-Solr-6.4-Linux/lucene/common-build.xml:1955: Compile failed; see the compiler error output for details. This does not happen in master and branch_6x. - Uwe Schindler Achterdiek 19, D-28357 Bremen http://www.thetaphi.de eMail: u...@thetaphi.de From: Ishan Chattopadhyaya [mailto:is...@apache.org] Sent: Wednesday, March 1, 2017 9:42 PM To: dev@lucene.apache.org Subject: [VOTE] Release Lucene/Solr 6.4.2 RC1 Please vote for release candidate 1 for Lucene/Solr 6.4.2 The artifacts can be downloaded from:https://dist.apache.org/repos/dist/dev/lucene/lucene-solr-6.4.2-RC1-rev34a975ca3d4bd7fa121340e5bcbf165929e0542f You can run the smoke tester directly with this command: python3 -u dev-tools/scripts/smokeTestRelease.py \https://dist.apache.org/repos/dist/dev/lucene/lucene-solr-6.4.2-RC1-rev34a975ca3d4bd7fa121340e5bcbf165929e0542f Here's my +1 SUCCESS! [0:52:41.429385]
