[jira] [Updated] (SOLR-2894) Implement distributed pivot faceting
[
https://issues.apache.org/jira/browse/SOLR-2894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Rowe updated SOLR-2894:
-
Attachment: 48.pivotfails.log.bz2
I ran the test using Mark's repro line a couple times on my Mac (OS X 10.9.4;
Oracle Java 1.7.0_55) and it didn't fail for me.
So overnight I set up the suite to run 500 iterations (via -Dtests.iters=500;
that option doesn't allow -Dtests.method to be supplied), otherwise same repro
line, and it ran 148 iterations before a suite timeout killed the run. Out of
those 148 iterations, there were 46 failures, one error, and the suite timeout:
~30% failure rate.
I'm attaching the log: {{48.pivotfails.log.bz2}} (warning: 130MB uncompressed).
Implement distributed pivot faceting
Key: SOLR-2894
URL: https://issues.apache.org/jira/browse/SOLR-2894
Project: Solr
Issue Type: Improvement
Reporter: Erik Hatcher
Assignee: Hoss Man
Fix For: 4.9, 5.0
Attachments: 48.pivotfails.log.bz2,
SOLR-2894-mincount-minification.patch, SOLR-2894-reworked.patch,
SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch,
SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch,
SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch,
SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch,
SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch,
SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch,
SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch,
SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch,
SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch,
SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch,
SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch,
SOLR-2894.patch, SOLR-2894_cloud_test.patch, dateToObject.patch,
pivot_mincount_problem.sh, pivotfail.log
Following up on SOLR-792, pivot faceting currently only supports
undistributed mode. Distributed pivot faceting needs to be implemented.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-2894) Implement distributed pivot faceting
[ https://issues.apache.org/jira/browse/SOLR-2894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hoss Man updated SOLR-2894: --- Attachment: SOLR-2894.patch Patch updated to trunk to deal with some minor compilation failures introduced by a (largely) unrelated commit a few hours ago (BytesRefBuilder) I'm currently running precommit - but once that's done i'll push to trunk. Implement distributed pivot faceting Key: SOLR-2894 URL: https://issues.apache.org/jira/browse/SOLR-2894 Project: Solr Issue Type: Improvement Reporter: Erik Hatcher Assignee: Hoss Man Fix For: 4.9, 5.0 Attachments: SOLR-2894-mincount-minification.patch, SOLR-2894-reworked.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894_cloud_test.patch, dateToObject.patch, pivot_mincount_problem.sh Following up on SOLR-792, pivot faceting currently only supports undistributed mode. Distributed pivot faceting needs to be implemented. -- This message was sent by Atlassian JIRA (v6.2#6252) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-2894) Implement distributed pivot faceting
[ https://issues.apache.org/jira/browse/SOLR-2894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Miller updated SOLR-2894: -- Attachment: pivotfail.log I just hit a fail. I've attatched the log. Implement distributed pivot faceting Key: SOLR-2894 URL: https://issues.apache.org/jira/browse/SOLR-2894 Project: Solr Issue Type: Improvement Reporter: Erik Hatcher Assignee: Hoss Man Fix For: 4.9, 5.0 Attachments: SOLR-2894-mincount-minification.patch, SOLR-2894-reworked.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894_cloud_test.patch, dateToObject.patch, pivot_mincount_problem.sh, pivotfail.log Following up on SOLR-792, pivot faceting currently only supports undistributed mode. Distributed pivot faceting needs to be implemented. -- This message was sent by Atlassian JIRA (v6.2#6252) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-2894) Implement distributed pivot faceting
[ https://issues.apache.org/jira/browse/SOLR-2894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hoss Man updated SOLR-2894: --- Attachment: SOLR-2894.patch No substance changes in this patch update, just needed updated to trunk since there has been a bunch of churn due to SOLR-4385. Implement distributed pivot faceting Key: SOLR-2894 URL: https://issues.apache.org/jira/browse/SOLR-2894 Project: Solr Issue Type: Improvement Reporter: Erik Hatcher Assignee: Hoss Man Fix For: 4.9, 5.0 Attachments: SOLR-2894-mincount-minification.patch, SOLR-2894-reworked.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894_cloud_test.patch, dateToObject.patch, pivot_mincount_problem.sh Following up on SOLR-792, pivot faceting currently only supports undistributed mode. Distributed pivot faceting needs to be implemented. -- This message was sent by Atlassian JIRA (v6.2#6252) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-2894) Implement distributed pivot faceting
[ https://issues.apache.org/jira/browse/SOLR-2894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hoss Man updated SOLR-2894: --- Attachment: SOLR-2894.patch Fingers crossed, this is the final patch. Not functional changes, just resolving hte prviously mentioned nocommits by renaming variables/methods or replacing comments about Jiras for future improvements with the actual jira numbers. ant precommit passes. Implement distributed pivot faceting Key: SOLR-2894 URL: https://issues.apache.org/jira/browse/SOLR-2894 Project: Solr Issue Type: Improvement Reporter: Erik Hatcher Assignee: Hoss Man Fix For: 4.9, 5.0 Attachments: SOLR-2894-mincount-minification.patch, SOLR-2894-reworked.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894_cloud_test.patch, dateToObject.patch, pivot_mincount_problem.sh Following up on SOLR-792, pivot faceting currently only supports undistributed mode. Distributed pivot faceting needs to be implemented. -- This message was sent by Atlassian JIRA (v6.2#6252) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-2894) Implement distributed pivot faceting
[
https://issues.apache.org/jira/browse/SOLR-2894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hoss Man updated SOLR-2894:
---
Attachment: SOLR-2894.patch
I've finished reviewing all the code and didn't find any new concerns. (woot!)
I was hoping that more refactoring could be done to share common logic between
the facet.field distributed code and the facet.pivot distributed code (akin to
what it seemed like applyToShardRequests() was aiming for in earlier patches)
but between the use of the DistribFieldFacet class and the anoying
discrepency between facet.mincount and facet.pivot.mincount that seemed
like more trouble then it's worth.
In addition, my little-laptop-that-could has been churning away of several
hundred iterations of TestCloudPivots using tests.nightly=true with this patch
for the past few days, w/o any signs of bugs in the refinement code.
At this point, there are only a handful of 'nocommit' comments left in the
patch, that fall into 2 basic categories:
* methods/variables I still want to rename
* reminders to create new jira's to track known issues / future improvements
I plan to deal with those over the next 24 hours, but none of those changes
should have any impact on the functionality / performance of the patch as it
currently stands.
[~brett.lucey] [~andrew.muldowney]: I'd really appreciate it if you guys
could take a gander at the latest version(s) of the patch and give my any
thoughts you have.
In particular: i know you've been using an older patch in production for a
while now, could you take this latest version for a spin using some of your
real data queries and set my mind at ease that i haven't introduced any
horrible performance problems with any of hte refacotring/code cleanup / bug
fixes i've made?
{panel:title=Changes in this patch}
* TestCloudPivotFacet
** a bit more logging
** dial back overrequest w/ comment (we're focused on refinement here)
** fix the num iters = 5 (no need to be higher on nightly runs, already
increase the index size num values per field)
* DistributedFacetPivotLargeTest
** new commented out test of limit=0 + mincount=0 + missing=true
*** i had a concern about this edge case w/refinement, but it turns out this
isn't evensupported in the existing pivot code.
* FacetComponent
** minor formatting comment cleanup
** use PIVOT_KEY consistently throughout file
** rename pivotPrefix - PIVOT_REFINE_PREFIX; and move to top of file
** move pivotRefinementCounter to top of file and add javadocs
** tweaked handleResponses:
*** check PURPOSE_REFINE_FACETS and PURPOSE_REFINE_PIVOT_FACETS in seperate if
blocks (instead of else if
*** doesn't change much at the moment, but smelled like a time bomb if/when we
ever do pivot refinement in the same requests as facet.field refinement.
** refactor away sanityCheckRefinements method
*** all it was doing was a single null check, so I inlined that
** use emptyList() in createPivotFacetOutput
** tweak variable names in createPivotFacetOutput
* PivotFacetProcessor
** clean up nocommits related to using FieldType methods where appropriate
** javadoc linting
* PivotFacetField
** trim() javadocs comment about future optimization
** javadoc linting
* PivotFacetValueCollection
** trim() javadocs
** javadoc linting
* PivotFacet
** javadoc linting
* PivotFacetValue
** javadoc linting
{panel}
Implement distributed pivot faceting
Key: SOLR-2894
URL: https://issues.apache.org/jira/browse/SOLR-2894
Project: Solr
Issue Type: Improvement
Reporter: Erik Hatcher
Assignee: Hoss Man
Fix For: 4.9, 5.0
Attachments: SOLR-2894-mincount-minification.patch,
SOLR-2894-reworked.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch,
SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch,
SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch,
SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch,
SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch,
SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch,
SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch,
SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch,
SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch,
SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch,
SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch,
SOLR-2894_cloud_test.patch, dateToObject.patch, pivot_mincount_problem.sh
Following up on SOLR-792, pivot faceting currently only supports
undistributed mode. Distributed pivot faceting needs to be implemented.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
[jira] [Updated] (SOLR-2894) Implement distributed pivot faceting
[
https://issues.apache.org/jira/browse/SOLR-2894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hoss Man updated SOLR-2894:
---
Attachment: SOLR-2894.patch
Ater working through the fix the the refinement logic in
PivotFacetField.queuePivotRefinementRequests the previously failing seed for
TestCloudPivotFacet started to pass, but some sort=index tests still weren't
working, which lead me to realize 2 things:
* some of my tests were absurd -- i've gotten use to using overrequest=0 as a
way to force refinement, but with facet.sort=index combined with limit (and
offset) ad mincount it ment that it was impossible for the sort=index facet
logic to ever find the results we're looking for. We *have* to allow some
overrequest when mincount1 or the initial shard requests won't find the values
(that will ultimately have a cumulative mincount high enough) in order to even
try refining them.
* offset wasn't being added to the limit in the per-shard requests, so w/o
overrequest enabled you would never get teh values you needed even in ideal
situations
* the shard query logic in FacetComponent was ignoring overrequest when
sort=index ... this seems broken to me, but from what i can tell, it comes
straight form the existing facet.field logic as well.
I'll open a bug to track the existing broken logic overrequest logic in
facet.field -- even though i hope that once we're done with this issue, it may
be fixed via refactoring and shared code with pivots (i'm not 100% certain: the
FacetComponent diff is the bulk of what i still need to review more closely on
this issue)
There's still a failure in DistributedFacetPivotLargeTest (mismatch comapred to
control) when i tried using mincount=0 that i'm not certain if/how we can
solve...
{code}
// :nocommit: broken honda?
rsp = query( params( q, *:*,
rows, 0,
facet,true,
facet.sort,index,
f.place_s.facet.limit, 20,
f.place_s.facet.offset, 40,
FacetParams.FACET_PIVOT_MINCOUNT,0,
facet.pivot, place_s,company_t) );
{code}
From what I can tell, the gist of the issue is that when dealing with
sub-fields of the pivot, the coordination code doesn't know about some of the
0 values if no shard which has the value for the parent field even knows
about the existence of the term.
The simplest example of this discrepency (compared to single node pivots) is to
consider an index with only 2 docs...
{noformat}
[{id:1,top_s:foo,sub_s:bar}
{id:2,top_s:xxx,sub_s:yyy}]
{noformat}
If those two docs exist in a single node index, and you pivot on
{{top_s,sub_s}} using mincount=0 you get a response like this...
{noformat}
$ curl -sS
'http://localhost:8881/solr/select?q=*:*rows=0facet=truefacet.pivot.mincount=0facet.pivot=top_s,sub_somitHeader=truewt=jsonindent=true'
{
response:{numFound:2,start:0,docs:[]
},
facet_counts:{
facet_queries:{},
facet_fields:{},
facet_dates:{},
facet_ranges:{},
facet_intervals:{},
facet_pivot:{
top_s,sub_s:[{
field:top_s,
value:foo,
count:1,
pivot:[{
field:sub_s,
value:bar,
count:1},
{
field:sub_s,
value:yyy,
count:0}]},
{
field:top_s,
value:xxx,
count:1,
pivot:[{
field:sub_s,
value:yyy,
count:1},
{
field:sub_s,
value:bar,
count:0}]}]}}}
{noformat}
If however you index each of those docs on a seperate shard, the response comes
back like this...
{noformat}
$ curl -sS
'http://localhost:8881/solr/select?q=*:*rows=0facet=truefacet.pivot.mincount=0facet.pivot=top_s,sub_somitHeader=truewt=jsonindent=trueshards=localhost:8881/solr,localhost:8882/solr'
{
response:{numFound:2,start:0,maxScore:1.0,docs:[]
},
facet_counts:{
facet_queries:{},
facet_fields:{},
facet_dates:{},
facet_ranges:{},
facet_intervals:{},
facet_pivot:{
top_s,sub_s:[{
field:top_s,
value:foo,
count:1,
pivot:[{
field:sub_s,
value:bar,
count:1}]},
{
field:top_s,
value:xxx,
count:1,
pivot:[{
field:sub_s,
value:yyy,
count:1}]}]}}}
{noformat}
The only solution i can think of, would be an extra (special to mincount=0)
stage of logic, after each PivotFacetField is refined, that would:
* iterate over all the values of the current pivot
* build up a Set of all all the known values for the child-pivots of of those
values
* iterate over all the values again, merging in a 0-count child value for
every value in the set
...ie: At least one shard knows about value 'v_x' in
[jira] [Updated] (SOLR-2894) Implement distributed pivot faceting
[
https://issues.apache.org/jira/browse/SOLR-2894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hoss Man updated SOLR-2894:
---
Attachment: SOLR-2894.patch
Quick update...
bq. ...i need to dig into this a bit more tommorow.
a restless night sleep and semi-fresh eyes make scary bugs shallow: the problem
was that PivotFacetField.queuePivotRefinementRequests had a short circuit
optimization when {{valueCollection.refinableCollection().isEmpty()}} that was
preventing the child pivots of the facet.missing count from being refined if
there were no matching values in the field.
This patch fixes that bug and adds an explicit test for this situation to
DistributedFacetPivotLargeTest.
Implement distributed pivot faceting
Key: SOLR-2894
URL: https://issues.apache.org/jira/browse/SOLR-2894
Project: Solr
Issue Type: Improvement
Reporter: Erik Hatcher
Assignee: Hoss Man
Fix For: 4.9, 5.0
Attachments: SOLR-2894-mincount-minification.patch,
SOLR-2894-reworked.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch,
SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch,
SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch,
SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch,
SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch,
SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch,
SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch,
SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch,
SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch,
SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch,
SOLR-2894_cloud_test.patch, dateToObject.patch, pivot_mincount_problem.sh
Following up on SOLR-792, pivot faceting currently only supports
undistributed mode. Distributed pivot faceting needs to be implemented.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-2894) Implement distributed pivot faceting
[
https://issues.apache.org/jira/browse/SOLR-2894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hoss Man updated SOLR-2894:
---
Attachment: SOLR-2894.patch
I let my laptop hammer away on TestCloudPivotFacet while i was looking at some
other stuff, and got a new reproducible failure...
{noformat}
[junit4] 2 NOTE: reproduce with: ant test -Dtestcase=TestCloudPivotFacet
-Dtests.method=testDistribSearch -Dtests.seed=EE02505B2F4046AC
-Dtests.nightly=true -Dtests.slow=true -Dtests.locale=fi
-Dtests.timezone=Asia/Aqtobe -Dtests.file.encoding=UTF-8
[junit4] FAILURE 56.9s | TestCloudPivotFacet.testDistribSearch
[junit4] Throwable #1: java.lang.AssertionError:
{main(facet=truefacet.pivot=pivot_y_s%2Cpivot_bfacet.pivot=pivot_tdt1facet.limit=4facet.offset=5facet.pivot.mincount=17facet.missing=falsefacet.sort=index),extra(rows=0q=id%3A%5B*+TO+786%5D_test_min=17_test_miss=false_test_sort=index)}
== pivot_y_s,pivot_b:
{params(rows=0),defaults({main(rows=0q=id%3A%5B*+TO+786%5D_test_min=17_test_miss=false_test_sort=index),extra(fq=%7B%21term+f%3Dpivot_y_s%7Dg)})}
expected:22 but was:50
[junit4]at
__randomizedtesting.SeedInfo.seed([EE02505B2F4046AC:6FE4DE43581F2690]:0)
[junit4]at
org.apache.solr.cloud.TestCloudPivotFacet.assertPivotCountsAreCorrect(TestCloudPivotFacet.java:239)
[junit4]at
org.apache.solr.cloud.TestCloudPivotFacet.doTest(TestCloudPivotFacet.java:187)
[junit4]at
org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:865)
[junit4]at java.lang.Thread.run(Thread.java:744)
[junit4] Caused by: java.lang.AssertionError: pivot_y_s,pivot_b:
{params(rows=0),defaults({main(rows=0q=id%3A%5B*+TO+786%5D_test_min=17_test_miss=false_test_sort=index),extra(fq=%7B%21term+f%3Dpivot_y_s%7Dg)})}
expected:22 but was:50
[junit4]at
org.apache.solr.cloud.TestCloudPivotFacet.assertNumFound(TestCloudPivotFacet.java:507)
[junit4]at
org.apache.solr.cloud.TestCloudPivotFacet.assertPivotCountsAreCorrect(TestCloudPivotFacet.java:257)
[junit4]at
org.apache.solr.cloud.TestCloudPivotFacet.assertPivotCountsAreCorrect(TestCloudPivotFacet.java:229)
[junit4]... 42 more
{noformat}
At first i thought this was simply an issue in how needRefinementAtThisLevel
assumed we never need refinement for sort=index -- that's too general of an
assertion, we can only asume no refinement is needed if mincount=0. But fixing
that still didn't solve the problem.
Thinking about the PivotFacetField.queuePivotRefinementRequests logic however
made me realize that all of the logic in that method (and it's use of
countThreshold) really only works with sort=count ... for sort=index we
shouldn't make any assumptions about the cutoff based on the count.
Before digging into a fix, I started working on more sort=index tests to try
and better excercise this code, and quickly encountered a new (unrelated?)
failure that seems to related to mincount==0 on sub pivots...
I distilled the new mincount failure out into a new isolated test query (that
doesn't use sort=index) in DistributedFacetPivotLargeTest:
{code}
rsp = query( q, *:*,
rows, 0,
facet,true,
facet.pivot,place_s,company_t,
FacetParams.FACET_LIMIT, 50,
FacetParams.FACET_PIVOT_MINCOUNT,0);
{code}
...which leads to...
{noformat}
[junit4] 2 NOTE: reproduce with: ant test
-Dtestcase=DistributedFacetPivotLargeTest -Dtests.method=testDistribSearch
-Dtests.seed=63DFE6A839DD2C9F -Dtests.slow=true -Dtests.locale=es_NI
-Dtests.timezone=Asia/Bishkek -Dtests.file.encoding=UTF-8
[junit4] FAILURE 43.5s | DistributedFacetPivotLargeTest.testDistribSearch
[junit4] Throwable #1: junit.framework.AssertionFailedError:
.facet_counts.facet_pivot.place_s,company_t[1].pivot.length:3!=50
[junit4]at
__randomizedtesting.SeedInfo.seed([63DFE6A839DD2C9F:E23968B04E824CA3]:0)
{noformat}
...i haven't dug into what exactly is going on here, i've been focusng on more
tests for the sort=index refinement bug first (since it's easy to reproduce
even w/o sub-pivots)
In addition to the above mentioned addition to DistributedFacetPivotLargeTest,
this new patch also adds some new queries/assertions to
DistributedFacetPivotSmallTest that seem to demo the problem with
facet.sort=index as the randomized failure (at least ... i *think* it's the
same problem).
i'm going to work on fixing queuePivotRefinementRequests to account for
sort=index tomorow.
Andrew, Brett: I don't suppose the mincount=0 bug jumps out at you guys as
something with an obvious fix?
Implement distributed pivot faceting
Key: SOLR-2894
URL:
[jira] [Updated] (SOLR-2894) Implement distributed pivot faceting
[
https://issues.apache.org/jira/browse/SOLR-2894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hoss Man updated SOLR-2894:
---
Attachment: SOLR-2894.patch
I've been focusing on more tests using facet.offset...
bq. I haven't looed into this closely, but i noticed the refinement code seems
to only refine things started at the facetFieldOffset, of the current
collection don't we need to refine all the values, starting from the beginging
of the list?
There was in fact a bug with refinement when using facet.offset -- but i was
looking in the wrong place. the code i was refering to before was involved in
deciding which values to drilldown into when recursively refining the
sub-pivots. that logic was already (mostly) correct because by that point
we've already refined the _current_ levle completly, so we can skip past the
offset when doing the recursion (the only glitch was a boundary check causing
an IOOBE, see detials below). Earlier on in the code however, there was a
mistake where only the limit (not the limit+offset) was being used to decide
the threshold value for refinement.
New improvements in this patch...
* TestCloudPivotFacet
** increase the odds of overrequest==0
** randonly include a facet.offset param to sanity check refinement in that case
* PivotFacetField
** fix refineNextLevelOfFacets not to ask for a sublist with a start offset
bigger then the size of the collection
*** this was causing an IndexOutOfBoundsException pretty quickly when offset
was mixed into the random test
** fix queuePivotRefinementRequests to respect offset when picking the
indexOfCountThreshold
*** before it was only looking at limit, with offset in the randomized test
this was causing failures even when pivots only had one field in them!
A few more things to consider in the future...
* PivotFacetFieldValueCollection.refinableSubList is only use to deal with
offset+limit sublisting from PivotFacetField.refineNextLevelOfFacets -- but
PivotFacetFieldValueCollection already knows the offsetlimit so maybe it
should be a smarter special purpose method with 0 args:
{{getNextLevelValuesToRefine()}}
* trim earlier?
** the way refinement currently works in PivotFacetField, after we've refined
our values, we mark that we no longer need refinement, and then on the next
call we recursively refine the subpivots of each value -- and in both cases we
do the offset+limit calculations and hang on to all of the values (both below
offset and above limit) as we keep iterating down hte pivots -- they don't get
thrown away until the final trim() call just before building up the final
result.
** i previously suggested folding the trim() logic into the NamedList response
logic -- but now i'm wondering if the trim() logic should instead be folded
into refinement? so once we're sure a level is fully refined, we go ahead and
trim that level before drilling down and refining it's kids?
Unfortunately, with this new patch, i did uncover a new random failure i can't
easily explain (doesn't seem related ot the offset changes since facet.offset
isn't evne used in these random params -- but it's possible i broke something
while fixing that) ...
{noformat}
[junit4] 2 NOTE: reproduce with: ant test -Dtestcase=TestCloudPivotFacet
-Dtests.method=testDistribSearch -Dtests.seed=775F7BCA685BBC22
-Dtests.nightly=true -Dtests.slow=true -Dtests.locale=da_DK
-Dtests.timezone=America/Montserrat -Dtests.file.encoding=UTF-8
[junit4] FAILURE 65.9s | TestCloudPivotFacet.testDistribSearch
[junit4] Throwable #1: java.lang.AssertionError:
{main(facet=truefacet.pivot=pivot_tl%2Cpivot_tl%2Cpivot_y_sfacet.pivot=bogus_not_in_any_doc_s%2Cpivot_l1%2Cpivot_tdfacet.limit=13facet.missing=truefacet.sort=countfacet.overrequest.count=2),extra(rows=0q=*%3A*fq=id%3A%5B*+TO+383%5D_test_miss=true_test_sort=count)}
== bogus_not_in_any_doc_s,pivot_l1,pivot_td:
{params(rows=0),defaults({main({main(rows=0q=*%3A*fq=id%3A%5B*+TO+383%5D_test_miss=true_test_sort=count),extra(fq=-bogus_not_in_any_doc_s%3A%5B*+TO+*%5D)}),extra(fq=%7B%21term+f%3Dpivot_l1%7D5098)})}
expected:7 but was:9
[junit4]at
__randomizedtesting.SeedInfo.seed([775F7BCA685BBC22:F6B9F5D21F04DC1E]:0)
[junit4]at
org.apache.solr.cloud.TestCloudPivotFacet.assertPivotCountsAreCorrect(TestCloudPivotFacet.java:239)
[junit4]at
org.apache.solr.cloud.TestCloudPivotFacet.doTest(TestCloudPivotFacet.java:187)
[junit4]at
org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:865)
[junit4]at java.lang.Thread.run(Thread.java:744)
[junit4] Caused by: java.lang.AssertionError:
bogus_not_in_any_doc_s,pivot_l1,pivot_td:
[jira] [Updated] (SOLR-2894) Implement distributed pivot faceting
[ https://issues.apache.org/jira/browse/SOLR-2894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hoss Man updated SOLR-2894: --- Attachment: SOLR-2894.patch Making good progress (only ~1600 lines of diff left to review!) updates in this patch... * PivotFacetFieldValueCollection ** some javadocs ** refactor away method: nonNullValueIterator() *** only called in one place * PivotFacetField ** some javaddocs ** made createFromListOfNamedLists smart enough to return null on null input *** simplified PivotFacetValue.createFromNamedList ** made contributeFromShard smart enough to be a no-op on null input *** simplified all callers (PivotFacet PivotFacetValue) ** made some vars final where possible via refactoring constructor createFromListOfNamedLists ** refactor skipRefinementAtThisLevel out of the method an up to an instance var since it never changes once the facet params are set in the constructor ** consolidate skipRefinementAtThisLevel + hasBeenRefined into a single var: needRefinementAtThisLevel ** simplify BitSet iteration (nextSetBit is always length) *** processDefiniteCandidateElement *** processPossibleCandidateElement * PivotFacetValue ** some javadocs ** made variables private and added method accessors (w/jdocs) as needed *** updated other classes as needed to call these new methods instead of the old pub vars ** made some vars final where possible via refactoring createFromNamedList constructor * PivotFacet ** some javadocs ** added getQueuedRefinements(int) ** made some variables final where possible ** renamed noRefinementsRequired - isRefinementsRequired ** eliminate unused method: areAnyRefinementsQueued * FacetComponent ** switched direct use of PivotFacet.queuedRefinements to use PivotFacet.getQueuedRefinements *** simplified error checking in several places One new question i want to go back and revisit later... * do we really need to track knownShards in PivotFacet ? ** ResponseBuilder already maintains a String[] of all shards, getShardNum derived from it ** can't we just loop from 0 to shards.length? does it ever matter if a shard hasn't participated? ** ie: is it really important that we skip any unset bits in knownShards when looping? (all the current usages seem safe even if a shard has no data for the current pivot) Implement distributed pivot faceting Key: SOLR-2894 URL: https://issues.apache.org/jira/browse/SOLR-2894 Project: Solr Issue Type: Improvement Reporter: Erik Hatcher Assignee: Hoss Man Fix For: 4.9, 5.0 Attachments: SOLR-2894-mincount-minification.patch, SOLR-2894-reworked.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894_cloud_test.patch, dateToObject.patch, pivot_mincount_problem.sh Following up on SOLR-792, pivot faceting currently only supports undistributed mode. Distributed pivot faceting needs to be implemented. -- This message was sent by Atlassian JIRA (v6.2#6252) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-2894) Implement distributed pivot faceting
[
https://issues.apache.org/jira/browse/SOLR-2894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hoss Man updated SOLR-2894:
---
Attachment: SOLR-2894.patch
hey guys, stoked to see all these tests passing!
I've been slowly working my way through Andrew's latest patch, reviewing all
the code and making some tweaks/improvements as I go. Here's a checkpointed
update...
{panel}
Patch updates in attachment:
* fix FacetComponent to mirror refactoring done in SOLR-6216
* fixed up the String.format calls in various classes so they specify
Locale.ROOT
** removed some useless toString() calls in these format calls as well,
particularly since it looked like they could cause NPEs
* PivotFacetField
** javadocs:
*** createFromListOfNamedLists
*** convertToListOfNamedLists
** eliminate call to {{PivotFacetFieldValueCollection.contains(...)}} (see
below)
* PivotFacetValue...
** javadocs:
*** class
*** createFromNamedList
*** shardHasContributed
*** convertToNamedList
* PivotFacetFieldValueCollection...
** javadocs:
*** class
*** refinableCollection
*** refinableSubList
*** refinableSize
*** size
*** get
*** add
** remove unused methods
*** isEmpty()
*** getValue(Comparable)
*** contains(Comparable)
(this was used, but only in a case where it was immediately followed by a
call {{get(Comparable)}} so i just optimized it away and replaced it with a
null check.
** rename: isSorted - dirty
** rename: nullValue - missingValue
*** it was really confusing because nullValue could be null, or it could be a
PivotFacetValue whose value was null
** fix {{add(PivotFacetValue)}} to set dirty directly
** lock down some stuff...
*** methods for accessing some vars so they don't need to be public
*** make some things specified in constructor final
*** make {{refinableCollection}} and {{refinableSubList}} return immutable lists
{panel}
Some things i'm either confused by and/or debating in my head ...
comments/opinions from others would be apreciated:
* refinement and facet offset
** I haven't looed into this closely, but i noticed the refinement code seems
to only refine things started at the facetFieldOffset, of the current
collection
** don't we need to refine all the values, starting from the beginging of the
list?
** if if the offset is 1 and the first value X has a count of 100 and the
second value Y has an initial count of 50 but a post-refinement count of
150 pushing itself prior to the offset and putting X into the window, then
doesn't X miss out on refinement?
* {{refinableCollection()}}
** I think we probably want to rename {{refinableCollection()}} (and
{{refinableSize()}}) to something more like {{getExplicitValuesList()}}
(compared to the {{getMissingValue()}} method I just added) to make it more
clear what you are really getting form this method ... I recognize that this
name comes from the fact that we don't ever really need to refine the count for
the missing value, but that seems like an implementaion detail that doesn't
affect a lot of places this method is called (and particularly since the
childPivots of the missing value _do_ still need refined so even when it is
relevant, it's still missleading from a recursion standpoint.)
* trim
** from what i can understand of the {{trim}} methods - these are typically
destructive operations that:
*** should only be called after all refinement is completed
*** prune things that are no longer needed based on the limit/offset params,
making the objects unusable for any future modifications/refinement so that
it's only good for...
*** should be called just prior to asking for the final NamedList response
structure
** if my understanding is correct, then it seems like it might be safer more
straight forward to instead just refactor this functionality directly into the
corrisponding methods for converting to a NamedList, and clearly document those
methods as destructive?
*** or at the very least add a trimmed boolean and sprinkle arround some
asserts in the various methods related to wether the object has/has not already
been trimmed
Implement distributed pivot faceting
Key: SOLR-2894
URL: https://issues.apache.org/jira/browse/SOLR-2894
Project: Solr
Issue Type: Improvement
Reporter: Erik Hatcher
Assignee: Hoss Man
Fix For: 4.9, 5.0
Attachments: SOLR-2894-mincount-minification.patch,
SOLR-2894-reworked.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch,
SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch,
SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch,
SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch,
SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch,
SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch,
SOLR-2894.patch,
[jira] [Updated] (SOLR-2894) Implement distributed pivot faceting
[ https://issues.apache.org/jira/browse/SOLR-2894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Muldowney updated SOLR-2894: --- Attachment: SOLR-2894.patch Uploaded latest patch with the refinement optimizations and error check Implement distributed pivot faceting Key: SOLR-2894 URL: https://issues.apache.org/jira/browse/SOLR-2894 Project: Solr Issue Type: Improvement Reporter: Erik Hatcher Assignee: Hoss Man Fix For: 4.9, 5.0 Attachments: SOLR-2894-mincount-minification.patch, SOLR-2894-reworked.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894_cloud_test.patch, dateToObject.patch, pivot_mincount_problem.sh Following up on SOLR-792, pivot faceting currently only supports undistributed mode. Distributed pivot faceting needs to be implemented. -- This message was sent by Atlassian JIRA (v6.2#6252) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-2894) Implement distributed pivot faceting
[
https://issues.apache.org/jira/browse/SOLR-2894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Muldowney updated SOLR-2894:
---
Attachment: SOLR-2894.patch
bq. Just to clarify: you are saying that bbc isn't included in the top set in
the distrib call because overrequest is so low, which is inconcsistent with the
control where bbc is in the top – but all of the values returned by the distrib
call do in fact have accurate refined counts ... correct?
It *was* not refining properly, which I attributed to the lack of overrequest
but that was incorrect. The test is actually the only one that tests the
following criteria:
* A value that should be in the top elements is not because the
overrequesting didn't pick it up or too many shards had values too small. (In
this case {{bbc}} only has a value of 150 from the initial round, when its
actual value is 445, larger than {{microsoft}}'s inital value of 398)
* There is a shard that has responded with an empty response, aka it has no
documents (shard#3 is always empty in the long test file)
When those two things combine we had an error in our refinement code where we
would add {{Integer.MAX_VALUE}} to the possible count, overflowing the {{int}}
and causing it to go negative, and we would never ask for refinement. So we
would get {{microsoft:398}} over {{bbc}} *Fixed*
I have fixed the null issue that keeps away from counting towards the
{{facet.limit}}
I have fixed the null issue that keeps it around even when its less than
{{facet.mincount}}
I have fixed the issue where an empty response from a shard would render all
values on the cusp of making it into the top values never get refined.
Are you still seeing the infinite recursion problem? The seeds you provided
earlier pass locally for me.
Implement distributed pivot faceting
Key: SOLR-2894
URL: https://issues.apache.org/jira/browse/SOLR-2894
Project: Solr
Issue Type: Improvement
Reporter: Erik Hatcher
Assignee: Hoss Man
Fix For: 4.9, 5.0
Attachments: SOLR-2894-mincount-minification.patch,
SOLR-2894-reworked.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch,
SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch,
SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch,
SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch,
SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch,
SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch,
SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch,
SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch,
SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch,
SOLR-2894_cloud_test.patch, dateToObject.patch, pivot_mincount_problem.sh
Following up on SOLR-792, pivot faceting currently only supports
undistributed mode. Distributed pivot faceting needs to be implemented.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-2894) Implement distributed pivot faceting
[
https://issues.apache.org/jira/browse/SOLR-2894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Muldowney updated SOLR-2894:
---
Attachment: SOLR-2894.patch
I've uploaded a new file with my {{facet.missing}} changes. It's got the small
and longtail working.
{code:title=DistributedFacetPivotLargeTest.java}rsp = query( q, *:*,
fq, -place_s:0placeholder,
rows, 0,
facet,true,
facet.limit,1,
facet.missing,true,
//FacetParams.FACET_OVERREQUEST_RATIO, 0, // force refine
//FacetParams.FACET_OVERREQUEST_COUNT, 0, // force refine
facet.pivot,special_s,company_t);{code}
This test gets whacky when the {{OVERREQUEST}} options are uncommented. With
the {{OVERREQUEST}} options uncommented we do not get the proper {{bbc}} value
and so the distributed version diverges from the non-distrib. Your second
comment on this issue is exactly on point.
Another variance in that test is that on the distrib side we get
{code}
{field=special_s,value=,count=3,pivot=[
{field=company_t,value=microsoft,count=2},
{field=company_t,value=null,count=0}]}
{code}
whereas for the non-distrib we just get
{code}
{field=special_s,value=,count=3,pivot=[
{field=company_t,value=microsoft,count=2}]}
{code}
Should {{facet.missing}} respect the {{mincount}} (in this case it's 1)?
Implement distributed pivot faceting
Key: SOLR-2894
URL: https://issues.apache.org/jira/browse/SOLR-2894
Project: Solr
Issue Type: Improvement
Reporter: Erik Hatcher
Assignee: Hoss Man
Fix For: 4.9, 5.0
Attachments: SOLR-2894-mincount-minification.patch,
SOLR-2894-reworked.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch,
SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch,
SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch,
SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch,
SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch,
SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch,
SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch,
SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch,
SOLR-2894.patch, SOLR-2894.patch, SOLR-2894_cloud_test.patch,
dateToObject.patch, pivot_mincount_problem.sh
Following up on SOLR-792, pivot faceting currently only supports
undistributed mode. Distributed pivot faceting needs to be implemented.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-2894) Implement distributed pivot faceting
[
https://issues.apache.org/jira/browse/SOLR-2894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hoss Man updated SOLR-2894:
---
Attachment: SOLR-2894.patch
bq. Quick note on PivotFacetHelper's retrieve method ...
I haven't really been aware of those other issues until now (although SOLR-3583
may explain some of the unused code i pruned from PivotListEntry a few patches
ago) but i agree with your assessment: if/when enhancements to distributed
pivots start dealing with adding optional data to each level of the pivot, the
appraoch currently used will have to change.
(Personally: I'm not emotionally ready to put any serious thought into that
level of implementation detail in future pivot improvements - i want to focus
on getting the basics of distrib pivots solid released first)
Updated patch with most of the tests i had in mind that i mentioned before
(although i'd still like to add some more facet.missing tests)...
* TestCloudPivotFacet
** randomize overrequest amounts
** randomize facet.mincount usage assert never exceded
** randomize facet.missing usage assert that null values are only ever last
in list of values
*** make the odds of docs missing a field more randomized (across test runs)
** add in the possibility of trying to pivot on a field that is in 0 docs
** Dial back some constants to reduce OOM risk when running -Dtests.nightly=true
** example refine count failure from the facet.missing problem (unless there's
another bug that looks really similar) with these changes:
*** {{ant test -Dtestcase=TestCloudPivotFacet -Dtests.method=testDistribSearch
-Dtests.seed=98C12D5256897A09 -Dtests.nightly=true -Dtests.slow=true
-Dtests.locale=sr -Dtests.timezone=America/Louisville
-Dtests.file.encoding=UTF-8}}
* DistributedFacetPivotLongTailTest
** some data tweaks an additional assertion to ensure refinement is happening
* DistributedFacetPivotSmallTest
** s/honda/null/g - help test that the 4 character string null isn't
triggering any special behavior, or getting confused with a missing value in
docs.
* DistributedFacetPivotLargeTest
** comment assert noting that a shard is left empty (helps with edge case
testing of result merging refinement)
** added assertPivot helper method did a bit of refactoring
** added test of 2 diff pivots in the same request (swap field order)
** added test of same bi-level pivot with w/o a tagged fq exclusion in the
same request
** added test variants of facet.limit facet.index used as localparam
*** currently commented out because it doesn't work -- see SOLR-6193
The problem noted above with using {{facet.*}} params as local params in
{{facet.pivot}} is something i discovered earlier this week while writing up
these tests. I initially set the problem set it asside to keep working on
tests, with hte intention of looking into a fix once i had better coverage of
the problem -- but then when i came back to revisit it yesterdan and looked to
the existing {{facet.field}} shard request logic for guidance, i discovered
that didn't seem to work the way i expected either and realized John Gibson
recently filed SOLR-6193 because {{facet.field}} _does_ have the exact same
problem.
i don't think we should let this block adding distributed facet.pivot, let's
tackle it holisticly for all faceting in SOLR-6193.
Andrew/Brett: have you guys had a chance to look into the refinement bug when
{{facet.missing}} is used?
(BTW: my update patch only affected test files, so hopefully theres no
collision with anything you guys have been working on -- but if there is, feel
free to just post whatever patch you guys come up with and i'll handle the
merge)
Implement distributed pivot faceting
Key: SOLR-2894
URL: https://issues.apache.org/jira/browse/SOLR-2894
Project: Solr
Issue Type: Improvement
Reporter: Erik Hatcher
Assignee: Hoss Man
Fix For: 4.9, 5.0
Attachments: SOLR-2894-mincount-minification.patch,
SOLR-2894-reworked.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch,
SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch,
SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch,
SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch,
SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch,
SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch,
SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch,
SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch,
SOLR-2894.patch, SOLR-2894_cloud_test.patch, dateToObject.patch,
pivot_mincount_problem.sh
Following up on SOLR-792, pivot faceting currently only supports
undistributed mode. Distributed pivot faceting needs to be implemented.
--
This message was sent by
[jira] [Updated] (SOLR-2894) Implement distributed pivot faceting
[ https://issues.apache.org/jira/browse/SOLR-2894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hoss Man updated SOLR-2894: --- Attachment: SOLR-2894.patch bq. It looks like we need to rethink how the values are encoded into a path for the purpose of refinement so we can account for and differentiate between missing values, the empty string (0 chars), and the literal string null (4 chars) I've been working on this for the last few days - cleaning up how we deal with the refinement strings so that facet.missing and/or empty strings () in fields won't be problematic. It's been slow going as i tried to be systematic about refactoring documenting methods as i went along and started understanding more and more of the code. The bulk of the changes i made can be summarized as: # make the valuePath tracking more structured via ListString instead of building up single comma seperated refinement string right off the bat # refactor the encoding/decoding of the refinement strings into a utility method thta can handle null and empty string. # refactor the refinement count subset computation so that it can actually handle facet.missing correctly (before attempts at refining facet.missing were just looking for the term null (ie: 4 characters) Full details on how this patch differs from the lsat one are listed below -- but as things stand right now there is still a nasty bug somewhere in the facet.missing processing that i can't wrap my head arround... In short: when facet.missing is enabled in the SPECIAL test i mentioned in my last comment, it's somehow causing the refined counts of of the non-missing SPECIAL value to be wrong (even if the SPECIAL value is a regular string, and not ). I can't really wrap my head arround how that's happening -- it's going to involve some more manual testing some more unit tests to get to the bottom of it, but in the mean time I wanted to get this patch posted. If folks could review it sanity check that i'm not doing something stupid with the refinement that would be appreciated. Detailed changes in this patch iteration... * PivotFacetHelper ** add new encodeRefinementValuePath decodeRefinementValuePath methods *** special encoding to handle empty strings (should be valid when pivoting) and null values (needed for facet.missing refinement) ** add tests in TestPivotHelperCode * PivotFacetValue PivotFacetField ** in general, make these a bit more structured ** eliminate fieldPath since it's unused ** replace PivotFacetValue.field (String) with a ref to the actual parentPivot (PivotFacetField) ** add PivotFacetField.parentValue (PivotFacetValue) to ref the value this pivot field is nested under (if any) ** replace valuePath with getValuePath() (ListString) to track the full structure * FacetComponent ** prune some big chunks of commented out code (alt approaches no longer needed it looks like?) ** use new PivotFacetValue.getValuePath() + PivotFacetHelper.encodeRefinementValuePath instead of PivotFacetValue.valuePath * SimpleFacets ** make getListedTermCounts(String,String) private again add javadocs clarifing that it smarSplits the list of terms ** convert getListedTermCounts(String,String,DocSet) - getListedTermCounts(String,DocSet,ListString) *** ie: pull the split logic out of this method, since it's confusing, and some callers don't need it. *** add javadocs *** updated SimpleFacets callers to do the split themselves * PivotFacetProcessor ** refactor subset logic (that dealt with missing values via negatived range query) into getSubset helper method *** add complimentary getSubsetSize method as well ** update previous callers of getListedTermCounts(String,String,DocSet) to use getSubsetSize instead in order to correctly handle the refinements of null (ie: facet.missing) ** refactor cleanup processSingle: *** have caller do the field splitting validation (eliminates redundency when refining many values) *** stop treating empty string as special case, switch conditionals that were looking at first value to look at list size directly * misc new javadocs on various methods throughout hte above mentioned files Misc notes for the future: * even if/when we get the refinement logic fixed, we really need some safety check to ensure we've completely eliminated this possibility of an infinite loop on refinement: ** coordinator should assert that if if asks shard for a refinement, that refinement is returned ** shard should assert that if it's asked to refine, the #vals makes sense for the #fields in the pivot * we need to include more testing of facet.missing: ** randomized testing in in TestCloudPivotFacet ** more usage of it in the Small Large tests. * in general, we need more testing that we know triggers refinement ** ie: the Small test already does a bunch with facet.missing, but I guess that never caught ny of these bugs, because refinement was never needed?
[jira] [Updated] (SOLR-2894) Implement distributed pivot faceting
[
https://issues.apache.org/jira/browse/SOLR-2894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hoss Man updated SOLR-2894:
---
Attachment: SOLR-2894.patch
I started reviewing again this afternoon and made a few more tweaks but then
quickly encountered a troubling situation:
There seems to be some set of circumstances that can cause pivot refinement to
go into an (infinite?) ridiculously long loop.
Here's an example log snippet from a test run that i eventually had to
explicitly kill after several minutes (normally it finishes in ~40 seconds on
my laptop)..
{noformat}
...
[junit4] 2 365476 T48 C473 P35623 oasc.SolrCore.execute [collection1]
webapp= path=/select
params={facet.limit=14facet.pivot={!fpt%3D3557}pivot_y_s,pivot_l1isShard=truedistrib=falsefacet=trueshard.url=https://127.0.0.1:35623/collection1/|https://127.0.0.1:35174/collection1/version=2q=*:*NOW=1403905534861facet.pivot.mincount=-1rows=0fpt3557=-8197981690463795098fpt3557=-7333481702750443698fpt3557=-5750361150833026124fpt3557=-1254664925684537075fpt3557=-790491513359287891fpt3557=-259812169693239119fpt3557=5005fpt3557=5023fpt3557=434325197357513755fpt3557=1208379606676285112fpt3557=2157244738088160377fpt3557=4049867752092041147wt=javabin}
hits=384 status=0 QTime=3
[junit4] 2 365484 T53 C473 P35623 oasc.SolrCore.execute [collection1]
webapp= path=/select
params={facet.limit=14facet.pivot={!fpt%3D3558}pivot_y_s,pivot_l1isShard=truedistrib=falsefacet=trueshard.url=https://127.0.0.1:35623/collection1/|https://127.0.0.1:35174/collection1/version=2q=*:*NOW=1403905534861facet.pivot.mincount=-1rows=0fpt3558=-8197981690463795098fpt3558=-7333481702750443698fpt3558=-5750361150833026124fpt3558=-1254664925684537075fpt3558=-790491513359287891fpt3558=-259812169693239119fpt3558=5005fpt3558=5023fpt3558=434325197357513755fpt3558=1208379606676285112fpt3558=2157244738088160377fpt3558=4049867752092041147wt=javabin}
hits=384 status=0 QTime=3
[junit4] 2 365493 T50 C473 P35623 oasc.SolrCore.execute [collection1]
webapp= path=/select
params={facet.limit=14facet.pivot={!fpt%3D3559}pivot_y_s,pivot_l1isShard=truedistrib=falsefacet=trueshard.url=https://127.0.0.1:35623/collection1/|https://127.0.0.1:35174/collection1/version=2q=*:*NOW=1403905534861facet.pivot.mincount=-1rows=0fpt3559=-8197981690463795098fpt3559=-7333481702750443698fpt3559=-5750361150833026124fpt3559=-1254664925684537075fpt3559=-790491513359287891fpt3559=-259812169693239119fpt3559=5005fpt3559=5023fpt3559=434325197357513755fpt3559=1208379606676285112fpt3559=2157244738088160377fpt3559=4049867752092041147wt=javabin}
hits=384 status=0 QTime=5
...
{noformat}
A few things to note about those above log lines:
* with the seed used in this run there was only 740 total docs in the index
* all three of those requests were made to the same shard/core (C473) on the
same port (P35623)
* the pivot_l1 field being refined in these requests is a single valued long
field - which means even if every random value generated for it were unique, in
an index with 740 docs there can only be 740 possible long values here.
* these requests are already upto fpt=3559 -- _way_ more refinements then
should be neccessary for this field
* the shard is being asked to refine the same pivot values over and over again
(but with increasing fpt# keys)
Unfortunately while trying to get to the bottom of this, i realized the way the
test was picking the random pivots it used wasn't reproducible with a
consistent test seed. I've fixed that, but now i need to hammer on this test
some more to try and reproduce again with a reliable seed.
Small changes to the patch ...
* TestCloudPivotFacet
** added explicit sort to String[] fieldNames so buildRandomPivot would
reproduce with consistent seed
* SimpleFacets tweaks i made before encountering the test bug:
** more javadocs on some subtly diff methods
** change the new getTermCounts(String,Integer,DocSet) to private since it's
only used as a helper for the other public methods
Implement distributed pivot faceting
Key: SOLR-2894
URL: https://issues.apache.org/jira/browse/SOLR-2894
Project: Solr
Issue Type: Improvement
Reporter: Erik Hatcher
Fix For: 4.9, 5.0
Attachments: SOLR-2894-mincount-minification.patch,
SOLR-2894-reworked.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch,
SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch,
SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch,
SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch,
SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch,
SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch,
SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch,
SOLR-2894.patch, SOLR-2894.patch,
[jira] [Updated] (SOLR-2894) Implement distributed pivot faceting
[
https://issues.apache.org/jira/browse/SOLR-2894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hoss Man updated SOLR-2894:
---
Attachment: SOLR-2894.patch
bq. By the looks of things, it looks the problem seems to be poping up when a
refinement constraint in a multi-level pivot involves the empty string (and/or
missing values?)
Hmmm... both cases definitely seem to be problematic:
* refining on values that are the empty string {{}}
* refining against the {{null}} psuedo-values when using {{facet.missing}}
(Note: TestCloudPivotFacet currently doesn't even try facet.missing -- need to
rememedy that in a future patch)
The attached patch update modifies DistributedFacetPivotLargeTest to add a new
special_s field to a handful of docs -- some of which get the value of
SPECIAL ({{final String SPECIAL = ;}} and it goes into a loop here...
{code}
// refine on empty string
rsp = query( q, *:*,
rows, 0,
facet,true,
facet.limit,1,
FacetParams.FACET_OVERREQUEST_RATIO, 0, // force refine
FacetParams.FACET_OVERREQUEST_COUNT, 0, // force refine
facet.pivot,special_s,company_t);
{code}
{noformat}
[junit4] 2 32409 T43 C21 oasc.SolrCore.execute [collection1]
webapp=/po_cuf path=/select
params={shard.url=[ff01::083]:2/po_cuf|[ff01::213]:2/po_cuf|http://127.0.0.1:37920/po_cufNOW=1403920234230rows=0isShard=truedistrib=falsewt=javabinfpt2938=facet.pivot.mincount=-1facet.overrequest.count=0q=*:*version=2facet.pivot={!fpt%3D2938}special_s,company_tfacet.overrequest.ratio=0facet=truefacet.limit=1}
hits=357 status=0 QTime=0
[junit4] 2 32413 T42 C21 oasc.SolrCore.execute [collection1]
webapp=/po_cuf path=/select
params={shard.url=[ff01::083]:2/po_cuf|[ff01::213]:2/po_cuf|http://127.0.0.1:37920/po_cufNOW=1403920234230rows=0isShard=truedistrib=falsewt=javabinfpt2939=facet.pivot.mincount=-1facet.overrequest.count=0q=*:*version=2facet.pivot={!fpt%3D2939}special_s,company_tfacet.overrequest.ratio=0facet=truefacet.limit=1}
hits=357 status=0 QTime=0
{noformat}
(Note the {{...fpt2938=...}} and {{...fpt2939=...}})
Even if you redefine SPECIAL to be some other constant (ie: {{SPECIAL =
SPECIAL;}}) the code still goes into a loop in the next call, where
facet.missing is used and refinement is needed on the missing value...
{code}
// refine on empty string facet.missing
rsp = query( q, *:*,
fq, -place_s:0placeholder,
rows, 0,
facet,true,
facet.limit,1,
facet.missing,true,
FacetParams.FACET_OVERREQUEST_RATIO, 0, // force refine
FacetParams.FACET_OVERREQUEST_COUNT, 0, // force refine
facet.pivot,special_s,company_t);
{code}
{noformat}
[junit4] 2 26798 T53 C19 oasc.SolrCore.execute [collection1] webapp=/do_
path=/select
params={facet.overrequest.ratio=0wt=javabinfacet.missing=truefacet.limit=1facet.pivot.mincount=-1facet.pivot={!fpt%3D2151}special_s,company_tfpt2151=null,microsoftdistrib=falseversion=2shard.url=[ff01::083]:2/do_|[ff01::213]:2/do_|https://127.0.0.1:36955/do_|[ff01::114]:2/do_facet=trueq=*:*rows=0fq=-place_s:0placeholderNOW=1403920466501isShard=truefacet.overrequest.count=0}
hits=202 status=0 QTime=0
[junit4] 2 26802 T54 C19 oasc.SolrCore.execute [collection1] webapp=/do_
path=/select
params={facet.overrequest.ratio=0wt=javabinfacet.missing=truefacet.limit=1facet.pivot.mincount=-1facet.pivot={!fpt%3D2153}special_s,company_tdistrib=falseversion=2shard.url=[ff01::083]:2/do_|[ff01::213]:2/do_|https://127.0.0.1:36955/do_|[ff01::114]:2/do_facet=trueq=*:*rows=0fpt2153=null,microsoftfq=-place_s:0placeholderNOW=1403920466501isShard=truefacet.overrequest.count=0}
hits=202 status=0 QTime=1
{noformat}
(Note the {{...fpt2151=null,microsoft...}} and
{{...fpt2153=null,microsoft...}})
It looks like we need to rethink how the values are encoded into a path for the
purpose of refinement so we can account for and differentiate between missing
values, the empty string (0 chars), and the literal string null (4 chars)
Implement distributed pivot faceting
Key: SOLR-2894
URL: https://issues.apache.org/jira/browse/SOLR-2894
Project: Solr
Issue Type: Improvement
Reporter: Erik Hatcher
Assignee: Hoss Man
Fix For: 4.9, 5.0
Attachments: SOLR-2894-mincount-minification.patch,
SOLR-2894-reworked.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch,
SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch,
SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch,
SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch,
SOLR-2894.patch, SOLR-2894.patch,
[jira] [Updated] (SOLR-2894) Implement distributed pivot faceting
[ https://issues.apache.org/jira/browse/SOLR-2894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hoss Man updated SOLR-2894: --- Attachment: SOLR-2894.patch bq. I think it's very important to keep pivot's response values as objects. +1 bq. In the PivotFacetProcessor (shards) we .toObject each value. This is weird in the non-distributed mode because nothing clears up those into strings for the response -XML or JSON. This is a problem with dates, because 2012-11-01T12:30:00Z becomes Nov 1 4:30 EST 2012. I don't know what methods get run after process in the non-distrib mode that we could hook into to change these values back into what they should be. I don't think that's weird -- i think the toObject() call you have is exactly what it should be -- i'm not really following your point about the XML or JSON responses, the response writers already know how to handle the various Object types that (Like Dates, and Integers, etc...) that might be included. Based on your comment about PivotFacetValue's convertToNamedList, i *think* what you mean is that the main underlying problem with using the real Object representation of the values is that when you then want to build up the paths in PivotFacetValue's _createFromNamedList_ for the purposes of the refinement queries, there is no corrollary to toObject that can be used. This is very similar to the problem we encountered in SOLR-5354 -- the solution there was a new FieldType methd specific to marshalling and unmarshalling sort values. we can't simply re-use that new method as is because the Objects used as Sort values don't neccessarily have a 1-to-1 corrispondence with the Objects that matter here. Ideally there should be a similar method on the FieldType for doing this, that let's you round trip the output of FieldType.toObject() for the purposes of building up a simple query string -- but that doesn't exist at the moment. My vote would be to leave the code the way it is right now (assuming it can toString() anything except a Date object) and open a new issue to improve on this for custom FieldTypes at a later date. That way people who want to go ahead and use Distributed Pivot Faceting for out of the box field types like Strings/Dates/Numbers can, and have the benefits of well structured objects in the response -- w/o waiting on a more robust solution that can work with arbitrary custom field types. (which can come later) bq. Otherwise you've just included the CursorPagingTest which is probably from a different patch? CursorPagingTest is included in the patch because of methods refactored up into SolrTestCaseJ4 for use in this patch. --- I've been making my way further through the code review slowly -- Attaching a revised patch... * updated to trunk * added microsoft asserts to DistributedFacetPivotLargeTest (per Andrew) * make FacetParams.FACET_OVERREQUEST package-private since it's not a usable param (just a base) * StrUtils ** more javadocs ** new escapeTextWithSeparator test - TestUtils ** refactor duplicated code with existing join method into new private method * PivotListEntry ** more javadocs ** kill some dead code (multiple enums with same index?) ** refactored to leverage standard java Enum plumbing better * PivotFacetValue ** added a nocommit regarding custom fieldtypes to createFromNamedList that we either need a better solution to the Object-String problem, or we need to file a new issue prior to commiting and update the comment ** switched if-else-if-else-if on PivotListEntry instances to be an enum switch Implement distributed pivot faceting Key: SOLR-2894 URL: https://issues.apache.org/jira/browse/SOLR-2894 Project: Solr Issue Type: Improvement Reporter: Erik Hatcher Fix For: 4.9, 5.0 Attachments: SOLR-2894-mincount-minification.patch, SOLR-2894-reworked.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894_cloud_test.patch, dateToObject.patch, pivot_mincount_problem.sh Following up on SOLR-792, pivot faceting currently only supports undistributed mode. Distributed pivot faceting needs to be implemented. -- This message was sent by Atlassian JIRA (v6.2#6252) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-2894) Implement distributed pivot faceting
[ https://issues.apache.org/jira/browse/SOLR-2894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hoss Man updated SOLR-2894: --- Attachment: SOLR-2894.patch bq. I definitely want to add some test logic around that – if for no other reason then to prove that when the overrequesting is used, it can help with finding constraints in the long tail Updated patch... * new DistributedFacetPivotLongTailTest ** crafts the shard distribution specifically to demonstrate that overrequesting is affecting things as expected * split DistributedFacetPivotTest into DistributedFacetPivotSmallTest and DistributedFacetPivotLargeTest ** this was already 2 very different sets of data with two very differnet styles of asserting expected results -- so i went ahead and split it up ** Now there isn't the weird suprise that halfway through a test all the data is deleted and new data is added and more asertions are made. Brett Andrew: would really appreciate if you guys could review my changes to your existing test as well as the new LongTail test and help sanity check that the assertions all look correct. Assuming you guys don't spot any problems with the tests: next up i'll move back into reviewing the code more in depth, and documenting/refactoring/unit-testing as needed to help myself understand all this awesomeness you guys have added. Implement distributed pivot faceting Key: SOLR-2894 URL: https://issues.apache.org/jira/browse/SOLR-2894 Project: Solr Issue Type: Improvement Reporter: Erik Hatcher Fix For: 4.9, 5.0 Attachments: SOLR-2894-mincount-minification.patch, SOLR-2894-reworked.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894_cloud_test.patch, dateToObject.patch, pivot_mincount_problem.sh Following up on SOLR-792, pivot faceting currently only supports undistributed mode. Distributed pivot faceting needs to be implemented. -- This message was sent by Atlassian JIRA (v6.2#6252) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-2894) Implement distributed pivot faceting
[
https://issues.apache.org/jira/browse/SOLR-2894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hoss Man updated SOLR-2894:
---
Attachment: SOLR-2894.patch
Started getting back into this yesterday (i should have several large blocks of
time for this issue this week next week)...
bq. Me and Brett discovered serveral bugs with our mincount and the changes I
made to our refinement requests that resulted in the odd behavior you were
seeing.
Awesome! ... glad to see the test was useful.
bq. Not everything is super happy. I get what look like solrcloud errors when
running certain seeds
Hmmm... that is a weird error. People sometimes see errors in solr tests that
use threads related to timing and/or assertions of things that haven't happened
yet - but i don't remember ever seeing anything like this type of problem with
initialization of the cores.
do these failures reproduce for you with the same seeds? can you post the full
reproduce line that you get with these failures?
bq. I forgot this patch also comments out the randomUsableUnicodeString to just
be a simple string, BUT I've changed it back on my box and It seems to be fine.
yep -- it also still had one of my nocommits so that it was _only_ pivoting on
string fields, but even w/o that it's worked great for me on many iterations.
Revised patch - mostly cleaning up the lingering issues in TestCloudPivotFacet
but a few other minor fixes of stuff i noticed.
Detailed changes compared to previous patch...
* removed TestDistributedSearch.java.orig that seems to have been included in
patch by mistake
* cleanup TestCloudPivotFacet
** fixed randomUsableUnicodeString()
** fix nocommit about testing pivot on non-string fields
** fixed the depth checking (we can assert the *max* depth, but that's it)
** removed weird (unused) int ss = 2 that got added to assertNumFound
*** was also in some dead code in PivotFacetProcessor?
** refactored cut/pate methods from Cursor test into baseclass
* I removed the NullGoesLastComparator class and replaced it with a
compareWithNullLast helper method in PivotFacetField (and added a unit test for
it)
** the Comparator contract is pretty explicit about null, and this class
violated that
** it was only being used for simple method calls, not passed to anything that
explicitly needed a Comparator, so there wasn't a strong need for a standalone
class
My next step plans...
* review DistributedFacetPivotTest in depth more - add more strong assertions
** at first glance, it looks like a lot of the test is following the example of
most existing distrib tests of relying on comparisons between the controlClient
and the distrib client -- in my opinion that's a bad pattern, and i'd like to
add some explicit assertions on the results of all the {{this.query(...)}} calls
* re-review the new pivot code (and the changes to facet code) in general
** it's been a while since my last skim, and i know you've tweaked a bunch
based on my previous comments
** i'll take a stab at adding more javadocs to some of the new methods as i
make sense of them
** where possible, i'm going to try to add unit tests for some of the new low
level methods you've introduced -- largely as a way to help ensure i understand
what they do
Implement distributed pivot faceting
Key: SOLR-2894
URL: https://issues.apache.org/jira/browse/SOLR-2894
Project: Solr
Issue Type: Improvement
Reporter: Erik Hatcher
Fix For: 4.9, 5.0
Attachments: SOLR-2894-mincount-minification.patch,
SOLR-2894-reworked.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch,
SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch,
SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch,
SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch,
SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch,
SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch,
SOLR-2894.patch, SOLR-2894.patch, SOLR-2894_cloud_test.patch,
dateToObject.patch, pivot_mincount_problem.sh
Following up on SOLR-792, pivot faceting currently only supports
undistributed mode. Distributed pivot faceting needs to be implemented.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-2894) Implement distributed pivot faceting
[ https://issues.apache.org/jira/browse/SOLR-2894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hoss Man updated SOLR-2894: --- Attachment: SOLR-2894.patch bq. review DistributedFacetPivotTest in depth more - add more strong assertions Attaching updated patch with progress along this line: in addition so some new explicit assertions, it also includes some refactoring simplification of setupDistributedPivotFacetDocuments One thing that jumped out at me when reviewing this is even though the test does some queries with large overrequest params as well disabling overrequest, there doesn't seem to be any assertions about how the overrequesting affects the results -- in fact, because of how the controlClient is compared with the distributed client, it seems that with this sample data disabling overrequest doesn't even change the results at all. I definitely want to add some test logic around that -- if for no other reason then to prove that *when* the overrequesting is used, it can help with finding constraints in the long tail Implement distributed pivot faceting Key: SOLR-2894 URL: https://issues.apache.org/jira/browse/SOLR-2894 Project: Solr Issue Type: Improvement Reporter: Erik Hatcher Fix For: 4.9, 5.0 Attachments: SOLR-2894-mincount-minification.patch, SOLR-2894-reworked.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894_cloud_test.patch, dateToObject.patch, pivot_mincount_problem.sh Following up on SOLR-792, pivot faceting currently only supports undistributed mode. Distributed pivot faceting needs to be implemented. -- This message was sent by Atlassian JIRA (v6.2#6252) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-2894) Implement distributed pivot faceting
[
https://issues.apache.org/jira/browse/SOLR-2894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Muldowney updated SOLR-2894:
---
Attachment: SOLR-2894.patch
Me and Brett discovered serveral bugs with our mincount and the changes I made
to our refinement requests that resulted in the odd behavior you were seeing.
Not everything is super happy. I get what look like solrcloud errors when
running certain seeds
{code}
215 T12 oasc.SolrResourceLoader.locateSolrHome using system property
solr.solr.home:
..\..\C:\Users\AMULDO~1\AppData\Local\Temp\solr.cloud.TestCloudPivotFacet-A515DED004CF1660-001\tempDir-002
5216 T12 oasc.SolrResourceLoader.init new SolrResourceLoader for directory:
'..\..\C:\Users\AMULDO~1\AppData\Local\Temp\solr.cloud.TestCloudPivotFacet-A515DED004CF1660-001\tempDir-002\'
5421 T12 oasc.ConfigSolr.fromFile Loading container configuration from
D:\hmm\lucene-solr\..\..\C:\Users\AMULDO~1\AppData\Local\Temp\solr.cloud.TestCloudPivotFacet-A515DED004CF1660-001\tempDir-002\solr.xml
5422 T12 oass.SolrDispatchFilter.init ERROR Could not start Solr. Check
solr/home property and the logs
5483 T12 oasc.SolrException.log ERROR
null:org.apache.solr.common.SolrException: Could not load SOLR configuration
at org.apache.solr.core.ConfigSolr.fromFile(ConfigSolr.java:71)
at
org.apache.solr.core.ConfigSolr.fromSolrHome(ConfigSolr.java:96)
at
org.apache.solr.servlet.SolrDispatchFilter.loadConfigSolr(SolrDispatchFilter.java:157)
at
org.apache.solr.servlet.SolrDispatchFilter.createCoreContainer(SolrDispatchFilter.java:188)
at
org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:137)
at
org.eclipse.jetty.servlet.FilterHolder.doStart(FilterHolder.java:119)
at
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64)
at
org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:719)
at
org.eclipse.jetty.servlet.ServletHandler.updateMappings(ServletHandler.java:1309)
at
org.eclipse.jetty.servlet.ServletHandler.setFilterMappings(ServletHandler.java:1345)
at
org.eclipse.jetty.servlet.ServletHandler.addFilterMapping(ServletHandler.java:1085)
at
org.eclipse.jetty.servlet.ServletHandler.addFilterWithMapping(ServletHandler.java:931)
at
org.eclipse.jetty.servlet.ServletHandler.addFilterWithMapping(ServletHandler.java:888)
at
org.eclipse.jetty.servlet.ServletContextHandler.addFilter(ServletContextHandler.java:340)
at
org.apache.solr.client.solrj.embedded.JettySolrRunner$1.lifeCycleStarted(JettySolrRunner.java:327)
at
org.eclipse.jetty.util.component.AbstractLifeCycle.setStarted(AbstractLifeCycle.java:174)
at
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:65)
at
org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:432)
at
org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:405)
at
org.apache.solr.cloud.AbstractFullDistribZkTestBase.createJetty(AbstractFullDistribZkTestBase.java:481)
at
org.apache.solr.BaseDistributedSearchTestCase.createJetty(BaseDistributedSearchTestCase.java:351)
at
org.apache.solr.cloud.AbstractFullDistribZkTestBase.createServers(AbstractFullDistribZkTestBase.java:282)
at
org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:863)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
at
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
at
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
at
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
at
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at
org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
at
[jira] [Updated] (SOLR-2894) Implement distributed pivot faceting
[
https://issues.apache.org/jira/browse/SOLR-2894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hoss Man updated SOLR-2894:
---
Attachment: SOLR-2894.patch
I haven't had a lot of time to review the updatd patch in depth, but I did
spend some time trying to improve TestCloudPivotFacet to resolve some of the
nocommits -- but i'm still seeing failures...
1) I realized the depth check i was trying to do was bogus and commented it
out (still need to purge the code - didn't want to muck with that until the
rest of the test was passing more reliably)
2) the NPE I mentioned in QueryResponse.readPivots is still happening, but i
realized that it has nothing to do with the datatype of the fields being
pivoted on -- it only seemed that way because of the poor randomization of
values getting put in the single valued string fields vs the multivalued fields
in the old version of the test.
The bug seems to pop up in _some_ cases where a pivot constraint has no
sub-pivots. Normally this results in a NamedList with 3 keys
(field,value,count) -- the 4th pivot key is only included if there is a list
of at least 1 sub-pivot. But in some cases (I can't explain from looking at
the code why) the server is responding back with a 4th entry using hte key
pivot but the value is null
We need to get to the bottom of this -- it's not clear if there is a bug
preventing real sub-pivot constraints from being returned correctly, or if this
is just a mistake in the code where it's putting null in the NamedList
instead of not adding anything at all (in which case it might be tempting to
make QueryResponse.readPivots smart enough to deal with it, but if we did that
it would still be broken for older clients -- best to stick with teh current
API semantics)
In the attached patch update, this seed will fail showing the null sub-pivots
problem...
{noformat}
[junit4] 2 NOTE: reproduce with: ant test -Dtestcase=TestCloudPivotFacet
-Dtests.method=testDistribSearch -Dtests.seed=680E68425E7CA1BA
-Dtests.slow=true -Dtests.locale=es_US -Dtests.timezone=Canada/Eastern
-Dtests.file.encoding=UTF-8
[junit4] FAILURE 41.7s | TestCloudPivotFacet.testDistribSearch
[junit4] Throwable #1: java.lang.AssertionError: Server sent back
'null' for sub pivots?
[junit4]at
__randomizedtesting.SeedInfo.seed([680E68425E7CA1BA:E9E8E65A2923C186]:0)
[junit4]at
org.apache.solr.client.solrj.response.QueryResponse.readPivots(QueryResponse.java:383)
[junit4]at
org.apache.solr.client.solrj.response.QueryResponse.extractFacetInfo(QueryResponse.java:363)
[junit4]at
org.apache.solr.client.solrj.response.QueryResponse.setResponse(QueryResponse.java:148)
[junit4]at
org.apache.solr.client.solrj.response.QueryResponse.init(QueryResponse.java:91)
[junit4]at
org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.java:91)
[junit4]at
org.apache.solr.client.solrj.SolrServer.query(SolrServer.java:301)
[junit4]at
org.apache.solr.cloud.TestCloudPivotFacet.assertPivotCountsAreCorrect(TestCloudPivotFacet.java:161)
[junit4]at
org.apache.solr.cloud.TestCloudPivotFacet.doTest(TestCloudPivotFacet.java:145)
{noformat}
3) Independent (i think) from the NPE issue, there is still something wonky
with the refined counts when mincount is specified...
Here for example is a seed that gets based the QueryResponse.readPivots, but
then fails the numFound validation queries used to check the pivot counts...
{noformat}
[junit4] 2 NOTE: reproduce with: ant test -Dtestcase=TestCloudPivotFacet
-Dtests.method=testDistribSearch -Dtests.seed=F08A107C384690FC
-Dtests.slow=true -Dtests.locale=ar_LY -Dtests.timezone=Jamaica
-Dtests.file.encoding=UTF-8
[junit4] FAILURE 27.0s | TestCloudPivotFacet.testDistribSearch
[junit4] Throwable #1: java.lang.AssertionError:
{main({main(facet.pivot.mincount=9),extra({main(facet.limit=12),extra({main(facet.pivot=pivot_y_s%2Cpivot_x_s1),extra(facet=truefacet.pivot=pivot_x_s1%2Cpivot_x_s)})})}),extra(rows=0q=id%3A%5B*+TO+503%5D)}
== pivot_y_s,pivot_x_s1:
{params(rows=0),defaults({main(rows=0q=id%3A%5B*+TO+503%5D),extra(fq=%7B%21term+f%3Dpivot_y_s%7D)})}
expected:9 but was:14
[junit4]at
__randomizedtesting.SeedInfo.seed([F08A107C384690FC:716C9E644F19F0C0]:0)
[junit4]at
org.apache.solr.cloud.TestCloudPivotFacet.assertPivotCountsAreCorrect(TestCloudPivotFacet.java:190)
[junit4]at
org.apache.solr.cloud.TestCloudPivotFacet.doTest(TestCloudPivotFacet.java:145)
[junit4]at
org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:863)
[junit4]at java.lang.Thread.run(Thread.java:744)
[junit4] Caused by: java.lang.AssertionError: pivot_y_s,pivot_x_s1:
[jira] [Updated] (SOLR-2894) Implement distributed pivot faceting
[ https://issues.apache.org/jira/browse/SOLR-2894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Muldowney updated SOLR-2894: --- Attachment: SOLR-2894.patch The latest patch upload has included Brett's change, along with the changes I outlined earlier. Hoss, I think we've addressed everything up to this point. I've got time to correct any other issues you find. Implement distributed pivot faceting Key: SOLR-2894 URL: https://issues.apache.org/jira/browse/SOLR-2894 Project: Solr Issue Type: Improvement Reporter: Erik Hatcher Fix For: 4.9, 5.0 Attachments: SOLR-2894-mincount-minification.patch, SOLR-2894-reworked.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894_cloud_test.patch, dateToObject.patch, pivot_mincount_problem.sh Following up on SOLR-792, pivot faceting currently only supports undistributed mode. Distributed pivot faceting needs to be implemented. -- This message was sent by Atlassian JIRA (v6.2#6252) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-2894) Implement distributed pivot faceting
[
https://issues.apache.org/jira/browse/SOLR-2894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Muldowney updated SOLR-2894:
---
Attachment: SOLR-2894-mincount-minification.patch
Thank you Hoss, that explanation gave me everything I needed.
So this patch breaks up the modifyRequest block into three parts. First is
the global removal and then running of the modifyRequestForFieldFacets and
modifyRequestForPivotFacets and includes the changed mincount for pivot facet
fields.
This also changes the refinement queries from
facet.pivot={!terms=$cat,inStock_10__terms}cat,inStock
to
facet.pivot={!fpt=1}cat,inStock
This caused some problems since before each term had its own facet.pivot and
thusly its own context and PivotFacetProcessor.
Now that we only have one context for all the refinement requests we needed to
manage our DocSet since it gets messed with. But those issues seem to be fixed.
Implement distributed pivot faceting
Key: SOLR-2894
URL: https://issues.apache.org/jira/browse/SOLR-2894
Project: Solr
Issue Type: Improvement
Reporter: Erik Hatcher
Fix For: 4.9, 5.0
Attachments: SOLR-2894-mincount-minification.patch,
SOLR-2894-reworked.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch,
SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch,
SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch,
SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch,
SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch,
SOLR-2894.patch, SOLR-2894.patch, SOLR-2894_cloud_test.patch,
dateToObject.patch, pivot_mincount_problem.sh
Following up on SOLR-792, pivot faceting currently only supports
undistributed mode. Distributed pivot faceting needs to be implemented.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-2894) Implement distributed pivot faceting
[
https://issues.apache.org/jira/browse/SOLR-2894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hoss Man updated SOLR-2894:
---
Attachment: pivot_mincount_problem.sh
Something's wonky with the way mincount is handled - if you run the attached
pivot_mincount_problem.sh script while a 2 node cluster is running with th
example configs you can see the problem by comparing these 3 URLs...
* Pivot1:
http://localhost:8983/solr/select?rows=0wt=jsonindent=trueq=single_7_s:%284%205%206%29facet=truefacet.pivot=multi_50_ss,single_100_sfacet.limit=10
* Filter:
http://localhost:8983/solr/select?rows=0wt=jsonindent=trueq=single_7_s:%284%205%206%29fq=multi_50_ss:35fq=single_100_s:79
* Pivot2:
http://localhost:8983/solr/select?rows=0wt=jsonindent=trueq=single_7_s:%284%205%206%29facet=truefacet.pivot=multi_50_ss,single_100_sfacet.limit=10facet.pivot.mincount=10
According to the Pivot1 URL, there are 4244 total docs matching the query, of
those 586 match {{multi_50_ss:35}} and of those 13 match {{single_100_s:79}}
This all jives with what the Filter URL tells us (where we ignore the pivot
facets and just apply those as filters)
But if we add {{facet.pivot.mincount=10}} to the original pivot request to get
the Pivot2 URL, no values for single_100_s make the cut as sub-facets of the
586 {{multi_50_ss:35}} docs.
Looking at the logs of the shard queries, it appears that
{{facet.pivot.mincount=-1}} is set _only_ on the refinement queries, but non in
the initial sub-shard queries (where the limit over requesting happens to find
the top terms). So terms that don't match above the mincount on at least one
_single_ shard won't be considered at all for the cumulative total.
Implement distributed pivot faceting
Key: SOLR-2894
URL: https://issues.apache.org/jira/browse/SOLR-2894
Project: Solr
Issue Type: Improvement
Reporter: Erik Hatcher
Fix For: 4.9, 5.0
Attachments: SOLR-2894-reworked.patch, SOLR-2894.patch,
SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch,
SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch,
SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch,
SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch,
SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch,
dateToObject.patch, pivot_mincount_problem.sh
Following up on SOLR-792, pivot faceting currently only supports
undistributed mode. Distributed pivot faceting needs to be implemented.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-2894) Implement distributed pivot faceting
[
https://issues.apache.org/jira/browse/SOLR-2894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hoss Man updated SOLR-2894:
---
Attachment: SOLR-2894_cloud_test.patch
bq. I'm not able to reproduce this. Could you tell me a little more about your
setup?
trunk, with patch applied, build the example and then run the [Simple Two-Shard
Cluster|https://cwiki.apache.org/confluence/display/solr/Getting+Started+with+SolrCloud#GettingStartedwithSolrCloud-SimpleTwo-ShardClusterontheSameMachine]
...
{noformat}
hossman@frisbee:~/lucene/dev/solr$ cp -r example node1
hossman@frisbee:~/lucene/dev/solr$ cp -r example node2
# in term1...
hossman@frisbee:~/lucene/dev/solr/node1$ java -DzkRun -DnumShards=2
-Dbootstrap_confdir=./solr/collection1/conf -Dcollection.configName=myconf -jar
start.jar
# wait for node1 startup, then in term2...
hossman@frisbee:~/lucene/dev/solr/node2$ java -Djetty.port=7574
-DzkHost=localhost:9983 -jar start.jar
# wait for node2 startup, then in term3...
hossman@frisbee:~/lucene/dev/solr/example/exampledocs$ java -jar post.jar *.xml
SimplePostTool version 1.5
Posting files to base url http://localhost:8983/solr/update using content-type
application/xml..
...
14 files indexed.
COMMITting Solr index changes to http://localhost:8983/solr/update..
Time spent: 0:00:01.763
hossman@frisbee:~/lucene/dev/solr/example/exampledocs$ curl
'http://localhost:8983/solr/select?q=*:*sort=id+descrows=2facet=truefacet.pivot=cat,manu_+id_s,inStockfacet.limit=3'
/dev/null
# watch the logs in term1 and term2 go spinning like mad
{noformat}
bq. While the size of the shard parameters may not strictly be as efficient as
possible, is it such that we can run with that for now and circle back to this
at a later point, or are you uncomfortable with including the parameters as is
in the initial commit?
Hmm... not sure how i feel about it w/o more testing - from what i was seeing,
with non-trivial field names, term values, and facet.limit the refinements
requests were getting *HUGE* so I suspect it's something we're going to want to
tackle before releasing -- but refactoring it to be smaller definitely seems
like something that should be a lower priority to some of the correctness
related issues we're finding, and adding more tests (so we can be confident the
refactoring is correct)
I'm attaching a SOLR-2894_cloud_test.patch that contains a new cloud based
randomized test i've been working at off and on over the last few days (I
created it as a standalone patch because i didn't want to conflict with
anything Brett might be in the middle of, and it was easy to do - kept me
focused on the test and not dabbling with the internals).
The test builds up a bunch of random docs, then does a handfull of random pivot
facet queries. For each pivot query, it recursively walks the pivot response
executing verification queries using fq params it builds up from the pivot
constraints -- so if pivot.facet=a,b,c says that a has a term x with 4
matching docs, it adds an fq=a:x to the original query and checks the count;
then it looks a the pivot terms for field b under a:x and also executes a
query for each of them with another fq added, etc...
As is, the patch currently passes, but that's only because of a few nocommits...
* randomization of mincount is disabled due to the refinement bug i mentioned
before
* it's currently only doing pivots on 2 string fields (one multivalued and one
single valued) ... any attempts at pivot faceting the numeric/date/boolean
fields (already included in the docs) causes an NPE in the SolrJ QueryResponse
class (i haven't investigated why yet)
Implement distributed pivot faceting
Key: SOLR-2894
URL: https://issues.apache.org/jira/browse/SOLR-2894
Project: Solr
Issue Type: Improvement
Reporter: Erik Hatcher
Fix For: 4.9, 5.0
Attachments: SOLR-2894-reworked.patch, SOLR-2894.patch,
SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch,
SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch,
SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch,
SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch,
SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch,
SOLR-2894_cloud_test.patch, dateToObject.patch, pivot_mincount_problem.sh
Following up on SOLR-792, pivot faceting currently only supports
undistributed mode. Distributed pivot faceting needs to be implemented.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-2894) Implement distributed pivot faceting
[
https://issues.apache.org/jira/browse/SOLR-2894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hoss Man updated SOLR-2894:
---
Attachment: SOLR-2894.patch
I'm going to spend some time this week reviewing the state of things.
First up, some minor tweaks to the latest patch...
* fixed a typo in TestDistributedSearch (facet.fiedl - facet.field)
** this is my biggest anoyance about most of our existing distributed serach
tests -- they just assert that queries return the same thing as single node
tests, but don't assert anything about the response, so mistakes in the input,
or mistakes in indexing hte docs, resulting in a useless test aren't caught)
** this also relates to marks comment about removing ndate since that field
no longer exists in the test configs - using tdate_a tdate_b here should be
fine
* removed the nocommit mark mentioned regarding DateField - that method moved
to TrieDateField so his fix is correct.
Some comments/questions based on what I've reviewed so far (note: many of these
comments/questions come from a place of genuine ignorance since i've only
reviewed about 30% of the patch so far)...
* even at a glance, it's obvious the SimpleFacets changes are a simple
refactoring and totally fine.
* In FacetComponent - Setting asside the core pivot facet changes...
** Most of the other changes in seem like straight forward (and much needed!)
variable renaming (+1) to help eliminate ambiguity between the existing field
faceting refinement and the new pivot faceting refinement
** the new fieldsToOverRequestOn Map confuses me in a few ways...
*** As is, i don't understand why this is a Map and not a Set.
*** Some odd conditional logic is used when iterating over this Set to
determine the overrequest limit - i'm still trying to wrap my head arround this
but in particular the comment {{// dff has the info, the params have been
scrubbed}} confuses me -- where are these params scrubbed ?
*** I like these new explict overrequest count/ratio params, and i get that the
end-game here is that they can be used to affect the amount of overrequest done
for both fact.field and facet.pivot -- but i'm not understanding the value of
building up this fieldsToOverRequestOn set of names (for every shard request)
and then iterating over it and consulting *either* the DFF or the params to
decide which limit value to use on the shard requests, and then
(conditionally?) removing the limit/offset/overrequest params from the shard
requests. Wouldn't it be simplier to have modifyRequest *always* remove the
limit/offset/overrequest params from the shard params, and then have the
individual code paths (for both facet.field facet.pivot) take responsibility
for adding back the new limit params based on the overrequest calculations
using the *original* request params (ie: {{rb.req.getParams()}}).
*** My chief concern here being that (at first glance) this change seems like
it adds a small amount of overhead to the overrequest limit calculations, and
makes this bit of code more confusing, w/o any obvious (to me) advantage.
* I don't yet understand the need for the new PURPOSE_REFINE_PIVOT_FACETS
stage of shard requests? ... can someone clarify why pivot facets can't just
be refined during the existing PURPOSE_REFINE_FACETS stage?
* I notice that the new DistributedFacetPivotTest directly extends
BaseDistributedSearchTestCase and uses a fixed shard count, and indexes some
docs directly to certain clients
** is there something about the functionality (or about the test) that requires
certain data locality (ie: certain docs on same shard) to work?
** if not: is there any other reason we can't switch this over to a Cloud based
test with a variable numbers of shards and compairons against the control
collection?
Implement distributed pivot faceting
Key: SOLR-2894
URL: https://issues.apache.org/jira/browse/SOLR-2894
Project: Solr
Issue Type: Improvement
Reporter: Erik Hatcher
Fix For: 4.9, 5.0
Attachments: SOLR-2894-reworked.patch, SOLR-2894.patch,
SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch,
SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch,
SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch,
SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch,
SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch,
dateToObject.patch
Following up on SOLR-792, pivot faceting currently only supports
undistributed mode. Distributed pivot faceting needs to be implemented.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-2894) Implement distributed pivot faceting
[ https://issues.apache.org/jira/browse/SOLR-2894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Miller updated SOLR-2894: -- Attachment: SOLR-2894.patch I have not worked on faceting code in the past, so this is really not my area. However, here is a patch I just worked up to apply against 5x. I had to make some small changes - DateField is deprecated and there was an ndate field in the tests that could not be found. I removed it in this patch. I also fixed a few issues around licenses and formatting - this patch passes precommit except for a nocommit at the DateField change - someone should review if that change has any other ramifications. All tests pass. This code does touch some existing faceting code in a way that demands a deeper review I think, but until I have a lot more time, I'm not the man for that job. Perhaps [~hossman_luc...@fucit.org]? Implement distributed pivot faceting Key: SOLR-2894 URL: https://issues.apache.org/jira/browse/SOLR-2894 Project: Solr Issue Type: Improvement Reporter: Erik Hatcher Fix For: 4.9, 5.0 Attachments: SOLR-2894-reworked.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, dateToObject.patch Following up on SOLR-792, pivot faceting currently only supports undistributed mode. Distributed pivot faceting needs to be implemented. -- This message was sent by Atlassian JIRA (v6.2#6252) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-2894) Implement distributed pivot faceting
[ https://issues.apache.org/jira/browse/SOLR-2894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brett Lucey updated SOLR-2894: -- Attachment: SOLR-2894.patch I've uploaded a patch to include changes needed to patch against trunk. (Revision caccba783be7c9f4d7b25c992ed4c49e5a2bddf7). Additionally, this fixes the JSON output formatting issue discovered by AJ. Implement distributed pivot faceting Key: SOLR-2894 URL: https://issues.apache.org/jira/browse/SOLR-2894 Project: Solr Issue Type: Improvement Reporter: Erik Hatcher Fix For: 4.8 Attachments: SOLR-2894-reworked.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, dateToObject.patch Following up on SOLR-792, pivot faceting currently only supports undistributed mode. Distributed pivot faceting needs to be implemented. -- This message was sent by Atlassian JIRA (v6.2#6252) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-2894) Implement distributed pivot faceting
[ https://issues.apache.org/jira/browse/SOLR-2894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Smiley updated SOLR-2894: --- Fix Version/s: (was: 4.7) 4.8 Implement distributed pivot faceting Key: SOLR-2894 URL: https://issues.apache.org/jira/browse/SOLR-2894 Project: Solr Issue Type: Improvement Reporter: Erik Hatcher Fix For: 4.8 Attachments: SOLR-2894-reworked.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, dateToObject.patch Following up on SOLR-792, pivot faceting currently only supports undistributed mode. Distributed pivot faceting needs to be implemented. -- This message was sent by Atlassian JIRA (v6.2#6252) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-2894) Implement distributed pivot faceting
[ https://issues.apache.org/jira/browse/SOLR-2894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brett Lucey updated SOLR-2894: -- Attachment: SOLR-2894.patch I've uploaded the newest version of the patch. This includes a fix for the -1 facet limit when specified on a specific field and incorporates Elran's toObject fix. Implement distributed pivot faceting Key: SOLR-2894 URL: https://issues.apache.org/jira/browse/SOLR-2894 Project: Solr Issue Type: Improvement Reporter: Erik Hatcher Fix For: 4.7 Attachments: SOLR-2894-reworked.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, dateToObject.patch Following up on SOLR-792, pivot faceting currently only supports undistributed mode. Distributed pivot faceting needs to be implemented. -- This message was sent by Atlassian JIRA (v6.2#6252) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-2894) Implement distributed pivot faceting
[ https://issues.apache.org/jira/browse/SOLR-2894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elran Dvir updated SOLR-2894: - Attachment: dateToObject.patch Implement distributed pivot faceting Key: SOLR-2894 URL: https://issues.apache.org/jira/browse/SOLR-2894 Project: Solr Issue Type: Improvement Reporter: Erik Hatcher Fix For: 4.7 Attachments: SOLR-2894-reworked.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, dateToObject.patch Following up on SOLR-792, pivot faceting currently only supports undistributed mode. Distributed pivot faceting needs to be implemented. -- This message was sent by Atlassian JIRA (v6.2#6252) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-2894) Implement distributed pivot faceting
[ https://issues.apache.org/jira/browse/SOLR-2894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brett Lucey updated SOLR-2894: -- Attachment: SOLR-2894.patch This is an updated patch which should apply cleanly against trunk. I've used this against revision 885cdea13918fee0c49d5ac0c5fa1fd286d5b466. This should include a fix for the unlimited facet that Elran brought up. It does not address the toObject issue being discussed. Does anyone have additional input or thoughts as to which route to go with the toObject/string issue? Implement distributed pivot faceting Key: SOLR-2894 URL: https://issues.apache.org/jira/browse/SOLR-2894 Project: Solr Issue Type: Improvement Reporter: Erik Hatcher Fix For: 4.7 Attachments: SOLR-2894-reworked.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch Following up on SOLR-792, pivot faceting currently only supports undistributed mode. Distributed pivot faceting needs to be implemented. -- This message was sent by Atlassian JIRA (v6.2#6252) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-2894) Implement distributed pivot faceting
[ https://issues.apache.org/jira/browse/SOLR-2894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brett Lucey updated SOLR-2894: -- Attachment: SOLR-2894.patch This is an update to the previous patch I uploaded which excludes whitespace changes and eliminates dead code. This does not yet include a fix for the -1 facet limit. Implement distributed pivot faceting Key: SOLR-2894 URL: https://issues.apache.org/jira/browse/SOLR-2894 Project: Solr Issue Type: Improvement Reporter: Erik Hatcher Fix For: 4.7 Attachments: SOLR-2894-reworked.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch Following up on SOLR-792, pivot faceting currently only supports undistributed mode. Distributed pivot faceting needs to be implemented. -- This message was sent by Atlassian JIRA (v6.1.5#6160) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-2894) Implement distributed pivot faceting
[ https://issues.apache.org/jira/browse/SOLR-2894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brett Lucey updated SOLR-2894: -- Attachment: SOLR-2894.patch This is the updated version of our implementation of Pivot Facets, as mentioned by Trey. We have significantly improved performance for cases which involve a large number of shards through changing the underlying data structure and the way that data from the shards is merged together. Implement distributed pivot faceting Key: SOLR-2894 URL: https://issues.apache.org/jira/browse/SOLR-2894 Project: Solr Issue Type: Improvement Reporter: Erik Hatcher Fix For: 4.7 Attachments: SOLR-2894-reworked.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch Following up on SOLR-792, pivot faceting currently only supports undistributed mode. Distributed pivot faceting needs to be implemented. -- This message was sent by Atlassian JIRA (v6.1.5#6160) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-2894) Implement distributed pivot faceting
[ https://issues.apache.org/jira/browse/SOLR-2894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Muldowney updated SOLR-2894: --- Attachment: SOLR-2894.patch Fixed the run-away-but-eventually-coalesing refinement query issue At this point all known issues have been resolved. Implement distributed pivot faceting Key: SOLR-2894 URL: https://issues.apache.org/jira/browse/SOLR-2894 Project: Solr Issue Type: Improvement Reporter: Erik Hatcher Fix For: 4.5 Attachments: SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894-reworked.patch Following up on SOLR-792, pivot faceting currently only supports undistributed mode. Distributed pivot faceting needs to be implemented. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-2894) Implement distributed pivot faceting
[ https://issues.apache.org/jira/browse/SOLR-2894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Muldowney updated SOLR-2894: --- Attachment: SOLR-2894.patch Fixed an issue where commas in string fields would cause infinite refinement loops. Implement distributed pivot faceting Key: SOLR-2894 URL: https://issues.apache.org/jira/browse/SOLR-2894 Project: Solr Issue Type: Improvement Reporter: Erik Hatcher Fix For: 4.5 Attachments: SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894-reworked.patch Following up on SOLR-792, pivot faceting currently only supports undistributed mode. Distributed pivot faceting needs to be implemented. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-2894) Implement distributed pivot faceting
[ https://issues.apache.org/jira/browse/SOLR-2894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Muldowney updated SOLR-2894: --- Attachment: SOLR-2894.patch This version fixes the following: 1) Indecisive faceting not being respected on refinement queries 2) Key not being respected 3) Facet.offset not being respected 4) datetimes breaking when trying to refine One point of contention is this: The SolrExampleTests.java (for the SolrJ stuff) had a check that required pivot facet boolean results as strict Boolean.TRUE as opposed to the string true. This came about from the change that was required to fix datetime. I can't find anywhere else where we require a boolean field's value to equal Boolean.True so I think this test was just an artifact of how the original pivot facetting code was written. As it stands now the SolrExampleTests.doPivotFacetTest:1151 has been changed to true instead of Boolean.TRUE Implement distributed pivot faceting Key: SOLR-2894 URL: https://issues.apache.org/jira/browse/SOLR-2894 Project: Solr Issue Type: Improvement Reporter: Erik Hatcher Fix For: 4.5 Attachments: SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894-reworked.patch Following up on SOLR-792, pivot faceting currently only supports undistributed mode. Distributed pivot faceting needs to be implemented. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-2894) Implement distributed pivot faceting
[ https://issues.apache.org/jira/browse/SOLR-2894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Otis Gospodnetic updated SOLR-2894: --- Fix Version/s: (was: 4.4) 4.5 Implement distributed pivot faceting Key: SOLR-2894 URL: https://issues.apache.org/jira/browse/SOLR-2894 Project: Solr Issue Type: Improvement Reporter: Erik Hatcher Fix For: 4.5 Attachments: SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894-reworked.patch Following up on SOLR-792, pivot faceting currently only supports undistributed mode. Distributed pivot faceting needs to be implemented. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-2894) Implement distributed pivot faceting
[ https://issues.apache.org/jira/browse/SOLR-2894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Muldowney updated SOLR-2894: --- Attachment: 0001-Pivot-Faceting-and-refinement.patch This was built for 4_1_0 but git thinks it'll apply to trunk no problem. This solves a myriad of issues surrounding the formatting of the refinement requests, should support all field types and deals with jagged pivot facet result sets due to nulls or empty data on pivoted fields. Implement distributed pivot faceting Key: SOLR-2894 URL: https://issues.apache.org/jira/browse/SOLR-2894 Project: Solr Issue Type: Improvement Reporter: Erik Hatcher Fix For: 4.3 Attachments: 0001-Pivot-Faceting-and-refinement.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894-reworked.patch Following up on SOLR-792, pivot faceting currently only supports undistributed mode. Distributed pivot faceting needs to be implemented. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-2894) Implement distributed pivot faceting
[ https://issues.apache.org/jira/browse/SOLR-2894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Muldowney updated SOLR-2894: --- Attachment: (was: 0001-Pivot-Faceting-and-refinement.patch) Implement distributed pivot faceting Key: SOLR-2894 URL: https://issues.apache.org/jira/browse/SOLR-2894 Project: Solr Issue Type: Improvement Reporter: Erik Hatcher Fix For: 4.3 Attachments: SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894-reworked.patch Following up on SOLR-792, pivot faceting currently only supports undistributed mode. Distributed pivot faceting needs to be implemented. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-2894) Implement distributed pivot faceting
[ https://issues.apache.org/jira/browse/SOLR-2894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Muldowney updated SOLR-2894: --- Attachment: SOLR-2894.patch Implement distributed pivot faceting Key: SOLR-2894 URL: https://issues.apache.org/jira/browse/SOLR-2894 Project: Solr Issue Type: Improvement Reporter: Erik Hatcher Fix For: 4.3 Attachments: SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894-reworked.patch Following up on SOLR-792, pivot faceting currently only supports undistributed mode. Distributed pivot faceting needs to be implemented. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-2894) Implement distributed pivot faceting
[ https://issues.apache.org/jira/browse/SOLR-2894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Muldowney updated SOLR-2894: --- Attachment: SOLR-2894.patch Fixes the NPEs in 3pivot along with solving a hidden issue with null values when .missing was false and dealt with an issue where the facet.mincount is different from the facet.pivot.mincount (which is true for the default). The mincount issue only showed itself during index sorting. Implement distributed pivot faceting Key: SOLR-2894 URL: https://issues.apache.org/jira/browse/SOLR-2894 Project: Solr Issue Type: Improvement Reporter: Erik Hatcher Fix For: 4.2, 5.0 Attachments: SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894-reworked.patch Following up on SOLR-792, pivot faceting currently only supports undistributed mode. Distributed pivot faceting needs to be implemented. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-2894) Implement distributed pivot faceting
[ https://issues.apache.org/jira/browse/SOLR-2894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Muldowney updated SOLR-2894: --- Attachment: SOLR-2894.patch This patch applies cleanly to trunk for me, apologies as the last one upon review was flawed. Implement distributed pivot faceting Key: SOLR-2894 URL: https://issues.apache.org/jira/browse/SOLR-2894 Project: Solr Issue Type: Improvement Reporter: Erik Hatcher Fix For: 4.2, 5.0 Attachments: SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894-reworked.patch Following up on SOLR-792, pivot faceting currently only supports undistributed mode. Distributed pivot faceting needs to be implemented. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-2894) Implement distributed pivot faceting
[ https://issues.apache.org/jira/browse/SOLR-2894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Muldowney updated SOLR-2894: --- Attachment: SOLR-2894.patch New patch file for trunk, its a git patch from trunk solr pulled today, let me know if there are any issues applying. Implement distributed pivot faceting Key: SOLR-2894 URL: https://issues.apache.org/jira/browse/SOLR-2894 Project: Solr Issue Type: Improvement Reporter: Erik Hatcher Fix For: 4.2, 5.0 Attachments: SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894-reworked.patch Following up on SOLR-792, pivot faceting currently only supports undistributed mode. Distributed pivot faceting needs to be implemented. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-2894) Implement distributed pivot faceting
[ https://issues.apache.org/jira/browse/SOLR-2894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Muldowney updated SOLR-2894: --- Attachment: (was: SOLR-2894.patch) Implement distributed pivot faceting Key: SOLR-2894 URL: https://issues.apache.org/jira/browse/SOLR-2894 Project: Solr Issue Type: Improvement Reporter: Erik Hatcher Fix For: 4.2, 5.0 Attachments: SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894-reworked.patch Following up on SOLR-792, pivot faceting currently only supports undistributed mode. Distributed pivot faceting needs to be implemented. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-2894) Implement distributed pivot faceting
[ https://issues.apache.org/jira/browse/SOLR-2894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Muldowney updated SOLR-2894: --- Attachment: SOLR-2894.patch An update to my previous path. After more consideration, I've pulled the pivot facet logic out of DistributedProcess and call those specific parts when doing the iterative refinement process to avoid any side effects of other people putting code into DistributedProcess. I've also added better support for null values and added tests to ensure that you can send refinement requests for null values and not blow up the stack. Implement distributed pivot faceting Key: SOLR-2894 URL: https://issues.apache.org/jira/browse/SOLR-2894 Project: Solr Issue Type: Improvement Reporter: Erik Hatcher Fix For: 4.2, 5.0 Attachments: SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894-reworked.patch Following up on SOLR-792, pivot faceting currently only supports undistributed mode. Distributed pivot faceting needs to be implemented. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-2894) Implement distributed pivot faceting
[ https://issues.apache.org/jira/browse/SOLR-2894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Muldowney updated SOLR-2894: --- Attachment: SOLR-2894.patch Distributed pivot faceting, along with testing Implement distributed pivot faceting Key: SOLR-2894 URL: https://issues.apache.org/jira/browse/SOLR-2894 Project: Solr Issue Type: Improvement Reporter: Erik Hatcher Fix For: 4.2, 5.0 Attachments: SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894-reworked.patch Following up on SOLR-792, pivot faceting currently only supports undistributed mode. Distributed pivot faceting needs to be implemented. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-2894) Implement distributed pivot faceting
[ https://issues.apache.org/jira/browse/SOLR-2894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Russell updated SOLR-2894: Attachment: SOLR-2894.patch Corrected null aggregation issues when docs contain null values for fields pivoting on. Added logic to remove local params from pivot QS vars when determining over-request. Implement distributed pivot faceting Key: SOLR-2894 URL: https://issues.apache.org/jira/browse/SOLR-2894 Project: Solr Issue Type: Improvement Reporter: Erik Hatcher Fix For: 4.2, 5.0 Attachments: SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894-reworked.patch Following up on SOLR-792, pivot faceting currently only supports undistributed mode. Distributed pivot faceting needs to be implemented. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-2894) Implement distributed pivot faceting
[ https://issues.apache.org/jira/browse/SOLR-2894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Russell updated SOLR-2894: Attachment: SOLR-2894.patch Fixed NRE when using facet.missing. Added test for over-requesting / refinement. Fixed issue that broke over-requesting when local params were present in facet.field. Implement distributed pivot faceting Key: SOLR-2894 URL: https://issues.apache.org/jira/browse/SOLR-2894 Project: Solr Issue Type: Improvement Reporter: Erik Hatcher Fix For: 4.2, 5.0 Attachments: SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894-reworked.patch Following up on SOLR-792, pivot faceting currently only supports undistributed mode. Distributed pivot faceting needs to be implemented. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-2894) Implement distributed pivot faceting
[ https://issues.apache.org/jira/browse/SOLR-2894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Russell updated SOLR-2894: Attachment: SOLR-2894.patch Implemented default pivot facet sort. Implemented per-field pivot facet sorting. Fixed NRE with sorting when facet.missing is on. Implement distributed pivot faceting Key: SOLR-2894 URL: https://issues.apache.org/jira/browse/SOLR-2894 Project: Solr Issue Type: Improvement Reporter: Erik Hatcher Fix For: 4.1 Attachments: SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894-reworked.patch Following up on SOLR-792, pivot faceting currently only supports undistributed mode. Distributed pivot faceting needs to be implemented. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-2894) Implement distributed pivot faceting
[ https://issues.apache.org/jira/browse/SOLR-2894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Russell updated SOLR-2894: Attachment: SOLR-2894.patch Implement distributed pivot faceting Key: SOLR-2894 URL: https://issues.apache.org/jira/browse/SOLR-2894 Project: Solr Issue Type: Improvement Reporter: Erik Hatcher Fix For: 4.1 Attachments: distributed_pivot.patch, distributed_pivot.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894-reworked.patch Following up on SOLR-792, pivot faceting currently only supports undistributed mode. Distributed pivot faceting needs to be implemented. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-2894) Implement distributed pivot faceting
[ https://issues.apache.org/jira/browse/SOLR-2894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Russell updated SOLR-2894: Attachment: (was: distributed_pivot.patch) Implement distributed pivot faceting Key: SOLR-2894 URL: https://issues.apache.org/jira/browse/SOLR-2894 Project: Solr Issue Type: Improvement Reporter: Erik Hatcher Fix For: 4.1 Attachments: SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894-reworked.patch Following up on SOLR-792, pivot faceting currently only supports undistributed mode. Distributed pivot faceting needs to be implemented. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-2894) Implement distributed pivot faceting
[ https://issues.apache.org/jira/browse/SOLR-2894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Russell updated SOLR-2894: Attachment: (was: distributed_pivot.patch) Implement distributed pivot faceting Key: SOLR-2894 URL: https://issues.apache.org/jira/browse/SOLR-2894 Project: Solr Issue Type: Improvement Reporter: Erik Hatcher Fix For: 4.1 Attachments: SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894-reworked.patch Following up on SOLR-792, pivot faceting currently only supports undistributed mode. Distributed pivot faceting needs to be implemented. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-2894) Implement distributed pivot faceting
[ https://issues.apache.org/jira/browse/SOLR-2894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Russell updated SOLR-2894: Attachment: SOLR-2894.patch Implement distributed pivot faceting Key: SOLR-2894 URL: https://issues.apache.org/jira/browse/SOLR-2894 Project: Solr Issue Type: Improvement Reporter: Erik Hatcher Fix For: 4.1 Attachments: SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894-reworked.patch Following up on SOLR-792, pivot faceting currently only supports undistributed mode. Distributed pivot faceting needs to be implemented. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-2894) Implement distributed pivot faceting
[ https://issues.apache.org/jira/browse/SOLR-2894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Russell updated SOLR-2894: Attachment: (was: SOLR-2894.patch) Implement distributed pivot faceting Key: SOLR-2894 URL: https://issues.apache.org/jira/browse/SOLR-2894 Project: Solr Issue Type: Improvement Reporter: Erik Hatcher Fix For: 4.1 Attachments: SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894-reworked.patch Following up on SOLR-792, pivot faceting currently only supports undistributed mode. Distributed pivot faceting needs to be implemented. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-2894) Implement distributed pivot faceting
[ https://issues.apache.org/jira/browse/SOLR-2894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dzmitry Zhemchuhou updated SOLR-2894: - Attachment: SOLR-2894-reworked.patch I have reapplied the SOLR-2894 patch from Jun 14th to the trunk while removing most of code formatting changes that were in it. On top of that I changed the FacetComponent.refineFacets() method to add facet_pivot key-value only when there are values in the pivotFacets map, which fixes distributed search unit tests. Implement distributed pivot faceting Key: SOLR-2894 URL: https://issues.apache.org/jira/browse/SOLR-2894 Project: Solr Issue Type: Improvement Reporter: Erik Hatcher Fix For: 4.0 Attachments: SOLR-2894-reworked.patch, SOLR-2894.patch, SOLR-2894.patch, distributed_pivot.patch, distributed_pivot.patch Following up on SOLR-792, pivot faceting currently only supports undistributed mode. Distributed pivot faceting needs to be implemented. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-2894) Implement distributed pivot faceting
[ https://issues.apache.org/jira/browse/SOLR-2894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erik Hatcher updated SOLR-2894: --- Attachment: SOLR-2894.patch Patch updated to 4x branch. Simon, just for you, I removed NamedListHelper as well :) (folded its one method into PivotFacetHelper) Tests pass. Implement distributed pivot faceting Key: SOLR-2894 URL: https://issues.apache.org/jira/browse/SOLR-2894 Project: Solr Issue Type: Improvement Reporter: Erik Hatcher Assignee: Erik Hatcher Fix For: 4.0 Attachments: SOLR-2894.patch, SOLR-2894.patch, distributed_pivot.patch, distributed_pivot.patch Following up on SOLR-792, pivot faceting currently only supports undistributed mode. Distributed pivot faceting needs to be implemented. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-2894) Implement distributed pivot faceting
[ https://issues.apache.org/jira/browse/SOLR-2894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Russell updated SOLR-2894: Attachment: distributed_pivot.patch facet_pivot will not show up in distrib search if no contents, reversed behavior of sorting to comply with solr standard for facet.sort Implement distributed pivot faceting Key: SOLR-2894 URL: https://issues.apache.org/jira/browse/SOLR-2894 Project: Solr Issue Type: Improvement Reporter: Erik Hatcher Fix For: 4.0 Attachments: SOLR-2894.patch, distributed_pivot.patch, distributed_pivot.patch Following up on SOLR-792, pivot faceting currently only supports undistributed mode. Distributed pivot faceting needs to be implemented. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-2894) Implement distributed pivot faceting
[ https://issues.apache.org/jira/browse/SOLR-2894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Russell updated SOLR-2894: Attachment: distributed_pivot.patch Some modifications to SOLR-2894.patch that I made while trying to get it to patch on rev 1297102. Implement distributed pivot faceting Key: SOLR-2894 URL: https://issues.apache.org/jira/browse/SOLR-2894 Project: Solr Issue Type: Improvement Reporter: Erik Hatcher Fix For: 4.0 Attachments: SOLR-2894.patch, distributed_pivot.patch Following up on SOLR-792, pivot faceting currently only supports undistributed mode. Distributed pivot faceting needs to be implemented. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-2894) Implement distributed pivot faceting
[ https://issues.apache.org/jira/browse/SOLR-2894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Miller updated SOLR-2894: -- Affects Version/s: (was: 4.0) Fix Version/s: 4.0 Implement distributed pivot faceting Key: SOLR-2894 URL: https://issues.apache.org/jira/browse/SOLR-2894 Project: Solr Issue Type: Improvement Reporter: Erik Hatcher Fix For: 4.0 Attachments: SOLR-2894.patch Following up on SOLR-792, pivot faceting currently only supports undistributed mode. Distributed pivot faceting needs to be implemented. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-2894) Implement distributed pivot faceting
[ https://issues.apache.org/jira/browse/SOLR-2894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dan Cooper updated SOLR-2894: - Attachment: distribPatch-05-03-12.txt Implement distributed pivot faceting Key: SOLR-2894 URL: https://issues.apache.org/jira/browse/SOLR-2894 Project: Solr Issue Type: Improvement Affects Versions: 4.0 Reporter: Erik Hatcher Attachments: distribPatch-05-03-12.txt Following up on SOLR-792, pivot faceting currently only supports undistributed mode. Distributed pivot faceting needs to be implemented. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-2894) Implement distributed pivot faceting
[ https://issues.apache.org/jira/browse/SOLR-2894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dan Cooper updated SOLR-2894: - Attachment: (was: distribPatch-05-03-12.txt) Implement distributed pivot faceting Key: SOLR-2894 URL: https://issues.apache.org/jira/browse/SOLR-2894 Project: Solr Issue Type: Improvement Affects Versions: 4.0 Reporter: Erik Hatcher Following up on SOLR-792, pivot faceting currently only supports undistributed mode. Distributed pivot faceting needs to be implemented. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-2894) Implement distributed pivot faceting
[ https://issues.apache.org/jira/browse/SOLR-2894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dan Cooper updated SOLR-2894: - Attachment: (was: SOLR-2894.patch) Implement distributed pivot faceting Key: SOLR-2894 URL: https://issues.apache.org/jira/browse/SOLR-2894 Project: Solr Issue Type: Improvement Affects Versions: 4.0 Reporter: Erik Hatcher Following up on SOLR-792, pivot faceting currently only supports undistributed mode. Distributed pivot faceting needs to be implemented. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-2894) Implement distributed pivot faceting
[ https://issues.apache.org/jira/browse/SOLR-2894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dan Cooper updated SOLR-2894: - Attachment: SOLR-2894.patch Implement distributed pivot faceting Key: SOLR-2894 URL: https://issues.apache.org/jira/browse/SOLR-2894 Project: Solr Issue Type: Improvement Affects Versions: 4.0 Reporter: Erik Hatcher Following up on SOLR-792, pivot faceting currently only supports undistributed mode. Distributed pivot faceting needs to be implemented. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-2894) Implement distributed pivot faceting
[ https://issues.apache.org/jira/browse/SOLR-2894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dan Cooper updated SOLR-2894: - Attachment: SOLR-2894.patch Implement distributed pivot faceting Key: SOLR-2894 URL: https://issues.apache.org/jira/browse/SOLR-2894 Project: Solr Issue Type: Improvement Affects Versions: 4.0 Reporter: Erik Hatcher Attachments: SOLR-2894.patch Following up on SOLR-792, pivot faceting currently only supports undistributed mode. Distributed pivot faceting needs to be implemented. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
