[jira] [Updated] (SOLR-6351) Let Stats Hang off of Pivots (via 'tag')
[
https://issues.apache.org/jira/browse/SOLR-6351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hoss Man updated SOLR-6351:
---
Attachment: SOLR-6351.patch
small API tweak...
* renamed PivotField.getFieldStatsInfo_s_() to PivotField.getFieldStatsInfo()
** this makes it consistent with the same method name in QueryResponse
** updated all callers
..planning to commit to trunk a little later today.
Let Stats Hang off of Pivots (via 'tag')
Key: SOLR-6351
URL: https://issues.apache.org/jira/browse/SOLR-6351
Project: Solr
Issue Type: Sub-task
Reporter: Hoss Man
Attachments: SOLR-6351.patch, SOLR-6351.patch, SOLR-6351.patch,
SOLR-6351.patch, SOLR-6351.patch, SOLR-6351.patch, SOLR-6351.patch,
SOLR-6351.patch, SOLR-6351.patch, SOLR-6351.patch, SOLR-6351.patch,
SOLR-6351.patch, SOLR-6351.patch, SOLR-6351.patch, SOLR-6351.patch,
SOLR-6351.patch, SOLR-6351.patch, SOLR-6351.patch, SOLR-6351.patch
he goal here is basically flip the notion of stats.facet on it's head, so
that instead of asking the stats component to also do some faceting
(something that's never worked well with the variety of field types and has
never worked in distributed mode) we instead ask the PivotFacet code to
compute some stats X for each leaf in a pivot. We'll do this with the
existing {{stats.field}} params, but we'll leverage the {{tag}} local param
of the {{stats.field}} instances to be able to associate which stats we want
hanging off of which {{facet.pivot}}
Example...
{noformat}
facet.pivot={!stats=s1}category,manufacturer
stats.field={!key=avg_price tag=s1 mean=true}price
stats.field={!tag=s1 min=true max=true}user_rating
{noformat}
...with the request above, in addition to computing the min/max user_rating
and mean price (labeled avg_price) over the entire result set, the
PivotFacet component will also include those stats for every node of the tree
it builds up when generating a pivot of the fields category,manufacturer
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-6351) Let Stats Hang off of Pivots (via 'tag')
[
https://issues.apache.org/jira/browse/SOLR-6351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hoss Man updated SOLR-6351:
---
Attachment: SOLR-6351.patch
bq. ...i can't understand how/why
DistributedFacetPivotSmallAdvancedTest.doTestTopStatsWithRefinement passes in
this patch, and that scares me (the refinement call should pollute the top
level stats, double counting at least one shard) so i really want to get to the
bottom of it.
After doing a bunch of manual testing to confirm that the bug really did exist,
i started pouring over the test logs and confirmed there was a stupid bug in
the test itself...
{code}
// i had...
ModifiableSolrParams facetForceRefineParams = new
ModifiableSolrParams(coreParams);
// should have been...
ModifiableSolrParams facetForceRefineParams = new
ModifiableSolrParams(facetParams);
{code}
...so in the request where i was trying to force facet refinement by adding
{{FacetParams.FACET_OVERREQUEST_COUNT=0}} to the params, i wasn't even faceting
at all.
Once that silly mistake was fixed, the test started failing as expected.
I then beefed up Vitaliy's new DistributedFacetPivotWhiteBoxTest to include
some asserts on the top level stats, and confirmed those also failed because
we're re-computing them on both the initial request, and the subsequent
refinement request.
Once i had those tests failing in a satisfactory way, finding a solution was
fairly straight forward: stop including {{stats=true}} in pivot refinement
requests, and instead make PivotFacetProcessor recognise when it needs to build
up a StatsInfo object to handle refinement requests (rather then relying on
StatsComponent to do it for us).
I think this is basically ready to commit - i'm going to give the patch issue
comments another going over tomorrow to see if anything pops out at me while my
laptop hammers away at the tests, but if no one has any other feedback i'll
commit to trunk start backporting.
{panel:title=summary of changes}
* DistributedFacetPivotSmallAdvancedTest
** fixed broken test of top level stats
** added some assertion msg strings
* StatsComponent
** removed the last remaining nocommit hack that set stats=true on pivot
refinements
* PivotFacetProcessor
** changed process() method:
*** init a {{StatsInfo}} object on demand in cases where it's a refinement
request there is a stats local param
*** call getTaggedStatsFields() to build up the {{ListStatsField}} once per
facet.pivot param (was being redundently called once per refinement value in
old code)
** changed processSingle() to take in the {{ListStatsField}} as a method param
** tweaked getTaggedStatsFields() method to take in just hte stats local param
instead of the full set of localparams -- simplified the parsing error
handling
* DistributedFacetPivotWhiteBoxTest
** beefed up test to check top level stats
** tweaked refine params to set {{stats=false}} to match what the code now does
* TestCloudPivotFacet
** fixed javadoc (lint error)
* PivotField
** fixed javadoc (lint error)
{panel}
Let Stats Hang off of Pivots (via 'tag')
Key: SOLR-6351
URL: https://issues.apache.org/jira/browse/SOLR-6351
Project: Solr
Issue Type: Sub-task
Reporter: Hoss Man
Attachments: SOLR-6351.patch, SOLR-6351.patch, SOLR-6351.patch,
SOLR-6351.patch, SOLR-6351.patch, SOLR-6351.patch, SOLR-6351.patch,
SOLR-6351.patch, SOLR-6351.patch, SOLR-6351.patch, SOLR-6351.patch,
SOLR-6351.patch, SOLR-6351.patch, SOLR-6351.patch, SOLR-6351.patch,
SOLR-6351.patch, SOLR-6351.patch, SOLR-6351.patch
he goal here is basically flip the notion of stats.facet on it's head, so
that instead of asking the stats component to also do some faceting
(something that's never worked well with the variety of field types and has
never worked in distributed mode) we instead ask the PivotFacet code to
compute some stats X for each leaf in a pivot. We'll do this with the
existing {{stats.field}} params, but we'll leverage the {{tag}} local param
of the {{stats.field}} instances to be able to associate which stats we want
hanging off of which {{facet.pivot}}
Example...
{noformat}
facet.pivot={!stats=s1}category,manufacturer
stats.field={!key=avg_price tag=s1 mean=true}price
stats.field={!tag=s1 min=true max=true}user_rating
{noformat}
...with the request above, in addition to computing the min/max user_rating
and mean price (labeled avg_price) over the entire result set, the
PivotFacet component will also include those stats for every node of the tree
it builds up when generating a pivot of the fields category,manufacturer
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail:
[jira] [Updated] (SOLR-6351) Let Stats Hang off of Pivots (via 'tag')
[
https://issues.apache.org/jira/browse/SOLR-6351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vitaliy Zhovtyuk updated SOLR-6351:
---
Attachment: SOLR-6351.patch
Added whitebox DistributedFacetPivotWhiteBoxTest test simulating pivot stats
shard requests in cases: get top level pivots and refinement requests. Both
contains stats on pivots.
Let Stats Hang off of Pivots (via 'tag')
Key: SOLR-6351
URL: https://issues.apache.org/jira/browse/SOLR-6351
Project: Solr
Issue Type: Sub-task
Reporter: Hoss Man
Attachments: SOLR-6351.patch, SOLR-6351.patch, SOLR-6351.patch,
SOLR-6351.patch, SOLR-6351.patch, SOLR-6351.patch, SOLR-6351.patch,
SOLR-6351.patch, SOLR-6351.patch, SOLR-6351.patch, SOLR-6351.patch,
SOLR-6351.patch, SOLR-6351.patch, SOLR-6351.patch, SOLR-6351.patch,
SOLR-6351.patch, SOLR-6351.patch
he goal here is basically flip the notion of stats.facet on it's head, so
that instead of asking the stats component to also do some faceting
(something that's never worked well with the variety of field types and has
never worked in distributed mode) we instead ask the PivotFacet code to
compute some stats X for each leaf in a pivot. We'll do this with the
existing {{stats.field}} params, but we'll leverage the {{tag}} local param
of the {{stats.field}} instances to be able to associate which stats we want
hanging off of which {{facet.pivot}}
Example...
{noformat}
facet.pivot={!stats=s1}category,manufacturer
stats.field={!key=avg_price tag=s1 mean=true}price
stats.field={!tag=s1 min=true max=true}user_rating
{noformat}
...with the request above, in addition to computing the min/max user_rating
and mean price (labeled avg_price) over the entire result set, the
PivotFacet component will also include those stats for every node of the tree
it builds up when generating a pivot of the fields category,manufacturer
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-6351) Let Stats Hang off of Pivots (via 'tag')
[
https://issues.apache.org/jira/browse/SOLR-6351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hoss Man updated SOLR-6351:
---
Attachment: SOLR-6351.patch
review some updates to the the latest tests Vitaliy added.
On monday i'll dig into how we're dealing with stas in pivot refinements (the
last remaining nocommit ... i can't understand how/why
DistributedFacetPivotSmallAdvancedTest.doTestTopStatsWithRefinement passes in
this patch, and that scares me (the refinement call should pollute the top
level stats, double counting at least one shard) so i really want to get to the
bottom of it.
{panel:title=patch changes}
* FacetPivotSmallTest
** new testBogusStatsTag: check that a bogus stats tag is ignored
** new testStatsTagHasComma: check that comma causes error for now (SOLR-6663)
* SolrExampleTests
** swap two comments that were on the wrong asserts
* DistributedFacetPivotSmallAdvancedTest
** cleaned up some formatting
** doTestTopStats
*** beefed up the assertions
*** added sanity checks that refinement really was happening in these requests
*** suprisingly this test still passes ... based on the nocommit hack i put
in StatsComponent, I was farily certain this was going to fail ... i need to
dig in more and make sure i understand why it doesn't fail.
*** renamed to doTestTopStatsWithRefinement to make it a bit more clear what
it's doing
** doTestDeepPivotStatsOnOverrequest
*** removed this test - i wasn't really clear on the purpose, and in asking
Vitaliy about it on IRC we realized he had missunderstood some advice i'd given
him on IRC a few days ago about how to sanity check that refinement was
happening (ie: what i just added to doTestTopStatsWithRefinement) so it wasn't
given us any value add.
{panel}
Let Stats Hang off of Pivots (via 'tag')
Key: SOLR-6351
URL: https://issues.apache.org/jira/browse/SOLR-6351
Project: Solr
Issue Type: Sub-task
Reporter: Hoss Man
Attachments: SOLR-6351.patch, SOLR-6351.patch, SOLR-6351.patch,
SOLR-6351.patch, SOLR-6351.patch, SOLR-6351.patch, SOLR-6351.patch,
SOLR-6351.patch, SOLR-6351.patch, SOLR-6351.patch, SOLR-6351.patch,
SOLR-6351.patch, SOLR-6351.patch, SOLR-6351.patch, SOLR-6351.patch,
SOLR-6351.patch
he goal here is basically flip the notion of stats.facet on it's head, so
that instead of asking the stats component to also do some faceting
(something that's never worked well with the variety of field types and has
never worked in distributed mode) we instead ask the PivotFacet code to
compute some stats X for each leaf in a pivot. We'll do this with the
existing {{stats.field}} params, but we'll leverage the {{tag}} local param
of the {{stats.field}} instances to be able to associate which stats we want
hanging off of which {{facet.pivot}}
Example...
{noformat}
facet.pivot={!stats=s1}category,manufacturer
stats.field={!key=avg_price tag=s1 mean=true}price
stats.field={!tag=s1 min=true max=true}user_rating
{noformat}
...with the request above, in addition to computing the min/max user_rating
and mean price (labeled avg_price) over the entire result set, the
PivotFacet component will also include those stats for every node of the tree
it builds up when generating a pivot of the fields category,manufacturer
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-6351) Let Stats Hang off of Pivots (via 'tag')
[
https://issues.apache.org/jira/browse/SOLR-6351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hoss Man updated SOLR-6351:
---
Attachment: SOLR-6351.patch
I started out focusing on fixing the +/-Infinity issue i mentioned yesterday,
but while working on beefing up the asserts for this in
DistributedFacetPivotSmallTest i realized the PivotField.getStatsInfos() method
was returning ListFieldStatsInfo which didn't really make a lot of sense - so
i changed that to return MapString,FieldStatsInfo and tweaked the method name
(consistent with how top level stats method works/is-named) and updated all of
the affected code/tests as well.
{panel:title=summay of patch changes}
* StatsValuesFactory
** fixed NumericStatsValues so that min/max are 'null' when no values
accumulated
* changed PivotField.getStatsInfos() to getFieldStatsInfos() and made it return
a map -- updated the following classes...
** PivotField
** QueryResponse
** SolrExampleTests
** TestCloudPivotFacet
** DistributedFacetPivotSmallTest
** DistributedFacetPivotLargeTest
** DistributedFacetPivotLongTailTest
** DistributedFacetPivotSmallAdvancedTest
* DistributedFacetPivotSmallTest
** fixed some +/-Infinity assumptions based on NumericStatsValues fixes
** beefed up testing of diff data types in doTestPivotStatsFromOneShard
* FacetPivotSmallTest
** fixed some +/-Infinity assumptions based on NumericStatsValues fixes
{panel}
Let Stats Hang off of Pivots (via 'tag')
Key: SOLR-6351
URL: https://issues.apache.org/jira/browse/SOLR-6351
Project: Solr
Issue Type: Sub-task
Reporter: Hoss Man
Attachments: SOLR-6351.patch, SOLR-6351.patch, SOLR-6351.patch,
SOLR-6351.patch, SOLR-6351.patch, SOLR-6351.patch, SOLR-6351.patch,
SOLR-6351.patch, SOLR-6351.patch, SOLR-6351.patch, SOLR-6351.patch,
SOLR-6351.patch, SOLR-6351.patch, SOLR-6351.patch, SOLR-6351.patch
he goal here is basically flip the notion of stats.facet on it's head, so
that instead of asking the stats component to also do some faceting
(something that's never worked well with the variety of field types and has
never worked in distributed mode) we instead ask the PivotFacet code to
compute some stats X for each leaf in a pivot. We'll do this with the
existing {{stats.field}} params, but we'll leverage the {{tag}} local param
of the {{stats.field}} instances to be able to associate which stats we want
hanging off of which {{facet.pivot}}
Example...
{noformat}
facet.pivot={!stats=s1}category,manufacturer
stats.field={!key=avg_price tag=s1 mean=true}price
stats.field={!tag=s1 min=true max=true}user_rating
{noformat}
...with the request above, in addition to computing the min/max user_rating
and mean price (labeled avg_price) over the entire result set, the
PivotFacet component will also include those stats for every node of the tree
it builds up when generating a pivot of the fields category,manufacturer
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-6351) Let Stats Hang off of Pivots (via 'tag')
[
https://issues.apache.org/jira/browse/SOLR-6351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hoss Man updated SOLR-6351:
---
Attachment: SOLR-6351.patch
My main focus the last day or so has been reviewing PivotFacetHelper
PivotFacetValue with an eye towards simplifying the amount of redundent code
between them and StatsComponent. Some details posted below but one key thing i
wanted to point out...
Even as (relatively) familiar as i am with the exsting Pivot code, it took me a
long time to understand how PivotFacetHelper.getStats + PivotListEntry.STATS
were working in the case of leaf level pivot values -- short answer:
PivotFacetHelper.getStats totally ignores the Enum value of
PivotListEntry.STATS and uses 0 (something PivotFacetHelper.getPivots also
does that i've never noticed before).
Given that we plan to add more data to pivots in issues like SOLR-4212
SOLR-6353, i really wanted to come up with a pattern for dealing with this that
was less likeely to trip people up when looking at the code.
{panel:title=Changes in this patch}
* StatsComponent
** refactored out tiny little reusable unwrapStats utility
** refactored out reusable convertToResponse utility
*** i was hoping this would help encapsulate simplify the way the count==0
rules are applied, to make top level consistent with pivots, but that lead me
down a rabbit hole of pain as far as testing and backcompat and solrj - so i
just captured it in a 'force' method param.
*** But at least now, the method is consistently called everywhere that outputs
stats, so if/when we change the rules for how empty stats are returned (see
comments in SOLR-6349) we won't need to audit/change multiple pieces of code,
we can just focus on callers of this method
** Added a StatsInfo.getStatsField(key) method for use by
PivotFacetHelper.mergeStats so it wouldn't need to constantly loop over every
possible stats.field
* PivotFacetValue
** removed an unneccessary level of wrapping arround the MapString,StatsValues
** switched to using StatsComponent.convertToResponse directly instead of
PivotFacetHelper.convertStatsValuesToNamedList
* PivotListEntry
** renamed index to minIndex
** added an extract method that knows how to correctly deal with the diff
between optional entries that may exist starting at the minIndex, and
mandatory entires (field,value,count) that *must* exist at the expected index.
* PivotFacetHelper
** changed the various getFoo methods to use PivotListEntry.FOO.extract
*** these methods now exact mainly just for convinience with the Object casting
*** this also ment the retrieve method could be removed
** simplified mergeStats via:
*** StatsComponent.unwrapStats
*** StatsInfo.getStatsField
** mergeStats javadocs
** removed convertStatsValuesToNamedList
* PivotFacetProcessor
** switch using StatsComponent.convertToResponse
* TestCloudPivots
** update nocommit comment regarding 'null' actualStats based on pain
encountered working on StastComponent.convertToResponse
*** added some more sanity check assertions in this case as well
* DistributedFacetPivotSmallTest
** added doTestPivotStatsFromOneShard to account for an edge case in merging
that occured to me while reviewing PivotFacetHelper.mergeStats
*** this fails because of how +/-Infinity are treated as the min/max - i'll
working on fixing this next
*** currently commented out + has some nocommits to beef up this test w/other
types
* merged my working changes with Vitaliy's additions (but have not yet actually
reviewed the new tests)...
** FacetPivotSmallTest
** DistributedFacetPivotSmallAdvancedTest
** PivotFacetValue.getStatsValues ... allthough it's not clear to me yet what
purpose/value this adds?
{panel}
Let Stats Hang off of Pivots (via 'tag')
Key: SOLR-6351
URL: https://issues.apache.org/jira/browse/SOLR-6351
Project: Solr
Issue Type: Sub-task
Reporter: Hoss Man
Attachments: SOLR-6351.patch, SOLR-6351.patch, SOLR-6351.patch,
SOLR-6351.patch, SOLR-6351.patch, SOLR-6351.patch, SOLR-6351.patch,
SOLR-6351.patch, SOLR-6351.patch, SOLR-6351.patch, SOLR-6351.patch,
SOLR-6351.patch, SOLR-6351.patch, SOLR-6351.patch
he goal here is basically flip the notion of stats.facet on it's head, so
that instead of asking the stats component to also do some faceting
(something that's never worked well with the variety of field types and has
never worked in distributed mode) we instead ask the PivotFacet code to
compute some stats X for each leaf in a pivot. We'll do this with the
existing {{stats.field}} params, but we'll leverage the {{tag}} local param
of the {{stats.field}} instances to be able to associate which stats we want
hanging off of which {{facet.pivot}}
Example...
{noformat}
facet.pivot={!stats=s1}category,manufacturer
[jira] [Updated] (SOLR-6351) Let Stats Hang off of Pivots (via 'tag')
[
https://issues.apache.org/jira/browse/SOLR-6351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vitaliy Zhovtyuk updated SOLR-6351:
---
Attachment: SOLR-6351.patch
Restored FacetPivotSmallTest, was lost between patches.
Added distributed test
org.apache.solr.handler.component.DistributedFacetPivotSmallAdvancedTestcovering
3additional cases
1. Getting pivot stats in string stats field
2. Getting top level stats on pivot stats
3. Pivot stats on each shard are not the same
Added getter to check stats values presence on
org.apache.solr.handler.component.PivotFacetValue#getStatsValues
Whitebox test assertions are not yet completed. Still working on it.
Let Stats Hang off of Pivots (via 'tag')
Key: SOLR-6351
URL: https://issues.apache.org/jira/browse/SOLR-6351
Project: Solr
Issue Type: Sub-task
Reporter: Hoss Man
Attachments: SOLR-6351.patch, SOLR-6351.patch, SOLR-6351.patch,
SOLR-6351.patch, SOLR-6351.patch, SOLR-6351.patch, SOLR-6351.patch,
SOLR-6351.patch, SOLR-6351.patch, SOLR-6351.patch, SOLR-6351.patch,
SOLR-6351.patch, SOLR-6351.patch
he goal here is basically flip the notion of stats.facet on it's head, so
that instead of asking the stats component to also do some faceting
(something that's never worked well with the variety of field types and has
never worked in distributed mode) we instead ask the PivotFacet code to
compute some stats X for each leaf in a pivot. We'll do this with the
existing {{stats.field}} params, but we'll leverage the {{tag}} local param
of the {{stats.field}} instances to be able to associate which stats we want
hanging off of which {{facet.pivot}}
Example...
{noformat}
facet.pivot={!stats=s1}category,manufacturer
stats.field={!key=avg_price tag=s1 mean=true}price
stats.field={!tag=s1 min=true max=true}user_rating
{noformat}
...with the request above, in addition to computing the min/max user_rating
and mean price (labeled avg_price) over the entire result set, the
PivotFacet component will also include those stats for every node of the tree
it builds up when generating a pivot of the fields category,manufacturer
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-6351) Let Stats Hang off of Pivots (via 'tag')
[
https://issues.apache.org/jira/browse/SOLR-6351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hoss Man updated SOLR-6351:
---
Attachment: SOLR-6351.patch
bq. ...Should we have a syntax to apply stats/queries/ranges only at specific
levels in the pivot hierarchy?...
I spun this off into it's own issue: SOLR-6663
For now, i think we should focus on requiring example one tag value in the
stats local param -- i incorporated that into some of the code/tests as i was
reviewing...
{panel:title=changes since last patch}
* PivotFacetProcessor
** brought back getSubsetSize - it gives us a nice optimization in the cases
where the actual subset isn't needed.
** getStatsFields
*** renamed getTaggedStatsFields so it's a little more clear it's a subset
relative this pivot
*** made static so it's use of localparams wasn't soo obtuse (i hate how much
stateful variables there are in SimpleFacets)
*** added error throwing if/when user specifies multiple tags seperated by
commas since we want to reserve for now and may use it for controlling where in
the pivot tree stats are computed (SOLR-6663)
*** javadocs
** doPivots
*** refactored the new stats logic to all be in one place, and not to do
anything (or allocate any extra objects) unless there are docs to compute stats
over, and stats to compute for those docs
* SolrExampleTests
** merged testPivotFacetsStatsParsed into testPivotFacetsStats
*** testPivotFacetsStats didn't need any distinct setup from what
testPivotFacetsStatsParsed, so i just moved the querys/assertions done by
testPivotFacetsStats into testPivotFacetsStatsParsed and simplified the name
*** removed the SOLR-6349 style local params from the stats.fields -- not
supported yet, and if/when it is this test asserts that more stats are there
then what those params were asking for, so we don't want it to suddenly break
in the future.
*** enhance the test a bit to sanity check these assertions still pass even
when extra levels of pivots are requested.
** merge testPivotFacetsStatsNotSupportedBoolean +
testPivotFacetsStatsNotSupportedString = testPivotFacetsStatsNotSupported
*** String was not accurate, TextField is more specific
*** also simplified the setup - no need for so many docs, and no need for it to
be diff between the two diff checks
*** added ignoreException -- the junit logs shouldn't misslead the user when an
exception is expected
*** added some additional assertions about the error messages
*** added another assertion if multiple tags seperated by commas (SOLR-6663)
* TestCloudPivots
** replaced nocommit comment with a comment refering to SOLR-6663
{panel}
Let Stats Hang off of Pivots (via 'tag')
Key: SOLR-6351
URL: https://issues.apache.org/jira/browse/SOLR-6351
Project: Solr
Issue Type: Sub-task
Reporter: Hoss Man
Attachments: SOLR-6351.patch, SOLR-6351.patch, SOLR-6351.patch,
SOLR-6351.patch, SOLR-6351.patch, SOLR-6351.patch, SOLR-6351.patch,
SOLR-6351.patch, SOLR-6351.patch, SOLR-6351.patch, SOLR-6351.patch,
SOLR-6351.patch
he goal here is basically flip the notion of stats.facet on it's head, so
that instead of asking the stats component to also do some faceting
(something that's never worked well with the variety of field types and has
never worked in distributed mode) we instead ask the PivotFacet code to
compute some stats X for each leaf in a pivot. We'll do this with the
existing {{stats.field}} params, but we'll leverage the {{tag}} local param
of the {{stats.field}} instances to be able to associate which stats we want
hanging off of which {{facet.pivot}}
Example...
{noformat}
facet.pivot={!stats=s1}category,manufacturer
stats.field={!key=avg_price tag=s1 mean=true}price
stats.field={!tag=s1 min=true max=true}user_rating
{noformat}
...with the request above, in addition to computing the min/max user_rating
and mean price (labeled avg_price) over the entire result set, the
PivotFacet component will also include those stats for every node of the tree
it builds up when generating a pivot of the fields category,manufacturer
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-6351) Let Stats Hang off of Pivots (via 'tag')
[
https://issues.apache.org/jira/browse/SOLR-6351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hoss Man updated SOLR-6351:
---
Attachment: SOLR-6351.patch
quick question for [~vzhovtiuk]: in an earlier comment, you mentioned creating
a new FacetPivotSmallTest.java test based on my earlier outline of how to
move forward -- but in later patches that test wasn't included. was there a
reason for removing it ? or was that just an oversight when generating the
later patches? Can we resurect that test from some of your earlier patches?
Been focusing on TestCloudPivotFacet. One particularly larg change to note...
After doing a bit of refactoring of some stuff i've mentioned in previous
comments, i realized that buildRandomPivotStatsFields buildStatsTagString
didn't really make a lot of sense -- notably due to some leftover cut/paste
comment cruft. digging in a bit, i realized that the way buildStatsTagString
was being called, we were only ever using the first stats tag w/ the first
facet.pivot -- and obliterating the second pivot from the params if there was
one.
So i ripped those methods out, and re-vamped the way the random stats.field
params were generated, and how the tags were associated with the pivots using
some slightly diff helper methods.
{panel:title=changes since last patch}
* TestCloudPivotFacet
** moved stats stats.field params from pivotP to baseP
*** this simplified a lot of request logic in assertPivotStats
** buildRandomPivotStatsFields buildStatsTagString
*** replaced with simpler logic via pickRandomStatsFields buildPivotParamValue
this uncovered an NPE in StatsInfo.getStatsFieldsByTag (see below)
*** fixed pickRandomStatsFields to include strings (not sure why they were
excluded - string stats work fine in StatsComponent)
** assertPivotStats
*** switched from one verification query per stat field, to a single
verification query that loops over each stats
*** shortened some variable names simplified assert msgs
*** added some hack-ish sanity checks on which stats were found for each pivot
** assertPivotData
*** wrapped up both assertNumFound assertPivotStats so that a single query is
executed and then each of those methods validates the data in the response that
they care about
** added assertDoubles() and sanityCheckAssertDoubles()
*** hammering on the test lead to a situation where stddev's were off due to
double rounding because of the order that the sumOfQuares were accumulated from
each shared (expected:2.3005390038169265E9 but was:2.300539003816927E9)
*** so i added a helper method to compare these types of stats with a small
epsilon relative to the size of the expected value, and a simple sanity checker
to test-the-test.
* QueryResponse
** refactored tightened up the pivot case statement a bit to assert on
unexpected keys or value types
* StatsComponent
** fixed NPE in StatsInfo.getStatsFieldsByTag - if someone asks for
{{facet.pivot=\{!stats=bogus\}foo}} (where 'bogus' is not a valid tag on a
stats.field) that was causing an NPE - should be ignored just like ex=bogus
{panel}
Let Stats Hang off of Pivots (via 'tag')
Key: SOLR-6351
URL: https://issues.apache.org/jira/browse/SOLR-6351
Project: Solr
Issue Type: Sub-task
Reporter: Hoss Man
Attachments: SOLR-6351.patch, SOLR-6351.patch, SOLR-6351.patch,
SOLR-6351.patch, SOLR-6351.patch, SOLR-6351.patch, SOLR-6351.patch,
SOLR-6351.patch, SOLR-6351.patch, SOLR-6351.patch, SOLR-6351.patch
he goal here is basically flip the notion of stats.facet on it's head, so
that instead of asking the stats component to also do some faceting
(something that's never worked well with the variety of field types and has
never worked in distributed mode) we instead ask the PivotFacet code to
compute some stats X for each leaf in a pivot. We'll do this with the
existing {{stats.field}} params, but we'll leverage the {{tag}} local param
of the {{stats.field}} instances to be able to associate which stats we want
hanging off of which {{facet.pivot}}
Example...
{noformat}
facet.pivot={!stats=s1}category,manufacturer
stats.field={!key=avg_price tag=s1 mean=true}price
stats.field={!tag=s1 min=true max=true}user_rating
{noformat}
...with the request above, in addition to computing the min/max user_rating
and mean price (labeled avg_price) over the entire result set, the
PivotFacet component will also include those stats for every node of the tree
it builds up when generating a pivot of the fields category,manufacturer
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-6351) Let Stats Hang off of Pivots (via 'tag')
[
https://issues.apache.org/jira/browse/SOLR-6351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hoss Man updated SOLR-6351:
---
Attachment: SOLR-6351.patch
small bits of progress on the patch review...
* PivotField
** fixed backcompt with constructor
** simplified write method fixed NPE risk + indenting
* DistributedFacetPivotLargeTest
** refactored query + assertFieldStats into doTestDeepPivotStats
*** only called once, and wasn't general in anyway - only usable for checking
one specific query
** made the epsilon used in assertEqual(double) a tiny float (0.1E-7) instead
of a large float, so we don't overlook any bugs
* DistributedFacetPivotSmallTest
** simplified ComparablePivotField
*** it isn't used with stat assertions, and didn't check stats when it was
used, so it doesn't need to know about stats in it's constructor
** refactored query + assertFieldStats into doTestDeepPivotStats
*** only called once, and wasn't general in anyway - only usable for checking
one specific query
*** simplified params (no need to muck with setDistributedParams
** made the epsilon used in assertEqual(double) a tiny float (0.1E-7) instead
of a large float, so we don't overlook any bugs
* DistributedFacetPivotLongTailTest
** did the match to fill in the expected values of the remaining nocommits
** made the epsilon used in assertEqual(double) a tiny float (0.1E-7) instead
of a large float, so we don't overlook any bugs
Let Stats Hang off of Pivots (via 'tag')
Key: SOLR-6351
URL: https://issues.apache.org/jira/browse/SOLR-6351
Project: Solr
Issue Type: Sub-task
Reporter: Hoss Man
Attachments: SOLR-6351.patch, SOLR-6351.patch, SOLR-6351.patch,
SOLR-6351.patch, SOLR-6351.patch, SOLR-6351.patch, SOLR-6351.patch,
SOLR-6351.patch, SOLR-6351.patch, SOLR-6351.patch
he goal here is basically flip the notion of stats.facet on it's head, so
that instead of asking the stats component to also do some faceting
(something that's never worked well with the variety of field types and has
never worked in distributed mode) we instead ask the PivotFacet code to
compute some stats X for each leaf in a pivot. We'll do this with the
existing {{stats.field}} params, but we'll leverage the {{tag}} local param
of the {{stats.field}} instances to be able to associate which stats we want
hanging off of which {{facet.pivot}}
Example...
{noformat}
facet.pivot={!stats=s1}category,manufacturer
stats.field={!key=avg_price tag=s1 mean=true}price
stats.field={!tag=s1 min=true max=true}user_rating
{noformat}
...with the request above, in addition to computing the min/max user_rating
and mean price (labeled avg_price) over the entire result set, the
PivotFacet component will also include those stats for every node of the tree
it builds up when generating a pivot of the fields category,manufacturer
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-6351) Let Stats Hang off of Pivots (via 'tag')
[
https://issues.apache.org/jira/browse/SOLR-6351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hoss Man updated SOLR-6351:
---
Attachment: SOLR-6351.patch
bq. It looks like Vitaliy's new code doesn't account for stats returned by a
shard in response to refinement requests.
I updated DistributedFacetPivotLongTailTest to check stats on a request where
refinement was required for correct results and was able to reliable reproduce.
Looking at the logs i realized that in the refinement requests stats=false
was explicitly set, and traced this to an optimization in StatsComponent -- it
was assuming that only the initial (PURPOSE_GET_TOP_IDS) shard request needed
stats computed, so i modified that to recognize (PURPOSE_REFINE_PIVOT_FACETS)
as another situation where we need to leave stats=true
This gets the tests to pass (still hammering on TestCloudPivot, but so far
looks good) but i'm not really liking this solution, for reasons i noted in a
StatsComponent nocommi comment...
{noformat}
// nocommit: PURPOSE_REFINE_PIVOT_FACETS by itself shouldn't be enough for
this...
//
// we need to check if the pivots actually have stats hanging off of them,
// if they don't then we still should supress the stats param
// (no need to compute the top level stats over and over)
//
// actually ... even if we do have stats hanging off of pivots,
// we need to make stats component smart enough not to waste time re-computing
// top level stats on every refinement request.
//
// so maybe StatsCOmponent should be left alone, and FacetComponent's prepare
method
// should be modified so that *if* isShard there are pivot refine params
those
// pivots have stats, then set some variable so that stats logic happens even
if stats=false?
{noformat}
I also made a few other various changes as i was reviewing the test (noted
below).
My plan is to move forward and continue reviewing more of the patch, starting
with the other tests, and then dig into the code changes -- writting additional
test cases if/when i notice things that looks like they may not be adequately
covered -- and then come back and revist the question of the stats=false
during refinement requests later (i'm certainly open to suggestions)
{panel:title=changes in this iteration of the patch}
* TestCloudPivots
** removed the bogus param cleanup vitaliy mentioned
** added some nocommits as reminders for the future
* DistributedFacetPivotLongTailTest
** refactored query + assertFieldStats into doTestDeepPivotStats
*** only called once, and wasn't general in anyway - only usable for checking
one specific query
** renamed foo_i to stat_i so it's a bit more obvious why that field is
there
** added stat_i to some long tail docs updated the existing stats assertions
** modified existing query that required refinement to show stats aren't correct
*** bbb0 on shard2 only gets included with refinement, but the min stat
trivially demonstrates that the -1 from shard2 isn't included
* StatsComponent
** check for PURPOSE_REFINE_PIVOT_FACETS in modifyRequest.
*NOTE:* This patch is significantly smaller then the last one because i
generated it using svn diff -x --ignore-all-space to supress a bunch of small
formatted changes from the previous patches.
{panel}
Let Stats Hang off of Pivots (via 'tag')
Key: SOLR-6351
URL: https://issues.apache.org/jira/browse/SOLR-6351
Project: Solr
Issue Type: Sub-task
Reporter: Hoss Man
Attachments: SOLR-6351.patch, SOLR-6351.patch, SOLR-6351.patch,
SOLR-6351.patch, SOLR-6351.patch, SOLR-6351.patch, SOLR-6351.patch,
SOLR-6351.patch, SOLR-6351.patch
he goal here is basically flip the notion of stats.facet on it's head, so
that instead of asking the stats component to also do some faceting
(something that's never worked well with the variety of field types and has
never worked in distributed mode) we instead ask the PivotFacet code to
compute some stats X for each leaf in a pivot. We'll do this with the
existing {{stats.field}} params, but we'll leverage the {{tag}} local param
of the {{stats.field}} instances to be able to associate which stats we want
hanging off of which {{facet.pivot}}
Example...
{noformat}
facet.pivot={!stats=s1}category,manufacturer
stats.field={!key=avg_price tag=s1 mean=true}price
stats.field={!tag=s1 min=true max=true}user_rating
{noformat}
...with the request above, in addition to computing the min/max user_rating
and mean price (labeled avg_price) over the entire result set, the
PivotFacet component will also include those stats for every node of the tree
it builds up when generating a pivot of the fields category,manufacturer
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail:
[jira] [Updated] (SOLR-6351) Let Stats Hang off of Pivots (via 'tag')
[
https://issues.apache.org/jira/browse/SOLR-6351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vitaliy Zhovtyuk updated SOLR-6351:
---
Attachment: SOLR-6351.patch
Fixed TestCloudPivotFacet. The reason for previous random test failures were
facet.limit, facet.offset, facet.overrequest.count, facet.overrequest.ratio
parameters generated randomly,
this was leading to inconsistent stats with pivot stats. Added cleanup for
those parameters before stats on pivots test. All tests are passing.
Let Stats Hang off of Pivots (via 'tag')
Key: SOLR-6351
URL: https://issues.apache.org/jira/browse/SOLR-6351
Project: Solr
Issue Type: Sub-task
Reporter: Hoss Man
Attachments: SOLR-6351.patch, SOLR-6351.patch, SOLR-6351.patch,
SOLR-6351.patch, SOLR-6351.patch, SOLR-6351.patch, SOLR-6351.patch,
SOLR-6351.patch
he goal here is basically flip the notion of stats.facet on it's head, so
that instead of asking the stats component to also do some faceting
(something that's never worked well with the variety of field types and has
never worked in distributed mode) we instead ask the PivotFacet code to
compute some stats X for each leaf in a pivot. We'll do this with the
existing {{stats.field}} params, but we'll leverage the {{tag}} local param
of the {{stats.field}} instances to be able to associate which stats we want
hanging off of which {{facet.pivot}}
Example...
{noformat}
facet.pivot={!stats=s1}category,manufacturer
stats.field={!key=avg_price tag=s1 mean=true}price
stats.field={!tag=s1 min=true max=true}user_rating
{noformat}
...with the request above, in addition to computing the min/max user_rating
and mean price (labeled avg_price) over the entire result set, the
PivotFacet component will also include those stats for every node of the tree
it builds up when generating a pivot of the fields category,manufacturer
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-6351) Let Stats Hang off of Pivots (via 'tag')
[
https://issues.apache.org/jira/browse/SOLR-6351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vitaliy Zhovtyuk updated SOLR-6351:
---
Attachment: SOLR-6351.patch
During work on
{code}
org.apache.solr.handler.component.DistributedFacetPivotLargeTest found:
junit.framework.AssertionFailedError:
.facet_counts.facet_pivot.place_s,company_t[0].stats!=pivot (unordered or
missing)
at
__randomizedtesting.SeedInfo.seed([705F7E1C2B9679AA:F1B9F0045CC91996]:0)
at
org.apache.solr.BaseDistributedSearchTestCase.compareSolrResponses(BaseDistributedSearchTestCase.java:842)
at
org.apache.solr.BaseDistributedSearchTestCase.compareResponses(BaseDistributedSearchTestCase.java:861)
at
org.apache.solr.BaseDistributedSearchTestCase.query(BaseDistributedSearchTestCase.java:562)
{code}
This mean difference in named list order between control shard and random shard.
Found the reason in
org.apache.solr.handler.component.PivotFacetValue#convertToNamedList
Due to this reason also updated DistributedFacetPivotSmallTest to use call
org.apache.solr.BaseDistributedSearchTestCase#query(org.apache.solr.common.params.SolrParams)
with response comparison.
Test org.apache.solr.handler.component.DistributedFacetPivotLongTailTest works
only on string fields, added int field to make stats on it.
org.apache.solr.cloud.TestCloudPivotFacet: added buildRandomPivotStatsFields to
build stats.filed list, this methods skip fields of string and boolean type
since they are not supported.
added tag string random generation on stat fields generated and if stats active
tags also added to pivot fields.
Added handling for not present controls stats (count=0) in TestCloudPivotFacet.
About skipping stats if count=0, it is not really good cause we losing missing
stats distribution.
Case with count=0, but missing=1 is real.
There are still some failures for TestCloudPivotFacet for date and double
(precision?). Will check it tomorrow.
Let Stats Hang off of Pivots (via 'tag')
Key: SOLR-6351
URL: https://issues.apache.org/jira/browse/SOLR-6351
Project: Solr
Issue Type: Sub-task
Reporter: Hoss Man
Attachments: SOLR-6351.patch, SOLR-6351.patch, SOLR-6351.patch,
SOLR-6351.patch, SOLR-6351.patch, SOLR-6351.patch, SOLR-6351.patch
he goal here is basically flip the notion of stats.facet on it's head, so
that instead of asking the stats component to also do some faceting
(something that's never worked well with the variety of field types and has
never worked in distributed mode) we instead ask the PivotFacet code to
compute some stats X for each leaf in a pivot. We'll do this with the
existing {{stats.field}} params, but we'll leverage the {{tag}} local param
of the {{stats.field}} instances to be able to associate which stats we want
hanging off of which {{facet.pivot}}
Example...
{noformat}
facet.pivot={!stats=s1}category,manufacturer
stats.field={!key=avg_price tag=s1 mean=true}price
stats.field={!tag=s1 min=true max=true}user_rating
{noformat}
...with the request above, in addition to computing the min/max user_rating
and mean price (labeled avg_price) over the entire result set, the
PivotFacet component will also include those stats for every node of the tree
it builds up when generating a pivot of the fields category,manufacturer
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-6351) Let Stats Hang off of Pivots (via 'tag')
[
https://issues.apache.org/jira/browse/SOLR-6351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vitaliy Zhovtyuk updated SOLR-6351:
---
Attachment: SOLR-6351.patch
1. Added stats fields pivot distribution to DistributedFacetPivotSmallTest
2. Fixed LinkedHashMap cannot be casted to NamedList exception, occurring on
stats distribution (changed
org.apache.solr.handler.component.PivotFacetHelper#convertStatsValuesToNamedList)
3. About Testing for not supported types, i think we need to cover/document
limitations as well.I'm not happy with asserting error message.
Added http code 400 assertion and error message substring assertion.
Let Stats Hang off of Pivots (via 'tag')
Key: SOLR-6351
URL: https://issues.apache.org/jira/browse/SOLR-6351
Project: Solr
Issue Type: Sub-task
Reporter: Hoss Man
Attachments: SOLR-6351.patch, SOLR-6351.patch, SOLR-6351.patch,
SOLR-6351.patch, SOLR-6351.patch, SOLR-6351.patch
he goal here is basically flip the notion of stats.facet on it's head, so
that instead of asking the stats component to also do some faceting
(something that's never worked well with the variety of field types and has
never worked in distributed mode) we instead ask the PivotFacet code to
compute some stats X for each leaf in a pivot. We'll do this with the
existing {{stats.field}} params, but we'll leverage the {{tag}} local param
of the {{stats.field}} instances to be able to associate which stats we want
hanging off of which {{facet.pivot}}
Example...
{noformat}
facet.pivot={!stats=s1}category,manufacturer
stats.field={!key=avg_price tag=s1 mean=true}price
stats.field={!tag=s1 min=true max=true}user_rating
{noformat}
...with the request above, in addition to computing the min/max user_rating
and mean price (labeled avg_price) over the entire result set, the
PivotFacet component will also include those stats for every node of the tree
it builds up when generating a pivot of the fields category,manufacturer
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-6351) Let Stats Hang off of Pivots (via 'tag')
[
https://issues.apache.org/jira/browse/SOLR-6351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Molloy updated SOLR-6351:
---
Attachment: SOLR-6351.patch
Addressing some comments. Remove unused for-loop and CommonParams.STATS. Didn't
touch the notSupprted test methods, will let Vitaliy a chance to speak for
their usefulness. Also reverted the hasValues logic to replace it with checking
if current pivot has positive count. Although it does produce some stats
entries with Infinity minimum/maximum and NaN mean. This is what I was asking
about before, I think I misunderstood the answer, but it still seems
error-prone to have such entries...
Finally, I updated some of the outputs to use NamedList instead of maps so that
solrj binary works better. Did have to sort fields in QueryResponse to get
tests to pass. Not sure this is the best way, but would sometimes get them out
of order if I didn't.
Let Stats Hang off of Pivots (via 'tag')
Key: SOLR-6351
URL: https://issues.apache.org/jira/browse/SOLR-6351
Project: Solr
Issue Type: Sub-task
Reporter: Hoss Man
Attachments: SOLR-6351.patch, SOLR-6351.patch, SOLR-6351.patch,
SOLR-6351.patch, SOLR-6351.patch
he goal here is basically flip the notion of stats.facet on it's head, so
that instead of asking the stats component to also do some faceting
(something that's never worked well with the variety of field types and has
never worked in distributed mode) we instead ask the PivotFacet code to
compute some stats X for each leaf in a pivot. We'll do this with the
existing {{stats.field}} params, but we'll leverage the {{tag}} local param
of the {{stats.field}} instances to be able to associate which stats we want
hanging off of which {{facet.pivot}}
Example...
{noformat}
facet.pivot={!stats=s1}category,manufacturer
stats.field={!key=avg_price tag=s1 mean=true}price
stats.field={!tag=s1 min=true max=true}user_rating
{noformat}
...with the request above, in addition to computing the min/max user_rating
and mean price (labeled avg_price) over the entire result set, the
PivotFacet component will also include those stats for every node of the tree
it builds up when generating a pivot of the fields category,manufacturer
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-6351) Let Stats Hang off of Pivots (via 'tag')
[
https://issues.apache.org/jira/browse/SOLR-6351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Molloy updated SOLR-6351:
---
Attachment: SOLR-6351.patch
Augmented previous 3 patches, added logic to not include stats entry if it's
empty, fixed distributed logic by actually merging stats from shards. Currently
have unit tests failing in solrj that I need to look at.
Let Stats Hang off of Pivots (via 'tag')
Key: SOLR-6351
URL: https://issues.apache.org/jira/browse/SOLR-6351
Project: Solr
Issue Type: Sub-task
Reporter: Hoss Man
Attachments: SOLR-6351.patch, SOLR-6351.patch, SOLR-6351.patch,
SOLR-6351.patch
he goal here is basically flip the notion of stats.facet on it's head, so
that instead of asking the stats component to also do some faceting
(something that's never worked well with the variety of field types and has
never worked in distributed mode) we instead ask the PivotFacet code to
compute some stats X for each leaf in a pivot. We'll do this with the
existing {{stats.field}} params, but we'll leverage the {{tag}} local param
of the {{stats.field}} instances to be able to associate which stats we want
hanging off of which {{facet.pivot}}
Example...
{noformat}
facet.pivot={!stats=s1}category,manufacturer
stats.field={!key=avg_price tag=s1 mean=true}price
stats.field={!tag=s1 min=true max=true}user_rating
{noformat}
...with the request above, in addition to computing the min/max user_rating
and mean price (labeled avg_price) over the entire result set, the
PivotFacet component will also include those stats for every node of the tree
it builds up when generating a pivot of the fields category,manufacturer
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-6351) Let Stats Hang off of Pivots (via 'tag')
[
https://issues.apache.org/jira/browse/SOLR-6351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Molloy updated SOLR-6351:
---
Attachment: SOLR-6351.patch
Adapted patch a bit to avoid NPE when no stats are asked, modified
PivotFacetValue to propagate info and other small tweaks. Tests seem to be
happy and got some requests to work, so I am too... :)
Let Stats Hang off of Pivots (via 'tag')
Key: SOLR-6351
URL: https://issues.apache.org/jira/browse/SOLR-6351
Project: Solr
Issue Type: Sub-task
Reporter: Hoss Man
Attachments: SOLR-6351.patch, SOLR-6351.patch
he goal here is basically flip the notion of stats.facet on it's head, so
that instead of asking the stats component to also do some faceting
(something that's never worked well with the variety of field types and has
never worked in distributed mode) we instead ask the PivotFacet code to
compute some stats X for each leaf in a pivot. We'll do this with the
existing {{stats.field}} params, but we'll leverage the {{tag}} local param
of the {{stats.field}} instances to be able to associate which stats we want
hanging off of which {{facet.pivot}}
Example...
{noformat}
facet.pivot={!stats=s1}category,manufacturer
stats.field={!key=avg_price tag=s1 mean=true}price
stats.field={!tag=s1 min=true max=true}user_rating
{noformat}
...with the request above, in addition to computing the min/max user_rating
and mean price (labeled avg_price) over the entire result set, the
PivotFacet component will also include those stats for every node of the tree
it builds up when generating a pivot of the fields category,manufacturer
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-6351) Let Stats Hang off of Pivots (via 'tag')
[
https://issues.apache.org/jira/browse/SOLR-6351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vitaliy Zhovtyuk updated SOLR-6351:
---
Attachment: SOLR-6351.patch
Combined with previous patch.
1. Added more solrj tests for stats on pivots
2. Fixed stats result
3. Minor tweaks
Let Stats Hang off of Pivots (via 'tag')
Key: SOLR-6351
URL: https://issues.apache.org/jira/browse/SOLR-6351
Project: Solr
Issue Type: Sub-task
Reporter: Hoss Man
Attachments: SOLR-6351.patch, SOLR-6351.patch, SOLR-6351.patch
he goal here is basically flip the notion of stats.facet on it's head, so
that instead of asking the stats component to also do some faceting
(something that's never worked well with the variety of field types and has
never worked in distributed mode) we instead ask the PivotFacet code to
compute some stats X for each leaf in a pivot. We'll do this with the
existing {{stats.field}} params, but we'll leverage the {{tag}} local param
of the {{stats.field}} instances to be able to associate which stats we want
hanging off of which {{facet.pivot}}
Example...
{noformat}
facet.pivot={!stats=s1}category,manufacturer
stats.field={!key=avg_price tag=s1 mean=true}price
stats.field={!tag=s1 min=true max=true}user_rating
{noformat}
...with the request above, in addition to computing the min/max user_rating
and mean price (labeled avg_price) over the entire result set, the
PivotFacet component will also include those stats for every node of the tree
it builds up when generating a pivot of the fields category,manufacturer
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-6351) Let Stats Hang off of Pivots (via 'tag')
[
https://issues.apache.org/jira/browse/SOLR-6351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vitaliy Zhovtyuk updated SOLR-6351:
---
Attachment: SOLR-6351.patch
Intermediate results:
1. Added pivot facet test to SolrExampleTests, extended
org.apache.solr.client.solrj.SolrQuery to provide multipls stats.field parameter
2. Added FacetPivotSmallTest and moved all asserts from
DistributedFacetPivotSmallTest to XPath assertions, separated it to few test
methods
3. tag local parameter parsing added
4. added org.apache.solr.handler.component.StatsInfo#tagToStatsFields and
org.apache.solr.handler.component.StatsInfo#getStatsFieldsByTag
to lookup list of stats fields by tag
5. Modified PivotFacetProcessor to collect and put StatValues for ever pivot
field, added test to assert stats value of pivots
6. Updated PivotField and org.apache.solr.client.solrj.response.QueryResponse
to read stats values on pivots
Let Stats Hang off of Pivots (via 'tag')
Key: SOLR-6351
URL: https://issues.apache.org/jira/browse/SOLR-6351
Project: Solr
Issue Type: Sub-task
Reporter: Hoss Man
Attachments: SOLR-6351.patch
he goal here is basically flip the notion of stats.facet on it's head, so
that instead of asking the stats component to also do some faceting
(something that's never worked well with the variety of field types and has
never worked in distributed mode) we instead ask the PivotFacet code to
compute some stats X for each leaf in a pivot. We'll do this with the
existing {{stats.field}} params, but we'll leverage the {{tag}} local param
of the {{stats.field}} instances to be able to associate which stats we want
hanging off of which {{facet.pivot}}
Example...
{noformat}
facet.pivot={!stats=s1}category,manufacturer
stats.field={!key=avg_price tag=s1 mean=true}price
stats.field={!tag=s1 min=true max=true}user_rating
{noformat}
...with the request above, in addition to computing the min/max user_rating
and mean price (labeled avg_price) over the entire result set, the
PivotFacet component will also include those stats for every node of the tree
it builds up when generating a pivot of the fields category,manufacturer
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
