date:20140929


[ 
https://issues.apache.org/jira/browse/LUCENE-5977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14151639#comment-14151639
 ] 

Adrien Grand commented on LUCENE-5977:
--

+1

 IW should safeguard against token streams returning invalid offsets for 
 multi-valued fields
 ---

 Key: LUCENE-5977
 URL: https://issues.apache.org/jira/browse/LUCENE-5977
 Project: Lucene - Core
  Issue Type: Bug
Affects Versions: 4.9, 4.9.1, 4.10, 4.10.1
Reporter: Dawid Weiss
Priority: Minor
 Fix For: 4.10.2

 Attachments: LUCENE-5977.patch


 We have a custom token stream that emits information about offsets of each 
 token. My (wrong) assumption was that for multi-valued fields a token 
 stream's offset information is magically shifted, much like this is the case 
 with positions. It's not the case -- offsets should be increasing and 
 monotonic across all instances of a field, even if it has custom token 
 streams. So, something like this:
 {code}
 doc.add(new Field(field-foo, new CannedTokenStream(token(bar, 1, 
 150, 160)), ftype));
 doc.add(new Field(field-foo, new CannedTokenStream(token(bar, 1,  
 50,  60)), ftype));
 {code}
 where the token function is defined as:
 {code}
 token(String image, int positionIncrement, int startOffset, int endOffset)
 {code}
 will result in either a cryptic assertion thrown from IW:
 {code}
 Exception in thread main java.lang.AssertionError
   at 
 org.apache.lucene.index.FreqProxTermsWriterPerField.writeOffsets(FreqProxTermsWriterPerField.java:99)
 {code}
 or nothing (or a codec error) if run without assertions.
 Obviously returning non-shifted offsets from subsequent token streams makes 
 little sense but I wonder if it could be made more explicit (or asserted) 
 that offsets need to be increasing between multiple-values. The minimum is to 
 add some documentation to OffsetAttribute. I don't know if offsets should be 
 shifted automatically, as it's the case with positions -- this would change 
 the semantics of existing tokenizers and filters which implement such 
 shifting internally already.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-5.x-Linux (32bit/jdk1.8.0_40-ea-b04) - Build # 11196 - Still Failing!

Build: http://jenkins.thetaphi.de/job/Lucene-Solr-5.x-Linux/11196/
Java: 32bit/jdk1.8.0_40-ea-b04 -server -XX:+UseParallelGC

All tests passed

Build Log:
[...truncated 4491 lines...]
[javac] Compiling 2 source files to 
/mnt/ssd/jenkins/workspace/Lucene-Solr-5.x-Linux/lucene/build/memory/classes/test
[javac] 
/mnt/ssd/jenkins/workspace/Lucene-Solr-5.x-Linux/lucene/memory/src/test/org/apache/lucene/index/memory/TestMemoryIndexAgainstRAMDir.java:434:
 error: incompatible types: IndexableField cannot be converted to Field
[javac]   for (Field field : nextDoc.getFields()) {
[javac]   ^
[javac] 
/mnt/ssd/jenkins/workspace/Lucene-Solr-5.x-Linux/lucene/memory/src/test/org/apache/lucene/index/memory/TestMemoryIndexAgainstRAMDir.java:445:
 error: cannot find symbol
[javac]   for (IndexableField field : doc.indexableFields()) {
[javac]  ^
[javac]   symbol:   method indexableFields()
[javac]   location: variable doc of type Document
[javac] Note: 
/mnt/ssd/jenkins/workspace/Lucene-Solr-5.x-Linux/lucene/memory/src/test/org/apache/lucene/index/memory/TestMemoryIndexAgainstRAMDir.java
 uses or overrides a deprecated API.
[javac] Note: Recompile with -Xlint:deprecation for details.
[javac] 2 errors

BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-5.x-Linux/build.xml:524: The following 
error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-5.x-Linux/build.xml:472: The following 
error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-5.x-Linux/build.xml:61: The following 
error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-5.x-Linux/extra-targets.xml:39: The 
following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-5.x-Linux/lucene/build.xml:447: The 
following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-5.x-Linux/lucene/common-build.xml:2142: 
The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-5.x-Linux/lucene/module-build.xml:55: 
The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-5.x-Linux/lucene/common-build.xml:785: 
The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-5.x-Linux/lucene/common-build.xml:799: 
The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-5.x-Linux/lucene/common-build.xml:1867: 
Compile failed; see the compiler error output for details.

Total time: 29 minutes 42 seconds
Build step 'Invoke Ant' marked build as failure
[description-setter] Description set: Java: 32bit/jdk1.8.0_40-ea-b04 -server 
-XX:+UseParallelGC
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-6261) Run ZK watch event callbacks in parallel to the event thread


[ 
https://issues.apache.org/jira/browse/SOLR-6261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14151666#comment-14151666
 ] 

Ramkumar Aiyengar commented on SOLR-6261:
-

 Ah, confused it with the second patch on the other issue when I glanced at 
 it. Yeah, I can commit it here.

Hey [~markrmil...@gmail.com], just checking since I don't see an automated 
message here, was this committed? Thanks!

 Run ZK watch event callbacks in parallel to the event thread
 

 Key: SOLR-6261
 URL: https://issues.apache.org/jira/browse/SOLR-6261
 Project: Solr
  Issue Type: Improvement
  Components: SolrCloud
Affects Versions: 4.9
Reporter: Ramkumar Aiyengar
Assignee: Mark Miller
Priority: Minor
 Fix For: 4.10, Trunk

 Attachments: thread-compare.jpg


 Currently checking for leadership (due to the leader's ephemeral node going 
 away) happens in ZK's event thread. If there are many cores and all of them 
 are due leadership, then they would have to serially go through the two-way 
 sync and leadership takeover.
 For tens of cores, this could mean 30-40s without leadership before the last 
 in the list even gets to start the leadership process. If the leadership 
 process happens in a separate thread, then the cores could all take over in 
 parallel.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Comment Edited] (SOLR-6261) Run ZK watch event callbacks in parallel to the event thread

[
https://issues.apache.org/jira/browse/SOLR-6261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14151666#comment-14151666
]

Ramkumar Aiyengar edited comment on SOLR-6261 at 9/29/14 1:13 PM:
--

bq. Ah, confused it with the second patch on the other issue when I glanced at
it. Yeah, I can commit it here.

Hey [~markrmil...@gmail.com], just checking since I don't see an automated
message here, was this committed? Thanks!

was (Author: andyetitmoves):
Ah, confused it with the second patch on the other issue when I glanced at
it. Yeah, I can commit it here.

Hey [~markrmil...@gmail.com], just checking since I don't see an automated
message here, was this committed? Thanks!

Run ZK watch event callbacks in parallel to the event thread

Key: SOLR-6261
URL: https://issues.apache.org/jira/browse/SOLR-6261
Project: Solr
Issue Type: Improvement
Components: SolrCloud
Affects Versions: 4.9
Reporter: Ramkumar Aiyengar
Assignee: Mark Miller
Priority: Minor
Fix For: 4.10, Trunk

Attachments: thread-compare.jpg

Currently checking for leadership (due to the leader's ephemeral node going
away) happens in ZK's event thread. If there are many cores and all of them
are due leadership, then they would have to serially go through the two-way
sync and leadership takeover.
For tens of cores, this could mean 30-40s without leadership before the last
in the list even gets to start the leadership process. If the leadership
process happens in a separate thread, then the cores could all take over in
parallel.

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-6261) Run ZK watch event callbacks in parallel to the event thread


[ 
https://issues.apache.org/jira/browse/SOLR-6261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14151675#comment-14151675
 ] 

Mark Miller commented on SOLR-6261:
---

Hmm, my fault. Should have reopened the issue then. We will have to make a new 
JIRA it sounds.

 Run ZK watch event callbacks in parallel to the event thread
 

 Key: SOLR-6261
 URL: https://issues.apache.org/jira/browse/SOLR-6261
 Project: Solr
  Issue Type: Improvement
  Components: SolrCloud
Affects Versions: 4.9
Reporter: Ramkumar Aiyengar
Assignee: Mark Miller
Priority: Minor
 Fix For: 4.10, Trunk

 Attachments: thread-compare.jpg


 Currently checking for leadership (due to the leader's ephemeral node going 
 away) happens in ZK's event thread. If there are many cores and all of them 
 are due leadership, then they would have to serially go through the two-way 
 sync and leadership takeover.
 For tens of cores, this could mean 30-40s without leadership before the last 
 in the list even gets to start the leadership process. If the leadership 
 process happens in a separate thread, then the cores could all take over in 
 parallel.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (LUCENE-5938) New DocIdSet implementation with random write access

2014-09-29 Thread ASF subversion and git services (JIRA)


 [ 
https://issues.apache.org/jira/browse/LUCENE-5938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Adrien Grand updated LUCENE-5938:
-
Attachment: LUCENE-5938.patch

I updated the patch to recent trunk modifications and ran the benchmark again, 
I think it is ready. In summary this patch:
 - introduces a new doc-id set impl similar to FixedBitSet but which is much 
faster in the sparse case and a bit slower in the dense case (between 1.5x and 
4x slower according to benchmarks).
 - introduces a doc-id set builder that supports random write access by 
starting with a sparse bit set and upgrading to a dense FixedBitSet when the 
cardinality of the set increases
 - changes MultiTermQueryWrapperFilter and TermsFilter to use this new builder
 - removes CONSTANT_SCORE_AUTO_REWRITE and makes CONSTANT_SCORE_FILTER_REWRITE 
the default

For queries that match many documents ({{wikimedium10m.tasks}}, the builder 
always ends up building a FixedBitSet), this new patch can be slower or faster 
depending on the cost to iterate the matching terms (since they are enumerated 
only once now) vs. the cost of building the doc-id set.

For queries that match few documents ({{low_freq.tasks}}, attached to this 
issue), this new patch makes things faster since it just sets a couple of bits 
in random order and then iterates over them instead of merging documents coming 
from several other iterators on the fly using a priority queue.

Independently of the benchmarks, I think it's a good simplification to remove 
the constant-score auto rewrite mode.

{noformat}
wikimedium10m.tasks

TaskQPS baseline  StdDev   QPS patch  StdDev
Pct diff
  IntNRQ8.79  (9.6%)8.41  (3.4%)   
-4.3% ( -15% -9%)
  Fuzzy2   60.83 (11.1%)   58.34  (8.7%)   
-4.1% ( -21% -   17%)
OrNotHighMed   98.35 (13.8%)   97.12 (10.9%)   
-1.3% ( -22% -   27%)
   OrHighNotHigh   18.88 (13.7%)   18.71 (11.1%)   
-0.9% ( -22% -   27%)
   OrNotHighHigh   17.10 (13.4%)   17.03 (11.2%)   
-0.4% ( -22% -   27%)
OrNotHighLow  126.52 (13.6%)  126.85 (10.9%)
0.3% ( -21% -   28%)
   OrHighMed   76.90 (14.0%)   77.25 (11.4%)
0.5% ( -21% -   30%)
OrHighNotLow   41.29 (14.3%)   41.51 (12.4%)
0.5% ( -22% -   31%)
OrHighNotMed   57.70 (13.6%)   58.03 (11.6%)
0.6% ( -21% -   29%)
   OrHighLow   73.14 (14.7%)   73.74 (12.0%)
0.8% ( -22% -   32%)
 LowSloppyPhrase  127.22  (8.6%)  128.43  (3.8%)
1.0% ( -10% -   14%)
  OrHighHigh   29.11 (15.1%)   29.41 (12.2%)
1.0% ( -22% -   33%)
HighSloppyPhrase   12.83 (10.4%)   13.02  (5.3%)
1.4% ( -12% -   19%)
 Prefix3  113.92  (9.9%)  115.71  (2.4%)
1.6% (  -9% -   15%)
PKLookup  297.13  (9.2%)  302.03  (3.5%)
1.6% ( -10% -   15%)
 MedSloppyPhrase   38.60  (8.8%)   39.26  (3.7%)
1.7% (  -9% -   15%)
 AndHighHigh   71.39  (6.9%)   72.67  (0.9%)
1.8% (  -5% -   10%)
HighTerm   87.17  (7.9%)   88.85  (2.1%)
1.9% (  -7% -   12%)
   MedPhrase   74.60  (9.3%)   76.10  (4.3%)
2.0% ( -10% -   17%)
   LowPhrase   21.67  (9.6%)   22.12  (4.0%)
2.1% ( -10% -   17%)
  AndHighMed  297.13  (9.4%)  303.73  (2.1%)
2.2% (  -8% -   15%)
  HighPhrase   16.65  (8.2%)   17.04  (3.7%)
2.3% (  -8% -   15%)
HighSpanNear4.56 (10.7%)4.67  (6.1%)
2.4% ( -12% -   21%)
 LowTerm  769.53  (7.8%)  788.24  (2.0%)
2.4% (  -6% -   13%)
  AndHighLow  726.76 (10.6%)  744.51  (4.2%)
2.4% ( -11% -   19%)
 MedSpanNear   65.27  (9.3%)   67.00  (3.2%)
2.6% (  -9% -   16%)
Wildcard  114.28  (9.1%)  118.05  (7.4%)
3.3% ( -12% -   21%)
 LowSpanNear  174.75 (10.3%)  180.83  (3.5%)
3.5% (  -9% -   19%)
  Fuzzy1   67.63 (11.3%)   70.08  (3.2%)
3.6% (  -9% -   20%)
 MedTerm  209.00  (9.3%)  216.71  (1.9%)
3.7% (  -6% -   16%)
 Respell   48.30 (10.6%)   50.58  (1.7%)
4.7% (  -6% -   18%)

low_freq.tasks

TaskQPS baseline  StdDev   QPS patch  StdDev
Pct diff
PKLookup  278.50  (8.8%)  297.48 (13.9%)
6.8% ( -14% -

[jira] [Commented] (SOLR-6249) Schema API changes return success before all cores are updated


[ 
https://issues.apache.org/jira/browse/SOLR-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14151690#comment-14151690
 ] 

ASF subversion and git services commented on SOLR-6249:
---

Commit 1628181 from [~thelabdude] in branch 'dev/trunk'
[ https://svn.apache.org/r1628181 ]

SOLR-6249: Schema API changes return success before all cores are updated

 Schema API changes return success before all cores are updated
 --

 Key: SOLR-6249
 URL: https://issues.apache.org/jira/browse/SOLR-6249
 Project: Solr
  Issue Type: Improvement
  Components: Schema and Analysis, SolrCloud
Reporter: Gregory Chanan
Assignee: Timothy Potter
 Attachments: SOLR-6249.patch, SOLR-6249.patch, SOLR-6249.patch


 See SOLR-6137 for more details.
 The basic issue is that Schema API changes return success when the first core 
 is updated, but other cores asynchronously read the updated schema from 
 ZooKeeper.
 So a client application could make a Schema API change and then index some 
 documents based on the new schema that may fail on other nodes.
 Possible fixes:
 1) Make the Schema API calls synchronous
 2) Give the client some ability to track the state of the schema.  They can 
 already do this to a certain extent by checking the Schema API on all the 
 replicas and verifying that the field has been added, though this is pretty 
 cumbersome.  Maybe it makes more sense to do this sort of thing on the 
 collection level, i.e. Schema API changes return the zk version to the 
 client.  We add an API to return the current zk version.  On a replica, if 
 the zk version is = the version the client has, the client knows that 
 replica has at least seen the schema change.  We could also provide an API to 
 do the distribution and checking across the different replicas of the 
 collection so that clients don't need ot do that themselves.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-6249) Schema API changes return success before all cores are updated

2014-09-29 Thread ASF subversion and git services (JIRA)


[ 
https://issues.apache.org/jira/browse/SOLR-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14151702#comment-14151702
 ] 

Noble Paul commented on SOLR-6249:
--

Does it take care of zkclient losing connections? I

 Schema API changes return success before all cores are updated
 --

 Key: SOLR-6249
 URL: https://issues.apache.org/jira/browse/SOLR-6249
 Project: Solr
  Issue Type: Improvement
  Components: Schema and Analysis, SolrCloud
Reporter: Gregory Chanan
Assignee: Timothy Potter
 Attachments: SOLR-6249.patch, SOLR-6249.patch, SOLR-6249.patch


 See SOLR-6137 for more details.
 The basic issue is that Schema API changes return success when the first core 
 is updated, but other cores asynchronously read the updated schema from 
 ZooKeeper.
 So a client application could make a Schema API change and then index some 
 documents based on the new schema that may fail on other nodes.
 Possible fixes:
 1) Make the Schema API calls synchronous
 2) Give the client some ability to track the state of the schema.  They can 
 already do this to a certain extent by checking the Schema API on all the 
 replicas and verifying that the field has been added, though this is pretty 
 cumbersome.  Maybe it makes more sense to do this sort of thing on the 
 collection level, i.e. Schema API changes return the zk version to the 
 client.  We add an API to return the current zk version.  On a replica, if 
 the zk version is = the version the client has, the client knows that 
 replica has at least seen the schema change.  We could also provide an API to 
 do the distribution and checking across the different replicas of the 
 collection so that clients don't need ot do that themselves.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-5977) IW should safeguard against token streams returning invalid offsets for multi-valued fields


[ 
https://issues.apache.org/jira/browse/LUCENE-5977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14151730#comment-14151730
 ] 

ASF subversion and git services commented on LUCENE-5977:
-

Commit 1628192 from [~rcmuir] in branch 'dev/trunk'
[ https://svn.apache.org/r1628192 ]

LUCENE-5977: Fix tokenstream safety checks in IndexWriter to work across 
multi-valued fields

 IW should safeguard against token streams returning invalid offsets for 
 multi-valued fields
 ---

 Key: LUCENE-5977
 URL: https://issues.apache.org/jira/browse/LUCENE-5977
 Project: Lucene - Core
  Issue Type: Bug
Affects Versions: 4.9, 4.9.1, 4.10, 4.10.1
Reporter: Dawid Weiss
Priority: Minor
 Fix For: 4.10.2

 Attachments: LUCENE-5977.patch


 We have a custom token stream that emits information about offsets of each 
 token. My (wrong) assumption was that for multi-valued fields a token 
 stream's offset information is magically shifted, much like this is the case 
 with positions. It's not the case -- offsets should be increasing and 
 monotonic across all instances of a field, even if it has custom token 
 streams. So, something like this:
 {code}
 doc.add(new Field(field-foo, new CannedTokenStream(token(bar, 1, 
 150, 160)), ftype));
 doc.add(new Field(field-foo, new CannedTokenStream(token(bar, 1,  
 50,  60)), ftype));
 {code}
 where the token function is defined as:
 {code}
 token(String image, int positionIncrement, int startOffset, int endOffset)
 {code}
 will result in either a cryptic assertion thrown from IW:
 {code}
 Exception in thread main java.lang.AssertionError
   at 
 org.apache.lucene.index.FreqProxTermsWriterPerField.writeOffsets(FreqProxTermsWriterPerField.java:99)
 {code}
 or nothing (or a codec error) if run without assertions.
 Obviously returning non-shifted offsets from subsequent token streams makes 
 little sense but I wonder if it could be made more explicit (or asserted) 
 that offsets need to be increasing between multiple-values. The minimum is to 
 add some documentation to OffsetAttribute. I don't know if offsets should be 
 shifted automatically, as it's the case with positions -- this would change 
 the semantics of existing tokenizers and filters which implement such 
 shifting internally already.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-2907) java.lang.IllegalArgumentException: deltaQuery has no column to resolve to declared primary key pk='ITEM_ID, CATEGORY_ID'

2014-09-29 Thread Aleksandr Ivanov (JIRA)


[ 
https://issues.apache.org/jira/browse/SOLR-2907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14151734#comment-14151734
 ] 

Aleksandr Ivanov commented on SOLR-2907:


Hi to everyone. I've had this problem. But I've solved it by adding  name of 
field (ITEM_ID and CATEGORY_ID) in schema.xml file on server to (just after 
/fields): 
uniqueKeyITEM_ID/uniqueKey
uniqueKeyCATEGORY_ID/uniqueKey
But I think the main problem is using wrong value for pk= in data-config.xml 
file.
You should use TemplateTransformer, in root entity. Something like this:
entity name=account datasource=MyDS pk=id 
transformer=TemplateTransformer rootEntity=true ...
...
field column=ITEM_ID name=item_id /
field column=CATEGORY_ID name=cid
field column=id name=id template=${account.item_id}_${account.cid} // 
and it should be 123_232e23
...
/entity



 java.lang.IllegalArgumentException: deltaQuery has no column to resolve to 
 declared primary key pk='ITEM_ID, CATEGORY_ID'
 -

 Key: SOLR-2907
 URL: https://issues.apache.org/jira/browse/SOLR-2907
 Project: Solr
  Issue Type: Bug
  Components: contrib - DataImportHandler, Schema and Analysis
Affects Versions: 3.4
Reporter: Alan Baker

 We are using solr for our site and ran into this error in our own schema and 
 I was able to reproduce it using the dataimport example code in the solr 
 project.  We do not get this error in SOLR 1.4 only started seeing it as we 
 are working to upgrade to 3.4.0.  It fails when delta-importing linked tables.
 Complete trace:
 Nov 18, 2011 5:21:02 PM org.apache.solr.handler.dataimport.DataImporter 
 doDeltaImport
 SEVERE: Delta Import Failed
 java.lang.IllegalArgumentException: deltaQuery has no column to resolve to 
 declared primary key pk='ITEM_ID, CATEGORY_ID'
   at 
 org.apache.solr.handler.dataimport.DocBuilder.findMatchingPkColumn(DocBuilder.java:849)
   at 
 org.apache.solr.handler.dataimport.DocBuilder.collectDelta(DocBuilder.java:900)
   at 
 org.apache.solr.handler.dataimport.DocBuilder.collectDelta(DocBuilder.java:879)
   at 
 org.apache.solr.handler.dataimport.DocBuilder.doDelta(DocBuilder.java:285)
   at 
 org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:179)
   at 
 org.apache.solr.handler.dataimport.DataImporter.doDeltaImport(DataImporter.java:390)
   at 
 org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:429)
   at 
 org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:408)
 I used this dataConfig from the wiki on the data import:
 dataConfig
 dataSource driver=org.hsqldb.jdbcDriver 
 url=jdbc:hsqldb:./example-DIH/hsqldb/ex user=sa /
 document
entity  name=item pk=ID 
   query=select * from item 
 deltaImportQuery=select * from item where 
 ID=='${dataimporter.delta.id}'
 deltaQuery=select id from item where last_modified gt; 
 '${dataimporter.last_index_time}'
 entity name=item_category pk=ITEM_ID, CATEGORY_ID
 query=select CATEGORY_ID from item_category where 
 ITEM_ID='${item.ID}'
 deltaQuery=select ITEM_ID, CATEGORY_ID from 
 item_category where last_modified  '${dataimporter.last_index_time}'
 parentDeltaQuery=select ID from item where 
 ID=${item_category.ITEM_ID}
 
   entity name=category pk=ID
 query=select DESCRIPTION as cat from category where 
 ID = '${item_category.CATEGORY_ID}'
 deltaQuery=select ID from category where 
 last_modified gt; '${dataimporter.last_index_time}'
 parentDeltaQuery=select ITEM_ID, CATEGORY_ID from 
 item_category where CATEGORY_ID=${category.ID}/
   /entity
 /entity
 
 /document
 /dataConfig
 To reproduce use the data config from above and set the dataimport.properties 
 last update times to before the last_modifed date in the example data.  I my 
 case I had to set the year to 1969.  Then run a delta-import and the 
 exception occurs.  Thanks.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-trunk-Linux (64bit/jdk1.7.0_67) - Build # 11349 - Still Failing!

2014-09-29 Thread ASF subversion and git services (JIRA)

Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/11349/
Java: 64bit/jdk1.7.0_67 -XX:+UseCompressedOops -XX:+UseSerialGC

1 tests failed.
REGRESSION:  
org.apache.solr.cloud.CloudExitableDirectoryReaderTest.testDistribSearch

Error Message:
no exception matching expected: 400: Request took too long during query 
expansion. Terminating request.

Stack Trace:
java.lang.AssertionError: no exception matching expected: 400: Request took too 
long during query expansion. Terminating request.
at 
__randomizedtesting.SeedInfo.seed([19D3E66939E34CC4:983568714EBC2CF8]:0)
at org.junit.Assert.fail(Assert.java:93)
at 
org.apache.solr.cloud.CloudExitableDirectoryReaderTest.assertFail(CloudExitableDirectoryReaderTest.java:101)
at 
org.apache.solr.cloud.CloudExitableDirectoryReaderTest.doTimeoutTests(CloudExitableDirectoryReaderTest.java:75)
at 
org.apache.solr.cloud.CloudExitableDirectoryReaderTest.doTest(CloudExitableDirectoryReaderTest.java:54)
at 
org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:869)
at sun.reflect.GeneratedMethodAccessor51.invoke(Unknown Source)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)

[jira] [Comment Edited] (SOLR-2907) java.lang.IllegalArgumentException: deltaQuery has no column to resolve to declared primary key pk='ITEM_ID, CATEGORY_ID'

2014-09-29 Thread Aleksandr Ivanov (JIRA)


[ 
https://issues.apache.org/jira/browse/SOLR-2907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14151734#comment-14151734
 ] 

Aleksandr Ivanov edited comment on SOLR-2907 at 9/29/14 2:36 PM:
-

Hi to everyone. I've had this problem. But I've solved it by adding  name of 
field (ITEM_ID and CATEGORY_ID) in schema.xml file on server to (just after 
/fields): 
uniqueKeyITEM_ID/uniqueKey
uniqueKeyCATEGORY_ID/uniqueKey
But I think the main problem is using wrong value for pk= in data-config.xml 
file.
You should use TemplateTransformer, in root entity. Something like this:
entity name=account datasource=MyDS pk=id 
transformer=TemplateTransformer rootEntity=true ...
...
field column=ITEM_ID name=item_id /
field column=CATEGORY_ID name=cid
field column=id name=id template=${account.item_id}_${account.cid} // 
and it should be 123_232e23
...
/entity




was (Author: aleksandr.ivanov):
Hi to everyone. I've had this problem. But I've solved it by adding  name of 
field (ITEM_ID and CATEGORY_ID) in schema.xml file on server to (just after 
/fields): 
uniqueKeyITEM_ID/uniqueKey
uniqueKeyCATEGORY_ID/uniqueKey
But I think the main problem is using wrong value for pk= in data-config.xml 
file.
You should use TemplateTransformer, in root entity. Something like this:
entity name=account datasource=MyDS pk=id 
transformer=TemplateTransformer rootEntity=true ...
...
field column=ITEM_ID name=item_id /
field column=CATEGORY_ID name=cid
field column=id name=id template=${account.item_id}_${account.cid} // 
and it should be 123_232e23
...
/entity



 java.lang.IllegalArgumentException: deltaQuery has no column to resolve to 
 declared primary key pk='ITEM_ID, CATEGORY_ID'
 -

 Key: SOLR-2907
 URL: https://issues.apache.org/jira/browse/SOLR-2907
 Project: Solr
  Issue Type: Bug
  Components: contrib - DataImportHandler, Schema and Analysis
Affects Versions: 3.4
Reporter: Alan Baker

 We are using solr for our site and ran into this error in our own schema and 
 I was able to reproduce it using the dataimport example code in the solr 
 project.  We do not get this error in SOLR 1.4 only started seeing it as we 
 are working to upgrade to 3.4.0.  It fails when delta-importing linked tables.
 Complete trace:
 Nov 18, 2011 5:21:02 PM org.apache.solr.handler.dataimport.DataImporter 
 doDeltaImport
 SEVERE: Delta Import Failed
 java.lang.IllegalArgumentException: deltaQuery has no column to resolve to 
 declared primary key pk='ITEM_ID, CATEGORY_ID'
   at 
 org.apache.solr.handler.dataimport.DocBuilder.findMatchingPkColumn(DocBuilder.java:849)
   at 
 org.apache.solr.handler.dataimport.DocBuilder.collectDelta(DocBuilder.java:900)
   at 
 org.apache.solr.handler.dataimport.DocBuilder.collectDelta(DocBuilder.java:879)
   at 
 org.apache.solr.handler.dataimport.DocBuilder.doDelta(DocBuilder.java:285)
   at 
 org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:179)
   at 
 org.apache.solr.handler.dataimport.DataImporter.doDeltaImport(DataImporter.java:390)
   at 
 org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:429)
   at 
 org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:408)
 I used this dataConfig from the wiki on the data import:
 dataConfig
 dataSource driver=org.hsqldb.jdbcDriver 
 url=jdbc:hsqldb:./example-DIH/hsqldb/ex user=sa /
 document
entity  name=item pk=ID 
   query=select * from item 
 deltaImportQuery=select * from item where 
 ID=='${dataimporter.delta.id}'
 deltaQuery=select id from item where last_modified gt; 
 '${dataimporter.last_index_time}'
 entity name=item_category pk=ITEM_ID, CATEGORY_ID
 query=select CATEGORY_ID from item_category where 
 ITEM_ID='${item.ID}'
 deltaQuery=select ITEM_ID, CATEGORY_ID from 
 item_category where last_modified  '${dataimporter.last_index_time}'
 parentDeltaQuery=select ID from item where 
 ID=${item_category.ITEM_ID}
 
   entity name=category pk=ID
 query=select DESCRIPTION as cat from category where 
 ID = '${item_category.CATEGORY_ID}'
 deltaQuery=select ID from category where 
 last_modified gt; '${dataimporter.last_index_time}'
 parentDeltaQuery=select ITEM_ID, CATEGORY_ID from 
 item_category where CATEGORY_ID=${category.ID}/
   /entity
 /entity
 
 /document
 /dataConfig
 To reproduce use the data config from above and set the dataimport.properties 
 last update times to before the last_modifed date in the example

[jira] [Commented] (LUCENE-5911) Make MemoryIndex thread-safe for queries


[ 
https://issues.apache.org/jira/browse/LUCENE-5911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14151738#comment-14151738
 ] 

ASF subversion and git services commented on LUCENE-5911:
-

Commit 1628194 from [~rcmuir] in branch 'dev/branches/branch_5x'
[ https://svn.apache.org/r1628194 ]

LUCENE-5911: fix compile

 Make MemoryIndex thread-safe for queries
 

 Key: LUCENE-5911
 URL: https://issues.apache.org/jira/browse/LUCENE-5911
 Project: Lucene - Core
  Issue Type: Improvement
Reporter: Alan Woodward
Assignee: Alan Woodward
Priority: Minor
 Fix For: 5.0, Trunk

 Attachments: LUCENE-5911.patch, LUCENE-5911.patch


 We want to be able to run multiple queries at once over a MemoryIndex in 
 luwak (see 
 https://github.com/flaxsearch/luwak/commit/49a8fba5764020c2f0e4dc29d80d93abb0231191),
  but this isn't possible with the current implementation.  However, looking 
 at the code, it seems that it would be relatively simple to make MemoryIndex 
 thread-safe for reads/queries.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Comment Edited] (SOLR-2907) java.lang.IllegalArgumentException: deltaQuery has no column to resolve to declared primary key pk='ITEM_ID, CATEGORY_ID'

2014-09-29 Thread Aleksandr Ivanov (JIRA)


[ 
https://issues.apache.org/jira/browse/SOLR-2907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14151734#comment-14151734
 ] 

Aleksandr Ivanov edited comment on SOLR-2907 at 9/29/14 2:37 PM:
-

Hi to everyone. I've had this problem. But I've solved it by adding  name of 
field (ITEM_ID and CATEGORY_ID) in schema.xml file on server to (just after 
/fields): 
uniqueKeyITEM_ID/uniqueKey
uniqueKeyCATEGORY_ID/uniqueKey
But I think the main problem is using wrong value for pk= in data-config.xml 
file.
You should use TemplateTransformer, in root entity. Something like this:
entity name=account datasource=MyDS pk=id 
transformer=TemplateTransformer rootEntity=true ...
...
field column=ITEM_ID name=item_id /
field column=CATEGORY_ID name=cid
field column=id name=id template=${account.item_id}_${account.cid}  // 
and it should be 123_232e23
...
/entity




was (Author: aleksandr.ivanov):
Hi to everyone. I've had this problem. But I've solved it by adding  name of 
field (ITEM_ID and CATEGORY_ID) in schema.xml file on server to (just after 
/fields): 
uniqueKeyITEM_ID/uniqueKey
uniqueKeyCATEGORY_ID/uniqueKey
But I think the main problem is using wrong value for pk= in data-config.xml 
file.
You should use TemplateTransformer, in root entity. Something like this:
entity name=account datasource=MyDS pk=id 
transformer=TemplateTransformer rootEntity=true ...
...
field column=ITEM_ID name=item_id /
field column=CATEGORY_ID name=cid
field column=id name=id template=${account.item_id}_${account.cid} // 
and it should be 123_232e23
...
/entity



 java.lang.IllegalArgumentException: deltaQuery has no column to resolve to 
 declared primary key pk='ITEM_ID, CATEGORY_ID'
 -

 Key: SOLR-2907
 URL: https://issues.apache.org/jira/browse/SOLR-2907
 Project: Solr
  Issue Type: Bug
  Components: contrib - DataImportHandler, Schema and Analysis
Affects Versions: 3.4
Reporter: Alan Baker

 We are using solr for our site and ran into this error in our own schema and 
 I was able to reproduce it using the dataimport example code in the solr 
 project.  We do not get this error in SOLR 1.4 only started seeing it as we 
 are working to upgrade to 3.4.0.  It fails when delta-importing linked tables.
 Complete trace:
 Nov 18, 2011 5:21:02 PM org.apache.solr.handler.dataimport.DataImporter 
 doDeltaImport
 SEVERE: Delta Import Failed
 java.lang.IllegalArgumentException: deltaQuery has no column to resolve to 
 declared primary key pk='ITEM_ID, CATEGORY_ID'
   at 
 org.apache.solr.handler.dataimport.DocBuilder.findMatchingPkColumn(DocBuilder.java:849)
   at 
 org.apache.solr.handler.dataimport.DocBuilder.collectDelta(DocBuilder.java:900)
   at 
 org.apache.solr.handler.dataimport.DocBuilder.collectDelta(DocBuilder.java:879)
   at 
 org.apache.solr.handler.dataimport.DocBuilder.doDelta(DocBuilder.java:285)
   at 
 org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:179)
   at 
 org.apache.solr.handler.dataimport.DataImporter.doDeltaImport(DataImporter.java:390)
   at 
 org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:429)
   at 
 org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:408)
 I used this dataConfig from the wiki on the data import:
 dataConfig
 dataSource driver=org.hsqldb.jdbcDriver 
 url=jdbc:hsqldb:./example-DIH/hsqldb/ex user=sa /
 document
entity  name=item pk=ID 
   query=select * from item 
 deltaImportQuery=select * from item where 
 ID=='${dataimporter.delta.id}'
 deltaQuery=select id from item where last_modified gt; 
 '${dataimporter.last_index_time}'
 entity name=item_category pk=ITEM_ID, CATEGORY_ID
 query=select CATEGORY_ID from item_category where 
 ITEM_ID='${item.ID}'
 deltaQuery=select ITEM_ID, CATEGORY_ID from 
 item_category where last_modified  '${dataimporter.last_index_time}'
 parentDeltaQuery=select ID from item where 
 ID=${item_category.ITEM_ID}
 
   entity name=category pk=ID
 query=select DESCRIPTION as cat from category where 
 ID = '${item_category.CATEGORY_ID}'
 deltaQuery=select ID from category where 
 last_modified gt; '${dataimporter.last_index_time}'
 parentDeltaQuery=select ITEM_ID, CATEGORY_ID from 
 item_category where CATEGORY_ID=${category.ID}/
   /entity
 /entity
 
 /document
 /dataConfig
 To reproduce use the data config from above and set the dataimport.properties 
 last update times to before the last_modifed date in the example

[jira] [Commented] (LUCENE-5977) IW should safeguard against token streams returning invalid offsets for multi-valued fields

2014-09-29 Thread ASF subversion and git services (JIRA)


[ 
https://issues.apache.org/jira/browse/LUCENE-5977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14151743#comment-14151743
 ] 

ASF subversion and git services commented on LUCENE-5977:
-

Commit 1628196 from [~rcmuir] in branch 'dev/branches/branch_5x'
[ https://svn.apache.org/r1628196 ]

LUCENE-5977: Fix tokenstream safety checks in IndexWriter to work across 
multi-valued fields

 IW should safeguard against token streams returning invalid offsets for 
 multi-valued fields
 ---

 Key: LUCENE-5977
 URL: https://issues.apache.org/jira/browse/LUCENE-5977
 Project: Lucene - Core
  Issue Type: Bug
Affects Versions: 4.9, 4.9.1, 4.10, 4.10.1
Reporter: Dawid Weiss
Priority: Minor
 Fix For: 4.10.2

 Attachments: LUCENE-5977.patch


 We have a custom token stream that emits information about offsets of each 
 token. My (wrong) assumption was that for multi-valued fields a token 
 stream's offset information is magically shifted, much like this is the case 
 with positions. It's not the case -- offsets should be increasing and 
 monotonic across all instances of a field, even if it has custom token 
 streams. So, something like this:
 {code}
 doc.add(new Field(field-foo, new CannedTokenStream(token(bar, 1, 
 150, 160)), ftype));
 doc.add(new Field(field-foo, new CannedTokenStream(token(bar, 1,  
 50,  60)), ftype));
 {code}
 where the token function is defined as:
 {code}
 token(String image, int positionIncrement, int startOffset, int endOffset)
 {code}
 will result in either a cryptic assertion thrown from IW:
 {code}
 Exception in thread main java.lang.AssertionError
   at 
 org.apache.lucene.index.FreqProxTermsWriterPerField.writeOffsets(FreqProxTermsWriterPerField.java:99)
 {code}
 or nothing (or a codec error) if run without assertions.
 Obviously returning non-shifted offsets from subsequent token streams makes 
 little sense but I wonder if it could be made more explicit (or asserted) 
 that offsets need to be increasing between multiple-values. The minimum is to 
 add some documentation to OffsetAttribute. I don't know if offsets should be 
 shifted automatically, as it's the case with positions -- this would change 
 the semantics of existing tokenizers and filters which implement such 
 shifting internally already.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-4212) Support for facet pivot query for filtered count

2014-09-29 Thread Steve Molloy (JIRA)


 [ 
https://issues.apache.org/jira/browse/SOLR-4212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Steve Molloy updated SOLR-4212:
---
Description: 
Facet pivot provide hierarchical support for computing data used to populate a 
treemap or similar visualization. TreeMaps usually offer users extra 
information by applying an overlay color on top of the existing square sizes 
based on hierarchical counts. This second count is based on user choices, 
representing, usually with gradient, the proportion of the square that fits the 
user's choices.

The proposition is to add a facet.pivot.q parameter that would allow to specify 
one or more queries (per field) that would be intersected with DocSet used to 
calculate pivot count, stored in separate qcounts list, each entry keyed by the 
query.

  was:
Facet pivot provide hierarchical support for computing data used to populate a 
treemap or similar visualization. TreeMaps usually offer users extra 
information by applying an overlay color on top of the existing square sizes 
based on hierarchical counts. This second count is based on user choices, 
representing, usually with gradient, the proportion of the square that fits the 
user's choices.

The proposition is to add a facet.pivot.q parameter that would allow to specify 
a query (per field) that would be intersected with DocSet used to calculate 
pivot count, stored in separate q-count.


 Support for facet pivot query for filtered count
 

 Key: SOLR-4212
 URL: https://issues.apache.org/jira/browse/SOLR-4212
 Project: Solr
  Issue Type: Improvement
  Components: search
Affects Versions: 4.0
Reporter: Steve Molloy
 Fix For: 4.9, Trunk

 Attachments: SOLR-4212.patch, SOLR-4212.patch, SOLR-4212.patch, 
 patch-4212.txt


 Facet pivot provide hierarchical support for computing data used to populate 
 a treemap or similar visualization. TreeMaps usually offer users extra 
 information by applying an overlay color on top of the existing square sizes 
 based on hierarchical counts. This second count is based on user choices, 
 representing, usually with gradient, the proportion of the square that fits 
 the user's choices.
 The proposition is to add a facet.pivot.q parameter that would allow to 
 specify one or more queries (per field) that would be intersected with DocSet 
 used to calculate pivot count, stored in separate qcounts list, each entry 
 keyed by the query.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS-MAVEN] Lucene-Solr-Maven-trunk #1222: POMs out of sync

Build: https://builds.apache.org/job/Lucene-Solr-Maven-trunk/1222/

1 tests failed.
REGRESSION:  
org.apache.solr.cloud.CloudExitableDirectoryReaderTest.testDistribSearch

Error Message:
no exception matching expected: 400: Request took too long during query 
expansion. Terminating request.

Stack Trace:
java.lang.AssertionError: no exception matching expected: 400: Request took too 
long during query expansion. Terminating request.
at 
__randomizedtesting.SeedInfo.seed([D39863050D5490D9:527EED1D7A0BF0E5]:0)
at org.junit.Assert.fail(Assert.java:93)
at 
org.apache.solr.cloud.CloudExitableDirectoryReaderTest.assertFail(CloudExitableDirectoryReaderTest.java:101)
at 
org.apache.solr.cloud.CloudExitableDirectoryReaderTest.doTimeoutTests(CloudExitableDirectoryReaderTest.java:81)
at 
org.apache.solr.cloud.CloudExitableDirectoryReaderTest.doTest(CloudExitableDirectoryReaderTest.java:54)




Build Log:
[...truncated 52905 lines...]
BUILD FAILED
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Maven-trunk/build.xml:547:
 The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Maven-trunk/build.xml:199:
 The following error occurred while executing this line:
: Java returned: 1

Total time: 229 minutes 20 seconds
Build step 'Invoke Ant' marked build as failure
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Resolved] (LUCENE-5977) IW should safeguard against token streams returning invalid offsets for multi-valued fields

2014-09-29 Thread ASF subversion and git services (JIRA)


 [ 
https://issues.apache.org/jira/browse/LUCENE-5977?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Muir resolved LUCENE-5977.
-
   Resolution: Fixed
Fix Version/s: Trunk
   5.0

Thank you for debugging this Dawid!


 IW should safeguard against token streams returning invalid offsets for 
 multi-valued fields
 ---

 Key: LUCENE-5977
 URL: https://issues.apache.org/jira/browse/LUCENE-5977
 Project: Lucene - Core
  Issue Type: Bug
Affects Versions: 4.9, 4.9.1, 4.10, 4.10.1
Reporter: Dawid Weiss
Priority: Minor
 Fix For: 4.10.2, 5.0, Trunk

 Attachments: LUCENE-5977.patch


 We have a custom token stream that emits information about offsets of each 
 token. My (wrong) assumption was that for multi-valued fields a token 
 stream's offset information is magically shifted, much like this is the case 
 with positions. It's not the case -- offsets should be increasing and 
 monotonic across all instances of a field, even if it has custom token 
 streams. So, something like this:
 {code}
 doc.add(new Field(field-foo, new CannedTokenStream(token(bar, 1, 
 150, 160)), ftype));
 doc.add(new Field(field-foo, new CannedTokenStream(token(bar, 1,  
 50,  60)), ftype));
 {code}
 where the token function is defined as:
 {code}
 token(String image, int positionIncrement, int startOffset, int endOffset)
 {code}
 will result in either a cryptic assertion thrown from IW:
 {code}
 Exception in thread main java.lang.AssertionError
   at 
 org.apache.lucene.index.FreqProxTermsWriterPerField.writeOffsets(FreqProxTermsWriterPerField.java:99)
 {code}
 or nothing (or a codec error) if run without assertions.
 Obviously returning non-shifted offsets from subsequent token streams makes 
 little sense but I wonder if it could be made more explicit (or asserted) 
 that offsets need to be increasing between multiple-values. The minimum is to 
 add some documentation to OffsetAttribute. I don't know if offsets should be 
 shifted automatically, as it's the case with positions -- this would change 
 the semantics of existing tokenizers and filters which implement such 
 shifting internally already.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-5977) IW should safeguard against token streams returning invalid offsets for multi-valued fields


[ 
https://issues.apache.org/jira/browse/LUCENE-5977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14151763#comment-14151763
 ] 

ASF subversion and git services commented on LUCENE-5977:
-

Commit 1628200 from [~rcmuir] in branch 'dev/branches/lucene_solr_4_10'
[ https://svn.apache.org/r1628200 ]

LUCENE-5977: Fix tokenstream safety checks in IndexWriter to work across 
multi-valued fields

 IW should safeguard against token streams returning invalid offsets for 
 multi-valued fields
 ---

 Key: LUCENE-5977
 URL: https://issues.apache.org/jira/browse/LUCENE-5977
 Project: Lucene - Core
  Issue Type: Bug
Affects Versions: 4.9, 4.9.1, 4.10, 4.10.1
Reporter: Dawid Weiss
Priority: Minor
 Fix For: 4.10.2

 Attachments: LUCENE-5977.patch


 We have a custom token stream that emits information about offsets of each 
 token. My (wrong) assumption was that for multi-valued fields a token 
 stream's offset information is magically shifted, much like this is the case 
 with positions. It's not the case -- offsets should be increasing and 
 monotonic across all instances of a field, even if it has custom token 
 streams. So, something like this:
 {code}
 doc.add(new Field(field-foo, new CannedTokenStream(token(bar, 1, 
 150, 160)), ftype));
 doc.add(new Field(field-foo, new CannedTokenStream(token(bar, 1,  
 50,  60)), ftype));
 {code}
 where the token function is defined as:
 {code}
 token(String image, int positionIncrement, int startOffset, int endOffset)
 {code}
 will result in either a cryptic assertion thrown from IW:
 {code}
 Exception in thread main java.lang.AssertionError
   at 
 org.apache.lucene.index.FreqProxTermsWriterPerField.writeOffsets(FreqProxTermsWriterPerField.java:99)
 {code}
 or nothing (or a codec error) if run without assertions.
 Obviously returning non-shifted offsets from subsequent token streams makes 
 little sense but I wonder if it could be made more explicit (or asserted) 
 that offsets need to be increasing between multiple-values. The minimum is to 
 add some documentation to OffsetAttribute. I don't know if offsets should be 
 shifted automatically, as it's the case with positions -- this would change 
 the semantics of existing tokenizers and filters which implement such 
 shifting internally already.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-5.x-Linux (32bit/jdk1.8.0_40-ea-b04) - Build # 11197 - Still Failing!

2014-09-29 Thread ASF subversion and git services (JIRA)

Build: http://jenkins.thetaphi.de/job/Lucene-Solr-5.x-Linux/11197/
Java: 32bit/jdk1.8.0_40-ea-b04 -server -XX:+UseG1GC

All tests passed

Build Log:
[...truncated 4413 lines...]
[javac] Compiling 2 source files to 
/mnt/ssd/jenkins/workspace/Lucene-Solr-5.x-Linux/lucene/build/memory/classes/test
[javac] 
/mnt/ssd/jenkins/workspace/Lucene-Solr-5.x-Linux/lucene/memory/src/test/org/apache/lucene/index/memory/TestMemoryIndexAgainstRAMDir.java:434:
 error: incompatible types: IndexableField cannot be converted to Field
[javac]   for (Field field : nextDoc.getFields()) {
[javac]   ^
[javac] 
/mnt/ssd/jenkins/workspace/Lucene-Solr-5.x-Linux/lucene/memory/src/test/org/apache/lucene/index/memory/TestMemoryIndexAgainstRAMDir.java:445:
 error: cannot find symbol
[javac]   for (IndexableField field : doc.indexableFields()) {
[javac]  ^
[javac]   symbol:   method indexableFields()
[javac]   location: variable doc of type Document
[javac] Note: 
/mnt/ssd/jenkins/workspace/Lucene-Solr-5.x-Linux/lucene/memory/src/test/org/apache/lucene/index/memory/TestMemoryIndexAgainstRAMDir.java
 uses or overrides a deprecated API.
[javac] Note: Recompile with -Xlint:deprecation for details.
[javac] 2 errors

BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-5.x-Linux/build.xml:524: The following 
error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-5.x-Linux/build.xml:472: The following 
error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-5.x-Linux/build.xml:61: The following 
error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-5.x-Linux/extra-targets.xml:39: The 
following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-5.x-Linux/lucene/build.xml:447: The 
following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-5.x-Linux/lucene/common-build.xml:2142: 
The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-5.x-Linux/lucene/module-build.xml:55: 
The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-5.x-Linux/lucene/common-build.xml:785: 
The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-5.x-Linux/lucene/common-build.xml:799: 
The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-5.x-Linux/lucene/common-build.xml:1867: 
Compile failed; see the compiler error output for details.

Total time: 32 minutes 32 seconds
Build step 'Invoke Ant' marked build as failure
[description-setter] Description set: Java: 32bit/jdk1.8.0_40-ea-b04 -server 
-XX:+UseG1GC
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-6511) Fencepost error in LeaderInitiatedRecoveryThread


[ 
https://issues.apache.org/jira/browse/SOLR-6511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14151781#comment-14151781
 ] 

ASF subversion and git services commented on SOLR-6511:
---

Commit 1628203 from [~thelabdude] in branch 'dev/trunk'
[ https://svn.apache.org/r1628203 ]

SOLR-6511: adjust test logic to account for timing issues in zk session 
expiration scenario.

 Fencepost error in LeaderInitiatedRecoveryThread
 

 Key: SOLR-6511
 URL: https://issues.apache.org/jira/browse/SOLR-6511
 Project: Solr
  Issue Type: Bug
Reporter: Alan Woodward
Assignee: Timothy Potter
 Attachments: SOLR-6511.patch, SOLR-6511.patch


 At line 106:
 {code}
 while (continueTrying  ++tries  maxTries) {
 {code}
 should be
 {code}
 while (continueTrying  ++tries = maxTries) {
 {code}
 This is only a problem when called from DistributedUpdateProcessor, as it can 
 have maxTries set to 1, which means the loop is never actually run.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-6249) Schema API changes return success before all cores are updated


[ 
https://issues.apache.org/jira/browse/SOLR-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14151776#comment-14151776
 ] 

Timothy Potter commented on SOLR-6249:
--

It uses ZkIndexSchemaReader, which under the covers calls:

{{byte[] data = zkClient.getData(managedSchemaPath, watcher, stat, 
true);   }}

The last param says to retry on connection loss. Do you think we need to do 
more than this?

 Schema API changes return success before all cores are updated
 --

 Key: SOLR-6249
 URL: https://issues.apache.org/jira/browse/SOLR-6249
 Project: Solr
  Issue Type: Improvement
  Components: Schema and Analysis, SolrCloud
Reporter: Gregory Chanan
Assignee: Timothy Potter
 Attachments: SOLR-6249.patch, SOLR-6249.patch, SOLR-6249.patch


 See SOLR-6137 for more details.
 The basic issue is that Schema API changes return success when the first core 
 is updated, but other cores asynchronously read the updated schema from 
 ZooKeeper.
 So a client application could make a Schema API change and then index some 
 documents based on the new schema that may fail on other nodes.
 Possible fixes:
 1) Make the Schema API calls synchronous
 2) Give the client some ability to track the state of the schema.  They can 
 already do this to a certain extent by checking the Schema API on all the 
 replicas and verifying that the field has been added, though this is pretty 
 cumbersome.  Maybe it makes more sense to do this sort of thing on the 
 collection level, i.e. Schema API changes return the zk version to the 
 client.  We add an API to return the current zk version.  On a replica, if 
 the zk version is = the version the client has, the client knows that 
 replica has at least seen the schema change.  We could also provide an API to 
 do the distribution and checking across the different replicas of the 
 collection so that clients don't need ot do that themselves.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-6249) Schema API changes return success before all cores are updated


[ 
https://issues.apache.org/jira/browse/SOLR-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14151786#comment-14151786
 ] 

Noble Paul commented on SOLR-6249:
--

I'm afraid , it is not sufficient. It just retries if the connection is lost in 
the getData() call. It does not take care of the case where the watcher is lost 
because of a zkclient failure. 

The correct example is the constructor of ZkStateReader

{code:java}
   zkClient = new SolrZkClient(zkServerAddress, zkClientTimeout, 
zkClientConnectTimeout,
// on reconnect, reload cloud info
new OnReconnect() {
//implement stuff here
 }
});
{code}

this ensures that the watchers are persisted across reconnect.


 Schema API changes return success before all cores are updated
 --

 Key: SOLR-6249
 URL: https://issues.apache.org/jira/browse/SOLR-6249
 Project: Solr
  Issue Type: Improvement
  Components: Schema and Analysis, SolrCloud
Reporter: Gregory Chanan
Assignee: Timothy Potter
 Attachments: SOLR-6249.patch, SOLR-6249.patch, SOLR-6249.patch


 See SOLR-6137 for more details.
 The basic issue is that Schema API changes return success when the first core 
 is updated, but other cores asynchronously read the updated schema from 
 ZooKeeper.
 So a client application could make a Schema API change and then index some 
 documents based on the new schema that may fail on other nodes.
 Possible fixes:
 1) Make the Schema API calls synchronous
 2) Give the client some ability to track the state of the schema.  They can 
 already do this to a certain extent by checking the Schema API on all the 
 replicas and verifying that the field has been added, though this is pretty 
 cumbersome.  Maybe it makes more sense to do this sort of thing on the 
 collection level, i.e. Schema API changes return the zk version to the 
 client.  We add an API to return the current zk version.  On a replica, if 
 the zk version is = the version the client has, the client knows that 
 replica has at least seen the schema change.  We could also provide an API to 
 do the distribution and checking across the different replicas of the 
 collection so that clients don't need ot do that themselves.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Comment Edited] (SOLR-6249) Schema API changes return success before all cores are updated


[ 
https://issues.apache.org/jira/browse/SOLR-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14151786#comment-14151786
 ] 

Noble Paul edited comment on SOLR-6249 at 9/29/14 3:25 PM:
---

I'm afraid , it is not sufficient. It just retries if the connection is lost in 
the getData() call. It does not take care of the case where the watcher is lost 
because of a zkclient failure. 

The correct example is the constructor of ZkStateReader

{code:java}
   zkClient = new SolrZkClient(zkServerAddress, zkClientTimeout, 
zkClientConnectTimeout,
// on reconnect, reload cloud info
new OnReconnect() {
//implement stuff here
 }
});
{code}

this ensures that the watchers are persisted across reconnect.

This is why I suggested the ZkStateReader is the right place to set watchers


was (Author: noble.paul):
I'm afraid , it is not sufficient. It just retries if the connection is lost in 
the getData() call. It does not take care of the case where the watcher is lost 
because of a zkclient failure. 

The correct example is the constructor of ZkStateReader

{code:java}
   zkClient = new SolrZkClient(zkServerAddress, zkClientTimeout, 
zkClientConnectTimeout,
// on reconnect, reload cloud info
new OnReconnect() {
//implement stuff here
 }
});
{code}

this ensures that the watchers are persisted across reconnect.


 Schema API changes return success before all cores are updated
 --

 Key: SOLR-6249
 URL: https://issues.apache.org/jira/browse/SOLR-6249
 Project: Solr
  Issue Type: Improvement
  Components: Schema and Analysis, SolrCloud
Reporter: Gregory Chanan
Assignee: Timothy Potter
 Attachments: SOLR-6249.patch, SOLR-6249.patch, SOLR-6249.patch


 See SOLR-6137 for more details.
 The basic issue is that Schema API changes return success when the first core 
 is updated, but other cores asynchronously read the updated schema from 
 ZooKeeper.
 So a client application could make a Schema API change and then index some 
 documents based on the new schema that may fail on other nodes.
 Possible fixes:
 1) Make the Schema API calls synchronous
 2) Give the client some ability to track the state of the schema.  They can 
 already do this to a certain extent by checking the Schema API on all the 
 replicas and verifying that the field has been added, though this is pretty 
 cumbersome.  Maybe it makes more sense to do this sort of thing on the 
 collection level, i.e. Schema API changes return the zk version to the 
 client.  We add an API to return the current zk version.  On a replica, if 
 the zk version is = the version the client has, the client knows that 
 replica has at least seen the schema change.  We could also provide an API to 
 do the distribution and checking across the different replicas of the 
 collection so that clients don't need ot do that themselves.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-6476) Create a bulk mode for schema API


[ 
https://issues.apache.org/jira/browse/SOLR-6476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14151790#comment-14151790
 ] 

Noble Paul commented on SOLR-6476:
--

The only outstanding issue AFAIK is the no:of reties. whether it should be 
finite or infinite. I believe all user interacting APIs should be coded to fail 
gracefully. If we can resolve that I can commit this

 Create a bulk mode for schema API
 -

 Key: SOLR-6476
 URL: https://issues.apache.org/jira/browse/SOLR-6476
 Project: Solr
  Issue Type: Bug
  Components: Schema and Analysis
Reporter: Noble Paul
Assignee: Noble Paul
  Labels: managedResource
 Attachments: SOLR-6476.patch, SOLR-6476.patch, SOLR-6476.patch, 
 SOLR-6476.patch, SOLR-6476.patch, SOLR-6476.patch, SOLR-6476.patch


 The current schema API does one operation at a time and the normal usecase is 
 that users add multiple fields/fieldtypes/copyFields etc in one shot.
 example 
 {code:javascript}
 curl http://localhost:8983/solr/collection1/schema -H 
 'Content-type:application/json'  -d '{
 add-field: {
 name:sell-by,
 type:tdate,
 stored:true
 },
 add-field:{
 name:catchall,
 type:text_general,
 stored:false
 }
 }
 {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Created] (SOLR-6570) Run SolrZkClient session watch asynchronously

Ramkumar Aiyengar created SOLR-6570:
---

 Summary: Run SolrZkClient session watch asynchronously
 Key: SOLR-6570
 URL: https://issues.apache.org/jira/browse/SOLR-6570
 Project: Solr
  Issue Type: Improvement
  Components: SolrCloud
Reporter: Ramkumar Aiyengar
Priority: Minor


Spin off from SOLR-6261.

This kind of already happens because the only session watcher in 
{{ConnectionManager}} does it's processing async (changed in SOLR-5615), but 
this is more consistent and avoids the possibility that a second session 
watcher or a change to that code re-surfaces the issue again.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[GitHub] lucene-solr pull request: Wrap connectionManager in SolrZkClient c...

2014-09-29 Thread andyetitmoves

Github user andyetitmoves commented on the pull request:

https://github.com/apache/lucene-solr/pull/78#issuecomment-57178936
  
Moved to SOLR-6570


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-6570) Run SolrZkClient session watch asynchronously

2014-09-29 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/SOLR-6570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14151799#comment-14151799
 ] 

ASF GitHub Bot commented on SOLR-6570:
--

Github user andyetitmoves commented on the pull request:

https://github.com/apache/lucene-solr/pull/78#issuecomment-57178936
  
Moved to SOLR-6570


 Run SolrZkClient session watch asynchronously
 -

 Key: SOLR-6570
 URL: https://issues.apache.org/jira/browse/SOLR-6570
 Project: Solr
  Issue Type: Improvement
  Components: SolrCloud
Reporter: Ramkumar Aiyengar
Priority: Minor

 Spin off from SOLR-6261.
 This kind of already happens because the only session watcher in 
 {{ConnectionManager}} does it's processing async (changed in SOLR-5615), but 
 this is more consistent and avoids the possibility that a second session 
 watcher or a change to that code re-surfaces the issue again.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-6261) Run ZK watch event callbacks in parallel to the event thread


[ 
https://issues.apache.org/jira/browse/SOLR-6261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14151800#comment-14151800
 ] 

Ramkumar Aiyengar commented on SOLR-6261:
-

np, I have now opened SOLR-6570 for this, this is for pull #78 on GitHub.

 Run ZK watch event callbacks in parallel to the event thread
 

 Key: SOLR-6261
 URL: https://issues.apache.org/jira/browse/SOLR-6261
 Project: Solr
  Issue Type: Improvement
  Components: SolrCloud
Affects Versions: 4.9
Reporter: Ramkumar Aiyengar
Assignee: Mark Miller
Priority: Minor
 Fix For: 4.10, Trunk

 Attachments: thread-compare.jpg


 Currently checking for leadership (due to the leader's ephemeral node going 
 away) happens in ZK's event thread. If there are many cores and all of them 
 are due leadership, then they would have to serially go through the two-way 
 sync and leadership takeover.
 For tens of cores, this could mean 30-40s without leadership before the last 
 in the list even gets to start the leadership process. If the leadership 
 process happens in a separate thread, then the cores could all take over in 
 parallel.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-trunk-MacOSX (64bit/jdk1.8.0) - Build # 1857 - Still Failing!

Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-MacOSX/1857/
Java: 64bit/jdk1.8.0 -XX:-UseCompressedOops -XX:+UseConcMarkSweepGC

1 tests failed.
REGRESSION:  org.apache.solr.TestDistributedMissingSort.testDistribSearch

Error Message:
Timeout occured while waiting response from server at: https://127.0.0.1:51348

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: Timeout occured while waiting 
response from server at: https://127.0.0.1:51348
at 
__randomizedtesting.SeedInfo.seed([ABDE7FF6831C9F0F:2A38F1EEF443FF33]:0)
at 
org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:582)
at 
org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:215)
at 
org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:211)
at 
org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(AbstractUpdateRequest.java:124)
at org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:116)
at org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:102)
at 
org.apache.solr.BaseDistributedSearchTestCase.indexDoc(BaseDistributedSearchTestCase.java:438)
at 
org.apache.solr.BaseDistributedSearchTestCase.indexr(BaseDistributedSearchTestCase.java:420)
at 
org.apache.solr.TestDistributedMissingSort.index(TestDistributedMissingSort.java:90)
at 
org.apache.solr.TestDistributedMissingSort.doTest(TestDistributedMissingSort.java:42)
at 
org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:875)
at sun.reflect.GeneratedMethodAccessor61.invoke(Unknown Source)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at

[jira] [Commented] (SOLR-6249) Schema API changes return success before all cores are updated


[ 
https://issues.apache.org/jira/browse/SOLR-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14151809#comment-14151809
 ] 

Timothy Potter commented on SOLR-6249:
--

Ok, that makes sense but seems like a generic issue vs. specific to this class. 
We could solve this specifically right now doing what you suggest but rather 
than polluting ZkStateReader with utility methods for setting watchers, I think 
we should have a generic watcher support class. ZkStateReader's job is to 
handle collection state right? It's not some generic ZK utility class that the 
rest of the code should use. There should be a ticket for refactoring any code 
that sets a watcher to do it correctly, including ZkIndexSchemaReader, or is 
this the only place in the code that doesn't set a watcher correctly?


 Schema API changes return success before all cores are updated
 --

 Key: SOLR-6249
 URL: https://issues.apache.org/jira/browse/SOLR-6249
 Project: Solr
  Issue Type: Improvement
  Components: Schema and Analysis, SolrCloud
Reporter: Gregory Chanan
Assignee: Timothy Potter
 Attachments: SOLR-6249.patch, SOLR-6249.patch, SOLR-6249.patch


 See SOLR-6137 for more details.
 The basic issue is that Schema API changes return success when the first core 
 is updated, but other cores asynchronously read the updated schema from 
 ZooKeeper.
 So a client application could make a Schema API change and then index some 
 documents based on the new schema that may fail on other nodes.
 Possible fixes:
 1) Make the Schema API calls synchronous
 2) Give the client some ability to track the state of the schema.  They can 
 already do this to a certain extent by checking the Schema API on all the 
 replicas and verifying that the field has been added, though this is pretty 
 cumbersome.  Maybe it makes more sense to do this sort of thing on the 
 collection level, i.e. Schema API changes return the zk version to the 
 client.  We add an API to return the current zk version.  On a replica, if 
 the zk version is = the version the client has, the client knows that 
 replica has at least seen the schema change.  We could also provide an API to 
 do the distribution and checking across the different replicas of the 
 collection so that clients don't need ot do that themselves.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-6249) Schema API changes return success before all cores are updated


[ 
https://issues.apache.org/jira/browse/SOLR-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14151818#comment-14151818
 ] 

Noble Paul commented on SOLR-6249:
--

I was just giving out an easy solution.This logic can be pushed to SolrZkClient 
and ZkStateReader and everyone else can use it from there. 

 Schema API changes return success before all cores are updated
 --

 Key: SOLR-6249
 URL: https://issues.apache.org/jira/browse/SOLR-6249
 Project: Solr
  Issue Type: Improvement
  Components: Schema and Analysis, SolrCloud
Reporter: Gregory Chanan
Assignee: Timothy Potter
 Attachments: SOLR-6249.patch, SOLR-6249.patch, SOLR-6249.patch


 See SOLR-6137 for more details.
 The basic issue is that Schema API changes return success when the first core 
 is updated, but other cores asynchronously read the updated schema from 
 ZooKeeper.
 So a client application could make a Schema API change and then index some 
 documents based on the new schema that may fail on other nodes.
 Possible fixes:
 1) Make the Schema API calls synchronous
 2) Give the client some ability to track the state of the schema.  They can 
 already do this to a certain extent by checking the Schema API on all the 
 replicas and verifying that the field has been added, though this is pretty 
 cumbersome.  Maybe it makes more sense to do this sort of thing on the 
 collection level, i.e. Schema API changes return the zk version to the 
 client.  We add an API to return the current zk version.  On a replica, if 
 the zk version is = the version the client has, the client knows that 
 replica has at least seen the schema change.  We could also provide an API to 
 do the distribution and checking across the different replicas of the 
 collection so that clients don't need ot do that themselves.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Created] (SOLR-6571) Need a znode watcher support class

Timothy Potter created SOLR-6571:


 Summary: Need a znode watcher support class
 Key: SOLR-6571
 URL: https://issues.apache.org/jira/browse/SOLR-6571
 Project: Solr
  Issue Type: Improvement
  Components: SolrCloud
Reporter: Timothy Potter


When implementing SOLR-6249, [~noble.paul] points out that ZkIndexSchemaReader 
doesn't watch the managed schema znode correctly and should use a strategy 
similar to what ZkStateReader does so that watchers persist across zk client 
connection failures:

The correct example is the constructor of ZkStateReader
   zkClient = new SolrZkClient(zkServerAddress, zkClientTimeout, 
zkClientConnectTimeout,
// on reconnect, reload cloud info
new OnReconnect() {
//implement stuff here
 }
});
this ensures that the watchers are persisted across reconnect.

We need a watch support class to help developers implement watchers correctly 
instead of having some code do it correctly and other code not.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-6249) Schema API changes return success before all cores are updated


[ 
https://issues.apache.org/jira/browse/SOLR-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14151842#comment-14151842
 ] 

Timothy Potter commented on SOLR-6249:
--

Sure but I'm tired of all this tribal knowledge around how to do something 
right in the code. We need a clear path for future coders to follow and it 
seems like a watcher support class is the right solution. I've opened SOLR-6571 
please make comments there.

 Schema API changes return success before all cores are updated
 --

 Key: SOLR-6249
 URL: https://issues.apache.org/jira/browse/SOLR-6249
 Project: Solr
  Issue Type: Improvement
  Components: Schema and Analysis, SolrCloud
Reporter: Gregory Chanan
Assignee: Timothy Potter
 Attachments: SOLR-6249.patch, SOLR-6249.patch, SOLR-6249.patch


 See SOLR-6137 for more details.
 The basic issue is that Schema API changes return success when the first core 
 is updated, but other cores asynchronously read the updated schema from 
 ZooKeeper.
 So a client application could make a Schema API change and then index some 
 documents based on the new schema that may fail on other nodes.
 Possible fixes:
 1) Make the Schema API calls synchronous
 2) Give the client some ability to track the state of the schema.  They can 
 already do this to a certain extent by checking the Schema API on all the 
 replicas and verifying that the field has been added, though this is pretty 
 cumbersome.  Maybe it makes more sense to do this sort of thing on the 
 collection level, i.e. Schema API changes return the zk version to the 
 client.  We add an API to return the current zk version.  On a replica, if 
 the zk version is = the version the client has, the client knows that 
 replica has at least seen the schema change.  We could also provide an API to 
 do the distribution and checking across the different replicas of the 
 collection so that clients don't need ot do that themselves.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-6249) Schema API changes return success before all cores are updated


[ 
https://issues.apache.org/jira/browse/SOLR-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14151850#comment-14151850
 ] 

Mark Miller commented on SOLR-6249:
---

This is not really tribal knowledge - it's ZooKeeper 101. Watchers do no 
persist across session timeouts and you need to re-create any watchers on 
reconnecting after a session timeout.

 Schema API changes return success before all cores are updated
 --

 Key: SOLR-6249
 URL: https://issues.apache.org/jira/browse/SOLR-6249
 Project: Solr
  Issue Type: Improvement
  Components: Schema and Analysis, SolrCloud
Reporter: Gregory Chanan
Assignee: Timothy Potter
 Attachments: SOLR-6249.patch, SOLR-6249.patch, SOLR-6249.patch


 See SOLR-6137 for more details.
 The basic issue is that Schema API changes return success when the first core 
 is updated, but other cores asynchronously read the updated schema from 
 ZooKeeper.
 So a client application could make a Schema API change and then index some 
 documents based on the new schema that may fail on other nodes.
 Possible fixes:
 1) Make the Schema API calls synchronous
 2) Give the client some ability to track the state of the schema.  They can 
 already do this to a certain extent by checking the Schema API on all the 
 replicas and verifying that the field has been added, though this is pretty 
 cumbersome.  Maybe it makes more sense to do this sort of thing on the 
 collection level, i.e. Schema API changes return the zk version to the 
 client.  We add an API to return the current zk version.  On a replica, if 
 the zk version is = the version the client has, the client knows that 
 replica has at least seen the schema change.  We could also provide an API to 
 do the distribution and checking across the different replicas of the 
 collection so that clients don't need ot do that themselves.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-6278) add admin/collections?action=DELETEREPLICAcore=... support, make collection=... and shard=... parameters optional

2014-09-29 Thread Christine Poerschke (JIRA)


[ 
https://issues.apache.org/jira/browse/SOLR-6278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14151857#comment-14151857
 ] 

Christine Poerschke commented on SOLR-6278:
---

Yes, the use of 

{{admin/cores?action=UNLOADdeleteInstanceDir=truecore=eoe1_shard8_replica1}}

would work for us as equivalent of

{{admin/collections?action=DELETEREPLICAcore=eoe1_shard8_replica1}}

The latter would merely hide the action=UNLOADdeleteInstanceDir=true 
implementation detail but not exposing core names at collections API level also 
makes sense.

Our typical use case is to decommission all replicas under a given solr home or 
another way to describe it would be everything living in a particular host/port 
solr instance. Collection name, shard name, replica name, core name, they are 
strictly speaking unnecessary detail and just a

{{host:port/solr/admin/something?action=DELETE_ALL_REPLICAS}}

could do the job or if one wanted to send admin commands not directly to the 
solr instance being wiped out then a

{{host:port/solr/admin/something?action=DELETE_ALL_REPLICAShost=...port=...}}

So, in short, yes, a won't fix JIRA resolution would be fine for our use case.

 add admin/collections?action=DELETEREPLICAcore=... support, make 
 collection=... and shard=... parameters optional
 --

 Key: SOLR-6278
 URL: https://issues.apache.org/jira/browse/SOLR-6278
 Project: Solr
  Issue Type: Improvement
Reporter: Christine Poerschke
Assignee: Erick Erickson
 Attachments: SOLR-6278.patch


 To add {{core=...}} as an alternative to {{replica=...}} way of identifying 
 what is to be deleted, {{collection=...}} and {{shard=...}} to be optional 
 provided the other parameters uniquely identify exactly one deletion target.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-6249) Schema API changes return success before all cores are updated


[ 
https://issues.apache.org/jira/browse/SOLR-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14151861#comment-14151861
 ] 

Mark Miller commented on SOLR-6249:
---

bq. The last param says to retry on connection loss. Do you think we need to do 
more than this?

Again, ZooKeeper 101.

You can lose the ZK connection generally in two ways:

* ConnectionLoss
* SessionExpiration

When you lose the connection due to ConnectionLoss, all the Watches remain. You 
have a couple choices - you can retry and hope the connection is reestablished, 
or do something else (see the leader election algorithm for an example of 
needing to do something else). Usually, you want to retry until you get a 
session expiration. That is what passing true as the final param does for you - 
it handles ConnectionLoss in the way you want to handle it 99% of the time.

SessionExpiration means the connection was lost for too long. Watches do not 
span sessions. In this case, when you reconnect to ZooKeeper, it's pretty use 
case specific how you need to handle things, but at a minimum, it usually means 
re-creating any watches.

 Schema API changes return success before all cores are updated
 --

 Key: SOLR-6249
 URL: https://issues.apache.org/jira/browse/SOLR-6249
 Project: Solr
  Issue Type: Improvement
  Components: Schema and Analysis, SolrCloud
Reporter: Gregory Chanan
Assignee: Timothy Potter
 Attachments: SOLR-6249.patch, SOLR-6249.patch, SOLR-6249.patch


 See SOLR-6137 for more details.
 The basic issue is that Schema API changes return success when the first core 
 is updated, but other cores asynchronously read the updated schema from 
 ZooKeeper.
 So a client application could make a Schema API change and then index some 
 documents based on the new schema that may fail on other nodes.
 Possible fixes:
 1) Make the Schema API calls synchronous
 2) Give the client some ability to track the state of the schema.  They can 
 already do this to a certain extent by checking the Schema API on all the 
 replicas and verifying that the field has been added, though this is pretty 
 cumbersome.  Maybe it makes more sense to do this sort of thing on the 
 collection level, i.e. Schema API changes return the zk version to the 
 client.  We add an API to return the current zk version.  On a replica, if 
 the zk version is = the version the client has, the client knows that 
 replica has at least seen the schema change.  We could also provide an API to 
 do the distribution and checking across the different replicas of the 
 collection so that clients don't need ot do that themselves.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-6249) Schema API changes return success before all cores are updated


[ 
https://issues.apache.org/jira/browse/SOLR-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14151860#comment-14151860
 ] 

Steve Rowe commented on SOLR-6249:
--

The fault is all mine. 

 Schema API changes return success before all cores are updated
 --

 Key: SOLR-6249
 URL: https://issues.apache.org/jira/browse/SOLR-6249
 Project: Solr
  Issue Type: Improvement
  Components: Schema and Analysis, SolrCloud
Reporter: Gregory Chanan
Assignee: Timothy Potter
 Attachments: SOLR-6249.patch, SOLR-6249.patch, SOLR-6249.patch


 See SOLR-6137 for more details.
 The basic issue is that Schema API changes return success when the first core 
 is updated, but other cores asynchronously read the updated schema from 
 ZooKeeper.
 So a client application could make a Schema API change and then index some 
 documents based on the new schema that may fail on other nodes.
 Possible fixes:
 1) Make the Schema API calls synchronous
 2) Give the client some ability to track the state of the schema.  They can 
 already do this to a certain extent by checking the Schema API on all the 
 replicas and verifying that the field has been added, though this is pretty 
 cumbersome.  Maybe it makes more sense to do this sort of thing on the 
 collection level, i.e. Schema API changes return the zk version to the 
 client.  We add an API to return the current zk version.  On a replica, if 
 the zk version is = the version the client has, the client knows that 
 replica has at least seen the schema change.  We could also provide an API to 
 do the distribution and checking across the different replicas of the 
 collection so that clients don't need ot do that themselves.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-6249) Schema API changes return success before all cores are updated


[ 
https://issues.apache.org/jira/browse/SOLR-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14151854#comment-14151854
 ] 

Timothy Potter commented on SOLR-6249:
--

Ok fair enough  clearly the implementer of ZkIndexSchemaReader (not me) 
missed that day.

 Schema API changes return success before all cores are updated
 --

 Key: SOLR-6249
 URL: https://issues.apache.org/jira/browse/SOLR-6249
 Project: Solr
  Issue Type: Improvement
  Components: Schema and Analysis, SolrCloud
Reporter: Gregory Chanan
Assignee: Timothy Potter
 Attachments: SOLR-6249.patch, SOLR-6249.patch, SOLR-6249.patch


 See SOLR-6137 for more details.
 The basic issue is that Schema API changes return success when the first core 
 is updated, but other cores asynchronously read the updated schema from 
 ZooKeeper.
 So a client application could make a Schema API change and then index some 
 documents based on the new schema that may fail on other nodes.
 Possible fixes:
 1) Make the Schema API calls synchronous
 2) Give the client some ability to track the state of the schema.  They can 
 already do this to a certain extent by checking the Schema API on all the 
 replicas and verifying that the field has been added, though this is pretty 
 cumbersome.  Maybe it makes more sense to do this sort of thing on the 
 collection level, i.e. Schema API changes return the zk version to the 
 client.  We add an API to return the current zk version.  On a replica, if 
 the zk version is = the version the client has, the client knows that 
 replica has at least seen the schema change.  We could also provide an API to 
 do the distribution and checking across the different replicas of the 
 collection so that clients don't need ot do that themselves.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-trunk-Linux (64bit/jdk1.9.0-ea-b28) - Build # 11350 - Still Failing!

Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/11350/
Java: 64bit/jdk1.9.0-ea-b28 -XX:+UseCompressedOops -XX:+UseParallelGC

3 tests failed.
REGRESSION:  org.apache.solr.cloud.DeleteReplicaTest.testDistribSearch

Error Message:
No live SolrServers available to handle this 
request:[http://127.0.0.1:47512/vqqpo, http://127.0.0.1:59264/vqqpo, 
http://127.0.0.1:59549/vqqpo, http://127.0.0.1:46014/vqqpo, 
http://127.0.0.1:56419/vqqpo]

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: No live SolrServers available 
to handle this request:[http://127.0.0.1:47512/vqqpo, 
http://127.0.0.1:59264/vqqpo, http://127.0.0.1:59549/vqqpo, 
http://127.0.0.1:46014/vqqpo, http://127.0.0.1:56419/vqqpo]
at 
__randomizedtesting.SeedInfo.seed([48532332BF07798A:C9B5AD2AC85819B6]:0)
at 
org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:322)
at 
org.apache.solr.client.solrj.impl.CloudSolrServer.sendRequest(CloudSolrServer.java:880)
at 
org.apache.solr.client.solrj.impl.CloudSolrServer.requestWithRetryOnStaleState(CloudSolrServer.java:658)
at 
org.apache.solr.client.solrj.impl.CloudSolrServer.request(CloudSolrServer.java:601)
at 
org.apache.solr.cloud.DeleteReplicaTest.removeAndWaitForReplicaGone(DeleteReplicaTest.java:171)
at 
org.apache.solr.cloud.DeleteReplicaTest.deleteLiveReplicaTest(DeleteReplicaTest.java:144)
at 
org.apache.solr.cloud.DeleteReplicaTest.doTest(DeleteReplicaTest.java:88)
at 
org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:869)
at sun.reflect.GeneratedMethodAccessor39.invoke(Unknown Source)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:484)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at

[jira] [Closed] (SOLR-6571) Need a znode watcher support class


 [ 
https://issues.apache.org/jira/browse/SOLR-6571?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Timothy Potter closed SOLR-6571.

Resolution: Won't Fix

Looks like ZkIndexSchemaReader is the only place where this isn't done 
correctly so there's not really any general problem here. Will fix the schema 
reader as part of SOLR-6249

 Need a znode watcher support class
 --

 Key: SOLR-6571
 URL: https://issues.apache.org/jira/browse/SOLR-6571
 Project: Solr
  Issue Type: Improvement
  Components: SolrCloud
Reporter: Timothy Potter

 When implementing SOLR-6249, [~noble.paul] points out that 
 ZkIndexSchemaReader doesn't watch the managed schema znode correctly and 
 should use a strategy similar to what ZkStateReader does so that watchers 
 persist across zk client connection failures:
 The correct example is the constructor of ZkStateReader
zkClient = new SolrZkClient(zkServerAddress, zkClientTimeout, 
 zkClientConnectTimeout,
 // on reconnect, reload cloud info
 new OnReconnect() {
 //implement stuff here
  }
 });
 this ensures that the watchers are persisted across reconnect.
 We need a watch support class to help developers implement watchers correctly 
 instead of having some code do it correctly and other code not.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-6249) Schema API changes return success before all cores are updated


[ 
https://issues.apache.org/jira/browse/SOLR-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14151926#comment-14151926
 ] 

Timothy Potter commented on SOLR-6249:
--

I'll commit another fix to address the watcher re-connect issue for 
ZkIndexSchemaReader.

 Schema API changes return success before all cores are updated
 --

 Key: SOLR-6249
 URL: https://issues.apache.org/jira/browse/SOLR-6249
 Project: Solr
  Issue Type: Improvement
  Components: Schema and Analysis, SolrCloud
Reporter: Gregory Chanan
Assignee: Timothy Potter
 Attachments: SOLR-6249.patch, SOLR-6249.patch, SOLR-6249.patch


 See SOLR-6137 for more details.
 The basic issue is that Schema API changes return success when the first core 
 is updated, but other cores asynchronously read the updated schema from 
 ZooKeeper.
 So a client application could make a Schema API change and then index some 
 documents based on the new schema that may fail on other nodes.
 Possible fixes:
 1) Make the Schema API calls synchronous
 2) Give the client some ability to track the state of the schema.  They can 
 already do this to a certain extent by checking the Schema API on all the 
 replicas and verifying that the field has been added, though this is pretty 
 cumbersome.  Maybe it makes more sense to do this sort of thing on the 
 collection level, i.e. Schema API changes return the zk version to the 
 client.  We add an API to return the current zk version.  On a replica, if 
 the zk version is = the version the client has, the client knows that 
 replica has at least seen the schema change.  We could also provide an API to 
 do the distribution and checking across the different replicas of the 
 collection so that clients don't need ot do that themselves.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-Tests-5.x-Java7 - Build # 2142 - Still Failing

Build: https://builds.apache.org/job/Lucene-Solr-Tests-5.x-Java7/2142/

1 tests failed.
REGRESSION:  
org.apache.solr.cloud.DeleteLastCustomShardedReplicaTest.testDistribSearch

Error Message:
No live SolrServers available to handle this 
request:[https://127.0.0.1:30539/yec/na, https://127.0.0.1:30475/yec/na, 
https://127.0.0.1:30484/yec/na]

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: No live SolrServers available 
to handle this request:[https://127.0.0.1:30539/yec/na, 
https://127.0.0.1:30475/yec/na, https://127.0.0.1:30484/yec/na]
at 
__randomizedtesting.SeedInfo.seed([18C89EF967239DBA:992E10E1107CFD86]:0)
at 
org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:322)
at 
org.apache.solr.client.solrj.impl.CloudSolrServer.sendRequest(CloudSolrServer.java:880)
at 
org.apache.solr.client.solrj.impl.CloudSolrServer.requestWithRetryOnStaleState(CloudSolrServer.java:658)
at 
org.apache.solr.client.solrj.impl.CloudSolrServer.request(CloudSolrServer.java:601)
at 
org.apache.solr.cloud.DeleteLastCustomShardedReplicaTest.removeAndWaitForLastReplicaGone(DeleteLastCustomShardedReplicaTest.java:117)
at 
org.apache.solr.cloud.DeleteLastCustomShardedReplicaTest.doTest(DeleteLastCustomShardedReplicaTest.java:107)
at 
org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:869)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at

[jira] [Commented] (SOLR-6476) Create a bulk mode for schema API


[ 
https://issues.apache.org/jira/browse/SOLR-6476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14151974#comment-14151974
 ] 

Steve Rowe commented on SOLR-6476:
--

bq. The only outstanding issue AFAIK is the no:of reties. whether it should be 
finite or infinite.

(There are other outstanding issues - I'll list them in another comment after 
this one.)

I still think continuous retrying when there are competing updates is the right 
thing to do.

How about this: in SOLR-6249, [~thelabdude] added request param 
{{updateTimeoutSecs}} to fail Schema API requests unless they succeed within 
the given time.  We could add checking of this timeout to the update-retry loop 
if it's provided, but when it's not, we allow the update-retry loop to continue 
ad infinitum.

In any case, the patch on this issue needs to be changed to make bulk Schema 
API requests aware of the new {{updateTimeoutSecs}} param and perform the same 
all-replicas-in-sync check that the other Schema API methods now have.


 Create a bulk mode for schema API
 -

 Key: SOLR-6476
 URL: https://issues.apache.org/jira/browse/SOLR-6476
 Project: Solr
  Issue Type: Bug
  Components: Schema and Analysis
Reporter: Noble Paul
Assignee: Noble Paul
  Labels: managedResource
 Attachments: SOLR-6476.patch, SOLR-6476.patch, SOLR-6476.patch, 
 SOLR-6476.patch, SOLR-6476.patch, SOLR-6476.patch, SOLR-6476.patch


 The current schema API does one operation at a time and the normal usecase is 
 that users add multiple fields/fieldtypes/copyFields etc in one shot.
 example 
 {code:javascript}
 curl http://localhost:8983/solr/collection1/schema -H 
 'Content-type:application/json'  -d '{
 add-field: {
 name:sell-by,
 type:tdate,
 stored:true
 },
 add-field:{
 name:catchall,
 type:text_general,
 stored:false
 }
 }
 {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-5.x-Linux (64bit/jdk1.8.0_20) - Build # 11198 - Still Failing!

Build: http://jenkins.thetaphi.de/job/Lucene-Solr-5.x-Linux/11198/
Java: 64bit/jdk1.8.0_20 -XX:-UseCompressedOops -XX:+UseG1GC

2 tests failed.
FAILED:  
org.apache.solr.cloud.CloudExitableDirectoryReaderTest.testDistribSearch

Error Message:
no exception matching expected: 400: Request took too long during query 
expansion. Terminating request.

Stack Trace:
java.lang.AssertionError: no exception matching expected: 400: Request took too 
long during query expansion. Terminating request.
at 
__randomizedtesting.SeedInfo.seed([370D36ADAF140588:B6EBB8B5D84B65B4]:0)
at org.junit.Assert.fail(Assert.java:93)
at 
org.apache.solr.cloud.CloudExitableDirectoryReaderTest.assertFail(CloudExitableDirectoryReaderTest.java:101)
at 
org.apache.solr.cloud.CloudExitableDirectoryReaderTest.doTimeoutTests(CloudExitableDirectoryReaderTest.java:81)
at 
org.apache.solr.cloud.CloudExitableDirectoryReaderTest.doTest(CloudExitableDirectoryReaderTest.java:54)
at 
org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:869)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at

[jira] [Commented] (SOLR-6476) Create a bulk mode for schema API


[ 
https://issues.apache.org/jira/browse/SOLR-6476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14152025#comment-14152025
 ] 

Steve Rowe commented on SOLR-6476:
--

Previously mentioned outstanding issues:

# One of the tests should have a non-trivial fieldtype definition, like at 
least one analyzer.
# SchemaManager has lots of non-generic collections - I looked at a couple, and 
they could be generified - maybe they all could?
# Most of the {{add\*()}} javadocs in {{IndexSchema}} say that persistence 
always happens, but it doesn't if persist=false

New things I noticed today:

# {{SchemaManager}} has zero javadocs.  More would be good.
# AFAICT, a bulk Schema API request with a new field using a new field type in 
the same request can fail depending on the order of the specified operations, 
e.g. this will fail because newtype won't exist when {{SchemaManager}} tries 
to add newfield (caveat: untested):
\\
{code:javascript}
{
  add-field : { name:newfield, type:newtype }, 
  add-field-type : { name:newtype, class:solr.StrField }
}
{code}

 Create a bulk mode for schema API
 -

 Key: SOLR-6476
 URL: https://issues.apache.org/jira/browse/SOLR-6476
 Project: Solr
  Issue Type: Bug
  Components: Schema and Analysis
Reporter: Noble Paul
Assignee: Noble Paul
  Labels: managedResource
 Attachments: SOLR-6476.patch, SOLR-6476.patch, SOLR-6476.patch, 
 SOLR-6476.patch, SOLR-6476.patch, SOLR-6476.patch, SOLR-6476.patch


 The current schema API does one operation at a time and the normal usecase is 
 that users add multiple fields/fieldtypes/copyFields etc in one shot.
 example 
 {code:javascript}
 curl http://localhost:8983/solr/collection1/schema -H 
 'Content-type:application/json'  -d '{
 add-field: {
 name:sell-by,
 type:tdate,
 stored:true
 },
 add-field:{
 name:catchall,
 type:text_general,
 stored:false
 }
 }
 {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-5938) New DocIdSet implementation with random write access

2014-09-29 Thread Michael McCandless (JIRA)


[ 
https://issues.apache.org/jira/browse/LUCENE-5938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14152040#comment-14152040
 ] 

Michael McCandless commented on LUCENE-5938:


This is a nice change; I like simplifying MTQ's rewrite methods, to
push sparse/dense handling lower.  It's hacky now how the auto
method tries to switch from Query to FixedBitSet backed filter
depending on term/doc count...

Maybe fix word to be long instead?  (In javadocs, variable names,
etc.).  word is kind of low-level, platform dependent term ... I
found SparseFixedBitSet somewhat hard to understand :)  Maybe rename
wordCount to nonZeroLongCount or something?

approximateCardinality / linear counting algorithm is cool ...  do we
need to guard against zeroWords being 0?  I guess this is allowed with
doubles in Java?  At least add a comment explaining this corner case
works, and I think add an explicit test case that sets a bit in
every long?

Spelling: in TestSparseBitSet.copyOf, change sensible - sensitive

Maybe add some more comments around tricky parts of SparseFixedBitSet.
E.g., the different branches inside set?  And, it looks strange doing
1L  i, but in fact the JVM/processor make that 1L  (i % 64).  And
Iterator.currentOrNextDoc is scary looking... do we have enough tests
here?

I hit this test failure, which reproduces with the patch, but not on
trunk ... not sure if it's somehow related ... but the test case seems
buggy (it doesn't try to unwrap an ExecutionException to get the ACE
root cause ... yet I can't get it to fail on trunk w/ beasting):

{noformat}
NOTE: reproduce with: ant test  -Dtestcase=TestReaderClosed 
-Dtests.method=testReaderChaining -Dtests.seed=89DF4A597D3C8CB1 
-Dtests.slow=true 
-Dtests.linedocsfile=/lucenedata/hudson.enwiki.random.lines.txt 
-Dtests.locale=sk -Dtests.timezone=America/Scoresbysund 
-Dtests.file.encoding=UTF-8
NOTE: test params are: 
codec=HighCompressionCompressingStoredFields(storedFieldsFormat=CompressingStoredFieldsFormat(compressionMode=HIGH_COMPRESSION,
 chunkSize=248), 
termVectorsFormat=CompressingTermVectorsFormat(compressionMode=HIGH_COMPRESSION,
 chunkSize=248)), sim=RandomSimilarityProvider(queryNorm=true,coord=yes): {}, 
locale=sk, timezone=America/Scoresbysund
NOTE: Linux 3.13.0-32-generic amd64/Oracle Corporation 1.7.0_55 
(64-bit)/cpus=8,threads=1,free=453126896,total=518979584
NOTE: All tests run in this JVM: [TestReaderClosed]

Time: 0.485
There was 1 failure:
1) testReaderChaining(org.apache.lucene.index.TestReaderClosed)
java.lang.RuntimeException: java.util.concurrent.ExecutionException: 
org.apache.lucene.store.AlreadyClosedException: this IndexReader cannot be used 
anymore as one of its child readers was closed
at 
__randomizedtesting.SeedInfo.seed([89DF4A597D3C8CB1:1EE91FD6CAA6CE7C]:0)
at 
org.apache.lucene.search.IndexSearcher$ExecutionHelper.next(IndexSearcher.java:836)
at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:452)
at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:273)
at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:261)
at 
org.apache.lucene.index.TestReaderClosed.testReaderChaining(TestReaderClosed.java:83)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
at

[jira] [Commented] (SOLR-6476) Create a bulk mode for schema API


[ 
https://issues.apache.org/jira/browse/SOLR-6476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14152063#comment-14152063
 ] 

Noble Paul commented on SOLR-6476:
--

bq.AFAICT, a bulk Schema API request with a new field using a new field type in 
the same request can fail depending on the order of the specified operations

Yes it fails. Works as designed. This is exactly the same behavior you will see 
in an RDBMS as well

 Create a bulk mode for schema API
 -

 Key: SOLR-6476
 URL: https://issues.apache.org/jira/browse/SOLR-6476
 Project: Solr
  Issue Type: Bug
  Components: Schema and Analysis
Reporter: Noble Paul
Assignee: Noble Paul
  Labels: managedResource
 Attachments: SOLR-6476.patch, SOLR-6476.patch, SOLR-6476.patch, 
 SOLR-6476.patch, SOLR-6476.patch, SOLR-6476.patch, SOLR-6476.patch


 The current schema API does one operation at a time and the normal usecase is 
 that users add multiple fields/fieldtypes/copyFields etc in one shot.
 example 
 {code:javascript}
 curl http://localhost:8983/solr/collection1/schema -H 
 'Content-type:application/json'  -d '{
 add-field: {
 name:sell-by,
 type:tdate,
 stored:true
 },
 add-field:{
 name:catchall,
 type:text_general,
 stored:false
 }
 }
 {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-4212) Support for facet pivot query for filtered count

2014-09-29 Thread Steve Molloy (JIRA)


 [ 
https://issues.apache.org/jira/browse/SOLR-4212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Steve Molloy updated SOLR-4212:
---
Attachment: SOLR-4212-multiple-q.patch

Add support for multiple queries per field.

 Support for facet pivot query for filtered count
 

 Key: SOLR-4212
 URL: https://issues.apache.org/jira/browse/SOLR-4212
 Project: Solr
  Issue Type: Improvement
  Components: search
Affects Versions: 4.0
Reporter: Steve Molloy
 Fix For: 4.9, Trunk

 Attachments: SOLR-4212-multiple-q.patch, SOLR-4212.patch, 
 SOLR-4212.patch, SOLR-4212.patch, patch-4212.txt


 Facet pivot provide hierarchical support for computing data used to populate 
 a treemap or similar visualization. TreeMaps usually offer users extra 
 information by applying an overlay color on top of the existing square sizes 
 based on hierarchical counts. This second count is based on user choices, 
 representing, usually with gradient, the proportion of the square that fits 
 the user's choices.
 The proposition is to add a facet.pivot.q parameter that would allow to 
 specify one or more queries (per field) that would be intersected with DocSet 
 used to calculate pivot count, stored in separate qcounts list, each entry 
 keyed by the query.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Comment Edited] (SOLR-6476) Create a bulk mode for schema API


[ 
https://issues.apache.org/jira/browse/SOLR-6476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14152063#comment-14152063
 ] 

Noble Paul edited comment on SOLR-6476 at 9/29/14 6:55 PM:
---

bq.AFAICT, a bulk Schema API request with a new field using a new field type in 
the same request can fail depending on the order of the specified operations

Yes it fails. Works as designed. This is exactly the same behavior you will see 
in an RDBMS as well. Order is important


was (Author: noble.paul):
bq.AFAICT, a bulk Schema API request with a new field using a new field type in 
the same request can fail depending on the order of the specified operations

Yes it fails. Works as designed. This is exactly the same behavior you will see 
in an RDBMS as well

 Create a bulk mode for schema API
 -

 Key: SOLR-6476
 URL: https://issues.apache.org/jira/browse/SOLR-6476
 Project: Solr
  Issue Type: Bug
  Components: Schema and Analysis
Reporter: Noble Paul
Assignee: Noble Paul
  Labels: managedResource
 Attachments: SOLR-6476.patch, SOLR-6476.patch, SOLR-6476.patch, 
 SOLR-6476.patch, SOLR-6476.patch, SOLR-6476.patch, SOLR-6476.patch


 The current schema API does one operation at a time and the normal usecase is 
 that users add multiple fields/fieldtypes/copyFields etc in one shot.
 example 
 {code:javascript}
 curl http://localhost:8983/solr/collection1/schema -H 
 'Content-type:application/json'  -d '{
 add-field: {
 name:sell-by,
 type:tdate,
 stored:true
 },
 add-field:{
 name:catchall,
 type:text_general,
 stored:false
 }
 }
 {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-6476) Create a bulk mode for schema API


[ 
https://issues.apache.org/jira/browse/SOLR-6476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14152117#comment-14152117
 ] 

Noble Paul commented on SOLR-6476:
--

bq.One of the tests should have a non-trivial fieldtype definition, like at 
least one analyzer.

It is not supported yet. Pls correct me if I am wrong. does the current REST 
API support it? 

bq.Most of the add*() javadocs in IndexSchema say that persistence always 
happens, but it doesn't if persist=false

It says * @param persist to persist or not . Isn't it right?


bq.SchemaManager has zero javadocs. More would be good.

It is not a class for others to use . But , it will be added


 Create a bulk mode for schema API
 -

 Key: SOLR-6476
 URL: https://issues.apache.org/jira/browse/SOLR-6476
 Project: Solr
  Issue Type: Bug
  Components: Schema and Analysis
Reporter: Noble Paul
Assignee: Noble Paul
  Labels: managedResource
 Attachments: SOLR-6476.patch, SOLR-6476.patch, SOLR-6476.patch, 
 SOLR-6476.patch, SOLR-6476.patch, SOLR-6476.patch, SOLR-6476.patch


 The current schema API does one operation at a time and the normal usecase is 
 that users add multiple fields/fieldtypes/copyFields etc in one shot.
 example 
 {code:javascript}
 curl http://localhost:8983/solr/collection1/schema -H 
 'Content-type:application/json'  -d '{
 add-field: {
 name:sell-by,
 type:tdate,
 stored:true
 },
 add-field:{
 name:catchall,
 type:text_general,
 stored:false
 }
 }
 {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-6571) Need a znode watcher support class


[ 
https://issues.apache.org/jira/browse/SOLR-6571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14152173#comment-14152173
 ] 

Ramkumar Aiyengar commented on SOLR-6571:
-

FWIW, I still think this is an interesting idea. SolrZkClient already wraps all 
watchers, so probably that should store and fire all watchers on expiry. Most 
things (except when you use it to do things like trigger recovery) shouldn't 
really worry about expiry, having such code exposes us to rare edge cases which 
happen only when some zk misconfiguration or issue happens.

 Need a znode watcher support class
 --

 Key: SOLR-6571
 URL: https://issues.apache.org/jira/browse/SOLR-6571
 Project: Solr
  Issue Type: Improvement
  Components: SolrCloud
Reporter: Timothy Potter

 When implementing SOLR-6249, [~noble.paul] points out that 
 ZkIndexSchemaReader doesn't watch the managed schema znode correctly and 
 should use a strategy similar to what ZkStateReader does so that watchers 
 persist across zk client connection failures:
 The correct example is the constructor of ZkStateReader
zkClient = new SolrZkClient(zkServerAddress, zkClientTimeout, 
 zkClientConnectTimeout,
 // on reconnect, reload cloud info
 new OnReconnect() {
 //implement stuff here
  }
 });
 this ensures that the watchers are persisted across reconnect.
 We need a watch support class to help developers implement watchers correctly 
 instead of having some code do it correctly and other code not.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[ANNOUNCE] Apache Lucene 4.10.1 released

2014-09-29 Thread Michael McCandless

September 2014, Apache Lucene™ 4.10.1 available

The Lucene PMC is pleased to announce the release of Apache Lucene 4.10.1

Apache Lucene is a high-performance, full-featured text search engine
library written entirely in Java. It is a technology suitable for
nearly any application that requires full-text search, especially
cross-platform.

The release is available for immediate download at:

http://lucene.apache.org/core/mirrors-core-latest-redir.html

Lucene 4.10.1 includes 7 bug fixes.

See the CHANGES.txt file included with the release for a full list of
changes and further details.

Please report any feedback to the mailing lists
(http://lucene.apache.org/core/discussion.html)

Note: The Apache Software Foundation uses an extensive mirroring
network for distributing releases. It is possible that the mirror you
are using may not have replicated the release yet. If that is the
case, please try another mirror. This also goes for Maven access.

Mike McCandless

http://blog.mikemccandless.com

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[ANNOUNCE] Apache Solr 4.10.1 released

2014-09-29 Thread Michael McCandless

September 2014, Apache Solr™ 4.10.1 available

The Lucene PMC is pleased to announce the release of Apache Solr 4.10.1

Solr is the popular, blazing fast, open source NoSQL search platform
from the Apache Lucene project. Its major features include powerful
full-text search, hit highlighting, faceted search, dynamic
clustering, database integration, rich document (e.g., Word, PDF)
handling, and geospatial search. Solr is highly scalable, providing
fault tolerant distributed search and indexing, and powers the search
and navigation features of many of the world's largest internet sites.

Solr 4.10.1 is available for immediate download at:

http://lucene.apache.org/solr/mirrors-solr-latest-redir.html

Solr 4.10.1 includes 6 bug fixes, as well as Lucene 4.10.1 and its 7 bug fixes.

See the CHANGES.txt file included with the release for a full list of
changes and further details.

Please report any feedback to the mailing lists
(http://lucene.apache.org/solr/discussion.html)

Note: The Apache Software Foundation uses an extensive mirroring
network for distributing releases. It is possible that the mirror you
are using may not have replicated the release yet. If that is the
case, please try another mirror. This also goes for Maven access.

Mike McCandless

http://blog.mikemccandless.com

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-6571) Need a znode watcher support class

[
https://issues.apache.org/jira/browse/SOLR-6571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14152223#comment-14152223
]

Mark Miller commented on SOLR-6571:
---

To note, nothing I said in the issue related to this is meant to mean there are
not improvements or helper classes or documentation improvements to be made.
It's also not meant to mean this stuff is easy - it's not. You usually learn it
properly by burning your fingers off one by one. We have some code that try's
to make things better. Things like handling an expired ZooKeeper instance for
you (you can't keep using the same one). Handling the common ConnectionLoss
retry for you. Etc. There is still a lot we can probably do. Still, to properly
code for ZooKeeper you have to understand ConnectionLoss and SessionExpiration
pretty well - you can't really count on the code you are writing otherwise - or
test it properly. Perhaps there are more javadocs that can be beefed up. The
tiny corners around this stuff was not easy to glean from the ZK doc back when
I tackled it either.

Need a znode watcher support class
--

Key: SOLR-6571
URL: https://issues.apache.org/jira/browse/SOLR-6571
Project: Solr
Issue Type: Improvement
Components: SolrCloud
Reporter: Timothy Potter

When implementing SOLR-6249, [~noble.paul] points out that
ZkIndexSchemaReader doesn't watch the managed schema znode correctly and
should use a strategy similar to what ZkStateReader does so that watchers
persist across zk client connection failures:
The correct example is the constructor of ZkStateReader
zkClient = new SolrZkClient(zkServerAddress, zkClientTimeout,
zkClientConnectTimeout,
// on reconnect, reload cloud info
new OnReconnect() {
//implement stuff here
}
});
this ensures that the watchers are persisted across reconnect.
We need a watch support class to help developers implement watchers correctly
instead of having some code do it correctly and other code not.

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-6571) Need a znode watcher support class


[ 
https://issues.apache.org/jira/browse/SOLR-6571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14152232#comment-14152232
 ] 

Mark Miller commented on SOLR-6571:
---

One of my favorite subtleties was that some Watcher events don't require that 
you re-create the Watcher. That's a bid insidious given the API ;)

 Need a znode watcher support class
 --

 Key: SOLR-6571
 URL: https://issues.apache.org/jira/browse/SOLR-6571
 Project: Solr
  Issue Type: Improvement
  Components: SolrCloud
Reporter: Timothy Potter

 When implementing SOLR-6249, [~noble.paul] points out that 
 ZkIndexSchemaReader doesn't watch the managed schema znode correctly and 
 should use a strategy similar to what ZkStateReader does so that watchers 
 persist across zk client connection failures:
 The correct example is the constructor of ZkStateReader
zkClient = new SolrZkClient(zkServerAddress, zkClientTimeout, 
 zkClientConnectTimeout,
 // on reconnect, reload cloud info
 new OnReconnect() {
 //implement stuff here
  }
 });
 this ensures that the watchers are persisted across reconnect.
 We need a watch support class to help developers implement watchers correctly 
 instead of having some code do it correctly and other code not.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-NightlyTests-trunk - Build # 642 - Still Failing

Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-trunk/642/

1 tests failed.
REGRESSION:  
org.apache.solr.handler.component.DistributedFacetPivotSmallTest.testDistribSearch

Error Message:
Captured an uncaught exception in thread: Thread[id=11082, name=Thread-4297, 
state=RUNNABLE, group=TGRP-DistributedFacetPivotSmallTest]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught 
exception in thread: Thread[id=11082, name=Thread-4297, state=RUNNABLE, 
group=TGRP-DistributedFacetPivotSmallTest]
at 
__randomizedtesting.SeedInfo.seed([3DD00CB100ACFBAF:BC3682A977F39B93]:0)
Caused by: java.lang.RuntimeException: 
org.apache.solr.client.solrj.SolrServerException: Timeout occured while waiting 
response from server at: https://127.0.0.1:13614/dk_tb/bq
at __randomizedtesting.SeedInfo.seed([3DD00CB100ACFBAF]:0)
at 
org.apache.solr.BaseDistributedSearchTestCase$5.run(BaseDistributedSearchTestCase.java:580)
Caused by: org.apache.solr.client.solrj.SolrServerException: Timeout occured 
while waiting response from server at: https://127.0.0.1:13614/dk_tb/bq
at 
org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:580)
at 
org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:215)
at 
org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:211)
at 
org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.java:91)
at org.apache.solr.client.solrj.SolrServer.query(SolrServer.java:301)
at 
org.apache.solr.BaseDistributedSearchTestCase$5.run(BaseDistributedSearchTestCase.java:575)
Caused by: java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:152)
at java.net.SocketInputStream.read(SocketInputStream.java:122)
at sun.security.ssl.InputRecord.readFully(InputRecord.java:442)
at sun.security.ssl.InputRecord.read(InputRecord.java:480)
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:927)
at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:884)
at sun.security.ssl.AppInputStream.read(AppInputStream.java:102)
at 
org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:160)
at 
org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:84)
at 
org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:273)
at 
org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140)
at 
org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
at 
org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:260)
at 
org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:283)
at 
org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:251)
at 
org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:197)
at 
org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:271)
at 
org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123)
at 
org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:682)
at 
org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:486)
at 
org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:863)
at 
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
at 
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:106)
at 
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:57)
at 
org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:466)
... 5 more




Build Log:
[...truncated 12549 lines...]
   [junit4] Suite: 
org.apache.solr.handler.component.DistributedFacetPivotSmallTest
   [junit4]   2 Creating dataDir: 
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.handler.component.DistributedFacetPivotSmallTest-3DD00CB100ACFBAF-001/init-core-data-001
   [junit4]   2 2668501 T10193 oas.SolrTestCaseJ4.buildSSLConfig Randomized 
ssl (true) and clientAuth (true)
   [junit4]   2 2668502 T10193 
oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system 
property: /dk_tb/bq
   [junit4]   2 2668506 T10193 oas.SolrTestCaseJ4.setUp ###Starting 
testDistribSearch
   [junit4]   2 2668509 T10193 oejs.Server.doStart

Re: queryResultMaxDocsCached vs. queryResultWindowSize

2014-09-29 Thread Tom Burton-West

Thanks for your help Yonik and Tomas,

I had several mistaken assumptions about how caching worked which were
resolved by walking through the code in the debugger after reading your
replies.

Tom


On Fri, Sep 26, 2014 at 4:55 PM, Yonik Seeley yo...@heliosearch.com wrote:

 On Fri, Sep 26, 2014 at 4:38 PM, Tom Burton-West tburt...@umich.edu
 wrote:
  Hi Yonik,
 
  I'm still confused.
 
   suspect don't understand how paging and caching interact.  I probably
 need
  to walk through the code.  Is there a unit test that exercises
  SolrIndexSearcher.getDocListC or a good unit test to use as a base to
 write
  one?
 
 
  Part of what confuses me is whether what gets cached always starts at
 row 1
  of results.

 Yes, we always cache from the first row.
 Asking for rows 91-100 requires collecting 1-100 (and it's the latter
 we cache - ignoring deep paging).
 It's also just ids (and optionally scores) that are cached... so
 either 4 bytes or 8 bytes per document cached, depending on if you ask
 for scores back.

 queryWindowSize just rounds up the upper bound.

  I'll try to explain my confusion.
  Using the defaults in the solrconfig example:
  queryResultWindowSize20/queryResultWindowSize
  queryResultMaxDocsCached200/queryResultMaxDocsCached
 
  If I query for start=0, rows =10  Solr fetches 20 results and caches
 them.
  If I query for start =11 rows =10 Solr read rows 11-20 from cache

 Correct.

  What happens when I query for start = 21 rows= 10?
  I thought that Solr would then fetch rows 21-40 into the
 queryResultCache.
  Is this wrong?

 It will result in a cache miss and we'll collect 0-40 and cache that.

  If I query for start =195 rows =10  does Solr cache rows 195-200 but go
 to
  disk for rows over 200 (queryResultMaxDocsCached=200)?   Or does Solr
 skip
  caching altogether for rows over 200

 Probably the latter... it's an edge case so I'd have to check the code
 to know for sure if the check is pre or post rounding up.

 -Yonik
 http://heliosearch.org - native code faceting, facet functions,
 sub-facets, off-heap data

 -
 To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
 For additional commands, e-mail: dev-h...@lucene.apache.org

Re: [JENKINS] Lucene-Solr-5.x-Linux (32bit/jdk1.8.0_40-ea-b04) - Build # 11197 - Still Failing!

2014-09-29 Thread Alan Woodward

Bah, I'll fix

Alan Woodward
www.flax.co.uk


On 29 Sep 2014, at 16:07, Policeman Jenkins Server wrote:

 Build: http://jenkins.thetaphi.de/job/Lucene-Solr-5.x-Linux/11197/
 Java: 32bit/jdk1.8.0_40-ea-b04 -server -XX:+UseG1GC
 
 All tests passed
 
 Build Log:
 [...truncated 4413 lines...]
[javac] Compiling 2 source files to 
 /mnt/ssd/jenkins/workspace/Lucene-Solr-5.x-Linux/lucene/build/memory/classes/test
[javac] 
 /mnt/ssd/jenkins/workspace/Lucene-Solr-5.x-Linux/lucene/memory/src/test/org/apache/lucene/index/memory/TestMemoryIndexAgainstRAMDir.java:434:
  error: incompatible types: IndexableField cannot be converted to Field
[javac]   for (Field field : nextDoc.getFields()) {
[javac]   ^
[javac] 
 /mnt/ssd/jenkins/workspace/Lucene-Solr-5.x-Linux/lucene/memory/src/test/org/apache/lucene/index/memory/TestMemoryIndexAgainstRAMDir.java:445:
  error: cannot find symbol
[javac]   for (IndexableField field : doc.indexableFields()) {
[javac]  ^
[javac]   symbol:   method indexableFields()
[javac]   location: variable doc of type Document
[javac] Note: 
 /mnt/ssd/jenkins/workspace/Lucene-Solr-5.x-Linux/lucene/memory/src/test/org/apache/lucene/index/memory/TestMemoryIndexAgainstRAMDir.java
  uses or overrides a deprecated API.
[javac] Note: Recompile with -Xlint:deprecation for details.
[javac] 2 errors
 
 BUILD FAILED
 /mnt/ssd/jenkins/workspace/Lucene-Solr-5.x-Linux/build.xml:524: The following 
 error occurred while executing this line:
 /mnt/ssd/jenkins/workspace/Lucene-Solr-5.x-Linux/build.xml:472: The following 
 error occurred while executing this line:
 /mnt/ssd/jenkins/workspace/Lucene-Solr-5.x-Linux/build.xml:61: The following 
 error occurred while executing this line:
 /mnt/ssd/jenkins/workspace/Lucene-Solr-5.x-Linux/extra-targets.xml:39: The 
 following error occurred while executing this line:
 /mnt/ssd/jenkins/workspace/Lucene-Solr-5.x-Linux/lucene/build.xml:447: The 
 following error occurred while executing this line:
 /mnt/ssd/jenkins/workspace/Lucene-Solr-5.x-Linux/lucene/common-build.xml:2142:
  The following error occurred while executing this line:
 /mnt/ssd/jenkins/workspace/Lucene-Solr-5.x-Linux/lucene/module-build.xml:55: 
 The following error occurred while executing this line:
 /mnt/ssd/jenkins/workspace/Lucene-Solr-5.x-Linux/lucene/common-build.xml:785: 
 The following error occurred while executing this line:
 /mnt/ssd/jenkins/workspace/Lucene-Solr-5.x-Linux/lucene/common-build.xml:799: 
 The following error occurred while executing this line:
 /mnt/ssd/jenkins/workspace/Lucene-Solr-5.x-Linux/lucene/common-build.xml:1867:
  Compile failed; see the compiler error output for details.
 
 Total time: 32 minutes 32 seconds
 Build step 'Invoke Ant' marked build as failure
 [description-setter] Description set: Java: 32bit/jdk1.8.0_40-ea-b04 -server 
 -XX:+UseG1GC
 Archiving artifacts
 Recording test results
 Email was triggered for: Failure - Any
 Sending email for trigger: Failure - Any
 
 
 
 -
 To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
 For additional commands, e-mail: dev-h...@lucene.apache.org

Re: svn commit: r1628194 - /lucene/dev/branches/branch_5x/lucene/memory/src/test/org/apache/lucene/index/memory/TestMemoryIndexAgainstRAMDir.java

2014-09-29 Thread Alan Woodward

Oh, Rob already fixed it.  Thanks!  

Alan Woodward
www.flax.co.uk


On 29 Sep 2014, at 15:37, rm...@apache.org wrote:

 Author: rmuir
 Date: Mon Sep 29 14:37:55 2014
 New Revision: 1628194
 
 URL: http://svn.apache.org/r1628194
 Log:
 LUCENE-5911: fix compile
 
 Modified:

 lucene/dev/branches/branch_5x/lucene/memory/src/test/org/apache/lucene/index/memory/TestMemoryIndexAgainstRAMDir.java
 
 Modified: 
 lucene/dev/branches/branch_5x/lucene/memory/src/test/org/apache/lucene/index/memory/TestMemoryIndexAgainstRAMDir.java
 URL: 
 http://svn.apache.org/viewvc/lucene/dev/branches/branch_5x/lucene/memory/src/test/org/apache/lucene/index/memory/TestMemoryIndexAgainstRAMDir.java?rev=1628194r1=1628193r2=1628194view=diff
 ==
 --- 
 lucene/dev/branches/branch_5x/lucene/memory/src/test/org/apache/lucene/index/memory/TestMemoryIndexAgainstRAMDir.java
  (original)
 +++ 
 lucene/dev/branches/branch_5x/lucene/memory/src/test/org/apache/lucene/index/memory/TestMemoryIndexAgainstRAMDir.java
  Mon Sep 29 14:37:55 2014
 @@ -431,7 +431,7 @@ public class TestMemoryIndexAgainstRAMDi
   IndexWriter writer = new IndexWriter(dir, 
 newIndexWriterConfig(random(), mockAnalyzer));
   Document nextDoc = lineFileDocs.nextDoc();
   Document doc = new Document();
 -  for (Field field : nextDoc.getFields()) {
 +  for (IndexableField field : nextDoc.getFields()) {
 if (field.fieldType().indexed()) {
   doc.add(field);
   if (random().nextInt(3) == 0) {
 @@ -442,7 +442,7 @@ public class TestMemoryIndexAgainstRAMDi
 
   writer.addDocument(doc);
   writer.close();
 -  for (IndexableField field : doc.indexableFields()) {
 +  for (IndexableField field : doc.getFields()) {
   memory.addField(field.name(), ((Field)field).stringValue(), 
 mockAnalyzer);  
   }
   DirectoryReader competitor = DirectoryReader.open(dir);

[jira] [Commented] (SOLR-1632) Distributed IDF

2014-09-29 Thread Anshum Gupta (JIRA)


[ 
https://issues.apache.org/jira/browse/SOLR-1632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14152329#comment-14152329
 ] 

Anshum Gupta commented on SOLR-1632:


Thanks for updating the patch [~vzhovtiuk].
The tests pass now. I'm looking at the updated patch.

 Distributed IDF
 ---

 Key: SOLR-1632
 URL: https://issues.apache.org/jira/browse/SOLR-1632
 Project: Solr
  Issue Type: New Feature
  Components: search
Affects Versions: 1.5
Reporter: Andrzej Bialecki 
Assignee: Mark Miller
 Fix For: 4.9, Trunk

 Attachments: 3x_SOLR-1632_doesntwork.patch, SOLR-1632.patch, 
 SOLR-1632.patch, SOLR-1632.patch, SOLR-1632.patch, SOLR-1632.patch, 
 SOLR-1632.patch, SOLR-1632.patch, SOLR-1632.patch, SOLR-1632.patch, 
 SOLR-1632.patch, SOLR-1632.patch, SOLR-1632.patch, SOLR-1632.patch, 
 distrib-2.patch, distrib.patch


 Distributed IDF is a valuable enhancement for distributed search across 
 non-uniform shards. This issue tracks the proposed implementation of an API 
 to support this functionality in Solr.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-6476) Create a bulk mode for schema API


[ 
https://issues.apache.org/jira/browse/SOLR-6476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14152339#comment-14152339
 ] 

Steve Rowe commented on SOLR-6476:
--

{quote}
bq. One of the tests should have a non-trivial fieldtype definition, like at 
least one analyzer.

It is not supported yet. Pls correct me if I am wrong. does the current REST 
API support it?
{quote}

Yes.  See {{TestManagedSchemaFieldTypeResource}}.

{quote}
bq. Most of the add*() javadocs in IndexSchema say that persistence always 
happens, but it doesn't if persist=false

It says * @param persist to persist or not . Isn't it right?
{quote}

This is what I'm talking about: Copies this schema, adds the given fields to 
the copy, then persists the new schema.

{quote}
bq. {{SchemaManager}} has zero javadocs. More would be good.

It is not a class for others to use . But , it will be added
{quote}

Thanks.  Javadocs (or rather comments of any kind) are for maintainers too, not 
just users.  Here's an example where javadocs/comments would help a maintainer: 
{{SchemaManager.Operation.getMetaWithout()}}.  What does that thing do?  (Hint: 
the name of the method doesn't tell you.)

 Create a bulk mode for schema API
 -

 Key: SOLR-6476
 URL: https://issues.apache.org/jira/browse/SOLR-6476
 Project: Solr
  Issue Type: Bug
  Components: Schema and Analysis
Reporter: Noble Paul
Assignee: Noble Paul
  Labels: managedResource
 Attachments: SOLR-6476.patch, SOLR-6476.patch, SOLR-6476.patch, 
 SOLR-6476.patch, SOLR-6476.patch, SOLR-6476.patch, SOLR-6476.patch


 The current schema API does one operation at a time and the normal usecase is 
 that users add multiple fields/fieldtypes/copyFields etc in one shot.
 example 
 {code:javascript}
 curl http://localhost:8983/solr/collection1/schema -H 
 'Content-type:application/json'  -d '{
 add-field: {
 name:sell-by,
 type:tdate,
 stored:true
 },
 add-field:{
 name:catchall,
 type:text_general,
 stored:false
 }
 }
 {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-6476) Create a bulk mode for schema API


[ 
https://issues.apache.org/jira/browse/SOLR-6476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14152356#comment-14152356
 ] 

Steve Rowe commented on SOLR-6476:
--

{quote}
bq.  AFAICT, a bulk Schema API request with a new field using a new field type 
in the same request can fail depending on the order of the specified operations

Yes it fails. Works as designed. This is exactly the same behavior you will see 
in an RDBMS as well. Order is important
{quote}

Order is not important in {{schema.xml}}, and in plenty of other contexts.  
This order dependence will need to be explicitly documented.

 Create a bulk mode for schema API
 -

 Key: SOLR-6476
 URL: https://issues.apache.org/jira/browse/SOLR-6476
 Project: Solr
  Issue Type: Bug
  Components: Schema and Analysis
Reporter: Noble Paul
Assignee: Noble Paul
  Labels: managedResource
 Attachments: SOLR-6476.patch, SOLR-6476.patch, SOLR-6476.patch, 
 SOLR-6476.patch, SOLR-6476.patch, SOLR-6476.patch, SOLR-6476.patch


 The current schema API does one operation at a time and the normal usecase is 
 that users add multiple fields/fieldtypes/copyFields etc in one shot.
 example 
 {code:javascript}
 curl http://localhost:8983/solr/collection1/schema -H 
 'Content-type:application/json'  -d '{
 add-field: {
 name:sell-by,
 type:tdate,
 stored:true
 },
 add-field:{
 name:catchall,
 type:text_general,
 stored:false
 }
 }
 {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Created] (LUCENE-5978) don't write a norm of infinity when analyzer returns no tokens

Robert Muir created LUCENE-5978:
---

 Summary: don't write a norm of infinity when analyzer returns no 
tokens
 Key: LUCENE-5978
 URL: https://issues.apache.org/jira/browse/LUCENE-5978
 Project: Lucene - Core
  Issue Type: Bug
Reporter: Robert Muir


When a document doesn't have the field, we fill with zero. when a segment 
doesn't have the field, we also fill with zero.

however, when the analyzer doesn't return any terms for the field, we still 
call similarity.computeNorm(0)... with the default similarity this encodes 
infinity... -1

in such a case, it doesnt really matter what the norm is, since it has no 
terms. But its more efficient for e.g. compression if we consistently use zero.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-5978) don't write a norm of infinity when analyzer returns no tokens

2014-09-29 Thread Michael McCandless (JIRA)


[ 
https://issues.apache.org/jira/browse/LUCENE-5978?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14152408#comment-14152408
 ] 

Michael McCandless commented on LUCENE-5978:


+1

 don't write a norm of infinity when analyzer returns no tokens
 --

 Key: LUCENE-5978
 URL: https://issues.apache.org/jira/browse/LUCENE-5978
 Project: Lucene - Core
  Issue Type: Bug
Reporter: Robert Muir

 When a document doesn't have the field, we fill with zero. when a segment 
 doesn't have the field, we also fill with zero.
 however, when the analyzer doesn't return any terms for the field, we still 
 call similarity.computeNorm(0)... with the default similarity this encodes 
 infinity... -1
 in such a case, it doesnt really matter what the norm is, since it has no 
 terms. But its more efficient for e.g. compression if we consistently use 
 zero.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-5978) don't write a norm of infinity when analyzer returns no tokens


[ 
https://issues.apache.org/jira/browse/LUCENE-5978?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14152412#comment-14152412
 ] 

Adrien Grand commented on LUCENE-5978:
--

+1 as well

 don't write a norm of infinity when analyzer returns no tokens
 --

 Key: LUCENE-5978
 URL: https://issues.apache.org/jira/browse/LUCENE-5978
 Project: Lucene - Core
  Issue Type: Bug
Reporter: Robert Muir

 When a document doesn't have the field, we fill with zero. when a segment 
 doesn't have the field, we also fill with zero.
 however, when the analyzer doesn't return any terms for the field, we still 
 call similarity.computeNorm(0)... with the default similarity this encodes 
 infinity... -1
 in such a case, it doesnt really matter what the norm is, since it has no 
 terms. But its more efficient for e.g. compression if we consistently use 
 zero.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-5814) CoreContainer reports incorrect missleading path for solrconfig.xml when there are loading problems

2014-09-29 Thread JIRA


 [ 
https://issues.apache.org/jira/browse/SOLR-5814?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jan Høydahl updated SOLR-5814:
--
Fix Version/s: 4.10.1

 CoreContainer reports incorrect  missleading path for solrconfig.xml when 
 there are loading problems
 -

 Key: SOLR-5814
 URL: https://issues.apache.org/jira/browse/SOLR-5814
 Project: Solr
  Issue Type: Bug
Reporter: Hoss Man
Assignee: Hoss Man
 Fix For: 4.10.1, 5.0, Trunk

 Attachments: SOLR-5814.patch, SOLR-5814.patch


 The error messages thrown by CoreContainer when there is a problem loading 
 solrconfig.xml refer to the wrong path (leaves out conf/).
 This is missleading users (who may not notice the root cause) into thinking 
 they need to move their solrconfig.xml from 
 {{collection_name/conf/solrconfig.xml}} to {{collection_name/solrconfig.xml}} 
 at which point they still get the same error message because solr still can't 
 find the file in the conf dir



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (LUCENE-5978) don't write a norm of infinity when analyzer returns no tokens


 [ 
https://issues.apache.org/jira/browse/LUCENE-5978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Muir updated LUCENE-5978:

Attachment: LUCENE-5978.patch

here's a patch with a test.

 don't write a norm of infinity when analyzer returns no tokens
 --

 Key: LUCENE-5978
 URL: https://issues.apache.org/jira/browse/LUCENE-5978
 Project: Lucene - Core
  Issue Type: Bug
Reporter: Robert Muir
 Attachments: LUCENE-5978.patch


 When a document doesn't have the field, we fill with zero. when a segment 
 doesn't have the field, we also fill with zero.
 however, when the analyzer doesn't return any terms for the field, we still 
 call similarity.computeNorm(0)... with the default similarity this encodes 
 infinity... -1
 in such a case, it doesnt really matter what the norm is, since it has no 
 terms. But its more efficient for e.g. compression if we consistently use 
 zero.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (LUCENE-5938) New DocIdSet implementation with random write access

[
https://issues.apache.org/jira/browse/LUCENE-5938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Adrien Grand updated LUCENE-5938:
-
Attachment: LUCENE-5938.patch

Thanks for the review Mike, here is a new patch that tries to address your
concerns.

bq. Maybe fix word to be long instead

Done.

bq. do we need to guard against zeroWords being 0?

I added a comment as well as a test as you suggested.

{quote}
Maybe add some more comments around tricky parts of SparseFixedBitSet.
E.g., the different branches inside set? And, it looks strange doing
1L i, but in fact the JVM/processor make that 1L (i % 64). And
Iterator.currentOrNextDoc is scary looking... do we have enough tests
here?
{quote}

I added more comments, hopefully they make sense. Please let me know if there
are still things that are not clear. currentOrNextDoc is a bit complicated
because of the different cases that need to be handled but the structure is
actually quite simple so at least get and set should be easy to understand. I
extracted the insertion of a new long to a separate method, this should make
set easier to read?

Regarding the shift, indeed it relies on the fact that shifts are mod 64
(FixedBitSet does the same). I added some comments about it.

Regarding the tests, BaseDocIdSetTestCase.testAgainstBitSet tests various
densities and assertEquals checks nextDoc(), docId(), interleaved calls to
nextDoc() and advance(), and that the oal.util.Bits representation is
consistent with the iterator. I think that is good?

bq. I hit this test failure, which reproduces with the patch

I dug that one, and the reason is that the searcher is created with threads, so
the exception is indeed wrapped into an ExecutionException, which is in turn
wrapped into a RuntimeException to by-pass the fact that ExecutionException is
checked. It doesn't reproduce on trunk because the default rewrite method reads
data from the index in MultiTermQuery.rewrite (collectTerms) which does not run
in a thread pool. You can reproduce the issue on trunk by setting the rewrite
method of the term range query to {{CONSTANT_SCORE_FILTER_REWRITE}}. I fixed
the test in the patch to walk down the causes of the exception that is thrown.

New DocIdSet implementation with random write access

Key: LUCENE-5938
URL: https://issues.apache.org/jira/browse/LUCENE-5938
Project: Lucene - Core
Issue Type: Improvement
Reporter: Adrien Grand
Assignee: Adrien Grand
Attachments: LUCENE-5938.patch, LUCENE-5938.patch, LUCENE-5938.patch,
LUCENE-5938.patch, LUCENE-5938.patch, low_freq.tasks

We have a great cost API that is supposed to help make decisions about how to
best execute queries. However, due to the fact that several of our filter
implementations (eg. TermsFilter and BooleanFilter) return FixedBitSets,
either we use the cost API and make bad decisions, or need to fall back to
heuristics which are not as good such as
RandomAccessFilterStrategy.useRandomAccess which decides that random access
should be used if the first doc in the set is less than 100.
On the other hand, we also have some nice compressed and cacheable DocIdSet
implementation but we cannot make use of them because TermsFilter requires a
DocIdSet that has random write access, and FixedBitSet is the only DocIdSet
that we have that supports random access.
I think it would be nice to replace FixedBitSet in those filters with another
DocIdSet that would also support random write access but would have a better
cost?

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-5978) don't write a norm of infinity when analyzer returns no tokens


[ 
https://issues.apache.org/jira/browse/LUCENE-5978?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14152473#comment-14152473
 ] 

Adrien Grand commented on LUCENE-5978:
--

+1

 don't write a norm of infinity when analyzer returns no tokens
 --

 Key: LUCENE-5978
 URL: https://issues.apache.org/jira/browse/LUCENE-5978
 Project: Lucene - Core
  Issue Type: Bug
Reporter: Robert Muir
 Attachments: LUCENE-5978.patch


 When a document doesn't have the field, we fill with zero. when a segment 
 doesn't have the field, we also fill with zero.
 however, when the analyzer doesn't return any terms for the field, we still 
 call similarity.computeNorm(0)... with the default similarity this encodes 
 infinity... -1
 in such a case, it doesnt really matter what the norm is, since it has no 
 terms. But its more efficient for e.g. compression if we consistently use 
 zero.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Resolved] (SOLR-6278) add admin/collections?action=DELETEREPLICAcore=... support, make collection=... and shard=... parameters optional

2014-09-29 Thread Erick Erickson (JIRA)


 [ 
https://issues.apache.org/jira/browse/SOLR-6278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Erick Erickson resolved SOLR-6278.
--
Resolution: Won't Fix

See discussion. There's still room for being able to do node-type operations, 
think of a physical machine going down and you want to remove all traces of it 
from the cluster state, but that's for another ticket.

 add admin/collections?action=DELETEREPLICAcore=... support, make 
 collection=... and shard=... parameters optional
 --

 Key: SOLR-6278
 URL: https://issues.apache.org/jira/browse/SOLR-6278
 Project: Solr
  Issue Type: Improvement
Reporter: Christine Poerschke
Assignee: Erick Erickson
 Attachments: SOLR-6278.patch


 To add {{core=...}} as an alternative to {{replica=...}} way of identifying 
 what is to be deleted, {{collection=...}} and {{shard=...}} to be optional 
 provided the other parameters uniquely identify exactly one deletion target.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-6512) Add a collections API call to add/delete arbitrary properties to a specific replica

2014-09-29 Thread Erick Erickson (JIRA)

[
https://issues.apache.org/jira/browse/SOLR-6512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Erick Erickson updated SOLR-6512:
-
Summary: Add a collections API call to add/delete arbitrary properties to a
specific replica (was: Add a collections API call to add/delete arbitrary
roles to a specific replica)

Add a collections API call to add/delete arbitrary properties to a specific
replica
---

Key: SOLR-6512
URL: https://issues.apache.org/jira/browse/SOLR-6512
Project: Solr
Issue Type: Improvement
Reporter: Erick Erickson
Assignee: Erick Erickson
Attachments: SOLR-6512.patch, SOLR-6512.patch

This is a sub-task for SOLR-6491, but seems generally useful.
Since this is in support of the preferredLeader functionality, I've run
into some considerations that I wanted some feedback on how to handle.
preferredLeader has the restriction that there should only be one per
slice, so setting this for a particular node means removing the property for
all the other replicas on the slice. Not a problem to do, my question is more
whether this is something reasonable to enforce on an arbitrary property
based on what that property is? Perfectly do-able, but semantically
challenged. Currently, this is never a node with preferedLeader set to
false, it is forcibly removed from other nodes in the slice when this
property is assigned.
The problem here is that there's nothing about assigning an arbitrary
property to a node that would reasonably imply this kind of behavior. One
could always control this with secondary flags on the command, e.g.
shardExclusive=true|false for instance, perhaps with safety checks in for
known one-per-shard properties like preferredLeader.
preferredLeader seems to fit more naturally into a role, but currently
ADDROLE and DELTEROLE have nothing to do with the notion of setting a role
for a particular node relative to a collection/shard. Easy enough to add, but
enforcing the only one node per slice may have this role rule there is
similarly arbitrary and overloads the ADDROLE/DELETEROLE in a way that seems
equally confusing. Plus, checking whether the required collection/shard/node
params are present becomes based on the value of the property being set,
which is all equally arbitrary.
The other interesting thing is that setting an arbitrary property on a node
would allow one to mess things up royally by, say, changing properties like
core, or base_url or node_name at will. Actually this is potentially
useful, but very, very dangerous and I'm not particularly interested in
supporting it ;). I suppose we could require a prefix, say the only settable
properties are property.whatever.
We could also add something specific to nodes, something like
ADDREPLICAROLE/DELETEREPLICAROLE, perhaps with sub-params like
onlyOneAllowedPerShard, but this gets messy and relies on the users doing
the right thing. I prefer enforcing rules like this based on the role I
think. Or at least enforcing these kinds of requirements on the
preferredLeader role if we go that way.
What are people's thoughts here? I think I'm tending towards the
ADDREPLICAROLE/DELETEREPLICAROLE way of doing this, but it's not set in
stone. I have code locally for arbitrary properties that I can modify for the
role bits.
So, if I'm going to summarize the points I'd like feedback on:
1 Is setting arbitrary properties on a node desirable? If so, should we
require a prefix like property to prevent resetting values SolrCloud
depends on?
2 Is it better to piggyback on ADDROLE/DELETEROLE? Personally I'm not in
favor of this one. Too messy with requiring additional parameters to work
right in this case
3 Is the best option to create new collections API calls for
ADDREPLICAROLE/DELETEREPLICAROLE that
3.1 require collection/slice/node parameters
3.2 enforces the onlyOnePerShard rule for certain known roles
3.3 v1 allows users to specify arbitrary roles something like
onlyOnePerShard as an optional T|F parameter, otherwise is totally open.
-or-
3.3 v2 No support other than preferredLeader, only roles that are
pre-defined are allowed, in which case the onlyOnePerShard is implicit in
the role.

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Review Request 26161: SOLR-6512: Add a collections API call to add/delete arbitrary properties to a specific replica

2014-09-29 Thread Erick Erickson


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/26161/
---

Review request for lucene.


Repository: lucene


Description
---

This allows us to add properties to a replica, it takes a 
collection/slice/replica triplet along with a property.

There is exactly one known sliceUnique property at present, preferredLeader. 
Arbitrary properties can be assigned a sliceUnique attribute though.

If sliceUnique==false, then there are no restrictions on how many replicas can 
have an property/value pair.


Diffs
-

  trunk/solr/core/src/java/org/apache/solr/cloud/Overseer.java 1628219 
  
trunk/solr/core/src/java/org/apache/solr/cloud/OverseerCollectionProcessor.java 
1628219 
  
trunk/solr/core/src/java/org/apache/solr/handler/admin/CollectionsHandler.java 
1628219 
  trunk/solr/core/src/test/org/apache/solr/cloud/DeleteReplicaTest.java 1628219 
  trunk/solr/core/src/test/org/apache/solr/cloud/TestCollectionAPI.java 1628219 
  trunk/solr/solrj/src/java/org/apache/solr/common/cloud/ZkStateReader.java 
1628219 
  trunk/solr/solrj/src/java/org/apache/solr/common/params/CollectionParams.java 
1628219 

Diff: https://reviews.apache.org/r/26161/diff/


Testing
---

Unit tests are in place.


Thanks,

Erick Erickson

[jira] [Updated] (SOLR-6512) Add a collections API call to add/delete arbitrary properties to a specific replica

2014-09-29 Thread Erick Erickson (JIRA)

[
https://issues.apache.org/jira/browse/SOLR-6512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Erick Erickson updated SOLR-6512:
-
Attachment: SOLR-6512.patch

Reworked patch for arbitrary property assignment rather than roles. Also on
review board, see: https://reviews.apache.org/r/26161/

the preferredLeader role is special in that there is a list of known
properties that we enforce the only one per slice rule. This list may be
added to in the future if necessary.

There is an optional parameter sliceUnique that can be specified with
arbitrary properties to enforce this rule without changing the code.
sliceUnique defaults to false, in which case properties can be added however
desired.

I'll probably commit this Wednesday unless there are objections.

Add a collections API call to add/delete arbitrary properties to a specific
replica
---

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[JENKINS] Lucene-Solr-5.x-Linux (32bit/jdk1.7.0_67) - Build # 11200 - Failure!

Build: http://jenkins.thetaphi.de/job/Lucene-Solr-5.x-Linux/11200/
Java: 32bit/jdk1.7.0_67 -server -XX:+UseConcMarkSweepGC

6 tests failed.
REGRESSION:  
org.apache.solr.client.solrj.SolrExampleBinaryTest.testChildDoctransformer

Error Message:
Expected mime type application/octet-stream but got text/html. html head 
meta http-equiv=Content-Type content=text/html;charset=ISO-8859-1/ 
titleError 500 Server Error/title /head body h2HTTP ERROR: 500/h2 
pProblem accessing /solr/collection1/select. Reason: preServer 
Error/pre/p hr /ismallPowered by Jetty:///small/i 












/body /html 

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Expected 
mime type application/octet-stream but got text/html. html
head
meta http-equiv=Content-Type content=text/html;charset=ISO-8859-1/
titleError 500 Server Error/title
/head
body
h2HTTP ERROR: 500/h2
pProblem accessing /solr/collection1/select. Reason:
preServer Error/pre/p
hr /ismallPowered by Jetty:///small/i




















/body
/html

at 
__randomizedtesting.SeedInfo.seed([DBB6A0751A5A2765:A86CBFEF96425063]:0)
at 
org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:530)
at 
org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:215)
at 
org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:211)
at 
org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.java:91)
at org.apache.solr.client.solrj.SolrServer.query(SolrServer.java:301)
at 
org.apache.solr.client.solrj.SolrExampleTests.testChildDoctransformer(SolrExampleTests.java:1373)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at

[JENKINS] Lucene-Solr-NightlyTests-5.x - Build # 636 - Still Failing

Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-5.x/636/

1 tests failed.
FAILED:  org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testDistribSearch

Error Message:
Error CREATEing SolrCore 'halfcollection_shard1_replica1': Unable to create 
core [halfcollection_shard1_replica1] Caused by: null

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Error 
CREATEing SolrCore 'halfcollection_shard1_replica1': Unable to create core 
[halfcollection_shard1_replica1] Caused by: null
at 
org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:568)
at 
org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:215)
at 
org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:211)
at 
org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testErrorHandling(CollectionsAPIDistributedZkTest.java:583)
at 
org.apache.solr.cloud.CollectionsAPIDistributedZkTest.doTest(CollectionsAPIDistributedZkTest.java:205)
at 
org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:869)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at

[JENKINS] Lucene-Solr-5.x-Linux (32bit/jdk1.8.0_40-ea-b04) - Build # 11201 - Still Failing!

Build: http://jenkins.thetaphi.de/job/Lucene-Solr-5.x-Linux/11201/
Java: 32bit/jdk1.8.0_40-ea-b04 -server -XX:+UseG1GC

2 tests failed.
REGRESSION:  
org.apache.solr.cloud.CloudExitableDirectoryReaderTest.testDistribSearch

Error Message:
no exception matching expected: 400: Request took too long during query 
expansion. Terminating request.

Stack Trace:
java.lang.AssertionError: no exception matching expected: 400: Request took too 
long during query expansion. Terminating request.
at 
__randomizedtesting.SeedInfo.seed([7178A0348F4483B2:F09E2E2CF81BE38E]:0)
at org.junit.Assert.fail(Assert.java:93)
at 
org.apache.solr.cloud.CloudExitableDirectoryReaderTest.assertFail(CloudExitableDirectoryReaderTest.java:101)
at 
org.apache.solr.cloud.CloudExitableDirectoryReaderTest.doTimeoutTests(CloudExitableDirectoryReaderTest.java:81)
at 
org.apache.solr.cloud.CloudExitableDirectoryReaderTest.doTest(CloudExitableDirectoryReaderTest.java:54)
at 
org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:869)
at sun.reflect.GeneratedMethodAccessor37.invoke(Unknown Source)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at

[JENKINS-MAVEN] Lucene-Solr-Maven-5.x #717: POMs out of sync

Build: https://builds.apache.org/job/Lucene-Solr-Maven-5.x/717/

2 tests failed.
REGRESSION:  
org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testDistribSearch

Error Message:
Error CREATEing SolrCore 'halfcollection_shard1_replica1': Unable to create 
core [halfcollection_shard1_replica1] Caused by: Could not get shard id for 
core: halfcollection_shard1_replica1

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Error 
CREATEing SolrCore 'halfcollection_shard1_replica1': Unable to create core 
[halfcollection_shard1_replica1] Caused by: Could not get shard id for core: 
halfcollection_shard1_replica1
at 
__randomizedtesting.SeedInfo.seed([52360C9922C634E6:D3D08281559954DA]:0)
at 
org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:568)
at 
org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:215)
at 
org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:211)
at 
org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testErrorHandling(CollectionsAPIDistributedZkTest.java:583)
at 
org.apache.solr.cloud.CollectionsAPIDistributedZkTest.doTest(CollectionsAPIDistributedZkTest.java:205)


REGRESSION:  
org.apache.solr.cloud.DeleteLastCustomShardedReplicaTest.testDistribSearch

Error Message:
No live SolrServers available to handle this request:[https://127.0.0.1:62006, 
https://127.0.0.1:11553, https://127.0.0.1:60126]

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: No live SolrServers available 
to handle this request:[https://127.0.0.1:62006, https://127.0.0.1:11553, 
https://127.0.0.1:60126]
at 
org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:568)
at 
org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:215)
at 
org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:211)
at 
org.apache.solr.client.solrj.impl.LBHttpSolrServer.doRequest(LBHttpSolrServer.java:343)
at 
org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:304)
at 
org.apache.solr.client.solrj.impl.CloudSolrServer.sendRequest(CloudSolrServer.java:880)
at 
org.apache.solr.client.solrj.impl.CloudSolrServer.requestWithRetryOnStaleState(CloudSolrServer.java:658)
at 
org.apache.solr.client.solrj.impl.CloudSolrServer.request(CloudSolrServer.java:601)
at 
org.apache.solr.cloud.DeleteLastCustomShardedReplicaTest.removeAndWaitForLastReplicaGone(DeleteLastCustomShardedReplicaTest.java:117)
at 
org.apache.solr.cloud.DeleteLastCustomShardedReplicaTest.doTest(DeleteLastCustomShardedReplicaTest.java:107)




Build Log:
[...truncated 52973 lines...]
BUILD FAILED
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Maven-5.x/build.xml:547: 
The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Maven-5.x/build.xml:199: 
The following error occurred while executing this line:
: Java returned: 1

Total time: 240 minutes 24 seconds
Build step 'Invoke Ant' marked build as failure
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-trunk-Linux (32bit/jdk1.9.0-ea-b28) - Build # 11347 - Still Failing!

Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/11347/
Java: 32bit/jdk1.9.0-ea-b28 -server -XX:+UseConcMarkSweepGC

1 tests failed.
FAILED:  org.apache.solr.cloud.HttpPartitionTest.testDistribSearch

Error Message:
Doc with id=2 not found in 
http://127.0.0.1:37774/nm_uc/c8n_1x2_leader_session_loss due to: Path not 
found: /id; rsp={doc=null}

Stack Trace:
java.lang.AssertionError: Doc with id=2 not found in 
http://127.0.0.1:37774/nm_uc/c8n_1x2_leader_session_loss due to: Path not 
found: /id; rsp={doc=null}
at 
__randomizedtesting.SeedInfo.seed([FC05035FA51DFD06:7DE38D47D2429D3A]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.assertTrue(Assert.java:43)
at 
org.apache.solr.cloud.HttpPartitionTest.assertDocExists(HttpPartitionTest.java:437)
at 
org.apache.solr.cloud.HttpPartitionTest.testLeaderZkSessionLoss(HttpPartitionTest.java:324)
at 
org.apache.solr.cloud.HttpPartitionTest.doTest(HttpPartitionTest.java:119)
at 
org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:869)
at sun.reflect.GeneratedMethodAccessor36.invoke(Unknown Source)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:484)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at

[JENKINS] Lucene-Solr-trunk-Windows (64bit/jdk1.8.0_40-ea-b04) - Build # 4343 - Still Failing!

Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Windows/4343/
Java: 64bit/jdk1.8.0_40-ea-b04 -XX:+UseCompressedOops -XX:+UseG1GC

2 tests failed.
REGRESSION:  org.apache.solr.cloud.HttpPartitionTest.testDistribSearch

Error Message:
Send doc 2 to old leader core_node2 should have failed! ClusterState: 
DocCollection(c8n_1x2_leader_session_loss)={   replicationFactor:2,   
shards:{shard1:{   range:8000-7fff,   state:active, 
  replicas:{ core_node1:{   
core:c8n_1x2_leader_session_loss_shard1_replica1,   
base_url:http://127.0.0.1:57338/iw_hj;,   
node_name:127.0.0.1:57338_iw_hj,   state:active,   
leader:true}, core_node2:{   
core:c8n_1x2_leader_session_loss_shard1_replica2,   
base_url:http://127.0.0.1:57322/iw_hj;,   
node_name:127.0.0.1:57322_iw_hj,   state:down,   
router:{name:compositeId},   maxShardsPerNode:1,   
autoAddReplicas:false}

Stack Trace:
java.lang.AssertionError: Send doc 2 to old leader core_node2 should have 
failed! ClusterState: DocCollection(c8n_1x2_leader_session_loss)={
  replicationFactor:2,
  shards:{shard1:{
  range:8000-7fff,
  state:active,
  replicas:{
core_node1:{
  core:c8n_1x2_leader_session_loss_shard1_replica1,
  base_url:http://127.0.0.1:57338/iw_hj;,
  node_name:127.0.0.1:57338_iw_hj,
  state:active,
  leader:true},
core_node2:{
  core:c8n_1x2_leader_session_loss_shard1_replica2,
  base_url:http://127.0.0.1:57322/iw_hj;,
  node_name:127.0.0.1:57322_iw_hj,
  state:down,
  router:{name:compositeId},
  maxShardsPerNode:1,
  autoAddReplicas:false}
at 
__randomizedtesting.SeedInfo.seed([299406B9921B386F:A87288A1E5445853]:0)
at org.junit.Assert.fail(Assert.java:93)
at 
org.apache.solr.cloud.HttpPartitionTest.testLeaderZkSessionLoss(HttpPartitionTest.java:329)
at 
org.apache.solr.cloud.HttpPartitionTest.doTest(HttpPartitionTest.java:119)
at 
org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:869)
at sun.reflect.GeneratedMethodAccessor41.invoke(Unknown Source)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at

[jira] [Updated] (LUCENE-5977) IW should safeguard against token streams returning invalid offsets for multi-valued fields


 [ 
https://issues.apache.org/jira/browse/LUCENE-5977?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Dawid Weiss updated LUCENE-5977:

Description: 
We have a custom token stream that emits information about offsets of each 
token. My (wrong) assumption was that for multi-valued fields a token stream's 
offset information is magically shifted, much like this is the case with 
positions. It's not the case -- offsets should be increasing and monotonic 
across all instances of a field, even if it has custom token streams. So, 
something like this:

{code}
doc.add(new Field(field-foo, new CannedTokenStream(token(bar, 1, 
150, 160)), ftype));
doc.add(new Field(field-foo, new CannedTokenStream(token(bar, 1,  
50,  60)), ftype));
{code}

where the token function is defined as:

{code}
token(String image, int positionIncrement, int startOffset, int endOffset)
{code}

will result in either a cryptic assertion thrown from IW:

{code}
Exception in thread main java.lang.AssertionError
at 
org.apache.lucene.index.FreqProxTermsWriterPerField.writeOffsets(FreqProxTermsWriterPerField.java:99)
{code}

or nothing (or a codec error) if run without assertions.

Obviously returning non-shifted offsets from subsequent token streams makes 
little sense but I wonder if it could be made more explicit (or asserted) that 
offsets need to be increasing between multiple-values. The minimum is to add 
some documentation to OffsetAttribute. I don't know if offsets should be 
shifted automatically, as it's the case with positions -- this would change the 
semantics of existing tokenizers and filters which implement such shifting 
internally already.

  was:
We have a custom token stream that emits information about offsets of each 
token. My (wrong) assumption was that for multi-valued fields a token stream's 
offset information is magically shifted, much like this is the case with 
positions. It's not the case -- offsets should be increasing and monotonic 
across all instances of a field, even if it has a custom token streams. So, 
something like this:

{code}
doc.add(new Field(field-foo, new CannedTokenStream(token(bar, 1, 
150, 160)), ftype));
doc.add(new Field(field-foo, new CannedTokenStream(token(bar, 1,  
50,  60)), ftype));
{code}

where the token function is defined as:

{code}
token(String image, int positionIncrement, int startOffset, int endOffset)
{code}

will result in either a cryptic assertion thrown from IW:

{code}
Exception in thread main java.lang.AssertionError
at 
org.apache.lucene.index.FreqProxTermsWriterPerField.writeOffsets(FreqProxTermsWriterPerField.java:99)
{code}

or nothing (or a codec error) if run without assertions.

Obviously returning non-shifted offsets from subsequent token streams makes 
little sense but I wonder if it could be made more explicit (or asserted) that 
offsets need to be increasing between multiple-values. The minimum is to add 
some documentation to OffsetAttribute. I don't know if offsets should be 
shifted automatically, as it's the case with positions -- this would change the 
semantics of existing tokenizers and filters which implement such shifting 
internally already.


 IW should safeguard against token streams returning invalid offsets for 
 multi-valued fields
 ---

 Key: LUCENE-5977
 URL: https://issues.apache.org/jira/browse/LUCENE-5977
 Project: Lucene - Core
  Issue Type: Improvement
Affects Versions: 4.9, 4.9.1, 4.10, 4.10.1
Reporter: Dawid Weiss
Priority: Minor

 We have a custom token stream that emits information about offsets of each 
 token. My (wrong) assumption was that for multi-valued fields a token 
 stream's offset information is magically shifted, much like this is the case 
 with positions. It's not the case -- offsets should be increasing and 
 monotonic across all instances of a field, even if it has custom token 
 streams. So, something like this:
 {code}
 doc.add(new Field(field-foo, new CannedTokenStream(token(bar, 1, 
 150, 160)), ftype));
 doc.add(new Field(field-foo, new CannedTokenStream(token(bar, 1,  
 50,  60)), ftype));
 {code}
 where the token function is defined as:
 {code}
 token(String image, int positionIncrement, int startOffset, int endOffset)
 {code}
 will result in either a cryptic assertion thrown from IW:
 {code}
 Exception in thread main java.lang.AssertionError
   at 
 org.apache.lucene.index.FreqProxTermsWriterPerField.writeOffsets(FreqProxTermsWriterPerField.java:99)
 {code}
 or nothing (or a codec error) if run without assertions.
 Obviously returning non-shifted offsets from subsequent token streams makes 
 little sense but I wonder if it could be made more explicit (or asserted) 
 that offsets need to be increasing between

[jira] [Created] (LUCENE-5977) IW should safeguard against token streams returning invalid offsets for multi-valued fields

Dawid Weiss created LUCENE-5977:
---

 Summary: IW should safeguard against token streams returning 
invalid offsets for multi-valued fields
 Key: LUCENE-5977
 URL: https://issues.apache.org/jira/browse/LUCENE-5977
 Project: Lucene - Core
  Issue Type: Improvement
Affects Versions: 4.10.1, 4.10, 4.9.1, 4.9
Reporter: Dawid Weiss
Priority: Minor


We have a custom token stream that emits information about offsets of each 
token. My (wrong) assumption was that for multi-valued fields a token stream's 
offset information is magically shifted, much like this is the case with 
positions. It's not the case -- offsets should be increasing and monotonic 
across all instances of a field, even if it has a custom token streams. So, 
something like this:

{code}
doc.add(new Field(field-foo, new CannedTokenStream(token(bar, 1, 
150, 160)), ftype));
doc.add(new Field(field-foo, new CannedTokenStream(token(bar, 1,  
50,  60)), ftype));
{code}

where the token function is defined as:

{code}
token(String image, int positionIncrement, int startOffset, int endOffset)
{code}

will result in either a cryptic assertion thrown from IW:

{code}
Exception in thread main java.lang.AssertionError
at 
org.apache.lucene.index.FreqProxTermsWriterPerField.writeOffsets(FreqProxTermsWriterPerField.java:99)
{code}

or nothing (or a codec error) if run without assertions.

Obviously returning non-shifted offsets from subsequent token streams makes 
little sense but I wonder if it could be made more explicit (or asserted) that 
offsets need to be increasing between multiple-values. The minimum is to add 
some documentation to OffsetAttribute. I don't know if offsets should be 
shifted automatically, as it's the case with positions -- this would change the 
semantics of existing tokenizers and filters which implement such shifting 
internally already.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-5977) IW should safeguard against token streams returning invalid offsets for multi-valued fields


[ 
https://issues.apache.org/jira/browse/LUCENE-5977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14151567#comment-14151567
 ] 

Dawid Weiss commented on LUCENE-5977:
-

A full example which shows the problem. Run it with -ea -- you'll get the 
assertion. Run it without assertions and it'll pass.

{code}
import org.apache.lucene.analysis.Token;
import org.apache.lucene.analysis.core.WhitespaceAnalyzer;
import org.apache.lucene.codecs.Codec;
import org.apache.lucene.codecs.simpletext.SimpleTextCodec;
import org.apache.lucene.document.Document;
import org.apache.lucene.document.Field;
import org.apache.lucene.document.FieldType;
import org.apache.lucene.index.FieldInfo.IndexOptions;
import org.apache.lucene.index.IndexWriter;
import org.apache.lucene.index.IndexWriterConfig;
import org.apache.lucene.store.Directory;
import org.apache.lucene.store.RAMDirectory;
import org.apache.lucene.util.Version;
import org.apache.lucene.analysis.CannedTokenStream;

public class OffsetIndexingBug {
  
  public static void main(String[] args) throws Exception {
Codec.setDefault(new SimpleTextCodec());
Version version = Version.LUCENE_CURRENT;
IndexWriterConfig conf = new IndexWriterConfig(version, new 
WhitespaceAnalyzer(version));
conf.setUseCompoundFile(false);
try (Directory d = new RAMDirectory()) {
  try (IndexWriter iw = new IndexWriter(d, conf)) {
iw.deleteAll();
iw.commit();

Document doc = new Document();

FieldType ftype = new FieldType();
ftype.setIndexed(true);
ftype.setStored(false);
ftype.setOmitNorms(true);

ftype.setStoreTermVectors(true);
ftype.setStoreTermVectorPositions(true);
ftype.setStoreTermVectorOffsets(true);

ftype.setTokenized(true);

ftype.setIndexOptions(IndexOptions.DOCS_AND_FREQS_AND_POSITIONS_AND_OFFSETS);
ftype.freeze();

// note use's offset is negative with respect to the first field 
value. 
doc.add(new Field(field-foo, new CannedTokenStream(token(use, 1, 
150, 160)), ftype));
doc.add(new Field(field-foo, new CannedTokenStream(token(use, 1,  
50,  60)), ftype));
iw.addDocument(doc);
  }
}
  }

  private static Token token(String image, int positionIncrement, int soffset, 
int eoffset) {
Token t = new Token();
t.setPositionIncrement(positionIncrement);
t.setOffset(soffset, eoffset);
t.append(image);
return t;
  }
}
{code}

 IW should safeguard against token streams returning invalid offsets for 
 multi-valued fields
 ---

 Key: LUCENE-5977
 URL: https://issues.apache.org/jira/browse/LUCENE-5977
 Project: Lucene - Core
  Issue Type: Improvement
Affects Versions: 4.9, 4.9.1, 4.10, 4.10.1
Reporter: Dawid Weiss
Priority: Minor

 We have a custom token stream that emits information about offsets of each 
 token. My (wrong) assumption was that for multi-valued fields a token 
 stream's offset information is magically shifted, much like this is the case 
 with positions. It's not the case -- offsets should be increasing and 
 monotonic across all instances of a field, even if it has custom token 
 streams. So, something like this:
 {code}
 doc.add(new Field(field-foo, new CannedTokenStream(token(bar, 1, 
 150, 160)), ftype));
 doc.add(new Field(field-foo, new CannedTokenStream(token(bar, 1,  
 50,  60)), ftype));
 {code}
 where the token function is defined as:
 {code}
 token(String image, int positionIncrement, int startOffset, int endOffset)
 {code}
 will result in either a cryptic assertion thrown from IW:
 {code}
 Exception in thread main java.lang.AssertionError
   at 
 org.apache.lucene.index.FreqProxTermsWriterPerField.writeOffsets(FreqProxTermsWriterPerField.java:99)
 {code}
 or nothing (or a codec error) if run without assertions.
 Obviously returning non-shifted offsets from subsequent token streams makes 
 little sense but I wonder if it could be made more explicit (or asserted) 
 that offsets need to be increasing between multiple-values. The minimum is to 
 add some documentation to OffsetAttribute. I don't know if offsets should be 
 shifted automatically, as it's the case with positions -- this would change 
 the semantics of existing tokenizers and filters which implement such 
 shifting internally already.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Comment Edited] (LUCENE-5977) IW should safeguard against token streams returning invalid offsets for multi-valued fields

2014-09-29 Thread ASF subversion and git services (JIRA)


[ 
https://issues.apache.org/jira/browse/LUCENE-5977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14151567#comment-14151567
 ] 

Dawid Weiss edited comment on LUCENE-5977 at 9/29/14 10:16 AM:
---

A full example which shows the problem. Run it with -ea -- you'll get the 
assertion. Run it without assertions and it'll pass.

{code}
import org.apache.lucene.analysis.Token;
import org.apache.lucene.analysis.core.WhitespaceAnalyzer;
import org.apache.lucene.codecs.Codec;
import org.apache.lucene.codecs.simpletext.SimpleTextCodec;
import org.apache.lucene.document.Document;
import org.apache.lucene.document.Field;
import org.apache.lucene.document.FieldType;
import org.apache.lucene.index.FieldInfo.IndexOptions;
import org.apache.lucene.index.IndexWriter;
import org.apache.lucene.index.IndexWriterConfig;
import org.apache.lucene.store.Directory;
import org.apache.lucene.store.RAMDirectory;
import org.apache.lucene.util.Version;
import org.apache.lucene.analysis.CannedTokenStream;

public class OffsetIndexingBug {
  
  public static void main(String[] args) throws Exception {
Codec.setDefault(new SimpleTextCodec());
Version version = Version.LUCENE_CURRENT;
IndexWriterConfig conf = new IndexWriterConfig(version, new 
WhitespaceAnalyzer(version));
conf.setUseCompoundFile(false);
try (Directory d = new RAMDirectory()) {
  try (IndexWriter iw = new IndexWriter(d, conf)) {
Document doc = new Document();

FieldType ftype = new FieldType();
ftype.setIndexed(true);
ftype.setStored(false);
ftype.setOmitNorms(true);

ftype.setStoreTermVectors(true);
ftype.setStoreTermVectorPositions(true);
ftype.setStoreTermVectorOffsets(true);

ftype.setTokenized(true);

ftype.setIndexOptions(IndexOptions.DOCS_AND_FREQS_AND_POSITIONS_AND_OFFSETS);
ftype.freeze();

// note use's offset is negative with respect to the first field 
value. 
doc.add(new Field(field-foo, new CannedTokenStream(token(use, 1, 
150, 160)), ftype));
doc.add(new Field(field-foo, new CannedTokenStream(token(use, 1,  
50,  60)), ftype));
iw.addDocument(doc);
  }
}
  }

  private static Token token(String image, int positionIncrement, int soffset, 
int eoffset) {
Token t = new Token();
t.setPositionIncrement(positionIncrement);
t.setOffset(soffset, eoffset);
t.append(image);
return t;
  }
}
{code}


was (Author: dweiss):
A full example which shows the problem. Run it with -ea -- you'll get the 
assertion. Run it without assertions and it'll pass.

{code}
import org.apache.lucene.analysis.Token;
import org.apache.lucene.analysis.core.WhitespaceAnalyzer;
import org.apache.lucene.codecs.Codec;
import org.apache.lucene.codecs.simpletext.SimpleTextCodec;
import org.apache.lucene.document.Document;
import org.apache.lucene.document.Field;
import org.apache.lucene.document.FieldType;
import org.apache.lucene.index.FieldInfo.IndexOptions;
import org.apache.lucene.index.IndexWriter;
import org.apache.lucene.index.IndexWriterConfig;
import org.apache.lucene.store.Directory;
import org.apache.lucene.store.RAMDirectory;
import org.apache.lucene.util.Version;
import org.apache.lucene.analysis.CannedTokenStream;

public class OffsetIndexingBug {
  
  public static void main(String[] args) throws Exception {
Codec.setDefault(new SimpleTextCodec());
Version version = Version.LUCENE_CURRENT;
IndexWriterConfig conf = new IndexWriterConfig(version, new 
WhitespaceAnalyzer(version));
conf.setUseCompoundFile(false);
try (Directory d = new RAMDirectory()) {
  try (IndexWriter iw = new IndexWriter(d, conf)) {
iw.deleteAll();
iw.commit();

Document doc = new Document();

FieldType ftype = new FieldType();
ftype.setIndexed(true);
ftype.setStored(false);
ftype.setOmitNorms(true);

ftype.setStoreTermVectors(true);
ftype.setStoreTermVectorPositions(true);
ftype.setStoreTermVectorOffsets(true);

ftype.setTokenized(true);

ftype.setIndexOptions(IndexOptions.DOCS_AND_FREQS_AND_POSITIONS_AND_OFFSETS);
ftype.freeze();

// note use's offset is negative with respect to the first field 
value. 
doc.add(new Field(field-foo, new CannedTokenStream(token(use, 1, 
150, 160)), ftype));
doc.add(new Field(field-foo, new CannedTokenStream(token(use, 1,  
50,  60)), ftype));
iw.addDocument(doc);
  }
}
  }

  private static Token token(String image, int positionIncrement, int soffset, 
int eoffset) {
Token t = new Token();
t.setPositionIncrement(positionIncrement);
t.setOffset(soffset, eoffset);
t.append(image);
return t;
  }
}
{code}

 IW should safeguard against token streams returning invalid offsets for 
 multi-valued fields

[jira] [Commented] (LUCENE-5911) Make MemoryIndex thread-safe for queries


[ 
https://issues.apache.org/jira/browse/LUCENE-5911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14151568#comment-14151568
 ] 

ASF subversion and git services commented on LUCENE-5911:
-

Commit 1628154 from [~romseygeek] in branch 'dev/trunk'
[ https://svn.apache.org/r1628154 ]

LUCENE-5911: Add MemoryIndex.freeze() to allow for thread-safe searching

 Make MemoryIndex thread-safe for queries
 

 Key: LUCENE-5911
 URL: https://issues.apache.org/jira/browse/LUCENE-5911
 Project: Lucene - Core
  Issue Type: Improvement
Reporter: Alan Woodward
Priority: Minor
 Attachments: LUCENE-5911.patch, LUCENE-5911.patch


 We want to be able to run multiple queries at once over a MemoryIndex in 
 luwak (see 
 https://github.com/flaxsearch/luwak/commit/49a8fba5764020c2f0e4dc29d80d93abb0231191),
  but this isn't possible with the current implementation.  However, looking 
 at the code, it seems that it would be relatively simple to make MemoryIndex 
 thread-safe for reads/queries.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-5911) Make MemoryIndex thread-safe for queries

2014-09-29 Thread ASF subversion and git services (JIRA)


[ 
https://issues.apache.org/jira/browse/LUCENE-5911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14151569#comment-14151569
 ] 

ASF subversion and git services commented on LUCENE-5911:
-

Commit 1628155 from [~romseygeek] in branch 'dev/branches/branch_5x'
[ https://svn.apache.org/r1628155 ]

LUCENE-5911: Add MemoryIndex.freeze() to allow for thread-safe searching

 Make MemoryIndex thread-safe for queries
 

 Key: LUCENE-5911
 URL: https://issues.apache.org/jira/browse/LUCENE-5911
 Project: Lucene - Core
  Issue Type: Improvement
Reporter: Alan Woodward
Priority: Minor
 Attachments: LUCENE-5911.patch, LUCENE-5911.patch


 We want to be able to run multiple queries at once over a MemoryIndex in 
 luwak (see 
 https://github.com/flaxsearch/luwak/commit/49a8fba5764020c2f0e4dc29d80d93abb0231191),
  but this isn't possible with the current implementation.  However, looking 
 at the code, it seems that it would be relatively simple to make MemoryIndex 
 thread-safe for reads/queries.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Resolved] (LUCENE-5911) Make MemoryIndex thread-safe for queries

2014-09-29 Thread Alan Woodward (JIRA)


 [ 
https://issues.apache.org/jira/browse/LUCENE-5911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alan Woodward resolved LUCENE-5911.
---
   Resolution: Fixed
Fix Version/s: Trunk
   5.0
 Assignee: Alan Woodward

 Make MemoryIndex thread-safe for queries
 

 Key: LUCENE-5911
 URL: https://issues.apache.org/jira/browse/LUCENE-5911
 Project: Lucene - Core
  Issue Type: Improvement
Reporter: Alan Woodward
Assignee: Alan Woodward
Priority: Minor
 Fix For: 5.0, Trunk

 Attachments: LUCENE-5911.patch, LUCENE-5911.patch


 We want to be able to run multiple queries at once over a MemoryIndex in 
 luwak (see 
 https://github.com/flaxsearch/luwak/commit/49a8fba5764020c2f0e4dc29d80d93abb0231191),
  but this isn't possible with the current implementation.  However, looking 
 at the code, it seems that it would be relatively simple to make MemoryIndex 
 thread-safe for reads/queries.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-5.x-Linux (32bit/jdk1.8.0_20) - Build # 11195 - Failure!

Build: http://jenkins.thetaphi.de/job/Lucene-Solr-5.x-Linux/11195/
Java: 32bit/jdk1.8.0_20 -server -XX:+UseSerialGC

2 tests failed.
REGRESSION:  
org.apache.solr.cloud.DeleteLastCustomShardedReplicaTest.testDistribSearch

Error Message:
No live SolrServers available to handle this request:[http://127.0.0.1:54590, 
http://127.0.0.1:38802, http://127.0.0.1:53259]

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: No live SolrServers available 
to handle this request:[http://127.0.0.1:54590, http://127.0.0.1:38802, 
http://127.0.0.1:53259]
at 
__randomizedtesting.SeedInfo.seed([4FE08E3460AF700C:CE06002C17F01030]:0)
at 
org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:322)
at 
org.apache.solr.client.solrj.impl.CloudSolrServer.sendRequest(CloudSolrServer.java:880)
at 
org.apache.solr.client.solrj.impl.CloudSolrServer.requestWithRetryOnStaleState(CloudSolrServer.java:658)
at 
org.apache.solr.client.solrj.impl.CloudSolrServer.request(CloudSolrServer.java:601)
at 
org.apache.solr.cloud.DeleteLastCustomShardedReplicaTest.removeAndWaitForLastReplicaGone(DeleteLastCustomShardedReplicaTest.java:117)
at 
org.apache.solr.cloud.DeleteLastCustomShardedReplicaTest.doTest(DeleteLastCustomShardedReplicaTest.java:107)
at 
org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:869)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at

[jira] [Commented] (LUCENE-5977) IW should safeguard against token streams returning invalid offsets for multi-valued fields


[ 
https://issues.apache.org/jira/browse/LUCENE-5977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14151584#comment-14151584
 ] 

Dawid Weiss commented on LUCENE-5977:
-

I looked a bit more closely at the logic of updating offsets in the 
DefaultIndexingChain and I see offsets *are* propagated for multiple fields, 
here:

{code}
// trigger streams to perform end-of-stream operations
stream.end();

// TODO: maybe add some safety? then again, its already checked 
// when we come back around to the field...
invertState.position += 
invertState.posIncrAttribute.getPositionIncrement();
invertState.offset += invertState.offsetAttribute.endOffset();
{code}

So the problem with my implementation was that it should have set offsets 
properly in end(). I still feel this should be verified / asserted cleaner 
somehow, so I'll leave this issue open looking for suggestions.

 IW should safeguard against token streams returning invalid offsets for 
 multi-valued fields
 ---

 Key: LUCENE-5977
 URL: https://issues.apache.org/jira/browse/LUCENE-5977
 Project: Lucene - Core
  Issue Type: Improvement
Affects Versions: 4.9, 4.9.1, 4.10, 4.10.1
Reporter: Dawid Weiss
Priority: Minor

 We have a custom token stream that emits information about offsets of each 
 token. My (wrong) assumption was that for multi-valued fields a token 
 stream's offset information is magically shifted, much like this is the case 
 with positions. It's not the case -- offsets should be increasing and 
 monotonic across all instances of a field, even if it has custom token 
 streams. So, something like this:
 {code}
 doc.add(new Field(field-foo, new CannedTokenStream(token(bar, 1, 
 150, 160)), ftype));
 doc.add(new Field(field-foo, new CannedTokenStream(token(bar, 1,  
 50,  60)), ftype));
 {code}
 where the token function is defined as:
 {code}
 token(String image, int positionIncrement, int startOffset, int endOffset)
 {code}
 will result in either a cryptic assertion thrown from IW:
 {code}
 Exception in thread main java.lang.AssertionError
   at 
 org.apache.lucene.index.FreqProxTermsWriterPerField.writeOffsets(FreqProxTermsWriterPerField.java:99)
 {code}
 or nothing (or a codec error) if run without assertions.
 Obviously returning non-shifted offsets from subsequent token streams makes 
 little sense but I wonder if it could be made more explicit (or asserted) 
 that offsets need to be increasing between multiple-values. The minimum is to 
 add some documentation to OffsetAttribute. I don't know if offsets should be 
 shifted automatically, as it's the case with positions -- this would change 
 the semantics of existing tokenizers and filters which implement such 
 shifting internally already.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-NightlyTests-5.x - Build # 635 - Still Failing

Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-5.x/635/

3 tests failed.
REGRESSION:  org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.testDistribSearch

Error Message:
Captured an uncaught exception in thread: Thread[id=13668, 
name=qtp1148022701-13668, state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderTest]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught 
exception in thread: Thread[id=13668, name=qtp1148022701-13668, state=RUNNABLE, 
group=TGRP-ChaosMonkeySafeLeaderTest]
Caused by: java.lang.OutOfMemoryError: unable to create new native thread
at __randomizedtesting.SeedInfo.seed([C76C029776C01205]:0)
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:714)
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1047)
at 
sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1312)
at 
sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1339)
at 
sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1323)
at 
org.eclipse.jetty.server.ssl.SslSocketConnector$SslConnectorEndPoint.run(SslSocketConnector.java:665)
at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
at 
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
at java.lang.Thread.run(Thread.java:745)


FAILED:  
junit.framework.TestSuite.org.apache.solr.handler.TestReplicationHandler

Error Message:
Resource in scope SUITE failed to close. Resource was registered from thread 
Thread[id=259, name=coreLoadExecutor-163-thread-1, state=RUNNABLE, 
group=TGRP-TestReplicationHandler], registration stack trace below.

Stack Trace:
com.carrotsearch.randomizedtesting.ResourceDisposalError: Resource in scope 
SUITE failed to close. Resource was registered from thread Thread[id=259, 
name=coreLoadExecutor-163-thread-1, state=RUNNABLE, 
group=TGRP-TestReplicationHandler], registration stack trace below.
at __randomizedtesting.SeedInfo.seed([C76C029776C01205]:0)
at java.lang.Thread.getStackTrace(Thread.java:1589)
at 
com.carrotsearch.randomizedtesting.RandomizedContext.closeAtEnd(RandomizedContext.java:166)
at 
org.apache.lucene.util.LuceneTestCase.closeAfterSuite(LuceneTestCase.java:688)
at 
org.apache.lucene.util.LuceneTestCase.wrapDirectory(LuceneTestCase.java:1231)
at 
org.apache.lucene.util.LuceneTestCase.newDirectory(LuceneTestCase.java:1122)
at 
org.apache.lucene.util.LuceneTestCase.newDirectory(LuceneTestCase.java:1114)
at 
org.apache.solr.core.MockDirectoryFactory.create(MockDirectoryFactory.java:47)
at 
org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:350)
at org.apache.solr.core.SolrCore.getNewIndexDir(SolrCore.java:275)
at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:487)
at org.apache.solr.core.SolrCore.init(SolrCore.java:793)
at org.apache.solr.core.SolrCore.init(SolrCore.java:651)
at org.apache.solr.core.CoreContainer.create(CoreContainer.java:491)
at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:255)
at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:249)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.AssertionError: Directory not closed: 
MockDirectoryWrapper(NIOFSDirectory@/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J2/temp/solr.handler.TestReplicationHandler-C76C029776C01205-001/index-NIOFSDirectory-028
 
lockFactory=NativeFSLockFactory@/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J2/temp/solr.handler.TestReplicationHandler-C76C029776C01205-001/index-NIOFSDirectory-028)
at org.junit.Assert.fail(Assert.java:93)
at 
org.apache.lucene.util.CloseableDirectory.close(CloseableDirectory.java:47)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$2$1.apply(RandomizedRunner.java:699)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$2$1.apply(RandomizedRunner.java:696)
at 
com.carrotsearch.randomizedtesting.RandomizedContext.closeResources(RandomizedContext.java:183)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$2.afterAlways(RandomizedRunner.java:712)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:43)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
... 1 more


FAILED:

[jira] [Commented] (LUCENE-5977) IW should safeguard against token streams returning invalid offsets for multi-valued fields


[ 
https://issues.apache.org/jira/browse/LUCENE-5977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14151598#comment-14151598
 ] 

Robert Muir commented on LUCENE-5977:
-

The bug is older than 4.9 actually: it existed in the previous indexing chain, 
too.

I ran the test program against 4.8, even with assertingcodec, it just silently 
succeeds.

But if you add checkindex call:
{noformat}
Segments file=segments_1 numSegments=1 version=4.8 format=
  1 of 1: name=_0 docCount=1
codec=Asserting
compound=false
numFiles=12
size (MB)=0.001
diagnostics = {timestamp=1411990327375, os=Linux, 
os.version=3.13.0-24-generic, source=flush, lucene.version=4.8-SNAPSHOT, 
os.arch=amd64, java.version=1.7.0_55, java.vendor=Oracle Corporation}
no deletions
test: open reader.OK
test: check integrity.OK
test: check live docs.OK
test: fields..OK [1 fields]
test: field norms.OK [0 fields]
test: terms, freq, prox...ERROR: java.lang.RuntimeException: term [75 73 
65]: doc 0: pos 1: startOffset -2147483597 is out of bounds
java.lang.RuntimeException: term [75 73 65]: doc 0: pos 1: startOffset 
-2147483597 is out of bounds
at org.apache.lucene.index.CheckIndex.checkFields(CheckIndex.java:944)
at org.apache.lucene.index.CheckIndex.testPostings(CheckIndex.java:1278)
at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:626)
at org.apache.lucene.util.TestUtil.checkIndex(TestUtil.java:199)
at org.apache.lucene.util.TestUtil.checkIndex(TestUtil.java:191)
at org.apache.lucene.OffsetIndexingBug.main(OffsetIndexingBug.java:48)
test: stored fields...OK [0 total field count; avg 0 fields per doc]
test: term vectorsERROR [vector term=[75 73 65] field=field-foo 
doc=0: startOffset=51 differs from postings startOffset=-2147483597]
java.lang.RuntimeException: vector term=[75 73 65] field=field-foo doc=0: 
startOffset=51 differs from postings startOffset=-2147483597
at 
org.apache.lucene.index.CheckIndex.testTermVectors(CheckIndex.java:1748)
at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:632)
at org.apache.lucene.util.TestUtil.checkIndex(TestUtil.java:199)
at org.apache.lucene.util.TestUtil.checkIndex(TestUtil.java:191)
at org.apache.lucene.OffsetIndexingBug.main(OffsetIndexingBug.java:48)
test: docvalues...OK [0 docvalues fields; 0 BINARY; 0 NUMERIC; 0 
SORTED; 0 SORTED_SET]
FAILED
WARNING: fixIndex() would remove reference to this segment; full exception:
java.lang.RuntimeException: Term Index test failed
at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:641)
at org.apache.lucene.util.TestUtil.checkIndex(TestUtil.java:199)
at org.apache.lucene.util.TestUtil.checkIndex(TestUtil.java:191)
at org.apache.lucene.OffsetIndexingBug.main(OffsetIndexingBug.java:48)

WARNING: 1 broken segments (containing 1 documents) detected
{noformat}

 IW should safeguard against token streams returning invalid offsets for 
 multi-valued fields
 ---

 Key: LUCENE-5977
 URL: https://issues.apache.org/jira/browse/LUCENE-5977
 Project: Lucene - Core
  Issue Type: Improvement
Affects Versions: 4.9, 4.9.1, 4.10, 4.10.1
Reporter: Dawid Weiss
Priority: Minor

 We have a custom token stream that emits information about offsets of each 
 token. My (wrong) assumption was that for multi-valued fields a token 
 stream's offset information is magically shifted, much like this is the case 
 with positions. It's not the case -- offsets should be increasing and 
 monotonic across all instances of a field, even if it has custom token 
 streams. So, something like this:
 {code}
 doc.add(new Field(field-foo, new CannedTokenStream(token(bar, 1, 
 150, 160)), ftype));
 doc.add(new Field(field-foo, new CannedTokenStream(token(bar, 1,  
 50,  60)), ftype));
 {code}
 where the token function is defined as:
 {code}
 token(String image, int positionIncrement, int startOffset, int endOffset)
 {code}
 will result in either a cryptic assertion thrown from IW:
 {code}
 Exception in thread main java.lang.AssertionError
   at 
 org.apache.lucene.index.FreqProxTermsWriterPerField.writeOffsets(FreqProxTermsWriterPerField.java:99)
 {code}
 or nothing (or a codec error) if run without assertions.
 Obviously returning non-shifted offsets from subsequent token streams makes 
 little sense but I wonder if it could be made more explicit (or asserted) 
 that offsets need to be increasing between multiple-values. The minimum is to 
 add some documentation to OffsetAttribute. I don't know if offsets should be 
 shifted automatically, as it's the case with positions --

[jira] [Updated] (LUCENE-5977) IW should safeguard against token streams returning invalid offsets for multi-valued fields


 [ 
https://issues.apache.org/jira/browse/LUCENE-5977?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Muir updated LUCENE-5977:

Attachment: LUCENE-5977.patch

The bug is some validation checks rely on lastStartOffset/lastPosition, but 
this is currently reset to 0 on each field instance.

We should instead track it across all instances of the field. I converted 
Dawid's test to a case in TestPostingsOffsets and moved these two state 
variables to FieldInvertState.

 IW should safeguard against token streams returning invalid offsets for 
 multi-valued fields
 ---

 Key: LUCENE-5977
 URL: https://issues.apache.org/jira/browse/LUCENE-5977
 Project: Lucene - Core
  Issue Type: Improvement
Affects Versions: 4.9, 4.9.1, 4.10, 4.10.1
Reporter: Dawid Weiss
Priority: Minor
 Attachments: LUCENE-5977.patch


 We have a custom token stream that emits information about offsets of each 
 token. My (wrong) assumption was that for multi-valued fields a token 
 stream's offset information is magically shifted, much like this is the case 
 with positions. It's not the case -- offsets should be increasing and 
 monotonic across all instances of a field, even if it has custom token 
 streams. So, something like this:
 {code}
 doc.add(new Field(field-foo, new CannedTokenStream(token(bar, 1, 
 150, 160)), ftype));
 doc.add(new Field(field-foo, new CannedTokenStream(token(bar, 1,  
 50,  60)), ftype));
 {code}
 where the token function is defined as:
 {code}
 token(String image, int positionIncrement, int startOffset, int endOffset)
 {code}
 will result in either a cryptic assertion thrown from IW:
 {code}
 Exception in thread main java.lang.AssertionError
   at 
 org.apache.lucene.index.FreqProxTermsWriterPerField.writeOffsets(FreqProxTermsWriterPerField.java:99)
 {code}
 or nothing (or a codec error) if run without assertions.
 Obviously returning non-shifted offsets from subsequent token streams makes 
 little sense but I wonder if it could be made more explicit (or asserted) 
 that offsets need to be increasing between multiple-values. The minimum is to 
 add some documentation to OffsetAttribute. I don't know if offsets should be 
 shifted automatically, as it's the case with positions -- this would change 
 the semantics of existing tokenizers and filters which implement such 
 shifting internally already.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (LUCENE-5977) IW should safeguard against token streams returning invalid offsets for multi-valued fields


 [ 
https://issues.apache.org/jira/browse/LUCENE-5977?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Muir updated LUCENE-5977:

Issue Type: Bug  (was: Improvement)

 IW should safeguard against token streams returning invalid offsets for 
 multi-valued fields
 ---

 Key: LUCENE-5977
 URL: https://issues.apache.org/jira/browse/LUCENE-5977
 Project: Lucene - Core
  Issue Type: Bug
Affects Versions: 4.9, 4.9.1, 4.10, 4.10.1
Reporter: Dawid Weiss
Priority: Minor
 Fix For: 4.10.2

 Attachments: LUCENE-5977.patch


 We have a custom token stream that emits information about offsets of each 
 token. My (wrong) assumption was that for multi-valued fields a token 
 stream's offset information is magically shifted, much like this is the case 
 with positions. It's not the case -- offsets should be increasing and 
 monotonic across all instances of a field, even if it has custom token 
 streams. So, something like this:
 {code}
 doc.add(new Field(field-foo, new CannedTokenStream(token(bar, 1, 
 150, 160)), ftype));
 doc.add(new Field(field-foo, new CannedTokenStream(token(bar, 1,  
 50,  60)), ftype));
 {code}
 where the token function is defined as:
 {code}
 token(String image, int positionIncrement, int startOffset, int endOffset)
 {code}
 will result in either a cryptic assertion thrown from IW:
 {code}
 Exception in thread main java.lang.AssertionError
   at 
 org.apache.lucene.index.FreqProxTermsWriterPerField.writeOffsets(FreqProxTermsWriterPerField.java:99)
 {code}
 or nothing (or a codec error) if run without assertions.
 Obviously returning non-shifted offsets from subsequent token streams makes 
 little sense but I wonder if it could be made more explicit (or asserted) 
 that offsets need to be increasing between multiple-values. The minimum is to 
 add some documentation to OffsetAttribute. I don't know if offsets should be 
 shifted automatically, as it's the case with positions -- this would change 
 the semantics of existing tokenizers and filters which implement such 
 shifting internally already.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (LUCENE-5977) IW should safeguard against token streams returning invalid offsets for multi-valued fields


 [ 
https://issues.apache.org/jira/browse/LUCENE-5977?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Muir updated LUCENE-5977:

Fix Version/s: 4.10.2

 IW should safeguard against token streams returning invalid offsets for 
 multi-valued fields
 ---

 Key: LUCENE-5977
 URL: https://issues.apache.org/jira/browse/LUCENE-5977
 Project: Lucene - Core
  Issue Type: Bug
Affects Versions: 4.9, 4.9.1, 4.10, 4.10.1
Reporter: Dawid Weiss
Priority: Minor
 Fix For: 4.10.2

 Attachments: LUCENE-5977.patch


 We have a custom token stream that emits information about offsets of each 
 token. My (wrong) assumption was that for multi-valued fields a token 
 stream's offset information is magically shifted, much like this is the case 
 with positions. It's not the case -- offsets should be increasing and 
 monotonic across all instances of a field, even if it has custom token 
 streams. So, something like this:
 {code}
 doc.add(new Field(field-foo, new CannedTokenStream(token(bar, 1, 
 150, 160)), ftype));
 doc.add(new Field(field-foo, new CannedTokenStream(token(bar, 1,  
 50,  60)), ftype));
 {code}
 where the token function is defined as:
 {code}
 token(String image, int positionIncrement, int startOffset, int endOffset)
 {code}
 will result in either a cryptic assertion thrown from IW:
 {code}
 Exception in thread main java.lang.AssertionError
   at 
 org.apache.lucene.index.FreqProxTermsWriterPerField.writeOffsets(FreqProxTermsWriterPerField.java:99)
 {code}
 or nothing (or a codec error) if run without assertions.
 Obviously returning non-shifted offsets from subsequent token streams makes 
 little sense but I wonder if it could be made more explicit (or asserted) 
 that offsets need to be increasing between multiple-values. The minimum is to 
 add some documentation to OffsetAttribute. I don't know if offsets should be 
 shifted automatically, as it's the case with positions -- this would change 
 the semantics of existing tokenizers and filters which implement such 
 shifting internally already.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-trunk-Linux (32bit/jdk1.8.0_40-ea-b04) - Build # 11348 - Still Failing!