[jira] [Assigned] (SOLR-5258) router.field support for compositeId router

2013-09-21 Thread Noble Paul (JIRA)

 [ 
https://issues.apache.org/jira/browse/SOLR-5258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Noble Paul reassigned SOLR-5258:


Assignee: Noble Paul

> router.field support for compositeId router
> ---
>
> Key: SOLR-5258
> URL: https://issues.apache.org/jira/browse/SOLR-5258
> Project: Solr
>  Issue Type: New Feature
>Reporter: Yonik Seeley
>Assignee: Noble Paul
>Priority: Minor
>
> Although there is code to support router.field for CompositeId, it only 
> calculates a simple (non-compound) hash, which isn't that useful unless you 
> don't use compound ids (this is why I changed the docs to say router.field is 
> only supported for the implicit router).  The field value should either
> - be used to calculate the full compound hash
> - be used to calculate the prefix bits, and the uniqueKey will still be used 
> for the lower bits.
> For consistency, I'd suggest the former.
> If we want to be able to specify a separate field that is only used for the 
> prefix bits, then perhaps that should be "router.prefixField"

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org



[jira] [Commented] (SOLR-3633) web UI reports an error if CoreAdminHandler says there are no SolrCores

2013-09-21 Thread Erick Erickson (JIRA)

[ 
https://issues.apache.org/jira/browse/SOLR-3633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13773973#comment-13773973
 ] 

Erick Erickson commented on SOLR-3633:
--

Aaron:

Please define "trouble", and I suggest you ask the question on the user's list, 
this has been working for others.

> web UI reports an error if CoreAdminHandler says there are no SolrCores
> ---
>
> Key: SOLR-3633
> URL: https://issues.apache.org/jira/browse/SOLR-3633
> Project: Solr
>  Issue Type: Bug
>  Components: web gui
>Affects Versions: 4.0-ALPHA
>Reporter: Hoss Man
>Assignee: Stefan Matheis (steffkes)
> Fix For: 4.5, 5.0
>
> Attachments: SOLR-3633.patch, SOLR-3633.patch, SOLR-3633.patch, 
> SOLR-3633.patch, SOLR-3633.patch, SOLR-3633.patch
>
>
> Spun off from SOLR-3591...
> * having no SolrCores is a valid situation
> * independent of what may happen in SOLR-3591, the web UI should cleanly deal 
> with there being no SolrCores, and just hide/grey out any tabs that can't be 
> supported w/o at least one core
> * even if there are no SolrCores the core admin features (ie: creating a new 
> core) should be accessible in the UI

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org



[jira] [Commented] (SOLR-3633) web UI reports an error if CoreAdminHandler says there are no SolrCores

2013-09-21 Thread Erick Erickson (JIRA)

[ 
https://issues.apache.org/jira/browse/SOLR-3633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13773972#comment-13773972
 ] 

Erick Erickson commented on SOLR-3633:
--

Aaron:

Please define "trouble", and I suggest you ask the question on the user's list, 
this has been working for others.

> web UI reports an error if CoreAdminHandler says there are no SolrCores
> ---
>
> Key: SOLR-3633
> URL: https://issues.apache.org/jira/browse/SOLR-3633
> Project: Solr
>  Issue Type: Bug
>  Components: web gui
>Affects Versions: 4.0-ALPHA
>Reporter: Hoss Man
>Assignee: Stefan Matheis (steffkes)
> Fix For: 4.5, 5.0
>
> Attachments: SOLR-3633.patch, SOLR-3633.patch, SOLR-3633.patch, 
> SOLR-3633.patch, SOLR-3633.patch, SOLR-3633.patch
>
>
> Spun off from SOLR-3591...
> * having no SolrCores is a valid situation
> * independent of what may happen in SOLR-3591, the web UI should cleanly deal 
> with there being no SolrCores, and just hide/grey out any tabs that can't be 
> supported w/o at least one core
> * even if there are no SolrCores the core admin features (ie: creating a new 
> core) should be accessible in the UI

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org



[jira] [Updated] (LUCENE-5235) throw illegalstate from Tokenizer (instead of NPE/IIOBE) if reset not called

2013-09-21 Thread Robert Muir (JIRA)

 [ 
https://issues.apache.org/jira/browse/LUCENE-5235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Muir updated LUCENE-5235:


Attachment: LUCENE-5235.patch

> throw illegalstate from Tokenizer (instead of NPE/IIOBE) if reset not called
> 
>
> Key: LUCENE-5235
> URL: https://issues.apache.org/jira/browse/LUCENE-5235
> Project: Lucene - Core
>  Issue Type: Improvement
>  Components: modules/analysis
>Reporter: Robert Muir
> Attachments: LUCENE-5235.patch, LUCENE-5235.patch, LUCENE-5235.patch, 
> LUCENE-5235.patch, LUCENE-5235.patch, LUCENE-5235.patch, 
> LUCENE-5235_test.patch
>
>
> We added these best effort checks, but it would be much better if we somehow 
> gave a clear exception... this comes up often

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org



[jira] [Commented] (LUCENE-5235) throw illegalstate from Tokenizer (instead of NPE/IIOBE) if reset not called

2013-09-21 Thread Robert Muir (JIRA)

[ 
https://issues.apache.org/jira/browse/LUCENE-5235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13773965#comment-13773965
 ] 

Robert Muir commented on LUCENE-5235:
-

updated patch: all tests pass

patch here: http://pastebin.com/raw.php?i=SyvM6dCZ

> throw illegalstate from Tokenizer (instead of NPE/IIOBE) if reset not called
> 
>
> Key: LUCENE-5235
> URL: https://issues.apache.org/jira/browse/LUCENE-5235
> Project: Lucene - Core
>  Issue Type: Improvement
>  Components: modules/analysis
>Reporter: Robert Muir
> Attachments: LUCENE-5235.patch, LUCENE-5235.patch, LUCENE-5235.patch, 
> LUCENE-5235.patch, LUCENE-5235.patch, LUCENE-5235.patch, 
> LUCENE-5235_test.patch
>
>
> We added these best effort checks, but it would be much better if we somehow 
> gave a clear exception... this comes up often

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org



[jira] [Updated] (LUCENE-5235) throw illegalstate from Tokenizer (instead of NPE/IIOBE) if reset not called

2013-09-21 Thread Robert Muir (JIRA)

 [ 
https://issues.apache.org/jira/browse/LUCENE-5235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Muir updated LUCENE-5235:


Attachment: LUCENE-5235.patch

current patch, now tests are passing in lucene, but there are problems in solr 
tests.

The test is big because we got confused with the assertAnalyzesToReuse vs 
assertAnalyzesTo (which is now useless) so we merged these.

> throw illegalstate from Tokenizer (instead of NPE/IIOBE) if reset not called
> 
>
> Key: LUCENE-5235
> URL: https://issues.apache.org/jira/browse/LUCENE-5235
> Project: Lucene - Core
>  Issue Type: Improvement
>  Components: modules/analysis
>Reporter: Robert Muir
> Attachments: LUCENE-5235.patch, LUCENE-5235.patch, LUCENE-5235.patch, 
> LUCENE-5235.patch, LUCENE-5235.patch, LUCENE-5235_test.patch
>
>
> We added these best effort checks, but it would be much better if we somehow 
> gave a clear exception... this comes up often

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org



[jira] [Updated] (LUCENE-5235) throw illegalstate from Tokenizer (instead of NPE/IIOBE) if reset not called

2013-09-21 Thread Uwe Schindler (JIRA)

 [ 
https://issues.apache.org/jira/browse/LUCENE-5235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Uwe Schindler updated LUCENE-5235:
--

Attachment: LUCENE-5235.patch

More Tokenizers fixed.

> throw illegalstate from Tokenizer (instead of NPE/IIOBE) if reset not called
> 
>
> Key: LUCENE-5235
> URL: https://issues.apache.org/jira/browse/LUCENE-5235
> Project: Lucene - Core
>  Issue Type: Improvement
>  Components: modules/analysis
>Reporter: Robert Muir
> Attachments: LUCENE-5235.patch, LUCENE-5235.patch, LUCENE-5235.patch, 
> LUCENE-5235.patch, LUCENE-5235_test.patch
>
>
> We added these best effort checks, but it would be much better if we somehow 
> gave a clear exception... this comes up often

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org



[jira] [Commented] (SOLR-3633) web UI reports an error if CoreAdminHandler says there are no SolrCores

2013-09-21 Thread Aaron Greenspan (JIRA)

[ 
https://issues.apache.org/jira/browse/SOLR-3633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13773926#comment-13773926
 ] 

Aaron Greenspan commented on SOLR-3633:
---

I'm still having this problem with 4.4.0.

> web UI reports an error if CoreAdminHandler says there are no SolrCores
> ---
>
> Key: SOLR-3633
> URL: https://issues.apache.org/jira/browse/SOLR-3633
> Project: Solr
>  Issue Type: Bug
>  Components: web gui
>Affects Versions: 4.0-ALPHA
>Reporter: Hoss Man
>Assignee: Stefan Matheis (steffkes)
> Fix For: 4.5, 5.0
>
> Attachments: SOLR-3633.patch, SOLR-3633.patch, SOLR-3633.patch, 
> SOLR-3633.patch, SOLR-3633.patch, SOLR-3633.patch
>
>
> Spun off from SOLR-3591...
> * having no SolrCores is a valid situation
> * independent of what may happen in SOLR-3591, the web UI should cleanly deal 
> with there being no SolrCores, and just hide/grey out any tabs that can't be 
> supported w/o at least one core
> * even if there are no SolrCores the core admin features (ie: creating a new 
> core) should be accessible in the UI

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org



[jira] [Commented] (LUCENE-5235) throw illegalstate from Tokenizer (instead of NPE/IIOBE) if reset not called

2013-09-21 Thread Michael McCandless (JIRA)

[ 
https://issues.apache.org/jira/browse/LUCENE-5235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13773911#comment-13773911
 ] 

Michael McCandless commented on LUCENE-5235:


+1, what a delightful solution!!

> throw illegalstate from Tokenizer (instead of NPE/IIOBE) if reset not called
> 
>
> Key: LUCENE-5235
> URL: https://issues.apache.org/jira/browse/LUCENE-5235
> Project: Lucene - Core
>  Issue Type: Improvement
>  Components: modules/analysis
>Reporter: Robert Muir
> Attachments: LUCENE-5235.patch, LUCENE-5235.patch, 
> LUCENE-5235_test.patch
>
>
> We added these best effort checks, but it would be much better if we somehow 
> gave a clear exception... this comes up often

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org



Re: [JENKINS] Lucene-Solr-NightlyTests-trunk - Build # 386 - Failure

2013-09-21 Thread Michael McCandless
Oh, I see ... OK +1 to remove the maxBufDocs=2.

Mike McCandless

http://blog.mikemccandless.com


On Sat, Sep 21, 2013 at 3:28 PM, Shai Erera  wrote:
> I thought about that, but then realized that the test does all the merging
> before the update threads start. This test only verifies that many
> concurrent updates (to same docs, fields etc.) works well.
> testSegmentMerging tests that concurrent updates w/ segment merging works
> well. So there really isn't a good reason for force many segments, and w/
> maxBufDocs=2 and numDocs=atLeast(2000), that means lots of merging, all
> happen before the update threads start anyway.
>
> Shai
>
>
> On Sat, Sep 21, 2013 at 8:01 PM, Michael McCandless
>  wrote:
>>
>> Maybe we can make maxBufDocs greater than 2 but still smallish?  Just
>> to make sure we are stressing merging + updating ...
>>
>> Mike McCandless
>>
>> http://blog.mikemccandless.com
>>
>>
>> On Sat, Sep 21, 2013 at 9:05 AM, Shai Erera  wrote:
>> > I don't hit an OOM with the reported parameters, but I do see the test
>> > takes
>> > extremely long time to finish (I killed it after >10 minutes!).
>> >
>> > With the reported parameters, I see that the test uses
>> > SerialMergeScheduler,
>> > spawns 18 threads and indexes 13K docs. Also, each thread calls
>> > IW.updateNDV
>> > 197 times. The test also hard-sets maxBufDocs to 2, which means a lot of
>> > segment merging. With these numbers, it just takes too long to complete
>> > (I
>> > see it makes progress, slowly). Maybe that, together with other tests
>> > running on Jenkins, can result in OOM.
>> >
>> > Anyway, I'd like to remove the hard-coded maxBufDocs=2. With
>> > numDocs=atLeast(2000), we should get few segments. Even with that, the
>> > test
>> > takes 4m to complete (w/ those settings). Maybe it's too stressing?
>> >
>> > Shai
>> >
>> >
>> > On Fri, Sep 20, 2013 at 3:48 PM, Apache Jenkins Server
>> >  wrote:
>> >>
>> >> Build:
>> >> https://builds.apache.org/job/Lucene-Solr-NightlyTests-trunk/386/
>> >>
>> >> 1 tests failed.
>> >> REGRESSION:
>> >>
>> >> org.apache.lucene.index.TestNumericDocValuesUpdates.testStressMultiThreading
>> >>
>> >> Error Message:
>> >> Captured an uncaught exception in thread: Thread[id=3130,
>> >> name=UpdateThread-1, state=RUNNABLE,
>> >> group=TGRP-TestNumericDocValuesUpdates]
>> >>
>> >> Stack Trace:
>> >> com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an
>> >> uncaught exception in thread: Thread[id=3130, name=UpdateThread-1,
>> >> state=RUNNABLE, group=TGRP-TestNumericDocValuesUpdates]
>> >> Caused by: java.lang.OutOfMemoryError: Java heap space
>> >> at __randomizedtesting.SeedInfo.seed([EA6A02F6820CB4B7]:0)
>> >> at java.util.Arrays.copyOf(Arrays.java:2367)
>> >> at
>> >>
>> >> java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:130)
>> >> at
>> >>
>> >> java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:114)
>> >> at
>> >> java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:415)
>> >> at java.lang.StringBuilder.append(StringBuilder.java:132)
>> >> at java.lang.StringBuilder.append(StringBuilder.java:128)
>> >> at
>> >> java.util.AbstractCollection.toString(AbstractCollection.java:450)
>> >> at java.lang.String.valueOf(String.java:2854)
>> >> at java.lang.StringBuilder.append(StringBuilder.java:128)
>> >> at
>> >> org.apache.lucene.index.IndexWriter.startCommit(IndexWriter.java:4239)
>> >> at
>> >>
>> >> org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:2834)
>> >> at
>> >>
>> >> org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:2922)
>> >> at
>> >> org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:2897)
>> >> at
>> >>
>> >> org.apache.lucene.index.TestNumericDocValuesUpdates$2.run(TestNumericDocValuesUpdates.java:957)
>> >>
>> >>
>> >>
>> >>
>> >> Build Log:
>> >> [...truncated 1672 lines...]
>> >>[junit4] Suite: org.apache.lucene.index.TestNumericDocValuesUpdates
>> >>[junit4]   2> 9 20, 0025 3:45:48 ??
>> >>
>> >> com.carrotsearch.randomizedtesting.RandomizedRunner$QueueUncaughtExceptionsHandler
>> >> uncaughtException
>> >>[junit4]   2> WARNING: Uncaught exception in thread:
>> >> Thread[UpdateThread-1,5,TGRP-TestNumericDocValuesUpdates]
>> >>[junit4]   2> java.lang.OutOfMemoryError: Java heap space
>> >>[junit4]   2>at
>> >> __randomizedtesting.SeedInfo.seed([EA6A02F6820CB4B7]:0)
>> >>[junit4]   2>at java.util.Arrays.copyOf(Arrays.java:2367)
>> >>[junit4]   2>at
>> >>
>> >> java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:130)
>> >>[junit4]   2>at
>> >>
>> >> java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:114)
>> >>[junit4]   2>at
>> >> java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:415)
>> >>

[jira] [Commented] (LUCENE-5234) Clarify FieldCache API around the use of NumericDocValues fields

2013-09-21 Thread ASF subversion and git services (JIRA)

[ 
https://issues.apache.org/jira/browse/LUCENE-5234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13773877#comment-13773877
 ] 

ASF subversion and git services commented on LUCENE-5234:
-

Commit 1525284 from [~shaie] in branch 'dev/trunk'
[ https://svn.apache.org/r1525284 ]

LUCENE-5234: improve FieldCache API

> Clarify FieldCache API around the use of NumericDocValues fields
> 
>
> Key: LUCENE-5234
> URL: https://issues.apache.org/jira/browse/LUCENE-5234
> Project: Lucene - Core
>  Issue Type: Improvement
>  Components: core/search
>Reporter: Shai Erera
>Assignee: Shai Erera
> Attachments: LUCENE-5234.patch, LUCENE-5234.patch
>
>
> Spinoff from this thread: http://lucene.markmail.org/thread/wxs6bzf2ul6go4pg. 
> FieldCache (and friends) API javadocs need some improvements.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org



Re: [JENKINS] Lucene-Solr-NightlyTests-trunk - Build # 386 - Failure

2013-09-21 Thread Shai Erera
I thought about that, but then realized that the test does all the merging
before the update threads start. This test only verifies that many
concurrent updates (to same docs, fields etc.) works well.
testSegmentMerging tests that concurrent updates w/ segment merging works
well. So there really isn't a good reason for force many segments, and w/
maxBufDocs=2 and numDocs=atLeast(2000), that means lots of merging, all
happen before the update threads start anyway.

Shai


On Sat, Sep 21, 2013 at 8:01 PM, Michael McCandless <
luc...@mikemccandless.com> wrote:

> Maybe we can make maxBufDocs greater than 2 but still smallish?  Just
> to make sure we are stressing merging + updating ...
>
> Mike McCandless
>
> http://blog.mikemccandless.com
>
>
> On Sat, Sep 21, 2013 at 9:05 AM, Shai Erera  wrote:
> > I don't hit an OOM with the reported parameters, but I do see the test
> takes
> > extremely long time to finish (I killed it after >10 minutes!).
> >
> > With the reported parameters, I see that the test uses
> SerialMergeScheduler,
> > spawns 18 threads and indexes 13K docs. Also, each thread calls
> IW.updateNDV
> > 197 times. The test also hard-sets maxBufDocs to 2, which means a lot of
> > segment merging. With these numbers, it just takes too long to complete
> (I
> > see it makes progress, slowly). Maybe that, together with other tests
> > running on Jenkins, can result in OOM.
> >
> > Anyway, I'd like to remove the hard-coded maxBufDocs=2. With
> > numDocs=atLeast(2000), we should get few segments. Even with that, the
> test
> > takes 4m to complete (w/ those settings). Maybe it's too stressing?
> >
> > Shai
> >
> >
> > On Fri, Sep 20, 2013 at 3:48 PM, Apache Jenkins Server
> >  wrote:
> >>
> >> Build:
> https://builds.apache.org/job/Lucene-Solr-NightlyTests-trunk/386/
> >>
> >> 1 tests failed.
> >> REGRESSION:
> >>
> org.apache.lucene.index.TestNumericDocValuesUpdates.testStressMultiThreading
> >>
> >> Error Message:
> >> Captured an uncaught exception in thread: Thread[id=3130,
> >> name=UpdateThread-1, state=RUNNABLE,
> group=TGRP-TestNumericDocValuesUpdates]
> >>
> >> Stack Trace:
> >> com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an
> >> uncaught exception in thread: Thread[id=3130, name=UpdateThread-1,
> >> state=RUNNABLE, group=TGRP-TestNumericDocValuesUpdates]
> >> Caused by: java.lang.OutOfMemoryError: Java heap space
> >> at __randomizedtesting.SeedInfo.seed([EA6A02F6820CB4B7]:0)
> >> at java.util.Arrays.copyOf(Arrays.java:2367)
> >> at
> >>
> java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:130)
> >> at
> >>
> java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:114)
> >> at
> >> java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:415)
> >> at java.lang.StringBuilder.append(StringBuilder.java:132)
> >> at java.lang.StringBuilder.append(StringBuilder.java:128)
> >> at
> >> java.util.AbstractCollection.toString(AbstractCollection.java:450)
> >> at java.lang.String.valueOf(String.java:2854)
> >> at java.lang.StringBuilder.append(StringBuilder.java:128)
> >> at
> >> org.apache.lucene.index.IndexWriter.startCommit(IndexWriter.java:4239)
> >> at
> >>
> org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:2834)
> >> at
> >>
> org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:2922)
> >> at
> >> org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:2897)
> >> at
> >>
> org.apache.lucene.index.TestNumericDocValuesUpdates$2.run(TestNumericDocValuesUpdates.java:957)
> >>
> >>
> >>
> >>
> >> Build Log:
> >> [...truncated 1672 lines...]
> >>[junit4] Suite: org.apache.lucene.index.TestNumericDocValuesUpdates
> >>[junit4]   2> 9 20, 0025 3:45:48 ??
> >>
> com.carrotsearch.randomizedtesting.RandomizedRunner$QueueUncaughtExceptionsHandler
> >> uncaughtException
> >>[junit4]   2> WARNING: Uncaught exception in thread:
> >> Thread[UpdateThread-1,5,TGRP-TestNumericDocValuesUpdates]
> >>[junit4]   2> java.lang.OutOfMemoryError: Java heap space
> >>[junit4]   2>at
> >> __randomizedtesting.SeedInfo.seed([EA6A02F6820CB4B7]:0)
> >>[junit4]   2>at java.util.Arrays.copyOf(Arrays.java:2367)
> >>[junit4]   2>at
> >>
> java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:130)
> >>[junit4]   2>at
> >>
> java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:114)
> >>[junit4]   2>at
> >> java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:415)
> >>[junit4]   2>at
> >> java.lang.StringBuilder.append(StringBuilder.java:132)
> >>[junit4]   2>at
> >> java.lang.StringBuilder.append(StringBuilder.java:128)
> >>[junit4]   2>at
> >> java.util.AbstractCollection.toString(AbstractCollection.java:450)
> >>[junit

[jira] [Updated] (LUCENE-5235) throw illegalstate from Tokenizer (instead of NPE/IIOBE) if reset not called

2013-09-21 Thread Robert Muir (JIRA)

 [ 
https://issues.apache.org/jira/browse/LUCENE-5235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Muir updated LUCENE-5235:


Attachment: LUCENE-5235.patch

here's an improved patch. I think this is easier and works for custom 
tokenizers too.

some tests should fail: not all tokenizers fixed yet (we have to revert the 
null/AIOOBE hacks and call super.reset)

> throw illegalstate from Tokenizer (instead of NPE/IIOBE) if reset not called
> 
>
> Key: LUCENE-5235
> URL: https://issues.apache.org/jira/browse/LUCENE-5235
> Project: Lucene - Core
>  Issue Type: Improvement
>  Components: modules/analysis
>Reporter: Robert Muir
> Attachments: LUCENE-5235.patch, LUCENE-5235.patch, 
> LUCENE-5235_test.patch
>
>
> We added these best effort checks, but it would be much better if we somehow 
> gave a clear exception... this comes up often

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org



[jira] [Commented] (LUCENE-5215) Add support for FieldInfos generation

2013-09-21 Thread Michael McCandless (JIRA)

[ 
https://issues.apache.org/jira/browse/LUCENE-5215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13773854#comment-13773854
 ] 

Michael McCandless commented on LUCENE-5215:


bq. BTW, I noticed that TestBackCompat suppresses Lucene41 and Lucene42. I ran 
it with -Dtests.codec=Lucene45 and it passed, so I'm not sure if I should add 
the now deprecated Lucene45Codec to the suppress list?

I think just add Lucene45Codec to the suppress list?  This test is already 
testing old indices ... it doesn't need to then test that writing (using the 
impersonators) to old formats works ok?  I think?

> Add support for FieldInfos generation
> -
>
> Key: LUCENE-5215
> URL: https://issues.apache.org/jira/browse/LUCENE-5215
> Project: Lucene - Core
>  Issue Type: New Feature
>  Components: core/index
>Reporter: Shai Erera
>Assignee: Shai Erera
> Attachments: LUCENE-5215.patch, LUCENE-5215.patch, LUCENE-5215.patch, 
> LUCENE-5215.patch
>
>
> In LUCENE-5189 we've identified few reasons to do that:
> # If you want to update docs' values of field 'foo', where 'foo' exists in 
> the index, but not in a specific segment (sparse DV), we cannot allow that 
> and have to throw a late UOE. If we could rewrite FieldInfos (with 
> generation), this would be possible since we'd also write a new generation of 
> FIS.
> # When we apply NDV updates, we call DVF.fieldsConsumer. Currently the 
> consumer isn't allowed to change FI.attributes because we cannot modify the 
> existing FIS. This is implicit however, and we silently ignore any modified 
> attributes. FieldInfos.gen will allow that too.
> The idea is to add to SIPC fieldInfosGen, add to each FieldInfo a dvGen and 
> add support for FIS generation in FieldInfosFormat, SegReader etc., like we 
> now do for DocValues. I'll work on a patch.
> Also on LUCENE-5189, Rob raised a concern about SegmentInfo.attributes that 
> have same limitation -- if a Codec modifies them, they are silently being 
> ignored, since we don't gen the .si files. I think we can easily solve that 
> by recording SI.attributes in SegmentInfos, so they are recorded per-commit. 
> But I think it should be handled in a separate issue.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org



[jira] [Commented] (LUCENE-5235) throw illegalstate from Tokenizer (instead of NPE/IIOBE) if reset not called

2013-09-21 Thread Uwe Schindler (JIRA)

[ 
https://issues.apache.org/jira/browse/LUCENE-5235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13773853#comment-13773853
 ] 

Uwe Schindler commented on LUCENE-5235:
---

First stab with fixing common Tokenizers. Robert will upload a test patch soon.

> throw illegalstate from Tokenizer (instead of NPE/IIOBE) if reset not called
> 
>
> Key: LUCENE-5235
> URL: https://issues.apache.org/jira/browse/LUCENE-5235
> Project: Lucene - Core
>  Issue Type: Improvement
>  Components: modules/analysis
>Reporter: Robert Muir
> Attachments: LUCENE-5235.patch, LUCENE-5235_test.patch
>
>
> We added these best effort checks, but it would be much better if we somehow 
> gave a clear exception... this comes up often

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org



[jira] [Updated] (LUCENE-5235) throw illegalstate from Tokenizer (instead of NPE/IIOBE) if reset not called

2013-09-21 Thread Robert Muir (JIRA)

 [ 
https://issues.apache.org/jira/browse/LUCENE-5235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Muir updated LUCENE-5235:


Attachment: LUCENE-5235_test.patch

modification of BaseTokenSTreamTestCase to check for this... I think its 
correct...

> throw illegalstate from Tokenizer (instead of NPE/IIOBE) if reset not called
> 
>
> Key: LUCENE-5235
> URL: https://issues.apache.org/jira/browse/LUCENE-5235
> Project: Lucene - Core
>  Issue Type: Improvement
>  Components: modules/analysis
>Reporter: Robert Muir
> Attachments: LUCENE-5235.patch, LUCENE-5235_test.patch
>
>
> We added these best effort checks, but it would be much better if we somehow 
> gave a clear exception... this comes up often

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org



Re: [JENKINS] Lucene-Solr-NightlyTests-trunk - Build # 386 - Failure

2013-09-21 Thread Michael McCandless
Maybe we can make maxBufDocs greater than 2 but still smallish?  Just
to make sure we are stressing merging + updating ...

Mike McCandless

http://blog.mikemccandless.com


On Sat, Sep 21, 2013 at 9:05 AM, Shai Erera  wrote:
> I don't hit an OOM with the reported parameters, but I do see the test takes
> extremely long time to finish (I killed it after >10 minutes!).
>
> With the reported parameters, I see that the test uses SerialMergeScheduler,
> spawns 18 threads and indexes 13K docs. Also, each thread calls IW.updateNDV
> 197 times. The test also hard-sets maxBufDocs to 2, which means a lot of
> segment merging. With these numbers, it just takes too long to complete (I
> see it makes progress, slowly). Maybe that, together with other tests
> running on Jenkins, can result in OOM.
>
> Anyway, I'd like to remove the hard-coded maxBufDocs=2. With
> numDocs=atLeast(2000), we should get few segments. Even with that, the test
> takes 4m to complete (w/ those settings). Maybe it's too stressing?
>
> Shai
>
>
> On Fri, Sep 20, 2013 at 3:48 PM, Apache Jenkins Server
>  wrote:
>>
>> Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-trunk/386/
>>
>> 1 tests failed.
>> REGRESSION:
>> org.apache.lucene.index.TestNumericDocValuesUpdates.testStressMultiThreading
>>
>> Error Message:
>> Captured an uncaught exception in thread: Thread[id=3130,
>> name=UpdateThread-1, state=RUNNABLE, group=TGRP-TestNumericDocValuesUpdates]
>>
>> Stack Trace:
>> com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an
>> uncaught exception in thread: Thread[id=3130, name=UpdateThread-1,
>> state=RUNNABLE, group=TGRP-TestNumericDocValuesUpdates]
>> Caused by: java.lang.OutOfMemoryError: Java heap space
>> at __randomizedtesting.SeedInfo.seed([EA6A02F6820CB4B7]:0)
>> at java.util.Arrays.copyOf(Arrays.java:2367)
>> at
>> java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:130)
>> at
>> java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:114)
>> at
>> java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:415)
>> at java.lang.StringBuilder.append(StringBuilder.java:132)
>> at java.lang.StringBuilder.append(StringBuilder.java:128)
>> at
>> java.util.AbstractCollection.toString(AbstractCollection.java:450)
>> at java.lang.String.valueOf(String.java:2854)
>> at java.lang.StringBuilder.append(StringBuilder.java:128)
>> at
>> org.apache.lucene.index.IndexWriter.startCommit(IndexWriter.java:4239)
>> at
>> org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:2834)
>> at
>> org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:2922)
>> at
>> org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:2897)
>> at
>> org.apache.lucene.index.TestNumericDocValuesUpdates$2.run(TestNumericDocValuesUpdates.java:957)
>>
>>
>>
>>
>> Build Log:
>> [...truncated 1672 lines...]
>>[junit4] Suite: org.apache.lucene.index.TestNumericDocValuesUpdates
>>[junit4]   2> 9 20, 0025 3:45:48 ??
>> com.carrotsearch.randomizedtesting.RandomizedRunner$QueueUncaughtExceptionsHandler
>> uncaughtException
>>[junit4]   2> WARNING: Uncaught exception in thread:
>> Thread[UpdateThread-1,5,TGRP-TestNumericDocValuesUpdates]
>>[junit4]   2> java.lang.OutOfMemoryError: Java heap space
>>[junit4]   2>at
>> __randomizedtesting.SeedInfo.seed([EA6A02F6820CB4B7]:0)
>>[junit4]   2>at java.util.Arrays.copyOf(Arrays.java:2367)
>>[junit4]   2>at
>> java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:130)
>>[junit4]   2>at
>> java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:114)
>>[junit4]   2>at
>> java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:415)
>>[junit4]   2>at
>> java.lang.StringBuilder.append(StringBuilder.java:132)
>>[junit4]   2>at
>> java.lang.StringBuilder.append(StringBuilder.java:128)
>>[junit4]   2>at
>> java.util.AbstractCollection.toString(AbstractCollection.java:450)
>>[junit4]   2>at java.lang.String.valueOf(String.java:2854)
>>[junit4]   2>at
>> java.lang.StringBuilder.append(StringBuilder.java:128)
>>[junit4]   2>at
>> org.apache.lucene.index.IndexWriter.startCommit(IndexWriter.java:4239)
>>[junit4]   2>at
>> org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:2834)
>>[junit4]   2>at
>> org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:2922)
>>[junit4]   2>at
>> org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:2897)
>>[junit4]   2>at
>> org.apache.lucene.index.TestNumericDocValuesUpdates$2.run(TestNumericDocValuesUpdates.java:957)
>>[junit4]   2>
>>[junit4]   2> 9 20, 0025 3:47:09 ??
>> com.carrotsearch.ran

[jira] [Updated] (LUCENE-5235) throw illegalstate from Tokenizer (instead of NPE/IIOBE) if reset not called

2013-09-21 Thread Uwe Schindler (JIRA)

 [ 
https://issues.apache.org/jira/browse/LUCENE-5235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Uwe Schindler updated LUCENE-5235:
--

Attachment: LUCENE-5235.patch

> throw illegalstate from Tokenizer (instead of NPE/IIOBE) if reset not called
> 
>
> Key: LUCENE-5235
> URL: https://issues.apache.org/jira/browse/LUCENE-5235
> Project: Lucene - Core
>  Issue Type: Improvement
>  Components: modules/analysis
>Reporter: Robert Muir
> Attachments: LUCENE-5235.patch
>
>
> We added these best effort checks, but it would be much better if we somehow 
> gave a clear exception... this comes up often

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org



[jira] [Commented] (LUCENE-5235) throw illegalstate from Tokenizer (instead of NPE/IIOBE) if reset not called

2013-09-21 Thread Michael McCandless (JIRA)

[ 
https://issues.apache.org/jira/browse/LUCENE-5235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13773851#comment-13773851
 ] 

Michael McCandless commented on LUCENE-5235:


+1

> throw illegalstate from Tokenizer (instead of NPE/IIOBE) if reset not called
> 
>
> Key: LUCENE-5235
> URL: https://issues.apache.org/jira/browse/LUCENE-5235
> Project: Lucene - Core
>  Issue Type: Improvement
>  Components: modules/analysis
>Reporter: Robert Muir
>
> We added these best effort checks, but it would be much better if we somehow 
> gave a clear exception... this comes up often

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org



[jira] [Commented] (LUCENE-5234) Clarify FieldCache API around the use of NumericDocValues fields

2013-09-21 Thread Robert Muir (JIRA)

[ 
https://issues.apache.org/jira/browse/LUCENE-5234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13773849#comment-13773849
 ] 

Robert Muir commented on LUCENE-5234:
-

+1

> Clarify FieldCache API around the use of NumericDocValues fields
> 
>
> Key: LUCENE-5234
> URL: https://issues.apache.org/jira/browse/LUCENE-5234
> Project: Lucene - Core
>  Issue Type: Improvement
>  Components: core/search
>Reporter: Shai Erera
>Assignee: Shai Erera
> Attachments: LUCENE-5234.patch, LUCENE-5234.patch
>
>
> Spinoff from this thread: http://lucene.markmail.org/thread/wxs6bzf2ul6go4pg. 
> FieldCache (and friends) API javadocs need some improvements.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org



[jira] [Created] (LUCENE-5235) throw illegalstate from Tokenizer (instead of NPE/IIOBE) if reset not called

2013-09-21 Thread Robert Muir (JIRA)
Robert Muir created LUCENE-5235:
---

 Summary: throw illegalstate from Tokenizer (instead of NPE/IIOBE) 
if reset not called
 Key: LUCENE-5235
 URL: https://issues.apache.org/jira/browse/LUCENE-5235
 Project: Lucene - Core
  Issue Type: Improvement
  Components: modules/analysis
Reporter: Robert Muir


We added these best effort checks, but it would be much better if we somehow 
gave a clear exception... this comes up often

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org



[jira] [Commented] (SOLR-4882) Restrict SolrResourceLoader to only classloader accessible files and instance dir

2013-09-21 Thread ASF subversion and git services (JIRA)

[ 
https://issues.apache.org/jira/browse/SOLR-4882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13773832#comment-13773832
 ] 

ASF subversion and git services commented on SOLR-4882:
---

Commit 1525248 from [~thetaphi] in branch 'dev/branches/branch_4x'
[ https://svn.apache.org/r1525248 ]

Merged revision(s) 1525246 from lucene/dev/trunk:
SOLR-4882: Restrict SolrResourceLoader to only allow access to resource files 
below the instance dir

> Restrict SolrResourceLoader to only classloader accessible files and instance 
> dir
> -
>
> Key: SOLR-4882
> URL: https://issues.apache.org/jira/browse/SOLR-4882
> Project: Solr
>  Issue Type: Improvement
>Affects Versions: 4.3
>Reporter: Uwe Schindler
>Assignee: Uwe Schindler
> Fix For: 4.5, 5.0
>
> Attachments: SOLR-4882.patch, SOLR-4882.patch, SOLR-4882.patch
>
>
> SolrResourceLoader currently allows to load files from any 
> absolute/CWD-relative path, which is used as a fallback if the resource 
> cannot be looked up via the class loader.
> We should limit this fallback to sub-dirs below the instanceDir passed into 
> the ctor. The CWD special case should be removed, too (the virtual CWD is 
> instance's config or root dir).
> The reason for this is security related. Some Solr components allow to pass 
> in resource paths via REST parameters (e.g. XSL stylesheets, velocity 
> templates,...) and load them via resource loader. By this it is possible to 
> limit the whole thing to
> not allow loading e.g. /etc/passwd as a stylesheet.
> In 4.4 we should add a solrconfig.xml setting to enable the old behaviour, 
> but disable it by default, if your existing installation requires the files 
> from outside the instance dir which are not available via the URLClassLoader 
> used internally. In Lucene 5.0 we should not support this anymore.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org



[jira] [Resolved] (SOLR-4882) Restrict SolrResourceLoader to only classloader accessible files and instance dir

2013-09-21 Thread Uwe Schindler (JIRA)

 [ 
https://issues.apache.org/jira/browse/SOLR-4882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Uwe Schindler resolved SOLR-4882.
-

   Resolution: Fixed
Fix Version/s: (was: 4.5)
   4.6

> Restrict SolrResourceLoader to only classloader accessible files and instance 
> dir
> -
>
> Key: SOLR-4882
> URL: https://issues.apache.org/jira/browse/SOLR-4882
> Project: Solr
>  Issue Type: Improvement
>Affects Versions: 4.3
>Reporter: Uwe Schindler
>Assignee: Uwe Schindler
> Fix For: 5.0, 4.6
>
> Attachments: SOLR-4882.patch, SOLR-4882.patch, SOLR-4882.patch
>
>
> SolrResourceLoader currently allows to load files from any 
> absolute/CWD-relative path, which is used as a fallback if the resource 
> cannot be looked up via the class loader.
> We should limit this fallback to sub-dirs below the instanceDir passed into 
> the ctor. The CWD special case should be removed, too (the virtual CWD is 
> instance's config or root dir).
> The reason for this is security related. Some Solr components allow to pass 
> in resource paths via REST parameters (e.g. XSL stylesheets, velocity 
> templates,...) and load them via resource loader. By this it is possible to 
> limit the whole thing to
> not allow loading e.g. /etc/passwd as a stylesheet.
> In 4.4 we should add a solrconfig.xml setting to enable the old behaviour, 
> but disable it by default, if your existing installation requires the files 
> from outside the instance dir which are not available via the URLClassLoader 
> used internally. In Lucene 5.0 we should not support this anymore.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org



[jira] [Updated] (SOLR-4882) Restrict SolrResourceLoader to only classloader accessible files and instance dir

2013-09-21 Thread Uwe Schindler (JIRA)

 [ 
https://issues.apache.org/jira/browse/SOLR-4882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Uwe Schindler updated SOLR-4882:


Attachment: SOLR-4882.patch

Here is the final patch for trunk. I also added a test that checks for escaping 
instance dir

> Restrict SolrResourceLoader to only classloader accessible files and instance 
> dir
> -
>
> Key: SOLR-4882
> URL: https://issues.apache.org/jira/browse/SOLR-4882
> Project: Solr
>  Issue Type: Improvement
>Affects Versions: 4.3
>Reporter: Uwe Schindler
>Assignee: Uwe Schindler
> Fix For: 4.5, 5.0
>
> Attachments: SOLR-4882.patch, SOLR-4882.patch, SOLR-4882.patch
>
>
> SolrResourceLoader currently allows to load files from any 
> absolute/CWD-relative path, which is used as a fallback if the resource 
> cannot be looked up via the class loader.
> We should limit this fallback to sub-dirs below the instanceDir passed into 
> the ctor. The CWD special case should be removed, too (the virtual CWD is 
> instance's config or root dir).
> The reason for this is security related. Some Solr components allow to pass 
> in resource paths via REST parameters (e.g. XSL stylesheets, velocity 
> templates,...) and load them via resource loader. By this it is possible to 
> limit the whole thing to
> not allow loading e.g. /etc/passwd as a stylesheet.
> In 4.4 we should add a solrconfig.xml setting to enable the old behaviour, 
> but disable it by default, if your existing installation requires the files 
> from outside the instance dir which are not available via the URLClassLoader 
> used internally. In Lucene 5.0 we should not support this anymore.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org



Re: [JENKINS] Lucene-Solr-NightlyTests-trunk - Build # 386 - Failure

2013-09-21 Thread Shai Erera
I don't hit an OOM with the reported parameters, but I do see the test
takes extremely long time to finish (I killed it after >10 minutes!).

With the reported parameters, I see that the test uses
SerialMergeScheduler, spawns 18 threads and indexes 13K docs. Also, each
thread calls IW.updateNDV 197 times. The test also hard-sets maxBufDocs to
2, which means a lot of segment merging. With these numbers, it just takes
too long to complete (I see it makes progress, slowly). Maybe that,
together with other tests running on Jenkins, can result in OOM.

Anyway, I'd like to remove the hard-coded maxBufDocs=2. With
numDocs=atLeast(2000), we should get few segments. Even with that, the test
takes 4m to complete (w/ those settings). Maybe it's too stressing?

Shai


On Fri, Sep 20, 2013 at 3:48 PM, Apache Jenkins Server <
jenk...@builds.apache.org> wrote:

> Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-trunk/386/
>
> 1 tests failed.
> REGRESSION:
>  org.apache.lucene.index.TestNumericDocValuesUpdates.testStressMultiThreading
>
> Error Message:
> Captured an uncaught exception in thread: Thread[id=3130,
> name=UpdateThread-1, state=RUNNABLE, group=TGRP-TestNumericDocValuesUpdates]
>
> Stack Trace:
> com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an
> uncaught exception in thread: Thread[id=3130, name=UpdateThread-1,
> state=RUNNABLE, group=TGRP-TestNumericDocValuesUpdates]
> Caused by: java.lang.OutOfMemoryError: Java heap space
> at __randomizedtesting.SeedInfo.seed([EA6A02F6820CB4B7]:0)
> at java.util.Arrays.copyOf(Arrays.java:2367)
> at
> java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:130)
> at
> java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:114)
> at
> java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:415)
> at java.lang.StringBuilder.append(StringBuilder.java:132)
> at java.lang.StringBuilder.append(StringBuilder.java:128)
> at
> java.util.AbstractCollection.toString(AbstractCollection.java:450)
> at java.lang.String.valueOf(String.java:2854)
> at java.lang.StringBuilder.append(StringBuilder.java:128)
> at
> org.apache.lucene.index.IndexWriter.startCommit(IndexWriter.java:4239)
> at
> org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:2834)
> at
> org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:2922)
> at
> org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:2897)
> at
> org.apache.lucene.index.TestNumericDocValuesUpdates$2.run(TestNumericDocValuesUpdates.java:957)
>
>
>
>
> Build Log:
> [...truncated 1672 lines...]
>[junit4] Suite: org.apache.lucene.index.TestNumericDocValuesUpdates
>[junit4]   2> 9 20, 0025 3:45:48 ??
> com.carrotsearch.randomizedtesting.RandomizedRunner$QueueUncaughtExceptionsHandler
> uncaughtException
>[junit4]   2> WARNING: Uncaught exception in thread:
> Thread[UpdateThread-1,5,TGRP-TestNumericDocValuesUpdates]
>[junit4]   2> java.lang.OutOfMemoryError: Java heap space
>[junit4]   2>at
> __randomizedtesting.SeedInfo.seed([EA6A02F6820CB4B7]:0)
>[junit4]   2>at java.util.Arrays.copyOf(Arrays.java:2367)
>[junit4]   2>at
> java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:130)
>[junit4]   2>at
> java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:114)
>[junit4]   2>at
> java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:415)
>[junit4]   2>at
> java.lang.StringBuilder.append(StringBuilder.java:132)
>[junit4]   2>at
> java.lang.StringBuilder.append(StringBuilder.java:128)
>[junit4]   2>at
> java.util.AbstractCollection.toString(AbstractCollection.java:450)
>[junit4]   2>at java.lang.String.valueOf(String.java:2854)
>[junit4]   2>at
> java.lang.StringBuilder.append(StringBuilder.java:128)
>[junit4]   2>at
> org.apache.lucene.index.IndexWriter.startCommit(IndexWriter.java:4239)
>[junit4]   2>at
> org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:2834)
>[junit4]   2>at
> org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:2922)
>[junit4]   2>at
> org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:2897)
>[junit4]   2>at
> org.apache.lucene.index.TestNumericDocValuesUpdates$2.run(TestNumericDocValuesUpdates.java:957)
>[junit4]   2>
>[junit4]   2> 9 20, 0025 3:47:09 ??
> com.carrotsearch.randomizedtesting.RandomizedRunner$QueueUncaughtExceptionsHandler
> uncaughtException
>[junit4]   2> WARNING: Uncaught exception in thread:
> Thread[UpdateThread-3,5,TGRP-TestNumericDocValuesUpdates]
>[junit4]   2> java.lang.IllegalStateException: this writer hit an
> OutOfMemoryError; cannot commit
>[jun

[jira] [Updated] (LUCENE-5234) Clarify FieldCache API around the use of NumericDocValues fields

2013-09-21 Thread Shai Erera (JIRA)

 [ 
https://issues.apache.org/jira/browse/LUCENE-5234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shai Erera updated LUCENE-5234:
---

Attachment: LUCENE-5234.patch

Patch improves the javadocs of Int/Float/Long/DoubleFieldSource as well as 
FieldCache.getInts/Floats/Longs/Doubles(). If there are no objections, I'll 
commit it later.

> Clarify FieldCache API around the use of NumericDocValues fields
> 
>
> Key: LUCENE-5234
> URL: https://issues.apache.org/jira/browse/LUCENE-5234
> Project: Lucene - Core
>  Issue Type: Improvement
>  Components: core/search
>Reporter: Shai Erera
>Assignee: Shai Erera
> Attachments: LUCENE-5234.patch, LUCENE-5234.patch
>
>
> Spinoff from this thread: http://lucene.markmail.org/thread/wxs6bzf2ul6go4pg. 
> FieldCache (and friends) API javadocs need some improvements.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org



[jira] [Commented] (LUCENE-5233) we should change the suggested search in the demo docs because the lucene code base is full of swear words

2013-09-21 Thread Michael McCandless (JIRA)

[ 
https://issues.apache.org/jira/browse/LUCENE-5233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13773803#comment-13773803
 ] 

Michael McCandless commented on LUCENE-5233:


bq.  (and i have a pretty extensive breadth of knowledge of profanity)

LOL :)

+1 to fix the demo docs!

> we should change the suggested search in the demo docs because the lucene 
> code base is full of swear words
> --
>
> Key: LUCENE-5233
> URL: https://issues.apache.org/jira/browse/LUCENE-5233
> Project: Lucene - Core
>  Issue Type: Bug
>Reporter: Hoss Man
>Priority: Trivial
>
> the javadocs for the lucene demo say...
> bq. You'll be prompted for a query. Type in a swear word and press the enter 
> key. You'll see that the Lucene developers are very well mannered and get no 
> results. Now try entering the word "string". That should return a whole bunch 
> of documents. The results will page at every tenth result and ask you whether 
> you want more results.
> ...but thanks to files like "KStemData*.java" and "Top50KWiki.utf8" i was 
> *really* hard pressed to find an (english) swear word that didn't result in a 
> match in any of the files in the lucene code base (and i have a pretty 
> extensive breadth of knowledge of profanity)
> We should change this paragraph to refer to something that is total giberish 
> ("supercalifragilisticexpialidocious")... or maybe just "nocommit"
> (side note: since this para exists in the javadoc package comments, it will 
> get picked up when they index the source -- so we should include an HTML 
> comment in the middle of whatever word is picked)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org