date:20130306


[ 
https://issues.apache.org/jira/browse/SOLR-4506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13595633#comment-13595633
 ] 

Mark Miller commented on SOLR-4506:
---

Yup, I can try and deal with this for 4.3. 4.2 is rolling now and it will take 
me some time to get this worked out.

> [solr4.0.0] many index.{date} dir in replication node 
> --
>
> Key: SOLR-4506
> URL: https://issues.apache.org/jira/browse/SOLR-4506
> Project: Solr
>  Issue Type: Bug
>  Components: SolrCloud
>Affects Versions: 4.0
> Environment: the solr4.0 runs on suse11.
> mem:32G
> cpu:16 cores
>Reporter: zhuojunjian
>Assignee: Mark Miller
> Fix For: 4.0, 4.3
>
>   Original Estimate: 12h
>  Remaining Estimate: 12h
>
> in our test,we used solrcloud feature in solr4.0(version detail 
> :4.0.0.2012.10.06.03.04.33).
> the solrcloud configuration is 3 shards and 2 replications each shard.
> we found that there are over than 25 dirs which named index.{date} in one 
> replication node belonging to shard 3. 
> for example:
> index.2013021725864  index.20130218012211880  index.20130218015714713  
> index.20130218023101958  index.20130218030424083  tlog
> index.20130218005648324  index.20130218012751078  index.20130218020141293  
> the issue seems like SOLR-1781. but it is fixed in 4.0-BETA,5.0. 
> so is solr4.0 ? if it is fixed too in solr4.0, why we find the issue again ?
> what can I do?   

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-4506) [solr4.0.0] many index.{date} dir in replication node


 [ 
https://issues.apache.org/jira/browse/SOLR-4506?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mark Miller updated SOLR-4506:
--

Fix Version/s: 4.3
 Assignee: Mark Miller

> [solr4.0.0] many index.{date} dir in replication node 
> --
>
> Key: SOLR-4506
> URL: https://issues.apache.org/jira/browse/SOLR-4506
> Project: Solr
>  Issue Type: Bug
>  Components: SolrCloud
>Affects Versions: 4.0
> Environment: the solr4.0 runs on suse11.
> mem:32G
> cpu:16 cores
>Reporter: zhuojunjian
>Assignee: Mark Miller
> Fix For: 4.0, 4.3
>
>   Original Estimate: 12h
>  Remaining Estimate: 12h
>
> in our test,we used solrcloud feature in solr4.0(version detail 
> :4.0.0.2012.10.06.03.04.33).
> the solrcloud configuration is 3 shards and 2 replications each shard.
> we found that there are over than 25 dirs which named index.{date} in one 
> replication node belonging to shard 3. 
> for example:
> index.2013021725864  index.20130218012211880  index.20130218015714713  
> index.20130218023101958  index.20130218030424083  tlog
> index.20130218005648324  index.20130218012751078  index.20130218020141293  
> the issue seems like SOLR-1781. but it is fixed in 4.0-BETA,5.0. 
> so is solr4.0 ? if it is fixed too in solr4.0, why we find the issue again ?
> what can I do?   

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

RE: JVM hanging when using G1GC on JDK8 b78 or b79 (Linux 32 bit)

Hi John,

 

I only have time to work on a setup this evening Germen time, because I am on a 
business trip today. Will come back to you. Unfortunately I failed to quickly 
setup an easy classpath without Ivy downloading the JARS. 

 

Uwe

 

-

Uwe Schindler

uschind...@apache.org 

Apache Lucene PMC Member / Committer

Bremen, Germany

http://lucene.apache.org/

 

From: John Cuthbertson [mailto:john.cuthbert...@oracle.com] 
Sent: Thursday, March 07, 2013 12:49 AM
To: Uwe Schindler
Cc: 'Bengt Rutisson'; hotspot-gc-...@openjdk.java.net; dev@lucene.apache.org
Subject: Re: JVM hanging when using G1GC on JDK8 b78 or b79 (Linux 32 bit)

 

Hi Uwe,

An update:

I have downloaded ant and the lucerne source.

I attempted the ivy-bootstrap but it failed to download the ivy=2.3.0.jar file 
- even after setting:

ANT_OPTS=-Dhttp.proxyHost=<...> -Dhttp.proxyPort=<...>

So I manually downloaded and placed it into the ANT library and now get:




ivy-bootstrap1:
[mkdir] Skipping /home/jcuthber/.ant/lib because it already exists.
 [echo] installing ivy 2.3.0 to /home/jcuthber/.ant/lib
  [get] Getting: 
http://repo1.maven.org/maven2/org/apache/ivy/ivy/2.3.0/ivy-2.3.0.jar
  [get] To: /home/jcuthber/.ant/lib/ivy-2.3.0.jar
  [get] Error getting 
http://repo1.maven.org/maven2/org/apache/ivy/ivy/2.3.0/ivy-2.3.0.jar to 
/home/jcuthber/.ant/lib/ivy-2.3.0.jar
[available] Found: /home/jcuthber/.ant/lib/ivy-2.3.0.jar

ivy-bootstrap2:
Skipped because property 'ivy.bootstrap1.success' set.

ivy-checksum:

ivy-bootstrap:

BUILD SUCCESSFUL
Total time: 3 minutes 46 seconds

Presumably I have to build the lucerne source before executing the tests. That 
seemed to go OK.

When I run the analysis/uima tests it seems to get hung up at the "resolve" 
target - even without specifying G1:




cairnapple{jcuthber}:408> cd analysis/uima/
cairnapple{jcuthber}:409> ls -l
total 29
-rw-r--r--   1 jcuthber staff   1473 Dec 10 10:39 build.xml
-rw-rw-r--   1 jcuthber staff   6895 Mar  6 15:20 hotspot.log
-rw-r--r--   1 jcuthber staff   1316 Mar 30  2012 ivy.xml
drwxr-xr-x   2 jcuthber staff  2 Mar  5 07:42 lib/
drwxr-xr-x   6 jcuthber staff  6 Mar  5 07:42 src/





ivy-configure:
[ivy:configure] Loading 
jar:file:/home/jcuthber/.ant/lib/ivy-2.3.0.jar!/org/apache/ivy/core/settings/ivy.properties
[ivy:configure] :: Apache Ivy 2.3.0 - 20130110142753 :: 
http://ant.apache.org/ivy/ ::
[ivy:configure] jakarta commons httpclient not found: using jdk url handling
[ivy:configure] :: loading settings :: file = 
/export/bugs/8009536/lucene-5.0-2013-03-05_15-37-06/ivy-settings.xml
[ivy:configure] no default ivy user dir defined: set to /home/jcuthber/.ivy2
[ivy:configure] including url: 
jar:file:/home/jcuthber/.ant/lib/ivy-2.3.0.jar!/org/apache/ivy/core/settings/ivysettings-public.xml
[ivy:configure] no default cache defined: set to /home/jcuthber/.ivy2/cache
[ivy:configure] including url: 
jar:file:/home/jcuthber/.ant/lib/ivy-2.3.0.jar!/org/apache/ivy/core/settings/ivysettings-shared.xml
[ivy:configure] including url: 
jar:file:/home/jcuthber/.ant/lib/ivy-2.3.0.jar!/org/apache/ivy/core/settings/ivysettings-local.xml
[ivy:configure] including url: 
jar:file:/home/jcuthber/.ant/lib/ivy-2.3.0.jar!/org/apache/ivy/core/settings/ivysettings-main-chain.xml
[ivy:configure] settings loaded (289ms)
[ivy:configure] default cache: /home/jcuthber/.ivy2/cache
[ivy:configure] default resolver: default
[ivy:configure] -- 7 resolvers:
[ivy:configure] working-chinese-mirror [ibiblio]
[ivy:configure] main [chain] [shared, public]
[ivy:configure] local [file]
[ivy:configure] shared [file]
[ivy:configure] sonatype-releases [ibiblio]
[ivy:configure] public [ibiblio]
[ivy:configure] default [chain] [local, main, sonatype-releases, 
working-chinese-mirror]

resolve:
[ivy:retrieve] no resolved descriptor found: launching default resolve
Overriding previous definition of property "ivy.version"
[ivy:retrieve] using ivy parser to parse 
file:/export/bugs/8009536/lucene-5.0-2013-03-05_15-37-06/analysis/uima/ivy.xml 

 
[ivy:retrieve] :: resolving dependencies :: 
org.apache.lucene#analyzers-uima;working@cairnapple
[ivy:retrieve]  confs: [default]
[ivy:retrieve]  validate = true
[ivy:retrieve]  refresh = false
[ivy:retrieve] resolving dependencies for configuration 'default'
[ivy:retrieve] == resolving dependencies for 
org.apache.lucene#analyzers-uima;working@cairnapple [default]
[ivy:retrieve] == resolving dependencies 
org.apache.lucene#analyzers-uima;working@cairnapple->org.apache.uima#Tagger;2.3.1
 [default->*]
[ivy:retrieve] default: Checking cache for: dependency: 
org.apache.uima#Tagger;2.3.1 {*=[*]}
[ivy:retrieve] don't use cache for org.apache.uima#Tagger;2.3.1: 
checkModified=true
[ivy:retrieve]  tried 
/home/jcuthber/.ivy2/local/org.apache.uima/Tagger/2.3.1/ivys/ivy.xml
[ivy:retrieve]  tried 
/home/jcuthber/.ivy2/local/org.apache.

[jira] [Commented] (SOLR-4497) Collection Aliasing.


[ 
https://issues.apache.org/jira/browse/SOLR-4497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13595587#comment-13595587
 ] 

Commit Tag Bot commented on SOLR-4497:
--

[branch_4x commit] Mark Robert Miller
http://svn.apache.org/viewvc?view=revision&revision=1453691

SOLR-4497: Collection Aliasing.


> Collection Aliasing.
> 
>
> Key: SOLR-4497
> URL: https://issues.apache.org/jira/browse/SOLR-4497
> Project: Solr
>  Issue Type: New Feature
>  Components: SolrCloud
>Reporter: Mark Miller
>Assignee: Mark Miller
> Fix For: 4.2, 5.0
>
> Attachments: CDH-4497.patch, SOLR-4497.patch
>
>
> We should bring back the old aliasing feature, but for SolrCloud and with the 
> ability to alias one collection to many.
> The old alias feature was of more limited use and had some problems, so we 
> dropped it, but I think we can do this in a more useful way with SolrCloud, 
> and at a level were it's not invasive to the CoreContainer.
> Initially, the search side will allowing mapping a single alias to multiple 
> collections, but the index side will only support mapping a single alias to a 
> single collection.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-3918) Port index sorter to trunk APIs


[ 
https://issues.apache.org/jira/browse/LUCENE-3918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13595580#comment-13595580
 ] 

Shai Erera commented on LUCENE-3918:


I think it's ready. If there are no objections, I'd like to commit it later 
today.

> Port index sorter to trunk APIs
> ---
>
> Key: LUCENE-3918
> URL: https://issues.apache.org/jira/browse/LUCENE-3918
> Project: Lucene - Core
>  Issue Type: Task
>  Components: modules/other
>Affects Versions: 4.0-ALPHA
>Reporter: Robert Muir
>Assignee: Shai Erera
> Fix For: 4.2, 5.0
>
> Attachments: LUCENE-3918.patch, LUCENE-3918.patch, LUCENE-3918.patch, 
> LUCENE-3918.patch, LUCENE-3918.patch, LUCENE-3918.patch, LUCENE-3918.patch, 
> LUCENE-3918.patch, LUCENE-3918.patch, LUCENE-3918.patch, LUCENE-3918.patch, 
> LUCENE-3918.patch, LUCENE-3918.patch, LUCENE-3918.patch
>
>
> LUCENE-2482 added an IndexSorter to 3.x, but we need to port this
> functionality to 4.0 apis.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-4497) Collection Aliasing.


[ 
https://issues.apache.org/jira/browse/SOLR-4497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13595577#comment-13595577
 ] 

Commit Tag Bot commented on SOLR-4497:
--

[trunk commit] Mark Robert Miller
http://svn.apache.org/viewvc?view=revision&revision=1453687

SOLR-4497: Collection Aliasing.


> Collection Aliasing.
> 
>
> Key: SOLR-4497
> URL: https://issues.apache.org/jira/browse/SOLR-4497
> Project: Solr
>  Issue Type: New Feature
>  Components: SolrCloud
>Reporter: Mark Miller
>Assignee: Mark Miller
> Fix For: 4.2, 5.0
>
> Attachments: CDH-4497.patch, SOLR-4497.patch
>
>
> We should bring back the old aliasing feature, but for SolrCloud and with the 
> ability to alias one collection to many.
> The old alias feature was of more limited use and had some problems, so we 
> dropped it, but I think we can do this in a more useful way with SolrCloud, 
> and at a level were it's not invasive to the CoreContainer.
> Initially, the search side will allowing mapping a single alias to multiple 
> collections, but the index side will only support mapping a single alias to a 
> single collection.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-4497) Collection Aliasing.


[ 
https://issues.apache.org/jira/browse/SOLR-4497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13595572#comment-13595572
 ] 

Mark Miller commented on SOLR-4497:
---

Okay - committing the current state now.

bq. This lead me to wondering (again without understanding implications and 
design goals) would it be best to have the Overseer push the change to all 
nodes (and write to ZK) vs relying only on the change in ZK, which has delays.

I think this is something we should consider. It actually came up before with 
the clusterstate.json as well - I think we are actually okay with how that 
works currently though, but it may make sense to have the overseer to 
distribute this. Let's make a new JIRA issue and investigate it as a future 
improvement.

> Collection Aliasing.
> 
>
> Key: SOLR-4497
> URL: https://issues.apache.org/jira/browse/SOLR-4497
> Project: Solr
>  Issue Type: New Feature
>  Components: SolrCloud
>Reporter: Mark Miller
>Assignee: Mark Miller
> Fix For: 4.2, 5.0
>
> Attachments: CDH-4497.patch, SOLR-4497.patch
>
>
> We should bring back the old aliasing feature, but for SolrCloud and with the 
> ability to alias one collection to many.
> The old alias feature was of more limited use and had some problems, so we 
> dropped it, but I think we can do this in a more useful way with SolrCloud, 
> and at a level were it's not invasive to the CoreContainer.
> Initially, the search side will allowing mapping a single alias to multiple 
> collections, but the index side will only support mapping a single alias to a 
> single collection.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-4.x-Linux (64bit/ibm-j9-jdk7) - Build # 4573 - Failure!

2013-03-06 Thread Policeman Jenkins Server

Build: http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-Linux/4573/
Java: 64bit/ibm-j9-jdk7 

2 tests failed.
REGRESSION:  org.apache.solr.rest.TestFieldCollectionResource.testGetAllFields

Error Message:
Exception during query

Stack Trace:
java.lang.RuntimeException: Exception during query
at 
__randomizedtesting.SeedInfo.seed([118807F236C56EBF:FD264D36CC369727]:0)
at org.apache.solr.util.RestTestBase.assertQ(RestTestBase.java:143)
at 
org.apache.solr.rest.TestFieldCollectionResource.testGetAllFields(TestFieldCollectionResource.java:26)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:88)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
at java.lang.reflect.Method.invoke(Method.java:613)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at 
org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at java.lang.Thread.run(Thread.java:780)
Caused by: java.lang.RuntimeException: REQUEST FAILED: 
xpath=(/response/arr[@name='fields']/lst/str[@name='name'])[1] = 'custstopfilt'
xml response was: 



  0
  2


  
lowerfilt
lowe

[jira] [Comment Edited] (SOLR-4497) Collection Aliasing.

2013-03-06 Thread Tim Vaillancourt (JIRA)


[ 
https://issues.apache.org/jira/browse/SOLR-4497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13595530#comment-13595530
 ] 

Tim Vaillancourt edited comment on SOLR-4497 at 3/7/13 4:20 AM:


I think that is a good start. The change wouldn't be 100% immediate on return 
of the call, but designing the client/app to expect this could be ok. If there 
was a STATUS call on the Cores API that could detail the node's running alias, 
one could write an external script to "check" that the change made it 
everywhere.

This lead me to wondering (again without understanding implications and design 
goals) would it be best to have the Overseer push the change to all nodes (and 
write to ZK) vs relying only on the change in ZK, which has delays.

  was (Author: tvaillancourt):
I think that is a good start. The change wouldn't be 100% immediate on 
return of the call, but designing the client/app to expect this could be ok. If 
there was a STATUS call on the Cores API that could detail the node's running 
alias, one could write an external script to "check" that the change made it 
everywhere.

This lead me to wondering (again without understanding implications and design 
goals) would it be best to have the Overseer push the change to all nodes vs 
write to ZK?
  
> Collection Aliasing.
> 
>
> Key: SOLR-4497
> URL: https://issues.apache.org/jira/browse/SOLR-4497
> Project: Solr
>  Issue Type: New Feature
>  Components: SolrCloud
>Reporter: Mark Miller
>Assignee: Mark Miller
> Fix For: 4.2, 5.0
>
> Attachments: CDH-4497.patch, SOLR-4497.patch
>
>
> We should bring back the old aliasing feature, but for SolrCloud and with the 
> ability to alias one collection to many.
> The old alias feature was of more limited use and had some problems, so we 
> dropped it, but I think we can do this in a more useful way with SolrCloud, 
> and at a level were it's not invasive to the CoreContainer.
> Initially, the search side will allowing mapping a single alias to multiple 
> collections, but the index side will only support mapping a single alias to a 
> single collection.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-4497) Collection Aliasing.

2013-03-06 Thread Tim Vaillancourt (JIRA)


[ 
https://issues.apache.org/jira/browse/SOLR-4497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13595530#comment-13595530
 ] 

Tim Vaillancourt commented on SOLR-4497:


I think that is a good start. The change wouldn't be 100% immediate on return 
of the call, but designing the client/app to expect this could be ok. If there 
was a STATUS call on the Cores API that could detail the node's running alias, 
one could write an external script to "check" that the change made it 
everywhere.

This lead me to wondering (again without understanding implications and design 
goals) would it be best to have the Overseer push the change to all nodes vs 
write to ZK?

> Collection Aliasing.
> 
>
> Key: SOLR-4497
> URL: https://issues.apache.org/jira/browse/SOLR-4497
> Project: Solr
>  Issue Type: New Feature
>  Components: SolrCloud
>Reporter: Mark Miller
>Assignee: Mark Miller
> Fix For: 4.2, 5.0
>
> Attachments: CDH-4497.patch, SOLR-4497.patch
>
>
> We should bring back the old aliasing feature, but for SolrCloud and with the 
> ability to alias one collection to many.
> The old alias feature was of more limited use and had some problems, so we 
> dropped it, but I think we can do this in a more useful way with SolrCloud, 
> and at a level were it's not invasive to the CoreContainer.
> Initially, the search side will allowing mapping a single alias to multiple 
> collections, but the index side will only support mapping a single alias to a 
> single collection.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-3706) Ship setup to log with log4j.


[ 
https://issues.apache.org/jira/browse/SOLR-3706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13595458#comment-13595458
 ] 

Mark Miller commented on SOLR-3706:
---

bq. Mark, have you tried Logback?

I have not, I'll take a look!

> Ship setup to log with log4j.
> -
>
> Key: SOLR-3706
> URL: https://issues.apache.org/jira/browse/SOLR-3706
> Project: Solr
>  Issue Type: Improvement
>Reporter: Mark Miller
>Assignee: Mark Miller
>Priority: Minor
> Fix For: 5.0, 4.3
>
>
> Currently we default to java util logging and it's terrible in my opinion.
> *It's simple built in logger is a 2 line logger.
> *You have to jump through hoops to use your own custom formatter with jetty - 
> either putting your class in the start.jar or other pain in the butt 
> solutions.
> *It can't roll files by date out of the box.
> I'm sure there are more issues, but those are the ones annoying me now. We 
> should switch to log4j - it's much nicer and it's easy to get a nice single 
> line format and roll by date, etc.
> If someone wants to use JUL they still can - but at least users could start 
> with something decent.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

We have confluence wikis!

2013-03-06 Thread Mark Miller

https://cwiki.apache.org/confluence/display/SOLR/Index

and

https://cwiki.apache.org/confluence/display/LUCENE/Index

I'm going to start like some fresh doc stuff. And with versioning. Though I 
don't yet know what I'm doing. So help appreciated. This is stuff that Hossman 
is a genius at.

- Mark
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-3706) Ship setup to log with log4j.

2013-03-06 Thread David Smiley (JIRA)


[ 
https://issues.apache.org/jira/browse/SOLR-3706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13595393#comment-13595393
 ] 

David Smiley commented on SOLR-3706:


I hear ya on JUL sucking.  My first involvement with the Solr community was on 
this very issue as I railed against it and pushed for SLF4J.  I'm glad me and 
those that pushed with me (Ryan, ...) won out.

Mark, have you tried Logback?  That's a good logging implementation; arguably a 
better one.

> Ship setup to log with log4j.
> -
>
> Key: SOLR-3706
> URL: https://issues.apache.org/jira/browse/SOLR-3706
> Project: Solr
>  Issue Type: Improvement
>Reporter: Mark Miller
>Assignee: Mark Miller
>Priority: Minor
> Fix For: 5.0, 4.3
>
>
> Currently we default to java util logging and it's terrible in my opinion.
> *It's simple built in logger is a 2 line logger.
> *You have to jump through hoops to use your own custom formatter with jetty - 
> either putting your class in the start.jar or other pain in the butt 
> solutions.
> *It can't roll files by date out of the box.
> I'm sure there are more issues, but those are the ones annoying me now. We 
> should switch to log4j - it's much nicer and it's easy to get a nice single 
> line format and roll by date, etc.
> If someone wants to use JUL they still can - but at least users could start 
> with something decent.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-4470) Support for basic http auth in internal solr requests


 [ 
https://issues.apache.org/jira/browse/SOLR-4470?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jan Høydahl updated SOLR-4470:
--

Labels: authentication https solrclient solrcloud ssl  (was: authentication 
solrclient solrcloud)

> Support for basic http auth in internal solr requests
> -
>
> Key: SOLR-4470
> URL: https://issues.apache.org/jira/browse/SOLR-4470
> Project: Solr
>  Issue Type: New Feature
>  Components: clients - java, multicore, replication (java), SolrCloud
>Affects Versions: 4.0
>Reporter: Per Steffensen
>  Labels: authentication, https, solrclient, solrcloud, ssl
> Fix For: 4.2, 5.0
>
> Attachments: SOLR-4470_branch_4x_r1452629.patch, 
> SOLR-4470_branch_4x_r1452629.patch, SOLR-4470.patch
>
>
> We want to protect any HTTP-resource (url). We want to require credentials no 
> matter what kind of HTTP-request you make to a Solr-node.
> It can faily easy be acheived as described on 
> http://wiki.apache.org/solr/SolrSecurity. This problem is that Solr-nodes 
> also make "internal" request to other Solr-nodes, and for it to work 
> credentials need to be provided here also.
> Ideally we would like to "forward" credentials from a particular request to 
> all the "internal" sub-requests it triggers. E.g. for search and update 
> request.
> But there are also "internal" requests
> * that only indirectly/asynchronously triggered from "outside" requests (e.g. 
> shard creation/deletion/etc based on calls to the "Collection API")
> * that do not in any way have relation to an "outside" "super"-request (e.g. 
> replica synching stuff)
> We would like to aim at a solution where "original" credentials are 
> "forwarded" when a request directly/synchronously trigger a subrequest, and 
> fallback to a configured "internal credentials" for the 
> asynchronous/non-rooted requests.
> In our solution we would aim at only supporting basic http auth, but we would 
> like to make a "framework" around it, so that not to much refactoring is 
> needed if you later want to make support for other kinds of auth (e.g. digest)
> We will work at a solution but create this JIRA issue early in order to get 
> input/comments from the community as early as possible.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-4470) Support for basic http auth in internal solr requests


 [ 
https://issues.apache.org/jira/browse/SOLR-4470?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jan Høydahl updated SOLR-4470:
--

Fix Version/s: 5.0

> Support for basic http auth in internal solr requests
> -
>
> Key: SOLR-4470
> URL: https://issues.apache.org/jira/browse/SOLR-4470
> Project: Solr
>  Issue Type: New Feature
>  Components: clients - java, multicore, replication (java), SolrCloud
>Affects Versions: 4.0
>Reporter: Per Steffensen
>  Labels: authentication, solrclient, solrcloud
> Fix For: 4.2, 5.0
>
> Attachments: SOLR-4470_branch_4x_r1452629.patch, 
> SOLR-4470_branch_4x_r1452629.patch, SOLR-4470.patch
>
>
> We want to protect any HTTP-resource (url). We want to require credentials no 
> matter what kind of HTTP-request you make to a Solr-node.
> It can faily easy be acheived as described on 
> http://wiki.apache.org/solr/SolrSecurity. This problem is that Solr-nodes 
> also make "internal" request to other Solr-nodes, and for it to work 
> credentials need to be provided here also.
> Ideally we would like to "forward" credentials from a particular request to 
> all the "internal" sub-requests it triggers. E.g. for search and update 
> request.
> But there are also "internal" requests
> * that only indirectly/asynchronously triggered from "outside" requests (e.g. 
> shard creation/deletion/etc based on calls to the "Collection API")
> * that do not in any way have relation to an "outside" "super"-request (e.g. 
> replica synching stuff)
> We would like to aim at a solution where "original" credentials are 
> "forwarded" when a request directly/synchronously trigger a subrequest, and 
> fallback to a configured "internal credentials" for the 
> asynchronous/non-rooted requests.
> In our solution we would aim at only supporting basic http auth, but we would 
> like to make a "framework" around it, so that not to much refactoring is 
> needed if you later want to make support for other kinds of auth (e.g. digest)
> We will work at a solution but create this JIRA issue early in order to get 
> input/comments from the community as early as possible.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-4470) Support for basic http auth in internal solr requests


 [ 
https://issues.apache.org/jira/browse/SOLR-4470?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jan Høydahl updated SOLR-4470:
--

Issue Type: New Feature  (was: Bug)

> Support for basic http auth in internal solr requests
> -
>
> Key: SOLR-4470
> URL: https://issues.apache.org/jira/browse/SOLR-4470
> Project: Solr
>  Issue Type: New Feature
>  Components: clients - java, multicore, replication (java), SolrCloud
>Affects Versions: 4.0
>Reporter: Per Steffensen
>  Labels: authentication, solrclient, solrcloud
> Fix For: 4.2
>
> Attachments: SOLR-4470_branch_4x_r1452629.patch, 
> SOLR-4470_branch_4x_r1452629.patch, SOLR-4470.patch
>
>
> We want to protect any HTTP-resource (url). We want to require credentials no 
> matter what kind of HTTP-request you make to a Solr-node.
> It can faily easy be acheived as described on 
> http://wiki.apache.org/solr/SolrSecurity. This problem is that Solr-nodes 
> also make "internal" request to other Solr-nodes, and for it to work 
> credentials need to be provided here also.
> Ideally we would like to "forward" credentials from a particular request to 
> all the "internal" sub-requests it triggers. E.g. for search and update 
> request.
> But there are also "internal" requests
> * that only indirectly/asynchronously triggered from "outside" requests (e.g. 
> shard creation/deletion/etc based on calls to the "Collection API")
> * that do not in any way have relation to an "outside" "super"-request (e.g. 
> replica synching stuff)
> We would like to aim at a solution where "original" credentials are 
> "forwarded" when a request directly/synchronously trigger a subrequest, and 
> fallback to a configured "internal credentials" for the 
> asynchronous/non-rooted requests.
> In our solution we would aim at only supporting basic http auth, but we would 
> like to make a "framework" around it, so that not to much refactoring is 
> needed if you later want to make support for other kinds of auth (e.g. digest)
> We will work at a solution but create this JIRA issue early in order to get 
> input/comments from the community as early as possible.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-4470) Support for basic http auth in internal solr requests


[ 
https://issues.apache.org/jira/browse/SOLR-4470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13595357#comment-13595357
 ] 

Jan Høydahl commented on SOLR-4470:
---

One comment

By adding a parameter to the utiliy methods without leaving the old signature 
in place, you break back-compat with people using the old signature.

{code}
-  public UpdateResponse add(SolrInputDocument doc, int commitWithinMs) throws 
SolrServerException, IOException {
+  public UpdateResponse add(SolrInputDocument doc, int commitWithinMs, 
AuthCredentials authCredentials) throws SolrServerException, IOException {
{code}

We should either ADD the new methods or simply skip adding this to convenience 
signatures. Users wanting auth can do so with the other (more elaborate) 
technique, they do not *need* a convenience method for auth.

{code}
 UpdateRequest req = new UpdateRequest();
 req.add(doc);
 req.setAuthCredentials(authCredentials);
{code}

One can argue that commitWithin is something most users want to / should use 
while auth will be more of a a corner case.

> Support for basic http auth in internal solr requests
> -
>
> Key: SOLR-4470
> URL: https://issues.apache.org/jira/browse/SOLR-4470
> Project: Solr
>  Issue Type: Bug
>  Components: clients - java, multicore, replication (java), SolrCloud
>Affects Versions: 4.0
>Reporter: Per Steffensen
>  Labels: authentication, solrclient, solrcloud
> Fix For: 4.2
>
> Attachments: SOLR-4470_branch_4x_r1452629.patch, 
> SOLR-4470_branch_4x_r1452629.patch, SOLR-4470.patch
>
>
> We want to protect any HTTP-resource (url). We want to require credentials no 
> matter what kind of HTTP-request you make to a Solr-node.
> It can faily easy be acheived as described on 
> http://wiki.apache.org/solr/SolrSecurity. This problem is that Solr-nodes 
> also make "internal" request to other Solr-nodes, and for it to work 
> credentials need to be provided here also.
> Ideally we would like to "forward" credentials from a particular request to 
> all the "internal" sub-requests it triggers. E.g. for search and update 
> request.
> But there are also "internal" requests
> * that only indirectly/asynchronously triggered from "outside" requests (e.g. 
> shard creation/deletion/etc based on calls to the "Collection API")
> * that do not in any way have relation to an "outside" "super"-request (e.g. 
> replica synching stuff)
> We would like to aim at a solution where "original" credentials are 
> "forwarded" when a request directly/synchronously trigger a subrequest, and 
> fallback to a configured "internal credentials" for the 
> asynchronous/non-rooted requests.
> In our solution we would aim at only supporting basic http auth, but we would 
> like to make a "framework" around it, so that not to much refactoring is 
> needed if you later want to make support for other kinds of auth (e.g. digest)
> We will work at a solution but create this JIRA issue early in order to get 
> input/comments from the community as early as possible.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-4470) Support for basic http auth in internal solr requests


 [ 
https://issues.apache.org/jira/browse/SOLR-4470?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jan Høydahl updated SOLR-4470:
--

Attachment: SOLR-4470.patch

Attached patch for trunk

> Support for basic http auth in internal solr requests
> -
>
> Key: SOLR-4470
> URL: https://issues.apache.org/jira/browse/SOLR-4470
> Project: Solr
>  Issue Type: Bug
>  Components: clients - java, multicore, replication (java), SolrCloud
>Affects Versions: 4.0
>Reporter: Per Steffensen
>  Labels: authentication, solrclient, solrcloud
> Fix For: 4.2
>
> Attachments: SOLR-4470_branch_4x_r1452629.patch, 
> SOLR-4470_branch_4x_r1452629.patch, SOLR-4470.patch
>
>
> We want to protect any HTTP-resource (url). We want to require credentials no 
> matter what kind of HTTP-request you make to a Solr-node.
> It can faily easy be acheived as described on 
> http://wiki.apache.org/solr/SolrSecurity. This problem is that Solr-nodes 
> also make "internal" request to other Solr-nodes, and for it to work 
> credentials need to be provided here also.
> Ideally we would like to "forward" credentials from a particular request to 
> all the "internal" sub-requests it triggers. E.g. for search and update 
> request.
> But there are also "internal" requests
> * that only indirectly/asynchronously triggered from "outside" requests (e.g. 
> shard creation/deletion/etc based on calls to the "Collection API")
> * that do not in any way have relation to an "outside" "super"-request (e.g. 
> replica synching stuff)
> We would like to aim at a solution where "original" credentials are 
> "forwarded" when a request directly/synchronously trigger a subrequest, and 
> fallback to a configured "internal credentials" for the 
> asynchronous/non-rooted requests.
> In our solution we would aim at only supporting basic http auth, but we would 
> like to make a "framework" around it, so that not to much refactoring is 
> needed if you later want to make support for other kinds of auth (e.g. digest)
> We will work at a solution but create this JIRA issue early in order to get 
> input/comments from the community as early as possible.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-4470) Support for basic http auth in internal solr requests


[ 
https://issues.apache.org/jira/browse/SOLR-4470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13595347#comment-13595347
 ] 

Jan Høydahl commented on SOLR-4470:
---

bq. I removed the solr.xml thing again because we not are going to use it and 
it would just make the patch even bigger (and we would not like that  ). 
Agree, keep it simple and iterate from there

I tried to apply the patch to trunk and there were not as many mismatches as I 
thought. So although I created a branch 
https://svn.apache.org/repos/asf/lucene/dev/branches/security/ we may not need 
to use it before we feel the need for it, as long as the patch keeps applying 
well for longer periods of time. Will attach trunk patch as SOLR-4470.patch

bq. Regarding running the test-suite: If not already granted in your policy 
file you need to grant "permission javax.security.auth.AuthPermission "*";" in 
general or at least for the JettySolrRunner (or the anonymous 
MappedLoginService in JettySolrRunner). This permission is needed in order to 
be able to set up the LoginModule in JettySolrRunner (test-only)

Hmm, it took some trial & error before I understood why tests failed, then I 
saw this comment and changed my java.policy file. But how can this be better 
integrated with Ant so that patching java.policy for all your JDKs is not a 
requirement for passing Lucene/Solr tests? I think such a requirement may be a 
no-starter. Is it an alternative to simply modify Jetty's xml file instead of 
doing it through API?

Regarding the use of "|" as separator in RegExpAuthorizationFilter
{code:xml}
  
search-constraint
1|update-role,admin-role|^.*/update$
  
{code}

Will it work with regexes containing "|"? Or would it be more readable (and 
cool) in these JSON-times to use a small object as param-value? Then we could 
keep the base object syntax for any future Filter implementations.
{noformat}
{"order": 1, "roles": ["update-role","admin-role"], "regex": "^.*/update$"}
{noformat}

> Support for basic http auth in internal solr requests
> -
>
> Key: SOLR-4470
> URL: https://issues.apache.org/jira/browse/SOLR-4470
> Project: Solr
>  Issue Type: Bug
>  Components: clients - java, multicore, replication (java), SolrCloud
>Affects Versions: 4.0
>Reporter: Per Steffensen
>  Labels: authentication, solrclient, solrcloud
> Fix For: 4.2
>
> Attachments: SOLR-4470_branch_4x_r1452629.patch, 
> SOLR-4470_branch_4x_r1452629.patch
>
>
> We want to protect any HTTP-resource (url). We want to require credentials no 
> matter what kind of HTTP-request you make to a Solr-node.
> It can faily easy be acheived as described on 
> http://wiki.apache.org/solr/SolrSecurity. This problem is that Solr-nodes 
> also make "internal" request to other Solr-nodes, and for it to work 
> credentials need to be provided here also.
> Ideally we would like to "forward" credentials from a particular request to 
> all the "internal" sub-requests it triggers. E.g. for search and update 
> request.
> But there are also "internal" requests
> * that only indirectly/asynchronously triggered from "outside" requests (e.g. 
> shard creation/deletion/etc based on calls to the "Collection API")
> * that do not in any way have relation to an "outside" "super"-request (e.g. 
> replica synching stuff)
> We would like to aim at a solution where "original" credentials are 
> "forwarded" when a request directly/synchronously trigger a subrequest, and 
> fallback to a configured "internal credentials" for the 
> asynchronous/non-rooted requests.
> In our solution we would aim at only supporting basic http auth, but we would 
> like to make a "framework" around it, so that not to much refactoring is 
> needed if you later want to make support for other kinds of auth (e.g. digest)
> We will work at a solution but create this JIRA issue early in order to get 
> input/comments from the community as early as possible.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-4234) Add support for binary files in ZooKeeper.


[ 
https://issues.apache.org/jira/browse/SOLR-4234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13595318#comment-13595318
 ] 

Mark Miller commented on SOLR-4234:
---

Ping - still interested in this Eric? I'll get to it eventually anyway, but it 
may take a little longer.

> Add support for binary files in ZooKeeper.
> --
>
> Key: SOLR-4234
> URL: https://issues.apache.org/jira/browse/SOLR-4234
> Project: Solr
>  Issue Type: Bug
>  Components: SolrCloud
>Affects Versions: 4.0
>Reporter: Eric Pugh
>Assignee: Mark Miller
> Fix For: 4.2, 5.0
>
> Attachments: binary_upload_download.patch, 
> fix_show_file_handler_with_binaries.patch, SOLR4234_binary_files.patch, 
> solr.png
>
>
> I was attempting to get the ShowFileHandler to show a .png file, and it was 
> failing.  But in non-ZK mode it worked just fine!   It took a while, but it 
> seems that we upload to zk as a text, and download as well.  I've attached a 
> unit test that demonstrates the problem, and a fix.  You have to have a 
> binary file in the conf directory to make the test work, I put solr.png in 
> the collection1/conf/velocity directory.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Resolved] (SOLR-3186) You should be able to set numShards in solr.xml per core.


 [ 
https://issues.apache.org/jira/browse/SOLR-3186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mark Miller resolved SOLR-3186.
---

Resolution: Won't Fix

> You should be able to set numShards in solr.xml per core.
> -
>
> Key: SOLR-3186
> URL: https://issues.apache.org/jira/browse/SOLR-3186
> Project: Solr
>  Issue Type: Improvement
>Reporter: Mark Miller
>Assignee: Mark Miller
>
> If you are bootstrapping a multi-core setup, you currently have to settle for 
> the same numShards for every core. We really should support a way of 
> specifying this per core without having to use the CoreAdminHandler to 
> dynamically create the cores. Currently I am thinking we just add numShards 
> to what you can specify per core in solr.xml. I know it's a little odd 
> because the param is only used once on the very first node for the collection 
> - but as long as we doc it correctly, I don't think its much of a problem.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Resolved] (SOLR-4416) Upgrade to Tika 1.3


 [ 
https://issues.apache.org/jira/browse/SOLR-4416?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mark Miller resolved SOLR-4416.
---

Resolution: Fixed

Seems the jenkins issues have been cleaned up?

> Upgrade to Tika 1.3
> ---
>
> Key: SOLR-4416
> URL: https://issues.apache.org/jira/browse/SOLR-4416
> Project: Solr
>  Issue Type: Task
>  Components: contrib - Solr Cell (Tika extraction)
>Affects Versions: 4.1
>Reporter: Erik Hatcher
>Assignee: Mark Miller
>Priority: Critical
> Fix For: 4.2, 5.0
>
> Attachments: SOLR-4416-trunk-1.patch
>
>
> Tika 1.3 was recently released with these improvements: 
> http://www.apache.org/dist/tika/CHANGES-1.3.txt

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: JVM hanging when using G1GC on JDK8 b78 or b79 (Linux 32 bit)

2013-03-06 Thread John Cuthbertson


Hi Uwe,

An update:

I have downloaded ant and the lucerne source.

I attempted the ivy-bootstrap but it failed to download the 
ivy=2.3.0.jar file - even after setting:


ANT_OPTS=-Dhttp.proxyHost=<...> -Dhttp.proxyPort=<...>

So I manually downloaded and placed it into the ANT library and now get:


ivy-bootstrap1:
[mkdir] Skipping /home/jcuthber/.ant/lib because it already exists.
 [echo] installing ivy 2.3.0 to /home/jcuthber/.ant/lib
  [get] Getting: 
http://repo1.maven.org/maven2/org/apache/ivy/ivy/2.3.0/ivy-2.3.0.jar

  [get] To: /home/jcuthber/.ant/lib/ivy-2.3.0.jar
  [get] Error getting 
http://repo1.maven.org/maven2/org/apache/ivy/ivy/2.3.0/ivy-2.3.0.jar 
to /home/jcuthber/.ant/lib/ivy-2.3.0.jar

[available] Found: /home/jcuthber/.ant/lib/ivy-2.3.0.jar

ivy-bootstrap2:
Skipped because property 'ivy.bootstrap1.success' set.

ivy-checksum:

ivy-bootstrap:

BUILD SUCCESSFUL
Total time: 3 minutes 46 seconds
Presumably I have to build the lucerne source before executing the 
tests. That seemed to go OK.


When I run the analysis/uima tests it seems to get hung up at the 
"resolve" target - even without specifying G1:



cairnapple{jcuthber}:408> cd analysis/uima/
cairnapple{jcuthber}:409> ls -l
total 29
-rw-r--r--   1 jcuthber staff   1473 Dec 10 10:39 build.xml
-rw-rw-r--   1 jcuthber staff   6895 Mar  6 15:20 hotspot.log
-rw-r--r--   1 jcuthber staff   1316 Mar 30  2012 ivy.xml
drwxr-xr-x   2 jcuthber staff  2 Mar  5 07:42 lib/
drwxr-xr-x   6 jcuthber staff  6 Mar  5 07:42 src/



ivy-configure:
[ivy:configure] Loading 
jar:file:/home/jcuthber/.ant/lib/ivy-2.3.0.jar!/org/apache/ivy/core/settings/ivy.properties
[ivy:configure] :: Apache Ivy 2.3.0 - 20130110142753 :: 
http://ant.apache.org/ivy/ ::
[ivy:configure] jakarta commons httpclient not found: using jdk url 
handling
[ivy:configure] :: loading settings :: file = 
/export/bugs/8009536/lucene-5.0-2013-03-05_15-37-06/ivy-settings.xml
[ivy:configure] no default ivy user dir defined: set to 
/home/jcuthber/.ivy2
[ivy:configure] including url: 
jar:file:/home/jcuthber/.ant/lib/ivy-2.3.0.jar!/org/apache/ivy/core/settings/ivysettings-public.xml
[ivy:configure] no default cache defined: set to 
/home/jcuthber/.ivy2/cache
[ivy:configure] including url: 
jar:file:/home/jcuthber/.ant/lib/ivy-2.3.0.jar!/org/apache/ivy/core/settings/ivysettings-shared.xml
[ivy:configure] including url: 
jar:file:/home/jcuthber/.ant/lib/ivy-2.3.0.jar!/org/apache/ivy/core/settings/ivysettings-local.xml
[ivy:configure] including url: 
jar:file:/home/jcuthber/.ant/lib/ivy-2.3.0.jar!/org/apache/ivy/core/settings/ivysettings-main-chain.xml

[ivy:configure] settings loaded (289ms)
[ivy:configure] default cache: /home/jcuthber/.ivy2/cache
[ivy:configure] default resolver: default
[ivy:configure] -- 7 resolvers:
[ivy:configure] working-chinese-mirror [ibiblio]
[ivy:configure] main [chain] [shared, public]
[ivy:configure] local [file]
[ivy:configure] shared [file]
[ivy:configure] sonatype-releases [ibiblio]
[ivy:configure] public [ibiblio]
[ivy:configure] default [chain] [local, main, 
sonatype-releases, working-chinese-mirror]


resolve:
[ivy:retrieve] no resolved descriptor found: launching default resolve
Overriding previous definition of property "ivy.version"
[ivy:retrieve] using ivy parser to parse 
file:/export/bugs/8009536/lucene-5.0-2013-03-05_15-37-06/analysis/uima/ivy.xml
[ivy:retrieve] :: resolving dependencies :: 
org.apache.lucene#analyzers-uima;working@cairnapple

[ivy:retrieve]  confs: [default]
[ivy:retrieve]  validate = true
[ivy:retrieve]  refresh = false
[ivy:retrieve] resolving dependencies for configuration 'default'
[ivy:retrieve] == resolving dependencies for 
org.apache.lucene#analyzers-uima;working@cairnapple [default]
[ivy:retrieve] == resolving dependencies 
org.apache.lucene#analyzers-uima;working@cairnapple->org.apache.uima#Tagger;2.3.1 
[default->*]
[ivy:retrieve] default: Checking cache for: dependency: 
org.apache.uima#Tagger;2.3.1 {*=[*]}
[ivy:retrieve] don't use cache for org.apache.uima#Tagger;2.3.1: 
checkModified=true
[ivy:retrieve]  tried 
/home/jcuthber/.ivy2/local/org.apache.uima/Tagger/2.3.1/ivys/ivy.xml
[ivy:retrieve]  tried 
/home/jcuthber/.ivy2/local/org.apache.uima/Tagger/2.3.1/jars/Tagger.jar
[ivy:retrieve]  local: no ivy file nor artifact found for 
org.apache.uima#Tagger;2.3.1
[ivy:retrieve] main: Checking cache for: dependency: 
org.apache.uima#Tagger;2.3.1 {*=[*]}
[ivy:retrieve]  tried 
/home/jcuthber/.ivy2/shared/org.apache.uima/Tagger/2.3.1/ivys/ivy.xml
[ivy:retrieve]  tried 
/home/jcuthber/.ivy2/shared/org.apache.uima/Tagger/2.3.1/jars/Tagger.jar
[ivy:retrieve]  shared: no ivy file nor artifact found for 
org.apache.uima#Tagger;2.3.1
[ivy:retrieve]  tried 
http://repo1.maven.org/maven2/org/apache/uima/Tagger/2.3.1/Tagger-2.3.1.pom
and there it hangs - presumably try

[jira] [Commented] (SOLR-4497) Collection Aliasing.


[ 
https://issues.apache.org/jira/browse/SOLR-4497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13595229#comment-13595229
 ] 

Mark Miller commented on SOLR-4497:
---

I'm ready to commit this soon - I'd like to get it in for 4.2.

> Collection Aliasing.
> 
>
> Key: SOLR-4497
> URL: https://issues.apache.org/jira/browse/SOLR-4497
> Project: Solr
>  Issue Type: New Feature
>  Components: SolrCloud
>Reporter: Mark Miller
>Assignee: Mark Miller
> Fix For: 4.2, 5.0
>
> Attachments: CDH-4497.patch, SOLR-4497.patch
>
>
> We should bring back the old aliasing feature, but for SolrCloud and with the 
> ability to alias one collection to many.
> The old alias feature was of more limited use and had some problems, so we 
> dropped it, but I think we can do this in a more useful way with SolrCloud, 
> and at a level were it's not invasive to the CoreContainer.
> Initially, the search side will allowing mapping a single alias to multiple 
> collections, but the index side will only support mapping a single alias to a 
> single collection.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-1604) Wildcards, ORs etc inside Phrase Queries

2013-03-06 Thread Paul Morris (JIRA)


[ 
https://issues.apache.org/jira/browse/SOLR-1604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13595230#comment-13595230
 ] 

Paul Morris commented on SOLR-1604:
---

Hi Ahmet and fellow ComplexPhrase users,

We have been using the ComplexPhraseQueryParser for a number of years with our 
SOLR installation, but we were hoping to upgrade to SOLR 4.1 and I was just 
wondering if ComplexPhrase works with the latest SOLR release? 

We are still on SOLR 1.4.1 so keen to take advantage of all that SOLR 4.1 can 
offer, but we can't go there without ComplexPhrase :-) 

Much thanks if anyone has anyone has any thoughts. 



> Wildcards, ORs etc inside Phrase Queries
> 
>
> Key: SOLR-1604
> URL: https://issues.apache.org/jira/browse/SOLR-1604
> Project: Solr
>  Issue Type: Improvement
>  Components: query parsers, search
>Affects Versions: 1.4
>Reporter: Ahmet Arslan
>Priority: Minor
> Attachments: ASF.LICENSE.NOT.GRANTED--ComplexPhrase.zip, 
> ComplexPhraseQueryParser.java, ComplexPhrase_solr_3.4.zip, ComplexPhrase.zip, 
> ComplexPhrase.zip, ComplexPhrase.zip, ComplexPhrase.zip, ComplexPhrase.zip, 
> ComplexPhrase.zip, SOLR-1604-alternative.patch, SOLR-1604.patch, 
> SOLR-1604.patch
>
>
> Solr Plugin for ComplexPhraseQueryParser (LUCENE-1486) which supports 
> wildcards, ORs, ranges, fuzzies inside phrase queries.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-4534) Shard Aliasing


 [ 
https://issues.apache.org/jira/browse/SOLR-4534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mark Miller updated SOLR-4534:
--

Description: Collection Aliasing lets you do some interesting things, but 
all of it requires multiple collections. It's also interesting to be able to 
alias shards within a collection so that you don't always have to use multiple 
collections to take advantage of aliasing.

> Shard Aliasing
> --
>
> Key: SOLR-4534
> URL: https://issues.apache.org/jira/browse/SOLR-4534
> Project: Solr
>  Issue Type: New Feature
>  Components: SolrCloud
>Reporter: Mark Miller
>
> Collection Aliasing lets you do some interesting things, but all of it 
> requires multiple collections. It's also interesting to be able to alias 
> shards within a collection so that you don't always have to use multiple 
> collections to take advantage of aliasing.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-4497) Collection Aliasing.


 [ 
https://issues.apache.org/jira/browse/SOLR-4497?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mark Miller updated SOLR-4497:
--

Attachment: CDH-4497.patch

New patch.

* Additional testing
* Fixes the case when the node you are hitting does not have a piece of the 
first collection in your alias list
* Updates aliases and does a retry on alias miss
* Tries to wait for alias changes as mentioned in my comment above.

> Collection Aliasing.
> 
>
> Key: SOLR-4497
> URL: https://issues.apache.org/jira/browse/SOLR-4497
> Project: Solr
>  Issue Type: New Feature
>  Components: SolrCloud
>Reporter: Mark Miller
>Assignee: Mark Miller
> Fix For: 4.2, 5.0
>
> Attachments: CDH-4497.patch, SOLR-4497.patch
>
>
> We should bring back the old aliasing feature, but for SolrCloud and with the 
> ability to alias one collection to many.
> The old alias feature was of more limited use and had some problems, so we 
> dropped it, but I think we can do this in a more useful way with SolrCloud, 
> and at a level were it's not invasive to the CoreContainer.
> Initially, the search side will allowing mapping a single alias to multiple 
> collections, but the index side will only support mapping a single alias to a 
> single collection.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-2894) Implement distributed pivot faceting

2013-03-06 Thread Monica Skidmore (JIRA)


[ 
https://issues.apache.org/jira/browse/SOLR-2894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13595202#comment-13595202
 ] 

Monica Skidmore commented on SOLR-2894:
---

Thanks, Andrew!  We're upgrading our version of Solr, and this will make our 
users very happy.  I'm hoping it will be committed now...

> Implement distributed pivot faceting
> 
>
> Key: SOLR-2894
> URL: https://issues.apache.org/jira/browse/SOLR-2894
> Project: Solr
>  Issue Type: Improvement
>Reporter: Erik Hatcher
> Fix For: 4.2, 5.0
>
> Attachments: SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, 
> SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, 
> SOLR-2894.patch, SOLR-2894-reworked.patch
>
>
> Following up on SOLR-792, pivot faceting currently only supports 
> undistributed mode.  Distributed pivot faceting needs to be implemented.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-2894) Implement distributed pivot faceting


[ 
https://issues.apache.org/jira/browse/SOLR-2894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13595174#comment-13595174
 ] 

Mark Miller commented on SOLR-2894:
---

Thanks Andrew - this isn't my area of expertise, but hopefully someone will get 
to this eventually. Lot's of votes and watchers :)

> Implement distributed pivot faceting
> 
>
> Key: SOLR-2894
> URL: https://issues.apache.org/jira/browse/SOLR-2894
> Project: Solr
>  Issue Type: Improvement
>Reporter: Erik Hatcher
> Fix For: 4.2, 5.0
>
> Attachments: SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, 
> SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, 
> SOLR-2894.patch, SOLR-2894-reworked.patch
>
>
> Following up on SOLR-792, pivot faceting currently only supports 
> undistributed mode.  Distributed pivot faceting needs to be implemented.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-2894) Implement distributed pivot faceting

2013-03-06 Thread Andrew Muldowney (JIRA)


 [ 
https://issues.apache.org/jira/browse/SOLR-2894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Muldowney updated SOLR-2894:
---

Attachment: SOLR-2894.patch

New patch file for trunk, its a git patch from trunk solr pulled today, let me 
know if there are any issues applying.

> Implement distributed pivot faceting
> 
>
> Key: SOLR-2894
> URL: https://issues.apache.org/jira/browse/SOLR-2894
> Project: Solr
>  Issue Type: Improvement
>Reporter: Erik Hatcher
> Fix For: 4.2, 5.0
>
> Attachments: SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, 
> SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, 
> SOLR-2894.patch, SOLR-2894-reworked.patch
>
>
> Following up on SOLR-792, pivot faceting currently only supports 
> undistributed mode.  Distributed pivot faceting needs to be implemented.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-4200) Significant log noise at INFO from CachingDirectoryFactory#close


[ 
https://issues.apache.org/jira/browse/SOLR-4200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13595148#comment-13595148
 ] 

Commit Tag Bot commented on SOLR-4200:
--

[branch_4x commit] Chris M. Hostetter
http://svn.apache.org/viewvc?view=revision&revision=1453578

SOLR-4200: Reduce INFO level logging from CachingDirectoryFactory (merge 
r1453560)


> Significant log noise at INFO from CachingDirectoryFactory#close
> 
>
> Key: SOLR-4200
> URL: https://issues.apache.org/jira/browse/SOLR-4200
> Project: Solr
>  Issue Type: Bug
>Affects Versions: 4.1
> Environment: solr-impl4.1-SNAPSHOT 1421496 - ncindex - 2012-12-13 
> 14:56:25
>Reporter: Shawn Heisey
>Assignee: Shawn Heisey
> Fix For: 4.2, 5.0
>
> Attachments: SOLR-4200.patch, SOLR-4200.patch
>
>
> The following line in CachingDirectoryFactory is resulting in a lot of noise 
> in my branch_4x logs.  It was added by Yonik's recent major sync from trunk 
> to branch_4x, r1420992.
> log.info("Releasing directory:" + cacheValue.path);
> This was probably added to debug a problem, but it seems to get called a lot. 
>  The specific thing that led me to bring it up is that it is logged four 
> times for every call to /admin/mbeans.  When you've got seven cores and you 
> hit them all, it increases the logging from 7 lines to 35.  IMHO, it should 
> be moved to debug instead of info.
> INFO  - 2012-12-15 13:36:01.674; 
> org.apache.solr.core.CachingDirectoryFactory; Releasing 
> directory:/index/solr4/cores/s0_0/../../data/s0_0
> INFO  - 2012-12-15 13:36:01.676; 
> org.apache.solr.core.CachingDirectoryFactory; Releasing 
> directory:/index/solr4/cores/s0_0/../../data/s0_0/index
> INFO  - 2012-12-15 13:36:01.676; 
> org.apache.solr.core.CachingDirectoryFactory; Releasing 
> directory:/index/solr4/cores/s0_0/../../data/s0_0
> INFO  - 2012-12-15 13:36:01.678; 
> org.apache.solr.core.CachingDirectoryFactory; Releasing 
> directory:/index/solr4/cores/s0_0/../../data/s0_0/index
> INFO  - 2012-12-15 13:36:01.679; org.apache.solr.core.SolrCore; [s0build] 
> webapp=/solr path=/admin/mbeans 
> params={qt.path=/admin/mbeans&wt=javabin&stats=true&version=2} status=0 
> QTime=6

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Resolved] (SOLR-4200) Significant log noise at INFO from CachingDirectoryFactory#close

2013-03-06 Thread Hoss Man (JIRA)


 [ 
https://issues.apache.org/jira/browse/SOLR-4200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hoss Man resolved SOLR-4200.


Resolution: Fixed
  Assignee: Shawn Heisey

Committed revision 1453560.
Committed revision 1453578.


> Significant log noise at INFO from CachingDirectoryFactory#close
> 
>
> Key: SOLR-4200
> URL: https://issues.apache.org/jira/browse/SOLR-4200
> Project: Solr
>  Issue Type: Bug
>Affects Versions: 4.1
> Environment: solr-impl4.1-SNAPSHOT 1421496 - ncindex - 2012-12-13 
> 14:56:25
>Reporter: Shawn Heisey
>Assignee: Shawn Heisey
> Fix For: 4.2, 5.0
>
> Attachments: SOLR-4200.patch, SOLR-4200.patch
>
>
> The following line in CachingDirectoryFactory is resulting in a lot of noise 
> in my branch_4x logs.  It was added by Yonik's recent major sync from trunk 
> to branch_4x, r1420992.
> log.info("Releasing directory:" + cacheValue.path);
> This was probably added to debug a problem, but it seems to get called a lot. 
>  The specific thing that led me to bring it up is that it is logged four 
> times for every call to /admin/mbeans.  When you've got seven cores and you 
> hit them all, it increases the logging from 7 lines to 35.  IMHO, it should 
> be moved to debug instead of info.
> INFO  - 2012-12-15 13:36:01.674; 
> org.apache.solr.core.CachingDirectoryFactory; Releasing 
> directory:/index/solr4/cores/s0_0/../../data/s0_0
> INFO  - 2012-12-15 13:36:01.676; 
> org.apache.solr.core.CachingDirectoryFactory; Releasing 
> directory:/index/solr4/cores/s0_0/../../data/s0_0/index
> INFO  - 2012-12-15 13:36:01.676; 
> org.apache.solr.core.CachingDirectoryFactory; Releasing 
> directory:/index/solr4/cores/s0_0/../../data/s0_0
> INFO  - 2012-12-15 13:36:01.678; 
> org.apache.solr.core.CachingDirectoryFactory; Releasing 
> directory:/index/solr4/cores/s0_0/../../data/s0_0/index
> INFO  - 2012-12-15 13:36:01.679; org.apache.solr.core.SolrCore; [s0build] 
> webapp=/solr path=/admin/mbeans 
> params={qt.path=/admin/mbeans&wt=javabin&stats=true&version=2} status=0 
> QTime=6

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Assigned] (LUCENE-3918) Port index sorter to trunk APIs


 [ 
https://issues.apache.org/jira/browse/LUCENE-3918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shai Erera reassigned LUCENE-3918:
--

Assignee: Shai Erera

> Port index sorter to trunk APIs
> ---
>
> Key: LUCENE-3918
> URL: https://issues.apache.org/jira/browse/LUCENE-3918
> Project: Lucene - Core
>  Issue Type: Task
>  Components: modules/other
>Affects Versions: 4.0-ALPHA
>Reporter: Robert Muir
>Assignee: Shai Erera
> Fix For: 4.2, 5.0
>
> Attachments: LUCENE-3918.patch, LUCENE-3918.patch, LUCENE-3918.patch, 
> LUCENE-3918.patch, LUCENE-3918.patch, LUCENE-3918.patch, LUCENE-3918.patch, 
> LUCENE-3918.patch, LUCENE-3918.patch, LUCENE-3918.patch, LUCENE-3918.patch, 
> LUCENE-3918.patch, LUCENE-3918.patch, LUCENE-3918.patch
>
>
> LUCENE-2482 added an IndexSorter to 3.x, but we need to port this
> functionality to 4.0 apis.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-4200) Significant log noise at INFO from CachingDirectoryFactory#close


[ 
https://issues.apache.org/jira/browse/SOLR-4200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13595125#comment-13595125
 ] 

Commit Tag Bot commented on SOLR-4200:
--

[trunk commit] Chris M. Hostetter
http://svn.apache.org/viewvc?view=revision&revision=1453560

SOLR-4200: Reduce INFO level logging from CachingDirectoryFactory


> Significant log noise at INFO from CachingDirectoryFactory#close
> 
>
> Key: SOLR-4200
> URL: https://issues.apache.org/jira/browse/SOLR-4200
> Project: Solr
>  Issue Type: Bug
>Affects Versions: 4.1
> Environment: solr-impl4.1-SNAPSHOT 1421496 - ncindex - 2012-12-13 
> 14:56:25
>Reporter: Shawn Heisey
> Fix For: 4.2, 5.0
>
> Attachments: SOLR-4200.patch, SOLR-4200.patch
>
>
> The following line in CachingDirectoryFactory is resulting in a lot of noise 
> in my branch_4x logs.  It was added by Yonik's recent major sync from trunk 
> to branch_4x, r1420992.
> log.info("Releasing directory:" + cacheValue.path);
> This was probably added to debug a problem, but it seems to get called a lot. 
>  The specific thing that led me to bring it up is that it is logged four 
> times for every call to /admin/mbeans.  When you've got seven cores and you 
> hit them all, it increases the logging from 7 lines to 35.  IMHO, it should 
> be moved to debug instead of info.
> INFO  - 2012-12-15 13:36:01.674; 
> org.apache.solr.core.CachingDirectoryFactory; Releasing 
> directory:/index/solr4/cores/s0_0/../../data/s0_0
> INFO  - 2012-12-15 13:36:01.676; 
> org.apache.solr.core.CachingDirectoryFactory; Releasing 
> directory:/index/solr4/cores/s0_0/../../data/s0_0/index
> INFO  - 2012-12-15 13:36:01.676; 
> org.apache.solr.core.CachingDirectoryFactory; Releasing 
> directory:/index/solr4/cores/s0_0/../../data/s0_0
> INFO  - 2012-12-15 13:36:01.678; 
> org.apache.solr.core.CachingDirectoryFactory; Releasing 
> directory:/index/solr4/cores/s0_0/../../data/s0_0/index
> INFO  - 2012-12-15 13:36:01.679; org.apache.solr.core.SolrCore; [s0build] 
> webapp=/solr path=/admin/mbeans 
> params={qt.path=/admin/mbeans&wt=javabin&stats=true&version=2} status=0 
> QTime=6

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-4200) Significant log noise at INFO from CachingDirectoryFactory#close

2013-03-06 Thread Hoss Man (JIRA)


 [ 
https://issues.apache.org/jira/browse/SOLR-4200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hoss Man updated SOLR-4200:
---

Attachment: SOLR-4200.patch

updated patch to trunk (looks like some code moved around).

i also swaped which of the two duplicate "Closing directory" was kept/removed 
(seems more prudent to keep the log message in the inner private method doing 
the actual closing since it is called from multiple places and might be called 
by others in the future)

> Significant log noise at INFO from CachingDirectoryFactory#close
> 
>
> Key: SOLR-4200
> URL: https://issues.apache.org/jira/browse/SOLR-4200
> Project: Solr
>  Issue Type: Bug
>Affects Versions: 4.1
> Environment: solr-impl4.1-SNAPSHOT 1421496 - ncindex - 2012-12-13 
> 14:56:25
>Reporter: Shawn Heisey
> Fix For: 4.2, 5.0
>
> Attachments: SOLR-4200.patch, SOLR-4200.patch
>
>
> The following line in CachingDirectoryFactory is resulting in a lot of noise 
> in my branch_4x logs.  It was added by Yonik's recent major sync from trunk 
> to branch_4x, r1420992.
> log.info("Releasing directory:" + cacheValue.path);
> This was probably added to debug a problem, but it seems to get called a lot. 
>  The specific thing that led me to bring it up is that it is logged four 
> times for every call to /admin/mbeans.  When you've got seven cores and you 
> hit them all, it increases the logging from 7 lines to 35.  IMHO, it should 
> be moved to debug instead of info.
> INFO  - 2012-12-15 13:36:01.674; 
> org.apache.solr.core.CachingDirectoryFactory; Releasing 
> directory:/index/solr4/cores/s0_0/../../data/s0_0
> INFO  - 2012-12-15 13:36:01.676; 
> org.apache.solr.core.CachingDirectoryFactory; Releasing 
> directory:/index/solr4/cores/s0_0/../../data/s0_0/index
> INFO  - 2012-12-15 13:36:01.676; 
> org.apache.solr.core.CachingDirectoryFactory; Releasing 
> directory:/index/solr4/cores/s0_0/../../data/s0_0
> INFO  - 2012-12-15 13:36:01.678; 
> org.apache.solr.core.CachingDirectoryFactory; Releasing 
> directory:/index/solr4/cores/s0_0/../../data/s0_0/index
> INFO  - 2012-12-15 13:36:01.679; org.apache.solr.core.SolrCore; [s0build] 
> webapp=/solr path=/admin/mbeans 
> params={qt.path=/admin/mbeans&wt=javabin&stats=true&version=2} status=0 
> QTime=6

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (LUCENE-3918) Port index sorter to trunk APIs


 [ 
https://issues.apache.org/jira/browse/LUCENE-3918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shai Erera updated LUCENE-3918:
---

Attachment: LUCENE-3918.patch

Patch removes IndexSorter (but keeps IndexSortingTest). I also:

* Moved ReverseDocIDSorter to a singleton on Sorter, and made IndexSortingTest 
randomly pick it or NumericDVSorter.

* Removed Payload and StoredFields sorter. As a consequence, removed SorterTest 
(sorters are covered by IndexSortingTest).

* Added example code to SortingAtomicReader jdocs.

> Port index sorter to trunk APIs
> ---
>
> Key: LUCENE-3918
> URL: https://issues.apache.org/jira/browse/LUCENE-3918
> Project: Lucene - Core
>  Issue Type: Task
>  Components: modules/other
>Affects Versions: 4.0-ALPHA
>Reporter: Robert Muir
> Fix For: 4.2, 5.0
>
> Attachments: LUCENE-3918.patch, LUCENE-3918.patch, LUCENE-3918.patch, 
> LUCENE-3918.patch, LUCENE-3918.patch, LUCENE-3918.patch, LUCENE-3918.patch, 
> LUCENE-3918.patch, LUCENE-3918.patch, LUCENE-3918.patch, LUCENE-3918.patch, 
> LUCENE-3918.patch, LUCENE-3918.patch, LUCENE-3918.patch
>
>
> LUCENE-2482 added an IndexSorter to 3.x, but we need to port this
> functionality to 4.0 apis.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-4497) Collection Aliasing.


[ 
https://issues.apache.org/jira/browse/SOLR-4497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13595084#comment-13595084
 ] 

Mark Miller commented on SOLR-4497:
---

bq. how to deal with that case.

So, it's not really as solid a solution as I'd like, but I've handled it this 
way:

The Overseer that actually writes the new Alias will wait until it locally gets 
pinged by ZK and gets the latest Aliases. It then waits a small fudge factor 
for other nodes that may get updated slightly behind and returns.

> Collection Aliasing.
> 
>
> Key: SOLR-4497
> URL: https://issues.apache.org/jira/browse/SOLR-4497
> Project: Solr
>  Issue Type: New Feature
>  Components: SolrCloud
>Reporter: Mark Miller
>Assignee: Mark Miller
> Fix For: 4.2, 5.0
>
> Attachments: SOLR-4497.patch
>
>
> We should bring back the old aliasing feature, but for SolrCloud and with the 
> ability to alias one collection to many.
> The old alias feature was of more limited use and had some problems, so we 
> dropped it, but I think we can do this in a more useful way with SolrCloud, 
> and at a level were it's not invasive to the CoreContainer.
> Initially, the search side will allowing mapping a single alias to multiple 
> collections, but the index side will only support mapping a single alias to a 
> single collection.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-608) scripts using curl should support authentication params


[ 
https://issues.apache.org/jira/browse/SOLR-608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13595060#comment-13595060
 ] 

Per Steffensen commented on SOLR-608:
-

http://wiki.apache.org/solr/SolrSecurity#curl_and_base64_on_linux

> scripts using curl should support authentication params
> ---
>
> Key: SOLR-608
> URL: https://issues.apache.org/jira/browse/SOLR-608
> Project: Solr
>  Issue Type: Improvement
>  Components: replication (scripts)
>Reporter: Hoss Man
>
> All scripts that utilize "curl" should be enhanced such that user 
> authentication based params can be specified and used by curl.  This would 
> make it possible for people to "secure" their Solr servers using Servlet 
> Container authentication features, but still interact with those Solr 
> instances using the scripts out of the box.
> The most straight forward approach would probably be to add a new "curl_args" 
> option in scripts.conf that could could contain any legal curl command line 
> options and would be prepended to the args for all usages of curl in the Solr 
> scripts.
>  

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Comment Edited] (SOLR-4465) Configurable Collectors


[ 
https://issues.apache.org/jira/browse/SOLR-4465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13594852#comment-13594852
 ] 

Joel Bernstein edited comment on SOLR-4465 at 3/6/13 8:10 PM:
--

New patch with a couple of additions:

1) Added code during initialization of the collectorFactories to handle 
condition where default collectorFactory is not defined in solrconfig.xml. This 
makes this patch backwords compatible with 4.* solrconfigs.

Tests now run without null pointers.

2) Added the CollectorSpec class to hold the collector http parameters. This 
class will implement hashCode and equals and will be added to QueryResultKey. 
The QueryResultKey implementation is not in this patch so more work still needs 
be done on getting this cache ready.

This patch has been tested with the default collector factory with sorting and 
ranking queries and in distributed mode.

  was (Author: joel.bernstein):
New patch with a couple of additions:

1) Added code during initialization of the collectorFactories to handle 
condition where default collectorFactory is not defined in solrconfig.xml. This 
makes this patch backwords compatible with 4.* solrconfigs.

Tests now run without null pointers.

2) Added the CollectorSpec class to hold the collector http parameters. This 
class will implement hashCode and equals and will be added to QueryResultKey. 
The QueryResultKey implementation of this is not in this patch so more work 
still needs be done on getting this cache ready.

This patch has been tested with the default collector factory with sorting and 
ranking queries and in distributed mode.
  
> Configurable Collectors
> ---
>
> Key: SOLR-4465
> URL: https://issues.apache.org/jira/browse/SOLR-4465
> Project: Solr
>  Issue Type: New Feature
>  Components: search
>Affects Versions: 4.1
>Reporter: Joel Bernstein
> Fix For: 4.2, 5.0
>
> Attachments: SOLR-4465.patch, SOLR-4465.patch, SOLR-4465.patch, 
> SOLR-4465.patch, SOLR-4465.patch, SOLR-4465.patch, SOLR-4465.patch, 
> SOLR-4465.patch, SOLR-4465.patch, SOLR-4465.patch
>
>
> This issue is to add configurable custom collectors to Solr. This expands the 
> design and work done in issue SOLR-1680 to include:
> 1) CollectorFactory configuration in solconfig.xml
> 2) Http parameters to allow clients to dynamically select a CollectorFactory 
> and construct a custom Collector.
> 3) Make aspects of QueryComponent pluggable so that the output from 
> distributed search can conform with custom collectors at the shard level.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: [jira] [Commented] (LUCENE-3918) Port index sorter to trunk APIs

2013-03-06 Thread eksdev

humpf, do we actually need stored fields?

What is wrong with having byte[] DV  that stores all document  fields,  e.g.  
avro or something simpler  to serialise all document fields into one byte[]?

I am definitely missing something about DV/Stored fields diff,  not sure what?



On Mar 6, 2013, at 8:18 PM, "Andrzej Bialecki  (JIRA)"  wrote:

> 
>[ 
> https://issues.apache.org/jira/browse/LUCENE-3918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13594997#comment-13594997
>  ] 
> 
> Andrzej Bialecki  commented on LUCENE-3918:
> ---
> 
> bq. I still don't get why someone would use stored fields rather than doc 
> values (either binary, sorted or numeric) to sort his index. I think it's 
> important to make users understand that stored fields are only useful to 
> display results?
> 
> This is a legacy of the original usage of this tool in Nutch - indexes would 
> use a PageRank value as a document boost, and that was the value to be used 
> for sorting - but since the doc boost is not recoverable from an existing 
> index the value itself was stored in a stored field.
> 
> And definitely DV didn't exist yet at that time :)
> 
>> Port index sorter to trunk APIs
>> ---
>> 
>>Key: LUCENE-3918
>>URL: https://issues.apache.org/jira/browse/LUCENE-3918
>>Project: Lucene - Core
>> Issue Type: Task
>> Components: modules/other
>>   Affects Versions: 4.0-ALPHA
>>   Reporter: Robert Muir
>>Fix For: 4.2, 5.0
>> 
>>Attachments: LUCENE-3918.patch, LUCENE-3918.patch, LUCENE-3918.patch, 
>> LUCENE-3918.patch, LUCENE-3918.patch, LUCENE-3918.patch, LUCENE-3918.patch, 
>> LUCENE-3918.patch, LUCENE-3918.patch, LUCENE-3918.patch, LUCENE-3918.patch, 
>> LUCENE-3918.patch, LUCENE-3918.patch
>> 
>> 
>> LUCENE-2482 added an IndexSorter to 3.x, but we need to port this
>> functionality to 4.0 apis.
> 
> --
> This message is automatically generated by JIRA.
> If you think it was sent incorrectly, please contact your JIRA administrators
> For more information on JIRA, see: http://www.atlassian.com/software/jira
> 
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org
> 


-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

RE: JVM hanging when using G1GC on JDK8 b78 or b79 (Linux 32 bit)

Hi,

If you want the command line of the JVM running the tests, there is one trick:

Run the ANT test (the final step in my howto). When it hangs (you should see 
ANT printing some "tests stalled" messages on stdout, you go to a separate 
console and kill the worker JVM with kill -9 (the PID is printed by ANT before 
starting the tests).
Once you killed the JVM, the ANT build should proceed and print debugging 
information about the "failed" JVM (the one you killed from the other 
terminal), including the full cmd line with full classpath. So it should be 
possible to spawn this in isolation using the printed command line.

Uwe

-
Uwe Schindler
uschind...@apache.org 
Apache Lucene PMC Member / Committer
Bremen, Germany
http://lucene.apache.org/


> -Original Message-
> From: John Cuthbertson [mailto:john.cuthbert...@oracle.com]
> Sent: Wednesday, March 06, 2013 8:21 PM
> To: Uwe Schindler
> Cc: 'Bengt Rutisson'; hotspot-gc-...@openjdk.java.net;
> dev@lucene.apache.org
> Subject: Re: JVM hanging when using G1GC on JDK8 b78 or b79 (Linux 32 bit)
> 
> Hi Uwe,
> 
> Let me try with your detailed instructions below before you go to all of that
> trouble. I will let you know how I get on.
> 
> Thanks,
> 
> JohnC
> 
> On 3/6/2013 11:15 AM, Uwe Schindler wrote:
> > Hi,
> >
> > That's unfortunately not so easy, because of project dependencies. To run
> the test you have to compile Lucene Core then the specific module + the test
> framework (which is special for Lucene) and download some JARs from
> Maven central (JAR hell, as usual).
> > If you give me some time, I would collect all needed JAR files from my local
> checkout and provide you the correct cmd line + a ZIP file with maybe a shell
> script to startup. It should be doable, but needs some work to collect all
> dependencies for the classpath.
> >
> > If you want to do it quicker (should be quite fast to do):
> > - Download ANT 1.8.2 binary zip (unfortunately ANT 1.8.4 has a bug making
> it not working out of the box with Java 8):
> http://archive.apache.org/dist/ant/binaries/apache-ant-1.8.2-bin.tar.gz - I
> just wonder about the fact: isn't ANT needed to build the JDK classlib by
> itself? I remember that the FreeBSD OpenJDK build downloads ANT and does
> a large part of the compilation using ANT...
> > - put the ANT bin/ dir into your PATH
> > - download the Apache Lucene source code from Jenkins:
> > https://builds.apache.org/job/Lucene-Artifacts-trunk/2212/artifact/luc
> > ene/dist/lucene-5.0-2013-03-05_15-37-06-src.tgz
> > - go to extracted lucene source dir, call "ant ivy-bootstrap" (this
> > will download Apache IVY, so all dependencies can be downloaded from
> > Maven Central)
> > - change to the module that fails: # cd analysis/uima
> > - execute: # ant -Dargs="-server -XX:+UseG1GC" -Dtests.multiplier=3
> > -Dtests.jvms=1 test
> > - In a parallel console you might be able to attach to the process, the 
> > build
> in the main console using ANT runs inside ANT and the test framework
> spawns separate worker instances of the JVM to execute the tests. This
> makes it hard to reproduce in standalone (the command line passed to the
> child JVM is very long).
> >
> > I will work on putting together a precompiled ZIP file with all needed JARs 
> > +
> the command line. Just tell me if you got it managed with the above howto,
> then I don’t need to do this.
> > Uwe
> >
> > -
> > Uwe Schindler
> > uschind...@apache.org
> > Apache Lucene PMC Member / Committer
> > Bremen, Germany
> > http://lucene.apache.org/
> >
> >
> >> -Original Message-
> >> From: John Cuthbertson [mailto:john.cuthbert...@oracle.com]
> >> Sent: Wednesday, March 06, 2013 7:51 PM
> >> To: Uwe Schindler
> >> Cc: 'Bengt Rutisson'; hotspot-gc-...@openjdk.java.net;
> >> dev@lucene.apache.org
> >> Subject: Re: JVM hanging when using G1GC on JDK8 b78 or b79 (Linux 32
> >> bit)
> >>
> >> Hi Uwe,
> >>
> >> I've downloaded  lucene-5.0-2013-03-05_15-37-06.zip from
> >> https://builds.apache.org/job/Lucene-Artifacts-
> >> trunk/2212/artifact/lucene/dist/
> >>
> >> I don't have ant on my workstation so do you have a java command line
> >> to run the test(s) that generate the error?
> >>
> >> Thanks,
> >>
> >> JohnC
> >>
> >> On 3/6/2013 3:16 AM, Uwe Schindler wrote:
> >>> Hi,
> >>>
>  I think this is a VM bug and the thread dumps that Uwe produced are
>  enough to start tracking down the root cause.
> >>> I hope it is enough! If I can help with more details, tell me what I
> >>> should do
> >> to track this down. Unfortunately, we have no isolated test case
> >> (like a small java class that triggers this bug) - you have to run
> >> the test cases of this Lucene's module. It only happens there, not in
> >> any other Lucene test suite. It may be caused by a lot of GC activity in 
> >> this
> "UIMA" module or a specific test.
>  On 3/6/13 8:52 AM, David Holmes wrote:
> > If the VM is completely unresponsive then it suggests we are at a
> > safepoint.
>

[jira] [Created] (SOLR-4534) Shard Aliasing

Mark Miller created SOLR-4534:
-

 Summary: Shard Aliasing
 Key: SOLR-4534
 URL: https://issues.apache.org/jira/browse/SOLR-4534
 Project: Solr
  Issue Type: New Feature
  Components: SolrCloud
Reporter: Mark Miller




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-4497) Collection Aliasing.


[ 
https://issues.apache.org/jira/browse/SOLR-4497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13595013#comment-13595013
 ] 

Mark Miller commented on SOLR-4497:
---

Good point Tim. I imagine I will have to setting for the former initially, but 
def thinking about how to deal with that case.

> Collection Aliasing.
> 
>
> Key: SOLR-4497
> URL: https://issues.apache.org/jira/browse/SOLR-4497
> Project: Solr
>  Issue Type: New Feature
>  Components: SolrCloud
>Reporter: Mark Miller
>Assignee: Mark Miller
> Fix For: 4.2, 5.0
>
> Attachments: SOLR-4497.patch
>
>
> We should bring back the old aliasing feature, but for SolrCloud and with the 
> ability to alias one collection to many.
> The old alias feature was of more limited use and had some problems, so we 
> dropped it, but I think we can do this in a more useful way with SolrCloud, 
> and at a level were it's not invasive to the CoreContainer.
> Initially, the search side will allowing mapping a single alias to multiple 
> collections, but the index side will only support mapping a single alias to a 
> single collection.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: JVM hanging when using G1GC on JDK8 b78 or b79 (Linux 32 bit)

2013-03-06 Thread John Cuthbertson


Hi Uwe,

Let me try with your detailed instructions below before you go to all of 
that trouble. I will let you know how I get on.


Thanks,

JohnC

On 3/6/2013 11:15 AM, Uwe Schindler wrote:

Hi,

That's unfortunately not so easy, because of project dependencies. To run the 
test you have to compile Lucene Core then the specific module + the test 
framework (which is special for Lucene) and download some JARs from Maven 
central (JAR hell, as usual).
If you give me some time, I would collect all needed JAR files from my local 
checkout and provide you the correct cmd line + a ZIP file with maybe a shell 
script to startup. It should be doable, but needs some work to collect all 
dependencies for the classpath.

If you want to do it quicker (should be quite fast to do):
- Download ANT 1.8.2 binary zip (unfortunately ANT 1.8.4 has a bug making it 
not working out of the box with Java 8): 
http://archive.apache.org/dist/ant/binaries/apache-ant-1.8.2-bin.tar.gz - I 
just wonder about the fact: isn't ANT needed to build the JDK classlib by 
itself? I remember that the FreeBSD OpenJDK build downloads ANT and does a 
large part of the compilation using ANT...
- put the ANT bin/ dir into your PATH
- download the Apache Lucene source code from Jenkins: 
https://builds.apache.org/job/Lucene-Artifacts-trunk/2212/artifact/lucene/dist/lucene-5.0-2013-03-05_15-37-06-src.tgz
- go to extracted lucene source dir, call "ant ivy-bootstrap" (this will 
download Apache IVY, so all dependencies can be downloaded from Maven Central)
- change to the module that fails: # cd analysis/uima
- execute: # ant -Dargs="-server -XX:+UseG1GC" -Dtests.multiplier=3 
-Dtests.jvms=1 test
- In a parallel console you might be able to attach to the process, the build 
in the main console using ANT runs inside ANT and the test framework spawns 
separate worker instances of the JVM to execute the tests. This makes it hard 
to reproduce in standalone (the command line passed to the child JVM is 
very long).

I will work on putting together a precompiled ZIP file with all needed JARs + 
the command line. Just tell me if you got it managed with the above howto, then 
I don’t need to do this.
Uwe

-
Uwe Schindler
uschind...@apache.org
Apache Lucene PMC Member / Committer
Bremen, Germany
http://lucene.apache.org/



-Original Message-
From: John Cuthbertson [mailto:john.cuthbert...@oracle.com]
Sent: Wednesday, March 06, 2013 7:51 PM
To: Uwe Schindler
Cc: 'Bengt Rutisson'; hotspot-gc-...@openjdk.java.net;
dev@lucene.apache.org
Subject: Re: JVM hanging when using G1GC on JDK8 b78 or b79 (Linux 32 bit)

Hi Uwe,

I've downloaded  lucene-5.0-2013-03-05_15-37-06.zip from
https://builds.apache.org/job/Lucene-Artifacts-
trunk/2212/artifact/lucene/dist/

I don't have ant on my workstation so do you have a java command line to
run the test(s) that generate the error?

Thanks,

JohnC

On 3/6/2013 3:16 AM, Uwe Schindler wrote:

Hi,


I think this is a VM bug and the thread dumps that Uwe produced are
enough to start tracking down the root cause.

I hope it is enough! If I can help with more details, tell me what I should do

to track this down. Unfortunately, we have no isolated test case (like a small
java class that triggers this bug) - you have to run the test cases of this
Lucene's module. It only happens there, not in any other Lucene test suite. It
may be caused by a lot of GC activity in this "UIMA" module or a specific test.

On 3/6/13 8:52 AM, David Holmes wrote:

If the VM is completely unresponsive then it suggests we are at a
safepoint.

Yes, we are hanging during a stop-the-world GC, so we are at a safepoint.


The GC threads are not "hung" in os::parK, they are parked - waiting
to be notified of something.

It looks like the reference processing thread is stuck in a loop
where it does wait(). So, the VM is hanging even if that stack trace
also ends up in os::park().


The thing is to find out why they are not being woken up.

Actually, in this case we should probably not even be calling wait...


Can the gdb log be posted somewhere? I don't know if the attachment
made it to the original posting on hotspot-gc but it's no longer
available on hotspot-dev.

I received the attachment with the original email. I've attached it
to the bug report that I created: 8009536. You can find it there if
you want to. But I think we have a fairly good idea of what change
caused the hang.

If it helps: Unfortunately, we had some problems with recent JDK builds,

because javac and javadoc tools were not working correctly, failing to build
our source code. Since b78 this was fixed. Until this was fixed, we used build
b65 (which was the last one working) and the G1GC hangs did not appear on
this version. So it must have happened by a change after b65 till b78.

Uwe


Bengt


Thanks,
David

On 6/03/2013 4:07 PM, Krystal Mok wrote:

Hi Uwe,

If you can attach gdb onto it, and jstack -m and jstack -F should
also work; that'll get you the Java stack trace.
(Bu

[jira] [Commented] (LUCENE-3918) Port index sorter to trunk APIs

2013-03-06 Thread Andrzej Bialecki (JIRA)


[ 
https://issues.apache.org/jira/browse/LUCENE-3918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13594997#comment-13594997
 ] 

Andrzej Bialecki  commented on LUCENE-3918:
---

bq. I still don't get why someone would use stored fields rather than doc 
values (either binary, sorted or numeric) to sort his index. I think it's 
important to make users understand that stored fields are only useful to 
display results?

This is a legacy of the original usage of this tool in Nutch - indexes would 
use a PageRank value as a document boost, and that was the value to be used for 
sorting - but since the doc boost is not recoverable from an existing index the 
value itself was stored in a stored field.

And definitely DV didn't exist yet at that time :)

> Port index sorter to trunk APIs
> ---
>
> Key: LUCENE-3918
> URL: https://issues.apache.org/jira/browse/LUCENE-3918
> Project: Lucene - Core
>  Issue Type: Task
>  Components: modules/other
>Affects Versions: 4.0-ALPHA
>Reporter: Robert Muir
> Fix For: 4.2, 5.0
>
> Attachments: LUCENE-3918.patch, LUCENE-3918.patch, LUCENE-3918.patch, 
> LUCENE-3918.patch, LUCENE-3918.patch, LUCENE-3918.patch, LUCENE-3918.patch, 
> LUCENE-3918.patch, LUCENE-3918.patch, LUCENE-3918.patch, LUCENE-3918.patch, 
> LUCENE-3918.patch, LUCENE-3918.patch
>
>
> LUCENE-2482 added an IndexSorter to 3.x, but we need to port this
> functionality to 4.0 apis.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

RE: JVM hanging when using G1GC on JDK8 b78 or b79 (Linux 32 bit)

Hi,

That's unfortunately not so easy, because of project dependencies. To run the 
test you have to compile Lucene Core then the specific module + the test 
framework (which is special for Lucene) and download some JARs from Maven 
central (JAR hell, as usual).
If you give me some time, I would collect all needed JAR files from my local 
checkout and provide you the correct cmd line + a ZIP file with maybe a shell 
script to startup. It should be doable, but needs some work to collect all 
dependencies for the classpath.

If you want to do it quicker (should be quite fast to do):
- Download ANT 1.8.2 binary zip (unfortunately ANT 1.8.4 has a bug making it 
not working out of the box with Java 8): 
http://archive.apache.org/dist/ant/binaries/apache-ant-1.8.2-bin.tar.gz - I 
just wonder about the fact: isn't ANT needed to build the JDK classlib by 
itself? I remember that the FreeBSD OpenJDK build downloads ANT and does a 
large part of the compilation using ANT...
- put the ANT bin/ dir into your PATH
- download the Apache Lucene source code from Jenkins: 
https://builds.apache.org/job/Lucene-Artifacts-trunk/2212/artifact/lucene/dist/lucene-5.0-2013-03-05_15-37-06-src.tgz
- go to extracted lucene source dir, call "ant ivy-bootstrap" (this will 
download Apache IVY, so all dependencies can be downloaded from Maven Central)
- change to the module that fails: # cd analysis/uima
- execute: # ant -Dargs="-server -XX:+UseG1GC" -Dtests.multiplier=3 
-Dtests.jvms=1 test
- In a parallel console you might be able to attach to the process, the build 
in the main console using ANT runs inside ANT and the test framework spawns 
separate worker instances of the JVM to execute the tests. This makes it hard 
to reproduce in standalone (the command line passed to the child JVM is 
very long).

I will work on putting together a precompiled ZIP file with all needed JARs + 
the command line. Just tell me if you got it managed with the above howto, then 
I don’t need to do this.
Uwe

-
Uwe Schindler
uschind...@apache.org 
Apache Lucene PMC Member / Committer
Bremen, Germany
http://lucene.apache.org/


> -Original Message-
> From: John Cuthbertson [mailto:john.cuthbert...@oracle.com]
> Sent: Wednesday, March 06, 2013 7:51 PM
> To: Uwe Schindler
> Cc: 'Bengt Rutisson'; hotspot-gc-...@openjdk.java.net;
> dev@lucene.apache.org
> Subject: Re: JVM hanging when using G1GC on JDK8 b78 or b79 (Linux 32 bit)
> 
> Hi Uwe,
> 
> I've downloaded  lucene-5.0-2013-03-05_15-37-06.zip from
> https://builds.apache.org/job/Lucene-Artifacts-
> trunk/2212/artifact/lucene/dist/
> 
> I don't have ant on my workstation so do you have a java command line to
> run the test(s) that generate the error?
> 
> Thanks,
> 
> JohnC
> 
> On 3/6/2013 3:16 AM, Uwe Schindler wrote:
> > Hi,
> >
> >> I think this is a VM bug and the thread dumps that Uwe produced are
> >> enough to start tracking down the root cause.
> > I hope it is enough! If I can help with more details, tell me what I should 
> > do
> to track this down. Unfortunately, we have no isolated test case (like a small
> java class that triggers this bug) - you have to run the test cases of this
> Lucene's module. It only happens there, not in any other Lucene test suite. It
> may be caused by a lot of GC activity in this "UIMA" module or a specific 
> test.
> >
> >> On 3/6/13 8:52 AM, David Holmes wrote:
> >>> If the VM is completely unresponsive then it suggests we are at a
> >>> safepoint.
> >> Yes, we are hanging during a stop-the-world GC, so we are at a safepoint.
> >>
> >>> The GC threads are not "hung" in os::parK, they are parked - waiting
> >>> to be notified of something.
> >> It looks like the reference processing thread is stuck in a loop
> >> where it does wait(). So, the VM is hanging even if that stack trace
> >> also ends up in os::park().
> >>
> >>> The thing is to find out why they are not being woken up.
> >> Actually, in this case we should probably not even be calling wait...
> >>
> >>> Can the gdb log be posted somewhere? I don't know if the attachment
> >>> made it to the original posting on hotspot-gc but it's no longer
> >>> available on hotspot-dev.
> >> I received the attachment with the original email. I've attached it
> >> to the bug report that I created: 8009536. You can find it there if
> >> you want to. But I think we have a fairly good idea of what change
> >> caused the hang.
> > If it helps: Unfortunately, we had some problems with recent JDK builds,
> because javac and javadoc tools were not working correctly, failing to build
> our source code. Since b78 this was fixed. Until this was fixed, we used build
> b65 (which was the last one working) and the G1GC hangs did not appear on
> this version. So it must have happened by a change after b65 till b78.
> >
> > Uwe
> >
> >> Bengt
> >>
> >>> Thanks,
> >>> David
> >>>
> >>> On 6/03/2013 4:07 PM, Krystal Mok wrote:
>  Hi Uwe,
> 
>  If you can attach gdb onto it, and jstack -m and jstack -F sh

[jira] [Commented] (SOLR-4470) Support for basic http auth in internal solr requests


[ 
https://issues.apache.org/jira/browse/SOLR-4470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13594988#comment-13594988
 ] 

Per Steffensen commented on SOLR-4470:
--

Actually I started out making support for providing "internal credentials" in 
either solr.xml or as VM params. I removed the solr.xml thing again because we 
not are going to use it and it would just make the patch even bigger (and we 
would not like that :-) ). I can dig the solr.xml solution up again, but why 
not make that as a separate jira on top of the current VM params only solution. 
No problem.

Actually in the long run we are not going to use the VM params solution either, 
because credentials can be seen by issuing a "ps" command. But dealing with 
that will also be another jira.

We are not going to use the simple realm.properties based realm that comes with 
jetty. We have made our own realm using crypto hashed credentials persisted in 
Zk. Seemed to be a good solution now that Zk is around anyway, and because Zk 
has a nice watcher feature where realms running in many jettys can receive push 
information about changes in the underlying credentials/roles.

> Support for basic http auth in internal solr requests
> -
>
> Key: SOLR-4470
> URL: https://issues.apache.org/jira/browse/SOLR-4470
> Project: Solr
>  Issue Type: Bug
>  Components: clients - java, multicore, replication (java), SolrCloud
>Affects Versions: 4.0
>Reporter: Per Steffensen
>  Labels: authentication, solrclient, solrcloud
> Fix For: 4.2
>
> Attachments: SOLR-4470_branch_4x_r1452629.patch, 
> SOLR-4470_branch_4x_r1452629.patch
>
>
> We want to protect any HTTP-resource (url). We want to require credentials no 
> matter what kind of HTTP-request you make to a Solr-node.
> It can faily easy be acheived as described on 
> http://wiki.apache.org/solr/SolrSecurity. This problem is that Solr-nodes 
> also make "internal" request to other Solr-nodes, and for it to work 
> credentials need to be provided here also.
> Ideally we would like to "forward" credentials from a particular request to 
> all the "internal" sub-requests it triggers. E.g. for search and update 
> request.
> But there are also "internal" requests
> * that only indirectly/asynchronously triggered from "outside" requests (e.g. 
> shard creation/deletion/etc based on calls to the "Collection API")
> * that do not in any way have relation to an "outside" "super"-request (e.g. 
> replica synching stuff)
> We would like to aim at a solution where "original" credentials are 
> "forwarded" when a request directly/synchronously trigger a subrequest, and 
> fallback to a configured "internal credentials" for the 
> asynchronous/non-rooted requests.
> In our solution we would aim at only supporting basic http auth, but we would 
> like to make a "framework" around it, so that not to much refactoring is 
> needed if you later want to make support for other kinds of auth (e.g. digest)
> We will work at a solution but create this JIRA issue early in order to get 
> input/comments from the community as early as possible.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: JVM hanging when using G1GC on JDK8 b78 or b79 (Linux 32 bit)

2013-03-06 Thread John Cuthbertson


Hi Uwe,

I've downloaded  lucene-5.0-2013-03-05_15-37-06.zip from 
https://builds.apache.org/job/Lucene-Artifacts-trunk/2212/artifact/lucene/dist/


I don't have ant on my workstation so do you have a java command line to 
run the test(s) that generate the error?


Thanks,

JohnC

On 3/6/2013 3:16 AM, Uwe Schindler wrote:

Hi,
  

I think this is a VM bug and the thread dumps that Uwe produced are enough
to start tracking down the root cause.

I hope it is enough! If I can help with more details, tell me what I should do to track 
this down. Unfortunately, we have no isolated test case (like a small java class that 
triggers this bug) - you have to run the test cases of this Lucene's module. It only 
happens there, not in any other Lucene test suite. It may be caused by a lot of GC 
activity in this "UIMA" module or a specific test.


On 3/6/13 8:52 AM, David Holmes wrote:

If the VM is completely unresponsive then it suggests we are at a
safepoint.

Yes, we are hanging during a stop-the-world GC, so we are at a safepoint.


The GC threads are not "hung" in os::parK, they are parked - waiting
to be notified of something.

It looks like the reference processing thread is stuck in a loop where it does
wait(). So, the VM is hanging even if that stack trace also ends up in
os::park().


The thing is to find out why they are not being woken up.

Actually, in this case we should probably not even be calling wait...


Can the gdb log be posted somewhere? I don't know if the attachment
made it to the original posting on hotspot-gc but it's no longer
available on hotspot-dev.

I received the attachment with the original email. I've attached it to
the bug report that I created: 8009536. You can find it there if you
want to. But I think we have a fairly good idea of what change caused
the hang.

If it helps: Unfortunately, we had some problems with recent JDK builds, 
because javac and javadoc tools were not working correctly, failing to build 
our source code. Since b78 this was fixed. Until this was fixed, we used build 
b65 (which was the last one working) and the G1GC hangs did not appear on this 
version. So it must have happened by a change after b65 till b78.

Uwe


Bengt


Thanks,
David

On 6/03/2013 4:07 PM, Krystal Mok wrote:

Hi Uwe,

If you can attach gdb onto it, and jstack -m and jstack -F should also
work; that'll get you the Java stack trace.
(But it probably doesn't matter in this case, because the hang is
probably bug in the VM).

- Kris

On Wed, Mar 6, 2013 at 5:48 AM, Uwe Schindler



wrote:

Hi,

since a few month we are extensively testing various preview builds
of JDK 8 for compatibility with Apache Lucene and Solr, so we can
find any bugs early and prevent the problems we had with the release
of Java 7 two years ago. Currently we have a Linux (Ubuntu 64bit)
Jenkins machine that has various JDKs (JDK 6, JDK 7, JDK 8 snapshot,
IBM J9, older JRockit) installed, choosing a different one with
different hotspot and garbage collector settings on every run of the
test suite (which takes approx. 30-45 minutes).

JDK 8 b79 works so far very well on Linux, we found some strange
behavior in early versions (maybe compiler errors), but no longer at
the moment. There is one configuration that constantly and
reproducibly hangs in one module that is tested: The configuration
uses JDK 8 b79 (same for b78), 32 bit, and G1GC (server or client
does not matter). The JVM running the tests hangs irresponsible
(jstack or kill -3 have no effect/cannot connect, standard kill does
not stop it, only kill -9 actually kills it). It can be reproduced
in this Lucene module 100% (it hangs always).

I was able to connect with GDB to the JVM and get a stack trace on
all threads (see attachment, dump.txt). As you see all threads of
G1GC seem to hang in a syscall (os:park(), a conditional wait in
pthread library). Unfortunately that’s all I can give you. A Java
stacktrace is not possible because the JVM reacts on neither kill -3
nor jstack. With all other garbage collectors it passes the test
without hangs in a few seconds, with 32 bit G1GC it can stand still
for hours. The 64 bit JVM passes with G1GC, so only the 32 bit
variant is affected. Client or Server VM makes no difference.

To reproduce:
- Use a 32 bit JDK 8 b78 or b79 (tested on Linux 64 bit, but this
should not matter)
- Download Lucene Source code (e.g. the snapshot version we were
testing with:
https://builds.apache.org/job/Lucene-Artifacts-

trunk/2212/artifact/lucene/dist/)

- change to directory lucene/analysis/uima and run:
  ant -Dargs="-server -XX:+UseG1GC" -Dtests.multiplier=3
-Dtests.jvms=1 test
After a while the test framework prints "stalled" messages (because
the child VM actually running the test no longer responds). The PID
is also printed. Try to get a stack trace or kill it, no response.
Only kill -9 helps. Choosing another garbage collector in the above
command line makes the test finish after a few seconds, e.g.
-Dargs="-server -XX:+UseConcMa

Re: JVM hanging when using G1GC on JDK8 b78 or b79 (Linux 32 bit)

2013-03-06 Thread Mark Miller

Awesome work Uwe! Nice job getting this some attention.

- mark

On Mar 6, 2013, at 10:41 AM, Uwe Schindler  wrote:

> It seems that there is already an explanation from the Oracle engineer:
> 
>> -Original Message-
>> From: John Cuthbertson [mailto:john.cuthbert...@oracle.com]
>> Sent: Wednesday, March 06, 2013 7:04 PM
>> To: Thomas Schatzl
>> Cc: Uwe Schindler; hotspot-gc-...@openjdk.java.net; 'David Holmes';
>> 'Dawid Weiss'; hotspot-...@openjdk.java.net
>> Subject: Re: JVM hanging when using G1GC on JDK8 b78 or b79 (Linux 32 bit)
>> 
>> Hi Everyone,
>> 
>> All:
>> I've looked at the bug report (haven't tried to reproduce it yet) and Bengt's
>> analysis is correct. The concurrent mark thread is entering the
>> synchronization protocol in a marking step call. That code is waiting for 
>> some
>> non-existent workers to terminate before proceeding. Normally we
>> shouldn't be entering that code but I think we overflowed the global marking
>> stack (I updated the CR at ~1am my time with that conjecture). I think I
>> missed a set_phase() call to tell the parallel terminator that we only have 
>> one
>> thread and it's picking up the number of workers that executed the remark
>> parallel task.
>> 
>> Thomas: you were on the right track with your comment about the marking
>> stack size.
>> 
>> David:
>> Thanks for helping out here. The stack trace you mentioned was for one the
>> refinement threads - a concurrent GC thread. When a concurrent GC thread
>> "joins" the suspendible thread set, it means that it will observe and
>> participate in safepoint operations, i.e. the thread will notice that it 
>> should
>> reach a safepoint and the safepoint synchronizer code will wait for it to 
>> block.
>> When we wish a concurrent GC thread to not observe safepoints, that
>> thread leaves the suspendible thread set. I think the name could be a bit
>> better and Tony, before he left, had a change that used a scoped object to
>> join and leave the STS that hasn't been integrated yet. IIRC Tony wasn't
>> happy with the name he chose for that also.
>> 
>> Uwe:
>> Thanks for bringing this up and my apologies for not replying sooner. I will
>> have a fix fairly soon. If I'm correct about it being caused by overflowing 
>> the
>> marking stack you can work around the issue by increasing the
>> MarkStackSize.you could try increasing it to 2M or 4M entries (which is the
>> current max size).
>> 
>> Cheers,
>> 
>> JohnC
> 
> -
> Uwe Schindler
> H.-H.-Meier-Allee 63, D-28213 Bremen
> http://www.thetaphi.de
> eMail: u...@thetaphi.de
> 
> 
>> -Original Message-
>> From: Uwe Schindler [mailto:u...@thetaphi.de]
>> Sent: Wednesday, March 06, 2013 1:35 PM
>> To: dev@lucene.apache.org
>> Subject: FW: JVM hanging when using G1GC on JDK8 b78 or b79 (Linux 32 bit)
>> 
>> They already understood the G1GC problem with JDK 8 b78/b79 and working
>> on a fix. This was really fast:
>> http://mail.openjdk.java.net/pipermail/hotspot-gc-dev/2013-
>> March/006128.html
>> 
>> Uwe
>> 
>> -
>> Uwe Schindler
>> H.-H.-Meier-Allee 63, D-28213 Bremen
>> http://www.thetaphi.de
>> eMail: u...@thetaphi.de
>> 
> 
> 
> 
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org
> 


-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-3918) Port index sorter to trunk APIs


[ 
https://issues.apache.org/jira/browse/LUCENE-3918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13594956#comment-13594956
 ] 

Shai Erera commented on LUCENE-3918:


Thanks Rob - I didn't know we can check these things :). Certainly better than 
suppressing the entire Codec.

Adrien, thanks for the update as well. So if someone loads NumericDV (default), 
indeed there's no need to copy the values again into an array. If someone uses 
DiskDVFormat though, list.get(i) will access the disk on every call ... but I 
guess that's fine since if someone wanted to save RAM, he should be ready to 
pay the price, and we should respect him.

bq. I still don't get why someone would use stored fields rather than doc 
values (either binary, sorted or numeric) to sort his index. I think it's 
important to make users understand that stored fields are only useful to 
display results?

Someone might have an existing index without DV. Also, who said that a stored 
field used for display cannot be used to sort the index?
But, since it's quite trivial to implement, I'll remove both Payload and 
StoredFields. I'll also make Reverse and Numeric sorters inner classes (though 
public) of Sorter.

I added a check in SortingAtomicReader ctor that old2new.length == 
reader.maxDoc(), to ensure that sorters provide a mapping for every document in 
the index.
I'll get rid of IndexSorter, but keep a test around + add to SortingAR javadocs 
code example how to use it for addIndexes.

Will upload a new patch later.

> Port index sorter to trunk APIs
> ---
>
> Key: LUCENE-3918
> URL: https://issues.apache.org/jira/browse/LUCENE-3918
> Project: Lucene - Core
>  Issue Type: Task
>  Components: modules/other
>Affects Versions: 4.0-ALPHA
>Reporter: Robert Muir
> Fix For: 4.2, 5.0
>
> Attachments: LUCENE-3918.patch, LUCENE-3918.patch, LUCENE-3918.patch, 
> LUCENE-3918.patch, LUCENE-3918.patch, LUCENE-3918.patch, LUCENE-3918.patch, 
> LUCENE-3918.patch, LUCENE-3918.patch, LUCENE-3918.patch, LUCENE-3918.patch, 
> LUCENE-3918.patch, LUCENE-3918.patch
>
>
> LUCENE-2482 added an IndexSorter to 3.x, but we need to port this
> functionality to 4.0 apis.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

RE: JVM hanging when using G1GC on JDK8 b78 or b79 (Linux 32 bit)

It seems that there is already an explanation from the Oracle engineer:

> -Original Message-
> From: John Cuthbertson [mailto:john.cuthbert...@oracle.com]
> Sent: Wednesday, March 06, 2013 7:04 PM
> To: Thomas Schatzl
> Cc: Uwe Schindler; hotspot-gc-...@openjdk.java.net; 'David Holmes';
> 'Dawid Weiss'; hotspot-...@openjdk.java.net
> Subject: Re: JVM hanging when using G1GC on JDK8 b78 or b79 (Linux 32 bit)
> 
> Hi Everyone,
> 
> All:
> I've looked at the bug report (haven't tried to reproduce it yet) and Bengt's
> analysis is correct. The concurrent mark thread is entering the
> synchronization protocol in a marking step call. That code is waiting for some
> non-existent workers to terminate before proceeding. Normally we
> shouldn't be entering that code but I think we overflowed the global marking
> stack (I updated the CR at ~1am my time with that conjecture). I think I
> missed a set_phase() call to tell the parallel terminator that we only have 
> one
> thread and it's picking up the number of workers that executed the remark
> parallel task.
> 
> Thomas: you were on the right track with your comment about the marking
> stack size.
> 
> David:
> Thanks for helping out here. The stack trace you mentioned was for one the
> refinement threads - a concurrent GC thread. When a concurrent GC thread
> "joins" the suspendible thread set, it means that it will observe and
> participate in safepoint operations, i.e. the thread will notice that it 
> should
> reach a safepoint and the safepoint synchronizer code will wait for it to 
> block.
> When we wish a concurrent GC thread to not observe safepoints, that
> thread leaves the suspendible thread set. I think the name could be a bit
> better and Tony, before he left, had a change that used a scoped object to
> join and leave the STS that hasn't been integrated yet. IIRC Tony wasn't
> happy with the name he chose for that also.
> 
> Uwe:
> Thanks for bringing this up and my apologies for not replying sooner. I will
> have a fix fairly soon. If I'm correct about it being caused by overflowing 
> the
> marking stack you can work around the issue by increasing the
> MarkStackSize.you could try increasing it to 2M or 4M entries (which is the
> current max size).
> 
> Cheers,
> 
> JohnC

-
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: u...@thetaphi.de


> -Original Message-
> From: Uwe Schindler [mailto:u...@thetaphi.de]
> Sent: Wednesday, March 06, 2013 1:35 PM
> To: dev@lucene.apache.org
> Subject: FW: JVM hanging when using G1GC on JDK8 b78 or b79 (Linux 32 bit)
> 
> They already understood the G1GC problem with JDK 8 b78/b79 and working
> on a fix. This was really fast:
> http://mail.openjdk.java.net/pipermail/hotspot-gc-dev/2013-
> March/006128.html
> 
> Uwe
> 
> -
> Uwe Schindler
> H.-H.-Meier-Allee 63, D-28213 Bremen
> http://www.thetaphi.de
> eMail: u...@thetaphi.de
> 



-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-3706) Ship setup to log with log4j.


[ 
https://issues.apache.org/jira/browse/SOLR-3706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13594911#comment-13594911
 ] 

Mark Miller commented on SOLR-3706:
---

Thanks for the details Shawn!

> Ship setup to log with log4j.
> -
>
> Key: SOLR-3706
> URL: https://issues.apache.org/jira/browse/SOLR-3706
> Project: Solr
>  Issue Type: Improvement
>Reporter: Mark Miller
>Assignee: Mark Miller
>Priority: Minor
> Fix For: 5.0, 4.3
>
>
> Currently we default to java util logging and it's terrible in my opinion.
> *It's simple built in logger is a 2 line logger.
> *You have to jump through hoops to use your own custom formatter with jetty - 
> either putting your class in the start.jar or other pain in the butt 
> solutions.
> *It can't roll files by date out of the box.
> I'm sure there are more issues, but those are the ones annoying me now. We 
> should switch to log4j - it's much nicer and it's easy to get a nice single 
> line format and roll by date, etc.
> If someone wants to use JUL they still can - but at least users could start 
> with something decent.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (LUCENE-3918) Port index sorter to trunk APIs


 [ 
https://issues.apache.org/jira/browse/LUCENE-3918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Adrien Grand updated LUCENE-3918:
-

Attachment: LUCENE-3918.patch

bq. I use two parallel arrays to sort the documents (docs and values)

I updated the patch to use doc IDs as ords so that values are never swapped 
(only doc IDs) and the numeric doc values don't need to be all loaded in memory.

bq. So one option is to remove the class, but still keep a test around which 
does the addIndexes to make sure it works.

+1

bq. I don't want however to add a main that is limited to NumericDV ... and I 
do think that stored fields / payload value are viable options.

I still don't get why someone would use stored fields rather than doc values 
(either binary, sorted or numeric) to sort his index. I think it's important to 
make users understand that stored fields are only useful to display results?

> Port index sorter to trunk APIs
> ---
>
> Key: LUCENE-3918
> URL: https://issues.apache.org/jira/browse/LUCENE-3918
> Project: Lucene - Core
>  Issue Type: Task
>  Components: modules/other
>Affects Versions: 4.0-ALPHA
>Reporter: Robert Muir
> Fix For: 4.2, 5.0
>
> Attachments: LUCENE-3918.patch, LUCENE-3918.patch, LUCENE-3918.patch, 
> LUCENE-3918.patch, LUCENE-3918.patch, LUCENE-3918.patch, LUCENE-3918.patch, 
> LUCENE-3918.patch, LUCENE-3918.patch, LUCENE-3918.patch, LUCENE-3918.patch, 
> LUCENE-3918.patch, LUCENE-3918.patch
>
>
> LUCENE-2482 added an IndexSorter to 3.x, but we need to port this
> functionality to 4.0 apis.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-4470) Support for basic http auth in internal solr requests


[ 
https://issues.apache.org/jira/browse/SOLR-4470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13594887#comment-13594887
 ] 

Jan Høydahl commented on SOLR-4470:
---

bq. I might do that, but its only a few lines, and will only reduce the patch a 
few percent, so not even you believe that will make a difference 

Actually, the reason I comment at all is that I'm fairly interested in this 
feature myself on behalf of a few customers, so I'd love to see it succeed, but 
right now this is too big for my time and priorities to follow through all the 
way. If it were split in 3 I could probably take one of them.

Back to discussing the architecture here:

To me the approach taken looks sane and not too intrusive.

Even if solr.xml is going away, I think it would make sense to include username 
& password config options in solr.xml as an alternative to passing them as 
JAVA_OPTS. You'll quickly see how that is done, and using ${var} syntax with 
fallback to a pre-defined default, you can choose whether to supply them as 
JAVA_OPTS or directly in solr.xml. The solr.xml approach would be less 
controversial to some I guess. Once solr.xml is nuked, the params will be moved 
to whatever takes its place.

> Support for basic http auth in internal solr requests
> -
>
> Key: SOLR-4470
> URL: https://issues.apache.org/jira/browse/SOLR-4470
> Project: Solr
>  Issue Type: Bug
>  Components: clients - java, multicore, replication (java), SolrCloud
>Affects Versions: 4.0
>Reporter: Per Steffensen
>  Labels: authentication, solrclient, solrcloud
> Fix For: 4.2
>
> Attachments: SOLR-4470_branch_4x_r1452629.patch, 
> SOLR-4470_branch_4x_r1452629.patch
>
>
> We want to protect any HTTP-resource (url). We want to require credentials no 
> matter what kind of HTTP-request you make to a Solr-node.
> It can faily easy be acheived as described on 
> http://wiki.apache.org/solr/SolrSecurity. This problem is that Solr-nodes 
> also make "internal" request to other Solr-nodes, and for it to work 
> credentials need to be provided here also.
> Ideally we would like to "forward" credentials from a particular request to 
> all the "internal" sub-requests it triggers. E.g. for search and update 
> request.
> But there are also "internal" requests
> * that only indirectly/asynchronously triggered from "outside" requests (e.g. 
> shard creation/deletion/etc based on calls to the "Collection API")
> * that do not in any way have relation to an "outside" "super"-request (e.g. 
> replica synching stuff)
> We would like to aim at a solution where "original" credentials are 
> "forwarded" when a request directly/synchronously trigger a subrequest, and 
> fallback to a configured "internal credentials" for the 
> asynchronous/non-rooted requests.
> In our solution we would aim at only supporting basic http auth, but we would 
> like to make a "framework" around it, so that not to much refactoring is 
> needed if you later want to make support for other kinds of auth (e.g. digest)
> We will work at a solution but create this JIRA issue early in order to get 
> input/comments from the community as early as possible.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Comment Edited] (SOLR-4465) Configurable Collectors


[ 
https://issues.apache.org/jira/browse/SOLR-4465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13594852#comment-13594852
 ] 

Joel Bernstein edited comment on SOLR-4465 at 3/6/13 5:24 PM:
--

New patch with a couple of additions:

1) Added code during initialization of the collectorFactories to handle 
condition where default collectorFactory is not defined in solrconfig.xml. This 
makes this patch backwords compatible with 4.* solrconfigs.

Tests now run without null pointers.

2) Added the CollectorSpec class to hold the collector http parameters. This 
class will implement hashCode and equals and will be added to QueryResultKey. 
The QueryResultKey implementation of this is not in this patch so more work 
still needs be done on getting this cache ready.

This patch has been tested with the default collector factory with sorting and 
ranking queries and in distributed mode.

  was (Author: joel.bernstein):
New patch with a couple of additions:

1) Added code during initialization of the collectorFactories to handle 
condition where default collectorFactory is not defined in solrconfig.xml. This 
makes this patch backwords compatible with 4.* solrconfigs.

Tests now run without null pointers.

2) Added the CollectorSpec class to hold the collector http parameters. This 
class will implement hashCode and equals and will be added to QueryResultKey. 
The QueryResultKey implementation of this is not in this patch so more work 
still needs be done on getting this cache ready.

I have tested this patch with the default collector factory with sorting and 
ranking queries and in distributed mode.
  
> Configurable Collectors
> ---
>
> Key: SOLR-4465
> URL: https://issues.apache.org/jira/browse/SOLR-4465
> Project: Solr
>  Issue Type: New Feature
>  Components: search
>Affects Versions: 4.1
>Reporter: Joel Bernstein
> Fix For: 4.2, 5.0
>
> Attachments: SOLR-4465.patch, SOLR-4465.patch, SOLR-4465.patch, 
> SOLR-4465.patch, SOLR-4465.patch, SOLR-4465.patch, SOLR-4465.patch, 
> SOLR-4465.patch, SOLR-4465.patch, SOLR-4465.patch
>
>
> This issue is to add configurable custom collectors to Solr. This expands the 
> design and work done in issue SOLR-1680 to include:
> 1) CollectorFactory configuration in solconfig.xml
> 2) Http parameters to allow clients to dynamically select a CollectorFactory 
> and construct a custom Collector.
> 3) Make aspects of QueryComponent pluggable so that the output from 
> distributed search can conform with custom collectors at the shard level.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Comment Edited] (SOLR-4465) Configurable Collectors


[ 
https://issues.apache.org/jira/browse/SOLR-4465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13594852#comment-13594852
 ] 

Joel Bernstein edited comment on SOLR-4465 at 3/6/13 5:22 PM:
--

New patch with a couple of additions:

1) Added code during initialization of the collectorFactories to handle 
condition where default collectorFactory is not defined in solrconfig.xml. This 
makes this patch backwords compatible with 4.* solrconfigs.

Tests now run without null pointers.

2) Added the CollectorSpec class to hold the collector http parameters. This 
class will implement hashCode and equals and will be added to QueryResultKey. 
The QueryResultKey implementation of this is not in this patch so more work 
still needs be done on getting this cache ready.

I have tested this patch with the default collector factory with sorting and 
ranking queries and in distributed mode.

  was (Author: joel.bernstein):
New patch with a couple of additions:

1) Added coded during initialization of the collectorFactories to handle 
condition where default collectorFactory is not defined in solrconfig. This 
makes this patch backwords compatible with 4.* solrconfigs.

Tests now run with out null pointers.

2) Added the CollectorSpec class to hold the collector http parameters. This 
class will implement hashCode and equals and will be added to QueryResultKey. 
The QueryResultKey implementation of this is not in this patch so more work 
still needs be done on getting this cache ready.

I have tested this patch with the default collector factory with sorting and 
ranking queries and in distributed mode.
  
> Configurable Collectors
> ---
>
> Key: SOLR-4465
> URL: https://issues.apache.org/jira/browse/SOLR-4465
> Project: Solr
>  Issue Type: New Feature
>  Components: search
>Affects Versions: 4.1
>Reporter: Joel Bernstein
> Fix For: 4.2, 5.0
>
> Attachments: SOLR-4465.patch, SOLR-4465.patch, SOLR-4465.patch, 
> SOLR-4465.patch, SOLR-4465.patch, SOLR-4465.patch, SOLR-4465.patch, 
> SOLR-4465.patch, SOLR-4465.patch, SOLR-4465.patch
>
>
> This issue is to add configurable custom collectors to Solr. This expands the 
> design and work done in issue SOLR-1680 to include:
> 1) CollectorFactory configuration in solconfig.xml
> 2) Http parameters to allow clients to dynamically select a CollectorFactory 
> and construct a custom Collector.
> 3) Make aspects of QueryComponent pluggable so that the output from 
> distributed search can conform with custom collectors at the shard level.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-4465) Configurable Collectors


 [ 
https://issues.apache.org/jira/browse/SOLR-4465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joel Bernstein updated SOLR-4465:
-

Attachment: SOLR-4465.patch

> Configurable Collectors
> ---
>
> Key: SOLR-4465
> URL: https://issues.apache.org/jira/browse/SOLR-4465
> Project: Solr
>  Issue Type: New Feature
>  Components: search
>Affects Versions: 4.1
>Reporter: Joel Bernstein
> Fix For: 4.2, 5.0
>
> Attachments: SOLR-4465.patch, SOLR-4465.patch, SOLR-4465.patch, 
> SOLR-4465.patch, SOLR-4465.patch, SOLR-4465.patch, SOLR-4465.patch, 
> SOLR-4465.patch, SOLR-4465.patch, SOLR-4465.patch
>
>
> This issue is to add configurable custom collectors to Solr. This expands the 
> design and work done in issue SOLR-1680 to include:
> 1) CollectorFactory configuration in solconfig.xml
> 2) Http parameters to allow clients to dynamically select a CollectorFactory 
> and construct a custom Collector.
> 3) Make aspects of QueryComponent pluggable so that the output from 
> distributed search can conform with custom collectors at the shard level.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (LUCENE-4813) Allow DirectSpellchecker to use totalTermFrequency rather than docFrequency

2013-03-06 Thread Simon Willnauer (JIRA)


 [ 
https://issues.apache.org/jira/browse/LUCENE-4813?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Simon Willnauer updated LUCENE-4813:


Attachment: LUCENE-4813.patch

here is an initial patch that adds this as the default yet optional statistics.

> Allow DirectSpellchecker to use totalTermFrequency rather than docFrequency
> ---
>
> Key: LUCENE-4813
> URL: https://issues.apache.org/jira/browse/LUCENE-4813
> Project: Lucene - Core
>  Issue Type: Bug
>  Components: modules/spellchecker
>Affects Versions: 4.1
>Reporter: Simon Willnauer
> Fix For: 4.2, 5.0
>
> Attachments: LUCENE-4813.patch
>
>
> we have a bunch of new statistics in on our term dictionaries that we should 
> make use of where it makes sense. For DirectSpellChecker totalTermFreq and 
> sumTotalTermFreq might be better suited for spell correction on top of a 
> fulltext index than docFreq and maxDoc

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Created] (LUCENE-4813) Allow DirectSpellchecker to use totalTermFrequency rather than docFrequency

2013-03-06 Thread Simon Willnauer (JIRA)

Simon Willnauer created LUCENE-4813:
---

 Summary: Allow DirectSpellchecker to use totalTermFrequency rather 
than docFrequency
 Key: LUCENE-4813
 URL: https://issues.apache.org/jira/browse/LUCENE-4813
 Project: Lucene - Core
  Issue Type: Bug
  Components: modules/spellchecker
Affects Versions: 4.1
Reporter: Simon Willnauer
 Fix For: 4.2, 5.0


we have a bunch of new statistics in on our term dictionaries that we should 
make use of where it makes sense. For DirectSpellChecker totalTermFreq and 
sumTotalTermFreq might be better suited for spell correction on top of a 
fulltext index than docFreq and maxDoc

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (LUCENE-3918) Port index sorter to trunk APIs

2013-03-06 Thread Robert Muir (JIRA)


 [ 
https://issues.apache.org/jira/browse/LUCENE-3918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Muir updated LUCENE-3918:


Attachment: LUCENE-3918.patch

patch fixing tests to not suppress whole codecs.

instead the testSortedSet() has an assume (and is ignored for ancient codecs).

in the case of offsets, ancient codecs just index and test docs/freqs/positions 
without offsets

> Port index sorter to trunk APIs
> ---
>
> Key: LUCENE-3918
> URL: https://issues.apache.org/jira/browse/LUCENE-3918
> Project: Lucene - Core
>  Issue Type: Task
>  Components: modules/other
>Affects Versions: 4.0-ALPHA
>Reporter: Robert Muir
> Fix For: 4.2, 5.0
>
> Attachments: LUCENE-3918.patch, LUCENE-3918.patch, LUCENE-3918.patch, 
> LUCENE-3918.patch, LUCENE-3918.patch, LUCENE-3918.patch, LUCENE-3918.patch, 
> LUCENE-3918.patch, LUCENE-3918.patch, LUCENE-3918.patch, LUCENE-3918.patch, 
> LUCENE-3918.patch
>
>
> LUCENE-2482 added an IndexSorter to 3.x, but we need to port this
> functionality to 4.0 apis.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-4465) Configurable Collectors


 [ 
https://issues.apache.org/jira/browse/SOLR-4465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joel Bernstein updated SOLR-4465:
-

Attachment: (was: SOLR-4465.patch)

> Configurable Collectors
> ---
>
> Key: SOLR-4465
> URL: https://issues.apache.org/jira/browse/SOLR-4465
> Project: Solr
>  Issue Type: New Feature
>  Components: search
>Affects Versions: 4.1
>Reporter: Joel Bernstein
> Fix For: 4.2, 5.0
>
> Attachments: SOLR-4465.patch, SOLR-4465.patch, SOLR-4465.patch, 
> SOLR-4465.patch, SOLR-4465.patch, SOLR-4465.patch, SOLR-4465.patch, 
> SOLR-4465.patch, SOLR-4465.patch
>
>
> This issue is to add configurable custom collectors to Solr. This expands the 
> design and work done in issue SOLR-1680 to include:
> 1) CollectorFactory configuration in solconfig.xml
> 2) Http parameters to allow clients to dynamically select a CollectorFactory 
> and construct a custom Collector.
> 3) Make aspects of QueryComponent pluggable so that the output from 
> distributed search can conform with custom collectors at the shard level.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-4465) Configurable Collectors


 [ 
https://issues.apache.org/jira/browse/SOLR-4465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joel Bernstein updated SOLR-4465:
-

Attachment: SOLR-4465.patch

New patch with a couple of additions:

1) Added coded during initialization of the collectorFactories to handle 
condition where default collectorFactory is not defined in solrconfig. This 
makes this patch backwords compatible with 4.* solrconfigs.

Tests now run with out null pointers.

2) Added the CollectorSpec class to hold the collector http parameters. This 
class will implement hashCode and equals and will be added to QueryResultKey. 
The QueryResultKey implementation of this is not in this patch so more work 
still needs be done on getting this cache ready.

I have tested this patch with the default collector factory with sorting and 
ranking queries and in distributed mode.

> Configurable Collectors
> ---
>
> Key: SOLR-4465
> URL: https://issues.apache.org/jira/browse/SOLR-4465
> Project: Solr
>  Issue Type: New Feature
>  Components: search
>Affects Versions: 4.1
>Reporter: Joel Bernstein
> Fix For: 4.2, 5.0
>
> Attachments: SOLR-4465.patch, SOLR-4465.patch, SOLR-4465.patch, 
> SOLR-4465.patch, SOLR-4465.patch, SOLR-4465.patch, SOLR-4465.patch, 
> SOLR-4465.patch, SOLR-4465.patch, SOLR-4465.patch
>
>
> This issue is to add configurable custom collectors to Solr. This expands the 
> design and work done in issue SOLR-1680 to include:
> 1) CollectorFactory configuration in solconfig.xml
> 2) Http parameters to allow clients to dynamically select a CollectorFactory 
> and construct a custom Collector.
> 3) Make aspects of QueryComponent pluggable so that the output from 
> distributed search can conform with custom collectors at the shard level.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-4200) Significant log noise at INFO from CachingDirectoryFactory#close

2013-03-06 Thread Otis Gospodnetic (JIRA)


 [ 
https://issues.apache.org/jira/browse/SOLR-4200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Otis Gospodnetic updated SOLR-4200:
---

Priority: Major  (was: Minor)

> Significant log noise at INFO from CachingDirectoryFactory#close
> 
>
> Key: SOLR-4200
> URL: https://issues.apache.org/jira/browse/SOLR-4200
> Project: Solr
>  Issue Type: Bug
>Affects Versions: 4.1
> Environment: solr-impl4.1-SNAPSHOT 1421496 - ncindex - 2012-12-13 
> 14:56:25
>Reporter: Shawn Heisey
> Fix For: 4.2, 5.0
>
> Attachments: SOLR-4200.patch
>
>
> The following line in CachingDirectoryFactory is resulting in a lot of noise 
> in my branch_4x logs.  It was added by Yonik's recent major sync from trunk 
> to branch_4x, r1420992.
> log.info("Releasing directory:" + cacheValue.path);
> This was probably added to debug a problem, but it seems to get called a lot. 
>  The specific thing that led me to bring it up is that it is logged four 
> times for every call to /admin/mbeans.  When you've got seven cores and you 
> hit them all, it increases the logging from 7 lines to 35.  IMHO, it should 
> be moved to debug instead of info.
> INFO  - 2012-12-15 13:36:01.674; 
> org.apache.solr.core.CachingDirectoryFactory; Releasing 
> directory:/index/solr4/cores/s0_0/../../data/s0_0
> INFO  - 2012-12-15 13:36:01.676; 
> org.apache.solr.core.CachingDirectoryFactory; Releasing 
> directory:/index/solr4/cores/s0_0/../../data/s0_0/index
> INFO  - 2012-12-15 13:36:01.676; 
> org.apache.solr.core.CachingDirectoryFactory; Releasing 
> directory:/index/solr4/cores/s0_0/../../data/s0_0
> INFO  - 2012-12-15 13:36:01.678; 
> org.apache.solr.core.CachingDirectoryFactory; Releasing 
> directory:/index/solr4/cores/s0_0/../../data/s0_0/index
> INFO  - 2012-12-15 13:36:01.679; org.apache.solr.core.SolrCore; [s0build] 
> webapp=/solr path=/admin/mbeans 
> params={qt.path=/admin/mbeans&wt=javabin&stats=true&version=2} status=0 
> QTime=6

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-4200) Significant log noise at INFO from CachingDirectoryFactory#close

2013-03-06 Thread Otis Gospodnetic (JIRA)


[ 
https://issues.apache.org/jira/browse/SOLR-4200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13594851#comment-13594851
 ] 

Otis Gospodnetic commented on SOLR-4200:


For what it's worth, I heard several users on the ML and off the ML complain 
about this filling up their logs.


> Significant log noise at INFO from CachingDirectoryFactory#close
> 
>
> Key: SOLR-4200
> URL: https://issues.apache.org/jira/browse/SOLR-4200
> Project: Solr
>  Issue Type: Bug
>Affects Versions: 4.1
> Environment: solr-impl4.1-SNAPSHOT 1421496 - ncindex - 2012-12-13 
> 14:56:25
>Reporter: Shawn Heisey
>Priority: Minor
> Fix For: 4.2, 5.0
>
> Attachments: SOLR-4200.patch
>
>
> The following line in CachingDirectoryFactory is resulting in a lot of noise 
> in my branch_4x logs.  It was added by Yonik's recent major sync from trunk 
> to branch_4x, r1420992.
> log.info("Releasing directory:" + cacheValue.path);
> This was probably added to debug a problem, but it seems to get called a lot. 
>  The specific thing that led me to bring it up is that it is logged four 
> times for every call to /admin/mbeans.  When you've got seven cores and you 
> hit them all, it increases the logging from 7 lines to 35.  IMHO, it should 
> be moved to debug instead of info.
> INFO  - 2012-12-15 13:36:01.674; 
> org.apache.solr.core.CachingDirectoryFactory; Releasing 
> directory:/index/solr4/cores/s0_0/../../data/s0_0
> INFO  - 2012-12-15 13:36:01.676; 
> org.apache.solr.core.CachingDirectoryFactory; Releasing 
> directory:/index/solr4/cores/s0_0/../../data/s0_0/index
> INFO  - 2012-12-15 13:36:01.676; 
> org.apache.solr.core.CachingDirectoryFactory; Releasing 
> directory:/index/solr4/cores/s0_0/../../data/s0_0
> INFO  - 2012-12-15 13:36:01.678; 
> org.apache.solr.core.CachingDirectoryFactory; Releasing 
> directory:/index/solr4/cores/s0_0/../../data/s0_0/index
> INFO  - 2012-12-15 13:36:01.679; org.apache.solr.core.SolrCore; [s0build] 
> webapp=/solr path=/admin/mbeans 
> params={qt.path=/admin/mbeans&wt=javabin&stats=true&version=2} status=0 
> QTime=6

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (LUCENE-4734) FastVectorHighlighter Overlapping Proximity Queries Do Not Highlight

2013-03-06 Thread Ryan Lauck (JIRA)


 [ 
https://issues.apache.org/jira/browse/LUCENE-4734?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ryan Lauck updated LUCENE-4734:
---

Attachment: (was: lucene-fvh-slop.patch)

> FastVectorHighlighter Overlapping Proximity Queries Do Not Highlight
> 
>
> Key: LUCENE-4734
> URL: https://issues.apache.org/jira/browse/LUCENE-4734
> Project: Lucene - Core
>  Issue Type: Bug
>  Components: modules/highlighter
>Affects Versions: 4.0, 4.1, 5.0
>Reporter: Ryan Lauck
>  Labels: fastvectorhighlighter, highlighter
> Fix For: 4.2, 5.0
>
> Attachments: lucene-4734.patch
>
>
> If a proximity phrase query overlaps with any other query term it will not be 
> highlighted.
> Example Text:  A B C D E F G
> Example Queries: 
> "B E"~10 D
> (D will be highlighted instead of "B C D E")
> "B E"~10 "C F"~10
> (nothing will be highlighted)
> This can be traced to the FieldPhraseList constructor's inner while loop. 
> From the first example query, the first TermInfo popped off the stack will be 
> "B". The second TermInfo will be "D" which will not be found in the submap 
> for "B E"~10 and will trigger a failed match.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (LUCENE-4734) FastVectorHighlighter Overlapping Proximity Queries Do Not Highlight

2013-03-06 Thread Ryan Lauck (JIRA)


 [ 
https://issues.apache.org/jira/browse/LUCENE-4734?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ryan Lauck updated LUCENE-4734:
---

Attachment: (was: lucene-fvh-slop-reverse.patch)

> FastVectorHighlighter Overlapping Proximity Queries Do Not Highlight
> 
>
> Key: LUCENE-4734
> URL: https://issues.apache.org/jira/browse/LUCENE-4734
> Project: Lucene - Core
>  Issue Type: Bug
>  Components: modules/highlighter
>Affects Versions: 4.0, 4.1, 5.0
>Reporter: Ryan Lauck
>  Labels: fastvectorhighlighter, highlighter
> Fix For: 4.2, 5.0
>
> Attachments: lucene-4734.patch
>
>
> If a proximity phrase query overlaps with any other query term it will not be 
> highlighted.
> Example Text:  A B C D E F G
> Example Queries: 
> "B E"~10 D
> (D will be highlighted instead of "B C D E")
> "B E"~10 "C F"~10
> (nothing will be highlighted)
> This can be traced to the FieldPhraseList constructor's inner while loop. 
> From the first example query, the first TermInfo popped off the stack will be 
> "B". The second TermInfo will be "D" which will not be found in the submap 
> for "B E"~10 and will trigger a failed match.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (LUCENE-4734) FastVectorHighlighter Overlapping Proximity Queries Do Not Highlight

2013-03-06 Thread Ryan Lauck (JIRA)


 [ 
https://issues.apache.org/jira/browse/LUCENE-4734?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ryan Lauck updated LUCENE-4734:
---

Attachment: lucene-4734.patch

Store the max possible slop on the QueryPhraseMap rather than the entire 
FieldQuery. This limits unnecessary matching when a PhraseQuery with a large 
slop is combined with other PhraseQuerys.

Also, I added a fragment of slop recalculation code from 
WeightedSpanTermExtractor that handles PhraseQuerys with position gaps. The 
most common way this is encountered is by searching a phrase that contains stop 
words while using an analyzer that filters them.

Also cleaned up the test cases a little, and added a few comments. 

> FastVectorHighlighter Overlapping Proximity Queries Do Not Highlight
> 
>
> Key: LUCENE-4734
> URL: https://issues.apache.org/jira/browse/LUCENE-4734
> Project: Lucene - Core
>  Issue Type: Bug
>  Components: modules/highlighter
>Affects Versions: 4.0, 4.1, 5.0
>Reporter: Ryan Lauck
>  Labels: fastvectorhighlighter, highlighter
> Fix For: 4.2, 5.0
>
> Attachments: lucene-4734.patch, lucene-fvh-slop.patch, 
> lucene-fvh-slop-reverse.patch
>
>
> If a proximity phrase query overlaps with any other query term it will not be 
> highlighted.
> Example Text:  A B C D E F G
> Example Queries: 
> "B E"~10 D
> (D will be highlighted instead of "B C D E")
> "B E"~10 "C F"~10
> (nothing will be highlighted)
> This can be traced to the FieldPhraseList constructor's inner while loop. 
> From the first example query, the first TermInfo popped off the stack will be 
> "B". The second TermInfo will be "D" which will not be found in the submap 
> for "B E"~10 and will trigger a failed match.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-4470) Support for basic http auth in internal solr requests


[ 
https://issues.apache.org/jira/browse/SOLR-4470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13594817#comment-13594817
 ] 

Per Steffensen commented on SOLR-4470:
--

Added a few (well a lot) words to http://wiki.apache.org/solr/SolrSecurity. 
Both elaborating on setting up and using security in general, but also how my 
patch will change things. The "with SOLR-4470" and "without SOLR-4470" parts 
can be removed when/if the patch gets committed. Please do not delete the 
SOLR-4470 related descriptions, because someone might want to use and 
understand what the patch does for you, even though it is never committed.

bq. Please read the Do's and Don'ts in 
http://wiki.apache.org/solr/HowToContribute#Creating_the_patch_file

I did, but just because things are mentioned there, it does not make them less 
"bad" or "good". Actually I am trying to change the "rules" there and in the 
minds of committers in general

bq. It's not about forbidding this or that, it's just that if a single patch 
touches too much, history shows that it is hard to maintain and relate to for 
all the different persons, both committers and contributors.

I understand and agree that this is a valid downside of big patches, but the 
alternative apparently is that refactoring is not done - enough.

bq. There are many examples of patches in the past doing only refactoring or 
only formatting changes

Glad to hear that this happens. But apparently not enough - look at the code! - 
structure, reparation of concerns, etc.

bq. Also, if a change is considered major or risky we have historically started 
in a branch

That is a nice strategy, and my patch might be considered major or risky to 
some. I am completely in on the branch thingy. You are very welcome to branch 
and add the patch - let me know if I can help in any way. I just fear that it 
will sit on the branch and rot - and basically I do not care where the patch 
rots :-) I am interested in getting it committed and used in a future release, 
so that others can benefit, and so that there is less potential merging 
conflicts when we want to upgrade our "local version of Solr" (which includes 
this patch, because we need it in production) to include new code from Apache 
Solr.

bq. Rather than trying to book a certain committer up-front to promise to work 
on the code you should incrementally continue work on it yourself, listening to 
feedback, rework the patch/branch

I would like to, but I do not want to put in the work as long as I fear that no 
one want to work with me, no matter what I do. I didnt ask for a promise, but 
just an indication that some committer (at least at the time being) has the 
intention to participate.

And this way of working will make it a very big and long-term process to get in 
the complete patch. I am not hired to work on Solr, but to deliver a product to 
a customer, a product which happens to be based on Solr. I can do a perfectly 
nice piece of work introducing this feature in a week or two, for it to be used 
in our application. My customer is glad to pay me doing that, but he does not 
understand how I can make the change in a week or two, see it work in 
production, but that I have to use half a year getting it in at Apache Solr - 
they ought to be happy that we already contributed a few weeks of work to 
introduce a feature that works, he thinks.

bq. So a first suggestion for one thing you could start with is factoring out 
the fix for the PeerSync.close issue into an own patch and get that committed, 
since it is a side-issue.

I might do that, but its only a few lines, and will only reduce the patch a few 
percent, so not even you believe that will make a difference :-)


> Support for basic http auth in internal solr requests
> -
>
> Key: SOLR-4470
> URL: https://issues.apache.org/jira/browse/SOLR-4470
> Project: Solr
>  Issue Type: Bug
>  Components: clients - java, multicore, replication (java), SolrCloud
>Affects Versions: 4.0
>Reporter: Per Steffensen
>  Labels: authentication, solrclient, solrcloud
> Fix For: 4.2
>
> Attachments: SOLR-4470_branch_4x_r1452629.patch, 
> SOLR-4470_branch_4x_r1452629.patch
>
>
> We want to protect any HTTP-resource (url). We want to require credentials no 
> matter what kind of HTTP-request you make to a Solr-node.
> It can faily easy be acheived as described on 
> http://wiki.apache.org/solr/SolrSecurity. This problem is that Solr-nodes 
> also make "internal" request to other Solr-nodes, and for it to work 
> credentials need to be provided here also.
> Ideally we would like to "forward" credentials from a particular request to 
> all the "internal" sub-requests it triggers. E.g. for search and update 
> request.
> But there are also "internal" requests

[jira] [Commented] (LUCENE-3918) Port index sorter to trunk APIs


[ 
https://issues.apache.org/jira/browse/LUCENE-3918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13594814#comment-13594814
 ] 

Shai Erera commented on LUCENE-3918:


bq. Maybe a List instead of an array would be even better

The thing is, I use two parallel arrays to sort the documents (docs and values) 
and SorterTemplate swaps entries in the arrays, so I think that kinda means we 
need to load everything into memory (well, certainly we cannot swap entries on 
disk)? But this is just the initial loading used to determine the docs order 
... and also, someone can implement a Sorter.oldToNew without loading anything 
into RAM, but I doubt if anyone will.

As for IndexSorter utility, I thought about adding main(), but then sortField 
is not enough (only works for NumericDV). On the other hand, I do get your 
point that there's not much use for a Java API that takes Directory-ies... and 
I don't want to add API that takes reader/writer and only calls .addIndexes. So 
one option is to remove the class, but still keep a test around which does the 
addIndexes to make sure it works.

I don't want however to add a main that is limited to NumericDV ... and I do 
think that stored fields / payload value are viable options. At least, stored 
field is, I can let go of the payload, if that really bothers anyone.

> Port index sorter to trunk APIs
> ---
>
> Key: LUCENE-3918
> URL: https://issues.apache.org/jira/browse/LUCENE-3918
> Project: Lucene - Core
>  Issue Type: Task
>  Components: modules/other
>Affects Versions: 4.0-ALPHA
>Reporter: Robert Muir
> Fix For: 4.2, 5.0
>
> Attachments: LUCENE-3918.patch, LUCENE-3918.patch, LUCENE-3918.patch, 
> LUCENE-3918.patch, LUCENE-3918.patch, LUCENE-3918.patch, LUCENE-3918.patch, 
> LUCENE-3918.patch, LUCENE-3918.patch, LUCENE-3918.patch, LUCENE-3918.patch
>
>
> LUCENE-2482 added an IndexSorter to 3.x, but we need to port this
> functionality to 4.0 apis.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-3918) Port index sorter to trunk APIs


[ 
https://issues.apache.org/jira/browse/LUCENE-3918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13594799#comment-13594799
 ] 

Shai Erera commented on LUCENE-3918:


Robert, now that I think about it again, I'm not sure if SortingAtomicReader 
should do anything to prevent usage over such Codecs. The list of codecs that 
do not support indexing offsets are there because the test indexes offsets. But 
SortingAtomicReader itself doesn't care about whether the Codec supported or 
not offsets -- it will get -1 for offsets (that's what happened before I set 
IndexOptions correctly!). I think that's fine?

As for the new DV features, again, the reader itself is immune to this? I.e. 
you would get same exception if you opened a 4.0 index and called 
getSortedSetDV(i)?

I think it's fine that the reader doesn't yell at you, and I wish we could test 
those codecs somehow, ignoring the unsupported methods, but I also think that 
these tests are not the ones that just *must* test all Codecs. It's a generic 
filter reader and it will yell at you just like any other filter reader will, 
if you call unsupported API?

> Port index sorter to trunk APIs
> ---
>
> Key: LUCENE-3918
> URL: https://issues.apache.org/jira/browse/LUCENE-3918
> Project: Lucene - Core
>  Issue Type: Task
>  Components: modules/other
>Affects Versions: 4.0-ALPHA
>Reporter: Robert Muir
> Fix For: 4.2, 5.0
>
> Attachments: LUCENE-3918.patch, LUCENE-3918.patch, LUCENE-3918.patch, 
> LUCENE-3918.patch, LUCENE-3918.patch, LUCENE-3918.patch, LUCENE-3918.patch, 
> LUCENE-3918.patch, LUCENE-3918.patch, LUCENE-3918.patch, LUCENE-3918.patch
>
>
> LUCENE-2482 added an IndexSorter to 3.x, but we need to port this
> functionality to 4.0 apis.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-3918) Port index sorter to trunk APIs


[ 
https://issues.apache.org/jira/browse/LUCENE-3918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13594787#comment-13594787
 ] 

Adrien Grand commented on LUCENE-3918:
--

Regarding PayloadSorter and StoredFieldsSorter I'm just afraid that the fact 
that they exist might make users think these are viable options...

bq. IndexSorter is a convenient utility for sorting a Directory end-to-end. Why 
remove it?

I think taking an AtomicReader as an argument (instead of a Directory) and 
feeding an IndexWriter (instead of another Directory) would be much more 
flexible but then it would just be a call to IndexWriter.addIndexes... If we 
want an utility to sort indexes, maybe it should rather be something callable 
from command-line? (java oal.index.sorter.IndexSorter fromDir toDir sortField)

bq. Get rid of SortDoc. Sorter is now abstract class with a helper int[] 
compute(int[] docs, T[] values)

I think it's better! Maybe a List instead of an array would be even better so 
that NumericDocValuesSorter could use a view over the doc values instead of 
loading all of them into memory?

Reusage of DocsEnum looks great!






> Port index sorter to trunk APIs
> ---
>
> Key: LUCENE-3918
> URL: https://issues.apache.org/jira/browse/LUCENE-3918
> Project: Lucene - Core
>  Issue Type: Task
>  Components: modules/other
>Affects Versions: 4.0-ALPHA
>Reporter: Robert Muir
> Fix For: 4.2, 5.0
>
> Attachments: LUCENE-3918.patch, LUCENE-3918.patch, LUCENE-3918.patch, 
> LUCENE-3918.patch, LUCENE-3918.patch, LUCENE-3918.patch, LUCENE-3918.patch, 
> LUCENE-3918.patch, LUCENE-3918.patch, LUCENE-3918.patch, LUCENE-3918.patch
>
>
> LUCENE-2482 added an IndexSorter to 3.x, but we need to port this
> functionality to 4.0 apis.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-3918) Port index sorter to trunk APIs

2013-03-06 Thread Uwe Schindler (JIRA)


[ 
https://issues.apache.org/jira/browse/LUCENE-3918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13594779#comment-13594779
 ] 

Uwe Schindler commented on LUCENE-3918:
---

bq.  Since we get an AtomicReader, what will leaves() return for a SlowWrapper?

Itsself - because its atomic :-) With a slow wrapper you have no chance to get 
the underlying codec information. With atomic readers implemented by 
sementreader there might be solutions.

> Port index sorter to trunk APIs
> ---
>
> Key: LUCENE-3918
> URL: https://issues.apache.org/jira/browse/LUCENE-3918
> Project: Lucene - Core
>  Issue Type: Task
>  Components: modules/other
>Affects Versions: 4.0-ALPHA
>Reporter: Robert Muir
> Fix For: 4.2, 5.0
>
> Attachments: LUCENE-3918.patch, LUCENE-3918.patch, LUCENE-3918.patch, 
> LUCENE-3918.patch, LUCENE-3918.patch, LUCENE-3918.patch, LUCENE-3918.patch, 
> LUCENE-3918.patch, LUCENE-3918.patch, LUCENE-3918.patch, LUCENE-3918.patch
>
>
> LUCENE-2482 added an IndexSorter to 3.x, but we need to port this
> functionality to 4.0 apis.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-2894) Implement distributed pivot faceting

2013-03-06 Thread Shawn Heisey (JIRA)


[ 
https://issues.apache.org/jira/browse/SOLR-2894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13594772#comment-13594772
 ] 

Shawn Heisey commented on SOLR-2894:


Andrew, someone on IRC is trying to apply your patch, but we can't find a 
revision that it will apply to successfully, and there's no revision 
information in the patch.  Also, whatever diff utility you used has put 
backslashes into the paths and that has to be manually fixed before it'll find 
the files on Linux.  Can you update your source tree to the current version and 
use 'svn diff' (or 'git diff' if you've checked out with git) to create the 
patch and re-upload?


> Implement distributed pivot faceting
> 
>
> Key: SOLR-2894
> URL: https://issues.apache.org/jira/browse/SOLR-2894
> Project: Solr
>  Issue Type: Improvement
>Reporter: Erik Hatcher
> Fix For: 4.2, 5.0
>
> Attachments: SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, 
> SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, SOLR-2894.patch, 
> SOLR-2894-reworked.patch
>
>
> Following up on SOLR-792, pivot faceting currently only supports 
> undistributed mode.  Distributed pivot faceting needs to be implemented.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-3918) Port index sorter to trunk APIs


[ 
https://issues.apache.org/jira/browse/LUCENE-3918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13594765#comment-13594765
 ] 

Shai Erera commented on LUCENE-3918:


Good idea. Is there a quick way to identify such indexes? Basically I need to 
iterate over leaves() and check version/Codec of each? Since we get an 
AtomicReader, what will leaves() return for a SlowWrapper?

> Port index sorter to trunk APIs
> ---
>
> Key: LUCENE-3918
> URL: https://issues.apache.org/jira/browse/LUCENE-3918
> Project: Lucene - Core
>  Issue Type: Task
>  Components: modules/other
>Affects Versions: 4.0-ALPHA
>Reporter: Robert Muir
> Fix For: 4.2, 5.0
>
> Attachments: LUCENE-3918.patch, LUCENE-3918.patch, LUCENE-3918.patch, 
> LUCENE-3918.patch, LUCENE-3918.patch, LUCENE-3918.patch, LUCENE-3918.patch, 
> LUCENE-3918.patch, LUCENE-3918.patch, LUCENE-3918.patch, LUCENE-3918.patch
>
>
> LUCENE-2482 added an IndexSorter to 3.x, but we need to port this
> functionality to 4.0 apis.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-3918) Port index sorter to trunk APIs

2013-03-06 Thread Robert Muir (JIRA)


[ 
https://issues.apache.org/jira/browse/LUCENE-3918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13594758#comment-13594758
 ] 

Robert Muir commented on LUCENE-3918:
-

The current tests suppress a long list of codecs because some methods test full 
functionality including some newer index features.

I think if we are going to test this way, then sorting reader itself should 
throw unsupported operation exception (refuse to sort segments from 3.x, 4.0, 
4.1 codecs).

Because its really unsupported/untested. otherwise, we should pull these newer 
methods out.

> Port index sorter to trunk APIs
> ---
>
> Key: LUCENE-3918
> URL: https://issues.apache.org/jira/browse/LUCENE-3918
> Project: Lucene - Core
>  Issue Type: Task
>  Components: modules/other
>Affects Versions: 4.0-ALPHA
>Reporter: Robert Muir
> Fix For: 4.2, 5.0
>
> Attachments: LUCENE-3918.patch, LUCENE-3918.patch, LUCENE-3918.patch, 
> LUCENE-3918.patch, LUCENE-3918.patch, LUCENE-3918.patch, LUCENE-3918.patch, 
> LUCENE-3918.patch, LUCENE-3918.patch, LUCENE-3918.patch, LUCENE-3918.patch
>
>
> LUCENE-2482 added an IndexSorter to 3.x, but we need to port this
> functionality to 4.0 apis.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-2660) omitPositions improvements

2013-03-06 Thread Robert Muir (JIRA)


[ 
https://issues.apache.org/jira/browse/SOLR-2660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13594751#comment-13594751
 ] 

Robert Muir commented on SOLR-2660:
---

There is no exception to fix. I think people discussing in that thread have a 
misunderstanding of what this issue is about.
If you ask to omit positions, and then you ask for a phrase query, or configure 
a stupid query parser that generates them automatically, then you deserve an 
exception.



> omitPositions improvements
> --
>
> Key: SOLR-2660
> URL: https://issues.apache.org/jira/browse/SOLR-2660
> Project: Solr
>  Issue Type: Improvement
>Affects Versions: 3.3, 4.0-ALPHA
>Reporter: Robert Muir
>Priority: Minor
> Attachments: SOLR-2660.patch
>
>
> followup to LUCENE-2048:
> Adds factory methods getPhraseQuery/getMultiPhraseQuery to QP, this way you 
> can subclass it and customize behavior, particularly
> * by default, Solr throws exception here if the fieldtype omits positions: 
> rather than 3.x's silent failure of no results, and even for trunk its nicer 
> to fail during query parsing rather than waiting for lucene's failure during 
> execution.
> * adds phraseAsBoolean, which allows you to downgrade these 
> phrase/multiphrase queries to boolean queries: this is a nice option in 
> conjunction with our word n-gram filters (shingle/commongrams/etc)for a fast 
> "approximation", if your application can tolerate some false positives, e.g. 
> "foo bar" -> termQuery(foo_bar), "foo bar baz" -> BQ(foo_bar AND bar_baz)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (LUCENE-3918) Port index sorter to trunk APIs


 [ 
https://issues.apache.org/jira/browse/LUCENE-3918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shai Erera updated LUCENE-3918:
---

Attachment: LUCENE-3918.patch

Patch does:

* Get rid of SortDoc. Sorter is now abstract class with a helper {{int[] 
compute(int[] docs, T[] values)}} method which sorter implementations can call 
to implement oldToNew().
** Thanks Uwe for helping me sort out the generics signatures!!

* Folded all SortingXYZ classes as private inner classes of 
SortingAtomicReader. While it increases SortingAR's code size, I feel that's 
the right thing to do, as these classes have no business on their own (even as 
package-private). If in the future we'll want to open them up for extension, we 
can make them public or something.

* Add {{getWrapped}} to both SortingDocsEnum and Positions, and implemented 
reuse as suggested by Adrien.

> Port index sorter to trunk APIs
> ---
>
> Key: LUCENE-3918
> URL: https://issues.apache.org/jira/browse/LUCENE-3918
> Project: Lucene - Core
>  Issue Type: Task
>  Components: modules/other
>Affects Versions: 4.0-ALPHA
>Reporter: Robert Muir
> Fix For: 4.2, 5.0
>
> Attachments: LUCENE-3918.patch, LUCENE-3918.patch, LUCENE-3918.patch, 
> LUCENE-3918.patch, LUCENE-3918.patch, LUCENE-3918.patch, LUCENE-3918.patch, 
> LUCENE-3918.patch, LUCENE-3918.patch, LUCENE-3918.patch, LUCENE-3918.patch
>
>
> LUCENE-2482 added an IndexSorter to 3.x, but we need to port this
> functionality to 4.0 apis.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-4470) Support for basic http auth in internal solr requests


[ 
https://issues.apache.org/jira/browse/SOLR-4470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13594699#comment-13594699
 ] 

Jan Høydahl commented on SOLR-4470:
---

So a first suggestion for one thing you could start with is factoring out the 
fix for the PeerSync.close issue into an own patch and get that committed, 
since it is a side-issue.

> Support for basic http auth in internal solr requests
> -
>
> Key: SOLR-4470
> URL: https://issues.apache.org/jira/browse/SOLR-4470
> Project: Solr
>  Issue Type: Bug
>  Components: clients - java, multicore, replication (java), SolrCloud
>Affects Versions: 4.0
>Reporter: Per Steffensen
>  Labels: authentication, solrclient, solrcloud
> Fix For: 4.2
>
> Attachments: SOLR-4470_branch_4x_r1452629.patch, 
> SOLR-4470_branch_4x_r1452629.patch
>
>
> We want to protect any HTTP-resource (url). We want to require credentials no 
> matter what kind of HTTP-request you make to a Solr-node.
> It can faily easy be acheived as described on 
> http://wiki.apache.org/solr/SolrSecurity. This problem is that Solr-nodes 
> also make "internal" request to other Solr-nodes, and for it to work 
> credentials need to be provided here also.
> Ideally we would like to "forward" credentials from a particular request to 
> all the "internal" sub-requests it triggers. E.g. for search and update 
> request.
> But there are also "internal" requests
> * that only indirectly/asynchronously triggered from "outside" requests (e.g. 
> shard creation/deletion/etc based on calls to the "Collection API")
> * that do not in any way have relation to an "outside" "super"-request (e.g. 
> replica synching stuff)
> We would like to aim at a solution where "original" credentials are 
> "forwarded" when a request directly/synchronously trigger a subrequest, and 
> fallback to a configured "internal credentials" for the 
> asynchronous/non-rooted requests.
> In our solution we would aim at only supporting basic http auth, but we would 
> like to make a "framework" around it, so that not to much refactoring is 
> needed if you later want to make support for other kinds of auth (e.g. digest)
> We will work at a solution but create this JIRA issue early in order to get 
> input/comments from the community as early as possible.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-4470) Support for basic http auth in internal solr requests


[ 
https://issues.apache.org/jira/browse/SOLR-4470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13594697#comment-13594697
 ] 

Jan Høydahl commented on SOLR-4470:
---

bq. It is like you are not allowed to do natural refactoring because it make a 
patch bigger than the absolute minimum to introduce the new feature.
Please read the Do's and Don'ts in 
http://wiki.apache.org/solr/HowToContribute#Creating_the_patch_file
It's not about forbidding this or that, it's just that if a single patch 
touches too much, history shows that it is hard to maintain and relate to for 
all the different persons, both committers and contributors. So cohesion 
matters. There are many examples of patches in the past doing only refactoring 
or only formatting changes. Those are welcome too, but it tends to be more 
efficient to first check in an API change which is necessary for a new feature, 
and then check in the core parts of that feature, and later check in the rest.

Also, if a change is considered major or risky we have historically started in 
a branch, then when mature merged in to trunk, then let it bake with nightly 
builds and other feedback for a few months before back-porting to the current 
stable branch. Chances are we'd want something similar here too, and that's why 
I suggested a branch off of trunk to start with.

Rather than trying to book a certain committer up-front to promise to work on 
the code you should incrementally continue work on it yourself, listening to 
feedback, rework the patch/branch... It's more likely to attract a committer 
helping with the last mile of something which is obviously being in the makings 
and being improved for a long time, than up-front commitments.

My 2¢

> Support for basic http auth in internal solr requests
> -
>
> Key: SOLR-4470
> URL: https://issues.apache.org/jira/browse/SOLR-4470
> Project: Solr
>  Issue Type: Bug
>  Components: clients - java, multicore, replication (java), SolrCloud
>Affects Versions: 4.0
>Reporter: Per Steffensen
>  Labels: authentication, solrclient, solrcloud
> Fix For: 4.2
>
> Attachments: SOLR-4470_branch_4x_r1452629.patch, 
> SOLR-4470_branch_4x_r1452629.patch
>
>
> We want to protect any HTTP-resource (url). We want to require credentials no 
> matter what kind of HTTP-request you make to a Solr-node.
> It can faily easy be acheived as described on 
> http://wiki.apache.org/solr/SolrSecurity. This problem is that Solr-nodes 
> also make "internal" request to other Solr-nodes, and for it to work 
> credentials need to be provided here also.
> Ideally we would like to "forward" credentials from a particular request to 
> all the "internal" sub-requests it triggers. E.g. for search and update 
> request.
> But there are also "internal" requests
> * that only indirectly/asynchronously triggered from "outside" requests (e.g. 
> shard creation/deletion/etc based on calls to the "Collection API")
> * that do not in any way have relation to an "outside" "super"-request (e.g. 
> replica synching stuff)
> We would like to aim at a solution where "original" credentials are 
> "forwarded" when a request directly/synchronously trigger a subrequest, and 
> fallback to a configured "internal credentials" for the 
> asynchronous/non-rooted requests.
> In our solution we would aim at only supporting basic http auth, but we would 
> like to make a "framework" around it, so that not to much refactoring is 
> needed if you later want to make support for other kinds of auth (e.g. digest)
> We will work at a solution but create this JIRA issue early in order to get 
> input/comments from the community as early as possible.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-4470) Support for basic http auth in internal solr requests


[ 
https://issues.apache.org/jira/browse/SOLR-4470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13594687#comment-13594687
 ] 

Jan Høydahl commented on SOLR-4470:
---

Please use JIRA to discuss the implementation.

I'm tired of long off-topic discussions here in JIRA, if you need to ventilate, 
please start a new thread on the dev-list.

> Support for basic http auth in internal solr requests
> -
>
> Key: SOLR-4470
> URL: https://issues.apache.org/jira/browse/SOLR-4470
> Project: Solr
>  Issue Type: Bug
>  Components: clients - java, multicore, replication (java), SolrCloud
>Affects Versions: 4.0
>Reporter: Per Steffensen
>  Labels: authentication, solrclient, solrcloud
> Fix For: 4.2
>
> Attachments: SOLR-4470_branch_4x_r1452629.patch, 
> SOLR-4470_branch_4x_r1452629.patch
>
>
> We want to protect any HTTP-resource (url). We want to require credentials no 
> matter what kind of HTTP-request you make to a Solr-node.
> It can faily easy be acheived as described on 
> http://wiki.apache.org/solr/SolrSecurity. This problem is that Solr-nodes 
> also make "internal" request to other Solr-nodes, and for it to work 
> credentials need to be provided here also.
> Ideally we would like to "forward" credentials from a particular request to 
> all the "internal" sub-requests it triggers. E.g. for search and update 
> request.
> But there are also "internal" requests
> * that only indirectly/asynchronously triggered from "outside" requests (e.g. 
> shard creation/deletion/etc based on calls to the "Collection API")
> * that do not in any way have relation to an "outside" "super"-request (e.g. 
> replica synching stuff)
> We would like to aim at a solution where "original" credentials are 
> "forwarded" when a request directly/synchronously trigger a subrequest, and 
> fallback to a configured "internal credentials" for the 
> asynchronous/non-rooted requests.
> In our solution we would aim at only supporting basic http auth, but we would 
> like to make a "framework" around it, so that not to much refactoring is 
> needed if you later want to make support for other kinds of auth (e.g. digest)
> We will work at a solution but create this JIRA issue early in order to get 
> input/comments from the community as early as possible.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-3918) Port index sorter to trunk APIs


[ 
https://issues.apache.org/jira/browse/LUCENE-3918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13594675#comment-13594675
 ] 

Shai Erera commented on LUCENE-3918:


bq. or make Sorter an abstract class (parameterized with T exnteds 
Comparable which does the sorting for you?

Hmm, Sorter should not be parameterized since it only returns an old2new[] 
mapping. The interface doesn't care about the values at all.

> Port index sorter to trunk APIs
> ---
>
> Key: LUCENE-3918
> URL: https://issues.apache.org/jira/browse/LUCENE-3918
> Project: Lucene - Core
>  Issue Type: Task
>  Components: modules/other
>Affects Versions: 4.0-ALPHA
>Reporter: Robert Muir
> Fix For: 4.2, 5.0
>
> Attachments: LUCENE-3918.patch, LUCENE-3918.patch, LUCENE-3918.patch, 
> LUCENE-3918.patch, LUCENE-3918.patch, LUCENE-3918.patch, LUCENE-3918.patch, 
> LUCENE-3918.patch, LUCENE-3918.patch, LUCENE-3918.patch
>
>
> LUCENE-2482 added an IndexSorter to 3.x, but we need to port this
> functionality to 4.0 apis.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-3918) Port index sorter to trunk APIs


[ 
https://issues.apache.org/jira/browse/LUCENE-3918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13594674#comment-13594674
 ] 

Shai Erera commented on LUCENE-3918:


bq. Are there actual use-cases for sorting by stored fields or payloads? If not 
I think we should remove StoredFieldsSorter and PayloadSorter?

The original 3x implementation allowed sorting only by stored fields, so we 
thought if it was useful to someone then, it might be useful now. And sort by 
payload value was useful in the past, before DV days, and e.g. we still have 
code that stores sort-by values in payloads. I don't think it's harmful to keep 
PayloadSorter, but I don't mind removing it as well.

bq. Remove IndexSorter.java

IndexSorter is a convenient utility for sorting a Directory end-to-end. Why 
remove it? On the opposite, I was thinking if we want to make it more of an 
instance utility, i.e. that you do something like {{new IndexSorter().sort()}} 
and perhaps allow to sort a Directory in-place (by opening an IW, deleteAll() 
and addIndexes SortingReader...).

bq. make SortDoc package-private

SortDoc, while not visible on the API today, might be useful for Sorter 
implementations, as a convenient way to create the old2new[]. Otherwise, if you 
need to sort both 'docs' and 'values', you need to work with something similar 
to SorterTemplate. I can remove it and implement sorting docs[] and values[] by 
each Sorter impl, or make Sorter an abstract class (parameterized with {{T 
exnteds Comparable}} which does the sorting for you?

bq. Maybe we could check if the docs enum to reuse is an instance of 
SortingDocsEnum and reuse its wrapped DocEnum?

Yeah, I've been thinking about it while writing the comment. Since 
FilterDocsEnum doesn't expose 'in' in a public way, I was wondering if I should 
expose the wrapped 'in' in SortingXYZ just for that purpose. How important is 
it to reuse DocsEnum/AndPositions?

> Port index sorter to trunk APIs
> ---
>
> Key: LUCENE-3918
> URL: https://issues.apache.org/jira/browse/LUCENE-3918
> Project: Lucene - Core
>  Issue Type: Task
>  Components: modules/other
>Affects Versions: 4.0-ALPHA
>Reporter: Robert Muir
> Fix For: 4.2, 5.0
>
> Attachments: LUCENE-3918.patch, LUCENE-3918.patch, LUCENE-3918.patch, 
> LUCENE-3918.patch, LUCENE-3918.patch, LUCENE-3918.patch, LUCENE-3918.patch, 
> LUCENE-3918.patch, LUCENE-3918.patch, LUCENE-3918.patch
>
>
> LUCENE-2482 added an IndexSorter to 3.x, but we need to port this
> functionality to 4.0 apis.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

FW: JVM hanging when using G1GC on JDK8 b78 or b79 (Linux 32 bit)

They already understood the G1GC problem with JDK 8 b78/b79 and working on a 
fix. This was really fast:
http://mail.openjdk.java.net/pipermail/hotspot-gc-dev/2013-March/006128.html

Uwe

-
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: u...@thetaphi.de


--- Begin Message ---


Hi all,

I sent this email earlier, but I did "reply list" instead of "reply 
all". Sorry about that.


The hang is due to the fact that we are using single threaded reference 
processing but end up in the multi threaded code path and get stuck in a 
loop that waits for the other processing threads to terminate.


John Cuthbertson is working on a fix for this. I think we have all the 
information we need to solve this.


Bengt

On 3/6/13 9:04 AM, Bengt Rutisson wrote:


David,

I think this is a VM bug and the thread dumps that Uwe produced are 
enough to start tracking down the root cause.


On 3/6/13 8:52 AM, David Holmes wrote:
If the VM is completely unresponsive then it suggests we are at a 
safepoint.

Yes, we are hanging during a stop-the-world GC, so we are at a safepoint.



The GC threads are not "hung" in os::parK, they are parked - waiting 
to be notified of something.


It looks like the reference processing thread is stuck in a loop where 
it does wait(). So, the VM is hanging even if that stack trace also 
ends up in os::park().




The thing is to find out why they are not being woken up.


Actually, in this case we should probably not even be calling wait...



Can the gdb log be posted somewhere? I don't know if the attachment 
made it to the original posting on hotspot-gc but it's no longer 
available on hotspot-dev.


I received the attachment with the original email. I've attached it to 
the bug report that I created: 8009536. You can find it there if you 
want to. But I think we have a fairly good idea of what change caused 
the hang.


Bengt



Thanks,
David

On 6/03/2013 4:07 PM, Krystal Mok wrote:

Hi Uwe,

If you can attach gdb onto it, and jstack -m and jstack -F should also
work; that'll get you the Java stack trace.
(But it probably doesn't matter in this case, because the hang is
probably bug in the VM).

- Kris

On Wed, Mar 6, 2013 at 5:48 AM, Uwe Schindler 
 wrote:

Hi,

since a few month we are extensively testing various preview builds 
of JDK 8 for compatibility with Apache Lucene and Solr, so we can 
find any bugs early and prevent the problems we had with the 
release of Java 7 two years ago. Currently we have a Linux (Ubuntu 
64bit) Jenkins machine that has various JDKs (JDK 6, JDK 7, JDK 8 
snapshot, IBM J9, older JRockit) installed, choosing a different 
one with different hotspot and garbage collector settings on every 
run of the test suite (which takes approx. 30-45 minutes).


JDK 8 b79 works so far very well on Linux, we found some strange 
behavior in early versions (maybe compiler errors), but no longer 
at the moment. There is one configuration that constantly and 
reproducibly hangs in one module that is tested: The configuration 
uses JDK 8 b79 (same for b78), 32 bit, and G1GC (server or client 
does not matter). The JVM running the tests hangs irresponsible 
(jstack or kill -3 have no effect/cannot connect, standard kill 
does not stop it, only kill -9 actually kills it). It can be 
reproduced in this Lucene module 100% (it hangs always).


I was able to connect with GDB to the JVM and get a stack trace on 
all threads (see attachment, dump.txt). As you see all threads of 
G1GC seem to hang in a syscall (os:park(), a conditional wait in 
pthread library). Unfortunately that’s all I can give you. A Java 
stacktrace is not possible because the JVM reacts on neither kill 
-3 nor jstack. With all other garbage collectors it passes the test 
without hangs in a few seconds, with 32 bit G1GC it can stand still 
for hours. The 64 bit JVM passes with G1GC, so only the 32 bit 
variant is affected. Client or Server VM makes no difference.


To reproduce:
- Use a 32 bit JDK 8 b78 or b79 (tested on Linux 64 bit, but this 
should not matter)
- Download Lucene Source code (e.g. the snapshot version we were 
testing with: 


https://builds.apache.org/job/Lucene-Artifacts-trunk/2212/artifact/lucene/di
st/)

- change to directory lucene/analysis/uima and run:
 ant -Dargs="-server -XX:+UseG1GC" -Dtests.multiplier=3 
-Dtests.jvms=1 test
After a while the test framework prints "stalled" messages (because 
the child VM actually running the test no longer responds). The PID 
is also printed. Try to get a stack trace or kill it, no response. 
Only kill -9 helps. Choosing another garbage collector in the above 
command line makes the test finish after a few seconds, e.g. 
-Dargs="-server -XX:+UseConcMarkSweepGC"


I posted this bug report directly to the mailing list, because with 
earlier bug reports, there seem to be a problem with bugs.sun.com - 
there is no response from any reviewer after several weeks and we 
were able to help to find and fix javadoc and

[jira] [Commented] (LUCENE-3918) Port index sorter to trunk APIs


[ 
https://issues.apache.org/jira/browse/LUCENE-3918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13594650#comment-13594650
 ] 

Adrien Grand commented on LUCENE-3918:
--

Thanks for your work Shai. Indeed it looks really good now! Here a a few 
suggestions/questions:
 - Are there actual use-cases for sorting by stored fields or payloads? If not 
I think we should remove StoredFieldsSorter and PayloadSorter?
 - Remove IndexSorter.java and make SortDoc package-private?

{code}
+  // we cannot reuse the given DocsAndPositionsEnum because we return our
+  // own wrapper, and not all Codecs like it.
{code}

Maybe we could check if the docs enum to reuse is an instance of 
SortingDocsEnum and reuse its wrapped DocEnum?



> Port index sorter to trunk APIs
> ---
>
> Key: LUCENE-3918
> URL: https://issues.apache.org/jira/browse/LUCENE-3918
> Project: Lucene - Core
>  Issue Type: Task
>  Components: modules/other
>Affects Versions: 4.0-ALPHA
>Reporter: Robert Muir
> Fix For: 4.2, 5.0
>
> Attachments: LUCENE-3918.patch, LUCENE-3918.patch, LUCENE-3918.patch, 
> LUCENE-3918.patch, LUCENE-3918.patch, LUCENE-3918.patch, LUCENE-3918.patch, 
> LUCENE-3918.patch, LUCENE-3918.patch, LUCENE-3918.patch
>
>
> LUCENE-2482 added an IndexSorter to 3.x, but we need to port this
> functionality to 4.0 apis.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Comment Edited] (SOLR-4470) Support for basic http auth in internal solr requests


[ 
https://issues.apache.org/jira/browse/SOLR-4470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13593830#comment-13593830
 ] 

Per Steffensen edited comment on SOLR-4470 at 3/6/13 12:09 PM:
---

About SOLR-4470_branch_4x_r1452629.patch
* Fits on top of branch_4x revision 1452629
* Fairly big patch but most of it is to provide credentials across all requests 
issued by BaseDistributedSearchTest tests or to just to forward decisions on 
credentials down the call-flow on server-side. Notes on the major changes in 
non-test code
** AuthCredentials: New class encapsulating the concept of credentials. 
Supports "http basic" only for now, but using this object across code wherever 
credentials are needed, the framework will be in place for supporting other 
kinds of credentials in the future
** InterSolrNodeAuthCredentialsFactory: Separation of concern with respect to 
"getting credentials for inter-solr-node requests". The implementation of the 
factory can be replaced to "change strategy", but this is only used in tests 
(for now)
** InterSolrNodeAuthCredentialsFactory.AuthCredentialsSource: A level of 
abstraction to be used server-side whenever you have to decide on "how to get 
credentials for sub-requests" - possible decisions are "use credentials from 
super-request" or "use internal credentials" but NOT "use no credentials". Any 
server-side code should be able to provide credentials in requests, because you 
never know if the solr-cluster has been set up with protection on the URL you 
are about to hit. Therefore there are basically only the two choices listed 
above. The credentials (internal or on super-request) might be empty/null in 
concrete cases if no internal credentials has been specified or of no 
credentials was provided in the super-request, but you as a programmer never 
know, so you have to decide on "how to get credentials for sub-requests" for 
your server-side case. Idea is to "use credentials from super-request" whenever 
such exists, so that a request from "the outside" can never trigger a 
sub-request that the outside user was not allowed to do directly by himself. 
Whenever there is no "super-request" use "internal credentials". There are 
gray-area-cases like e.g. the Observer that asynchronously issues requests to 
CoreAdmin API when the Collection API has been called "from the outside" - to 
be true to the idea, in this case, we ought to actually use the credentials 
provided in the original request to Collection API, but that would require 
writing those credentials into the ZK-queue, and I wouldn't include that now, 
so in this case we use internal credentials. Server-side methods should 
typically not take AuthCredentials-parameters, because that will not 
encourage/remind developers to decide on the strategy when they use the method 
in the future - use AuthCredentialsSource instead.
** SolrRequest: Has been modified so that it can hold credentials to be used, 
and info on whether or not to provide them preemptively
** SolrQueryRequest: Has been modified to hold the credentials used in the 
request received
** SolrRequestParsers: Has been modified to extract the credentials from the 
request and add it to SolrQueryRequest
** HttpClientUtil: Has been changed to allow for specifying credentials and/or 
ClientConnectionManager when creating a HttpClient. Credentials to be used 
across all requests issued by this HttpClient. ClientConnectionManager to be 
able to share connection-manager across HttpClients - see reason below
** HttpShardHandler(Factory): It is used many places (and thats good) where 
sub-request are issued, places doing "use internal credentials" but also places 
"use credentials from super-request" (SolrCmdDistributor) and therefore 
HttpShardHandler(Factory) has been modified to support both.
** LBHttpSolrServer: Changed to allow for specific HttpClient for each request, 
so that even the loadbalancer in HttpShardHandler can use different HttpClients 
for different requests. It should have been done this way from the beginning - 
it is strange to use the defaultHttpClient for loadbalancing request through a 
HttpShardHandler which has been created with a specific HttpClient to use.
** Shared ClientConnectionManager instead of HttpClient: Several classes that 
used to have a HttpClient used for all requests (PeerSync, SyncStrategy and 
UpdateShardHandler), now have only a shared ClientConnetionManager. This in 
order to be able to specify different credentials (on HttpClient level) for 
different requests issued through those components. Actually it is not absolute 
necessary for PeerSync because it always uses "internal credentials", but it is 
done anyway in order to allow HttpShardHandlerFactory to insist on getting a 
decision on "how to get credentials for sub-requests" even when you "bring you 
own client" - shardhandlers ALWAYS need to have a strategy

[jira] [Commented] (LUCENE-4752) Merge segments to sort them


[ 
https://issues.apache.org/jira/browse/LUCENE-4752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13594638#comment-13594638
 ] 

Adrien Grand commented on LUCENE-4752:
--

bq.  I took a look at MP API, and maybe if we change OneMerge from holding a 
List to List, we could write an MP which sorts 
segments together by opening a SortingAtomicReader over the segments that were 
picked for merge?

Making the sorting stuff part of MergePolicy makes sense. However, I think that 
the (package-private) List in MergePolicy is only used to track 
the list of segment readers being used while merging (this reference is only 
used in IndexWriter). What MP actually manipulates is a list of 
SegmentInfoPerCommit, it is possible that no reader is open for a segment when 
MergePolicy picks it, and Lucene should not force a reader to be open until the 
merge actually starts. So maybe we should have an additional method in 
MergePolicy (or OneMerge for finer-grained control?) to tell IndexWriter how to 
view a list of segment readers? (either sequentially as today or a sorted view 
as suggested in this issue description).




> Merge segments to sort them
> ---
>
> Key: LUCENE-4752
> URL: https://issues.apache.org/jira/browse/LUCENE-4752
> Project: Lucene - Core
>  Issue Type: New Feature
>  Components: core/index
>Reporter: David Smiley
>Assignee: Adrien Grand
>
> It would be awesome if Lucene could write the documents out in a segment 
> based on a configurable order.  This of course applies to merging segments 
> to. The benefit is increased locality on disk of documents that are likely to 
> be accessed together.  This often applies to documents near each other in 
> time, but also spatially.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Comment Edited] (SOLR-4470) Support for basic http auth in internal solr requests


[ 
https://issues.apache.org/jira/browse/SOLR-4470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13594627#comment-13594627
 ] 

Per Steffensen edited comment on SOLR-4470 at 3/6/13 11:39 AM:
---

New patch. Passes precommit if you do the following
{code}
cd 
patch -p0 -i SOLR-4470_branch_4x_r1452629.patch
svn stat | grep "^?" | awk '{print $2}' | xargs svn add
svn propset svn:eol-style native 
solr/core/src/java/org/apache/solr/security/InterSolrNodeAuthCredentialsFactory.java
svn propset svn:eol-style native 
solr/core/src/java/org/apache/solr/servlet/security/RegExpAuthorizationFilter.java
svn propset svn:eol-style native 
solr/core/src/test/org/apache/solr/cloud/SecurityDistributedTest.java
svn propset svn:eol-style native 
solr/solrj/src/java/org/apache/solr/security/AuthCredentials.java
svn propset svn:eol-style native 
solr/test-framework/src/java/org/apache/solr/cloud/InterSolrNodeAuthCredentialsFactoryTestingHelper.java
ant precommit
{code}

Only problem is that it says "Linting documentation HTML is not supported on 
this Java version (1.6) / JVM (Java HotSpot(TM) 64-Bit Server VM).", but I 
guess that is because I am not "is.jenkins.build", and I do not know what to do 
about that on my local machine. If I comment out the stuff in target 
"-documentation-lint-unsupported" in lucene/common-build.xml it passes "ant 
precommit"

Besides that, the new patch contains a small modification to 
RegExpAuthorizationFilter so that it works in both a "real" jetty and in a 
"test" jetty (started by JettySolrRunning). Strange but 
HttpServletRequest.getPathInfo() returns the correct path in "test" jetty, but 
returns null in "real" jetty. And HttpServletRequest.getServletPath() returns 
correct path in "real" jetty, but returns empty/null in "test" jetty - so now 
we use whatever is non-null/empty.


  was (Author: steff1193):
New patch. Passes precommit if you do the following
{code}
cd 
patch -p0 -i SOLR-4470_branch_4x_r1452629.patch
svn stat | grep "^?" | awk '{print $2}' | xargs svn add
svn propset svn:eol-style native 
solr/core/src/java/org/apache/solr/security/InterSolrNodeAuthCredentialsFactory.java
svn propset svn:eol-style native 
solr/core/src/java/org/apache/solr/servlet/security/RegExpAuthorizationFilter.java
svn propset svn:eol-style native 
solr/core/src/test/org/apache/solr/cloud/SecurityDistributedTest.java
svn propset svn:eol-style native 
solr/solrj/src/java/org/apache/solr/security/AuthCredentials.java
svn propset svn:eol-style native 
solr/test-framework/src/java/org/apache/solr/cloud/InterSolrNodeAuthCredentialsFactoryTestingHelper.java
ant precommit
{code}

Only problem is that it says "Linting documentation HTML is not supported on 
this Java version (1.6) / JVM (Java HotSpot(TM) 64-Bit Server VM).", but I 
guess that is because I am not "is.jenkins.build", and I do not know what to do 
about that on my local machine. If I comment out the stuff in target 
"-documentation-lint-unsupported" in lucene/common-build.xml it passes "ant 
precommit"

  
> Support for basic http auth in internal solr requests
> -
>
> Key: SOLR-4470
> URL: https://issues.apache.org/jira/browse/SOLR-4470
> Project: Solr
>  Issue Type: Bug
>  Components: clients - java, multicore, replication (java), SolrCloud
>Affects Versions: 4.0
>Reporter: Per Steffensen
>  Labels: authentication, solrclient, solrcloud
> Fix For: 4.2
>
> Attachments: SOLR-4470_branch_4x_r1452629.patch, 
> SOLR-4470_branch_4x_r1452629.patch
>
>
> We want to protect any HTTP-resource (url). We want to require credentials no 
> matter what kind of HTTP-request you make to a Solr-node.
> It can faily easy be acheived as described on 
> http://wiki.apache.org/solr/SolrSecurity. This problem is that Solr-nodes 
> also make "internal" request to other Solr-nodes, and for it to work 
> credentials need to be provided here also.
> Ideally we would like to "forward" credentials from a particular request to 
> all the "internal" sub-requests it triggers. E.g. for search and update 
> request.
> But there are also "internal" requests
> * that only indirectly/asynchronously triggered from "outside" requests (e.g. 
> shard creation/deletion/etc based on calls to the "Collection API")
> * that do not in any way have relation to an "outside" "super"-request (e.g. 
> replica synching stuff)
> We would like to aim at a solution where "original" credentials are 
> "forwarded" when a request directly/synchronously trigger a subrequest, and 
> fallback to a configured "internal credentials" for the 
> asynchronous/non-rooted requests.
> In our solution we would aim at only supporting basic http auth, but we would 
> like to make a "framework" around it, so that not

[jira] [Updated] (SOLR-4470) Support for basic http auth in internal solr requests


 [ 
https://issues.apache.org/jira/browse/SOLR-4470?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Per Steffensen updated SOLR-4470:
-

Attachment: SOLR-4470_branch_4x_r1452629.patch

New patch. Passes precommit if you do the following
{code}
cd 
patch -p0 -i SOLR-4470_branch_4x_r1452629.patch
svn stat | grep "^?" | awk '{print $2}' | xargs svn add
svn propset svn:eol-style native 
solr/core/src/java/org/apache/solr/security/InterSolrNodeAuthCredentialsFactory.java
svn propset svn:eol-style native 
solr/core/src/java/org/apache/solr/servlet/security/RegExpAuthorizationFilter.java
svn propset svn:eol-style native 
solr/core/src/test/org/apache/solr/cloud/SecurityDistributedTest.java
svn propset svn:eol-style native 
solr/solrj/src/java/org/apache/solr/security/AuthCredentials.java
svn propset svn:eol-style native 
solr/test-framework/src/java/org/apache/solr/cloud/InterSolrNodeAuthCredentialsFactoryTestingHelper.java
ant precommit
{code}

Only problem is that it says "Linting documentation HTML is not supported on 
this Java version (1.6) / JVM (Java HotSpot(TM) 64-Bit Server VM).", but I 
guess that is because I am not "is.jenkins.build", and I do not know what to do 
about that on my local machine. If I comment out the stuff in target 
"-documentation-lint-unsupported" in lucene/common-build.xml it passes "ant 
precommit"


> Support for basic http auth in internal solr requests
> -
>
> Key: SOLR-4470
> URL: https://issues.apache.org/jira/browse/SOLR-4470
> Project: Solr
>  Issue Type: Bug
>  Components: clients - java, multicore, replication (java), SolrCloud
>Affects Versions: 4.0
>Reporter: Per Steffensen
>  Labels: authentication, solrclient, solrcloud
> Fix For: 4.2
>
> Attachments: SOLR-4470_branch_4x_r1452629.patch, 
> SOLR-4470_branch_4x_r1452629.patch
>
>
> We want to protect any HTTP-resource (url). We want to require credentials no 
> matter what kind of HTTP-request you make to a Solr-node.
> It can faily easy be acheived as described on 
> http://wiki.apache.org/solr/SolrSecurity. This problem is that Solr-nodes 
> also make "internal" request to other Solr-nodes, and for it to work 
> credentials need to be provided here also.
> Ideally we would like to "forward" credentials from a particular request to 
> all the "internal" sub-requests it triggers. E.g. for search and update 
> request.
> But there are also "internal" requests
> * that only indirectly/asynchronously triggered from "outside" requests (e.g. 
> shard creation/deletion/etc based on calls to the "Collection API")
> * that do not in any way have relation to an "outside" "super"-request (e.g. 
> replica synching stuff)
> We would like to aim at a solution where "original" credentials are 
> "forwarded" when a request directly/synchronously trigger a subrequest, and 
> fallback to a configured "internal credentials" for the 
> asynchronous/non-rooted requests.
> In our solution we would aim at only supporting basic http auth, but we would 
> like to make a "framework" around it, so that not to much refactoring is 
> needed if you later want to make support for other kinds of auth (e.g. digest)
> We will work at a solution but create this JIRA issue early in order to get 
> input/comments from the community as early as possible.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: JVM hanging when using G1GC on JDK8 b78 or b79 (Linux 32 bit)

2013-03-06 Thread Thomas Schatzl

Hi,

On Wed, 2013-03-06 at 20:23 +1000, David Holmes wrote:
> On 6/03/2013 5:55 PM, Dawid Weiss wrote:
> >
> > Here you go:
> > http://pastebin.com/raw.php?i=b2PHLm1e
> 
> Thanks. I would have to say this seems to be the suspicious part:
> 
> Thread 22 (Thread 0xf20ffb40 (LWP 22939)):
[...]
> from 
> /var/lib/jenkins/tools/java/32bit/jdk1.8.0-ea-b79/jre/lib/i386/server/libjvm.so
> #6  0xf6b5ea41 in ConcurrentG1RefineThread::run_young_rs_sampling() ()
> from 
> The suspendible thread set logic looks 'tricky". Time for the G1 experts 
> to take over. :)

The young gen rs sampling thread is a thread that does some statistical
updates while the application is running. So that in the STW pause not
so much work needs to be done.

At a safepoint it is always suspended, this is normal.

As Bengt mentioned, the problem seems to be thread 10, which is the VM
thread (the one responsible for bringing everything to a safepoint and
then distributing work).

According to the stack trace, this thread seems to be waiting for
synchronization with the marking threads because of a mark stack
overflow during weak reference processing.

However all marking threads are already waiting due to the safepointing
operation, and so it waits endlessly.

As Bengt mentioned, this thread shouldn't be waiting, and shouldn't need
to because it seems to be the only thread working on weak references
anyway (i.e. this phase is single threaded).

(All imo)

Thomas

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

RE: JVM hanging when using G1GC on JDK8 b78 or b79 (Linux 32 bit)

Hi,
 
> I think this is a VM bug and the thread dumps that Uwe produced are enough
> to start tracking down the root cause.

I hope it is enough! If I can help with more details, tell me what I should do 
to track this down. Unfortunately, we have no isolated test case (like a small 
java class that triggers this bug) - you have to run the test cases of this 
Lucene's module. It only happens there, not in any other Lucene test suite. It 
may be caused by a lot of GC activity in this "UIMA" module or a specific test.

> On 3/6/13 8:52 AM, David Holmes wrote:
> > If the VM is completely unresponsive then it suggests we are at a
> > safepoint.
> Yes, we are hanging during a stop-the-world GC, so we are at a safepoint.
> 
> >
> > The GC threads are not "hung" in os::parK, they are parked - waiting
> > to be notified of something.
> 
> It looks like the reference processing thread is stuck in a loop where it does
> wait(). So, the VM is hanging even if that stack trace also ends up in
> os::park().
> 
> >
> > The thing is to find out why they are not being woken up.
> 
> Actually, in this case we should probably not even be calling wait...
> 
> >
> > Can the gdb log be posted somewhere? I don't know if the attachment
> > made it to the original posting on hotspot-gc but it's no longer
> > available on hotspot-dev.
> 
> I received the attachment with the original email. I've attached it to
> the bug report that I created: 8009536. You can find it there if you
> want to. But I think we have a fairly good idea of what change caused
> the hang.

If it helps: Unfortunately, we had some problems with recent JDK builds, 
because javac and javadoc tools were not working correctly, failing to build 
our source code. Since b78 this was fixed. Until this was fixed, we used build 
b65 (which was the last one working) and the G1GC hangs did not appear on this 
version. So it must have happened by a change after b65 till b78.

Uwe

> Bengt
> 
> >
> > Thanks,
> > David
> >
> > On 6/03/2013 4:07 PM, Krystal Mok wrote:
> >> Hi Uwe,
> >>
> >> If you can attach gdb onto it, and jstack -m and jstack -F should also
> >> work; that'll get you the Java stack trace.
> >> (But it probably doesn't matter in this case, because the hang is
> >> probably bug in the VM).
> >>
> >> - Kris
> >>
> >> On Wed, Mar 6, 2013 at 5:48 AM, Uwe Schindler
> 
> >> wrote:
> >>> Hi,
> >>>
> >>> since a few month we are extensively testing various preview builds
> >>> of JDK 8 for compatibility with Apache Lucene and Solr, so we can
> >>> find any bugs early and prevent the problems we had with the release
> >>> of Java 7 two years ago. Currently we have a Linux (Ubuntu 64bit)
> >>> Jenkins machine that has various JDKs (JDK 6, JDK 7, JDK 8 snapshot,
> >>> IBM J9, older JRockit) installed, choosing a different one with
> >>> different hotspot and garbage collector settings on every run of the
> >>> test suite (which takes approx. 30-45 minutes).
> >>>
> >>> JDK 8 b79 works so far very well on Linux, we found some strange
> >>> behavior in early versions (maybe compiler errors), but no longer at
> >>> the moment. There is one configuration that constantly and
> >>> reproducibly hangs in one module that is tested: The configuration
> >>> uses JDK 8 b79 (same for b78), 32 bit, and G1GC (server or client
> >>> does not matter). The JVM running the tests hangs irresponsible
> >>> (jstack or kill -3 have no effect/cannot connect, standard kill does
> >>> not stop it, only kill -9 actually kills it). It can be reproduced
> >>> in this Lucene module 100% (it hangs always).
> >>>
> >>> I was able to connect with GDB to the JVM and get a stack trace on
> >>> all threads (see attachment, dump.txt). As you see all threads of
> >>> G1GC seem to hang in a syscall (os:park(), a conditional wait in
> >>> pthread library). Unfortunately that’s all I can give you. A Java
> >>> stacktrace is not possible because the JVM reacts on neither kill -3
> >>> nor jstack. With all other garbage collectors it passes the test
> >>> without hangs in a few seconds, with 32 bit G1GC it can stand still
> >>> for hours. The 64 bit JVM passes with G1GC, so only the 32 bit
> >>> variant is affected. Client or Server VM makes no difference.
> >>>
> >>> To reproduce:
> >>> - Use a 32 bit JDK 8 b78 or b79 (tested on Linux 64 bit, but this
> >>> should not matter)
> >>> - Download Lucene Source code (e.g. the snapshot version we were
> >>> testing with:
> >>> https://builds.apache.org/job/Lucene-Artifacts-
> trunk/2212/artifact/lucene/dist/)
> >>> - change to directory lucene/analysis/uima and run:
> >>>  ant -Dargs="-server -XX:+UseG1GC" -Dtests.multiplier=3
> >>> -Dtests.jvms=1 test
> >>> After a while the test framework prints "stalled" messages (because
> >>> the child VM actually running the test no longer responds). The PID
> >>> is also printed. Try to get a stack trace or kill it, no response.
> >>> Only kill -9 helps. Choosing another garbage collector in the above
> >>> command l

[jira] [Commented] (LUCENE-4752) Merge segments to sort them


[ 
https://issues.apache.org/jira/browse/LUCENE-4752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13594614#comment-13594614
 ] 

Shai Erera commented on LUCENE-4752:


When I thought about AtomicReaderFactory I thought about something that returns 
a single AtomicReader. IW-wise, that makes more sense (as an API) than 
something that reorders a list of segments. Reordering can be left to things 
like MergePolicy. In fact, I took a look at MP API, and maybe if we change 
OneMerge from holding a {{List}} to {{List}}, we 
could write an MP which sorts segments together by opening a 
SortingAtomicReader over the segments that were picked for merge?

I think, if we do that, we don't need any hook on IWC 
({{AtomicReaderFactory}}), although that might be useful in other cases, but 
outside the scope of this issue. All we'll need is a SortingMP?

> Merge segments to sort them
> ---
>
> Key: LUCENE-4752
> URL: https://issues.apache.org/jira/browse/LUCENE-4752
> Project: Lucene - Core
>  Issue Type: New Feature
>  Components: core/index
>Reporter: David Smiley
>Assignee: Adrien Grand
>
> It would be awesome if Lucene could write the documents out in a segment 
> based on a configurable order.  This of course applies to merging segments 
> to. The benefit is increased locality on disk of documents that are likely to 
> be accessed together.  This often applies to documents near each other in 
> time, but also spatially.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-4533) Synonyms, created in custom filters are ignored after tokenizers.

2013-03-06 Thread Artem Lukanin (JIRA)


 [ 
https://issues.apache.org/jira/browse/SOLR-4533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Artem Lukanin updated SOLR-4533:


Attachment: synonyms.patch

Fixed

> Synonyms, created in custom filters are ignored after tokenizers.
> -
>
> Key: SOLR-4533
> URL: https://issues.apache.org/jira/browse/SOLR-4533
> Project: Solr
>  Issue Type: Bug
>Affects Versions: 4.1
>Reporter: Artem Lukanin
> Attachments: synonyms.patch
>
>
> If a synonym token is added in a custom filter (e.g. this one: 
> http://solr.pl/en/2013/02/04/developing-your-own-solr-filter-part-2/) and the 
> default operator is AND, the query becomes broken, because 2 tokens at the 
> same position becomes required, which is impossible. Solution: place all 
> synonyms in a separate clause and assign these tokens occur=SHOULD.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Created] (SOLR-4533) Synonyms, created in custom filters are ignored after tokenizers.

2013-03-06 Thread Artem Lukanin (JIRA)

Artem Lukanin created SOLR-4533:
---

 Summary: Synonyms, created in custom filters are ignored after 
tokenizers.
 Key: SOLR-4533
 URL: https://issues.apache.org/jira/browse/SOLR-4533
 Project: Solr
  Issue Type: Bug
Affects Versions: 4.1
Reporter: Artem Lukanin


If a synonym token is added in a custom filter (e.g. this one: 
http://solr.pl/en/2013/02/04/developing-your-own-solr-filter-part-2/) and the 
default operator is AND, the query becomes broken, because 2 tokens at the same 
position becomes required, which is impossible. Solution: place all synonyms in 
a separate clause and assign these tokens occur=SHOULD.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: JVM hanging when using G1GC on JDK8 b78 or b79 (Linux 32 bit)

2013-03-06 Thread David Holmes


On 6/03/2013 5:55 PM, Dawid Weiss wrote:


Here you go:
http://pastebin.com/raw.php?i=b2PHLm1e


Thanks. I would have to say this seems to be the suspicious part:

Thread 22 (Thread 0xf20ffb40 (LWP 22939)):
#0  0xf7743430 in __kernel_vsyscall ()
#1  0xf771e96b in pthread_cond_wait@@GLIBC_2.3.2 () from 
/lib/i386-linux-gnu/libpthread.so.0

#2  0xf6ec849c in os::PlatformEvent::park() ()
   from 
/var/lib/jenkins/tools/java/32bit/jdk1.8.0-ea-b79/jre/lib/i386/server/libjvm.so

#3  0xf6e98b82 in Monitor::IWait(Thread*, long long) ()
   from 
/var/lib/jenkins/tools/java/32bit/jdk1.8.0-ea-b79/jre/lib/i386/server/libjvm.so

#4  0xf6e99370 in Monitor::wait(bool, long, bool) ()
   from 
/var/lib/jenkins/tools/java/32bit/jdk1.8.0-ea-b79/jre/lib/i386/server/libjvm.so

#5  0xf6b5fb16 in SuspendibleThreadSet::join() ()
   from 
/var/lib/jenkins/tools/java/32bit/jdk1.8.0-ea-b79/jre/lib/i386/server/libjvm.so

#6  0xf6b5ea41 in ConcurrentG1RefineThread::run_young_rs_sampling() ()
   from 
/var/lib/jenkins/tools/java/32bit/jdk1.8.0-ea-b79/jre/lib/i386/server/libjvm.so

#7  0xf6b5ef91 in ConcurrentG1RefineThread::run() ()

The suspendible thread set logic looks 'tricky". Time for the G1 experts 
to take over. :)


David


Dawid

On Wed, Mar 6, 2013 at 8:52 AM, David Holmes mailto:david.hol...@oracle.com>> wrote:

If the VM is completely unresponsive then it suggests we are at a
safepoint.

The GC threads are not "hung" in os::parK, they are parked - waiting
to be notified of something.

The thing is to find out why they are not being woken up.

Can the gdb log be posted somewhere? I don't know if the attachment
made it to the original posting on hotspot-gc but it's no longer
available on hotspot-dev.

Thanks,
David


On 6/03/2013 4:07 PM, Krystal Mok wrote:

Hi Uwe,

If you can attach gdb onto it, and jstack -m and jstack -F
should also
work; that'll get you the Java stack trace.
(But it probably doesn't matter in this case, because the hang is
probably bug in the VM).

- Kris

On Wed, Mar 6, 2013 at 5:48 AM, Uwe Schindler
mailto:uschind...@apache.org>> wrote:

Hi,

since a few month we are extensively testing various preview
builds of JDK 8 for compatibility with Apache Lucene and
Solr, so we can find any bugs early and prevent the problems
we had with the release of Java 7 two years ago. Currently
we have a Linux (Ubuntu 64bit) Jenkins machine that has
various JDKs (JDK 6, JDK 7, JDK 8 snapshot, IBM J9, older
JRockit) installed, choosing a different one with different
hotspot and garbage collector settings on every run of the
test suite (which takes approx. 30-45 minutes).

JDK 8 b79 works so far very well on Linux, we found some
strange behavior in early versions (maybe compiler errors),
but no longer at the moment. There is one configuration that
constantly and reproducibly hangs in one module that is
tested: The configuration uses JDK 8 b79 (same for b78), 32
bit, and G1GC (server or client does not matter). The JVM
running the tests hangs irresponsible (jstack or kill -3
have no effect/cannot connect, standard kill does not stop
it, only kill -9 actually kills it). It can be reproduced in
this Lucene module 100% (it hangs always).

I was able to connect with GDB to the JVM and get a stack
trace on all threads (see attachment, dump.txt). As you see
all threads of G1GC seem to hang in a syscall (os:park(), a
conditional wait in pthread library). Unfortunately that’s
all I can give you. A Java stacktrace is not possible
because the JVM reacts on neither kill -3 nor jstack. With
all other garbage collectors it passes the test without
hangs in a few seconds, with 32 bit G1GC it can stand still
for hours. The 64 bit JVM passes with G1GC, so only the 32
bit variant is affected. Client or Server VM makes no
difference.

To reproduce:
- Use a 32 bit JDK 8 b78 or b79 (tested on Linux 64 bit, but
this should not matter)
- Download Lucene Source code (e.g. the snapshot version we
were testing with:

https://builds.apache.org/job/__Lucene-Artifacts-trunk/2212/__artifact/lucene/dist/

)
- change to directory lucene/analysis/uima and run:
  ant -Dargs="-server -XX:+UseG1GC"
-Dtests.multiplier=3 -Dtests.jvms=1 test
After a while the test framework prints "stalled" messages
(because the child VM actually running the test no lon

RE: JVM hanging when using G1GC on JDK8 b78 or b79 (Linux 32 bit)

Thanks, I'll try to use jstack with -F or -m!

Uwe

-
Uwe Schindler
uschind...@apache.org 
Apache Lucene PMC Member / Committer
Bremen, Germany
http://lucene.apache.org/


> -Original Message-
> From: Krystal Mok [mailto:rednaxel...@gmail.com]
> Sent: Wednesday, March 06, 2013 7:08 AM
> To: Uwe Schindler
> Cc: hotspot-gc-...@openjdk.java.net; hotspot-...@openjdk.java.net;
> Dawid Weiss; dev@lucene.apache.org
> Subject: Re: JVM hanging when using G1GC on JDK8 b78 or b79 (Linux 32 bit)
> 
> Hi Uwe,
> 
> If you can attach gdb onto it, and jstack -m and jstack -F should also work;
> that'll get you the Java stack trace.
> (But it probably doesn't matter in this case, because the hang is probably bug
> in the VM).
> 
> - Kris
> 
> On Wed, Mar 6, 2013 at 5:48 AM, Uwe Schindler 
> wrote:
> > Hi,
> >
> > since a few month we are extensively testing various preview builds of JDK
> 8 for compatibility with Apache Lucene and Solr, so we can find any bugs
> early and prevent the problems we had with the release of Java 7 two years
> ago. Currently we have a Linux (Ubuntu 64bit) Jenkins machine that has
> various JDKs (JDK 6, JDK 7, JDK 8 snapshot, IBM J9, older JRockit) installed,
> choosing a different one with different hotspot and garbage collector
> settings on every run of the test suite (which takes approx. 30-45 minutes).
> >
> > JDK 8 b79 works so far very well on Linux, we found some strange behavior
> in early versions (maybe compiler errors), but no longer at the moment.
> There is one configuration that constantly and reproducibly hangs in one
> module that is tested: The configuration uses JDK 8 b79 (same for b78), 32
> bit, and G1GC (server or client does not matter). The JVM running the tests
> hangs irresponsible (jstack or kill -3 have no effect/cannot connect, standard
> kill does not stop it, only kill -9 actually kills it). It can be reproduced 
> in this
> Lucene module 100% (it hangs always).
> >
> > I was able to connect with GDB to the JVM and get a stack trace on all
> threads (see attachment, dump.txt). As you see all threads of G1GC seem to
> hang in a syscall (os:park(), a conditional wait in pthread library).
> Unfortunately that’s all I can give you. A Java stacktrace is not possible
> because the JVM reacts on neither kill -3 nor jstack. With all other garbage
> collectors it passes the test without hangs in a few seconds, with 32 bit G1GC
> it can stand still for hours. The 64 bit JVM passes with G1GC, so only the 32 
> bit
> variant is affected. Client or Server VM makes no difference.
> >
> > To reproduce:
> > - Use a 32 bit JDK 8 b78 or b79 (tested on Linux 64 bit, but this
> > should not matter)
> > - Download Lucene Source code (e.g. the snapshot version we were
> > testing with:
> > https://builds.apache.org/job/Lucene-Artifacts-trunk/2212/artifact/luc
> > ene/dist/)
> > - change to directory lucene/analysis/uima and run:
> > ant -Dargs="-server -XX:+UseG1GC" -Dtests.multiplier=3
> > -Dtests.jvms=1 test After a while the test framework prints "stalled"
> messages (because the child VM actually running the test no longer
> responds). The PID is also printed. Try to get a stack trace or kill it, no
> response. Only kill -9 helps. Choosing another garbage collector in the above
> command line makes the test finish after a few seconds, e.g. -Dargs="-server
> -XX:+UseConcMarkSweepGC"
> >
> > I posted this bug report directly to the mailing list, because with earlier 
> > bug
> reports, there seem to be a problem with bugs.sun.com - there is no
> response from any reviewer after several weeks and we were able to help to
> find and fix javadoc and javac-compiler bugs early. So I hope you can help for
> this bug, too.
> >
> > Uwe
> >
> > -
> > Uwe Schindler
> > uschind...@apache.org
> > Apache Lucene PMC Member / Committer
> > Bremen, Germany
> > http://lucene.apache.org/
> >
> >
> 
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional
> commands, e-mail: dev-h...@lucene.apache.org


-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (LUCENE-3918) Port index sorter to trunk APIs


 [ 
https://issues.apache.org/jira/browse/LUCENE-3918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shai Erera updated LUCENE-3918:
---

Attachment: LUCENE-3918.patch

I added to SuppressCodecs the ones Mike mentioned plus MockRandom since it also 
picks randomly SepWriter.

Improved some javadocs, more cleanups. Added CHANGES.

I think it's ready, would appreciate if someone can take another look.

> Port index sorter to trunk APIs
> ---
>
> Key: LUCENE-3918
> URL: https://issues.apache.org/jira/browse/LUCENE-3918
> Project: Lucene - Core
>  Issue Type: Task
>  Components: modules/other
>Affects Versions: 4.0-ALPHA
>Reporter: Robert Muir
> Fix For: 4.2, 5.0
>
> Attachments: LUCENE-3918.patch, LUCENE-3918.patch, LUCENE-3918.patch, 
> LUCENE-3918.patch, LUCENE-3918.patch, LUCENE-3918.patch, LUCENE-3918.patch, 
> LUCENE-3918.patch, LUCENE-3918.patch, LUCENE-3918.patch
>
>
> LUCENE-2482 added an IndexSorter to 3.x, but we need to port this
> functionality to 4.0 apis.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Comment Edited] (SOLR-4470) Support for basic http auth in internal solr requests

[
https://issues.apache.org/jira/browse/SOLR-4470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13594492#comment-13594492
]

Per Steffensen edited comment on SOLR-4470 at 3/6/13 8:38 AM:
--

bq. You could run it anyway.

I will bring a patch passing precommit shortly

bq. Perhaps you can catch another committers attention.

Hopefully

bq. It would in my eyes be wiser to split the elephant

I agree, but it is hard, because the changes in the actual non-test code is not
that comprehensive. I will try though, if a least one committer indicates that
he will take the patch seriously and spent some time on this issue.

bq. Is there some major part you can leave out and start with the simplest
possible codebase to enable simple auth on internal requests only?

Actually that is all it is already - besides I fixed the PeerSync.close issue,
which is just a few lines. Most of the patch is related to testing, and I dont
like to deliver parts that are not thoroughly tested.

bq. Another option is to do the finalization of auth in a new branch

We could do that, but will it make a difference? The end goal is to have it on
a branch from which an actual release will be build. The changes will become
hard to merge very soon anyway - wont they?

bq. Just a pity non-committers can't work with branches in svn

Well, I guess it wouldnt be that hard if the branch is made from branch_4x
revision 1452629 - the patch will fit without any problems.

bq. I could create a branch off trunk for you and commit your code to it if
you're willing to make it apply

I dont know the magnitude of the diff between branch_4x and trunk - according
to Mark they where almost equal some time ago, but that might not be the case
anymore? Besides that you indicated earlier that it wouldnt matter if it was a
bracn_4x or a trunk patch. But I think I would be able to create a patch
fitting trunk fairly easy. But again, there is no need I do this, if no
committer is willing to actually take this patch seriously and work a little
bit with it. But if you think it makes a difference Jan, I will do it? But
please also consider just making the branch from branch_4x and it will fit
without problems.

bq. Oh, I don't mind if he runs precommit or not - I don't think contributors
need to.

That was my impression too

bq. I've just read enough little negative jabs through all of the issues I've
interacted with Per on that I'm ready to let someone else work with him if they
desire. Getting into security like this has usually been avoided with Solr -
coming in with a bad attitude just seems counter productive.

Maybe it is just because I actually give a damn. Unfortunately I have to say
that I do not believe Solr(Cloud) will become a serious competition to
ElasticSearch (and others), if quality of code and committer attitude do not
change. The code is a mess and will not be maintainable for very long if
way-of-working do not change. The main reason is that apparently no one ever
refactor. It is all just small patches on top of already messy code making it
even more messy. It is like you are not allowed to do natural refactoring
because it make a patch bigger than the absolute minimum to introduce the new
feature. And apparently we do not trust the test-suite enough to say, that if
it is green after a patch (and you didnt actually delete, modify or ignore
existing tests in a suspicious manner), nothing was broken. It is an illusion
that people will ever do refactor/cleanup-only patches - refactoring/cleanup is
something you do as a natural thing when touching the code for other reasons.
And even if someone actually did a refactor/cleanup-only patch it would
probably be too big by itself to make it into the code anyway.

The small jabs are continuous attempt to try to influence things that I believe
is bad around Solr, and it IS because I give a damn, so IMHO the bad attitude
is not on my side, it is on (some of) the Solr-core-members side. Doing
retrospective sessions among Solr-core-members might be a good idea, to at
least consider what works well and what might needs some twisting (process- and
code-requirements-wise). And I am a little surprised that this single last line
it my thorough description above was the only one that really got some
attention. It is a one-line small jab in a big productive description.

Of course I accept current rules and agreed-ways-to-do-things, but it will not
stop me trying to change it wherever I think it is currently wrong. Because I
give a damn.

Maybe Solr-core-members also have something to learn from developers outside
the Solr-core. I have a lot of experience as a developer, architect, mentor,
teacher etc., and I have strong opinions based on experience. But I am aways
ready to listen to arguments and learn, and you have heard me (several time)
say "I stand corrected" when

[jira] [Commented] (SOLR-4470) Support for basic http auth in internal solr requests