[jira] [Commented] (SOLR-10249) Allow index fetching to return a detailed result instead of a true/false value

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-10249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15932157#comment-15932157
 ] 

ASF GitHub Bot commented on SOLR-10249:
---

Github user dsmiley commented on the issue:

https://github.com/apache/lucene-solr/pull/169
  
@millerjeff0  when you created this PR, you didn't put SOLR-10249 in *in 
the title* and therefore the PR and the Jira issue aren't linked.  Can you edit 
it and do that please?


> Allow index fetching to return a detailed result instead of a true/false value
> --
>
> Key: SOLR-10249
> URL: https://issues.apache.org/jira/browse/SOLR-10249
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: replication (java)
>Affects Versions: 6.4.1
> Environment: Any
>Reporter: Jeff Miller
>Priority: Trivial
>  Labels: easyfix, newbie
> Fix For: 6.5
>
>   Original Estimate: 3h
>  Remaining Estimate: 3h
>
> This gives us the ability to see into why a replication might of failed and 
> act on it if we need to.  We use this enhancement for logging conditions so 
> we can quantify what is happening with replication, get success rates, etc.
> The idea is to create a public static class IndexFetchResult as an inner 
> class to IndexFetcher that has strings that hold statuses that could occur 
> while fetching an index.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[GitHub] lucene-solr issue #169: Add status return for replication

[dsmiley]

Github user dsmiley commented on the issue:

https://github.com/apache/lucene-solr/pull/169
  
@millerjeff0  when you created this PR, you didn't put SOLR-10249 in *in 
the title* and therefore the PR and the Jira issue aren't linked.  Can you edit 
it and do that please?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-10317) Solr Nightly Benchmarks

[David Smiley (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-10317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15932153#comment-15932153
 ] 

David Smiley commented on SOLR-10317:
-

This would be an awesome GSoC project.

> Solr Nightly Benchmarks
> ---
>
> Key: SOLR-10317
> URL: https://issues.apache.org/jira/browse/SOLR-10317
> Project: Solr
>  Issue Type: Task
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Ishan Chattopadhyaya
>  Labels: gsoc2017, mentor
>
> Solr needs nightly benchmarks reporting. Similar Lucene benchmarks can be 
> found here, https://home.apache.org/~mikemccand/lucenebench/.
> Preferably, we need:
> # A suite of benchmarks that build Solr from a commit point, start Solr 
> nodes, both in SolrCloud and standalone mode, and record timing information 
> of various operations like indexing, querying, faceting, grouping, 
> replication etc.
> # It should be possible to run them either as an independent suite or as a 
> Jenkins job, and we should be able to report timings as graphs (Jenkins has 
> some charting plugins).
> # The code should eventually be integrated in the Solr codebase, so that it 
> never goes out of date.
> There is some prior work / discussion:
> # https://github.com/shalinmangar/solr-perf-tools (Shalin)
> # https://github.com/chatman/solr-upgrade-tests/blob/master/BENCHMARKS.md 
> (Ishan/Vivek)
> # SOLR-2646 & SOLR-9863 (Mark Miller)
> # https://home.apache.org/~mikemccand/lucenebench/ (Mike McCandless)
> There is support for building, starting, indexing/querying and stopping Solr 
> in some of these frameworks above. However, the benchmarks run are very 
> limited. Any of these can be a starting point, or a new framework can as well 
> be used. The motivation is to be able to cover every functionality of Solr 
> with a corresponding benchmark that is run every night.
> Proposing this as a GSoC 2017 project. I'm willing to mentor, and I'm sure 
> [~shalinmangar] and [~markrmil...@gmail.com] would help here.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-Tests-master - Build # 1750 - Unstable

[Apache Jenkins Server]

Build: https://builds.apache.org/job/Lucene-Solr-Tests-master/1750/

2 tests failed.
FAILED:  org.apache.solr.cloud.OverseerRolesTest.testOverseerRole

Error Message:
Timed out waiting for overseer state change

Stack Trace:
java.lang.AssertionError: Timed out waiting for overseer state change
at 
__randomizedtesting.SeedInfo.seed([EBA0DE45B66DBB6:EF71F07060D5ED67]:0)
at org.junit.Assert.fail(Assert.java:93)
at 
org.apache.solr.cloud.OverseerRolesTest.waitForNewOverseer(OverseerRolesTest.java:62)
at 
org.apache.solr.cloud.OverseerRolesTest.testOverseerRole(OverseerRolesTest.java:140)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.cloud.ShardSplitTest.testSplitWithChaosMonkey

Error Message:
There are still nodes recoverying - waited for 330 seconds

Stack Trace:
java.lang.AssertionError: There are still nodes recoverying - waited for 330 
seconds
at 

[jira] [Commented] (SOLR-10303) Add date/time Stream Evaluators

[Dennis Gove (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-10303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15932130#comment-15932130
 ] 

Dennis Gove commented on SOLR-10303:


For the function names, any concern with using pascal casing for dayOfYear, 
dayOfMonth? That will be consistent with other multi-word cases in streaming.

> Add date/time Stream Evaluators
> ---
>
> Key: SOLR-10303
> URL: https://issues.apache.org/jira/browse/SOLR-10303
> Project: Solr
>  Issue Type: New Feature
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Joel Bernstein
> Attachments: SOLR-10303.patch
>
>
> This ticket will add Stream Evaluators that extract date/time values from a 
> Solr date field. The following Evaluators will be supported:
> hour (date)
> minute (date)
> month (date) 
> monthname(date) 
> quarter(date) 
> second (date)
> year(date)
> Syntax:
> {code}
> select(id,
>year(recdate) as year,
>month(recdate) as month,
>day(recdate) as day,
>search(logs, q="blah", fl="id, recdate", sort="recdate asc", 
> qt="/export"))
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: python 3 support is checked into trunk

[Andi Vajda]

On Mon, 20 Mar 2017, Ruediger Meier wrote:


On Sunday 19 March 2017, Andi Vajda wrote:

I just now checked in support for Python 3 (3.5+),


Thanks a lot!


built and tested
on Mac OS X 10.12 only, with Python 3.6.


FYI I've tested a HelloWorld.jar using your svn trunk on travis build
farm for OSX 10.11 and xcode 7.3, 8, 8.1, 8.2:
https://travis-ci.org/rudimeier/jcc/builds/212820385


Linux support should be next.


It would work already on Linux with this patch
https://github.com/rudimeier/jcc/commit/2ccf7e4b828c678577fc0ace24bdb4680ede207a


I changed linux2 to linux but still need to see what kind of setuptools 
patching/monkeypatching hackery is needed to produce a usable libjcc.so.


Thanks !

Andi..



plus fixing -ljcc and -lpython soname.

BTW support for python >=3.3 would be nice and very easy. We just need
one define like this in macros.h:

#if PY_VERSION_HEX < 0x0305
#define Py_DecodeLocale(_arg_, _size_) _Py_char2wchar(_arg_, _size_)
#endif

... as you see here a huge Ubuntu 14.04 build matrix
https://travis-ci.org/rudimeier/jcc/builds/212830434


Someone with access to Windows, please help test/fix/finish support
for Python 3 on Windows, both with the MSVC and Mingw compilers. I
have no access to Windows anymore.


I know already about one MSVC issue:
https://github.com/rudimeier/jcc/issues/1

probably fixed by
https://github.com/rudimeier/jcc/commit/764ed0dc1f77c68e4a6998688d2e8340704fd237
(But this fix is also not tested yet.)

cu,
Rudi

Re: python 3 support is checked into trunk

[Andi Vajda]

On Mon, 20 Mar 2017, Ruediger Meier wrote:


On Sunday 19 March 2017, Andi Vajda wrote:

I just now checked in support for Python 3 (3.5+),


Thanks a lot!


built and tested
on Mac OS X 10.12 only, with Python 3.6.


FYI I've tested a HelloWorld.jar using your svn trunk on travis build
farm for OSX 10.11 and xcode 7.3, 8, 8.1, 8.2:
https://travis-ci.org/rudimeier/jcc/builds/212820385


Linux support should be next.


It would work already on Linux with this patch
https://github.com/rudimeier/jcc/commit/2ccf7e4b828c678577fc0ace24bdb4680ede207a

plus fixing -ljcc and -lpython soname.

BTW support for python >=3.3 would be nice and very easy. We just need
one define like this in macros.h:

#if PY_VERSION_HEX < 0x0305
#define Py_DecodeLocale(_arg_, _size_) _Py_char2wchar(_arg_, _size_)
#endif


Added to jcc.cpp since that't the only place this is used.


... as you see here a huge Ubuntu 14.04 build matrix
https://travis-ci.org/rudimeier/jcc/builds/212830434


Someone with access to Windows, please help test/fix/finish support
for Python 3 on Windows, both with the MSVC and Mingw compilers. I
have no access to Windows anymore.


I know already about one MSVC issue:
https://github.com/rudimeier/jcc/issues/1

probably fixed by
https://github.com/rudimeier/jcc/commit/764ed0dc1f77c68e4a6998688d2e8340704fd237
(But this fix is also not tested yet.)


I changed strhash to use Py_hash_t.

Andi..



cu,
Rudi

[jira] [Commented] (SOLR-10079) TestInPlaceUpdates(Distrib|Standalone) failures

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-10079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15932073#comment-15932073
 ] 

ASF subversion and git services commented on SOLR-10079:


Commit e09273943b41330ca224377e2e6b6ca54dce84eb in lucene-solr's branch 
refs/heads/master from [~caomanhdat]
[ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=e092739 ]

SOLR-10079: Speedup TestInPlaceUpdatesDistrib in new replication mode


> TestInPlaceUpdates(Distrib|Standalone) failures
> ---
>
> Key: SOLR-10079
> URL: https://issues.apache.org/jira/browse/SOLR-10079
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Steve Rowe
>Assignee: Cao Manh Dat
> Fix For: master (7.0), branch_6x
>
> Attachments: SOLR-10079.patch, stdout, tests-failures.txt
>
>
> From [https://jenkins.thetaphi.de/job/Lucene-Solr-master-Linux/18881/], 
> reproduces for me:
> {noformat}
> Checking out Revision d8d61ff61d1d798f5e3853ef66bc485d0d403f18 
> (refs/remotes/origin/master)
> [...]
>[junit4]   2> NOTE: reproduce with: ant test  
> -Dtestcase=TestInPlaceUpdatesDistrib -Dtests.method=test 
> -Dtests.seed=E1BB56269B8215B0 -Dtests.multiplier=3 -Dtests.slow=true 
> -Dtests.locale=sr-Latn-RS -Dtests.timezone=America/Grand_Turk 
> -Dtests.asserts=true -Dtests.file.encoding=UTF-8
>[junit4] FAILURE 77.7s J2 | TestInPlaceUpdatesDistrib.test <<<
>[junit4]> Throwable #1: java.lang.AssertionError: Earlier: [79, 79, 
> 79], now: [78, 78, 78]
>[junit4]>  at 
> __randomizedtesting.SeedInfo.seed([E1BB56269B8215B0:69EF69FC357E7848]:0)
>[junit4]>  at 
> org.apache.solr.update.TestInPlaceUpdatesDistrib.ensureRtgWorksWithPartialUpdatesTest(TestInPlaceUpdatesDistrib.java:425)
>[junit4]>  at 
> org.apache.solr.update.TestInPlaceUpdatesDistrib.test(TestInPlaceUpdatesDistrib.java:142)
>[junit4]>  at 
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>[junit4]>  at 
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>[junit4]>  at 
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>[junit4]>  at 
> java.base/java.lang.reflect.Method.invoke(Method.java:543)
>[junit4]>  at 
> org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:985)
>[junit4]>  at 
> org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:960)
>[junit4]>  at java.base/java.lang.Thread.run(Thread.java:844)
> [...]
>[junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): 
> {id_i=PostingsFormat(name=LuceneFixedGap), title_s=FSTOrd50, 
> id=PostingsFormat(name=Asserting), 
> id_field_copy_that_does_not_support_in_place_update_s=FSTOrd50}, 
> docValues:{inplace_updatable_float=DocValuesFormat(name=Asserting), 
> id_i=DocValuesFormat(name=Direct), _version_=DocValuesFormat(name=Asserting), 
> title_s=DocValuesFormat(name=Lucene70), id=DocValuesFormat(name=Lucene70), 
> id_field_copy_that_does_not_support_in_place_update_s=DocValuesFormat(name=Lucene70),
>  inplace_updatable_int_with_default=DocValuesFormat(name=Asserting), 
> inplace_updatable_int=DocValuesFormat(name=Direct), 
> inplace_updatable_float_with_default=DocValuesFormat(name=Direct)}, 
> maxPointsInLeafNode=1342, maxMBSortInHeap=6.368734895089348, 
> sim=RandomSimilarity(queryNorm=true): {}, locale=sr-Latn-RS, 
> timezone=America/Grand_Turk
>[junit4]   2> NOTE: Linux 4.4.0-53-generic i386/Oracle Corporation 9-ea 
> (32-bit)/cpus=12,threads=1,free=107734480,total=518979584
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-10079) TestInPlaceUpdates(Distrib|Standalone) failures

[Cao Manh Dat (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-10079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15932058#comment-15932058
 ] 

Cao Manh Dat commented on SOLR-10079:
-

[~ichattopadhyaya] Sure, I will take a look at this issue. I think I can make 
the test faster without adding @Nightly

> TestInPlaceUpdates(Distrib|Standalone) failures
> ---
>
> Key: SOLR-10079
> URL: https://issues.apache.org/jira/browse/SOLR-10079
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Steve Rowe
>Assignee: Cao Manh Dat
> Fix For: master (7.0), branch_6x
>
> Attachments: SOLR-10079.patch, stdout, tests-failures.txt
>
>
> From [https://jenkins.thetaphi.de/job/Lucene-Solr-master-Linux/18881/], 
> reproduces for me:
> {noformat}
> Checking out Revision d8d61ff61d1d798f5e3853ef66bc485d0d403f18 
> (refs/remotes/origin/master)
> [...]
>[junit4]   2> NOTE: reproduce with: ant test  
> -Dtestcase=TestInPlaceUpdatesDistrib -Dtests.method=test 
> -Dtests.seed=E1BB56269B8215B0 -Dtests.multiplier=3 -Dtests.slow=true 
> -Dtests.locale=sr-Latn-RS -Dtests.timezone=America/Grand_Turk 
> -Dtests.asserts=true -Dtests.file.encoding=UTF-8
>[junit4] FAILURE 77.7s J2 | TestInPlaceUpdatesDistrib.test <<<
>[junit4]> Throwable #1: java.lang.AssertionError: Earlier: [79, 79, 
> 79], now: [78, 78, 78]
>[junit4]>  at 
> __randomizedtesting.SeedInfo.seed([E1BB56269B8215B0:69EF69FC357E7848]:0)
>[junit4]>  at 
> org.apache.solr.update.TestInPlaceUpdatesDistrib.ensureRtgWorksWithPartialUpdatesTest(TestInPlaceUpdatesDistrib.java:425)
>[junit4]>  at 
> org.apache.solr.update.TestInPlaceUpdatesDistrib.test(TestInPlaceUpdatesDistrib.java:142)
>[junit4]>  at 
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>[junit4]>  at 
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>[junit4]>  at 
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>[junit4]>  at 
> java.base/java.lang.reflect.Method.invoke(Method.java:543)
>[junit4]>  at 
> org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:985)
>[junit4]>  at 
> org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:960)
>[junit4]>  at java.base/java.lang.Thread.run(Thread.java:844)
> [...]
>[junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): 
> {id_i=PostingsFormat(name=LuceneFixedGap), title_s=FSTOrd50, 
> id=PostingsFormat(name=Asserting), 
> id_field_copy_that_does_not_support_in_place_update_s=FSTOrd50}, 
> docValues:{inplace_updatable_float=DocValuesFormat(name=Asserting), 
> id_i=DocValuesFormat(name=Direct), _version_=DocValuesFormat(name=Asserting), 
> title_s=DocValuesFormat(name=Lucene70), id=DocValuesFormat(name=Lucene70), 
> id_field_copy_that_does_not_support_in_place_update_s=DocValuesFormat(name=Lucene70),
>  inplace_updatable_int_with_default=DocValuesFormat(name=Asserting), 
> inplace_updatable_int=DocValuesFormat(name=Direct), 
> inplace_updatable_float_with_default=DocValuesFormat(name=Direct)}, 
> maxPointsInLeafNode=1342, maxMBSortInHeap=6.368734895089348, 
> sim=RandomSimilarity(queryNorm=true): {}, locale=sr-Latn-RS, 
> timezone=America/Grand_Turk
>[junit4]   2> NOTE: Linux 4.4.0-53-generic i386/Oracle Corporation 9-ea 
> (32-bit)/cpus=12,threads=1,free=107734480,total=518979584
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-9835) Create another replication mode for SolrCloud

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15932053#comment-15932053
 ] 

ASF subversion and git services commented on SOLR-9835:
---

Commit 4bc75dbf235145fad5ec1001004c663e15449523 in lucene-solr's branch 
refs/heads/master from [~caomanhdat]
[ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=4bc75db ]

SOLR-9835: Fix OnlyLeaderIndexesTest failure, inplace updates is not copied 
over properly


> Create another replication mode for SolrCloud
> -
>
> Key: SOLR-9835
> URL: https://issues.apache.org/jira/browse/SOLR-9835
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Cao Manh Dat
>Assignee: Shalin Shekhar Mangar
> Attachments: OnlyLeaderIndexesTest-fail-1.zip, SOLR-9835.patch, 
> SOLR-9835.patch, SOLR-9835.patch, SOLR-9835.patch, SOLR-9835.patch, 
> SOLR-9835.patch, SOLR-9835.patch, SOLR-9835.patch, SOLR-9835.patch, 
> SOLR-9835.patch, SOLR-9835.patch, SOLR-9835.patch, SOLR-9835.patch, 
> SOLR-9835.patch, SOLR-9835.patch
>
>
> The current replication mechanism of SolrCloud is called state machine, which 
> replicas start in same initial state and for each input, the input is 
> distributed across replicas so all replicas will end up with same next state. 
> But this type of replication have some drawbacks
> - The commit (which costly) have to run on all replicas
> - Slow recovery, because if replica miss more than N updates on its down 
> time, the replica have to download entire index from its leader.
> So we create create another replication mode for SolrCloud called state 
> transfer, which acts like master/slave replication. In basically
> - Leader distribute the update to other replicas, but the leader only apply 
> the update to IW, other replicas just store the update to UpdateLog (act like 
> replication).
> - Replicas frequently polling the latest segments from leader.
> Pros:
> - Lightweight for indexing, because only leader are running the commit, 
> updates.
> - Very fast recovery, replicas just have to download the missing segments.
> On CAP point of view, this ticket will trying to promise to end users a 
> distributed systems :
> - Partition tolerance
> - Weak Consistency for normal query : clusters can serve stale data. This 
> happen when leader finish a commit and slave is fetching for latest segment. 
> This period can at most {{pollInterval + time to fetch latest segment}}.
> - Consistency for RTG : if we *do not use DQBs*, replicas will consistence 
> with master just like original SolrCloud mode
> - Weak Availability : just like original SolrCloud mode. If a leader down, 
> client must wait until new leader being elected.
> To use this new replication mode, a new collection must be created with an 
> additional parameter {{liveReplicas=1}}
> {code}
> http://localhost:8983/solr/admin/collections?action=CREATE=newCollection=2=1=1
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-10292) Add cartesian Streaming Expression to build cartesian products from multi-value fields and text fields

[Dennis Gove (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-10292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Dennis Gove updated SOLR-10292:
---
Attachment: SOLR-10292.patch

I think this is ready to go. I've decided to be explicit and register it under 
the function name 'cartesianProduct'.

Full suite of tests and precommit pass.

> Add cartesian Streaming Expression to build cartesian products from 
> multi-value fields and text fields
> --
>
> Key: SOLR-10292
> URL: https://issues.apache.org/jira/browse/SOLR-10292
> Project: Solr
>  Issue Type: New Feature
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Joel Bernstein
> Attachments: SOLR-10292.patch, SOLR-10292.patch, SOLR-10292.patch, 
> SOLR-10292.patch, SOLR-10292.patch
>
>
> Currently all the Streaming Expression such as rollups, intersections, fetch 
> etc, work on single value fields. The *cartesian* expression would create a 
> stream of tuples from a single tuple with a multi-value field. This would 
> allow multi-valued fields to be operated on by the wider library of Streaming 
> Expression.
> For example a single tuple with a multi-valued field:
> id: 1
> author: [Jim, Jack, Steve]
> Would be transformed in the following three tuples:
> id:1
> author:Jim
> id:1
> author:Jack
> id:1
> author:Steve



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: python 3 support is checked into trunk

[Ruediger Meier]

On Sunday 19 March 2017, Andi Vajda wrote:
> I just now checked in support for Python 3 (3.5+), 

Thanks a lot!

> built and tested 
> on Mac OS X 10.12 only, with Python 3.6.

FYI I've tested a HelloWorld.jar using your svn trunk on travis build 
farm for OSX 10.11 and xcode 7.3, 8, 8.1, 8.2:
https://travis-ci.org/rudimeier/jcc/builds/212820385

> Linux support should be next.

It would work already on Linux with this patch
https://github.com/rudimeier/jcc/commit/2ccf7e4b828c678577fc0ace24bdb4680ede207a

plus fixing -ljcc and -lpython soname.

BTW support for python >=3.3 would be nice and very easy. We just need 
one define like this in macros.h:

#if PY_VERSION_HEX < 0x0305
#define Py_DecodeLocale(_arg_, _size_) _Py_char2wchar(_arg_, _size_)
#endif

... as you see here a huge Ubuntu 14.04 build matrix
https://travis-ci.org/rudimeier/jcc/builds/212830434

> Someone with access to Windows, please help test/fix/finish support
> for Python 3 on Windows, both with the MSVC and Mingw compilers. I
> have no access to Windows anymore.

I know already about one MSVC issue:
https://github.com/rudimeier/jcc/issues/1

probably fixed by 
https://github.com/rudimeier/jcc/commit/764ed0dc1f77c68e4a6998688d2e8340704fd237
(But this fix is also not tested yet.)

cu,
Rudi

[JENKINS-EA] Lucene-Solr-master-Linux (32bit/jdk-9-ea+160) - Build # 19211 - Unstable!

[Policeman Jenkins Server]

Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-Linux/19211/
Java: 32bit/jdk-9-ea+160 -client -XX:+UseG1GC

1 tests failed.
FAILED:  org.apache.solr.cloud.OnlyLeaderIndexesTest.test

Error Message:


Stack Trace:
java.lang.AssertionError
at 
__randomizedtesting.SeedInfo.seed([5CAB98A6C52B8F98:D4FFA77C6BD7E260]:0)
at org.junit.Assert.fail(Assert.java:92)
at org.junit.Assert.assertTrue(Assert.java:43)
at org.junit.Assert.assertNotNull(Assert.java:526)
at org.junit.Assert.assertNotNull(Assert.java:537)
at 
org.apache.solr.cloud.OnlyLeaderIndexesTest.outOfOrderDBQWithInPlaceUpdatesTest(OnlyLeaderIndexesTest.java:295)
at 
org.apache.solr.cloud.OnlyLeaderIndexesTest.test(OnlyLeaderIndexesTest.java:78)
at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:547)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.base/java.lang.Thread.run(Thread.java:844)




Build Log:
[...truncated 10884 lines...]
   [junit4] Suite: org.apache.solr.cloud.OnlyLeaderIndexesTest
   [junit4]   2> Creating dataDir: 

[jira] [Commented] (SOLR-10265) Overseer can become the bottleneck in very large clusters

[Noble Paul (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-10265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15932032#comment-15932032
 ] 

Noble Paul commented on SOLR-10265:
---

Yeah, the overseer messages and collection action messages are already handled 
separately in two different queues. If we can  manage the  overseer messages 
without sending to zk it's a huuuge win because that's the bulk of the 
operations 

> Overseer can become the bottleneck in very large clusters
> -
>
> Key: SOLR-10265
> URL: https://issues.apache.org/jira/browse/SOLR-10265
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Varun Thacker
>
> Let's say we have a large cluster. Some numbers:
> - To ingest the data at the volume we want to I need roughly a 600 shard 
> collection.
> - Index into the collection for 1 hour and then create a new collection 
> - For a 30 days retention window with these numbers we would end up wth  
> ~400k cores in the cluster
> - Just a rolling restart of this cluster can take hours because the overseer 
> queue gets backed up. If a few nodes looses connectivity to ZooKeeper then 
> also we can end up with lots of messages in the Overseer queue
> With some tests here are the two high level problems we have identified:
> 1> How fast can the overseer process operations:
> The rate at which the overseer processes events is too slow at this scale. 
> I ran {{OverseerTest#testPerformance}} which creates 10 collections ( 1 shard 
> 1 replica ) and generates 20k state change events. The test took 119 seconds 
> to run on my machine which means ~170 events a second. Let's say a server can 
> process 5x of my machine so 1k events a second. 
> Total events generated by a 400k replica cluster = 400k * 4 ( state changes 
> till replica become active ) = 1.6M / 1k events a second will be 1600 minutes.
> Second observation was that the rate at which the overseer can process events 
> slows down when the number of items in the queue gets larger
> I ran the same {{OverseerTest#testPerformance}} but changed the number of 
> events generated to 2000 instead. The test took only 5 seconds to run. So it 
> was a lot faster than the test run which generated 20k events
> 2> State changes overwhelming ZK:
> For every state change Solr is writing out a big state.json to zookeeper. 
> This can lead to the zookeeper transaction logs going out of control even 
> with auto purging etc set . 
> I haven't debugged why the transaction logs ran into terabytes without taking 
> into snapshots but this was my assumption based on the other problems we 
> observed



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-6.x-MacOSX (64bit/jdk1.8.0) - Build # 771 - Still Unstable!

[Policeman Jenkins Server]

Build: https://jenkins.thetaphi.de/job/Lucene-Solr-6.x-MacOSX/771/
Java: 64bit/jdk1.8.0 -XX:-UseCompressedOops -XX:+UseSerialGC

35 tests failed.
FAILED:  
org.apache.solr.cloud.TestDeleteCollectionOnDownNodes.deleteCollectionWithDownNodes

Error Message:
Timed out waiting for leader elections null Last available state: null

Stack Trace:
java.lang.AssertionError: Timed out waiting for leader elections
null
Last available state: null
at 
__randomizedtesting.SeedInfo.seed([E1B1817A22BABE1A:7F84E5820499F292]:0)
at org.junit.Assert.fail(Assert.java:93)
at 
org.apache.solr.cloud.SolrCloudTestCase.waitForState(SolrCloudTestCase.java:265)
at 
org.apache.solr.cloud.TestDeleteCollectionOnDownNodes.deleteCollectionWithDownNodes(TestDeleteCollectionOnDownNodes.java:47)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.lang.Thread.run(Thread.java:745)


FAILED:  
org.apache.solr.cloud.TestLeaderElectionZkExpiry.testLeaderElectionWithZkExpiry

Error Message:


Stack 

[jira] [Updated] (SOLR-10317) Solr Nightly Benchmarks

[Ishan Chattopadhyaya (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-10317?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ishan Chattopadhyaya updated SOLR-10317:

Description: 
Solr needs nightly benchmarks reporting. Similar Lucene benchmarks can be found 
here, https://home.apache.org/~mikemccand/lucenebench/.

Preferably, we need:
# A suite of benchmarks that build Solr from a commit point, start Solr nodes, 
both in SolrCloud and standalone mode, and record timing information of various 
operations like indexing, querying, faceting, grouping, replication etc.
# It should be possible to run them either as an independent suite or as a 
Jenkins job, and we should be able to report timings as graphs (Jenkins has 
some charting plugins).
# The code should eventually be integrated in the Solr codebase, so that it 
never goes out of date.

There is some prior work / discussion:
# https://github.com/shalinmangar/solr-perf-tools (Shalin)
# https://github.com/chatman/solr-upgrade-tests/blob/master/BENCHMARKS.md 
(Ishan/Vivek)
# SOLR-2646 & SOLR-9863 (Mark Miller)
# https://home.apache.org/~mikemccand/lucenebench/ (Mike McCandless)

There is support for building, starting, indexing/querying and stopping Solr in 
some of these frameworks above. However, the benchmarks run are very limited. 
Any of these can be a starting point, or a new framework can as well be used. 
The motivation is to be able to cover every functionality of Solr with a 
corresponding benchmark that is run every night.

Proposing this as a GSoC 2017 project. I'm willing to mentor, and I'm sure 
[~shalinmangar] and [~markrmil...@gmail.com] would help here.

  was:
Solr needs nightly benchmarks reporting. Similar Lucene benchmarks can be found 
here, https://home.apache.org/~mikemccand/lucenebench/.

Preferably, we need:
# A suite of benchmarks that build Solr from a commit point, start Solr nodes, 
both in SolrCloud and standalone mode, and record timing information of various 
operations like indexing, querying, faceting, grouping, replication etc.
# It should be possible to run them either as an independent suite or as a 
Jenkins job, and we should be able to report timings as graphs (Jenkins has 
some charting plugins).
# The code should eventually be integrated in the Solr codebase, so that it 
never goes out of date.

There is some prior work / discussion:
# https://github.com/shalinmangar/solr-perf-tools (Shalin)
# https://github.com/chatman/solr-upgrade-tests/blob/master/BENCHMARKS.md 
(Ishan/Vivek)
# SOLR-2646 & SOLR-9863 (Mark Miller)
# https://home.apache.org/~mikemccand/lucenebench/ (Mike McCandless)

There is support for building, starting, indexing/querying and stopping Solr in 
some of these frameworks above. However, the benchmarks run are very limited. 
Any of these can be a starting point, or a new framework can as well be used. 
The motivation is to be able to cover every functionality of Solr with a 
corresponding benchmark that is run every night.


> Solr Nightly Benchmarks
> ---
>
> Key: SOLR-10317
> URL: https://issues.apache.org/jira/browse/SOLR-10317
> Project: Solr
>  Issue Type: Task
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Ishan Chattopadhyaya
>  Labels: gsoc2017, mentor
>
> Solr needs nightly benchmarks reporting. Similar Lucene benchmarks can be 
> found here, https://home.apache.org/~mikemccand/lucenebench/.
> Preferably, we need:
> # A suite of benchmarks that build Solr from a commit point, start Solr 
> nodes, both in SolrCloud and standalone mode, and record timing information 
> of various operations like indexing, querying, faceting, grouping, 
> replication etc.
> # It should be possible to run them either as an independent suite or as a 
> Jenkins job, and we should be able to report timings as graphs (Jenkins has 
> some charting plugins).
> # The code should eventually be integrated in the Solr codebase, so that it 
> never goes out of date.
> There is some prior work / discussion:
> # https://github.com/shalinmangar/solr-perf-tools (Shalin)
> # https://github.com/chatman/solr-upgrade-tests/blob/master/BENCHMARKS.md 
> (Ishan/Vivek)
> # SOLR-2646 & SOLR-9863 (Mark Miller)
> # https://home.apache.org/~mikemccand/lucenebench/ (Mike McCandless)
> There is support for building, starting, indexing/querying and stopping Solr 
> in some of these frameworks above. However, the benchmarks run are very 
> limited. Any of these can be a starting point, or a new framework can as well 
> be used. The motivation is to be able to cover every functionality of Solr 
> with a corresponding benchmark that is run every night.
> Proposing this as a GSoC 2017 project. I'm willing to mentor, and I'm sure 
> [~shalinmangar] and [~markrmil...@gmail.com] would help here.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (SOLR-10317) Solr Nightly Benchmarks

[Ishan Chattopadhyaya (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-10317?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ishan Chattopadhyaya updated SOLR-10317:

Description: 
Solr needs nightly benchmarks reporting. Similar Lucene benchmarks can be found 
here, https://home.apache.org/~mikemccand/lucenebench/.

Preferably, we need:
# A suite of benchmarks that build Solr from a commit point, start Solr nodes, 
both in SolrCloud and standalone mode, and record timing information of various 
operations like indexing, querying, faceting, grouping, replication etc.
# It should be possible to run them either as an independent suite or as a 
Jenkins job, and we should be able to report timings as graphs (Jenkins has 
some charting plugins).
# The code should eventually be integrated in the Solr codebase, so that it 
never goes out of date.

There is some prior work / discussion:
# https://github.com/shalinmangar/solr-perf-tools (Shalin)
# https://github.com/chatman/solr-upgrade-tests/blob/master/BENCHMARKS.md 
(Ishan/Vivek)
# SOLR-2646 & SOLR-9863 (Mark Miller)
# https://home.apache.org/~mikemccand/lucenebench/ (Mike McCandless)

There is support for building, starting, indexing/querying and stopping Solr in 
some of these frameworks above. However, the benchmarks run are very limited. 
Any of these can be a starting point, or a new framework can as well be used. 
The motivation is to be able to cover every functionality of Solr with a 
corresponding benchmark that is run every night.

  was:
Solr needs nightly benchmarks reporting. Similar Lucene benchmarks can be found 
here, https://home.apache.org/~mikemccand/lucenebench/.

Preferably, we need:
# A suite of benchmarks that build Solr from a commit point, start Solr nodes, 
both in SolrCloud and standalone mode, and record timing information of various 
operations like indexing, querying, faceting, grouping, replication etc.
# It should be possible to run them either as an independent suite or as a 
Jenkins job, and we should be able to report timings as graphs (Jenkins has 
some charting plugins).
# The code should eventually be integrated in the Solr codebase, so that it 
never goes out of date.

There is some prior work / discussion:
# https://github.com/shalinmangar/solr-perf-tools (Shalin)
# https://github.com/chatman/solr-upgrade-tests/blob/master/BENCHMARKS.md 
(Ishan/Vivek)
# SOLR-2646 (Mark Miller)
# https://home.apache.org/~mikemccand/lucenebench/ (Mike McCandless)

There is support for building, starting, indexing/querying and stopping Solr in 
some of these frameworks above. However, the benchmarks run are very limited. 
Any of these can be a starting point, or a new framework can as well be used. 
The motivation is to be able to cover every functionality of Solr with a 
corresponding benchmark that is run every night.


> Solr Nightly Benchmarks
> ---
>
> Key: SOLR-10317
> URL: https://issues.apache.org/jira/browse/SOLR-10317
> Project: Solr
>  Issue Type: Task
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Ishan Chattopadhyaya
>  Labels: gsoc2017, mentor
>
> Solr needs nightly benchmarks reporting. Similar Lucene benchmarks can be 
> found here, https://home.apache.org/~mikemccand/lucenebench/.
> Preferably, we need:
> # A suite of benchmarks that build Solr from a commit point, start Solr 
> nodes, both in SolrCloud and standalone mode, and record timing information 
> of various operations like indexing, querying, faceting, grouping, 
> replication etc.
> # It should be possible to run them either as an independent suite or as a 
> Jenkins job, and we should be able to report timings as graphs (Jenkins has 
> some charting plugins).
> # The code should eventually be integrated in the Solr codebase, so that it 
> never goes out of date.
> There is some prior work / discussion:
> # https://github.com/shalinmangar/solr-perf-tools (Shalin)
> # https://github.com/chatman/solr-upgrade-tests/blob/master/BENCHMARKS.md 
> (Ishan/Vivek)
> # SOLR-2646 & SOLR-9863 (Mark Miller)
> # https://home.apache.org/~mikemccand/lucenebench/ (Mike McCandless)
> There is support for building, starting, indexing/querying and stopping Solr 
> in some of these frameworks above. However, the benchmarks run are very 
> limited. Any of these can be a starting point, or a new framework can as well 
> be used. The motivation is to be able to cover every functionality of Solr 
> with a corresponding benchmark that is run every night.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-10317) Solr Nightly Benchmarks

[Ishan Chattopadhyaya (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-10317?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ishan Chattopadhyaya updated SOLR-10317:

Description: 
Solr needs nightly benchmarks reporting. Similar Lucene benchmarks can be found 
here, https://home.apache.org/~mikemccand/lucenebench/.

Preferably, we need:
# A suite of benchmarks that build Solr from a commit point, start Solr nodes, 
both in SolrCloud and standalone mode, and record timing information of various 
operations like indexing, querying, faceting, grouping, replication etc.
# It should be possible to run them either as an independent suite or as a 
Jenkins job, and we should be able to report timings as graphs (Jenkins has 
some charting plugins).
# The code should eventually be integrated in the Solr codebase, so that it 
never goes out of date.

There is some prior work / discussion:
# https://github.com/shalinmangar/solr-perf-tools (Shalin)
# https://github.com/chatman/solr-upgrade-tests/blob/master/BENCHMARKS.md 
(Ishan/Vivek)
# SOLR-2646 (Mark Miller)
# https://home.apache.org/~mikemccand/lucenebench/ (Mike McCandless)

There is support for building, starting, indexing/querying and stopping Solr in 
some of these frameworks above. However, the benchmarks run are very limited. 
Any of these can be a starting point, or a new framework can as well be used. 
The motivation is to be able to cover every functionality of Solr with a 
corresponding benchmark that is run every night.

  was:
Solr needs nightly benchmarks reporting. Similar Lucene benchmarks can be found 
here, https://home.apache.org/~mikemccand/lucenebench/.

Preferably, we need:
# A suite of benchmarks that build Solr from a commit point, start Solr nodes, 
both in SolrCloud and standalone mode, and record timing information of various 
operations like indexing, querying, faceting, grouping, replication etc.
# It should be possible to run them via Jenkins, and we should be able to 
leverage some reporting/charting plugins.
# The code should eventually be integrated in the Solr codebase, so that it 
never goes out of date.

There is some prior work / discussion:
# https://github.com/shalinmangar/solr-perf-tools (Shalin)
# https://github.com/chatman/solr-upgrade-tests/blob/master/BENCHMARKS.md 
(Ishan/Vivek)
# SOLR-2646 (Mark Miller)
# https://home.apache.org/~mikemccand/lucenebench/ (Mike McCandless)

There is support for building, starting, indexing/querying and stopping Solr in 
some of these frameworks above. However, the benchmarks run are very limited. 
Any of these can be a starting point, or a new framework can as well be used. 
The motivation is to be able to cover every functionality of Solr with a 
corresponding benchmark that is run every night.


> Solr Nightly Benchmarks
> ---
>
> Key: SOLR-10317
> URL: https://issues.apache.org/jira/browse/SOLR-10317
> Project: Solr
>  Issue Type: Task
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Ishan Chattopadhyaya
>  Labels: gsoc2017, mentor
>
> Solr needs nightly benchmarks reporting. Similar Lucene benchmarks can be 
> found here, https://home.apache.org/~mikemccand/lucenebench/.
> Preferably, we need:
> # A suite of benchmarks that build Solr from a commit point, start Solr 
> nodes, both in SolrCloud and standalone mode, and record timing information 
> of various operations like indexing, querying, faceting, grouping, 
> replication etc.
> # It should be possible to run them either as an independent suite or as a 
> Jenkins job, and we should be able to report timings as graphs (Jenkins has 
> some charting plugins).
> # The code should eventually be integrated in the Solr codebase, so that it 
> never goes out of date.
> There is some prior work / discussion:
> # https://github.com/shalinmangar/solr-perf-tools (Shalin)
> # https://github.com/chatman/solr-upgrade-tests/blob/master/BENCHMARKS.md 
> (Ishan/Vivek)
> # SOLR-2646 (Mark Miller)
> # https://home.apache.org/~mikemccand/lucenebench/ (Mike McCandless)
> There is support for building, starting, indexing/querying and stopping Solr 
> in some of these frameworks above. However, the benchmarks run are very 
> limited. Any of these can be a starting point, or a new framework can as well 
> be used. The motivation is to be able to cover every functionality of Solr 
> with a corresponding benchmark that is run every night.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Created] (SOLR-10317) Solr Nightly Benchmarks

[Ishan Chattopadhyaya (JIRA)]

Ishan Chattopadhyaya created SOLR-10317:
---

 Summary: Solr Nightly Benchmarks
 Key: SOLR-10317
 URL: https://issues.apache.org/jira/browse/SOLR-10317
 Project: Solr
  Issue Type: Task
  Security Level: Public (Default Security Level. Issues are Public)
Reporter: Ishan Chattopadhyaya


Solr needs nightly benchmarks reporting. Similar Lucene benchmarks can be found 
here, https://home.apache.org/~mikemccand/lucenebench/.

Preferably, we need:
# A suite of benchmarks that build Solr from a commit point, start Solr nodes, 
both in SolrCloud and standalone mode, and record timing information of various 
operations like indexing, querying, faceting, grouping, replication etc.
# It should be possible to run them via Jenkins, and we should be able to 
leverage some reporting/charting plugins.
# The code should eventually be integrated in the Solr codebase, so that it 
never goes out of date.

There is some prior work / discussion:
# https://github.com/shalinmangar/solr-perf-tools (Shalin)
# https://github.com/chatman/solr-upgrade-tests/blob/master/BENCHMARKS.md 
(Ishan/Vivek)
# SOLR-2646 (Mark Miller)
# https://home.apache.org/~mikemccand/lucenebench/ (Mike McCandless)

There is support for building, starting, indexing/querying and stopping Solr in 
some of these frameworks above. However, the benchmarks run are very limited. 
Any of these can be a starting point, or a new framework can as well be used. 
The motivation is to be able to cover every functionality of Solr with a 
corresponding benchmark that is run every night.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-7745) Explore GPU acceleration

[Ishan Chattopadhyaya (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-7745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15931975#comment-15931975
 ] 

Ishan Chattopadhyaya commented on LUCENE-7745:
--

Here are some ideas on things to start out with:
# Copy over and index lots of points and corresponding docids to the GPU as an 
offline, one time operation. Then, given a query point, return top-n nearest 
indexed points.
# Copy over and index lots of points and corresponding docids to the GPU as an 
offline, one time operation. Then, given a polygon (complex shape), return all 
points that lie inside the polygon.

In both the cases, compare performance against existing Lucene spatial search. 
One would need to choose the most suitable algorithm for doing these as 
efficiently as possible. Any GPGPU API can be used for now (OpenCL, CUDA) for 
initial exploration.

[~dsmiley], [~kwri...@metacarta.com], [~nknize], [~mikemccand], given your 
depth and expertise in this area, do you have any suggestions? Any other area 
of Lucene that comes to mind which should be easiest to start with, in terms of 
exploring GPU based parallelization?

> Explore GPU acceleration
> 
>
> Key: LUCENE-7745
> URL: https://issues.apache.org/jira/browse/LUCENE-7745
> Project: Lucene - Core
>  Issue Type: Improvement
>Reporter: Ishan Chattopadhyaya
>  Labels: gsoc2017, mentor
>
> There are parts of Lucene that can potentially be speeded up if computations 
> were to be offloaded from CPU to the GPU(s). With commodity GPUs having as 
> high as 12GB of high bandwidth RAM, we might be able to leverage GPUs to 
> speed parts of Lucene (indexing, search).
> First that comes to mind is spatial filtering, which is traditionally known 
> to be a good candidate for GPU based speedup (esp. when complex polygons are 
> involved). In the past, Mike McCandless has mentioned that "both initial 
> indexing and merging are CPU/IO intensive, but they are very amenable to 
> soaking up the hardware's concurrency."
> I'm opening this issue as an exploratory task, suitable for a GSoC project. I 
> volunteer to mentor any GSoC student willing to work on this this summer.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-master-MacOSX (64bit/jdk1.8.0) - Build # 3906 - Still Unstable!

[Policeman Jenkins Server]

Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-MacOSX/3906/
Java: 64bit/jdk1.8.0 -XX:-UseCompressedOops -XX:+UseG1GC

1 tests failed.
FAILED:  org.apache.solr.security.BasicAuthIntegrationTest.testBasicAuth

Error Message:
expected:<200> but was:<403>

Stack Trace:
java.lang.AssertionError: expected:<200> but was:<403>
at 
__randomizedtesting.SeedInfo.seed([A3A051FD5892AF36:1FCE27EFFCC12C4C]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.failNotEquals(Assert.java:647)
at org.junit.Assert.assertEquals(Assert.java:128)
at org.junit.Assert.assertEquals(Assert.java:472)
at org.junit.Assert.assertEquals(Assert.java:456)
at 
org.apache.solr.security.BasicAuthIntegrationTest.executeCommand(BasicAuthIntegrationTest.java:235)
at 
org.apache.solr.security.BasicAuthIntegrationTest.testBasicAuth(BasicAuthIntegrationTest.java:147)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.lang.Thread.run(Thread.java:745)

python 3 support is checked into trunk

[Andi Vajda]

I just now checked in support for Python 3 (3.5+), built and tested on
Mac OS X 10.12 only, with Python 3.6. Linux support should be next. I have no
access to Windows anymore and thus can't test support there.

I manually integrated/merged/changed/fixed the patches proposed by Rüdiger
Meier and Thomas Koch earlier as well as the much older experimental branch
I checked in many years ago.

I refactored the JCC egg's structure, splitting it into two parts, one for
Python 2 support (jcc2, _jcc2, helpers2) and one for Python 3 (jcc3, _jcc3,
helper3). That way, less conditional code is required and Python 3 support can
proceed unshackled.
There still is only one egg for JCC, called jcc.
MANIFEST.in was updated accordingly (but is still untested).

Similarly, the PyLucene tests got split into test2 and test3.

This is still work in progress but it seems to work well on Mac OS X, so far.
All tests pass.

I intend to work next on support for Linux.
Someone with access to Windows, please help test/fix/finish support for
Python 3 on Windows, both with the MSVC and Mingw compilers.
I have no access to Windows anymore. 
I expect to mostly work but with caveats on building the libjcc3.dll with the

version of setuptools included in Python 3.6.

Thank you all for your contributions and patience so far !

Andi..

Re: [JENKINS] Lucene-Solr-master-Windows (64bit/jdk1.8.0_121) - Build # 6460 - Still Unstable!

[Steve Rowe]

A more accurate/complete list of reserved Windows filenames, from the NTFS row 
in the table at 
:

> Reserved words: Only in root directory: $AttrDef $BadClus $Bitmap $Boot 
> $LogFile $MFT $MFTMirr pagefile.sys $Secure $UpCase $Volume $Extend 
> $Extend\$ObjId $Extend\$Quota $Extend\$Reparse ($Extend is a directory) […] 
> Windows forbids the use of the MS-DOS device names AUX, CLOCK$, COM1, …, 
> COM9, CON, LPT1, …, LPT9, NUL and PRN, as well as these names with any 
> extension (for example, AUX.txt)


--
Steve
www.lucidworks.com

> On Mar 19, 2017, at 4:15 PM, Steve Rowe  wrote:
> 
> Files named “con[0-9]?” (and “prn[0-9]?”, etc.?) anger Windows.
> 
> I thought this got fixed a while back?  I can’t find any JIRAs for it though.
> 
> --
> Steve
> www.lucidworks.com
> 
>> On Mar 19, 2017, at 1:06 PM, Policeman Jenkins Server  
>> wrote:
>> 
>> Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-Windows/6460/
>> Java: 64bit/jdk1.8.0_121 -XX:-UseCompressedOops -XX:+UseParallelGC
>> 
>> 3 tests failed.
>> FAILED:  org.apache.lucene.store.TestFileSwitchDirectory.testListAllIsSorted
>> 
>> Error Message:
>> access denied ("java.io.FilePermission" 
>> "C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\lucene\build\core\test\J1\temp\lucene.store.TestFileSwitchDirectory_121F5D1A61C5BA5C-001\bar-017\con"
>>  "write")
>> 
>> Stack Trace:
>> java.security.AccessControlException: access denied 
>> ("java.io.FilePermission" 
>> "C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\lucene\build\core\test\J1\temp\lucene.store.TestFileSwitchDirectory_121F5D1A61C5BA5C-001\bar-017\con"
>>  "write")
>>  at 
>> __randomizedtesting.SeedInfo.seed([121F5D1A61C5BA5C:8833FA68A1FA35D5]:0)
>>  at 
>> java.security.AccessControlContext.checkPermission(AccessControlContext.java:472)
>>  at 
>> java.security.AccessController.checkPermission(AccessController.java:884)
>>  at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
>>  at java.lang.SecurityManager.checkWrite(SecurityManager.java:979)
>>  at sun.nio.fs.WindowsChannelFactory.open(WindowsChannelFactory.java:295)
>>  at 
>> sun.nio.fs.WindowsChannelFactory.newFileChannel(WindowsChannelFactory.java:162)
>>  at 
>> sun.nio.fs.WindowsFileSystemProvider.newByteChannel(WindowsFileSystemProvider.java:225)
>>  at 
>> java.nio.file.spi.FileSystemProvider.newOutputStream(FileSystemProvider.java:434)
>>  at 
>> org.apache.lucene.mockfile.FilterFileSystemProvider.newOutputStream(FilterFileSystemProvider.java:197)
>>  at 
>> org.apache.lucene.mockfile.FilterFileSystemProvider.newOutputStream(FilterFileSystemProvider.java:197)
>>  at 
>> org.apache.lucene.mockfile.HandleTrackingFS.newOutputStream(HandleTrackingFS.java:129)
>>  at 
>> org.apache.lucene.mockfile.HandleTrackingFS.newOutputStream(HandleTrackingFS.java:129)
>>  at 
>> org.apache.lucene.mockfile.FilterFileSystemProvider.newOutputStream(FilterFileSystemProvider.java:197)
>>  at java.nio.file.Files.newOutputStream(Files.java:216)
>>  at 
>> org.apache.lucene.store.FSDirectory$FSIndexOutput.(FSDirectory.java:413)
>>  at 
>> org.apache.lucene.store.FSDirectory$FSIndexOutput.(FSDirectory.java:409)
>>  at 
>> org.apache.lucene.store.FSDirectory.createOutput(FSDirectory.java:253)
>>  at 
>> org.apache.lucene.store.FileSwitchDirectory.createOutput(FileSwitchDirectory.java:161)
>>  at 
>> org.apache.lucene.store.BaseDirectoryTestCase.testListAllIsSorted(BaseDirectoryTestCase.java:1198)
>>  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>  at 
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>>  at 
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>  at java.lang.reflect.Method.invoke(Method.java:498)
>>  at 
>> com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
>>  at 
>> com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
>>  at 
>> com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
>>  at 
>> com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
>>  at 
>> org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
>>  at 
>> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
>>  at 
>> org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
>>  at 
>> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
>>  at 
>> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
>>  at 
>> 

[jira] [Comment Edited] (SOLR-10292) Add cartesian Streaming Expression to build cartesian products from multi-value fields and text fields

[Dennis Gove (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-10292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15931943#comment-15931943
 ] 

Dennis Gove edited comment on SOLR-10292 at 3/19/17 8:31 PM:
-

Tests added and passing.

This does not add any additional evaluators. I think those can be added in 
other tickets. All evaluators are supported by this stream so anything you 
think to add (regex matching, sentence creation, etc...) will work. The stream 
works with both multi-valued and single-valued fields in so much that it will 
treat single-valued fields as a collection with a single item.


was (Author: dpgove):
Tests added and passing.

This does not add any additional evaluators. I think those can be added in 
other tickets. All evaluators are supported by this stream so anything you 
think to add (regex matching, sentence creation, etc...) will work. The stream 
works with both multi-valued and sing-valued fields in so much that it will 
treat single-valued fields as a collection with a single item.

> Add cartesian Streaming Expression to build cartesian products from 
> multi-value fields and text fields
> --
>
> Key: SOLR-10292
> URL: https://issues.apache.org/jira/browse/SOLR-10292
> Project: Solr
>  Issue Type: New Feature
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Joel Bernstein
> Attachments: SOLR-10292.patch, SOLR-10292.patch, SOLR-10292.patch, 
> SOLR-10292.patch
>
>
> Currently all the Streaming Expression such as rollups, intersections, fetch 
> etc, work on single value fields. The *cartesian* expression would create a 
> stream of tuples from a single tuple with a multi-value field. This would 
> allow multi-valued fields to be operated on by the wider library of Streaming 
> Expression.
> For example a single tuple with a multi-valued field:
> id: 1
> author: [Jim, Jack, Steve]
> Would be transformed in the following three tuples:
> id:1
> author:Jim
> id:1
> author:Jack
> id:1
> author:Steve



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-10292) Add cartesian Streaming Expression to build cartesian products from multi-value fields and text fields

[Dennis Gove (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-10292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Dennis Gove updated SOLR-10292:
---
Attachment: SOLR-10292.patch

Tests added and passing.

This does not add any additional evaluators. I think those can be added in 
other tickets. All evaluators are supported by this stream so anything you 
think to add (regex matching, sentence creation, etc...) will work. The stream 
works with both multi-valued and sing-valued fields in so much that it will 
treat single-valued fields as a collection with a single item.

> Add cartesian Streaming Expression to build cartesian products from 
> multi-value fields and text fields
> --
>
> Key: SOLR-10292
> URL: https://issues.apache.org/jira/browse/SOLR-10292
> Project: Solr
>  Issue Type: New Feature
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Joel Bernstein
> Attachments: SOLR-10292.patch, SOLR-10292.patch, SOLR-10292.patch, 
> SOLR-10292.patch
>
>
> Currently all the Streaming Expression such as rollups, intersections, fetch 
> etc, work on single value fields. The *cartesian* expression would create a 
> stream of tuples from a single tuple with a multi-value field. This would 
> allow multi-valued fields to be operated on by the wider library of Streaming 
> Expression.
> For example a single tuple with a multi-valued field:
> id: 1
> author: [Jim, Jack, Steve]
> Would be transformed in the following three tuples:
> id:1
> author:Jim
> id:1
> author:Jack
> id:1
> author:Steve



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: [JENKINS] Lucene-Solr-master-Windows (64bit/jdk1.8.0_121) - Build # 6460 - Still Unstable!

[Steve Rowe]

Files named “con[0-9]?” (and “prn[0-9]?”, etc.?) anger Windows.

I thought this got fixed a while back?  I can’t find any JIRAs for it though.

--
Steve
www.lucidworks.com

> On Mar 19, 2017, at 1:06 PM, Policeman Jenkins Server  
> wrote:
> 
> Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-Windows/6460/
> Java: 64bit/jdk1.8.0_121 -XX:-UseCompressedOops -XX:+UseParallelGC
> 
> 3 tests failed.
> FAILED:  org.apache.lucene.store.TestFileSwitchDirectory.testListAllIsSorted
> 
> Error Message:
> access denied ("java.io.FilePermission" 
> "C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\lucene\build\core\test\J1\temp\lucene.store.TestFileSwitchDirectory_121F5D1A61C5BA5C-001\bar-017\con"
>  "write")
> 
> Stack Trace:
> java.security.AccessControlException: access denied ("java.io.FilePermission" 
> "C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\lucene\build\core\test\J1\temp\lucene.store.TestFileSwitchDirectory_121F5D1A61C5BA5C-001\bar-017\con"
>  "write")
>   at 
> __randomizedtesting.SeedInfo.seed([121F5D1A61C5BA5C:8833FA68A1FA35D5]:0)
>   at 
> java.security.AccessControlContext.checkPermission(AccessControlContext.java:472)
>   at 
> java.security.AccessController.checkPermission(AccessController.java:884)
>   at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
>   at java.lang.SecurityManager.checkWrite(SecurityManager.java:979)
>   at sun.nio.fs.WindowsChannelFactory.open(WindowsChannelFactory.java:295)
>   at 
> sun.nio.fs.WindowsChannelFactory.newFileChannel(WindowsChannelFactory.java:162)
>   at 
> sun.nio.fs.WindowsFileSystemProvider.newByteChannel(WindowsFileSystemProvider.java:225)
>   at 
> java.nio.file.spi.FileSystemProvider.newOutputStream(FileSystemProvider.java:434)
>   at 
> org.apache.lucene.mockfile.FilterFileSystemProvider.newOutputStream(FilterFileSystemProvider.java:197)
>   at 
> org.apache.lucene.mockfile.FilterFileSystemProvider.newOutputStream(FilterFileSystemProvider.java:197)
>   at 
> org.apache.lucene.mockfile.HandleTrackingFS.newOutputStream(HandleTrackingFS.java:129)
>   at 
> org.apache.lucene.mockfile.HandleTrackingFS.newOutputStream(HandleTrackingFS.java:129)
>   at 
> org.apache.lucene.mockfile.FilterFileSystemProvider.newOutputStream(FilterFileSystemProvider.java:197)
>   at java.nio.file.Files.newOutputStream(Files.java:216)
>   at 
> org.apache.lucene.store.FSDirectory$FSIndexOutput.(FSDirectory.java:413)
>   at 
> org.apache.lucene.store.FSDirectory$FSIndexOutput.(FSDirectory.java:409)
>   at 
> org.apache.lucene.store.FSDirectory.createOutput(FSDirectory.java:253)
>   at 
> org.apache.lucene.store.FileSwitchDirectory.createOutput(FileSwitchDirectory.java:161)
>   at 
> org.apache.lucene.store.BaseDirectoryTestCase.testListAllIsSorted(BaseDirectoryTestCase.java:1198)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
>   at 
> com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
>   at 
> com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
>   at 
> com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
>   at 
> org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
>   at 
> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
>   at 
> org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
>   at 
> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
>   at 
> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
>   at 
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>   at 
> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
>   at 
> com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
>   at 
> com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
>   at 
> com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
>   at 
> com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
>   at 
> com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
>   at 
> 

Exception during integration of Solr with UIMA

[aruninfo100]

Hi All,

I am trying to integrate UIMA with Solr.I am following the steps mentioned
in https://cwiki.apache.org/confluence/display/solr/UIMA+Integration .But
when I try to index the  documents,exceptions are thrown in terminal and
solr log is also logged with error traces.I have been trying to work around
for some time,but unable to get a proper solution for the issue.

I am using solr 6.1.0

I have included all the jars mentioned in the document.
analyze field:

 
  content
 
This field hold all the extracted content(text content) from respective
documents indexed(some are large documents).
content field is of field type text_general.It is not a copy field.The field
holds the respective document contents.
  

I have created the three fields too in the config file(referred the
document).

I have generated valid keys for the API's.  Internet connection is 
available.

*solrconfig.xml:*





VALID_ALCHEMYAPI_KEY
VALID_ALCHEMYAPI_KEY
VALID_ALCHEMYAPI_KEY
VALID_ALCHEMYAPI_KEY
VALID_ALCHEMYAPI_KEY
VALID_OPENCALAIS_KEY

/org/apache/uima/desc/OverridingParamsExtServicesAE.xml

true
fileName


false

  content

  
  

  org.apache.uima.alchemy.ts.concept.ConceptFS
  
text
concept
  


  org.apache.uima.alchemy.ts.language.LanguageFS
  
language
language
  


  org.apache.uima.SentenceAnnotation
  
coveredText
sentence
  

  

  
  
  



 
  
uima
  




terminal error trace:

Mar 19, 2017 10:46:16 AM WhitespaceTokenizer typeSystemInit
INFO: "Whitespace tokenizer typesystem initialized"
Mar 19, 2017 10:46:16 AM WhitespaceTokenizer process
INFO: "Whitespace tokenizer starts processing"
Mar 19, 2017 10:46:16 AM WhitespaceTokenizer process
INFO: "Whitespace tokenizer finished processing"
Mar 19, 2017 10:46:16 AM
org.apache.uima.analysis_engine.impl.PrimitiveAnalysisE
ngine_impl callAnalysisComponentProcess(405)
SEVERE: Exception occurred
org.apache.uima.analysis_engine.AnalysisEngineProcessException
at
org.apache.uima.annotator.calais.OpenCalaisAnnotator.process(OpenCala
isAnnotator.java:206)
at
org.apache.uima.analysis_component.CasAnnotator_ImplBase.process(CasA
nnotator_ImplBase.java:56)
at
org.apache.uima.analysis_engine.impl.PrimitiveAnalysisEngine_impl.cal
lAnalysisComponentProcess(PrimitiveAnalysisEngine_impl.java:377)
at
org.apache.uima.analysis_engine.impl.PrimitiveAnalysisEngine_impl.pro
cessAndOutputNewCASes(PrimitiveAnalysisEngine_impl.java:295)
at
org.apache.uima.analysis_engine.asb.impl.ASB_impl$AggregateCasIterato
r.processUntilNextOutputCas(ASB_impl.java:567)
at
org.apache.uima.analysis_engine.asb.impl.ASB_impl$AggregateCasIterato
r.(ASB_impl.java:409)
at
org.apache.uima.analysis_engine.asb.impl.ASB_impl.process(ASB_impl.ja
va:342)
at
org.apache.uima.analysis_engine.impl.AggregateAnalysisEngine_impl.pro
cessAndOutputNewCASes(AggregateAnalysisEngine_impl.java:267)
at
org.apache.uima.analysis_engine.impl.AnalysisEngineImplBase.process(A
nalysisEngineImplBase.java:267)
at
org.apache.uima.analysis_engine.impl.AnalysisEngineImplBase.process(A
nalysisEngineImplBase.java:280)
at
org.apache.solr.uima.processor.UIMAUpdateRequestProcessor.processText
(UIMAUpdateRequestProcessor.java:176)
at
org.apache.solr.uima.processor.UIMAUpdateRequestProcessor.processAdd(
UIMAUpdateRequestProcessor.java:78)
at
org.apache.solr.handler.loader.JavabinLoader$1.update(JavabinLoader.j
ava:97)
at
org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$1.read
OuterMostDocIterator(JavaBinUpdateRequestCodec.java:179)
at
org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$1.read
Iterator(JavaBinUpdateRequestCodec.java:135)
at
org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCodec.java:27
4)
at
org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$1.read
NamedList(JavaBinUpdateRequestCodec.java:121)
at
org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCodec.java:23
9)
at
org.apache.solr.common.util.JavaBinCodec.unmarshal(JavaBinCodec.java:
157)
at
org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec.unmars
hal(JavaBinUpdateRequestCodec.java:186)
at

[JENKINS] Lucene-Solr-6.x-Linux (32bit/jdk1.8.0_121) - Build # 3099 - Unstable!

[Policeman Jenkins Server]

Build: https://jenkins.thetaphi.de/job/Lucene-Solr-6.x-Linux/3099/
Java: 32bit/jdk1.8.0_121 -server -XX:+UseSerialGC

1 tests failed.
FAILED:  org.apache.solr.TestDistributedSearch.test

Error Message:
Expected to find shardAddress in the up shard info

Stack Trace:
java.lang.AssertionError: Expected to find shardAddress in the up shard info
at 
__randomizedtesting.SeedInfo.seed([6A9C571829B52D8:8EFDFAAB2C673F20]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.assertTrue(Assert.java:43)
at 
org.apache.solr.TestDistributedSearch.comparePartialResponses(TestDistributedSearch.java:1176)
at 
org.apache.solr.TestDistributedSearch.queryPartialResults(TestDistributedSearch.java:1117)
at 
org.apache.solr.TestDistributedSearch.test(TestDistributedSearch.java:977)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsRepeatStatement.callStatement(BaseDistributedSearchTestCase.java:1018)
at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:967)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at 

[JENKINS] Lucene-Solr-master-Windows (64bit/jdk1.8.0_121) - Build # 6460 - Still Unstable!

[Policeman Jenkins Server]

Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-Windows/6460/
Java: 64bit/jdk1.8.0_121 -XX:-UseCompressedOops -XX:+UseParallelGC

3 tests failed.
FAILED:  org.apache.lucene.store.TestFileSwitchDirectory.testListAllIsSorted

Error Message:
access denied ("java.io.FilePermission" 
"C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\lucene\build\core\test\J1\temp\lucene.store.TestFileSwitchDirectory_121F5D1A61C5BA5C-001\bar-017\con"
 "write")

Stack Trace:
java.security.AccessControlException: access denied ("java.io.FilePermission" 
"C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\lucene\build\core\test\J1\temp\lucene.store.TestFileSwitchDirectory_121F5D1A61C5BA5C-001\bar-017\con"
 "write")
at 
__randomizedtesting.SeedInfo.seed([121F5D1A61C5BA5C:8833FA68A1FA35D5]:0)
at 
java.security.AccessControlContext.checkPermission(AccessControlContext.java:472)
at 
java.security.AccessController.checkPermission(AccessController.java:884)
at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
at java.lang.SecurityManager.checkWrite(SecurityManager.java:979)
at sun.nio.fs.WindowsChannelFactory.open(WindowsChannelFactory.java:295)
at 
sun.nio.fs.WindowsChannelFactory.newFileChannel(WindowsChannelFactory.java:162)
at 
sun.nio.fs.WindowsFileSystemProvider.newByteChannel(WindowsFileSystemProvider.java:225)
at 
java.nio.file.spi.FileSystemProvider.newOutputStream(FileSystemProvider.java:434)
at 
org.apache.lucene.mockfile.FilterFileSystemProvider.newOutputStream(FilterFileSystemProvider.java:197)
at 
org.apache.lucene.mockfile.FilterFileSystemProvider.newOutputStream(FilterFileSystemProvider.java:197)
at 
org.apache.lucene.mockfile.HandleTrackingFS.newOutputStream(HandleTrackingFS.java:129)
at 
org.apache.lucene.mockfile.HandleTrackingFS.newOutputStream(HandleTrackingFS.java:129)
at 
org.apache.lucene.mockfile.FilterFileSystemProvider.newOutputStream(FilterFileSystemProvider.java:197)
at java.nio.file.Files.newOutputStream(Files.java:216)
at 
org.apache.lucene.store.FSDirectory$FSIndexOutput.(FSDirectory.java:413)
at 
org.apache.lucene.store.FSDirectory$FSIndexOutput.(FSDirectory.java:409)
at 
org.apache.lucene.store.FSDirectory.createOutput(FSDirectory.java:253)
at 
org.apache.lucene.store.FileSwitchDirectory.createOutput(FileSwitchDirectory.java:161)
at 
org.apache.lucene.store.BaseDirectoryTestCase.testListAllIsSorted(BaseDirectoryTestCase.java:1198)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at 

[JENKINS] Lucene-Solr-NightlyTests-6.x - Build # 313 - Unstable

[Apache Jenkins Server]

Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-6.x/313/

2 tests failed.
FAILED:  org.apache.lucene.index.TestIndexSorting.testRandom3

Error Message:
Test abandoned because suite timeout was reached.

Stack Trace:
java.lang.Exception: Test abandoned because suite timeout was reached.
at __randomizedtesting.SeedInfo.seed([F793A635E9E1F98D]:0)


FAILED:  junit.framework.TestSuite.org.apache.lucene.index.TestIndexSorting

Error Message:
Suite timeout exceeded (>= 720 msec).

Stack Trace:
java.lang.Exception: Suite timeout exceeded (>= 720 msec).
at __randomizedtesting.SeedInfo.seed([F793A635E9E1F98D]:0)




Build Log:
[...truncated 1849 lines...]
   [junit4] Suite: org.apache.lucene.index.TestIndexSorting
   [junit4]   2> mrt 19, 2017 3:40:41 AM 
com.carrotsearch.randomizedtesting.ThreadLeakControl$2 evaluate
   [junit4]   2> WARNING: Suite execution timed out: 
org.apache.lucene.index.TestIndexSorting
   [junit4]   2>1) Thread[id=1, name=main, state=WAITING, group=main]
   [junit4]   2> at java.lang.Object.wait(Native Method)
   [junit4]   2> at java.lang.Thread.join(Thread.java:1249)
   [junit4]   2> at java.lang.Thread.join(Thread.java:1323)
   [junit4]   2> at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSuite(RandomizedRunner.java:608)
   [junit4]   2> at 
com.carrotsearch.randomizedtesting.RandomizedRunner.run(RandomizedRunner.java:457)
   [junit4]   2> at 
com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.execute(SlaveMain.java:244)
   [junit4]   2> at 
com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.main(SlaveMain.java:355)
   [junit4]   2> at 
com.carrotsearch.ant.tasks.junit4.slave.SlaveMainSafe.main(SlaveMainSafe.java:13)
   [junit4]   2>2) Thread[id=5711, 
name=TEST-TestIndexSorting.testRandom3-seed#[F793A635E9E1F98D], state=RUNNABLE, 
group=TGRP-TestIndexSorting]
   [junit4]   2> at sun.nio.ch.FileChannelImpl.position0(Native Method)
   [junit4]   2> at 
sun.nio.ch.FileChannelImpl.position(FileChannelImpl.java:288)
   [junit4]   2> at 
sun.nio.ch.FileChannelImpl.position(FileChannelImpl.java:50)
   [junit4]   2> at 
org.apache.lucene.mockfile.FilterSeekableByteChannel.position(FilterSeekableByteChannel.java:74)
   [junit4]   2> at 
org.apache.lucene.mockfile.FilterSeekableByteChannel.position(FilterSeekableByteChannel.java:74)
   [junit4]   2> at 
org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexInput.readInternal(SimpleFSDirectory.java:171)
   [junit4]   2> at 
org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:342)
   [junit4]   2> at 
org.apache.lucene.store.BufferedIndexInput.readBytes(BufferedIndexInput.java:140)
   [junit4]   2> at 
org.apache.lucene.store.BufferedIndexInput.readBytes(BufferedIndexInput.java:116)
   [junit4]   2> at 
org.apache.lucene.store.MockIndexInputWrapper.readBytes(MockIndexInputWrapper.java:146)
   [junit4]   2> at 
org.apache.lucene.store.BufferedChecksumIndexInput.readBytes(BufferedChecksumIndexInput.java:49)
   [junit4]   2> at 
org.apache.lucene.store.DataInput.readBytes(DataInput.java:87)
   [junit4]   2> at 
org.apache.lucene.store.DataInput.skipBytes(DataInput.java:350)
   [junit4]   2> at 
org.apache.lucene.store.ChecksumIndexInput.seek(ChecksumIndexInput.java:52)
   [junit4]   2> at 
org.apache.lucene.codecs.CodecUtil.checksumEntireFile(CodecUtil.java:519)
   [junit4]   2> at 
org.apache.lucene.codecs.memory.DirectDocValuesProducer.checkIntegrity(DirectDocValuesProducer.java:281)
   [junit4]   2> at 
org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat$FieldsReader.checkIntegrity(PerFieldDocValuesFormat.java:366)
   [junit4]   2> at 
org.apache.lucene.codecs.DocValuesConsumer.merge(DocValuesConsumer.java:137)
   [junit4]   2> at 
org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat$FieldsWriter.merge(PerFieldDocValuesFormat.java:153)
   [junit4]   2> at 
org.apache.lucene.index.SegmentMerger.mergeDocValues(SegmentMerger.java:167)
   [junit4]   2> at 
org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:111)
   [junit4]   2> at 
org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4353)
   [junit4]   2> at 
org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3928)
   [junit4]   2> at 
org.apache.lucene.index.SerialMergeScheduler.merge(SerialMergeScheduler.java:40)
   [junit4]   2> at 
org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:2081)
   [junit4]   2> at 
org.apache.lucene.index.IndexWriter.doAfterSegmentFlushed(IndexWriter.java:5002)
   [junit4]   2> at 
org.apache.lucene.index.DocumentsWriter$MergePendingEvent.process(DocumentsWriter.java:731)
   [junit4]   2> at 

[jira] [Commented] (SOLR-10265) Overseer can become the bottleneck in very large clusters

[Erick Erickson (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-10265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15931754#comment-15931754
 ] 

Erick Erickson commented on SOLR-10265:
---

Hmmm, could we take a step in that direction now? Off the top of my head on 
Sunday morning before coffee has really kicked in, so maybe nonsense.

Anyway, it seems like we have two sets of operations here, what about splitting 
them out? The state change messages go into memory and the Collections-level 
calls like "create collection", "addreplica" and the like continue to use the 
Overseer queue. Seems like that would allow us to then build robustness into 
the collection-level ops "sometime" and have them nicely isolated.

> Overseer can become the bottleneck in very large clusters
> -
>
> Key: SOLR-10265
> URL: https://issues.apache.org/jira/browse/SOLR-10265
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Varun Thacker
>
> Let's say we have a large cluster. Some numbers:
> - To ingest the data at the volume we want to I need roughly a 600 shard 
> collection.
> - Index into the collection for 1 hour and then create a new collection 
> - For a 30 days retention window with these numbers we would end up wth  
> ~400k cores in the cluster
> - Just a rolling restart of this cluster can take hours because the overseer 
> queue gets backed up. If a few nodes looses connectivity to ZooKeeper then 
> also we can end up with lots of messages in the Overseer queue
> With some tests here are the two high level problems we have identified:
> 1> How fast can the overseer process operations:
> The rate at which the overseer processes events is too slow at this scale. 
> I ran {{OverseerTest#testPerformance}} which creates 10 collections ( 1 shard 
> 1 replica ) and generates 20k state change events. The test took 119 seconds 
> to run on my machine which means ~170 events a second. Let's say a server can 
> process 5x of my machine so 1k events a second. 
> Total events generated by a 400k replica cluster = 400k * 4 ( state changes 
> till replica become active ) = 1.6M / 1k events a second will be 1600 minutes.
> Second observation was that the rate at which the overseer can process events 
> slows down when the number of items in the queue gets larger
> I ran the same {{OverseerTest#testPerformance}} but changed the number of 
> events generated to 2000 instead. The test took only 5 seconds to run. So it 
> was a lot faster than the test run which generated 20k events
> 2> State changes overwhelming ZK:
> For every state change Solr is writing out a big state.json to zookeeper. 
> This can lead to the zookeeper transaction logs going out of control even 
> with auto purging etc set . 
> I haven't debugged why the transaction logs ran into terabytes without taking 
> into snapshots but this was my assumption based on the other problems we 
> observed



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS-EA] Lucene-Solr-master-Linux (64bit/jdk-9-ea+160) - Build # 19208 - Unstable!

[Policeman Jenkins Server]

Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-Linux/19208/
Java: 64bit/jdk-9-ea+160 -XX:-UseCompressedOops -XX:+UseParallelGC

1 tests failed.
FAILED:  org.apache.solr.cloud.OnlyLeaderIndexesTest.test

Error Message:


Stack Trace:
java.lang.AssertionError
at 
__randomizedtesting.SeedInfo.seed([3CD659DC3F4B31AB:B482660691B75C53]:0)
at org.junit.Assert.fail(Assert.java:92)
at org.junit.Assert.assertTrue(Assert.java:43)
at org.junit.Assert.assertNotNull(Assert.java:526)
at org.junit.Assert.assertNotNull(Assert.java:537)
at 
org.apache.solr.cloud.OnlyLeaderIndexesTest.outOfOrderDBQWithInPlaceUpdatesTest(OnlyLeaderIndexesTest.java:295)
at 
org.apache.solr.cloud.OnlyLeaderIndexesTest.test(OnlyLeaderIndexesTest.java:78)
at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:547)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.base/java.lang.Thread.run(Thread.java:844)




Build Log:
[...truncated 11417 lines...]
   [junit4] Suite: org.apache.solr.cloud.OnlyLeaderIndexesTest
   [junit4]   2> Creating dataDir: 

[JENKINS] Lucene-Solr-6.x-MacOSX (64bit/jdk1.8.0) - Build # 770 - Still Unstable!

[Policeman Jenkins Server]

Build: https://jenkins.thetaphi.de/job/Lucene-Solr-6.x-MacOSX/770/
Java: 64bit/jdk1.8.0 -XX:-UseCompressedOops -XX:+UseParallelGC

1 tests failed.
FAILED:  org.apache.solr.cloud.CustomCollectionTest.testCustomCollectionsAPI

Error Message:
Error from server at https://127.0.0.1:57887/solr: Failed to create shard

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error 
from server at https://127.0.0.1:57887/solr: Failed to create shard
at 
__randomizedtesting.SeedInfo.seed([98F8C321050165A8:F2194D4A389BD3D0]:0)
at 
org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:610)
at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:279)
at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:268)
at 
org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:435)
at 
org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:387)
at 
org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1376)
at 
org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:1127)
at 
org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:1057)
at 
org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:160)
at 
org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:177)
at 
org.apache.solr.cloud.CustomCollectionTest.testCustomCollectionsAPI(CustomCollectionTest.java:103)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 

[JENKINS] Lucene-Solr-master-MacOSX (64bit/jdk1.8.0) - Build # 3905 - Still Unstable!

[Policeman Jenkins Server]

Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-MacOSX/3905/
Java: 64bit/jdk1.8.0 -XX:-UseCompressedOops -XX:+UseParallelGC

1 tests failed.
FAILED:  org.apache.solr.cloud.FullSolrCloudDistribCmdsTest.test

Error Message:
Could not find collection:collection2

Stack Trace:
java.lang.AssertionError: Could not find collection:collection2
at 
__randomizedtesting.SeedInfo.seed([C5B8394910EFB0B:840FBC4E3FF296F3]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.assertTrue(Assert.java:43)
at org.junit.Assert.assertNotNull(Assert.java:526)
at 
org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:159)
at 
org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:144)
at 
org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:139)
at 
org.apache.solr.cloud.AbstractFullDistribZkTestBase.waitForRecoveriesToFinish(AbstractFullDistribZkTestBase.java:870)
at 
org.apache.solr.cloud.FullSolrCloudDistribCmdsTest.testIndexingBatchPerRequestWithHttpSolrClient(FullSolrCloudDistribCmdsTest.java:620)
at 
org.apache.solr.cloud.FullSolrCloudDistribCmdsTest.test(FullSolrCloudDistribCmdsTest.java:152)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:985)
at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:960)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 

[JENKINS] Lucene-Solr-6.x-Windows (64bit/jdk1.8.0_121) - Build # 789 - Unstable!

[Policeman Jenkins Server]

Build: https://jenkins.thetaphi.de/job/Lucene-Solr-6.x-Windows/789/
Java: 64bit/jdk1.8.0_121 -XX:+UseCompressedOops -XX:+UseSerialGC

1 tests failed.
FAILED:  org.apache.solr.core.ExitableDirectoryReaderTest.testCacheAssumptions

Error Message:
Should have fewer docs than 100

Stack Trace:
java.lang.AssertionError: Should have fewer docs than 100
at 
__randomizedtesting.SeedInfo.seed([9A353E8C541DE478:ED4881AB3AE8A95A]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.assertTrue(Assert.java:43)
at 
org.apache.solr.core.ExitableDirectoryReaderTest.testCacheAssumptions(ExitableDirectoryReaderTest.java:102)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.lang.Thread.run(Thread.java:745)




Build Log:
[...truncated 11163 lines...]
   [junit4] Suite: org.apache.solr.core.ExitableDirectoryReaderTest
   [junit4]   2> Creating dataDir: 

[JENKINS] Lucene-Solr-master-Windows (64bit/jdk1.8.0_121) - Build # 6459 - Unstable!

[Policeman Jenkins Server]

Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-Windows/6459/
Java: 64bit/jdk1.8.0_121 -XX:-UseCompressedOops -XX:+UseSerialGC

1 tests failed.
FAILED:  org.apache.solr.morphlines.solr.SolrMorphlineZkAvroTest.test

Error Message:
Cannot download schema.xml from ZooKeeper near: { # String: 1 "solrUrl" 
: "", # String: 1 "collection" : "collection1", # String: 1 
"zkHost" : "127.0.0.1:56768/solr", # String: 1 "batchSize" : "1000",
 # String: 1 "solrHomeDir" : "" }

Stack Trace:
org.kitesdk.morphline.api.MorphlineCompilationException: Cannot download 
schema.xml from ZooKeeper near: {
# String: 1
"solrUrl" : "",
# String: 1
"collection" : "collection1",
# String: 1
"zkHost" : "127.0.0.1:56768/solr",
# String: 1
"batchSize" : "1000",
# String: 1
"solrHomeDir" : ""
}
at 
__randomizedtesting.SeedInfo.seed([5F83611FEBCF2D3D:D7D75EC5453340C5]:0)
at 
org.apache.solr.morphlines.solr.SolrLocator.getIndexSchema(SolrLocator.java:148)
at 
org.apache.solr.morphlines.solr.SanitizeUnknownSolrFieldsBuilder$SanitizeUnknownSolrFields.(SanitizeUnknownSolrFieldsBuilder.java:70)
at 
org.apache.solr.morphlines.solr.SanitizeUnknownSolrFieldsBuilder.build(SanitizeUnknownSolrFieldsBuilder.java:52)
at 
org.kitesdk.morphline.base.AbstractCommand.buildCommand(AbstractCommand.java:302)
at 
org.kitesdk.morphline.base.AbstractCommand.buildCommandChain(AbstractCommand.java:249)
at org.kitesdk.morphline.stdlib.Pipe.(Pipe.java:46)
at org.kitesdk.morphline.stdlib.PipeBuilder.build(PipeBuilder.java:40)
at 
org.apache.solr.morphlines.solr.AbstractSolrMorphlineZkTestBase.createMorphline(AbstractSolrMorphlineZkTestBase.java:112)
at 
org.apache.solr.morphlines.solr.AbstractSolrMorphlineZkTestBase.parse(AbstractSolrMorphlineZkTestBase.java:108)
at 
org.apache.solr.morphlines.solr.AbstractSolrMorphlineZkTestBase.parse(AbstractSolrMorphlineZkTestBase.java:97)
at 
org.apache.solr.morphlines.solr.SolrMorphlineZkAvroTest.test(SolrMorphlineZkAvroTest.java:58)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at