[jira] [Resolved] (SOLR-9896) Instrument and collect metrics from thread pools

[Shalin Shekhar Mangar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-9896?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shalin Shekhar Mangar resolved SOLR-9896.
-
Resolution: Fixed

> Instrument and collect metrics from thread pools
> 
>
> Key: SOLR-9896
> URL: https://issues.apache.org/jira/browse/SOLR-9896
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: metrics
>Reporter: Shalin Shekhar Mangar
>Assignee: Shalin Shekhar Mangar
> Fix For: master (7.0), 6.4
>
> Attachments: SOLR-9896.patch
>
>
> The metrics-core library has a InstrumentedExecutorService which collects 
> stats on submitted, running, completed tasks and durations. This issue will 
> expose such stats for all important thread pools in solr.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Resolved] (SOLR-9877) Use instrumented http client

[Shalin Shekhar Mangar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-9877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shalin Shekhar Mangar resolved SOLR-9877.
-
Resolution: Fixed

> Use instrumented http client
> 
>
> Key: SOLR-9877
> URL: https://issues.apache.org/jira/browse/SOLR-9877
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: metrics
>Reporter: Shalin Shekhar Mangar
>Assignee: Shalin Shekhar Mangar
>Priority: Minor
> Fix For: master (7.0), 6.4
>
> Attachments: SOLR-9877.patch, SOLR-9877.patch, 
> SOLR-9877_branch_6x.patch, solr-http-metrics.png
>
>
> Use instrumented equivalents of PooledHttpClientConnectionManager and others 
> from metrics-httpclient library.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-9896) Instrument and collect metrics from thread pools

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15797436#comment-15797436
 ] 

ASF subversion and git services commented on SOLR-9896:
---

Commit 084f7a0604da846c3ccc2c45d560d93403c883e5 in lucene-solr's branch 
refs/heads/branch_6x from [~shalinmangar]
[ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=084f7a0 ]

SOLR-9896: Instrument and collect metrics from query, update, core admin and 
core load thread pools

(cherry picked from commit 3c963967242aed73a906b7bc17c26a4b8b07083c)


> Instrument and collect metrics from thread pools
> 
>
> Key: SOLR-9896
> URL: https://issues.apache.org/jira/browse/SOLR-9896
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: metrics
>Reporter: Shalin Shekhar Mangar
>Assignee: Shalin Shekhar Mangar
> Fix For: master (7.0), 6.4
>
> Attachments: SOLR-9896.patch
>
>
> The metrics-core library has a InstrumentedExecutorService which collects 
> stats on submitted, running, completed tasks and durations. This issue will 
> expose such stats for all important thread pools in solr.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: 6.4 release

[Andrzej Białecki]

> On 4 Jan 2017, at 05:00, Walter Underwood  wrote:
> 
> If the metrics changes get in, I will be very, very happy. We’ve been running 
> with local hacks for four years.
> 


Most of the key instrumentation is already in place. Shalin and I are working 
now on cloud-related metrics (transaction log metrics, aggregated metrics from 
replicas and from shard leaders). There’s one improvement that would be good to 
put into 6.4: SOLR-9911 Add a way to filter metrics by prefix in the 
MetricsHandler API. This would help a lot in optimizing the retrieval of 
selected metrics, from among the hundreds that we track now.

Andrzej

> wunder
> Walter Underwood
> wun...@wunderwood.org 
> http://observer.wunderwood.org/  (my blog)
> 
> 
>> On Jan 3, 2017, at 9:25 AM, Michael McCandless > > wrote:
>> 
>> +1, lots of good stuff for 6.4 already!  Thanks for volunteering Jim :)
>> 
>> Mike McCandless
>> 
>> http://blog.mikemccandless.com 
>> 
>> 
>> On Tue, Jan 3, 2017 at 11:23 AM, jim ferenczi  wrote:
>>> Hi,
>>> I would like to volunteer to release 6.4. I can cut the release branch next
>>> Monday if everybody agrees.
>>> 
>>> Jim
>> 
>> -
>> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
>> For additional commands, e-mail: dev-h...@lucene.apache.org
>> 
> 

[jira] [Resolved] (SOLR-9915) PeerSync alreadyInSync check is not backwards compatible

[Noble Paul (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-9915?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Noble Paul resolved SOLR-9915.
--
   Resolution: Fixed
Fix Version/s: 6.4

> PeerSync alreadyInSync check is not backwards compatible
> 
>
> Key: SOLR-9915
> URL: https://issues.apache.org/jira/browse/SOLR-9915
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: replication (java)
>Affects Versions: 6.3
>Reporter: Tim Owen
>Assignee: Noble Paul
>Priority: Blocker
> Fix For: 6.4
>
> Attachments: SOLR-9915.patch
>
>
> The fingerprint check added to PeerSync in SOLR-9446 works fine when all 
> servers are running 6.3 but this means it's hard to do a rolling upgrade from 
> e.g. 6.2.1 to 6.3 because the 6.3 server sends a request to a 6.2.1 server to 
> get a fingerprint and then gets a NPE because the older server doesn't return 
> the expected field in its response.
> This leads to the PeerSync completely failing, and results in a full index 
> replication from scratch, copying all index files over the network. We 
> noticed this happening when we tried to do a rolling upgrade on one of our 
> 6.2.1 clusters to 6.3. Unfortunately this amount of replication was hammering 
> our disks and network, so we had to do a full shutdown, upgrade all to 6.3 
> and restart, which was not ideal for a production cluster.
> The attached patch should behave more gracefully in this situation, as it 
> will typically return false for alreadyInSync() and then carry on doing the 
> normal re-sync based on versions.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-master-MacOSX (64bit/jdk1.8.0) - Build # 3754 - Still Unstable!

[Policeman Jenkins Server]

Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-MacOSX/3754/
Java: 64bit/jdk1.8.0 -XX:-UseCompressedOops -XX:+UseConcMarkSweepGC

1 tests failed.
FAILED:  org.apache.solr.cloud.PeerSyncReplicationTest.test

Error Message:
timeout waiting to see all nodes active

Stack Trace:
java.lang.AssertionError: timeout waiting to see all nodes active
at 
__randomizedtesting.SeedInfo.seed([279F1AA02F8A725C:AFCB257A81761FA4]:0)
at org.junit.Assert.fail(Assert.java:93)
at 
org.apache.solr.cloud.PeerSyncReplicationTest.waitTillNodesActive(PeerSyncReplicationTest.java:311)
at 
org.apache.solr.cloud.PeerSyncReplicationTest.bringUpDeadNodeAndEnsureNoReplication(PeerSyncReplicationTest.java:262)
at 
org.apache.solr.cloud.PeerSyncReplicationTest.forceNodeFailureAndDoPeerSync(PeerSyncReplicationTest.java:244)
at 
org.apache.solr.cloud.PeerSyncReplicationTest.test(PeerSyncReplicationTest.java:133)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:985)
at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:960)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:811)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:462)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 

[jira] [Commented] (SOLR-9877) Use instrumented http client

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15797256#comment-15797256
 ] 

ASF subversion and git services commented on SOLR-9877:
---

Commit a50ebcb412b1a884b826b62418e9f5d8b3c1f40c in lucene-solr's branch 
refs/heads/branch_6x from [~shalinmangar]
[ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=a50ebcb ]

SOLR-9877: Use instrumented http client and connection pool


> Use instrumented http client
> 
>
> Key: SOLR-9877
> URL: https://issues.apache.org/jira/browse/SOLR-9877
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: metrics
>Reporter: Shalin Shekhar Mangar
>Assignee: Shalin Shekhar Mangar
>Priority: Minor
> Fix For: master (7.0), 6.4
>
> Attachments: SOLR-9877.patch, SOLR-9877.patch, 
> SOLR-9877_branch_6x.patch, solr-http-metrics.png
>
>
> Use instrumented equivalents of PooledHttpClientConnectionManager and others 
> from metrics-httpclient library.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-9877) Use instrumented http client

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15797257#comment-15797257
 ] 

ASF subversion and git services commented on SOLR-9877:
---

Commit f65dc06180bdb02cfbfa048e2f08d1183c250d5d in lucene-solr's branch 
refs/heads/branch_6x from [~shalinmangar]
[ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=f65dc06 ]

SOLR-9877: Null check for metric registry before attempting to use it

(cherry picked from commit 662be93)


> Use instrumented http client
> 
>
> Key: SOLR-9877
> URL: https://issues.apache.org/jira/browse/SOLR-9877
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: metrics
>Reporter: Shalin Shekhar Mangar
>Assignee: Shalin Shekhar Mangar
>Priority: Minor
> Fix For: master (7.0), 6.4
>
> Attachments: SOLR-9877.patch, SOLR-9877.patch, 
> SOLR-9877_branch_6x.patch, solr-http-metrics.png
>
>
> Use instrumented equivalents of PooledHttpClientConnectionManager and others 
> from metrics-httpclient library.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-9918) An UpdateRequestProcessor to skip duplicate inserts and ignore updates to missing docs

[David Smiley (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15797201#comment-15797201
 ] 

David Smiley commented on SOLR-9918:


Cool

> An UpdateRequestProcessor to skip duplicate inserts and ignore updates to 
> missing docs
> --
>
> Key: SOLR-9918
> URL: https://issues.apache.org/jira/browse/SOLR-9918
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: update
>Reporter: Tim Owen
> Attachments: SOLR-9918.patch
>
>
> This is an UpdateRequestProcessor and Factory that we have been using in 
> production, to handle 2 common cases that were awkward to achieve using the 
> existing update pipeline and current processor classes:
> * When inserting document(s), if some already exist then quietly skip the new 
> document inserts - do not churn the index by replacing the existing documents 
> and do not throw a noisy exception that breaks the batch of inserts. By 
> analogy with SQL, {{insert if not exists}}. In our use-case, multiple 
> application instances can (rarely) process the same input so it's easier for 
> us to de-dupe these at Solr insert time than to funnel them into a global 
> ordered queue first.
> * When applying AtomicUpdate documents, if a document being updated does not 
> exist, quietly do nothing - do not create a new partially-populated document 
> and do not throw a noisy exception about missing required fields. By analogy 
> with SQL, {{update where id = ..}}. Our use-case relies on this because we 
> apply updates optimistically and have best-effort knowledge about what 
> documents will exist, so it's easiest to skip the updates (in the same way a 
> Database would).
> I would have kept this in our own package hierarchy but it relies on some 
> package-scoped methods, and seems like it could be useful to others if they 
> choose to configure it. Some bits of the code were borrowed from 
> {{DocBasedVersionConstraintsProcessorFactory}}.
> Attached patch has unit tests to confirm the behaviour.
> This class can be used by configuring solrconfig.xml like so..
> {noformat}
>   
> 
>  class="org.apache.solr.update.processor.SkipExistingDocumentsProcessorFactory">
>   true
>   false 
> 
> 
> 
>   
> {noformat}
> and initParams defaults of
> {noformat}
>   skipexisting
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: 6.4 release

[Walter Underwood]

If the metrics changes get in, I will be very, very happy. We’ve been running 
with local hacks for four years.

wunder
Walter Underwood
wun...@wunderwood.org
http://observer.wunderwood.org/  (my blog)


> On Jan 3, 2017, at 9:25 AM, Michael McCandless  
> wrote:
> 
> +1, lots of good stuff for 6.4 already!  Thanks for volunteering Jim :)
> 
> Mike McCandless
> 
> http://blog.mikemccandless.com
> 
> 
> On Tue, Jan 3, 2017 at 11:23 AM, jim ferenczi  wrote:
>> Hi,
>> I would like to volunteer to release 6.4. I can cut the release branch next
>> Monday if everybody agrees.
>> 
>> Jim
> 
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org
> 

[jira] [Commented] (SOLR-9835) Create another replication mode for SolrCloud

[Noble Paul (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15796982#comment-15796982
 ] 

Noble Paul commented on SOLR-9835:
--

I would like this to be added to the description of the ticket

> Create another replication mode for SolrCloud
> -
>
> Key: SOLR-9835
> URL: https://issues.apache.org/jira/browse/SOLR-9835
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Cao Manh Dat
>Assignee: Shalin Shekhar Mangar
> Attachments: SOLR-9835.patch, SOLR-9835.patch, SOLR-9835.patch, 
> SOLR-9835.patch, SOLR-9835.patch
>
>
> The current replication mechanism of SolrCloud is called state machine, which 
> replicas start in same initial state and for each input, the input is 
> distributed across replicas so all replicas will end up with same next state. 
> But this type of replication have some drawbacks
> - The commit (which costly) have to run on all replicas
> - Slow recovery, because if replica miss more than N updates on its down 
> time, the replica have to download entire index from its leader.
> So we create create another replication mode for SolrCloud called state 
> transfer, which acts like master/slave replication. In basically
> - Leader distribute the update to other replicas, but the leader only apply 
> the update to IW, other replicas just store the update to UpdateLog (act like 
> replication).
> - Replicas frequently polling the latest segments from leader.
> Pros:
> - Lightweight for indexing, because only leader are running the commit, 
> updates.
> - Very fast recovery, replicas just have to download the missing segments.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Comment Edited] (SOLR-9835) Create another replication mode for SolrCloud

[Cao Manh Dat (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15796970#comment-15796970
 ] 

Cao Manh Dat edited comment on SOLR-9835 at 1/4/17 2:51 AM:


Right now, to create a collection in {{onlyLeaderIndexesMode}} we must use 
"Create Collection API" with additional parameter : onlyLeaderIndexes=true. 
When a collection is created on a mode, it will stick to that mode.

In further ticket, we can support switch between modes.


was (Author: caomanhdat):
Right now, to create a collection in {{onlyLeaderIndexesMode}} we must use 
"Create Collection API" with additional parameter : onlyLeaderIndexes=true. 
When a collection is created on a mode, it will stick to a mode.

In further ticket, we can support switch between modes.

> Create another replication mode for SolrCloud
> -
>
> Key: SOLR-9835
> URL: https://issues.apache.org/jira/browse/SOLR-9835
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Cao Manh Dat
>Assignee: Shalin Shekhar Mangar
> Attachments: SOLR-9835.patch, SOLR-9835.patch, SOLR-9835.patch, 
> SOLR-9835.patch, SOLR-9835.patch
>
>
> The current replication mechanism of SolrCloud is called state machine, which 
> replicas start in same initial state and for each input, the input is 
> distributed across replicas so all replicas will end up with same next state. 
> But this type of replication have some drawbacks
> - The commit (which costly) have to run on all replicas
> - Slow recovery, because if replica miss more than N updates on its down 
> time, the replica have to download entire index from its leader.
> So we create create another replication mode for SolrCloud called state 
> transfer, which acts like master/slave replication. In basically
> - Leader distribute the update to other replicas, but the leader only apply 
> the update to IW, other replicas just store the update to UpdateLog (act like 
> replication).
> - Replicas frequently polling the latest segments from leader.
> Pros:
> - Lightweight for indexing, because only leader are running the commit, 
> updates.
> - Very fast recovery, replicas just have to download the missing segments.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Comment Edited] (SOLR-9835) Create another replication mode for SolrCloud

[Cao Manh Dat (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15796970#comment-15796970
 ] 

Cao Manh Dat edited comment on SOLR-9835 at 1/4/17 2:51 AM:


Right now, to create a collection in {{onlyLeaderIndexesMode}} we must use 
"Create Collection API" with additional parameter : onlyLeaderIndexes=true. 
When a collection is created on a mode, it will stick to a mode.

In further ticket, we can support switch between modes.


was (Author: caomanhdat):
Right now, to create a collection in {{onlyLeaderIndexesMode}} we must use 
"Create Collection API" with additional parameter : onlyLeaderIndexes=true.

In further ticket, we can support switch between mode dynamically.

> Create another replication mode for SolrCloud
> -
>
> Key: SOLR-9835
> URL: https://issues.apache.org/jira/browse/SOLR-9835
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Cao Manh Dat
>Assignee: Shalin Shekhar Mangar
> Attachments: SOLR-9835.patch, SOLR-9835.patch, SOLR-9835.patch, 
> SOLR-9835.patch, SOLR-9835.patch
>
>
> The current replication mechanism of SolrCloud is called state machine, which 
> replicas start in same initial state and for each input, the input is 
> distributed across replicas so all replicas will end up with same next state. 
> But this type of replication have some drawbacks
> - The commit (which costly) have to run on all replicas
> - Slow recovery, because if replica miss more than N updates on its down 
> time, the replica have to download entire index from its leader.
> So we create create another replication mode for SolrCloud called state 
> transfer, which acts like master/slave replication. In basically
> - Leader distribute the update to other replicas, but the leader only apply 
> the update to IW, other replicas just store the update to UpdateLog (act like 
> replication).
> - Replicas frequently polling the latest segments from leader.
> Pros:
> - Lightweight for indexing, because only leader are running the commit, 
> updates.
> - Very fast recovery, replicas just have to download the missing segments.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-9835) Create another replication mode for SolrCloud

[Cao Manh Dat (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15796970#comment-15796970
 ] 

Cao Manh Dat commented on SOLR-9835:


Right now, to create a collection in {{onlyLeaderIndexesMode}} we must use 
"Create Collection API" with additional parameter : onlyLeaderIndexes=true.

In further ticket, we can support switch between mode dynamically.

> Create another replication mode for SolrCloud
> -
>
> Key: SOLR-9835
> URL: https://issues.apache.org/jira/browse/SOLR-9835
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Cao Manh Dat
>Assignee: Shalin Shekhar Mangar
> Attachments: SOLR-9835.patch, SOLR-9835.patch, SOLR-9835.patch, 
> SOLR-9835.patch, SOLR-9835.patch
>
>
> The current replication mechanism of SolrCloud is called state machine, which 
> replicas start in same initial state and for each input, the input is 
> distributed across replicas so all replicas will end up with same next state. 
> But this type of replication have some drawbacks
> - The commit (which costly) have to run on all replicas
> - Slow recovery, because if replica miss more than N updates on its down 
> time, the replica have to download entire index from its leader.
> So we create create another replication mode for SolrCloud called state 
> transfer, which acts like master/slave replication. In basically
> - Leader distribute the update to other replicas, but the leader only apply 
> the update to IW, other replicas just store the update to UpdateLog (act like 
> replication).
> - Replicas frequently polling the latest segments from leader.
> Pros:
> - Lightweight for indexing, because only leader are running the commit, 
> updates.
> - Very fast recovery, replicas just have to download the missing segments.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: Welcome Jim Ferenczi as a Lucene/Solr committer

[Koji Sekiguchi]

Congratulations and welcome Jim!

koji

On 2017/01/01 19:04, Michael McCandless wrote:

I'm pleased to announce that Jim Ferenczi has accepted the Lucene
PMC's invitation to become a committer.

Jim, it's tradition that you introduce yourself with a brief bio.

Your handle "jimczi" has been added to the “lucene" LDAP group, so you
now have commit privileges. Please test this by adding yourself to the
committers section of the Who We Are page on the website:
 (instructions here
).

The ASF dev page also has lots of useful links: .

Congratulations and welcome and Happy New Year,

Mike McCandless

http://blog.mikemccandless.com

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org






-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-9918) An UpdateRequestProcessor to skip duplicate inserts and ignore updates to missing docs

[Koji Sekiguchi (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15796928#comment-15796928
 ] 

Koji Sekiguchi commented on SOLR-9918:
--

I believe the proposal is very useful for users who need this function, but it 
is better for users if there is an additional explanation of the difference 
from the existing one that gives similar function.

How do users decide which UpdateRequestProcessor to use for their use cases as 
compared to SignatureUpdateProcessor?

> An UpdateRequestProcessor to skip duplicate inserts and ignore updates to 
> missing docs
> --
>
> Key: SOLR-9918
> URL: https://issues.apache.org/jira/browse/SOLR-9918
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: update
>Reporter: Tim Owen
> Attachments: SOLR-9918.patch
>
>
> This is an UpdateRequestProcessor and Factory that we have been using in 
> production, to handle 2 common cases that were awkward to achieve using the 
> existing update pipeline and current processor classes:
> * When inserting document(s), if some already exist then quietly skip the new 
> document inserts - do not churn the index by replacing the existing documents 
> and do not throw a noisy exception that breaks the batch of inserts. By 
> analogy with SQL, {{insert if not exists}}. In our use-case, multiple 
> application instances can (rarely) process the same input so it's easier for 
> us to de-dupe these at Solr insert time than to funnel them into a global 
> ordered queue first.
> * When applying AtomicUpdate documents, if a document being updated does not 
> exist, quietly do nothing - do not create a new partially-populated document 
> and do not throw a noisy exception about missing required fields. By analogy 
> with SQL, {{update where id = ..}}. Our use-case relies on this because we 
> apply updates optimistically and have best-effort knowledge about what 
> documents will exist, so it's easiest to skip the updates (in the same way a 
> Database would).
> I would have kept this in our own package hierarchy but it relies on some 
> package-scoped methods, and seems like it could be useful to others if they 
> choose to configure it. Some bits of the code were borrowed from 
> {{DocBasedVersionConstraintsProcessorFactory}}.
> Attached patch has unit tests to confirm the behaviour.
> This class can be used by configuring solrconfig.xml like so..
> {noformat}
>   
> 
>  class="org.apache.solr.update.processor.SkipExistingDocumentsProcessorFactory">
>   true
>   false 
> 
> 
> 
>   
> {noformat}
> and initParams defaults of
> {noformat}
>   skipexisting
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Resolved] (SOLR-8530) Add HavingStream to Streaming API and StreamingExpressions

[Joel Bernstein (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-8530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joel Bernstein resolved SOLR-8530.
--
Resolution: Implemented

> Add HavingStream to Streaming API and StreamingExpressions
> --
>
> Key: SOLR-8530
> URL: https://issues.apache.org/jira/browse/SOLR-8530
> Project: Solr
>  Issue Type: Improvement
>  Components: SolrJ
>Affects Versions: 6.0
>Reporter: Dennis Gove
>Priority: Minor
> Fix For: master (7.0), 6.4
>
> Attachments: SOLR-8530.patch
>
>
> The goal here is to support something similar to SQL's HAVING clause where 
> one can filter documents based on data that is not available in the index. 
> For example, filter the output of a reduce() based on the calculated 
> metrics.
> {code}
> having(
>   reduce(
> search(.),
> sum(cost),
> on=customerId
>   ),
>   q="sum(cost):[500 TO *]"
> )
> {code}
> This example would return all where the total spent by each distinct customer 
> is >= 500. The total spent is calculated via the sum(cost) metric in the 
> reduce stream.
> The intent is to support as the filters in the having(...) clause the full 
> query syntax of a search(...) clause. I see this being possible in one of two 
> ways. 
> 1. Use Lucene's MemoryIndex and as each tuple is read out of the underlying 
> stream creating an instance of MemoryIndex and apply the query to it. If the 
> result of that is >0 then the tuple should be returned from the HavingStream.
> 2. Create an in-memory solr index via something like RamDirectory, read all 
> tuples into that in-memory index using the UpdateStream, and then stream out 
> of that all the matching tuples from the query.
> There are benefits to each approach but I think the easiest and most direct 
> one is the MemoryIndex approach. With MemoryIndex it isn't necessary to read 
> all incoming tuples before returning a single tuple. With a MemoryIndex there 
> is a need to parse the solr query parameters and create a valid Lucene query 
> but I suspect that can be done using existing QParser implementations.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-8530) Add HavingStream to Streaming API and StreamingExpressions

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-8530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15796914#comment-15796914
 ] 

ASF subversion and git services commented on SOLR-8530:
---

Commit 93d1bba8f2194970ca736bee993cedea24e66b91 in lucene-solr's branch 
refs/heads/branch_6x from [~joel.bernstein]
[ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=93d1bba ]

SOLR-8530: Add support for single quoted aggregate HAVING comparisons


> Add HavingStream to Streaming API and StreamingExpressions
> --
>
> Key: SOLR-8530
> URL: https://issues.apache.org/jira/browse/SOLR-8530
> Project: Solr
>  Issue Type: Improvement
>  Components: SolrJ
>Affects Versions: 6.0
>Reporter: Dennis Gove
>Priority: Minor
> Fix For: master (7.0), 6.4
>
> Attachments: SOLR-8530.patch
>
>
> The goal here is to support something similar to SQL's HAVING clause where 
> one can filter documents based on data that is not available in the index. 
> For example, filter the output of a reduce() based on the calculated 
> metrics.
> {code}
> having(
>   reduce(
> search(.),
> sum(cost),
> on=customerId
>   ),
>   q="sum(cost):[500 TO *]"
> )
> {code}
> This example would return all where the total spent by each distinct customer 
> is >= 500. The total spent is calculated via the sum(cost) metric in the 
> reduce stream.
> The intent is to support as the filters in the having(...) clause the full 
> query syntax of a search(...) clause. I see this being possible in one of two 
> ways. 
> 1. Use Lucene's MemoryIndex and as each tuple is read out of the underlying 
> stream creating an instance of MemoryIndex and apply the query to it. If the 
> result of that is >0 then the tuple should be returned from the HavingStream.
> 2. Create an in-memory solr index via something like RamDirectory, read all 
> tuples into that in-memory index using the UpdateStream, and then stream out 
> of that all the matching tuples from the query.
> There are benefits to each approach but I think the easiest and most direct 
> one is the MemoryIndex approach. With MemoryIndex it isn't necessary to read 
> all incoming tuples before returning a single tuple. With a MemoryIndex there 
> is a need to parse the solr query parameters and create a valid Lucene query 
> but I suspect that can be done using existing QParser implementations.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-8530) Add HavingStream to Streaming API and StreamingExpressions

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-8530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15796890#comment-15796890
 ] 

ASF subversion and git services commented on SOLR-8530:
---

Commit ccdbb6ac0e0094985e5145c84b3cc2814ababf1d in lucene-solr's branch 
refs/heads/master from [~joel.bernstein]
[ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=ccdbb6a ]

SOLR-8530: Add support for single quoted aggregate HAVING comparisons


> Add HavingStream to Streaming API and StreamingExpressions
> --
>
> Key: SOLR-8530
> URL: https://issues.apache.org/jira/browse/SOLR-8530
> Project: Solr
>  Issue Type: Improvement
>  Components: SolrJ
>Affects Versions: 6.0
>Reporter: Dennis Gove
>Priority: Minor
> Fix For: master (7.0), 6.4
>
> Attachments: SOLR-8530.patch
>
>
> The goal here is to support something similar to SQL's HAVING clause where 
> one can filter documents based on data that is not available in the index. 
> For example, filter the output of a reduce() based on the calculated 
> metrics.
> {code}
> having(
>   reduce(
> search(.),
> sum(cost),
> on=customerId
>   ),
>   q="sum(cost):[500 TO *]"
> )
> {code}
> This example would return all where the total spent by each distinct customer 
> is >= 500. The total spent is calculated via the sum(cost) metric in the 
> reduce stream.
> The intent is to support as the filters in the having(...) clause the full 
> query syntax of a search(...) clause. I see this being possible in one of two 
> ways. 
> 1. Use Lucene's MemoryIndex and as each tuple is read out of the underlying 
> stream creating an instance of MemoryIndex and apply the query to it. If the 
> result of that is >0 then the tuple should be returned from the HavingStream.
> 2. Create an in-memory solr index via something like RamDirectory, read all 
> tuples into that in-memory index using the UpdateStream, and then stream out 
> of that all the matching tuples from the query.
> There are benefits to each approach but I think the easiest and most direct 
> one is the MemoryIndex approach. With MemoryIndex it isn't necessary to read 
> all incoming tuples before returning a single tuple. With a MemoryIndex there 
> is a need to parse the solr query parameters and create a valid Lucene query 
> but I suspect that can be done using existing QParser implementations.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-7115) UpdateLog can miss closing transaction log objects.

[Hoss Man (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-7115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15796887#comment-15796887
 ] 

Hoss Man commented on SOLR-7115:


just to clarify: when refering to SOLR-7912, i think folks actually ment 
SOLR-9712 ... correct?

> UpdateLog can miss closing transaction log objects.
> ---
>
> Key: SOLR-7115
> URL: https://issues.apache.org/jira/browse/SOLR-7115
> Project: Solr
>  Issue Type: Bug
>Reporter: Mark Miller
>Assignee: Yonik Seeley
> Fix For: 6.x, master (7.0)
>
> Attachments: SOLR-7115-LargeVolumeEmbeddedTest-fail.txt, 
> SOLR-7115.patch, SOLR-7115.patch, tests-failures-7115.txt
>
>
> I've seen this happen on YourKit and in various tests - especially since 
> adding resource release tracking to the log objects. Now I've got a test that 
> catches it in SOLR-7113.
> It seems that in precommit, if prevTlog is not null, we need to close it 
> because we are going to overwrite prevTlog with a new log.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS-EA] Lucene-Solr-6.x-Linux (64bit/jdk-9-ea+147) - Build # 2584 - Unstable!

[Policeman Jenkins Server]

Build: https://jenkins.thetaphi.de/job/Lucene-Solr-6.x-Linux/2584/
Java: 64bit/jdk-9-ea+147 -XX:-UseCompressedOops -XX:+UseParallelGC

1 tests failed.
FAILED:  org.apache.solr.cloud.ShardSplitTest.testSplitAfterFailedSplit

Error Message:
expected:<1> but was:<2>

Stack Trace:
java.lang.AssertionError: expected:<1> but was:<2>
at 
__randomizedtesting.SeedInfo.seed([281BA6645D06C4A:FBCC29C979A521C0]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.failNotEquals(Assert.java:647)
at org.junit.Assert.assertEquals(Assert.java:128)
at org.junit.Assert.assertEquals(Assert.java:472)
at org.junit.Assert.assertEquals(Assert.java:456)
at 
org.apache.solr.cloud.ShardSplitTest.testSplitAfterFailedSplit(ShardSplitTest.java:280)
at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:538)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:992)
at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:967)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:811)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:462)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 

[jira] [Moved] (SOLR-9920) error fixes included for did_you_mean.vm and richtext_doc.vm velocity examples

[Sergiu Dumitriu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-9920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergiu Dumitriu moved VELOCITY-879 to SOLR-9920:


Affects Version/s: (was: 1.7)
 Security: Public
  Component/s: (was: Scripting)
  Key: SOLR-9920  (was: VELOCITY-879)
  Project: Solr  (was: Velocity)

>  error fixes included for did_you_mean.vm and richtext_doc.vm velocity 
> examples
> ---
>
> Key: SOLR-9920
> URL: https://issues.apache.org/jira/browse/SOLR-9920
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
> Environment: openSUSE leap42.2, Java HotSpot(TM) 64-Bit Server VM 
> (build 25.112-b15, mixed mode), Solr 6.3.0 includes lucene 6.30 which 
> includes velocity
>Reporter: matthew grisius
>Priority: Trivial
> Attachments: jira-VELOCITY-979-bug-report.txt
>
>
> fixes included for velocity examples 1) did_you_mean.vm 2) and richtext_doc.vm
> 1) did_you_mean.vm - java ERROR in log get .size of null value
> #**
>  *  Hyperlinked spelling suggestions in results list
>  *#
> #set($collations = 
> $response.response.spellcheck.collations.getAll("collation"))
> ## make sure we get a non-null result before checking size . . .
> #if($collations && $collations.size() > 0)
>   Did you mean
>   #foreach($collation in $collations)
>  href="#{url_for_home}#{lensNoQ}=$esc.url($collation.collationQuery)">$esc.html($collation.collationQuery)
>  ($collation.hits)?
>   #end
> #end
> 2) richtext_doc.vm - Mime-Type/filetype mapping to reasonable display icon
> replace everything between line from: "## Sort out Mime-Type"
> to line "## Row 1: Icon and Title and mlt link" with:
> ... ## Sort out Mime-Type
> ## change 'content_type' to 'type', Nutch ootb default field per 
> https://wiki.apache.org/nutch/IndexStructure
> ## should change other Nutch/Solr config files too, e.g. schema.xml or 
> managed_schema . . .
> #set($ct = $list.get($doc.getFirstValue('type').split(";"),0))
> #set($filename = $doc.getFieldValue('resourcename'))
> #set($filetype = false)
> #set($filetype = $mimeExtensionsMap.get($ct))
> ## delete/update this block of comments to suit maintainer. . .
> ## TODO: falling back to file extension is convenient,
> ## except when you don't have an icon for that extension
> ## example 
> "application/vnd.openxmlformats-officedocument.wordprocessingml.document"
> ## document with a .docx extension.
> ## It'd be nice to fall back to an "unknown" or the existing "file" type
> ## We sort of do this below, but only if the filename has no extension
> ## (anything after the last dot).
> ## if we still don't have filetype...
> #if (!$filetype)
> ## get file extension if any
> #set ($dot = $url.lastIndexOf("."))
> #if ($dot > 0) ## ok, found a '.'
>   #set ($dot = $dot + 1) ## move past it
>   #set ($filetype = $url.substring($dot))
> #end
> ## still no filetype extension or no supported icon for extension
> #if (!$filetype || !$supportedMimeTypes.contains($filetype))
>   #set ($filetype = "file") ## use default for unknown
> #end
> ## could check for mere existence of a filetype file in img/ but that 
> would
> ## would be an expensive dir/file read, anyway user should config
> ## $supportedMimeTypes and $mimeExtensionsMap properly in 
> mime_type_lists.vm
> #end
> ... ## Row 1: Icon and Title and mlt link



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-8530) Add HavingStream to Streaming API and StreamingExpressions

[Joel Bernstein (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-8530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15796835#comment-15796835
 ] 

Joel Bernstein commented on SOLR-8530:
--

This is reopened to support having comparisons on aggregates.

> Add HavingStream to Streaming API and StreamingExpressions
> --
>
> Key: SOLR-8530
> URL: https://issues.apache.org/jira/browse/SOLR-8530
> Project: Solr
>  Issue Type: Improvement
>  Components: SolrJ
>Affects Versions: 6.0
>Reporter: Dennis Gove
>Priority: Minor
> Fix For: master (7.0), 6.4
>
> Attachments: SOLR-8530.patch
>
>
> The goal here is to support something similar to SQL's HAVING clause where 
> one can filter documents based on data that is not available in the index. 
> For example, filter the output of a reduce() based on the calculated 
> metrics.
> {code}
> having(
>   reduce(
> search(.),
> sum(cost),
> on=customerId
>   ),
>   q="sum(cost):[500 TO *]"
> )
> {code}
> This example would return all where the total spent by each distinct customer 
> is >= 500. The total spent is calculated via the sum(cost) metric in the 
> reduce stream.
> The intent is to support as the filters in the having(...) clause the full 
> query syntax of a search(...) clause. I see this being possible in one of two 
> ways. 
> 1. Use Lucene's MemoryIndex and as each tuple is read out of the underlying 
> stream creating an instance of MemoryIndex and apply the query to it. If the 
> result of that is >0 then the tuple should be returned from the HavingStream.
> 2. Create an in-memory solr index via something like RamDirectory, read all 
> tuples into that in-memory index using the UpdateStream, and then stream out 
> of that all the matching tuples from the query.
> There are benefits to each approach but I think the easiest and most direct 
> one is the MemoryIndex approach. With MemoryIndex it isn't necessary to read 
> all incoming tuples before returning a single tuple. With a MemoryIndex there 
> is a need to parse the solr query parameters and create a valid Lucene query 
> but I suspect that can be done using existing QParser implementations.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Reopened] (SOLR-8530) Add HavingStream to Streaming API and StreamingExpressions

[Joel Bernstein (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-8530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joel Bernstein reopened SOLR-8530:
--

> Add HavingStream to Streaming API and StreamingExpressions
> --
>
> Key: SOLR-8530
> URL: https://issues.apache.org/jira/browse/SOLR-8530
> Project: Solr
>  Issue Type: Improvement
>  Components: SolrJ
>Affects Versions: 6.0
>Reporter: Dennis Gove
>Priority: Minor
> Fix For: master (7.0), 6.4
>
> Attachments: SOLR-8530.patch
>
>
> The goal here is to support something similar to SQL's HAVING clause where 
> one can filter documents based on data that is not available in the index. 
> For example, filter the output of a reduce() based on the calculated 
> metrics.
> {code}
> having(
>   reduce(
> search(.),
> sum(cost),
> on=customerId
>   ),
>   q="sum(cost):[500 TO *]"
> )
> {code}
> This example would return all where the total spent by each distinct customer 
> is >= 500. The total spent is calculated via the sum(cost) metric in the 
> reduce stream.
> The intent is to support as the filters in the having(...) clause the full 
> query syntax of a search(...) clause. I see this being possible in one of two 
> ways. 
> 1. Use Lucene's MemoryIndex and as each tuple is read out of the underlying 
> stream creating an instance of MemoryIndex and apply the query to it. If the 
> result of that is >0 then the tuple should be returned from the HavingStream.
> 2. Create an in-memory solr index via something like RamDirectory, read all 
> tuples into that in-memory index using the UpdateStream, and then stream out 
> of that all the matching tuples from the query.
> There are benefits to each approach but I think the easiest and most direct 
> one is the MemoryIndex approach. With MemoryIndex it isn't necessary to read 
> all incoming tuples before returning a single tuple. With a MemoryIndex there 
> is a need to parse the solr query parameters and create a valid Lucene query 
> but I suspect that can be done using existing QParser implementations.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-9919) random Streaming Expression is not registered in /stream or /graph handler

[Joel Bernstein (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-9919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joel Bernstein updated SOLR-9919:
-
Description: 
The random streaming expression is not registered in the /stream or /graph 
handler, so cannot be used with these handlers.

This is a weakness of the test cases, which only exercise the /stream handler 
through the parallel test cases. Tests for functions that don't work with the 
parallel function, like the random function don't exercise the /stream handler. 
Typically scale testing catches unregistered functions but in this case it has 
fallen through the cracks.

To avoid this we should add a test that exercises the /stream handler for all 
functions that don't run in parallel.

  was:
The random streaming expression is not registered in the /stream or /graph 
handler, so cannot be used with these handlers.

This is a weakness of the test cases, which only exercise the /stream handler 
through the parallel test cases. Tests for functions that don't work with the 
parallel function, like the random function don't exercise the /stream handler. 
Typically scale testing catches unregistered functions but in this case it has 
fallen through the cracks.

To avoid this we should add a test the exercises the /stream handler for all 
functions that don't run in parallel.


> random Streaming Expression is not registered in /stream or /graph handler
> --
>
> Key: SOLR-9919
> URL: https://issues.apache.org/jira/browse/SOLR-9919
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Joel Bernstein
> Fix For: 6.4
>
>
> The random streaming expression is not registered in the /stream or /graph 
> handler, so cannot be used with these handlers.
> This is a weakness of the test cases, which only exercise the /stream handler 
> through the parallel test cases. Tests for functions that don't work with the 
> parallel function, like the random function don't exercise the /stream 
> handler. Typically scale testing catches unregistered functions but in this 
> case it has fallen through the cracks.
> To avoid this we should add a test that exercises the /stream handler for all 
> functions that don't run in parallel.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-9919) random Streaming Expression is not registered in /stream or /graph handler

[Joel Bernstein (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-9919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joel Bernstein updated SOLR-9919:
-
Description: 
The random streaming expression is not registered in the /stream or /graph 
handler, so cannot be used with these handlers.

This is a weakness of the test cases, which only exercise the /stream handler 
through the parallel test cases. Tests for functions that don't work with the 
parallel function, like the random function don't exercise the /stream handler. 
Typically scale testing catches unregistered functions but in this case it has 
fallen through the cracks.

To avoid this we should add a test the exercises the /stream handler for all 
functions that don't run in parallel.

  was:
The random streaming expression is not registered in the /stream or /graph 
handler, so cannot be used with these handlers.

This is a weakness of the test cases, which only exercise the /stream handler 
through the parallel test cases. Test for functions that don't work with the 
parallel function, like the random function don't exercise the /stream handler. 
Typically scale testing catches unregistered functions but in this case it has 
fallen through the cracks.

To avoid this we should add a test the exercises the /stream handler for all 
functions that don't run in parallel.


> random Streaming Expression is not registered in /stream or /graph handler
> --
>
> Key: SOLR-9919
> URL: https://issues.apache.org/jira/browse/SOLR-9919
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Joel Bernstein
> Fix For: 6.4
>
>
> The random streaming expression is not registered in the /stream or /graph 
> handler, so cannot be used with these handlers.
> This is a weakness of the test cases, which only exercise the /stream handler 
> through the parallel test cases. Tests for functions that don't work with the 
> parallel function, like the random function don't exercise the /stream 
> handler. Typically scale testing catches unregistered functions but in this 
> case it has fallen through the cracks.
> To avoid this we should add a test the exercises the /stream handler for all 
> functions that don't run in parallel.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Created] (SOLR-9919) random Streaming Expression is not registered in /stream or /graph handler

[Joel Bernstein (JIRA)]

Joel Bernstein created SOLR-9919:


 Summary: random Streaming Expression is not registered in /stream 
or /graph handler
 Key: SOLR-9919
 URL: https://issues.apache.org/jira/browse/SOLR-9919
 Project: Solr
  Issue Type: Bug
  Security Level: Public (Default Security Level. Issues are Public)
Reporter: Joel Bernstein
 Fix For: 6.4


The random streaming expression is not registered in the /stream or /graph 
handler, so cannot be used with these handlers.

This is a weakness of the test cases, which only exercise the /stream handler 
through the parallel test cases. Test for functions that don't work with the 
parallel function, like the random function don't exercise the /stream handler. 
Typically scale testing catches unregistered functions but in this case it has 
fallen through the cracks.

To avoid this we should add a test the exercises the /stream handler for all 
functions that don't run in parallel.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-6.x-Solaris (64bit/jdk1.8.0) - Build # 594 - Unstable!

[Policeman Jenkins Server]

Build: https://jenkins.thetaphi.de/job/Lucene-Solr-6.x-Solaris/594/
Java: 64bit/jdk1.8.0 -XX:-UseCompressedOops -XX:+UseConcMarkSweepGC

1 tests failed.
FAILED:  org.apache.solr.cloud.LeaderFailoverAfterPartitionTest.test

Error Message:
Expected 2 of 3 replicas to be active but only found 1; 
[core_node3:{"core":"c8n_1x3_lf_shard1_replica1","base_url":"http://127.0.0.1:60164/sp/wg","node_name":"127.0.0.1:60164_sp%2Fwg","state":"active","leader":"true"}];
 clusterState: DocCollection(c8n_1x3_lf//clusterstate.json/27)={   
"replicationFactor":"3",   "shards":{"shard1":{   
"range":"8000-7fff",   "state":"active",   "replicas":{ 
"core_node1":{   "core":"c8n_1x3_lf_shard1_replica3",   
"base_url":"http://127.0.0.1:51264/sp/wg;,   
"node_name":"127.0.0.1:51264_sp%2Fwg",   "state":"down"}, 
"core_node2":{   "state":"down",   
"base_url":"http://127.0.0.1:57200/sp/wg;,   
"core":"c8n_1x3_lf_shard1_replica2",   
"node_name":"127.0.0.1:57200_sp%2Fwg"}, "core_node3":{   
"core":"c8n_1x3_lf_shard1_replica1",   
"base_url":"http://127.0.0.1:60164/sp/wg;,   
"node_name":"127.0.0.1:60164_sp%2Fwg",   "state":"active",   
"leader":"true",   "router":{"name":"compositeId"},   
"maxShardsPerNode":"1",   "autoAddReplicas":"false"}

Stack Trace:
java.lang.AssertionError: Expected 2 of 3 replicas to be active but only found 
1; 
[core_node3:{"core":"c8n_1x3_lf_shard1_replica1","base_url":"http://127.0.0.1:60164/sp/wg","node_name":"127.0.0.1:60164_sp%2Fwg","state":"active","leader":"true"}];
 clusterState: DocCollection(c8n_1x3_lf//clusterstate.json/27)={
  "replicationFactor":"3",
  "shards":{"shard1":{
  "range":"8000-7fff",
  "state":"active",
  "replicas":{
"core_node1":{
  "core":"c8n_1x3_lf_shard1_replica3",
  "base_url":"http://127.0.0.1:51264/sp/wg;,
  "node_name":"127.0.0.1:51264_sp%2Fwg",
  "state":"down"},
"core_node2":{
  "state":"down",
  "base_url":"http://127.0.0.1:57200/sp/wg;,
  "core":"c8n_1x3_lf_shard1_replica2",
  "node_name":"127.0.0.1:57200_sp%2Fwg"},
"core_node3":{
  "core":"c8n_1x3_lf_shard1_replica1",
  "base_url":"http://127.0.0.1:60164/sp/wg;,
  "node_name":"127.0.0.1:60164_sp%2Fwg",
  "state":"active",
  "leader":"true",
  "router":{"name":"compositeId"},
  "maxShardsPerNode":"1",
  "autoAddReplicas":"false"}
at 
__randomizedtesting.SeedInfo.seed([B1E7954C79FE494E:39B3AA96D70224B6]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.assertTrue(Assert.java:43)
at 
org.apache.solr.cloud.LeaderFailoverAfterPartitionTest.testRf3WithLeaderFailover(LeaderFailoverAfterPartitionTest.java:168)
at 
org.apache.solr.cloud.LeaderFailoverAfterPartitionTest.test(LeaderFailoverAfterPartitionTest.java:55)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:992)
at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:967)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
at 

[jira] [Commented] (SOLR-9644) MoreLikeThis parsers SimpleMLTQParser and CloudMLTQParser don't handle boosts properly

[Anshum Gupta (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15796568#comment-15796568
 ] 

Anshum Gupta commented on SOLR-9644:


[~emaijala] can you update the patch ? I end up with a lot of conflicts when I 
try to apply this to the current master.

If you don't have the time right now, let me know and I'll take it up.

> MoreLikeThis parsers SimpleMLTQParser and CloudMLTQParser don't handle boosts 
> properly
> --
>
> Key: SOLR-9644
> URL: https://issues.apache.org/jira/browse/SOLR-9644
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: MoreLikeThis
>Affects Versions: 6.2.1
>Reporter: Ere Maijala
>  Labels: patch
>
> It seems SimpleMLTQParser and CloudMLTQParser should be able to handle boost 
> parameters, but it's not working properly. I'll make a pull request to add 
> tests and fix both.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-master-Solaris (64bit/jdk1.8.0) - Build # 1054 - Still Unstable!

[Policeman Jenkins Server]

Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-Solaris/1054/
Java: 64bit/jdk1.8.0 -XX:+UseCompressedOops -XX:+UseSerialGC

2 tests failed.
FAILED:  
org.apache.solr.cloud.TestSolrCloudWithDelegationTokens.testDelegationTokenCancelFail

Error Message:
expected:<200> but was:<404>

Stack Trace:
java.lang.AssertionError: expected:<200> but was:<404>
at 
__randomizedtesting.SeedInfo.seed([50577D0E7D517A41:38E84824ADCB68AD]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.failNotEquals(Assert.java:647)
at org.junit.Assert.assertEquals(Assert.java:128)
at org.junit.Assert.assertEquals(Assert.java:472)
at org.junit.Assert.assertEquals(Assert.java:456)
at 
org.apache.solr.cloud.TestSolrCloudWithDelegationTokens.cancelDelegationToken(TestSolrCloudWithDelegationTokens.java:140)
at 
org.apache.solr.cloud.TestSolrCloudWithDelegationTokens.testDelegationTokenCancelFail(TestSolrCloudWithDelegationTokens.java:294)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:811)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:462)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 

Re: Installing PyLucene

[Andi Vajda]

On Tue, 3 Jan 2017, marco turchi wrote:


Nope different linux kernels and python versions (2.7.11 and 2.7.12).

Btw I have also similar problem on the machine with Web access.


I suspect you don't have shared mode enabled when running tests.
The WindowsError exception is a bug that no one hits until the tests fails 
to remove its STORE_DIR tree. I need to remove this...

You can remove that try/except bracket since it doesn't apply to you anyway.

Usually, errors with test_PythonException.py have to do with missing shared 
mode. If PyLucene is otherwise working for you, you can ignore this or make 
sure shared mode is actually deployed with PyLucene.


Andi..



Best,
Marco

On Tue, Jan 3, 2017 at 10:05 PM, Andi Vajda  wrote:



On Tue, 3 Jan 2017, marco turchi wrote:

Dear Andi,

following you suggestions, I have first installed pyLucene on a machine
with access to the Web and than I have copied the tree on the cluster
machine where I have installed pyLucene.

Running the make test I have the following errors:
ERROR: testThroughLayerException (__main__.PythonExceptionTestCase)
--
Traceback (most recent call last):
 File "test/test_PythonException.py", line 34, in
testThroughLayerException
   qp.parse("foo bar")
JavaError: , >
   Java stacktrace:
java.lang.RuntimeException: TestException
at
org.apache.pylucene.queryparser.classic.PythonQueryParser.ge
tFieldQuery_quoted(Native
Method)
at
org.apache.pylucene.queryparser.classic.PythonQueryParser.ge
tFieldQuery(Unknown
Source
at
org.apache.lucene.queryparser.classic.QueryParser.MultiTerm(
QueryParser.java:585)
at
org.apache.lucene.queryparser.classic.QueryParser.Query(Quer
yParser.java:198)
at
org.apache.lucene.queryparser.classic.QueryParser.TopLevelQu
ery(QueryParser.java:187)
at
org.apache.lucene.queryparser.classic.QueryParserBase.parse(
QueryParserBase.java:111)

and:
NameError: global name 'WindowsError' is not defined
in
ERROR: test_FieldEnumeration (__main__.Test_PyLuceneWithFSStore)
ERROR: test_removeDocuments (__main__.Test_PyLuceneWithFSStore)
ERROR: test_searchDocuments (__main__.Test_PyLuceneWithFSStore)
...



Are both machines (the one on the web and the one offline) running the
samev version of Windows and Python ?

Andi..


I have checked on the mailing list but I have not found solutions that work

for me. I'm using:
python 2.7.12 (I have also tried python 3.5.2 but I have problems to
compile jcc)
java jdk1.8.0_60

do you have an idea of what it is not working?

Thanks a lot in advance for your help!
Marco




On Sat, Dec 31, 2016 at 4:09 AM, Andi Vajda  wrote:



On Dec 30, 2016, at 15:07, marco turchi  wrote:


Dear Andi,
thanks a lot for you answers!


You do not need root privileges if you don't modify the system python.



One



way to achieve that is to setup a python virtualenv first and install



jcc



and pylucene into it instead of the system python.



Do you mean to install a new version of python in one of my folders and



us


it for installing JCC and pyLucene?



No, I mean to setup a python virtualenv.

Andi..





I'm using a

version of python (2.7.5) available in anaconda and our cluster is not
connected to the WEB, so I cannot use setuptools.



You can use setuptools without a web connection, why not ?



Sorry, you are right I thought that setuptools needs to be connected to


the


Web to download the required libraries





Ah, here, to build Java Lucene, ivy is required and without a web
connection, it's going to be more difficult. You need to somehow make


sure



that all things ivy is going to download during the Lucene build (a one

time setup only) are already there when you build Lucene.
You could do this on an equivalent machine that has a web connection
and
then copy the local ivy tree to the machine that doesn't.



This is a great suggestion, thanks a lot! I'm going to try this in the


next


days!!

Best,
Marco




Andi..



resolve:

Am I doing anything wrong? do you have any suggestions to help me to
proceed with the installation?

Thanks a lot in advance for your help!

Best Regards,
Marco

[jira] [Commented] (SOLR-9901) Implement move in HdfsDirectoryFactory.

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15796442#comment-15796442
 ] 

ASF subversion and git services commented on SOLR-9901:
---

Commit 90902a610ba724c7f02096b167cfb921e383789c in lucene-solr's branch 
refs/heads/branch_6x from markrmiller
[ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=90902a6 ]

SOLR-9901: Implement move in HdfsDirectoryFactory.


> Implement move in HdfsDirectoryFactory.
> ---
>
> Key: SOLR-9901
> URL: https://issues.apache.org/jira/browse/SOLR-9901
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Mark Miller
>Assignee: Mark Miller
> Fix For: master (7.0), 6.4
>
> Attachments: SOLR-9901.patch
>
>
> Without this, you can end up with things like a 0 bytes segment file.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Resolved] (SOLR-8530) Add HavingStream to Streaming API and StreamingExpressions

[Joel Bernstein (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-8530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joel Bernstein resolved SOLR-8530.
--
   Resolution: Implemented
Fix Version/s: 6.4
   master (7.0)

> Add HavingStream to Streaming API and StreamingExpressions
> --
>
> Key: SOLR-8530
> URL: https://issues.apache.org/jira/browse/SOLR-8530
> Project: Solr
>  Issue Type: Improvement
>  Components: SolrJ
>Affects Versions: 6.0
>Reporter: Dennis Gove
>Priority: Minor
> Fix For: master (7.0), 6.4
>
> Attachments: SOLR-8530.patch
>
>
> The goal here is to support something similar to SQL's HAVING clause where 
> one can filter documents based on data that is not available in the index. 
> For example, filter the output of a reduce() based on the calculated 
> metrics.
> {code}
> having(
>   reduce(
> search(.),
> sum(cost),
> on=customerId
>   ),
>   q="sum(cost):[500 TO *]"
> )
> {code}
> This example would return all where the total spent by each distinct customer 
> is >= 500. The total spent is calculated via the sum(cost) metric in the 
> reduce stream.
> The intent is to support as the filters in the having(...) clause the full 
> query syntax of a search(...) clause. I see this being possible in one of two 
> ways. 
> 1. Use Lucene's MemoryIndex and as each tuple is read out of the underlying 
> stream creating an instance of MemoryIndex and apply the query to it. If the 
> result of that is >0 then the tuple should be returned from the HavingStream.
> 2. Create an in-memory solr index via something like RamDirectory, read all 
> tuples into that in-memory index using the UpdateStream, and then stream out 
> of that all the matching tuples from the query.
> There are benefits to each approach but I think the easiest and most direct 
> one is the MemoryIndex approach. With MemoryIndex it isn't necessary to read 
> all incoming tuples before returning a single tuple. With a MemoryIndex there 
> is a need to parse the solr query parameters and create a valid Lucene query 
> but I suspect that can be done using existing QParser implementations.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-9902) StandardDirectoryFactory should use Files API for it's move implementation.

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15796420#comment-15796420
 ] 

ASF subversion and git services commented on SOLR-9902:
---

Commit cbf96e0d7572a28141226d92f978b8aef8bd2509 in lucene-solr's branch 
refs/heads/branch_6x from markrmiller
[ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=cbf96e0 ]

SOLR-9902: StandardDirectoryFactory should use Files API for it's move 
implementation.


> StandardDirectoryFactory should use Files API for it's move implementation.
> ---
>
> Key: SOLR-9902
> URL: https://issues.apache.org/jira/browse/SOLR-9902
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Mark Miller
>Assignee: Mark Miller
> Fix For: master (7.0), 6.4
>
> Attachments: SOLR-9902.patch
>
>
> It's done in a platform independent way as opposed to the old File API.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: [JENKINS] Lucene-Solr-6.x-MacOSX (64bit/jdk1.8.0) - Build # 619 - Failure!

[Joel Bernstein]

A fix has been pushed.

Joel Bernstein
http://joelsolr.blogspot.com/

On Tue, Jan 3, 2017 at 5:22 PM, Joel Bernstein  wrote:

> I ran into this while running precommit on branch_6x. I'll be pushing a
> fix along with my commit.
>
> Joel Bernstein
> http://joelsolr.blogspot.com/
>
> On Tue, Jan 3, 2017 at 4:45 PM, Policeman Jenkins Server <
> jenk...@thetaphi.de> wrote:
>
>> Build: https://jenkins.thetaphi.de/job/Lucene-Solr-6.x-MacOSX/619/
>> Java: 64bit/jdk1.8.0 -XX:-UseCompressedOops -XX:+UseConcMarkSweepGC
>>
>> All tests passed
>>
>> Build Log:
>> [...truncated 68484 lines...]
>> -ecj-javadoc-lint-src:
>> [mkdir] Created dir: /var/folders/qg/h2dfw5s161s51l
>> 2bn79mrb7rgn/T/ecj2025476677
>>  [ecj-lint] Compiling 49 source files to /var/folders/qg/h2dfw5s161s51l
>> 2bn79mrb7rgn/T/ecj2025476677
>>  [ecj-lint] invalid Class-Path header in manifest of jar file:
>> /Users/jenkins/.ivy2/cache/org.restlet.jee/org.restlet/jars/
>> org.restlet-2.3.0.jar
>>  [ecj-lint] invalid Class-Path header in manifest of jar file:
>> /Users/jenkins/.ivy2/cache/org.restlet.jee/org.restlet.ext.
>> servlet/jars/org.restlet.ext.servlet-2.3.0.jar
>>  [ecj-lint] --
>>  [ecj-lint] 1. ERROR in /Users/jenkins/workspace/Lucen
>> e-Solr-6.x-MacOSX/solr/test-framework/src/java/org/apache/
>> solr/cloud/AbstractDistribZkTestBase.java (at line 22)
>>  [ecj-lint] import java.util.concurrent.TimeUnit;
>>  [ecj-lint]^
>>  [ecj-lint] The import java.util.concurrent.TimeUnit is never used
>>  [ecj-lint] --
>>  [ecj-lint] 1 problem (1 error)
>>
>> BUILD FAILED
>> /Users/jenkins/workspace/Lucene-Solr-6.x-MacOSX/build.xml:775: The
>> following error occurred while executing this line:
>> /Users/jenkins/workspace/Lucene-Solr-6.x-MacOSX/build.xml:101: The
>> following error occurred while executing this line:
>> /Users/jenkins/workspace/Lucene-Solr-6.x-MacOSX/solr/build.xml:671: The
>> following error occurred while executing this line:
>> /Users/jenkins/workspace/Lucene-Solr-6.x-MacOSX/lucene/common-build.xml:1992:
>> The following error occurred while executing this line:
>> /Users/jenkins/workspace/Lucene-Solr-6.x-MacOSX/lucene/common-build.xml:2031:
>> Compile failed; see the compiler error output for details.
>>
>> Total time: 101 minutes 46 seconds
>> Build step 'Invoke Ant' marked build as failure
>> Archiving artifacts
>> [WARNINGS] Skipping publisher since build result is FAILURE
>> Recording test results
>> Email was triggered for: Failure - Any
>> Sending email for trigger: Failure - Any
>>
>>
>>
>>
>> -
>> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
>> For additional commands, e-mail: dev-h...@lucene.apache.org
>>
>
>

[jira] [Commented] (SOLR-8530) Add HavingStream to Streaming API and StreamingExpressions

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-8530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15796408#comment-15796408
 ] 

ASF subversion and git services commented on SOLR-8530:
---

Commit 297b1789092f4f9b3a2cfb91da397e5034708486 in lucene-solr's branch 
refs/heads/branch_6x from [~joel.bernstein]
[ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=297b178 ]

SOLR-8530: Add tests from the HavingStream


> Add HavingStream to Streaming API and StreamingExpressions
> --
>
> Key: SOLR-8530
> URL: https://issues.apache.org/jira/browse/SOLR-8530
> Project: Solr
>  Issue Type: Improvement
>  Components: SolrJ
>Affects Versions: 6.0
>Reporter: Dennis Gove
>Priority: Minor
> Attachments: SOLR-8530.patch
>
>
> The goal here is to support something similar to SQL's HAVING clause where 
> one can filter documents based on data that is not available in the index. 
> For example, filter the output of a reduce() based on the calculated 
> metrics.
> {code}
> having(
>   reduce(
> search(.),
> sum(cost),
> on=customerId
>   ),
>   q="sum(cost):[500 TO *]"
> )
> {code}
> This example would return all where the total spent by each distinct customer 
> is >= 500. The total spent is calculated via the sum(cost) metric in the 
> reduce stream.
> The intent is to support as the filters in the having(...) clause the full 
> query syntax of a search(...) clause. I see this being possible in one of two 
> ways. 
> 1. Use Lucene's MemoryIndex and as each tuple is read out of the underlying 
> stream creating an instance of MemoryIndex and apply the query to it. If the 
> result of that is >0 then the tuple should be returned from the HavingStream.
> 2. Create an in-memory solr index via something like RamDirectory, read all 
> tuples into that in-memory index using the UpdateStream, and then stream out 
> of that all the matching tuples from the query.
> There are benefits to each approach but I think the easiest and most direct 
> one is the MemoryIndex approach. With MemoryIndex it isn't necessary to read 
> all incoming tuples before returning a single tuple. With a MemoryIndex there 
> is a need to parse the solr query parameters and create a valid Lucene query 
> but I suspect that can be done using existing QParser implementations.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-8530) Add HavingStream to Streaming API and StreamingExpressions

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-8530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15796410#comment-15796410
 ] 

ASF subversion and git services commented on SOLR-8530:
---

Commit d8a58146c3155a13f9bb8c46eb2d2878301426d3 in lucene-solr's branch 
refs/heads/branch_6x from [~joel.bernstein]
[ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=d8a5814 ]

SOLR-8530: Updated CHANGES.txt


> Add HavingStream to Streaming API and StreamingExpressions
> --
>
> Key: SOLR-8530
> URL: https://issues.apache.org/jira/browse/SOLR-8530
> Project: Solr
>  Issue Type: Improvement
>  Components: SolrJ
>Affects Versions: 6.0
>Reporter: Dennis Gove
>Priority: Minor
> Attachments: SOLR-8530.patch
>
>
> The goal here is to support something similar to SQL's HAVING clause where 
> one can filter documents based on data that is not available in the index. 
> For example, filter the output of a reduce() based on the calculated 
> metrics.
> {code}
> having(
>   reduce(
> search(.),
> sum(cost),
> on=customerId
>   ),
>   q="sum(cost):[500 TO *]"
> )
> {code}
> This example would return all where the total spent by each distinct customer 
> is >= 500. The total spent is calculated via the sum(cost) metric in the 
> reduce stream.
> The intent is to support as the filters in the having(...) clause the full 
> query syntax of a search(...) clause. I see this being possible in one of two 
> ways. 
> 1. Use Lucene's MemoryIndex and as each tuple is read out of the underlying 
> stream creating an instance of MemoryIndex and apply the query to it. If the 
> result of that is >0 then the tuple should be returned from the HavingStream.
> 2. Create an in-memory solr index via something like RamDirectory, read all 
> tuples into that in-memory index using the UpdateStream, and then stream out 
> of that all the matching tuples from the query.
> There are benefits to each approach but I think the easiest and most direct 
> one is the MemoryIndex approach. With MemoryIndex it isn't necessary to read 
> all incoming tuples before returning a single tuple. With a MemoryIndex there 
> is a need to parse the solr query parameters and create a valid Lucene query 
> but I suspect that can be done using existing QParser implementations.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-8530) Add HavingStream to Streaming API and StreamingExpressions

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-8530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15796407#comment-15796407
 ] 

ASF subversion and git services commented on SOLR-8530:
---

Commit 2f7d6fc0fa3de7e2f1d09823d9ef4c6ee08e9d44 in lucene-solr's branch 
refs/heads/branch_6x from [~joel.bernstein]
[ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=2f7d6fc ]

SOLR-8530: Add HavingStream to Streaming API and StreamingExpressions


> Add HavingStream to Streaming API and StreamingExpressions
> --
>
> Key: SOLR-8530
> URL: https://issues.apache.org/jira/browse/SOLR-8530
> Project: Solr
>  Issue Type: Improvement
>  Components: SolrJ
>Affects Versions: 6.0
>Reporter: Dennis Gove
>Priority: Minor
> Attachments: SOLR-8530.patch
>
>
> The goal here is to support something similar to SQL's HAVING clause where 
> one can filter documents based on data that is not available in the index. 
> For example, filter the output of a reduce() based on the calculated 
> metrics.
> {code}
> having(
>   reduce(
> search(.),
> sum(cost),
> on=customerId
>   ),
>   q="sum(cost):[500 TO *]"
> )
> {code}
> This example would return all where the total spent by each distinct customer 
> is >= 500. The total spent is calculated via the sum(cost) metric in the 
> reduce stream.
> The intent is to support as the filters in the having(...) clause the full 
> query syntax of a search(...) clause. I see this being possible in one of two 
> ways. 
> 1. Use Lucene's MemoryIndex and as each tuple is read out of the underlying 
> stream creating an instance of MemoryIndex and apply the query to it. If the 
> result of that is >0 then the tuple should be returned from the HavingStream.
> 2. Create an in-memory solr index via something like RamDirectory, read all 
> tuples into that in-memory index using the UpdateStream, and then stream out 
> of that all the matching tuples from the query.
> There are benefits to each approach but I think the easiest and most direct 
> one is the MemoryIndex approach. With MemoryIndex it isn't necessary to read 
> all incoming tuples before returning a single tuple. With a MemoryIndex there 
> is a need to parse the solr query parameters and create a valid Lucene query 
> but I suspect that can be done using existing QParser implementations.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-8530) Add HavingStream to Streaming API and StreamingExpressions

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-8530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15796409#comment-15796409
 ] 

ASF subversion and git services commented on SOLR-8530:
---

Commit 00af5fff4d096000b0cde9066a599f68076c1862 in lucene-solr's branch 
refs/heads/branch_6x from [~joel.bernstein]
[ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=00af5ff ]

SOLR-8530: Fixed javadoc


> Add HavingStream to Streaming API and StreamingExpressions
> --
>
> Key: SOLR-8530
> URL: https://issues.apache.org/jira/browse/SOLR-8530
> Project: Solr
>  Issue Type: Improvement
>  Components: SolrJ
>Affects Versions: 6.0
>Reporter: Dennis Gove
>Priority: Minor
> Attachments: SOLR-8530.patch
>
>
> The goal here is to support something similar to SQL's HAVING clause where 
> one can filter documents based on data that is not available in the index. 
> For example, filter the output of a reduce() based on the calculated 
> metrics.
> {code}
> having(
>   reduce(
> search(.),
> sum(cost),
> on=customerId
>   ),
>   q="sum(cost):[500 TO *]"
> )
> {code}
> This example would return all where the total spent by each distinct customer 
> is >= 500. The total spent is calculated via the sum(cost) metric in the 
> reduce stream.
> The intent is to support as the filters in the having(...) clause the full 
> query syntax of a search(...) clause. I see this being possible in one of two 
> ways. 
> 1. Use Lucene's MemoryIndex and as each tuple is read out of the underlying 
> stream creating an instance of MemoryIndex and apply the query to it. If the 
> result of that is >0 then the tuple should be returned from the HavingStream.
> 2. Create an in-memory solr index via something like RamDirectory, read all 
> tuples into that in-memory index using the UpdateStream, and then stream out 
> of that all the matching tuples from the query.
> There are benefits to each approach but I think the easiest and most direct 
> one is the MemoryIndex approach. With MemoryIndex it isn't necessary to read 
> all incoming tuples before returning a single tuple. With a MemoryIndex there 
> is a need to parse the solr query parameters and create a valid Lucene query 
> but I suspect that can be done using existing QParser implementations.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: [JENKINS] Lucene-Solr-6.x-MacOSX (64bit/jdk1.8.0) - Build # 619 - Failure!

[Joel Bernstein]

I ran into this while running precommit on branch_6x. I'll be pushing a fix
along with my commit.

Joel Bernstein
http://joelsolr.blogspot.com/

On Tue, Jan 3, 2017 at 4:45 PM, Policeman Jenkins Server <
jenk...@thetaphi.de> wrote:

> Build: https://jenkins.thetaphi.de/job/Lucene-Solr-6.x-MacOSX/619/
> Java: 64bit/jdk1.8.0 -XX:-UseCompressedOops -XX:+UseConcMarkSweepGC
>
> All tests passed
>
> Build Log:
> [...truncated 68484 lines...]
> -ecj-javadoc-lint-src:
> [mkdir] Created dir: /var/folders/qg/h2dfw5s161s51l2bn79mrb7rgn
> /T/ecj2025476677
>  [ecj-lint] Compiling 49 source files to /var/folders/qg/
> h2dfw5s161s51l2bn79mrb7rgn/T/ecj2025476677
>  [ecj-lint] invalid Class-Path header in manifest of jar file:
> /Users/jenkins/.ivy2/cache/org.restlet.jee/org.restlet/
> jars/org.restlet-2.3.0.jar
>  [ecj-lint] invalid Class-Path header in manifest of jar file:
> /Users/jenkins/.ivy2/cache/org.restlet.jee/org.restlet.
> ext.servlet/jars/org.restlet.ext.servlet-2.3.0.jar
>  [ecj-lint] --
>  [ecj-lint] 1. ERROR in /Users/jenkins/workspace/
> Lucene-Solr-6.x-MacOSX/solr/test-framework/src/java/org/apache/solr/cloud/AbstractDistribZkTestBase.java
> (at line 22)
>  [ecj-lint] import java.util.concurrent.TimeUnit;
>  [ecj-lint]^
>  [ecj-lint] The import java.util.concurrent.TimeUnit is never used
>  [ecj-lint] --
>  [ecj-lint] 1 problem (1 error)
>
> BUILD FAILED
> /Users/jenkins/workspace/Lucene-Solr-6.x-MacOSX/build.xml:775: The
> following error occurred while executing this line:
> /Users/jenkins/workspace/Lucene-Solr-6.x-MacOSX/build.xml:101: The
> following error occurred while executing this line:
> /Users/jenkins/workspace/Lucene-Solr-6.x-MacOSX/solr/build.xml:671: The
> following error occurred while executing this line:
> /Users/jenkins/workspace/Lucene-Solr-6.x-MacOSX/lucene/common-build.xml:1992:
> The following error occurred while executing this line:
> /Users/jenkins/workspace/Lucene-Solr-6.x-MacOSX/lucene/common-build.xml:2031:
> Compile failed; see the compiler error output for details.
>
> Total time: 101 minutes 46 seconds
> Build step 'Invoke Ant' marked build as failure
> Archiving artifacts
> [WARNINGS] Skipping publisher since build result is FAILURE
> Recording test results
> Email was triggered for: Failure - Any
> Sending email for trigger: Failure - Any
>
>
>
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org
>

[jira] [Commented] (SOLR-9917) NPE in JSON facet merger

[Yonik Seeley (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15796375#comment-15796375
 ] 

Yonik Seeley commented on SOLR-9917:


Thanks for the extra info, I've now reproduced it and am working on a fix.

> NPE in JSON facet merger
> 
>
> Key: SOLR-9917
> URL: https://issues.apache.org/jira/browse/SOLR-9917
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>Affects Versions: 6.3
>Reporter: Markus Jelsma
>Assignee: Yonik Seeley
> Fix For: master (7.0), 6.4
>
> Attachments: SOLR-9917.patch
>
>
> I've spotted this before, and just now one of my unit tests does it as well. 
> I believe this happens when there is no data for the requested field.
> {code}
> java.lang.NullPointerException
> at java.nio.ByteBuffer.wrap(ByteBuffer.java:396)
> at 
> org.apache.solr.search.facet.PercentileAgg$Merger.merge(PercentileAgg.java:195)
> at 
> org.apache.solr.search.facet.FacetBucket.mergeBucket(FacetBucket.java:90)
> at 
> org.apache.solr.search.facet.FacetRequestSortedMerger.mergeBucketList(FacetRequestSortedMerger.java:61)
> at 
> org.apache.solr.search.facet.FacetRangeMerger.mergeBucketList(FacetRangeMerger.java:27)
> at 
> org.apache.solr.search.facet.FacetRangeMerger.merge(FacetRangeMerger.java:91)
> at 
> org.apache.solr.search.facet.FacetRangeMerger.merge(FacetRangeMerger.java:43)
> at 
> org.apache.solr.search.facet.FacetBucket.mergeBucket(FacetBucket.java:90)
> at 
> org.apache.solr.search.facet.FacetQueryMerger.merge(FacetModule.java:444)
> at 
> org.apache.solr.search.facet.FacetModule.handleResponses(FacetModule.java:272)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-8530) Add HavingStream to Streaming API and StreamingExpressions

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-8530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15796350#comment-15796350
 ] 

ASF subversion and git services commented on SOLR-8530:
---

Commit db7d2ff1629e7ae45a405eebdcdde1c68664d01f in lucene-solr's branch 
refs/heads/master from [~joel.bernstein]
[ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=db7d2ff ]

SOLR-8530: Updated CHANGES.txt


> Add HavingStream to Streaming API and StreamingExpressions
> --
>
> Key: SOLR-8530
> URL: https://issues.apache.org/jira/browse/SOLR-8530
> Project: Solr
>  Issue Type: Improvement
>  Components: SolrJ
>Affects Versions: 6.0
>Reporter: Dennis Gove
>Priority: Minor
> Attachments: SOLR-8530.patch
>
>
> The goal here is to support something similar to SQL's HAVING clause where 
> one can filter documents based on data that is not available in the index. 
> For example, filter the output of a reduce() based on the calculated 
> metrics.
> {code}
> having(
>   reduce(
> search(.),
> sum(cost),
> on=customerId
>   ),
>   q="sum(cost):[500 TO *]"
> )
> {code}
> This example would return all where the total spent by each distinct customer 
> is >= 500. The total spent is calculated via the sum(cost) metric in the 
> reduce stream.
> The intent is to support as the filters in the having(...) clause the full 
> query syntax of a search(...) clause. I see this being possible in one of two 
> ways. 
> 1. Use Lucene's MemoryIndex and as each tuple is read out of the underlying 
> stream creating an instance of MemoryIndex and apply the query to it. If the 
> result of that is >0 then the tuple should be returned from the HavingStream.
> 2. Create an in-memory solr index via something like RamDirectory, read all 
> tuples into that in-memory index using the UpdateStream, and then stream out 
> of that all the matching tuples from the query.
> There are benefits to each approach but I think the easiest and most direct 
> one is the MemoryIndex approach. With MemoryIndex it isn't necessary to read 
> all incoming tuples before returning a single tuple. With a MemoryIndex there 
> is a need to parse the solr query parameters and create a valid Lucene query 
> but I suspect that can be done using existing QParser implementations.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-8530) Add HavingStream to Streaming API and StreamingExpressions

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-8530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15796347#comment-15796347
 ] 

ASF subversion and git services commented on SOLR-8530:
---

Commit 1da283ef2c673b2effac834da1de1cb94c0118bb in lucene-solr's branch 
refs/heads/master from [~joel.bernstein]
[ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=1da283e ]

SOLR-8530: Add HavingStream to Streaming API and StreamingExpressions


> Add HavingStream to Streaming API and StreamingExpressions
> --
>
> Key: SOLR-8530
> URL: https://issues.apache.org/jira/browse/SOLR-8530
> Project: Solr
>  Issue Type: Improvement
>  Components: SolrJ
>Affects Versions: 6.0
>Reporter: Dennis Gove
>Priority: Minor
> Attachments: SOLR-8530.patch
>
>
> The goal here is to support something similar to SQL's HAVING clause where 
> one can filter documents based on data that is not available in the index. 
> For example, filter the output of a reduce() based on the calculated 
> metrics.
> {code}
> having(
>   reduce(
> search(.),
> sum(cost),
> on=customerId
>   ),
>   q="sum(cost):[500 TO *]"
> )
> {code}
> This example would return all where the total spent by each distinct customer 
> is >= 500. The total spent is calculated via the sum(cost) metric in the 
> reduce stream.
> The intent is to support as the filters in the having(...) clause the full 
> query syntax of a search(...) clause. I see this being possible in one of two 
> ways. 
> 1. Use Lucene's MemoryIndex and as each tuple is read out of the underlying 
> stream creating an instance of MemoryIndex and apply the query to it. If the 
> result of that is >0 then the tuple should be returned from the HavingStream.
> 2. Create an in-memory solr index via something like RamDirectory, read all 
> tuples into that in-memory index using the UpdateStream, and then stream out 
> of that all the matching tuples from the query.
> There are benefits to each approach but I think the easiest and most direct 
> one is the MemoryIndex approach. With MemoryIndex it isn't necessary to read 
> all incoming tuples before returning a single tuple. With a MemoryIndex there 
> is a need to parse the solr query parameters and create a valid Lucene query 
> but I suspect that can be done using existing QParser implementations.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-8530) Add HavingStream to Streaming API and StreamingExpressions

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-8530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15796348#comment-15796348
 ] 

ASF subversion and git services commented on SOLR-8530:
---

Commit 5bbd4d6765d69d245131d049a2551c0534c1180d in lucene-solr's branch 
refs/heads/master from [~joel.bernstein]
[ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=5bbd4d6 ]

SOLR-8530: Add tests from the HavingStream


> Add HavingStream to Streaming API and StreamingExpressions
> --
>
> Key: SOLR-8530
> URL: https://issues.apache.org/jira/browse/SOLR-8530
> Project: Solr
>  Issue Type: Improvement
>  Components: SolrJ
>Affects Versions: 6.0
>Reporter: Dennis Gove
>Priority: Minor
> Attachments: SOLR-8530.patch
>
>
> The goal here is to support something similar to SQL's HAVING clause where 
> one can filter documents based on data that is not available in the index. 
> For example, filter the output of a reduce() based on the calculated 
> metrics.
> {code}
> having(
>   reduce(
> search(.),
> sum(cost),
> on=customerId
>   ),
>   q="sum(cost):[500 TO *]"
> )
> {code}
> This example would return all where the total spent by each distinct customer 
> is >= 500. The total spent is calculated via the sum(cost) metric in the 
> reduce stream.
> The intent is to support as the filters in the having(...) clause the full 
> query syntax of a search(...) clause. I see this being possible in one of two 
> ways. 
> 1. Use Lucene's MemoryIndex and as each tuple is read out of the underlying 
> stream creating an instance of MemoryIndex and apply the query to it. If the 
> result of that is >0 then the tuple should be returned from the HavingStream.
> 2. Create an in-memory solr index via something like RamDirectory, read all 
> tuples into that in-memory index using the UpdateStream, and then stream out 
> of that all the matching tuples from the query.
> There are benefits to each approach but I think the easiest and most direct 
> one is the MemoryIndex approach. With MemoryIndex it isn't necessary to read 
> all incoming tuples before returning a single tuple. With a MemoryIndex there 
> is a need to parse the solr query parameters and create a valid Lucene query 
> but I suspect that can be done using existing QParser implementations.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-NightlyTests-6.x - Build # 246 - Still Unstable

[Apache Jenkins Server]

Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-6.x/246/

10 tests failed.
FAILED:  org.apache.solr.cloud.LeaderFailoverAfterPartitionTest.test

Error Message:
Expected 2 of 3 replicas to be active but only found 1; 
[core_node3:{"core":"c8n_1x3_lf_shard1_replica2","base_url":"http://127.0.0.1:43460/wy","node_name":"127.0.0.1:43460_wy","state":"active","leader":"true"}];
 clusterState: DocCollection(c8n_1x3_lf//clusterstate.json/31)={   
"replicationFactor":"3",   "shards":{"shard1":{   
"range":"8000-7fff",   "state":"active",   "replicas":{ 
"core_node1":{   "core":"c8n_1x3_lf_shard1_replica3",   
"base_url":"http://127.0.0.1:32919/wy;,   
"node_name":"127.0.0.1:32919_wy",   "state":"down"}, 
"core_node2":{   "state":"down",   
"base_url":"http://127.0.0.1:43113/wy;,   
"core":"c8n_1x3_lf_shard1_replica1",   
"node_name":"127.0.0.1:43113_wy"}, "core_node3":{   
"core":"c8n_1x3_lf_shard1_replica2",   
"base_url":"http://127.0.0.1:43460/wy;,   
"node_name":"127.0.0.1:43460_wy",   "state":"active",   
"leader":"true",   "router":{"name":"compositeId"},   
"maxShardsPerNode":"1",   "autoAddReplicas":"false"}

Stack Trace:
java.lang.AssertionError: Expected 2 of 3 replicas to be active but only found 
1; 
[core_node3:{"core":"c8n_1x3_lf_shard1_replica2","base_url":"http://127.0.0.1:43460/wy","node_name":"127.0.0.1:43460_wy","state":"active","leader":"true"}];
 clusterState: DocCollection(c8n_1x3_lf//clusterstate.json/31)={
  "replicationFactor":"3",
  "shards":{"shard1":{
  "range":"8000-7fff",
  "state":"active",
  "replicas":{
"core_node1":{
  "core":"c8n_1x3_lf_shard1_replica3",
  "base_url":"http://127.0.0.1:32919/wy;,
  "node_name":"127.0.0.1:32919_wy",
  "state":"down"},
"core_node2":{
  "state":"down",
  "base_url":"http://127.0.0.1:43113/wy;,
  "core":"c8n_1x3_lf_shard1_replica1",
  "node_name":"127.0.0.1:43113_wy"},
"core_node3":{
  "core":"c8n_1x3_lf_shard1_replica2",
  "base_url":"http://127.0.0.1:43460/wy;,
  "node_name":"127.0.0.1:43460_wy",
  "state":"active",
  "leader":"true",
  "router":{"name":"compositeId"},
  "maxShardsPerNode":"1",
  "autoAddReplicas":"false"}
at 
__randomizedtesting.SeedInfo.seed([D715F336889356E:852560E9C6755896]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.assertTrue(Assert.java:43)
at 
org.apache.solr.cloud.LeaderFailoverAfterPartitionTest.testRf3WithLeaderFailover(LeaderFailoverAfterPartitionTest.java:168)
at 
org.apache.solr.cloud.LeaderFailoverAfterPartitionTest.test(LeaderFailoverAfterPartitionTest.java:55)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:992)
at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:967)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:811)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:462)
at 

[jira] [Comment Edited] (SOLR-9917) NPE in JSON facet merger

[Markus Jelsma (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15796293#comment-15796293
 ] 

Markus Jelsma edited comment on SOLR-9917 at 1/3/17 9:56 PM:
-

The NPE is thrown at the line where i execute the query with this test input 
data:

{code}
indexDoc(sdoc("document", "document_A", "uid", "1", "time", time(24), 
"type", "view", "key", "test_key", "coord", "53.22,6.56"));
{code}

It's gone when adding the requested data (view_time=50)
{code}
indexDoc(sdoc("document", "document_A", "uid", "1", "time", time(24), 
"type", "view", "key", "test_key", "coord", "53.22,6.56", "view_time", "50"));
{code}

Test extends AbstractFullDistribZkTestBase and has @ShardsFixed(num = 2) set


was (Author: markus17):
The NPE it thrown at the line where i execute the query with this test input 
data:

{code}
indexDoc(sdoc("document", "document_A", "uid", "1", "time", time(24), 
"type", "view", "key", "test_key", "coord", "53.22,6.56"));
{code}

It's gone when adding the requested data (view_time=50)
{code}
indexDoc(sdoc("document", "document_A", "uid", "1", "time", time(24), 
"type", "view", "key", "test_key", "coord", "53.22,6.56", "view_time", "50"));
{code}

Test extends AbstractFullDistribZkTestBase and has @ShardsFixed(num = 2) set

> NPE in JSON facet merger
> 
>
> Key: SOLR-9917
> URL: https://issues.apache.org/jira/browse/SOLR-9917
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>Affects Versions: 6.3
>Reporter: Markus Jelsma
>Assignee: Yonik Seeley
> Fix For: master (7.0), 6.4
>
> Attachments: SOLR-9917.patch
>
>
> I've spotted this before, and just now one of my unit tests does it as well. 
> I believe this happens when there is no data for the requested field.
> {code}
> java.lang.NullPointerException
> at java.nio.ByteBuffer.wrap(ByteBuffer.java:396)
> at 
> org.apache.solr.search.facet.PercentileAgg$Merger.merge(PercentileAgg.java:195)
> at 
> org.apache.solr.search.facet.FacetBucket.mergeBucket(FacetBucket.java:90)
> at 
> org.apache.solr.search.facet.FacetRequestSortedMerger.mergeBucketList(FacetRequestSortedMerger.java:61)
> at 
> org.apache.solr.search.facet.FacetRangeMerger.mergeBucketList(FacetRangeMerger.java:27)
> at 
> org.apache.solr.search.facet.FacetRangeMerger.merge(FacetRangeMerger.java:91)
> at 
> org.apache.solr.search.facet.FacetRangeMerger.merge(FacetRangeMerger.java:43)
> at 
> org.apache.solr.search.facet.FacetBucket.mergeBucket(FacetBucket.java:90)
> at 
> org.apache.solr.search.facet.FacetQueryMerger.merge(FacetModule.java:444)
> at 
> org.apache.solr.search.facet.FacetModule.handleResponses(FacetModule.java:272)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Comment Edited] (SOLR-9917) NPE in JSON facet merger

[Markus Jelsma (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15796293#comment-15796293
 ] 

Markus Jelsma edited comment on SOLR-9917 at 1/3/17 9:56 PM:
-

The NPE it thrown at the line where i execute the query with this test input 
data:

{code}
indexDoc(sdoc("document", "document_A", "uid", "1", "time", time(24), 
"type", "view", "key", "test_key", "coord", "53.22,6.56"));
{code}

It's gone when adding the requested data (view_time=50)
{code}
indexDoc(sdoc("document", "document_A", "uid", "1", "time", time(24), 
"type", "view", "key", "test_key", "coord", "53.22,6.56", "view_time", "50"));
{code}

Test extends AbstractFullDistribZkTestBase and has @ShardsFixed(num = 2) set


was (Author: markus17):
The NPE is throws leading to the line where i execute the query with this test 
input data:

{code}
indexDoc(sdoc("document", "document_A", "uid", "1", "time", time(24), 
"type", "view", "key", "test_key", "coord", "53.22,6.56"));
{code}

It's gone when adding the requested data (view_time=50)
{code}
indexDoc(sdoc("document", "document_A", "uid", "1", "time", time(24), 
"type", "view", "key", "test_key", "coord", "53.22,6.56", "view_time", "50"));
{code}

Test extends AbstractFullDistribZkTestBase and has @ShardsFixed(num = 2) set

> NPE in JSON facet merger
> 
>
> Key: SOLR-9917
> URL: https://issues.apache.org/jira/browse/SOLR-9917
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>Affects Versions: 6.3
>Reporter: Markus Jelsma
>Assignee: Yonik Seeley
> Fix For: master (7.0), 6.4
>
> Attachments: SOLR-9917.patch
>
>
> I've spotted this before, and just now one of my unit tests does it as well. 
> I believe this happens when there is no data for the requested field.
> {code}
> java.lang.NullPointerException
> at java.nio.ByteBuffer.wrap(ByteBuffer.java:396)
> at 
> org.apache.solr.search.facet.PercentileAgg$Merger.merge(PercentileAgg.java:195)
> at 
> org.apache.solr.search.facet.FacetBucket.mergeBucket(FacetBucket.java:90)
> at 
> org.apache.solr.search.facet.FacetRequestSortedMerger.mergeBucketList(FacetRequestSortedMerger.java:61)
> at 
> org.apache.solr.search.facet.FacetRangeMerger.mergeBucketList(FacetRangeMerger.java:27)
> at 
> org.apache.solr.search.facet.FacetRangeMerger.merge(FacetRangeMerger.java:91)
> at 
> org.apache.solr.search.facet.FacetRangeMerger.merge(FacetRangeMerger.java:43)
> at 
> org.apache.solr.search.facet.FacetBucket.mergeBucket(FacetBucket.java:90)
> at 
> org.apache.solr.search.facet.FacetQueryMerger.merge(FacetModule.java:444)
> at 
> org.apache.solr.search.facet.FacetModule.handleResponses(FacetModule.java:272)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Comment Edited] (SOLR-9917) NPE in JSON facet merger

[Markus Jelsma (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15796293#comment-15796293
 ] 

Markus Jelsma edited comment on SOLR-9917 at 1/3/17 9:55 PM:
-

The NPE is throws leading to the line where i execute the query with this test 
input data:

{code}
indexDoc(sdoc("document", "document_A", "uid", "1", "time", time(24), 
"type", "view", "key", "test_key", "coord", "53.22,6.56"));
{code}

It's gone when adding the requested data (view_time=50)
{code}
indexDoc(sdoc("document", "document_A", "uid", "1", "time", time(24), 
"type", "view", "key", "test_key", "coord", "53.22,6.56", "view_time", "50"));
{code}

Test extends AbstractFullDistribZkTestBase and has @ShardsFixed(num = 2) set


was (Author: markus17):
The NPE is throws leading to the line where i execute the query with this test 
input data:

{code]
indexDoc(sdoc("document", "document_A", "uid", "1", "time", time(24), 
"type", "view", "key", "test_key", "coord", "53.22,6.56"));
{code}

It's gone when adding the requested data (view_time=50)
{code}
indexDoc(sdoc("document", "document_A", "uid", "1", "time", time(24), 
"type", "view", "key", "test_key", "coord", "53.22,6.56", "view_time", "50"));
{code}

Test extends AbstractFullDistribZkTestBase and has @ShardsFixed(num = 2) set

> NPE in JSON facet merger
> 
>
> Key: SOLR-9917
> URL: https://issues.apache.org/jira/browse/SOLR-9917
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>Affects Versions: 6.3
>Reporter: Markus Jelsma
>Assignee: Yonik Seeley
> Fix For: master (7.0), 6.4
>
> Attachments: SOLR-9917.patch
>
>
> I've spotted this before, and just now one of my unit tests does it as well. 
> I believe this happens when there is no data for the requested field.
> {code}
> java.lang.NullPointerException
> at java.nio.ByteBuffer.wrap(ByteBuffer.java:396)
> at 
> org.apache.solr.search.facet.PercentileAgg$Merger.merge(PercentileAgg.java:195)
> at 
> org.apache.solr.search.facet.FacetBucket.mergeBucket(FacetBucket.java:90)
> at 
> org.apache.solr.search.facet.FacetRequestSortedMerger.mergeBucketList(FacetRequestSortedMerger.java:61)
> at 
> org.apache.solr.search.facet.FacetRangeMerger.mergeBucketList(FacetRangeMerger.java:27)
> at 
> org.apache.solr.search.facet.FacetRangeMerger.merge(FacetRangeMerger.java:91)
> at 
> org.apache.solr.search.facet.FacetRangeMerger.merge(FacetRangeMerger.java:43)
> at 
> org.apache.solr.search.facet.FacetBucket.mergeBucket(FacetBucket.java:90)
> at 
> org.apache.solr.search.facet.FacetQueryMerger.merge(FacetModule.java:444)
> at 
> org.apache.solr.search.facet.FacetModule.handleResponses(FacetModule.java:272)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-9917) NPE in JSON facet merger

[Markus Jelsma (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15796293#comment-15796293
 ] 

Markus Jelsma commented on SOLR-9917:
-

The NPE is throws leading to the line where i execute the query with this test 
input data:

{code]
indexDoc(sdoc("document", "document_A", "uid", "1", "time", time(24), 
"type", "view", "key", "test_key", "coord", "53.22,6.56"));
{code}

It's gone when adding the requested data (view_time=50)
{code}
indexDoc(sdoc("document", "document_A", "uid", "1", "time", time(24), 
"type", "view", "key", "test_key", "coord", "53.22,6.56", "view_time", "50"));
{code}

Test extends AbstractFullDistribZkTestBase and has @ShardsFixed(num = 2) set

> NPE in JSON facet merger
> 
>
> Key: SOLR-9917
> URL: https://issues.apache.org/jira/browse/SOLR-9917
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>Affects Versions: 6.3
>Reporter: Markus Jelsma
>Assignee: Yonik Seeley
> Fix For: master (7.0), 6.4
>
> Attachments: SOLR-9917.patch
>
>
> I've spotted this before, and just now one of my unit tests does it as well. 
> I believe this happens when there is no data for the requested field.
> {code}
> java.lang.NullPointerException
> at java.nio.ByteBuffer.wrap(ByteBuffer.java:396)
> at 
> org.apache.solr.search.facet.PercentileAgg$Merger.merge(PercentileAgg.java:195)
> at 
> org.apache.solr.search.facet.FacetBucket.mergeBucket(FacetBucket.java:90)
> at 
> org.apache.solr.search.facet.FacetRequestSortedMerger.mergeBucketList(FacetRequestSortedMerger.java:61)
> at 
> org.apache.solr.search.facet.FacetRangeMerger.mergeBucketList(FacetRangeMerger.java:27)
> at 
> org.apache.solr.search.facet.FacetRangeMerger.merge(FacetRangeMerger.java:91)
> at 
> org.apache.solr.search.facet.FacetRangeMerger.merge(FacetRangeMerger.java:43)
> at 
> org.apache.solr.search.facet.FacetBucket.mergeBucket(FacetBucket.java:90)
> at 
> org.apache.solr.search.facet.FacetQueryMerger.merge(FacetModule.java:444)
> at 
> org.apache.solr.search.facet.FacetModule.handleResponses(FacetModule.java:272)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-6.x-MacOSX (64bit/jdk1.8.0) - Build # 619 - Failure!

[Policeman Jenkins Server]

Build: https://jenkins.thetaphi.de/job/Lucene-Solr-6.x-MacOSX/619/
Java: 64bit/jdk1.8.0 -XX:-UseCompressedOops -XX:+UseConcMarkSweepGC

All tests passed

Build Log:
[...truncated 68484 lines...]
-ecj-javadoc-lint-src:
[mkdir] Created dir: 
/var/folders/qg/h2dfw5s161s51l2bn79mrb7rgn/T/ecj2025476677
 [ecj-lint] Compiling 49 source files to 
/var/folders/qg/h2dfw5s161s51l2bn79mrb7rgn/T/ecj2025476677
 [ecj-lint] invalid Class-Path header in manifest of jar file: 
/Users/jenkins/.ivy2/cache/org.restlet.jee/org.restlet/jars/org.restlet-2.3.0.jar
 [ecj-lint] invalid Class-Path header in manifest of jar file: 
/Users/jenkins/.ivy2/cache/org.restlet.jee/org.restlet.ext.servlet/jars/org.restlet.ext.servlet-2.3.0.jar
 [ecj-lint] --
 [ecj-lint] 1. ERROR in 
/Users/jenkins/workspace/Lucene-Solr-6.x-MacOSX/solr/test-framework/src/java/org/apache/solr/cloud/AbstractDistribZkTestBase.java
 (at line 22)
 [ecj-lint] import java.util.concurrent.TimeUnit;
 [ecj-lint]^
 [ecj-lint] The import java.util.concurrent.TimeUnit is never used
 [ecj-lint] --
 [ecj-lint] 1 problem (1 error)

BUILD FAILED
/Users/jenkins/workspace/Lucene-Solr-6.x-MacOSX/build.xml:775: The following 
error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-6.x-MacOSX/build.xml:101: The following 
error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-6.x-MacOSX/solr/build.xml:671: The 
following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-6.x-MacOSX/lucene/common-build.xml:1992: 
The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-6.x-MacOSX/lucene/common-build.xml:2031: 
Compile failed; see the compiler error output for details.

Total time: 101 minutes 46 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
[WARNINGS] Skipping publisher since build result is FAILURE
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-9917) NPE in JSON facet merger

[Markus Jelsma (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15796280#comment-15796280
 ] 

Markus Jelsma commented on SOLR-9917:
-

Oh, i should add that this is indeed in cloud mode. The unit tests also run in 
cloud mode.

> NPE in JSON facet merger
> 
>
> Key: SOLR-9917
> URL: https://issues.apache.org/jira/browse/SOLR-9917
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>Affects Versions: 6.3
>Reporter: Markus Jelsma
>Assignee: Yonik Seeley
> Fix For: master (7.0), 6.4
>
> Attachments: SOLR-9917.patch
>
>
> I've spotted this before, and just now one of my unit tests does it as well. 
> I believe this happens when there is no data for the requested field.
> {code}
> java.lang.NullPointerException
> at java.nio.ByteBuffer.wrap(ByteBuffer.java:396)
> at 
> org.apache.solr.search.facet.PercentileAgg$Merger.merge(PercentileAgg.java:195)
> at 
> org.apache.solr.search.facet.FacetBucket.mergeBucket(FacetBucket.java:90)
> at 
> org.apache.solr.search.facet.FacetRequestSortedMerger.mergeBucketList(FacetRequestSortedMerger.java:61)
> at 
> org.apache.solr.search.facet.FacetRangeMerger.mergeBucketList(FacetRangeMerger.java:27)
> at 
> org.apache.solr.search.facet.FacetRangeMerger.merge(FacetRangeMerger.java:91)
> at 
> org.apache.solr.search.facet.FacetRangeMerger.merge(FacetRangeMerger.java:43)
> at 
> org.apache.solr.search.facet.FacetBucket.mergeBucket(FacetBucket.java:90)
> at 
> org.apache.solr.search.facet.FacetQueryMerger.merge(FacetModule.java:444)
> at 
> org.apache.solr.search.facet.FacetModule.handleResponses(FacetModule.java:272)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-9917) NPE in JSON facet merger

[Markus Jelsma (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15796277#comment-15796277
 ] 

Markus Jelsma commented on SOLR-9917:
-

I only have access to the code right now, here's the SolrJ code causing it:

vars:
logHistory is just the number of days we look back in the history;
percentile is, well, the percentile, it is set to 90th percentile

{code}
SolrQuery query = new SolrQuery("{!term f=document}" + document);
query.setRows(0);
query.addFilterQuery("{!term f=type}view");
query.addFilterQuery("time:[NOW-" + logHistory + "DAY/DAY TO 
NOW+1DAY/DAY]");

query.setParam("json.facet", "{period:{type:range,field:time,start:\"NOW-" 
+ logHistory + "DAY/DAY\",end:\"NOW+1DAY/DAY\",gap:\"+" + logHistory + 
"DAY\",facet:{time_percentiles:\"percentile(view_time," + percentile + 
")\"}}}");

QueryResponse response = client.query(query);
{code}

Let me know if you need more information.


> NPE in JSON facet merger
> 
>
> Key: SOLR-9917
> URL: https://issues.apache.org/jira/browse/SOLR-9917
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>Affects Versions: 6.3
>Reporter: Markus Jelsma
>Assignee: Yonik Seeley
> Fix For: master (7.0), 6.4
>
> Attachments: SOLR-9917.patch
>
>
> I've spotted this before, and just now one of my unit tests does it as well. 
> I believe this happens when there is no data for the requested field.
> {code}
> java.lang.NullPointerException
> at java.nio.ByteBuffer.wrap(ByteBuffer.java:396)
> at 
> org.apache.solr.search.facet.PercentileAgg$Merger.merge(PercentileAgg.java:195)
> at 
> org.apache.solr.search.facet.FacetBucket.mergeBucket(FacetBucket.java:90)
> at 
> org.apache.solr.search.facet.FacetRequestSortedMerger.mergeBucketList(FacetRequestSortedMerger.java:61)
> at 
> org.apache.solr.search.facet.FacetRangeMerger.mergeBucketList(FacetRangeMerger.java:27)
> at 
> org.apache.solr.search.facet.FacetRangeMerger.merge(FacetRangeMerger.java:91)
> at 
> org.apache.solr.search.facet.FacetRangeMerger.merge(FacetRangeMerger.java:43)
> at 
> org.apache.solr.search.facet.FacetBucket.mergeBucket(FacetBucket.java:90)
> at 
> org.apache.solr.search.facet.FacetQueryMerger.merge(FacetModule.java:444)
> at 
> org.apache.solr.search.facet.FacetModule.handleResponses(FacetModule.java:272)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-9917) NPE in JSON facet merger

[Yonik Seeley (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-9917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yonik Seeley updated SOLR-9917:
---
Attachment: SOLR-9917.patch

Here's a test patch I tried using to force empty shards.  But I guess we need 
more along the lines of shards with documents but w/o the values.  I'll try 
that next.

> NPE in JSON facet merger
> 
>
> Key: SOLR-9917
> URL: https://issues.apache.org/jira/browse/SOLR-9917
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>Affects Versions: 6.3
>Reporter: Markus Jelsma
>Assignee: Yonik Seeley
> Fix For: master (7.0), 6.4
>
> Attachments: SOLR-9917.patch
>
>
> I've spotted this before, and just now one of my unit tests does it as well. 
> I believe this happens when there is no data for the requested field.
> {code}
> java.lang.NullPointerException
> at java.nio.ByteBuffer.wrap(ByteBuffer.java:396)
> at 
> org.apache.solr.search.facet.PercentileAgg$Merger.merge(PercentileAgg.java:195)
> at 
> org.apache.solr.search.facet.FacetBucket.mergeBucket(FacetBucket.java:90)
> at 
> org.apache.solr.search.facet.FacetRequestSortedMerger.mergeBucketList(FacetRequestSortedMerger.java:61)
> at 
> org.apache.solr.search.facet.FacetRangeMerger.mergeBucketList(FacetRangeMerger.java:27)
> at 
> org.apache.solr.search.facet.FacetRangeMerger.merge(FacetRangeMerger.java:91)
> at 
> org.apache.solr.search.facet.FacetRangeMerger.merge(FacetRangeMerger.java:43)
> at 
> org.apache.solr.search.facet.FacetBucket.mergeBucket(FacetBucket.java:90)
> at 
> org.apache.solr.search.facet.FacetQueryMerger.merge(FacetModule.java:444)
> at 
> org.apache.solr.search.facet.FacetModule.handleResponses(FacetModule.java:272)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: 6.4 release

[Anshum Gupta]

Thanks for volunteering Jim!

I have some stuff that I'd like to get in, but nothing that's a show
stopper.

-Anshum

On Tue, Jan 3, 2017 at 10:23 AM Joel Bernstein  wrote:

> +1. I still have a little more work to do for 6.4 but I should have it all
> committed by end of day tomorrow.
>
> Joel Bernstein
> http://joelsolr.blogspot.com/
>
> On Tue, Jan 3, 2017 at 12:25 PM, Michael McCandless <
> luc...@mikemccandless.com> wrote:
>
> +1, lots of good stuff for 6.4 already!  Thanks for volunteering Jim :)
>
> Mike McCandless
>
> http://blog.mikemccandless.com
>
>
> On Tue, Jan 3, 2017 at 11:23 AM, jim ferenczi 
> wrote:
> > Hi,
> > I would like to volunteer to release 6.4. I can cut the release branch
> next
> > Monday if everybody agrees.
> >
> > Jim
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org
>
>
>

Re: Welcome Jim Ferenczi as a Lucene/Solr committer

[Anshum Gupta]

Congratulations and welcome Jim!

On Tue, Jan 3, 2017 at 10:35 AM Kevin Risden 
wrote:

> Welcome Jim!
>
> Kevin Risden
>
> On Tue, Jan 3, 2017 at 12:30 PM, Varun Thacker  wrote:
>
> Welcome Jim!
>
> On Tue, Jan 3, 2017 at 8:38 AM, Erick Erickson 
> wrote:
>
> And then he immediately volunteers to cut 6.4. What a guy!
>
> On Tue, Jan 3, 2017 at 6:26 AM, Christine Poerschke (BLOOMBERG/
> LONDON)  wrote:
> > Welcome Jim!
> >
> > - Original Message -
> > From: dev@lucene.apache.org
> > To: dev@lucene.apache.org
> > At: 01/01/17 10:05:18
> >
> > I'm pleased to announce that Jim Ferenczi has accepted the Lucene
> > PMC's invitation to become a committer.
> >
> > Jim, it's tradition that you introduce yourself with a brief bio.
> >
> > Your handle "jimczi" has been added to the “lucene" LDAP group, so you
> > now have commit privileges. Please test this by adding yourself to the
> > committers section of the Who We Are page on the website:
> >  (instructions here
> > ).
> >
> > The ASF dev page also has lots of useful links: <
> http://www.apache.org/dev/>.
> >
> > Congratulations and welcome and Happy New Year,
> >
> > Mike McCandless
> >
> > http://blog.mikemccandless.com
> >
> > -
> > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> > For additional commands, e-mail: dev-h...@lucene.apache.org
> >
> >
> >
> > -
> > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> > For additional commands, e-mail: dev-h...@lucene.apache.org
> >
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org
>
>
>
>

Re: Welcome Mikhail Khludnev to the PMC

[Anshum Gupta]

Welcome Mikhail!

On Tue, Jan 3, 2017 at 11:04 AM Varun Thacker  wrote:

> Congrats Mikhail!
>
> On Tue, Jan 3, 2017 at 10:35 AM, Kevin Risden 
> wrote:
>
> Congrats Mikhail!
>
> Kevin Risden
>
> On Tue, Jan 3, 2017 at 3:58 AM, Ramkumar R. Aiyengar <
> andyetitmo...@gmail.com> wrote:
>
> Congratulations Mikhail!
>
> On 30 Dec 2016 15:16, "Adrien Grand"  wrote:
>
> I am pleased to announce that Mikhail Khludnev has accepted the PMC's
> invitation to join.
>
> Welcome Mikhail!
>
> Adrien
>
>
>
>

Re: Welcome Christine Poerschke to the PMC

[Anshum Gupta]

Welcome Christine!

On Tue, Jan 3, 2017 at 11:05 AM Varun Thacker  wrote:

> Congrats Christine!
>
> On Tue, Jan 3, 2017 at 10:35 AM, Kevin Risden 
> wrote:
>
> Congrats Christine!
>
> Kevin Risden
>
> On Tue, Jan 3, 2017 at 6:09 AM, Dennis Gove  wrote:
>
> Wahoo! Congratulations Christine!
>
> On Fri, Dec 30, 2016 at 7:46 AM, Adrien Grand  wrote:
>
> I am pleased to announce that Christine Poerschke has accepted the PMC's
> invitation to join.
>
> Welcome Christine!
>
> Adrien
>
>
>
>
>

[jira] [Commented] (SOLR-9917) NPE in JSON facet merger

[Yonik Seeley (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15796234#comment-15796234
 ] 

Yonik Seeley commented on SOLR-9917:


It's odd this never happens once in a while to TestJsonFacets, so the criteria 
may be more specific than having a shard w/o a field.  What does the relevant 
part of the query request look like (i.e. how is the percentile being used and 
what is it nested under?)

> NPE in JSON facet merger
> 
>
> Key: SOLR-9917
> URL: https://issues.apache.org/jira/browse/SOLR-9917
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>Affects Versions: 6.3
>Reporter: Markus Jelsma
>Assignee: Yonik Seeley
> Fix For: master (7.0), 6.4
>
>
> I've spotted this before, and just now one of my unit tests does it as well. 
> I believe this happens when there is no data for the requested field.
> {code}
> java.lang.NullPointerException
> at java.nio.ByteBuffer.wrap(ByteBuffer.java:396)
> at 
> org.apache.solr.search.facet.PercentileAgg$Merger.merge(PercentileAgg.java:195)
> at 
> org.apache.solr.search.facet.FacetBucket.mergeBucket(FacetBucket.java:90)
> at 
> org.apache.solr.search.facet.FacetRequestSortedMerger.mergeBucketList(FacetRequestSortedMerger.java:61)
> at 
> org.apache.solr.search.facet.FacetRangeMerger.mergeBucketList(FacetRangeMerger.java:27)
> at 
> org.apache.solr.search.facet.FacetRangeMerger.merge(FacetRangeMerger.java:91)
> at 
> org.apache.solr.search.facet.FacetRangeMerger.merge(FacetRangeMerger.java:43)
> at 
> org.apache.solr.search.facet.FacetBucket.mergeBucket(FacetBucket.java:90)
> at 
> org.apache.solr.search.facet.FacetQueryMerger.merge(FacetModule.java:444)
> at 
> org.apache.solr.search.facet.FacetModule.handleResponses(FacetModule.java:272)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (LUCENE-7618) Hypothetical perf improvements in DocValuesRangeQuery: reducing comparisons for some queries/segments

[Hoss Man (JIRA)]

 [ 
https://issues.apache.org/jira/browse/LUCENE-7618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hoss Man updated LUCENE-7618:
-
Attachment: LUCENE-7618.patch


*TL;DR:* based on some half assed benchmarking using junit timings, the results 
did *NOT* show any improvements -- for some seeds (on my laptop) the new code 
was actaully slower sometimes, leading me to suspect I may have a bug somewhere 
in either the test or the code, but I haven't had time to dig into it.



*Background...*

Just before I went on vacation, I had a conversation with someone who asked a 
question that can be summarized as:

bq. If I already know the smallest possible value ({{minF}}) in the field 
{{foo}}, which is faster: {{foo:\[\* TO X\]}} or {{foo:\[minF TO X\]}} ?  
_(where X will be diff in every query)_

My answer, based on recollection of how simple term indexing range queries use 
to work, was that any difference should be negligable -- in the {{\*}} case 
lucene can skip the call to {{seekTo(minF)}} but otherwise the executation 
would be identical after that.  For DocValues, I speculated that using {{\*}} 
should theoretically be faster, because as the code iterates ove the DocValues, 
it would only have to compare each doc's value to {{X}}, and not to {{minF}}.

When I dug into the code, I found that wasn't the case: while query rewriting 
had a special case optimization for {{foo:\[\* TO \*\]}} there were no special 
case optimizations for a range query open on one end -- instead the 
TwoPhaseIterator created for SortedNumericDocValues assigns 
Long.MIN_VALUE/Long.MAX_VALUE as needed anytime min/max are null, and does 2 
comparisons per doc value.  Likewise the codepath for SortedSetDocValues 
assigns minOrd ({{0}}) and maxOrd ({{values.getValueCount() - 1}}) when the 
user specified min/max are null; and again did 2 comparisons per value.

*Theory...*

Adding special case handling for the open ended range queries, to return 
TwoPhaseIterators that only did one comparison in these special cases, 
seem(ed|s) like an easy win?  In particular for the case of SortedSetDocValues: 
Even if the original query has non-null min & max values, after calling 
{{lookupTerm(...)}} on each we might find that one of them is completely out of 
range _for individual segments_ and optimize away one comparison on a 
per-segment level.  (an existing optimization already handled the case where 
the min/max were both completely below/above the range of values in the segment)

*Attempt...*

Step #1 was writting a {{TestDocValuesRangeQueryOptimizations}} (see patch) 
against the existing code that built up an index with identical values in 3 
diff fields (SortedNumeric, SortedSet, and Points) and then did a lot of 
randomized, open ended, range queries against that index, one field per test 
method.  

Once I had the basics of the test written, I then ran the test over and over 
with the same seed and noted down the junit test times for each.

I then added what seemed like the most straight forward optimizations to 
{{DocValuesRangeQuery}}, and re-ran the test a few times with the same seed.

*Results...*

The results were not good.  Looking back, didn't even bother to save the tmp 
file where I had been pasting the junit times -- that's how not good they were. 
 Even accounting for potential "noise" in the results from other stuff running 
on my laptop, there was no inkling of any speedup in the new code (as compared 
to the test against the points field which didn't involve any modified code 
paths) and IIRC the results were occasionally slower.


I set all this aside to come back to later to try and figure out if I'd made 
any obvious mistakes, but skimming the code today, and writting up my 
recollections, nothing jumps out at me.  My best guess is that HotSpot is 
already optimizing away some of these comparisons, and that the new code just 
complicates things for HotSpot to the point that it's sometimes slower.  I 
don't plan on taking this any further, but I wanted to open the issue to track 
the idea in case anyone else sees value in pursuing it futher.






> Hypothetical perf improvements in DocValuesRangeQuery: reducing comparisons 
> for some queries/segments
> -
>
> Key: LUCENE-7618
> URL: https://issues.apache.org/jira/browse/LUCENE-7618
> Project: Lucene - Core
>  Issue Type: Improvement
>Reporter: Hoss Man
> Attachments: LUCENE-7618.patch
>
>
> In reviewing the DocValuesRangeQuery code, it occured to me that there 
> _might_ be some potential performance optimizations possible in a few cases 
> relating queries that involve explicitly specified open ranges (ie: min or 
> max are null) or in the case of SortedSet: range queries that are 
> *effectively* open ended on particular segments, 

[jira] [Created] (LUCENE-7618) Hypothetical perf improvements in DocValuesRangeQuery: reducing comparisons for some queries/segments

[Hoss Man (JIRA)]

Hoss Man created LUCENE-7618:


 Summary: Hypothetical perf improvements in DocValuesRangeQuery: 
reducing comparisons for some queries/segments
 Key: LUCENE-7618
 URL: https://issues.apache.org/jira/browse/LUCENE-7618
 Project: Lucene - Core
  Issue Type: Improvement
Reporter: Hoss Man



In reviewing the DocValuesRangeQuery code, it occured to me that there _might_ 
be some potential performance optimizations possible in a few cases relating 
queries that involve explicitly specified open ranges (ie: min or max are null) 
or in the case of SortedSet: range queries that are *effectively* open ended on 
particular segments, because the min/max are below/above the minOrd/maxOrd for 
the segment.

Since these seemed like semi-common situations (open ended range queries are 
fairly common in my experience, i'm not sure about the secondary SortedSet 
"ord" case, but it seemd potentially promising particularly for fields like 
incrementing ids, or timestamps, where values are added sequentially and 
likeley to be clustered together) I did a bit of experimenting and wanted to 
post my findings in jira -- patch & details to follow in comments.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Comment Edited] (SOLR-8593) Integrate Apache Calcite into the SQLHandler

[Joel Bernstein (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-8593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15796107#comment-15796107
 ] 

Joel Bernstein edited comment on SOLR-8593 at 1/3/17 8:33 PM:
--

It doesn't look like we're going to get this in for Solr 6.4. So, I'm currently 
planning to take the HavingStream from this branch and commit it with SOLR-8530 
for Solr 6.4

This is going to cause merge conflicts between jira/solr-8593  and master. As 
soon as the Calcite release is cut I'll begin the work to merge jira/solr-8593  
to master and work out the conflicts. So Calcite will be in master at the very 
beginning of the dev cycle for Solr 6.5. Then we can work with the Calcite code 
in master and perform the backport to branch_6x when ready.


was (Author: joel.bernstein):
It doesn't look like we're going to get this in for Solr 6.4. So, I'm currently 
planning to take the HavingStream from this branch and commit it with SOLR-8530 
for Solr 6.4

This is going to cause merge conflicts between jira/solr-8593  and master. As 
soon as the Calcite release is cut I'll begin the work to merge jira/solr-8593  
to master and work out the conflicts. So Calcite will be in master at very 
beginning of the dev cycle for Solr 6.5. Then we can work with the Calcite code 
more in master and perform the backport to branch_6x when ready.

> Integrate Apache Calcite into the SQLHandler
> 
>
> Key: SOLR-8593
> URL: https://issues.apache.org/jira/browse/SOLR-8593
> Project: Solr
>  Issue Type: Improvement
>Reporter: Joel Bernstein
>Assignee: Joel Bernstein
> Attachments: SOLR-8593.patch, SOLR-8593.patch
>
>
>The Presto SQL Parser was perfect for phase one of the SQLHandler. It was 
> nicely split off from the larger Presto project and it did everything that 
> was needed for the initial implementation.
> Phase two of the SQL work though will require an optimizer. Here is where 
> Apache Calcite comes into play. It has a battle tested cost based optimizer 
> and has been integrated into Apache Drill and Hive.
> This work can begin in trunk following the 6.0 release. The final query plans 
> will continue to be translated to Streaming API objects (TupleStreams), so 
> continued work on the JDBC driver should plug in nicely with the Calcite work.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Comment Edited] (SOLR-8593) Integrate Apache Calcite into the SQLHandler

[Joel Bernstein (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-8593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15796107#comment-15796107
 ] 

Joel Bernstein edited comment on SOLR-8593 at 1/3/17 8:32 PM:
--

It doesn't look like we're going to get this in for Solr 6.4. So, I'm currently 
planning to take the HavingStream from this branch and commit it with SOLR-8530 
for Solr 6.4

This is going to cause merge conflicts between jira/solr-8593  and master. As 
soon as the Calcite release is cut I'll begin the work to merge jira/solr-8593  
to master and work out the conflicts. So Calcite will be in master at very 
beginning of the dev cycle for Solr 6.5. Then we can work with the Calcite code 
more in master and perform the backport to branch_6x when ready.


was (Author: joel.bernstein):
It doesn't look like we're going to get this in for Solr 6.4. So, I'm currently 
planning to take the HavingStream from this branch an commit it with SOLR-8530 
for Solr 6.4

This is going to cause merge conflicts between jira/solr-8593  and master. As 
soon as the Calcite release is cut I'll begin the work to merge jira/solr-8593  
to master and work out the conflicts. So Calcite will be in master at very 
beginning of the dev cycle for Solr 6.5. Then we can work with the Calcite code 
more in master and perform the backport to branch_6x when ready.

> Integrate Apache Calcite into the SQLHandler
> 
>
> Key: SOLR-8593
> URL: https://issues.apache.org/jira/browse/SOLR-8593
> Project: Solr
>  Issue Type: Improvement
>Reporter: Joel Bernstein
>Assignee: Joel Bernstein
> Attachments: SOLR-8593.patch, SOLR-8593.patch
>
>
>The Presto SQL Parser was perfect for phase one of the SQLHandler. It was 
> nicely split off from the larger Presto project and it did everything that 
> was needed for the initial implementation.
> Phase two of the SQL work though will require an optimizer. Here is where 
> Apache Calcite comes into play. It has a battle tested cost based optimizer 
> and has been integrated into Apache Drill and Hive.
> This work can begin in trunk following the 6.0 release. The final query plans 
> will continue to be translated to Streaming API objects (TupleStreams), so 
> continued work on the JDBC driver should plug in nicely with the Calcite work.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-8593) Integrate Apache Calcite into the SQLHandler

[Joel Bernstein (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-8593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15796107#comment-15796107
 ] 

Joel Bernstein commented on SOLR-8593:
--

It doesn't look like we're going to get this in for Solr 6.4. So, I'm currently 
planning to take the HavingStream from this branch an commit it with SOLR-8530 
for Solr 6.4

This is going to cause merge conflicts between jira/solr-8593  and master. As 
soon as the Calcite release is cut I'll begin the work to merge jira/solr-8593  
to master and work out the conflicts. So Calcite will be in master at very 
beginning of the dev cycle for Solr 6.5. Then we can work with the Calcite code 
more in master and perform the backport to branch_6x when ready.

> Integrate Apache Calcite into the SQLHandler
> 
>
> Key: SOLR-8593
> URL: https://issues.apache.org/jira/browse/SOLR-8593
> Project: Solr
>  Issue Type: Improvement
>Reporter: Joel Bernstein
>Assignee: Joel Bernstein
> Attachments: SOLR-8593.patch, SOLR-8593.patch
>
>
>The Presto SQL Parser was perfect for phase one of the SQLHandler. It was 
> nicely split off from the larger Presto project and it did everything that 
> was needed for the initial implementation.
> Phase two of the SQL work though will require an optimizer. Here is where 
> Apache Calcite comes into play. It has a battle tested cost based optimizer 
> and has been integrated into Apache Drill and Hive.
> This work can begin in trunk following the 6.0 release. The final query plans 
> will continue to be translated to Streaming API objects (TupleStreams), so 
> continued work on the JDBC driver should plug in nicely with the Calcite work.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-9877) Use instrumented http client

[Shalin Shekhar Mangar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-9877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shalin Shekhar Mangar updated SOLR-9877:

Attachment: SOLR-9877_branch_6x.patch

Here's a patch that applies on branch_6x. It required a different approach than 
master because 6.x uses an older deprecated http client API. I ended up 
extending DefaultHttpClient to override the createRequestExecutor() method 
which creates and sets up the InstrumentedHttpRequestExecutor.

This does not use the HttpClientFactory and its methods introduced in SOLR-6625 
but firstly the factory's static setter is never used in Solr and secondly, 
I'll open an issue to get rid of it completely.

> Use instrumented http client
> 
>
> Key: SOLR-9877
> URL: https://issues.apache.org/jira/browse/SOLR-9877
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: metrics
>Reporter: Shalin Shekhar Mangar
>Assignee: Shalin Shekhar Mangar
>Priority: Minor
> Fix For: master (7.0), 6.4
>
> Attachments: SOLR-9877.patch, SOLR-9877.patch, 
> SOLR-9877_branch_6x.patch, solr-http-metrics.png
>
>
> Use instrumented equivalents of PooledHttpClientConnectionManager and others 
> from metrics-httpclient library.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-master-MacOSX (64bit/jdk1.8.0) - Build # 3753 - Unstable!

[Policeman Jenkins Server]

Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-MacOSX/3753/
Java: 64bit/jdk1.8.0 -XX:-UseCompressedOops -XX:+UseG1GC

2 tests failed.
FAILED:  org.apache.solr.handler.extraction.TestExtractionDateUtil.testParseDate

Error Message:
Incorrect parsed timestamp: 1226583351000 != 1226579751000 (Thu Nov 13 04:35:51 
AKST 2008)

Stack Trace:
java.lang.AssertionError: Incorrect parsed timestamp: 1226583351000 != 
1226579751000 (Thu Nov 13 04:35:51 AKST 2008)
at 
__randomizedtesting.SeedInfo.seed([1E2E689D7B2EB0F5:543710A80087C740]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.assertTrue(Assert.java:43)
at 
org.apache.solr.handler.extraction.TestExtractionDateUtil.assertParsedDate(TestExtractionDateUtil.java:59)
at 
org.apache.solr.handler.extraction.TestExtractionDateUtil.testParseDate(TestExtractionDateUtil.java:54)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:811)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:462)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.cloud.PeerSyncReplicationTest.test

Error Message:
timeout waiting to see all nodes active

Stack Trace:
java.lang.AssertionError: timeout waiting to see all nodes active
at 
__randomizedtesting.SeedInfo.seed([9471CBEC3F4F517C:1C25F43691B33C84]:0)
at org.junit.Assert.fail(Assert.java:93)
at 

[jira] [Commented] (SOLR-9899) StandardDirectoryFactory should use optimizations for all FilterDirectorys not just NRTCachingDirectory.

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15795990#comment-15795990
 ] 

ASF subversion and git services commented on SOLR-9899:
---

Commit 6caefcda8b9aefb1f107e629e1c00de7d9104ec4 in lucene-solr's branch 
refs/heads/branch_6x from markrmiller
[ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=6caefcd ]

SOLR-9899: StandardDirectoryFactory should use optimizations for all 
FilterDirectorys not just NRTCachingDirectory.


> StandardDirectoryFactory should use optimizations for all FilterDirectorys 
> not just NRTCachingDirectory.
> 
>
> Key: SOLR-9899
> URL: https://issues.apache.org/jira/browse/SOLR-9899
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Mark Miller
>Assignee: Mark Miller
> Fix For: master (7.0), 6.4
>
> Attachments: SOLR-9899.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-9859) replication.properties cannot be updated after being written and neither replication.properties or index.properties are durable in the face of a crash

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15795946#comment-15795946
 ] 

ASF subversion and git services commented on SOLR-9859:
---

Commit c6ea1718675beeb988b08b2b213155d8b20f in lucene-solr's branch 
refs/heads/branch_6x from markrmiller
[ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=c6ea171 ]

SOLR-9859: replication.properties cannot be updated after being written and 
neither eplication.properties or ndex.properties are durable in the face of a 
crash.


> replication.properties cannot be updated after being written and neither 
> replication.properties or index.properties are durable in the face of a crash
> --
>
> Key: SOLR-9859
> URL: https://issues.apache.org/jira/browse/SOLR-9859
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>Affects Versions: 5.5.3, 6.3
>Reporter: Pushkar Raste
>Assignee: Mark Miller
>Priority: Minor
> Attachments: SOLR-9859.patch, SOLR-9859.patch, SOLR-9859.patch, 
> SOLR-9859.patch, SOLR-9859.patch, SOLR-9859.patch
>
>
> If a shard recovers via replication (vs PeerSync) a file named 
> {{replication.properties}} gets created. If the same shard recovers once more 
> via replication, IndexFetcher fails to write latest replication information 
> as it tries to create {{replication.properties}} but as file already exists. 
> Here is the stack trace I saw 
> {code}
> java.nio.file.FileAlreadyExistsException: 
> \shard-3-001\cores\collection1\data\replication.properties
>   at sun.nio.fs.WindowsException.translateToIOException(Unknown Source)
>   at sun.nio.fs.WindowsException.rethrowAsIOException(Unknown Source)
>   at sun.nio.fs.WindowsException.rethrowAsIOException(Unknown Source)
>   at sun.nio.fs.WindowsFileSystemProvider.newByteChannel(Unknown Source)
>   at java.nio.file.spi.FileSystemProvider.newOutputStream(Unknown Source)
>   at java.nio.file.Files.newOutputStream(Unknown Source)
>   at 
> org.apache.lucene.store.FSDirectory$FSIndexOutput.(FSDirectory.java:413)
>   at 
> org.apache.lucene.store.FSDirectory$FSIndexOutput.(FSDirectory.java:409)
>   at 
> org.apache.lucene.store.FSDirectory.createOutput(FSDirectory.java:253)
>   at 
> org.apache.solr.handler.IndexFetcher.logReplicationTimeAndConfFiles(IndexFetcher.java:689)
>   at 
> org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:501)
>   at 
> org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:265)
>   at 
> org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:397)
>   at 
> org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:157)
>   at 
> org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:409)
>   at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:222)
>   at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
>   at java.util.concurrent.FutureTask.run(Unknown Source)
>   at 
> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$0(ExecutorUtil.java:229)
>   at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
>   at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
>   at java.lang.Thread.run(Unknown Source)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-9835) Create another replication mode for SolrCloud

[Noble Paul (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15795916#comment-15795916
 ] 

Noble Paul commented on SOLR-9835:
--

What is the public interface for this feature? How do I enable/disable this 
feature? Is there a command?

> Create another replication mode for SolrCloud
> -
>
> Key: SOLR-9835
> URL: https://issues.apache.org/jira/browse/SOLR-9835
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Cao Manh Dat
>Assignee: Shalin Shekhar Mangar
> Attachments: SOLR-9835.patch, SOLR-9835.patch, SOLR-9835.patch, 
> SOLR-9835.patch, SOLR-9835.patch
>
>
> The current replication mechanism of SolrCloud is called state machine, which 
> replicas start in same initial state and for each input, the input is 
> distributed across replicas so all replicas will end up with same next state. 
> But this type of replication have some drawbacks
> - The commit (which costly) have to run on all replicas
> - Slow recovery, because if replica miss more than N updates on its down 
> time, the replica have to download entire index from its leader.
> So we create create another replication mode for SolrCloud called state 
> transfer, which acts like master/slave replication. In basically
> - Leader distribute the update to other replicas, but the leader only apply 
> the update to IW, other replicas just store the update to UpdateLog (act like 
> replication).
> - Replicas frequently polling the latest segments from leader.
> Pros:
> - Lightweight for indexing, because only leader are running the commit, 
> updates.
> - Very fast recovery, replicas just have to download the missing segments.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: Welcome Christine Poerschke to the PMC

[Varun Thacker]

Congrats Christine!

On Tue, Jan 3, 2017 at 10:35 AM, Kevin Risden 
wrote:

> Congrats Christine!
>
> Kevin Risden
>
> On Tue, Jan 3, 2017 at 6:09 AM, Dennis Gove  wrote:
>
>> Wahoo! Congratulations Christine!
>>
>> On Fri, Dec 30, 2016 at 7:46 AM, Adrien Grand  wrote:
>>
>>> I am pleased to announce that Christine Poerschke has accepted the PMC's
>>> invitation to join.
>>>
>>> Welcome Christine!
>>>
>>> Adrien
>>>
>>
>>
>

Re: Welcome Mikhail Khludnev to the PMC

[Varun Thacker]

Congrats Mikhail!

On Tue, Jan 3, 2017 at 10:35 AM, Kevin Risden 
wrote:

> Congrats Mikhail!
>
> Kevin Risden
>
> On Tue, Jan 3, 2017 at 3:58 AM, Ramkumar R. Aiyengar <
> andyetitmo...@gmail.com> wrote:
>
>> Congratulations Mikhail!
>>
>> On 30 Dec 2016 15:16, "Adrien Grand"  wrote:
>>
>>> I am pleased to announce that Mikhail Khludnev has accepted the PMC's
>>> invitation to join.
>>>
>>> Welcome Mikhail!
>>>
>>> Adrien
>>>
>>
>

[jira] [Commented] (SOLR-9906) Use better check to validate if node recovered via PeerSync or Replication

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15795845#comment-15795845
 ] 

ASF subversion and git services commented on SOLR-9906:
---

Commit 812070a77f483149e1d83b3d1bbc7ba80f0fd868 in lucene-solr's branch 
refs/heads/branch_6x from [~noble.paul]
[ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=812070a ]

SOLR-9906-Use better check to validate if node recovered via PeerSync or 
Replication


> Use better check to validate if node recovered via PeerSync or Replication
> --
>
> Key: SOLR-9906
> URL: https://issues.apache.org/jira/browse/SOLR-9906
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Pushkar Raste
>Priority: Minor
> Attachments: SOLR-9906.patch, SOLR-9906.patch, 
> SOLR-PeerSyncVsReplicationTest.diff
>
>
> Tests {{LeaderFailureAfterFreshStartTest}} and {{PeerSyncReplicationTest}} 
> currently rely on number of requests made to the leader's replication handler 
> to check if node recovered via PeerSync or replication. This check is not 
> very reliable and we have seen failures in the past. 
> While tinkering with different way to write a better test I found 
> [SOLR-9859|SOLR-9859]. Now that SOLR-9859 is fixed, here is idea for better 
> way to distinguish recovery via PeerSync vs Replication. 
> * For {{PeerSyncReplicationTest}}, if node successfully recovers via 
> PeerSync, then file {{replication.properties}} should not exist
> For {{LeaderFailureAfterFreshStartTest}}, if the freshly replicated node does 
> not go into replication recovery after the leader failure, contents 
> {{replication.properties}} should not change 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Assigned] (SOLR-9917) NPE in JSON facet merger

[Yonik Seeley (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-9917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yonik Seeley reassigned SOLR-9917:
--

Assignee: Yonik Seeley

> NPE in JSON facet merger
> 
>
> Key: SOLR-9917
> URL: https://issues.apache.org/jira/browse/SOLR-9917
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>Affects Versions: 6.3
>Reporter: Markus Jelsma
>Assignee: Yonik Seeley
> Fix For: master (7.0), 6.4
>
>
> I've spotted this before, and just now one of my unit tests does it as well. 
> I believe this happens when there is no data for the requested field.
> {code}
> java.lang.NullPointerException
> at java.nio.ByteBuffer.wrap(ByteBuffer.java:396)
> at 
> org.apache.solr.search.facet.PercentileAgg$Merger.merge(PercentileAgg.java:195)
> at 
> org.apache.solr.search.facet.FacetBucket.mergeBucket(FacetBucket.java:90)
> at 
> org.apache.solr.search.facet.FacetRequestSortedMerger.mergeBucketList(FacetRequestSortedMerger.java:61)
> at 
> org.apache.solr.search.facet.FacetRangeMerger.mergeBucketList(FacetRangeMerger.java:27)
> at 
> org.apache.solr.search.facet.FacetRangeMerger.merge(FacetRangeMerger.java:91)
> at 
> org.apache.solr.search.facet.FacetRangeMerger.merge(FacetRangeMerger.java:43)
> at 
> org.apache.solr.search.facet.FacetBucket.mergeBucket(FacetBucket.java:90)
> at 
> org.apache.solr.search.facet.FacetQueryMerger.merge(FacetModule.java:444)
> at 
> org.apache.solr.search.facet.FacetModule.handleResponses(FacetModule.java:272)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-9917) NPE in JSON facet merger

[Yonik Seeley (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15795820#comment-15795820
 ] 

Yonik Seeley commented on SOLR-9917:


Thanks for investigating Markus, I'll look into it now...

> NPE in JSON facet merger
> 
>
> Key: SOLR-9917
> URL: https://issues.apache.org/jira/browse/SOLR-9917
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>Affects Versions: 6.3
>Reporter: Markus Jelsma
> Fix For: master (7.0), 6.4
>
>
> I've spotted this before, and just now one of my unit tests does it as well. 
> I believe this happens when there is no data for the requested field.
> {code}
> java.lang.NullPointerException
> at java.nio.ByteBuffer.wrap(ByteBuffer.java:396)
> at 
> org.apache.solr.search.facet.PercentileAgg$Merger.merge(PercentileAgg.java:195)
> at 
> org.apache.solr.search.facet.FacetBucket.mergeBucket(FacetBucket.java:90)
> at 
> org.apache.solr.search.facet.FacetRequestSortedMerger.mergeBucketList(FacetRequestSortedMerger.java:61)
> at 
> org.apache.solr.search.facet.FacetRangeMerger.mergeBucketList(FacetRangeMerger.java:27)
> at 
> org.apache.solr.search.facet.FacetRangeMerger.merge(FacetRangeMerger.java:91)
> at 
> org.apache.solr.search.facet.FacetRangeMerger.merge(FacetRangeMerger.java:43)
> at 
> org.apache.solr.search.facet.FacetBucket.mergeBucket(FacetBucket.java:90)
> at 
> org.apache.solr.search.facet.FacetQueryMerger.merge(FacetModule.java:444)
> at 
> org.apache.solr.search.facet.FacetModule.handleResponses(FacetModule.java:272)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-9915) PeerSync alreadyInSync check is not backwards compatible

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15795803#comment-15795803
 ] 

ASF subversion and git services commented on SOLR-9915:
---

Commit 122fa6cf64a56dd5ab5aff84f7f5c9a1305bde4e in lucene-solr's branch 
refs/heads/branch_6x from [~noble.paul]
[ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=122fa6c ]

SOLR-9915: PeerSync alreadyInSync check is not backwards compatible and results 
in full replication during a rolling restart


> PeerSync alreadyInSync check is not backwards compatible
> 
>
> Key: SOLR-9915
> URL: https://issues.apache.org/jira/browse/SOLR-9915
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: replication (java)
>Affects Versions: 6.3
>Reporter: Tim Owen
>Assignee: Noble Paul
>Priority: Blocker
> Attachments: SOLR-9915.patch
>
>
> The fingerprint check added to PeerSync in SOLR-9446 works fine when all 
> servers are running 6.3 but this means it's hard to do a rolling upgrade from 
> e.g. 6.2.1 to 6.3 because the 6.3 server sends a request to a 6.2.1 server to 
> get a fingerprint and then gets a NPE because the older server doesn't return 
> the expected field in its response.
> This leads to the PeerSync completely failing, and results in a full index 
> replication from scratch, copying all index files over the network. We 
> noticed this happening when we tried to do a rolling upgrade on one of our 
> 6.2.1 clusters to 6.3. Unfortunately this amount of replication was hammering 
> our disks and network, so we had to do a full shutdown, upgrade all to 6.3 
> and restart, which was not ideal for a production cluster.
> The attached patch should behave more gracefully in this situation, as it 
> will typically return false for alreadyInSync() and then carry on doing the 
> normal re-sync based on versions.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-9915) PeerSync alreadyInSync check is not backwards compatible

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15795802#comment-15795802
 ] 

ASF subversion and git services commented on SOLR-9915:
---

Commit 1b9564a5dccb2938586f2f82f963bd1534b002cd in lucene-solr's branch 
refs/heads/branch_6x from [~noble.paul]
[ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=1b9564a ]

SOLR-9915: PeerSync alreadyInSync check is not backwards compatible and results 
in full replication during a rolling restart


> PeerSync alreadyInSync check is not backwards compatible
> 
>
> Key: SOLR-9915
> URL: https://issues.apache.org/jira/browse/SOLR-9915
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: replication (java)
>Affects Versions: 6.3
>Reporter: Tim Owen
>Assignee: Noble Paul
>Priority: Blocker
> Attachments: SOLR-9915.patch
>
>
> The fingerprint check added to PeerSync in SOLR-9446 works fine when all 
> servers are running 6.3 but this means it's hard to do a rolling upgrade from 
> e.g. 6.2.1 to 6.3 because the 6.3 server sends a request to a 6.2.1 server to 
> get a fingerprint and then gets a NPE because the older server doesn't return 
> the expected field in its response.
> This leads to the PeerSync completely failing, and results in a full index 
> replication from scratch, copying all index files over the network. We 
> noticed this happening when we tried to do a rolling upgrade on one of our 
> 6.2.1 clusters to 6.3. Unfortunately this amount of replication was hammering 
> our disks and network, so we had to do a full shutdown, upgrade all to 6.3 
> and restart, which was not ideal for a production cluster.
> The attached patch should behave more gracefully in this situation, as it 
> will typically return false for alreadyInSync() and then carry on doing the 
> normal re-sync based on versions.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: Welcome Mikhail Khludnev to the PMC

[Kevin Risden]

Congrats Mikhail!

Kevin Risden

On Tue, Jan 3, 2017 at 3:58 AM, Ramkumar R. Aiyengar <
andyetitmo...@gmail.com> wrote:

> Congratulations Mikhail!
>
> On 30 Dec 2016 15:16, "Adrien Grand"  wrote:
>
>> I am pleased to announce that Mikhail Khludnev has accepted the PMC's
>> invitation to join.
>>
>> Welcome Mikhail!
>>
>> Adrien
>>
>

Re: Welcome Christine Poerschke to the PMC

[Kevin Risden]

Congrats Christine!

Kevin Risden

On Tue, Jan 3, 2017 at 6:09 AM, Dennis Gove  wrote:

> Wahoo! Congratulations Christine!
>
> On Fri, Dec 30, 2016 at 7:46 AM, Adrien Grand  wrote:
>
>> I am pleased to announce that Christine Poerschke has accepted the PMC's
>> invitation to join.
>>
>> Welcome Christine!
>>
>> Adrien
>>
>
>

Re: Welcome Jim Ferenczi as a Lucene/Solr committer

[Kevin Risden]

Welcome Jim!

Kevin Risden

On Tue, Jan 3, 2017 at 12:30 PM, Varun Thacker  wrote:

> Welcome Jim!
>
> On Tue, Jan 3, 2017 at 8:38 AM, Erick Erickson 
> wrote:
>
>> And then he immediately volunteers to cut 6.4. What a guy!
>>
>> On Tue, Jan 3, 2017 at 6:26 AM, Christine Poerschke (BLOOMBERG/
>> LONDON)  wrote:
>> > Welcome Jim!
>> >
>> > - Original Message -
>> > From: dev@lucene.apache.org
>> > To: dev@lucene.apache.org
>> > At: 01/01/17 10:05:18
>> >
>> > I'm pleased to announce that Jim Ferenczi has accepted the Lucene
>> > PMC's invitation to become a committer.
>> >
>> > Jim, it's tradition that you introduce yourself with a brief bio.
>> >
>> > Your handle "jimczi" has been added to the “lucene" LDAP group, so you
>> > now have commit privileges. Please test this by adding yourself to the
>> > committers section of the Who We Are page on the website:
>> >  (instructions here
>> > ).
>> >
>> > The ASF dev page also has lots of useful links: <
>> http://www.apache.org/dev/>.
>> >
>> > Congratulations and welcome and Happy New Year,
>> >
>> > Mike McCandless
>> >
>> > http://blog.mikemccandless.com
>> >
>> > -
>> > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
>> > For additional commands, e-mail: dev-h...@lucene.apache.org
>> >
>> >
>> >
>> > -
>> > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
>> > For additional commands, e-mail: dev-h...@lucene.apache.org
>> >
>>
>> -
>> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
>> For additional commands, e-mail: dev-h...@lucene.apache.org
>>
>>
>

Re: Welcome Jim Ferenczi as a Lucene/Solr committer

[Varun Thacker]

Welcome Jim!

On Tue, Jan 3, 2017 at 8:38 AM, Erick Erickson 
wrote:

> And then he immediately volunteers to cut 6.4. What a guy!
>
> On Tue, Jan 3, 2017 at 6:26 AM, Christine Poerschke (BLOOMBERG/
> LONDON)  wrote:
> > Welcome Jim!
> >
> > - Original Message -
> > From: dev@lucene.apache.org
> > To: dev@lucene.apache.org
> > At: 01/01/17 10:05:18
> >
> > I'm pleased to announce that Jim Ferenczi has accepted the Lucene
> > PMC's invitation to become a committer.
> >
> > Jim, it's tradition that you introduce yourself with a brief bio.
> >
> > Your handle "jimczi" has been added to the “lucene" LDAP group, so you
> > now have commit privileges. Please test this by adding yourself to the
> > committers section of the Who We Are page on the website:
> >  (instructions here
> > ).
> >
> > The ASF dev page also has lots of useful links: <
> http://www.apache.org/dev/>.
> >
> > Congratulations and welcome and Happy New Year,
> >
> > Mike McCandless
> >
> > http://blog.mikemccandless.com
> >
> > -
> > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> > For additional commands, e-mail: dev-h...@lucene.apache.org
> >
> >
> >
> > -
> > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> > For additional commands, e-mail: dev-h...@lucene.apache.org
> >
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org
>
>

Re: 6.4 release

[Joel Bernstein]

+1. I still have a little more work to do for 6.4 but I should have it all
committed by end of day tomorrow.

Joel Bernstein
http://joelsolr.blogspot.com/

On Tue, Jan 3, 2017 at 12:25 PM, Michael McCandless <
luc...@mikemccandless.com> wrote:

> +1, lots of good stuff for 6.4 already!  Thanks for volunteering Jim :)
>
> Mike McCandless
>
> http://blog.mikemccandless.com
>
>
> On Tue, Jan 3, 2017 at 11:23 AM, jim ferenczi 
> wrote:
> > Hi,
> > I would like to volunteer to release 6.4. I can cut the release branch
> next
> > Monday if everybody agrees.
> >
> > Jim
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org
>
>

[jira] [Updated] (SOLR-8530) Add HavingStream to Streaming API and StreamingExpressions

[Joel Bernstein (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-8530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joel Bernstein updated SOLR-8530:
-
Attachment: SOLR-8530.patch

Patch with tests.

> Add HavingStream to Streaming API and StreamingExpressions
> --
>
> Key: SOLR-8530
> URL: https://issues.apache.org/jira/browse/SOLR-8530
> Project: Solr
>  Issue Type: Improvement
>  Components: SolrJ
>Affects Versions: 6.0
>Reporter: Dennis Gove
>Priority: Minor
> Attachments: SOLR-8530.patch
>
>
> The goal here is to support something similar to SQL's HAVING clause where 
> one can filter documents based on data that is not available in the index. 
> For example, filter the output of a reduce() based on the calculated 
> metrics.
> {code}
> having(
>   reduce(
> search(.),
> sum(cost),
> on=customerId
>   ),
>   q="sum(cost):[500 TO *]"
> )
> {code}
> This example would return all where the total spent by each distinct customer 
> is >= 500. The total spent is calculated via the sum(cost) metric in the 
> reduce stream.
> The intent is to support as the filters in the having(...) clause the full 
> query syntax of a search(...) clause. I see this being possible in one of two 
> ways. 
> 1. Use Lucene's MemoryIndex and as each tuple is read out of the underlying 
> stream creating an instance of MemoryIndex and apply the query to it. If the 
> result of that is >0 then the tuple should be returned from the HavingStream.
> 2. Create an in-memory solr index via something like RamDirectory, read all 
> tuples into that in-memory index using the UpdateStream, and then stream out 
> of that all the matching tuples from the query.
> There are benefits to each approach but I think the easiest and most direct 
> one is the MemoryIndex approach. With MemoryIndex it isn't necessary to read 
> all incoming tuples before returning a single tuple. With a MemoryIndex there 
> is a need to parse the solr query parameters and create a valid Lucene query 
> but I suspect that can be done using existing QParser implementations.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (LUCENE-7055) Better execution path for costly queries

[Adrien Grand (JIRA)]

 [ 
https://issues.apache.org/jira/browse/LUCENE-7055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Adrien Grand updated LUCENE-7055:
-
Attachment: LUCENE-7055.patch

Here is an updated patch. It adds docs, tests and improves the way BooleanQuery 
implements this new API so that cost estimations would be propagated lazily 
through deep query trees (similarly to the fact that two-phase iteration still 
executes slow bits in deep leaves after fast bits that are in the top-level 
BooleanQuery).

Only PointRangeQuery and BooleanQuery implement this API right now, since they 
are the two important ones to get started, but I think we might be able to 
generalize to other slow queries, like TermsQuery for instance, by leveraging 
the sumDocFreq index statistic. But this belongs to another issue.

Reviews are warmly welcome.

> Better execution path for costly queries
> 
>
> Key: LUCENE-7055
> URL: https://issues.apache.org/jira/browse/LUCENE-7055
> Project: Lucene - Core
>  Issue Type: Improvement
>Reporter: Adrien Grand
>Assignee: Adrien Grand
> Attachments: LUCENE-7055.patch, LUCENE-7055.patch
>
>
> In Lucene 5.0, we improved the execution path for queries that run costly 
> operations on a per-document basis, like phrase queries or doc values 
> queries. But we have another class of costly queries, that return fine 
> iterators, but these iterators are very expensive to build. This is typically 
> the case for queries that leverage DocIdSetBuilder, like TermsQuery, 
> multi-term queries or the new point queries. Intersecting such queries with a 
> selective query is very inefficient since these queries build a doc id set of 
> matching documents for the entire index.
> Is there something we could do to improve the execution path for these 
> queries?
> One idea that comes to mind is that most of these queries could also run on 
> doc values, so maybe we could come up with something that would help decide 
> how to run a query based on other parts of the query? (Just thinking out 
> loud, other ideas are very welcome)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-9915) PeerSync alreadyInSync check is not backwards compatible

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15795691#comment-15795691
 ] 

ASF subversion and git services commented on SOLR-9915:
---

Commit 5b1f6b2ba48f8afc6c822c097d0500eb2ed66815 in lucene-solr's branch 
refs/heads/master from [~noble.paul]
[ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=5b1f6b2 ]

SOLR-9915: PeerSync alreadyInSync check is not backwards compatible and results 
in full replication during a rolling restart


> PeerSync alreadyInSync check is not backwards compatible
> 
>
> Key: SOLR-9915
> URL: https://issues.apache.org/jira/browse/SOLR-9915
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: replication (java)
>Affects Versions: 6.3
>Reporter: Tim Owen
>Assignee: Noble Paul
>Priority: Blocker
> Attachments: SOLR-9915.patch
>
>
> The fingerprint check added to PeerSync in SOLR-9446 works fine when all 
> servers are running 6.3 but this means it's hard to do a rolling upgrade from 
> e.g. 6.2.1 to 6.3 because the 6.3 server sends a request to a 6.2.1 server to 
> get a fingerprint and then gets a NPE because the older server doesn't return 
> the expected field in its response.
> This leads to the PeerSync completely failing, and results in a full index 
> replication from scratch, copying all index files over the network. We 
> noticed this happening when we tried to do a rolling upgrade on one of our 
> 6.2.1 clusters to 6.3. Unfortunately this amount of replication was hammering 
> our disks and network, so we had to do a full shutdown, upgrade all to 6.3 
> and restart, which was not ideal for a production cluster.
> The attached patch should behave more gracefully in this situation, as it 
> will typically return false for alreadyInSync() and then carry on doing the 
> normal re-sync based on versions.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: [feature proposal] Segment sorted by CUSTOM SortField

[Adrien Grand]

Hi Eduardo,

I agree with the usefulness, but implementing it will raise issues with
backward compatibility since those custom sort fields would need to be
serialized in the index files, which I think will be controversial. I think
it would still be useful to open an issue to get the problem discussed, but
I wanted to warn you that this is not an easy issue to get started with. I
suspect we will look at use-cases one-by-one, like we did for multi-valued
fields (https://issues.apache.org/jira/browse/LUCENE-7537) rather than try
to solve the problem in the general case. If there is some specific type of
sort that you are interested in, that might be something easier to get in.

If you want something easier to get started with, something that comes to
my mind is this old comment about adding a token filter that protects
tokens based on their length. I don't think it has been implemented:
https://issues.apache.org/jira/browse/LUCENE-5208?focusedCommentId=13766576=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13766576
.



Le mar. 3 janv. 2017 à 18:27, Eduardo Alonso  a
écrit :

> Hi to all:
>
> I am Eduardo Alonso and i am new to Lucene.
>
> If you use *indexWriterConfig.setIndexSort(Sort sort)* lucene saves
> Segments sorted by Sort, but does not allows sorting by Custom SortFields
> (user defined class extending o.a.l.s.SortField).
>
> Do you think is this functionality (sort by custom SortField) useful?
>
> I would like to contribute to Lucene deploying this functionality.
>
> Should I open a Jira ticket?
>
> Regards
>
>
>
> Eduardo Alonso
> Vía de las dos Castillas, 33, Ática 4, 3ª Planta
> 28224 Pozuelo de Alarcón, Madrid
> Tel: +34 91 828 6473 <+34%20918%2028%2064%2073> // www.stratio.com // 
> *@stratiobd
> *
>

[jira] [Updated] (SOLR-9918) An UpdateRequestProcessor to skip duplicate inserts and ignore updates to missing docs

[Tim Owen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-9918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tim Owen updated SOLR-9918:
---
Attachment: SOLR-9918.patch

> An UpdateRequestProcessor to skip duplicate inserts and ignore updates to 
> missing docs
> --
>
> Key: SOLR-9918
> URL: https://issues.apache.org/jira/browse/SOLR-9918
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: update
>Reporter: Tim Owen
> Attachments: SOLR-9918.patch
>
>
> This is an UpdateRequestProcessor and Factory that we have been using in 
> production, to handle 2 common cases that were awkward to achieve using the 
> existing update pipeline and current processor classes:
> * When inserting document(s), if some already exist then quietly skip the new 
> document inserts - do not churn the index by replacing the existing documents 
> and do not throw a noisy exception that breaks the batch of inserts. By 
> analogy with SQL, {{insert if not exists}}. In our use-case, multiple 
> application instances can (rarely) process the same input so it's easier for 
> us to de-dupe these at Solr insert time than to funnel them into a global 
> ordered queue first.
> * When applying AtomicUpdate documents, if a document being updated does not 
> exist, quietly do nothing - do not create a new partially-populated document 
> and do not throw a noisy exception about missing required fields. By analogy 
> with SQL, {{update where id = ..}}. Our use-case relies on this because we 
> apply updates optimistically and have best-effort knowledge about what 
> documents will exist, so it's easiest to skip the updates (in the same way a 
> Database would).
> I would have kept this in our own package hierarchy but it relies on some 
> package-scoped methods, and seems like it could be useful to others if they 
> choose to configure it. Some bits of the code were borrowed from 
> {{DocBasedVersionConstraintsProcessorFactory}}.
> Attached patch has unit tests to confirm the behaviour.
> This class can be used by configuring solrconfig.xml like so..
> {noformat}
>   
> 
>  class="org.apache.solr.update.processor.SkipExistingDocumentsProcessorFactory">
>   true
>   false 
> 
> 
> 
>   
> {noformat}
> and initParams defaults of
> {noformat}
>   skipexisting
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Created] (SOLR-9918) An UpdateRequestProcessor to skip duplicate inserts and ignore updates to missing docs

[Tim Owen (JIRA)]

Tim Owen created SOLR-9918:
--

 Summary: An UpdateRequestProcessor to skip duplicate inserts and 
ignore updates to missing docs
 Key: SOLR-9918
 URL: https://issues.apache.org/jira/browse/SOLR-9918
 Project: Solr
  Issue Type: Improvement
  Security Level: Public (Default Security Level. Issues are Public)
  Components: update
Reporter: Tim Owen


This is an UpdateRequestProcessor and Factory that we have been using in 
production, to handle 2 common cases that were awkward to achieve using the 
existing update pipeline and current processor classes:

* When inserting document(s), if some already exist then quietly skip the new 
document inserts - do not churn the index by replacing the existing documents 
and do not throw a noisy exception that breaks the batch of inserts. By analogy 
with SQL, {{insert if not exists}}. In our use-case, multiple application 
instances can (rarely) process the same input so it's easier for us to de-dupe 
these at Solr insert time than to funnel them into a global ordered queue first.
* When applying AtomicUpdate documents, if a document being updated does not 
exist, quietly do nothing - do not create a new partially-populated document 
and do not throw a noisy exception about missing required fields. By analogy 
with SQL, {{update where id = ..}}. Our use-case relies on this because we 
apply updates optimistically and have best-effort knowledge about what 
documents will exist, so it's easiest to skip the updates (in the same way a 
Database would).

I would have kept this in our own package hierarchy but it relies on some 
package-scoped methods, and seems like it could be useful to others if they 
choose to configure it. Some bits of the code were borrowed from 
{{DocBasedVersionConstraintsProcessorFactory}}.

Attached patch has unit tests to confirm the behaviour.

This class can be used by configuring solrconfig.xml like so..

{noformat}
  


  true
  false 



  
{noformat}

and initParams defaults of

{noformat}
  skipexisting
{noformat}




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Assigned] (SOLR-9915) PeerSync alreadyInSync check is not backwards compatible

[Noble Paul (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-9915?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Noble Paul reassigned SOLR-9915:


Assignee: Noble Paul

> PeerSync alreadyInSync check is not backwards compatible
> 
>
> Key: SOLR-9915
> URL: https://issues.apache.org/jira/browse/SOLR-9915
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: replication (java)
>Affects Versions: 6.3
>Reporter: Tim Owen
>Assignee: Noble Paul
>Priority: Blocker
> Attachments: SOLR-9915.patch
>
>
> The fingerprint check added to PeerSync in SOLR-9446 works fine when all 
> servers are running 6.3 but this means it's hard to do a rolling upgrade from 
> e.g. 6.2.1 to 6.3 because the 6.3 server sends a request to a 6.2.1 server to 
> get a fingerprint and then gets a NPE because the older server doesn't return 
> the expected field in its response.
> This leads to the PeerSync completely failing, and results in a full index 
> replication from scratch, copying all index files over the network. We 
> noticed this happening when we tried to do a rolling upgrade on one of our 
> 6.2.1 clusters to 6.3. Unfortunately this amount of replication was hammering 
> our disks and network, so we had to do a full shutdown, upgrade all to 6.3 
> and restart, which was not ideal for a production cluster.
> The attached patch should behave more gracefully in this situation, as it 
> will typically return false for alreadyInSync() and then carry on doing the 
> normal re-sync based on versions.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-9917) NPE in JSON facet merger

[Markus Jelsma (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15795608#comment-15795608
 ] 

Markus Jelsma commented on SOLR-9917:
-

I can now also confirm that lack of data for a field causes the NPE to occur. I 
added data for a field to my failing unit test, it now passes.

> NPE in JSON facet merger
> 
>
> Key: SOLR-9917
> URL: https://issues.apache.org/jira/browse/SOLR-9917
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>Affects Versions: 6.3
>Reporter: Markus Jelsma
> Fix For: master (7.0), 6.4
>
>
> I've spotted this before, and just now one of my unit tests does it as well. 
> I believe this happens when there is no data for the requested field.
> {code}
> java.lang.NullPointerException
> at java.nio.ByteBuffer.wrap(ByteBuffer.java:396)
> at 
> org.apache.solr.search.facet.PercentileAgg$Merger.merge(PercentileAgg.java:195)
> at 
> org.apache.solr.search.facet.FacetBucket.mergeBucket(FacetBucket.java:90)
> at 
> org.apache.solr.search.facet.FacetRequestSortedMerger.mergeBucketList(FacetRequestSortedMerger.java:61)
> at 
> org.apache.solr.search.facet.FacetRangeMerger.mergeBucketList(FacetRangeMerger.java:27)
> at 
> org.apache.solr.search.facet.FacetRangeMerger.merge(FacetRangeMerger.java:91)
> at 
> org.apache.solr.search.facet.FacetRangeMerger.merge(FacetRangeMerger.java:43)
> at 
> org.apache.solr.search.facet.FacetBucket.mergeBucket(FacetBucket.java:90)
> at 
> org.apache.solr.search.facet.FacetQueryMerger.merge(FacetModule.java:444)
> at 
> org.apache.solr.search.facet.FacetModule.handleResponses(FacetModule.java:272)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Created] (SOLR-9917) NPE in JSON facet merger

[Markus Jelsma (JIRA)]

Markus Jelsma created SOLR-9917:
---

 Summary: NPE in JSON facet merger
 Key: SOLR-9917
 URL: https://issues.apache.org/jira/browse/SOLR-9917
 Project: Solr
  Issue Type: Bug
  Security Level: Public (Default Security Level. Issues are Public)
Affects Versions: 6.3
Reporter: Markus Jelsma
 Fix For: master (7.0), 6.4


I've spotted this before, and just now one of my unit tests does it as well. I 
believe this happens when there is no data for the requested field.

{code}
java.lang.NullPointerException
at java.nio.ByteBuffer.wrap(ByteBuffer.java:396)
at 
org.apache.solr.search.facet.PercentileAgg$Merger.merge(PercentileAgg.java:195)
at 
org.apache.solr.search.facet.FacetBucket.mergeBucket(FacetBucket.java:90)
at 
org.apache.solr.search.facet.FacetRequestSortedMerger.mergeBucketList(FacetRequestSortedMerger.java:61)
at 
org.apache.solr.search.facet.FacetRangeMerger.mergeBucketList(FacetRangeMerger.java:27)
at 
org.apache.solr.search.facet.FacetRangeMerger.merge(FacetRangeMerger.java:91)
at 
org.apache.solr.search.facet.FacetRangeMerger.merge(FacetRangeMerger.java:43)
at 
org.apache.solr.search.facet.FacetBucket.mergeBucket(FacetBucket.java:90)
at 
org.apache.solr.search.facet.FacetQueryMerger.merge(FacetModule.java:444)
at 
org.apache.solr.search.facet.FacetModule.handleResponses(FacetModule.java:272)
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[feature proposal] Segment sorted by CUSTOM SortField

[Eduardo Alonso]

Hi to all:

I am Eduardo Alonso and i am new to Lucene.

If you use *indexWriterConfig.setIndexSort(Sort sort)* lucene saves
Segments sorted by Sort, but does not allows sorting by Custom SortFields
(user defined class extending o.a.l.s.SortField).

Do you think is this functionality (sort by custom SortField) useful?

I would like to contribute to Lucene deploying this functionality.

Should I open a Jira ticket?

Regards



Eduardo Alonso
Vía de las dos Castillas, 33, Ática 4, 3ª Planta
28224 Pozuelo de Alarcón, Madrid
Tel: +34 91 828 6473 // www.stratio.com // *@stratiobd
*

[jira] [Updated] (SOLR-9915) PeerSync alreadyInSync check is not backwards compatible

[Erick Erickson (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-9915?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Erick Erickson updated SOLR-9915:
-
Priority: Blocker  (was: Major)

Marking as "blocker" so we render a considered opinion before we release 6.4.

> PeerSync alreadyInSync check is not backwards compatible
> 
>
> Key: SOLR-9915
> URL: https://issues.apache.org/jira/browse/SOLR-9915
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: replication (java)
>Affects Versions: 6.3
>Reporter: Tim Owen
>Priority: Blocker
> Attachments: SOLR-9915.patch
>
>
> The fingerprint check added to PeerSync in SOLR-9446 works fine when all 
> servers are running 6.3 but this means it's hard to do a rolling upgrade from 
> e.g. 6.2.1 to 6.3 because the 6.3 server sends a request to a 6.2.1 server to 
> get a fingerprint and then gets a NPE because the older server doesn't return 
> the expected field in its response.
> This leads to the PeerSync completely failing, and results in a full index 
> replication from scratch, copying all index files over the network. We 
> noticed this happening when we tried to do a rolling upgrade on one of our 
> 6.2.1 clusters to 6.3. Unfortunately this amount of replication was hammering 
> our disks and network, so we had to do a full shutdown, upgrade all to 6.3 
> and restart, which was not ideal for a production cluster.
> The attached patch should behave more gracefully in this situation, as it 
> will typically return false for alreadyInSync() and then carry on doing the 
> normal re-sync based on versions.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-8030) Transaction log does not store the update chain (or req params?) used for updates

[Hoss Man (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-8030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hoss Man updated SOLR-8030:
---
Description: 
Transaction Log does not store the update chain, or any other details from the 
original update request such as the request params, used during updates.

Therefore tLog uses the default update chain, and a synthetic request, during 
log replay.

If we implement custom update logic with multiple distinct update chains that 
use custom processors after DistributedUpdateProcessor, or if the default chain 
uses processors whose behavior depends on other request params, then log replay 
may be incorrect.

Potentially problematic scenerios (need test cases):

* DBQ where the main query string uses local param variables that refer to 
other request params
* custom Update chain set as {{default="true"}} using something like 
StatelessScriptUpdateProcessorFactory after DUP where the script depends on 
request params.
* multiple named update chains with diff processors configured after DUP and 
specific requests sent to diff chains -- ex: ParseDateProcessor w/ custom 
formats configured after DUP in some special chains, but not in the default 
chain



  was:
Transaction Log does not store the update chain used during updates.

Therefore tLog uses the default update chain during log replay.

If we implement custom update logic with multiple update chains, the log replay 
could break this logic.

Summary: Transaction log does not store the update chain (or req 
params?) used for updates  (was: Transaction log does not store the update 
chain used for updates)

FWIW: I have not looked into this issue in depth, nor have i had time to really 
read & think about the patch posted  -- but based on some recent discussion 
about this issue and SOLR-9883 it occured to me that the problem is almost 
cerainly broader then just keeping track of the chain name -- it's also 
potentially all the request params from the original request.

So i've updated the summary & description accordingly.

> Transaction log does not store the update chain (or req params?) used for 
> updates
> -
>
> Key: SOLR-8030
> URL: https://issues.apache.org/jira/browse/SOLR-8030
> Project: Solr
>  Issue Type: Bug
>  Components: SolrCloud
>Affects Versions: 5.3
>Reporter: Ludovic Boutros
> Attachments: SOLR-8030.patch
>
>
> Transaction Log does not store the update chain, or any other details from 
> the original update request such as the request params, used during updates.
> Therefore tLog uses the default update chain, and a synthetic request, during 
> log replay.
> If we implement custom update logic with multiple distinct update chains that 
> use custom processors after DistributedUpdateProcessor, or if the default 
> chain uses processors whose behavior depends on other request params, then 
> log replay may be incorrect.
> Potentially problematic scenerios (need test cases):
> * DBQ where the main query string uses local param variables that refer to 
> other request params
> * custom Update chain set as {{default="true"}} using something like 
> StatelessScriptUpdateProcessorFactory after DUP where the script depends on 
> request params.
> * multiple named update chains with diff processors configured after DUP and 
> specific requests sent to diff chains -- ex: ParseDateProcessor w/ custom 
> formats configured after DUP in some special chains, but not in the default 
> chain



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: 6.4 release

[Michael McCandless]

+1, lots of good stuff for 6.4 already!  Thanks for volunteering Jim :)

Mike McCandless

http://blog.mikemccandless.com


On Tue, Jan 3, 2017 at 11:23 AM, jim ferenczi  wrote:
> Hi,
> I would like to volunteer to release 6.4. I can cut the release branch next
> Monday if everybody agrees.
>
> Jim

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-9916) Add arithmetic operations to the SelectStream

[Joel Bernstein (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-9916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joel Bernstein updated SOLR-9916:
-
Summary: Add arithmetic operations to the SelectStream  (was: Add 
arithmetic operations for the SelectStream)

> Add arithmetic operations to the SelectStream
> -
>
> Key: SOLR-9916
> URL: https://issues.apache.org/jira/browse/SOLR-9916
> Project: Solr
>  Issue Type: New Feature
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Joel Bernstein
>
> One of the things that will be needed as the SQL implementation matures is 
> the ability to do arithmetic operations. For example:
> select (a+b) from x;
> select sum(a)+sum(b) from x;
> We will need to support arithmetic operations within the Streaming API to 
> support these types of operations.
> It looks like adding arithmetic operations to the SelectStream is the best 
> place to add this functionality.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Comment Edited] (SOLR-9916) Add arithmetic operations for the SelectStream

[Joel Bernstein (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15795511#comment-15795511
 ] 

Joel Bernstein edited comment on SOLR-9916 at 1/3/17 4:58 PM:
--

One possible approach would look like this:

{code}
plus(a, b, outField)
plus(plus(a,b), c, outField)
plus(sum(a), sum(b), outField)
{code}

In the first example two field names are used to represent operands.

In the second example the first operand is a nested arithmetic operation. 

In the third example the operands are aggregate function names.

The constructors of the arithmetic operations will need to do the work to 
distinguish the different types of operands.

As part of a select expression it would like this:
{code}
select(expr, plus(a,b, outField), minus(sum(a), sum(b), outField))
{code}


For simplicity arithmetic functions can only return doubles.

Suggested initial arithmetic operations:

plus
minus,
mult,
div,
abs,
mod












was (Author: joel.bernstein):
One possible approach would look like this:

{code}
plus(a, b, outField)
plus(plus(a,b), c, outField)
plus(sum(a), sum(b), outField)
{code}

In the first example two field names are used to represent operands.

In the second example the first operand is a nested arithmetic operation. 

In the third example the operands are aggregate function names.

The constructors of the arithmetic operations will need to do the work to 
distinguish the different types of operands.

For simplicity arithmetic functions can only return doubles.

Suggested initial arithmetic operations:

plus
minus,
mult,
div,
abs,
mod











> Add arithmetic operations for the SelectStream
> --
>
> Key: SOLR-9916
> URL: https://issues.apache.org/jira/browse/SOLR-9916
> Project: Solr
>  Issue Type: New Feature
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Joel Bernstein
>
> One of the things that will be needed as the SQL implementation matures is 
> the ability to do arithmetic operations. For example:
> select (a+b) from x;
> select sum(a)+sum(b) from x;
> We will need to support arithmetic operations within the Streaming API to 
> support these types of operations.
> It looks like adding arithmetic operations to the SelectStream is the best 
> place to add this functionality.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-9916) Add arithmetic operations for the SelectStream

[Joel Bernstein (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15795511#comment-15795511
 ] 

Joel Bernstein commented on SOLR-9916:
--

One possible approach would like this:

{code}
plus(a, b, outField)
plus(plus(a,b), c, outField)
plus(sum(a), sum(b), outField)
{code}

In the first example two field names are used to represent operands.

In the second example the first operand is a nested arithmetic operation. 

In the third example the operands are aggregate function names.

The constructors of the arithmetic operations will need to do the work to 
distinguish the different types of operands.

For simplicity arithmetic functions can only return doubles.

Suggested initial arithmetic operations:

plus
minus,
mult,
div,
abs,
mod











> Add arithmetic operations for the SelectStream
> --
>
> Key: SOLR-9916
> URL: https://issues.apache.org/jira/browse/SOLR-9916
> Project: Solr
>  Issue Type: New Feature
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Joel Bernstein
>
> One of the things that will be needed as the SQL implementation matures is 
> the ability to do arithmetic operations. For example:
> select (a+b) from x;
> select sum(a)+sum(b) from x;
> We will need to support arithmetic operations within the Streaming API to 
> support these types of operations.
> It looks like adding arithmetic operations to the SelectStream is the best 
> place to add this functionality.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Comment Edited] (SOLR-9916) Add arithmetic operations for the SelectStream

[Joel Bernstein (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15795511#comment-15795511
 ] 

Joel Bernstein edited comment on SOLR-9916 at 1/3/17 4:55 PM:
--

One possible approach would look like this:

{code}
plus(a, b, outField)
plus(plus(a,b), c, outField)
plus(sum(a), sum(b), outField)
{code}

In the first example two field names are used to represent operands.

In the second example the first operand is a nested arithmetic operation. 

In the third example the operands are aggregate function names.

The constructors of the arithmetic operations will need to do the work to 
distinguish the different types of operands.

For simplicity arithmetic functions can only return doubles.

Suggested initial arithmetic operations:

plus
minus,
mult,
div,
abs,
mod












was (Author: joel.bernstein):
One possible approach would like this:

{code}
plus(a, b, outField)
plus(plus(a,b), c, outField)
plus(sum(a), sum(b), outField)
{code}

In the first example two field names are used to represent operands.

In the second example the first operand is a nested arithmetic operation. 

In the third example the operands are aggregate function names.

The constructors of the arithmetic operations will need to do the work to 
distinguish the different types of operands.

For simplicity arithmetic functions can only return doubles.

Suggested initial arithmetic operations:

plus
minus,
mult,
div,
abs,
mod











> Add arithmetic operations for the SelectStream
> --
>
> Key: SOLR-9916
> URL: https://issues.apache.org/jira/browse/SOLR-9916
> Project: Solr
>  Issue Type: New Feature
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Joel Bernstein
>
> One of the things that will be needed as the SQL implementation matures is 
> the ability to do arithmetic operations. For example:
> select (a+b) from x;
> select sum(a)+sum(b) from x;
> We will need to support arithmetic operations within the Streaming API to 
> support these types of operations.
> It looks like adding arithmetic operations to the SelectStream is the best 
> place to add this functionality.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-9916) Add arithmetic operations for the SelectStream

[Joel Bernstein (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15795468#comment-15795468
 ] 

Joel Bernstein commented on SOLR-9916:
--

I think the implementation will not be difficult. The tricky part is going to 
be getting the syntax sorted out. I'll use the comments below to suggest some 
syntax options.

> Add arithmetic operations for the SelectStream
> --
>
> Key: SOLR-9916
> URL: https://issues.apache.org/jira/browse/SOLR-9916
> Project: Solr
>  Issue Type: New Feature
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Joel Bernstein
>
> One of the things that will be needed as the SQL implementation matures is 
> the ability to do arithmetic operations. For example:
> select (a+b) from x;
> select sum(a)+sum(b) from x;
> We will need to support arithmetic operations within the Streaming API to 
> support these types of operations.
> It looks like adding arithmetic operations to the SelectStream is the best 
> place to add this functionality.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: Welcome Jim Ferenczi as a Lucene/Solr committer

[Erick Erickson]

And then he immediately volunteers to cut 6.4. What a guy!

On Tue, Jan 3, 2017 at 6:26 AM, Christine Poerschke (BLOOMBERG/
LONDON)  wrote:
> Welcome Jim!
>
> - Original Message -
> From: dev@lucene.apache.org
> To: dev@lucene.apache.org
> At: 01/01/17 10:05:18
>
> I'm pleased to announce that Jim Ferenczi has accepted the Lucene
> PMC's invitation to become a committer.
>
> Jim, it's tradition that you introduce yourself with a brief bio.
>
> Your handle "jimczi" has been added to the “lucene" LDAP group, so you
> now have commit privileges. Please test this by adding yourself to the
> committers section of the Who We Are page on the website:
>  (instructions here
> ).
>
> The ASF dev page also has lots of useful links: .
>
> Congratulations and welcome and Happy New Year,
>
> Mike McCandless
>
> http://blog.mikemccandless.com
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org
>
>
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org
>

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-9916) Add arithmetic operations for the SelectStream

[Joel Bernstein (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-9916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joel Bernstein updated SOLR-9916:
-
Description: 
One of the things that will be needed as the SQL implementation matures is the 
ability to do arithmetic operations. For example:

select (a+b) from x;

select sum(a)+sum(b) from x;

We will need to support arithmetic operations within the Streaming API to 
support these types of operations.

It looks like adding arithmetic operations to the SelectStream is the best 
place to add this functionality.


  was:
One of the things that will be needed as the SQL implementation matures is the 
ability to do arithmetic operations. For example:

select (a+b) from x;

select sum(a)+sum(b) from x;

We will need to support arithmetic operations within the Streaming API to 
support these types of operations.

It looks like adding arithmetic operations to the SelectStream is best place to 
add this functionality.



> Add arithmetic operations for the SelectStream
> --
>
> Key: SOLR-9916
> URL: https://issues.apache.org/jira/browse/SOLR-9916
> Project: Solr
>  Issue Type: New Feature
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Joel Bernstein
>
> One of the things that will be needed as the SQL implementation matures is 
> the ability to do arithmetic operations. For example:
> select (a+b) from x;
> select sum(a)+sum(b) from x;
> We will need to support arithmetic operations within the Streaming API to 
> support these types of operations.
> It looks like adding arithmetic operations to the SelectStream is the best 
> place to add this functionality.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org