[jira] [Commented] (LUCENE-7355) Leverage MultiTermAwareComponent in query parsers

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-7355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16722299#comment-16722299
 ] 

ASF GitHub Bot commented on LUCENE-7355:


GitHub user xristy opened a pull request:

https://github.com/apache/jena/pull/512

merge analyzer-regression

per LUCENE-7355 add normalize to LowerCaseKeywordAnalyzer

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/buda-base/jena JENA-1652-analyzer-regression

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/jena/pull/512.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #512


commit e63b85fdc0514de7916998ef15e4760f2449574f
Author: Code Ferret 
Date:   2018-12-15T21:41:38Z

merge analyzer-regression




> Leverage MultiTermAwareComponent in query parsers
> -
>
> Key: LUCENE-7355
> URL: https://issues.apache.org/jira/browse/LUCENE-7355
> Project: Lucene - Core
>  Issue Type: Improvement
>Reporter: Adrien Grand
>Assignee: Adrien Grand
>Priority: Minor
> Fix For: 6.2, 7.0
>
> Attachments: LUCENE-7355.patch, LUCENE-7355.patch, LUCENE-7355.patch, 
> LUCENE-7355.patch, LUCENE-7355.patch, LUCENE-7355.patch, LUCENE-7355.patch, 
> LUCENE-7355.patch
>
>
> MultiTermAwareComponent is designed to make it possible to do the right thing 
> in query parsers when in comes to analysis of multi-term queries. However, 
> since query parsers just take an analyzer and since analyzers do not 
> propagate the information about what to do for multi-term analysis, query 
> parsers cannot do the right thing out of the box.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-11617) Expose Alias Metadata CRUD in REST API

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-11617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16338646#comment-16338646
 ] 

ASF GitHub Bot commented on SOLR-11617:
---

Github user dsmiley commented on the issue:

https://github.com/apache/lucene-solr/pull/310
  
Nice docs.  Lets not provide an option to configure whitespace/trim 
handling -- configuration-itis.  IMO we should be consistent with other parts 
of Solr on what to do (e.g. props on core creation) but you seem to feel 
strongly about it so whatever.  It's not that I don't disagree with your point 
of view, it's only that I think consistency is a higher virtue.

Can you add a test please?  It could be a simple modification to an 
existing test.


> Expose Alias Metadata CRUD in REST API
> --
>
> Key: SOLR-11617
> URL: https://issues.apache.org/jira/browse/SOLR-11617
> Project: Solr
>  Issue Type: Sub-task
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: SolrCloud
>Affects Versions: master (8.0)
>Reporter: Gus Heck
>Priority: Major
> Attachments: SOLR_11617.patch, SOLR_11617.patch
>
>
> SOLR-11487 is adding Java API for metadata on aliases, this task is to expose 
> that functionality to end-users via a REST API.
> Some proposed commands, for initial discussion:
>  - *SETALIASMETA* - upsert, or delete if blank/null/white-space provided.
>  - *GETALIASMETA* - read existing alias metadata
> Given that the parent ticket to this task is going to rely on the alias 
> metadata, and I suspect a user would potentially completely break their time 
> partitioned data configuration by editing system metadata directly, we should 
> either document these commands as "use at your own risk, great 
> power/responsibility etc" or consider protecting some subset of metadata.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-11898) ConcurrentModificationException when calling org.apache.solr.core.SolrInfoBean.getMetricsSnapshot(SolrInfoBean.java:62)

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-11898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16338163#comment-16338163
 ] 

ASF GitHub Bot commented on SOLR-11898:
---

GitHub user millerjeff0 opened a pull request:

https://github.com/apache/lucene-solr/pull/312

Solr 11898

SOLR-11898 adding classes missed in SOLR-11898

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/millerjeff0/lucene-solr SOLR-11898

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/lucene-solr/pull/312.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #312


commit af2f43f3d182f46b890db467257b6480b842a16c
Author: Jeff 
Date:   2018-01-24T18:55:24Z

SOLR-11898 Updating classes to use ConcurrentHashMap.newKeySet

commit 79ceffd56cad6e34815055f408b03e030e1505c6
Author: Jeff 
Date:   2018-01-24T20:20:42Z

SOLR-11898




> ConcurrentModificationException when calling 
> org.apache.solr.core.SolrInfoBean.getMetricsSnapshot(SolrInfoBean.java:62)
> ---
>
> Key: SOLR-11898
> URL: https://issues.apache.org/jira/browse/SOLR-11898
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: metrics
>Affects Versions: 7.1
>Reporter: Jeff Miller
>Priority: Major
>  Labels: newbie
>   Original Estimate: 0.5h
>  Remaining Estimate: 0.5h
>
> SOLR-11255 Updated to use  ConcurrentHashMap.newKeySet(); for all metricNames 
> implementation but missed a few classes.
> I see this exception happening once in a while:
> Cause0: java.util.ConcurrentModificationException Cause0-StackTrace: at 
> java.util.HashMap$KeySpliterator.forEachRemaining(HashMap.java:1558) at 
> java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:580) 
> at 
> org.apache.solr.util.stats.MetricUtils.convertMetrics(MetricUtils.java:248) 
> at 
> org.apache.solr.util.stats.MetricUtils.convertMetrics(MetricUtils.java:226) 
> at org.apache.solr.core.SolrInfoBean.getMetricsSnapshot(SolrInfoBean.java:62)
>  
> Three classes weren't updated from using a HashSet for metricNames: 
> FastLRUCache, LFUCache and LRUCache



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-11873) Use time based expiration cache in all places in HdfsDirectoryFactory

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-11873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16336971#comment-16336971
 ] 

ASF GitHub Bot commented on SOLR-11873:
---

GitHub user misutoth opened a pull request:

https://github.com/apache/lucene-solr/pull/311

SOLR-11873: Use a time based expiration cache for one off HDFS FileSy…

…stem instances in all functions.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/misutoth/lucene-solr master-hdfsfix

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/lucene-solr/pull/311.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #311


commit 53ba8bd68f025aa786c40e407b29f354bd12e7f5
Author: Mihaly Toth 
Date:   2018-01-23T18:58:11Z

SOLR-11873: Use a time based expiration cache for one off HDFS FileSystem 
instances in all functions.




> Use time based expiration cache in all places in HdfsDirectoryFactory
> -
>
> Key: SOLR-11873
> URL: https://issues.apache.org/jira/browse/SOLR-11873
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: hdfs
>Affects Versions: 7.2
>Reporter: Mihaly Toth
>Priority: Major
>
> {{HdfsDirectoryFactory.exists()}} method already applies caching on 
> FileSystem objects. This is not done yet in the {{size()}} method.
> This function is eventually used when querying the core status. Each and 
> every query will use the same configuration and start from the first 
> configured HDFS NameNode. If that is down Solr will always access this down 
> node first without "learning".
> It would be nice to apply the same caching on that function too.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-11617) Expose Alias Metadata CRUD in REST API

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-11617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16336864#comment-16336864
 ] 

ASF GitHub Bot commented on SOLR-11617:
---

GitHub user fsparv opened a pull request:

https://github.com/apache/lucene-solr/pull/310

SOLR-11617

Changes as mentioned in Jira comment, plus rather than decide I provided an 
option for the user to enable whitespace values. By default this is off and 
whitespace values are treated as null, but if the user really wants to set a 
value to 3 spaces they can by passing allowWhitespaceValues=true. I'm fairly 
set against pure whitespace keys unless someone can supply up with a good 
reason for that.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/nsoft/lucene-solr SOLR-11617

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/lucene-solr/pull/310.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #310


commit 959401b9ac6bac5e4d696a750c3e5eb42b7da9f5
Author: gus 
Date:   2018-01-23T19:28:11Z

SOLR 11617 - patch 2

commit 218890d7b8c01bddabd1de2a6bbe2f1735ac238c
Author: Gus Heck 
Date:   2018-01-23T20:43:54Z

Merge branch 'master' into SOLR-11617

commit 4b4c656a107eba82ec1bfe11e8f5b14b1ea751bf
Author: Gus Heck 
Date:   2018-01-24T03:44:46Z

adjust for package changes in master




> Expose Alias Metadata CRUD in REST API
> --
>
> Key: SOLR-11617
> URL: https://issues.apache.org/jira/browse/SOLR-11617
> Project: Solr
>  Issue Type: Sub-task
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: SolrCloud
>Affects Versions: master (8.0)
>Reporter: Gus Heck
>Priority: Major
> Attachments: SOLR_11617.patch, SOLR_11617.patch
>
>
> SOLR-11487 is adding Java API for metadata on aliases, this task is to expose 
> that functionality to end-users via a REST API.
> Some proposed commands, for initial discussion:
>  - *SETALIASMETA* - upsert, or delete if blank/null/white-space provided.
>  - *GETALIASMETA* - read existing alias metadata
> Given that the parent ticket to this task is going to rely on the alias 
> metadata, and I suspect a user would potentially completely break their time 
> partitioned data configuration by editing system metadata directly, we should 
> either document these commands as "use at your own risk, great 
> power/responsibility etc" or consider protecting some subset of metadata.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-11692) SolrDispatchFilter.closeShield passes the shielded response object back to jetty making the stream unclose able

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-11692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16335373#comment-16335373
 ] 

ASF GitHub Bot commented on SOLR-11692:
---

Github user tflobbe commented on the issue:

https://github.com/apache/lucene-solr/pull/281
  
This PR was already merged (SOLR-11692) but wasn't mentioned in the commit 
message. Could you close @millerjeff0 ?


> SolrDispatchFilter.closeShield passes the shielded response object back to 
> jetty making the stream unclose able
> ---
>
> Key: SOLR-11692
> URL: https://issues.apache.org/jira/browse/SOLR-11692
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: Response Writers
>Affects Versions: 7.1
> Environment: Linux/Mac tested
>Reporter: Jeff Miller
>Assignee: David Smiley
>Priority: Minor
>  Labels: dispatchlayer, jetty, newbie, streams
> Fix For: 7.3
>
> Attachments: SOLR-11692.patch, SOLR-11692.patch
>
>   Original Estimate: 3h
>  Remaining Estimate: 3h
>
> In test mode we trigger closeShield code in SolrDispatchFilter, however there 
> are code paths where we passthrough the objects to the DefaultHandler which 
> can no longer close the response.
> Example stack trace:
> java.lang.AssertionError: Attempted close of response output stream.
> at 
> org.apache.solr.servlet.SolrDispatchFilter$2$1.close(SolrDispatchFilter.java:528)
> at org.eclipse.jetty.server.Dispatcher.commitResponse(Dispatcher.java:315)
> at org.eclipse.jetty.server.Dispatcher.forward(Dispatcher.java:279)
> at org.eclipse.jetty.server.Dispatcher.forward(Dispatcher.java:103)
> at org.eclipse.jetty.servlet.DefaultServlet.doGet(DefaultServlet.java:566)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:734)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:847)
> at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:684)
> at 
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1448)
> at 
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:385)
> at 
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:326)
> at 
> searchserver.filter.SfdcDispatchFilter.doFilter(SfdcDispatchFilter.java:204)
> at 
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)
> at 
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:455)
> at 
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
> at 
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557)
> at 
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
> at 
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1075)
> at 
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:384)
> at 
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
> at 
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1009)
> at 
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
> at 
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255)
> at 
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154)
> at 
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
> at org.eclipse.jetty.server.Server.handle(Server.java:370)
> at 
> org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489)
> at 
> org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:949)
> at 
> org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1011)
> at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:644)
> at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
> at 
> org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
> at 
> org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:668)
> at 
> org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52)
> at 
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
> at 
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
> at java.lang.Thread.run(Thread.java:745)
> Related JIRA: SOLR-8933



--
This message was sent by Atlassian JIRA

[jira] [Commented] (SOLR-11459) AddUpdateCommand#prevVersion is not cleared which may lead to problem for in-place updates of non existed documents

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-11459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16328178#comment-16328178
 ] 

ASF GitHub Bot commented on SOLR-11459:
---

GitHub user werder06 opened a pull request:

https://github.com/apache/lucene-solr/pull/307

SOLR-11459 Clear AddUpdateCommand#prevVersion to fix in-place updates…

… for non existed documents

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/werder06/lucene-solr master

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/lucene-solr/pull/307.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #307


commit 38a689190e80860e9efcfc6bc121f86656801044
Author: Andrey 
Date:   2018-01-17T02:17:54Z

SOLR-11459 Clear AddUpdateCommand#prevVersion to fix in-place updates for 
non existed documents




> AddUpdateCommand#prevVersion is not cleared which may lead to problem for 
> in-place updates of non existed documents
> ---
>
> Key: SOLR-11459
> URL: https://issues.apache.org/jira/browse/SOLR-11459
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: SolrCloud
>Affects Versions: 7.0
>Reporter: Andrey Kudryavtsev
>Assignee: Ishan Chattopadhyaya
>Priority: Minor
>
> I have a 1_shard / *m*_replicas SolrCloud cluster with Solr 6.6.0 and run 
> batches of 5 - 10k in-place updates from time to time. 
> Once I noticed that job "hangs" - it started and couldn't finish for a a 
> while.
> Logs were full of messages like:
> {code} Missing update, on which current in-place update depends on, hasn't 
> arrived. id=__, looking for version=___, last found version=0"  {code}
> {code} 
> Tried to fetch document ___ from the leader, but the leader says document has 
> been deleted. Deleting the document here and skipping this update: Last found 
> version: 0, was looking for: ___",24,0,"but the leader says document has been 
> deleted. Deleting the document here and skipping this update: Last found 
> version: 0
> {code}
> Further analysis shows that:
> * There are 100-500 updates for non-existed documents among other updates 
> (something that I have to deal with)
> * Leader receives bunch of updates and executes this updates one by one. 
> {{JavabinLoader}} which is used by processing documents reuses same instance 
> of {{AddUpdateCommand}} for every update and just [clearing its state at the 
> end|https://github.com/apache/lucene-solr/blob/e2521b2a8baabdaf43b92192588f51e042d21e97/solr/core/src/java/org/apache/solr/handler/loader/JavabinLoader.java#L99].
>  Field [AddUpdateCommand#prevVersion| 
> https://github.com/apache/lucene-solr/blob/6396cb759f8c799f381b0730636fa412761030ce/solr/core/src/java/org/apache/solr/update/AddUpdateCommand.java#L76]
>  is not cleared.   
> * In case of update is in-place update, but specified document does not 
> exist, this update is processed as a regular atomic update (i.e. new doc is 
> created), but {{prevVersion}} is used as a {{distrib.inplace.prevversion}} 
> parameter in sequential calls to every slave in DistributedUpdateProcessor. 
> {{prevVersion}} wasn't cleared, so it may contain version from previous 
> processed update.
> * Slaves checks it's own version of documents which is 0 (cause doc does not 
> exist), slave thinks that some updates were missed and spends 5 seconds in 
> [DistributedUpdateProcessor#waitForDependentUpdates|https://github.com/apache/lucene-solr/blob/e2521b2a8baabdaf43b92192588f51e042d21e97/solr/core/src/java/org/apache/solr/handler/loader/JavabinLoader.java#L99]
>  waiting for missed updates (no luck) and also tries to get "correct" version 
> from leader (no luck as well) 
> * So update for non existed document costs *m* * 5 sec each
> I workarounded this by explicit check of doc existence, but it probably 
> should be fixed.
> Obviously first guess is that  prevVersion should be cleared in 
> {{AddUpdateCommand#clear}}, but have no clue how to test it.
> {code}
> +++ solr/core/src/java/org/apache/solr/update/AddUpdateCommand.java   
> (revision )
> @@ -78,6 +78,7 @@
>   updateTerm = null;
>   isLastDocInBatch = false;
>   version = 0;
> + prevVersion = -1;
> }
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-11853) Solr installer: Make check for used tool "service" compatible with SuSE Linux like OpenSuse or SLES

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-11853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16325426#comment-16325426
 ] 

ASF GitHub Bot commented on SOLR-11853:
---

GitHub user Mandalka opened a pull request:

https://github.com/apache/lucene-solr/pull/305

SOLR-11853: Make check for used tool "service" compatible with SuSE Linux

On current SuSE Linux releases like SLES or OpenSuSE the Solr installer 
stops with the error message "Script requires the 'service' command".

This happens because before installation the installer checks if the used 
command "service" exists by its option "service --version".

The command line option "--version" doesn't exist for "service" on current 
SuSE Linux stable releases.

Since the command "service" is there and has an option "--help", this 
option can be used as additional fallback.

So in the pull request i extended the check with "service --help" as second 
check / fallback before printing this error and exiting.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/Mandalka/lucene-solr master

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/lucene-solr/pull/305.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #305


commit 0aa573740b55027192440fc5997a93945ca9b9b3
Author: Markus Mandalka 
Date:   2018-01-14T02:14:47Z

Make check for used tool 'service' compatible with SuSE Linux like OpenSuse 
or SLES




> Solr installer: Make check for used tool "service" compatible with SuSE Linux 
> like OpenSuse or SLES
> ---
>
> Key: SOLR-11853
> URL: https://issues.apache.org/jira/browse/SOLR-11853
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>Affects Versions: 7.2
> Environment: SuSE Linux (SLES & OpenSuSE)
>Reporter: Markus Mandalka
>Priority: Minor
>  Labels: github-import, newbie
> Fix For: master (8.0), 7.3, 7.2.2
>
>
> On current SuSE Linux releases like SLES or OpenSuSE the Solr installer stops 
> with the error message "Script requires the 'service' command".
> This happens because before installation the installer checks if the used 
> command "service" exists by its option "service --version".
> The command line option "--version" doesn't exist for "service" on current 
> SuSE Linux stable releases.
> Since the command "service" is there and has an option "--help", this option 
> can be used as additional fallback.
> So in the pull request i extended the check with "service --help" as second 
> check / fallback before printing this error and exiting.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-11624) collection creation should not also overwrite/delete any configset but it can!

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-11624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16325318#comment-16325318
 ] 

ASF GitHub Bot commented on SOLR-11624:
---

Github user abhidemon closed the pull request at:

https://github.com/apache/lucene-solr/pull/297


> collection creation should not also overwrite/delete any configset but it can!
> --
>
> Key: SOLR-11624
> URL: https://issues.apache.org/jira/browse/SOLR-11624
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>Affects Versions: 7.2
>Reporter: Erick Erickson
>Assignee: Ishan Chattopadhyaya
> Attachments: SOLR-11624-2.patch, SOLR-11624.3.patch, 
> SOLR-11624.4.patch, SOLR-11624.patch, SOLR-11624.patch
>
>
> Looks like a problem that crept in when we changed the _default configset 
> stuff.
> setup:
> upload a configset named "wiki"
> collections?action=CREATE=wiki&.
> My custom configset "wiki" gets overwritten by _default and then used by the 
> "wiki" collection.
> Assigning to myself only because it really needs to be fixed IMO and I don't 
> want to lose track of it. Anyone else please feel free to take it.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-11722) API to create a Time Routed Alias and first collection

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-11722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16324348#comment-16324348
 ] 

ASF GitHub Bot commented on SOLR-11722:
---

Github user nsoft commented on a diff in the pull request:

https://github.com/apache/lucene-solr/pull/304#discussion_r161294218
  
--- Diff: 
solr/core/src/java/org/apache/solr/handler/admin/CollectionsHandler.java ---
@@ -476,6 +451,31 @@ private static void 
addStatusToResponse(NamedList results, RequestStatus
   SolrIdentifierValidator.validateAliasName(req.getParams().get(NAME));
   return req.getParams().required().getAll(null, NAME, "collections");
 }),
+CREATEROUTEDALIAS_OP(CREATEROUTEDALIAS, (req, rsp, h) -> {
+  String alias = req.getParams().get(NAME);
+  SolrIdentifierValidator.validateAliasName(alias);
+  Map params = req.getParams().required()
+  .getAll(null, REQUIRED_ROUTING_PARAMS.toArray(new 
String[REQUIRED_ROUTING_PARAMS.size()]));
+  req.getParams().getAll(params, NONREQUIRED_ROUTING_PARAMS);
+  // subset the params to reuse the collection creation/parsing code
+  ModifiableSolrParams collectionParams = 
extractPrefixedParams("create-collection.", req.getParams());
+  if (collectionParams.get(NAME) != null) {
+SolrException solrException = new SolrException(BAD_REQUEST, 
"routed aliases calculate names for their " +
+"dependent collections, you cannot specify the name.");
+log.error("Could not create routed alias",solrException);
+throw solrException;
+  }
+  SolrParams v1Params = convertToV1WhenRequired(req, collectionParams);
+
+  // We need to add this temporary name just to pass validation.
--- End diff --

ah that's actually checked here: 
https://github.com/apache/lucene-solr/pull/304/files#diff-3fe6a8aeb14a57e63507fa17f8346771R207,
 but It could be moved to this class (or done both places)


> API to create a Time Routed Alias and first collection
> --
>
> Key: SOLR-11722
> URL: https://issues.apache.org/jira/browse/SOLR-11722
> Project: Solr
>  Issue Type: Sub-task
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: SolrCloud
>Reporter: David Smiley
> Attachments: SOLR-11722.patch, SOLR-11722.patch
>
>
> This issue is about creating a single API command to create a "Time Routed 
> Alias" along with its first collection.  Need to decide what endpoint URL it 
> is and parameters.
> Perhaps in v2 it'd be {{/api/collections?command=create-routed-alias}} or 
> alternatively piggy-back off of command=create-alias but we add more options, 
> perhaps with a prefix like "router"?
> Inputs:
> * alias name
> * misc collection creation metadata (e.g. config, numShards, ...) perhaps in 
> this context with a prefix like "collection."
> * metadata for TimeRoutedAliasUpdateProcessor, currently: router.field
> * date specifier for first collection; can include "date math".
> We'll certainly add more options as future features unfold.
> I believe the collection needs to be created first (referring to the alias 
> name via a core property), and then the alias pointing to it which demands 
> collections exist first.  When figuring the collection name, you'll need to 
> reference the format in TimeRoutedAliasUpdateProcessor.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-11722) API to create a Time Routed Alias and first collection

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-11722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16323976#comment-16323976
 ] 

ASF GitHub Bot commented on SOLR-11722:
---

Github user dsmiley commented on a diff in the pull request:

https://github.com/apache/lucene-solr/pull/304#discussion_r161226045
  
--- Diff: 
solr/core/src/java/org/apache/solr/handler/admin/CollectionsHandler.java ---
@@ -476,6 +451,31 @@ private static void 
addStatusToResponse(NamedList results, RequestStatus
   SolrIdentifierValidator.validateAliasName(req.getParams().get(NAME));
   return req.getParams().required().getAll(null, NAME, "collections");
 }),
+CREATEROUTEDALIAS_OP(CREATEROUTEDALIAS, (req, rsp, h) -> {
+  String alias = req.getParams().get(NAME);
+  SolrIdentifierValidator.validateAliasName(alias);
+  Map params = req.getParams().required()
+  .getAll(null, REQUIRED_ROUTING_PARAMS.toArray(new 
String[REQUIRED_ROUTING_PARAMS.size()]));
+  req.getParams().getAll(params, NONREQUIRED_ROUTING_PARAMS);
+  // subset the params to reuse the collection creation/parsing code
+  ModifiableSolrParams collectionParams = 
extractPrefixedParams("create-collection.", req.getParams());
+  if (collectionParams.get(NAME) != null) {
+SolrException solrException = new SolrException(BAD_REQUEST, 
"routed aliases calculate names for their " +
+"dependent collections, you cannot specify the name.");
+log.error("Could not create routed alias",solrException);
+throw solrException;
+  }
+  SolrParams v1Params = convertToV1WhenRequired(req, collectionParams);
+
+  // We need to add this temporary name just to pass validation.
--- End diff --

I wasn't clear; I don't have any issue with this part of the patch. I'm 
suggesting _adding_ a check that we require the configSet.  I needed to click 
on some line to insert my comment; maybe I should have put my review comment on 
the blank line.


> API to create a Time Routed Alias and first collection
> --
>
> Key: SOLR-11722
> URL: https://issues.apache.org/jira/browse/SOLR-11722
> Project: Solr
>  Issue Type: Sub-task
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: SolrCloud
>Reporter: David Smiley
> Attachments: SOLR-11722.patch, SOLR-11722.patch
>
>
> This issue is about creating a single API command to create a "Time Routed 
> Alias" along with its first collection.  Need to decide what endpoint URL it 
> is and parameters.
> Perhaps in v2 it'd be {{/api/collections?command=create-routed-alias}} or 
> alternatively piggy-back off of command=create-alias but we add more options, 
> perhaps with a prefix like "router"?
> Inputs:
> * alias name
> * misc collection creation metadata (e.g. config, numShards, ...) perhaps in 
> this context with a prefix like "collection."
> * metadata for TimeRoutedAliasUpdateProcessor, currently: router.field
> * date specifier for first collection; can include "date math".
> We'll certainly add more options as future features unfold.
> I believe the collection needs to be created first (referring to the alias 
> name via a core property), and then the alias pointing to it which demands 
> collections exist first.  When figuring the collection name, you'll need to 
> reference the format in TimeRoutedAliasUpdateProcessor.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-11722) API to create a Time Routed Alias and first collection

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-11722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16323968#comment-16323968
 ] 

ASF GitHub Bot commented on SOLR-11722:
---

Github user dsmiley commented on a diff in the pull request:

https://github.com/apache/lucene-solr/pull/304#discussion_r161223342
  
--- Diff: solr/core/src/java/org/apache/solr/cloud/CreateAliasCmd.java ---
@@ -30,13 +44,101 @@
 import org.apache.solr.common.cloud.ZkStateReader;
 import org.apache.solr.common.util.NamedList;
 import org.apache.solr.common.util.StrUtils;
+import org.apache.solr.update.processor.TimeRoutedAliasUpdateProcessor;
+import org.apache.solr.util.DateMathParser;
+import org.apache.solr.util.TimeZoneUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
 
+import static java.time.format.DateTimeFormatter.ISO_INSTANT;
+import static 
org.apache.solr.cloud.OverseerCollectionMessageHandler.COLL_CONF;
+import static org.apache.solr.common.SolrException.ErrorCode.BAD_REQUEST;
 import static org.apache.solr.common.params.CommonParams.NAME;
+import static org.apache.solr.common.params.CommonParams.TZ;
+import static 
org.apache.solr.handler.admin.CollectionsHandler.ROUTED_ALIAS_COLLECTION_PROP_PFX;
+import static 
org.apache.solr.update.processor.TimeRoutedAliasUpdateProcessor.DATE_TIME_FORMATTER;
 
 
 public class CreateAliasCmd implements Cmd {
+
+  private static final Logger log = 
LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+  public static final String ROUTING_TYPE = "router.name";
+  public static final String ROUTING_FIELD = "router.field";
+  public static final String ROUTING_INCREMENT = "router.interval";
+  public static final String ROUTING_MAX_FUTURE = "router.max-future-ms";
+  public static final String START = "start";
+  // Collection constants should all reflect names in the v2 structured 
input for this command, not v1
+  // names used for CREATE
+  public static final String CREATE_COLLECTION_CONFIG = 
"create-collection.config";
--- End diff --

My concern isn't just about the duplication of the Strings/names, it's 
about the "set" of them here.  Even if we could refer to a constant in some 
other class accessible to SolrJ, it would still be a maintenance burden to 
refer to the set of those that exist since it's redundant with the collection 
creation code.  So can we eliminate this entirely (I'd love that) or failing 
that have a single place where the set of these strings are defined so we don't 
even need to list them in CreateAliasCmd.


> API to create a Time Routed Alias and first collection
> --
>
> Key: SOLR-11722
> URL: https://issues.apache.org/jira/browse/SOLR-11722
> Project: Solr
>  Issue Type: Sub-task
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: SolrCloud
>Reporter: David Smiley
> Attachments: SOLR-11722.patch, SOLR-11722.patch
>
>
> This issue is about creating a single API command to create a "Time Routed 
> Alias" along with its first collection.  Need to decide what endpoint URL it 
> is and parameters.
> Perhaps in v2 it'd be {{/api/collections?command=create-routed-alias}} or 
> alternatively piggy-back off of command=create-alias but we add more options, 
> perhaps with a prefix like "router"?
> Inputs:
> * alias name
> * misc collection creation metadata (e.g. config, numShards, ...) perhaps in 
> this context with a prefix like "collection."
> * metadata for TimeRoutedAliasUpdateProcessor, currently: router.field
> * date specifier for first collection; can include "date math".
> We'll certainly add more options as future features unfold.
> I believe the collection needs to be created first (referring to the alias 
> name via a core property), and then the alias pointing to it which demands 
> collections exist first.  When figuring the collection name, you'll need to 
> reference the format in TimeRoutedAliasUpdateProcessor.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-11722) API to create a Time Routed Alias and first collection

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-11722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16323657#comment-16323657
 ] 

ASF GitHub Bot commented on SOLR-11722:
---

Github user fsparv commented on a diff in the pull request:

https://github.com/apache/lucene-solr/pull/304#discussion_r161155289
  
--- Diff: 
solr/core/src/java/org/apache/solr/update/processor/TimeRoutedAliasUpdateProcessor.java
 ---
@@ -99,6 +101,8 @@
   .parseDefaulting(ChronoField.HOUR_OF_DAY, 0)
   .parseDefaulting(ChronoField.MINUTE_OF_HOUR, 0)
   .parseDefaulting(ChronoField.SECOND_OF_MINUTE, 0)
+  // Setting a timezone here is fine as a default, but generally need 
to clone with user's timezone, so that
+  // truncation of milliseconds is consistent.
--- End diff --

removed in update


> API to create a Time Routed Alias and first collection
> --
>
> Key: SOLR-11722
> URL: https://issues.apache.org/jira/browse/SOLR-11722
> Project: Solr
>  Issue Type: Sub-task
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: SolrCloud
>Reporter: David Smiley
> Attachments: SOLR-11722.patch, SOLR-11722.patch
>
>
> This issue is about creating a single API command to create a "Time Routed 
> Alias" along with its first collection.  Need to decide what endpoint URL it 
> is and parameters.
> Perhaps in v2 it'd be {{/api/collections?command=create-routed-alias}} or 
> alternatively piggy-back off of command=create-alias but we add more options, 
> perhaps with a prefix like "router"?
> Inputs:
> * alias name
> * misc collection creation metadata (e.g. config, numShards, ...) perhaps in 
> this context with a prefix like "collection."
> * metadata for TimeRoutedAliasUpdateProcessor, currently: router.field
> * date specifier for first collection; can include "date math".
> We'll certainly add more options as future features unfold.
> I believe the collection needs to be created first (referring to the alias 
> name via a core property), and then the alias pointing to it which demands 
> collections exist first.  When figuring the collection name, you'll need to 
> reference the format in TimeRoutedAliasUpdateProcessor.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-11722) API to create a Time Routed Alias and first collection

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-11722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16323659#comment-16323659
 ] 

ASF GitHub Bot commented on SOLR-11722:
---

Github user fsparv commented on a diff in the pull request:

https://github.com/apache/lucene-solr/pull/304#discussion_r161155141
  
--- Diff: solr/core/src/java/org/apache/solr/cloud/CreateAliasCmd.java ---
@@ -68,34 +249,100 @@ public void call(ClusterState state, ZkNodeProps 
message, NamedList results)
 Thread.sleep(100);
   }
 
+  private Map buildAliasMap(String routedField, String 
routingType, String tz, String increment, String maxFutureMs, ZkNodeProps 
collectionProps) {
+Map properties = collectionProps.getProperties();
+Map cleanMap = properties.entrySet().stream()
+.filter(stringObjectEntry ->
+!"fromApi".equals(stringObjectEntry.getKey())
+&& !"stateFormat".equals(stringObjectEntry.getKey())
+&& !"name".equals(stringObjectEntry.getKey()))
+.collect(Collectors.toMap((e) -> "collection-create." + 
e.getKey(), e -> String.valueOf(e.getValue(;
+cleanMap.put(ROUTING_FIELD, routedField);
+cleanMap.put(ROUTING_TYPE, routingType);
+cleanMap.put(ROUTING_INCREMENT, increment);
+cleanMap.put(ROUTING_MAX_FUTURE, maxFutureMs);
+cleanMap.put(TZ, tz);
+return cleanMap;
+  }
+
+  private Instant validateStart(TimeZone zone, DateTimeFormatter fmt, 
String start) {
--- End diff --

checking the ms was the main point, the format and reparse was meant to 
trap any timezone/formatter craziness, which mostly goes away with the 
collection creation. Simplified, in update


> API to create a Time Routed Alias and first collection
> --
>
> Key: SOLR-11722
> URL: https://issues.apache.org/jira/browse/SOLR-11722
> Project: Solr
>  Issue Type: Sub-task
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: SolrCloud
>Reporter: David Smiley
> Attachments: SOLR-11722.patch, SOLR-11722.patch
>
>
> This issue is about creating a single API command to create a "Time Routed 
> Alias" along with its first collection.  Need to decide what endpoint URL it 
> is and parameters.
> Perhaps in v2 it'd be {{/api/collections?command=create-routed-alias}} or 
> alternatively piggy-back off of command=create-alias but we add more options, 
> perhaps with a prefix like "router"?
> Inputs:
> * alias name
> * misc collection creation metadata (e.g. config, numShards, ...) perhaps in 
> this context with a prefix like "collection."
> * metadata for TimeRoutedAliasUpdateProcessor, currently: router.field
> * date specifier for first collection; can include "date math".
> We'll certainly add more options as future features unfold.
> I believe the collection needs to be created first (referring to the alias 
> name via a core property), and then the alias pointing to it which demands 
> collections exist first.  When figuring the collection name, you'll need to 
> reference the format in TimeRoutedAliasUpdateProcessor.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-11722) API to create a Time Routed Alias and first collection

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-11722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16323656#comment-16323656
 ] 

ASF GitHub Bot commented on SOLR-11722:
---

Github user fsparv commented on a diff in the pull request:

https://github.com/apache/lucene-solr/pull/304#discussion_r161155263
  
--- Diff: 
solr/core/src/java/org/apache/solr/handler/admin/CollectionsHandler.java ---
@@ -962,6 +982,76 @@ public static CollectionOperation get(CollectionAction 
action) {
 }
   }
 
+  private static SolrParams convertToV1WhenRequired(SolrQueryRequest req, 
ModifiableSolrParams params) {
+SolrParams v1Params = params; // (maybe...)
+
+// in the v2 world we get a data map based on the json request, from 
the CommandOperation associated
+// with the request, so locate that if we can.. if we find it we have 
to translate the v2 request
+// properties to v1 params, otherwise we're already good to go.
+List cmds = req.getCommands(true);
+if (cmds.size() > 1) {
+  // todo: not sure if this is the right thing to do here, but also 
not sure what to do if there is more than one...
+  throw new SolrException(BAD_REQUEST, "Only one command is allowed 
when creating a routed alias");
+}
+CommandOperation c = cmds.size() == 0 ? null : cmds.get(0);
+if (c != null) {  // v2 api, do conversion to v1
+  v1Params = new 
BaseHandlerApiSupport.V2ToV1SolrParams(CollectionApiMapping.Meta.CREATE_COLLECTION,
+  req.getPathTemplateValues(), true, params,
+  new CommandOperation("create", 
c.getDataMap().get("create-collection")));
+}
+return v1Params;
+  }
+
+  private static Map 
parseColletionCreationProps(CollectionsHandler h, SolrParams params, String 
prefix)
--- End diff --

fixed in update


> API to create a Time Routed Alias and first collection
> --
>
> Key: SOLR-11722
> URL: https://issues.apache.org/jira/browse/SOLR-11722
> Project: Solr
>  Issue Type: Sub-task
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: SolrCloud
>Reporter: David Smiley
> Attachments: SOLR-11722.patch, SOLR-11722.patch
>
>
> This issue is about creating a single API command to create a "Time Routed 
> Alias" along with its first collection.  Need to decide what endpoint URL it 
> is and parameters.
> Perhaps in v2 it'd be {{/api/collections?command=create-routed-alias}} or 
> alternatively piggy-back off of command=create-alias but we add more options, 
> perhaps with a prefix like "router"?
> Inputs:
> * alias name
> * misc collection creation metadata (e.g. config, numShards, ...) perhaps in 
> this context with a prefix like "collection."
> * metadata for TimeRoutedAliasUpdateProcessor, currently: router.field
> * date specifier for first collection; can include "date math".
> We'll certainly add more options as future features unfold.
> I believe the collection needs to be created first (referring to the alias 
> name via a core property), and then the alias pointing to it which demands 
> collections exist first.  When figuring the collection name, you'll need to 
> reference the format in TimeRoutedAliasUpdateProcessor.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-11722) API to create a Time Routed Alias and first collection

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-11722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16323658#comment-16323658
 ] 

ASF GitHub Bot commented on SOLR-11722:
---

Github user fsparv commented on a diff in the pull request:

https://github.com/apache/lucene-solr/pull/304#discussion_r161155153
  
--- Diff: solr/core/src/java/org/apache/solr/cloud/CreateAliasCmd.java ---
@@ -68,34 +249,100 @@ public void call(ClusterState state, ZkNodeProps 
message, NamedList results)
 Thread.sleep(100);
   }
 
+  private Map buildAliasMap(String routedField, String 
routingType, String tz, String increment, String maxFutureMs, ZkNodeProps 
collectionProps) {
+Map properties = collectionProps.getProperties();
+Map cleanMap = properties.entrySet().stream()
+.filter(stringObjectEntry ->
+!"fromApi".equals(stringObjectEntry.getKey())
+&& !"stateFormat".equals(stringObjectEntry.getKey())
+&& !"name".equals(stringObjectEntry.getKey()))
+.collect(Collectors.toMap((e) -> "collection-create." + 
e.getKey(), e -> String.valueOf(e.getValue(;
+cleanMap.put(ROUTING_FIELD, routedField);
+cleanMap.put(ROUTING_TYPE, routingType);
+cleanMap.put(ROUTING_INCREMENT, increment);
+cleanMap.put(ROUTING_MAX_FUTURE, maxFutureMs);
+cleanMap.put(TZ, tz);
+return cleanMap;
+  }
+
+  private Instant validateStart(TimeZone zone, DateTimeFormatter fmt, 
String start) {
+// This is the normal/easy case, if we can get away with this great!
+TemporalAccessor startTime = attemptTimeStampParsing(start, 
zone.toZoneId());
+if (startTime == null) {
+  // No luck, they gave us either date math, or garbage, so we have to 
do more work to figure out which and
+  // to make sure it's valid date math and that it doesn't encode any 
millisecond precision.
+  ZonedDateTime now = 
ZonedDateTime.now(zone.toZoneId()).truncatedTo(ChronoUnit.MILLIS);
+  try {
+Date date = DateMathParser.parseMath(Date.from(now.toInstant()), 
start);
+String reformatted = 
fmt.format(date.toInstant().truncatedTo(ChronoUnit.MILLIS));
+Date reparse = 
Date.from(Instant.from(DATE_TIME_FORMATTER.parse(reformatted)));
+if (!reparse.equals(date)) {
+  throw new SolrException(BAD_REQUEST,
+  "Formatted time did not have the same milliseconds as 
original: " + date.getTime() + " vs. " +
+  reparse.getTime() + " This indicates that you used date 
math that includes milliseconds. " +
+  "(Hint: 'NOW' used without rounding always has this 
problem)" );
+}
+return date.toInstant();
+  } catch (SolrException e) {
+throw new SolrException(BAD_REQUEST,
+"Start Time for the first collection must be a timestamp of 
the format -MM-dd_HH_mm_ss, " +
+"or a valid date math expression not containing specific 
milliseconds", e);
+  }
+}
+return Instant.from(startTime);
+  }
+
+  private TemporalAccessor attemptTimeStampParsing(String start, ZoneId 
zone) {
+try {
+  DATE_TIME_FORMATTER.withZone(zone);
+  return DATE_TIME_FORMATTER.parse(start);
+} catch (DateTimeParseException e) {
+  return null;
+}
+  }
+
+  private boolean anyRoutingParams(ZkNodeProps message) {
+
+return message.containsKey(ROUTING_FIELD) || 
message.containsKey(ROUTING_TYPE) || message.containsKey(START)
+|| message.containsKey(ROUTING_INCREMENT) || 
message.containsKey(TZ);
+  }
+
   private void validateAllCollectionsExistAndNoDups(List 
collectionList, ZkStateReader zkStateReader) {
 final String collectionStr = StrUtils.join(collectionList, ',');
 
 if (new HashSet<>(collectionList).size() != collectionList.size()) {
-  throw new SolrException(SolrException.ErrorCode.BAD_REQUEST,
+  throw new SolrException(BAD_REQUEST,
   String.format(Locale.ROOT,  "Can't create collection alias for 
collections='%s', since it contains duplicates", collectionStr));
 }
 ClusterState clusterState = zkStateReader.getClusterState();
 Set aliasNames = 
zkStateReader.getAliases().getCollectionAliasListMap().keySet();
 for (String collection : collectionList) {
   if (clusterState.getCollectionOrNull(collection) == null && 
!aliasNames.contains(collection)) {
-throw new SolrException(SolrException.ErrorCode.BAD_REQUEST,
+throw new SolrException(BAD_REQUEST,
 String.format(Locale.ROOT,  "Can't create collection alias for 
collections='%s', '%s' 

[jira] [Commented] (SOLR-11722) API to create a Time Routed Alias and first collection

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-11722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16323649#comment-16323649
 ] 

ASF GitHub Bot commented on SOLR-11722:
---

Github user fsparv commented on a diff in the pull request:

https://github.com/apache/lucene-solr/pull/304#discussion_r161155221
  
--- Diff: 
solr/core/src/java/org/apache/solr/handler/admin/CollectionsHandler.java ---
@@ -476,6 +451,31 @@ private static void 
addStatusToResponse(NamedList results, RequestStatus
   SolrIdentifierValidator.validateAliasName(req.getParams().get(NAME));
   return req.getParams().required().getAll(null, NAME, "collections");
 }),
+CREATEROUTEDALIAS_OP(CREATEROUTEDALIAS, (req, rsp, h) -> {
+  String alias = req.getParams().get(NAME);
+  SolrIdentifierValidator.validateAliasName(alias);
+  Map params = req.getParams().required()
+  .getAll(null, REQUIRED_ROUTING_PARAMS.toArray(new 
String[REQUIRED_ROUTING_PARAMS.size()]));
+  req.getParams().getAll(params, NONREQUIRED_ROUTING_PARAMS);
+  // subset the params to reuse the collection creation/parsing code
+  ModifiableSolrParams collectionParams = 
extractPrefixedParams("create-collection.", req.getParams());
+  if (collectionParams.get(NAME) != null) {
+SolrException solrException = new SolrException(BAD_REQUEST, 
"routed aliases calculate names for their " +
+"dependent collections, you cannot specify the name.");
+log.error("Could not create routed alias",solrException);
--- End diff --

I think I was finding that in running the tests it wasn't always showing 
up. I can take it out. Removed here and in cmd class in update


> API to create a Time Routed Alias and first collection
> --
>
> Key: SOLR-11722
> URL: https://issues.apache.org/jira/browse/SOLR-11722
> Project: Solr
>  Issue Type: Sub-task
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: SolrCloud
>Reporter: David Smiley
> Attachments: SOLR-11722.patch, SOLR-11722.patch
>
>
> This issue is about creating a single API command to create a "Time Routed 
> Alias" along with its first collection.  Need to decide what endpoint URL it 
> is and parameters.
> Perhaps in v2 it'd be {{/api/collections?command=create-routed-alias}} or 
> alternatively piggy-back off of command=create-alias but we add more options, 
> perhaps with a prefix like "router"?
> Inputs:
> * alias name
> * misc collection creation metadata (e.g. config, numShards, ...) perhaps in 
> this context with a prefix like "collection."
> * metadata for TimeRoutedAliasUpdateProcessor, currently: router.field
> * date specifier for first collection; can include "date math".
> We'll certainly add more options as future features unfold.
> I believe the collection needs to be created first (referring to the alias 
> name via a core property), and then the alias pointing to it which demands 
> collections exist first.  When figuring the collection name, you'll need to 
> reference the format in TimeRoutedAliasUpdateProcessor.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-11722) API to create a Time Routed Alias and first collection

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-11722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16323653#comment-16323653
 ] 

ASF GitHub Bot commented on SOLR-11722:
---

Github user fsparv commented on a diff in the pull request:

https://github.com/apache/lucene-solr/pull/304#discussion_r161155184
  
--- Diff: 
solr/core/src/java/org/apache/solr/cloud/OverseerCollectionMessageHandler.java 
---
@@ -218,6 +218,7 @@ public OverseerCollectionMessageHandler(ZkStateReader 
zkStateReader, String myId
 .put(RELOAD, this::reloadCollection)
 .put(DELETE, new DeleteCollectionCmd(this))
 .put(CREATEALIAS, new CreateAliasCmd(this))
+.put(CREATEROUTEDALIAS, new CreateAliasCmd(this))
--- End diff --

Seemed like a good idea at the time :) As you like it.


> API to create a Time Routed Alias and first collection
> --
>
> Key: SOLR-11722
> URL: https://issues.apache.org/jira/browse/SOLR-11722
> Project: Solr
>  Issue Type: Sub-task
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: SolrCloud
>Reporter: David Smiley
> Attachments: SOLR-11722.patch, SOLR-11722.patch
>
>
> This issue is about creating a single API command to create a "Time Routed 
> Alias" along with its first collection.  Need to decide what endpoint URL it 
> is and parameters.
> Perhaps in v2 it'd be {{/api/collections?command=create-routed-alias}} or 
> alternatively piggy-back off of command=create-alias but we add more options, 
> perhaps with a prefix like "router"?
> Inputs:
> * alias name
> * misc collection creation metadata (e.g. config, numShards, ...) perhaps in 
> this context with a prefix like "collection."
> * metadata for TimeRoutedAliasUpdateProcessor, currently: router.field
> * date specifier for first collection; can include "date math".
> We'll certainly add more options as future features unfold.
> I believe the collection needs to be created first (referring to the alias 
> name via a core property), and then the alias pointing to it which demands 
> collections exist first.  When figuring the collection name, you'll need to 
> reference the format in TimeRoutedAliasUpdateProcessor.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-11722) API to create a Time Routed Alias and first collection

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-11722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16323650#comment-16323650
 ] 

ASF GitHub Bot commented on SOLR-11722:
---

Github user fsparv commented on a diff in the pull request:

https://github.com/apache/lucene-solr/pull/304#discussion_r161155243
  
--- Diff: 
solr/core/src/java/org/apache/solr/handler/admin/CollectionsHandler.java ---
@@ -476,6 +451,31 @@ private static void 
addStatusToResponse(NamedList results, RequestStatus
   SolrIdentifierValidator.validateAliasName(req.getParams().get(NAME));
   return req.getParams().required().getAll(null, NAME, "collections");
 }),
+CREATEROUTEDALIAS_OP(CREATEROUTEDALIAS, (req, rsp, h) -> {
+  String alias = req.getParams().get(NAME);
+  SolrIdentifierValidator.validateAliasName(alias);
+  Map params = req.getParams().required()
+  .getAll(null, REQUIRED_ROUTING_PARAMS.toArray(new 
String[REQUIRED_ROUTING_PARAMS.size()]));
+  req.getParams().getAll(params, NONREQUIRED_ROUTING_PARAMS);
+  // subset the params to reuse the collection creation/parsing code
+  ModifiableSolrParams collectionParams = 
extractPrefixedParams("create-collection.", req.getParams());
+  if (collectionParams.get(NAME) != null) {
+SolrException solrException = new SolrException(BAD_REQUEST, 
"routed aliases calculate names for their " +
+"dependent collections, you cannot specify the name.");
+log.error("Could not create routed alias",solrException);
+throw solrException;
+  }
+  SolrParams v1Params = convertToV1WhenRequired(req, collectionParams);
+
+  // We need to add this temporary name just to pass validation.
--- End diff --

The name on this line that I am TMP-ing isn't the collection config name, 
it's the name of the collection. We don't want them to pass it but the 
collection param parsing code I'm reusing will barf (rightly) if there's no 
name for a collection. We'll fix that up later when we calculate the name for 
the initial collection in the Cmd class.


> API to create a Time Routed Alias and first collection
> --
>
> Key: SOLR-11722
> URL: https://issues.apache.org/jira/browse/SOLR-11722
> Project: Solr
>  Issue Type: Sub-task
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: SolrCloud
>Reporter: David Smiley
> Attachments: SOLR-11722.patch, SOLR-11722.patch
>
>
> This issue is about creating a single API command to create a "Time Routed 
> Alias" along with its first collection.  Need to decide what endpoint URL it 
> is and parameters.
> Perhaps in v2 it'd be {{/api/collections?command=create-routed-alias}} or 
> alternatively piggy-back off of command=create-alias but we add more options, 
> perhaps with a prefix like "router"?
> Inputs:
> * alias name
> * misc collection creation metadata (e.g. config, numShards, ...) perhaps in 
> this context with a prefix like "collection."
> * metadata for TimeRoutedAliasUpdateProcessor, currently: router.field
> * date specifier for first collection; can include "date math".
> We'll certainly add more options as future features unfold.
> I believe the collection needs to be created first (referring to the alias 
> name via a core property), and then the alias pointing to it which demands 
> collections exist first.  When figuring the collection name, you'll need to 
> reference the format in TimeRoutedAliasUpdateProcessor.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-11722) API to create a Time Routed Alias and first collection

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-11722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16323654#comment-16323654
 ] 

ASF GitHub Bot commented on SOLR-11722:
---

Github user fsparv commented on a diff in the pull request:

https://github.com/apache/lucene-solr/pull/304#discussion_r161155122
  
--- Diff: solr/core/src/java/org/apache/solr/cloud/CreateAliasCmd.java ---
@@ -68,34 +249,100 @@ public void call(ClusterState state, ZkNodeProps 
message, NamedList results)
 Thread.sleep(100);
   }
 
+  private Map buildAliasMap(String routedField, String 
routingType, String tz, String increment, String maxFutureMs, ZkNodeProps 
collectionProps) {
+Map properties = collectionProps.getProperties();
+Map cleanMap = properties.entrySet().stream()
+.filter(stringObjectEntry ->
+!"fromApi".equals(stringObjectEntry.getKey())
--- End diff --

Made my own constants & made set existing constants were too 
scattered/disorganized/nonexistant ... fixed in update


> API to create a Time Routed Alias and first collection
> --
>
> Key: SOLR-11722
> URL: https://issues.apache.org/jira/browse/SOLR-11722
> Project: Solr
>  Issue Type: Sub-task
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: SolrCloud
>Reporter: David Smiley
> Attachments: SOLR-11722.patch, SOLR-11722.patch
>
>
> This issue is about creating a single API command to create a "Time Routed 
> Alias" along with its first collection.  Need to decide what endpoint URL it 
> is and parameters.
> Perhaps in v2 it'd be {{/api/collections?command=create-routed-alias}} or 
> alternatively piggy-back off of command=create-alias but we add more options, 
> perhaps with a prefix like "router"?
> Inputs:
> * alias name
> * misc collection creation metadata (e.g. config, numShards, ...) perhaps in 
> this context with a prefix like "collection."
> * metadata for TimeRoutedAliasUpdateProcessor, currently: router.field
> * date specifier for first collection; can include "date math".
> We'll certainly add more options as future features unfold.
> I believe the collection needs to be created first (referring to the alias 
> name via a core property), and then the alias pointing to it which demands 
> collections exist first.  When figuring the collection name, you'll need to 
> reference the format in TimeRoutedAliasUpdateProcessor.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-11722) API to create a Time Routed Alias and first collection

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-11722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16323531#comment-16323531
 ] 

ASF GitHub Bot commented on SOLR-11722:
---

Github user fsparv commented on a diff in the pull request:

https://github.com/apache/lucene-solr/pull/304#discussion_r161138197
  
--- Diff: solr/core/src/java/org/apache/solr/cloud/CreateAliasCmd.java ---
@@ -45,13 +147,92 @@ public CreateAliasCmd(OverseerCollectionMessageHandler 
ocmh) {
   public void call(ClusterState state, ZkNodeProps message, NamedList 
results)
   throws Exception {
 final String aliasName = message.getStr(NAME);
-final List canonicalCollectionList = 
parseCollectionsParameter(message.get("collections"));
-final String canonicalCollectionsString = 
StrUtils.join(canonicalCollectionList, ',');
-
 ZkStateReader zkStateReader = ocmh.zkStateReader;
-validateAllCollectionsExistAndNoDups(canonicalCollectionList, 
zkStateReader);
+ZkStateReader.AliasesManager holder = zkStateReader.aliasesHolder;
+if (!anyRoutingParams(message)) {
+  final List canonicalCollectionList = 
parseCollectionsParameter(message.get("collections"));
+  final String canonicalCollectionsString = 
StrUtils.join(canonicalCollectionList, ',');
+  validateAllCollectionsExistAndNoDups(canonicalCollectionList, 
zkStateReader);
+  holder.applyModificationAndExportToZk(aliases -> 
aliases.cloneWithCollectionAlias(aliasName, canonicalCollectionsString));
+} else {
+  final String routedField = message.getStr(ROUTING_FIELD);
+  final String routingType = message.getStr(ROUTING_TYPE);
+  final String tz = message.getStr(TZ);
+  final String start = message.getStr(START);
+  final String increment = message.getStr(ROUTING_INCREMENT);
+  final String maxFutureMs = message.getStr(ROUTING_MAX_FUTURE);
+
+  try {
+if (0 > Long.valueOf(maxFutureMs)) {
+  throw new NumberFormatException("Negative value not allowed 
here");
+}
+  } catch (NumberFormatException e) {
+throw new SolrException(BAD_REQUEST, ROUTING_MAX_FUTURE + " must 
be a valid long integer representing a number " +
+"of milliseconds greater than or equal to zero");
+  }
 
-zkStateReader.aliasesHolder.applyModificationAndExportToZk(aliases -> 
aliases.cloneWithCollectionAlias(aliasName, canonicalCollectionsString));
+  // Validate we got everything we need
+  if (routedField == null || routingType == null || start == null || 
increment == null) {
+SolrException solrException = new SolrException(BAD_REQUEST, "If 
any of " + CREATE_ROUTED_ALIAS_PARAMS +
+" are supplied, then all of " + REQUIRED_ROUTING_PARAMS + " 
must be present.");
+log.error("Could not create routed alias",solrException);
+throw solrException;
+  }
+
+  if (!"time".equals(routingType)) {
+SolrException solrException = new SolrException(BAD_REQUEST, "Only 
time based routing is supported at this time");
+log.error("Could not create routed alias",solrException);
+throw solrException;
+  }
+  // Check for invalid timezone
+  if(tz != null && !TimeZoneUtils.KNOWN_TIMEZONE_IDS.contains(tz)) {
+SolrException solrException = new SolrException(BAD_REQUEST, 
"Invalid timezone:" + tz);
+log.error("Could not create routed alias",solrException);
+throw solrException;
+
+  }
+  TimeZone zone;
+  if (tz != null) {
+zone = TimeZoneUtils.getTimeZone(tz);
+  } else {
+zone = TimeZoneUtils.getTimeZone("UTC");
+  }
+  DateTimeFormatter fmt = 
DATE_TIME_FORMATTER.withZone(zone.toZoneId());
--- End diff --

An offline chat reveals that I misunderstood the intent some time back. 
Names for collections ARE meant to be in UTC, that will make things simpler I 
think.


> API to create a Time Routed Alias and first collection
> --
>
> Key: SOLR-11722
> URL: https://issues.apache.org/jira/browse/SOLR-11722
> Project: Solr
>  Issue Type: Sub-task
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: SolrCloud
>Reporter: David Smiley
> Attachments: SOLR-11722.patch, SOLR-11722.patch
>
>
> This issue is about creating a single API command to create a "Time Routed 
> Alias" along with its first collection.  Need to decide what endpoint URL it 
> is and parameters.
> Perhaps in v2 it'd be {{/api/collections?command=create-routed-alias}} or 
> alternatively piggy-back off of command=create-alias 

[jira] [Commented] (SOLR-11722) API to create a Time Routed Alias and first collection

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-11722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16323502#comment-16323502
 ] 

ASF GitHub Bot commented on SOLR-11722:
---

Github user dsmiley commented on a diff in the pull request:

https://github.com/apache/lucene-solr/pull/304#discussion_r161134845
  
--- Diff: solr/core/src/java/org/apache/solr/cloud/CreateAliasCmd.java ---
@@ -45,13 +147,92 @@ public CreateAliasCmd(OverseerCollectionMessageHandler 
ocmh) {
   public void call(ClusterState state, ZkNodeProps message, NamedList 
results)
   throws Exception {
 final String aliasName = message.getStr(NAME);
-final List canonicalCollectionList = 
parseCollectionsParameter(message.get("collections"));
-final String canonicalCollectionsString = 
StrUtils.join(canonicalCollectionList, ',');
-
 ZkStateReader zkStateReader = ocmh.zkStateReader;
-validateAllCollectionsExistAndNoDups(canonicalCollectionList, 
zkStateReader);
+ZkStateReader.AliasesManager holder = zkStateReader.aliasesHolder;
+if (!anyRoutingParams(message)) {
+  final List canonicalCollectionList = 
parseCollectionsParameter(message.get("collections"));
+  final String canonicalCollectionsString = 
StrUtils.join(canonicalCollectionList, ',');
+  validateAllCollectionsExistAndNoDups(canonicalCollectionList, 
zkStateReader);
+  holder.applyModificationAndExportToZk(aliases -> 
aliases.cloneWithCollectionAlias(aliasName, canonicalCollectionsString));
+} else {
+  final String routedField = message.getStr(ROUTING_FIELD);
+  final String routingType = message.getStr(ROUTING_TYPE);
+  final String tz = message.getStr(TZ);
+  final String start = message.getStr(START);
+  final String increment = message.getStr(ROUTING_INCREMENT);
+  final String maxFutureMs = message.getStr(ROUTING_MAX_FUTURE);
+
+  try {
+if (0 > Long.valueOf(maxFutureMs)) {
+  throw new NumberFormatException("Negative value not allowed 
here");
+}
+  } catch (NumberFormatException e) {
+throw new SolrException(BAD_REQUEST, ROUTING_MAX_FUTURE + " must 
be a valid long integer representing a number " +
+"of milliseconds greater than or equal to zero");
+  }
 
-zkStateReader.aliasesHolder.applyModificationAndExportToZk(aliases -> 
aliases.cloneWithCollectionAlias(aliasName, canonicalCollectionsString));
+  // Validate we got everything we need
+  if (routedField == null || routingType == null || start == null || 
increment == null) {
+SolrException solrException = new SolrException(BAD_REQUEST, "If 
any of " + CREATE_ROUTED_ALIAS_PARAMS +
+" are supplied, then all of " + REQUIRED_ROUTING_PARAMS + " 
must be present.");
+log.error("Could not create routed alias",solrException);
+throw solrException;
+  }
+
+  if (!"time".equals(routingType)) {
+SolrException solrException = new SolrException(BAD_REQUEST, "Only 
time based routing is supported at this time");
+log.error("Could not create routed alias",solrException);
+throw solrException;
+  }
+  // Check for invalid timezone
+  if(tz != null && !TimeZoneUtils.KNOWN_TIMEZONE_IDS.contains(tz)) {
+SolrException solrException = new SolrException(BAD_REQUEST, 
"Invalid timezone:" + tz);
+log.error("Could not create routed alias",solrException);
+throw solrException;
+
+  }
+  TimeZone zone;
+  if (tz != null) {
+zone = TimeZoneUtils.getTimeZone(tz);
+  } else {
+zone = TimeZoneUtils.getTimeZone("UTC");
+  }
+  DateTimeFormatter fmt = 
DATE_TIME_FORMATTER.withZone(zone.toZoneId());
--- End diff --

> you want TZ to adjust any dates that don't include a timezone, and not 
adjust any that do (where the only acceptable date included TZ is utc via the Z 
suffix)?

Not really because Solr doesn't accept ambiguous dates (dates without a 
'Z', == UTC) so that's kind of a non-issue.  _The TimeZone is needed to 
correctly interpret date math (if present)_.  Oh yeah, DateRangeField can 
accept very dates without 'Z' but lets ignore that :-)  Nonetheless DRF is 
strictly UTC.

The format of the collection name is an internal detail so it almost 
doesn't matter.  I'd rather it be permanently UTC interpreted to avoid problems 
in changing TZ somehow.  But I can sympathize that there's an argument to be 
made it would look nicer if it honored the TZ.  But again it's internal.


> API to create a Time Routed Alias and first collection
> --
>
> Key: SOLR-11722
>   

[jira] [Commented] (SOLR-11722) API to create a Time Routed Alias and first collection

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-11722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16323386#comment-16323386
 ] 

ASF GitHub Bot commented on SOLR-11722:
---

Github user fsparv commented on a diff in the pull request:

https://github.com/apache/lucene-solr/pull/304#discussion_r161122011
  
--- Diff: solr/core/src/java/org/apache/solr/cloud/CreateAliasCmd.java ---
@@ -45,13 +147,92 @@ public CreateAliasCmd(OverseerCollectionMessageHandler 
ocmh) {
   public void call(ClusterState state, ZkNodeProps message, NamedList 
results)
   throws Exception {
 final String aliasName = message.getStr(NAME);
-final List canonicalCollectionList = 
parseCollectionsParameter(message.get("collections"));
-final String canonicalCollectionsString = 
StrUtils.join(canonicalCollectionList, ',');
-
 ZkStateReader zkStateReader = ocmh.zkStateReader;
-validateAllCollectionsExistAndNoDups(canonicalCollectionList, 
zkStateReader);
+ZkStateReader.AliasesManager holder = zkStateReader.aliasesHolder;
+if (!anyRoutingParams(message)) {
+  final List canonicalCollectionList = 
parseCollectionsParameter(message.get("collections"));
+  final String canonicalCollectionsString = 
StrUtils.join(canonicalCollectionList, ',');
+  validateAllCollectionsExistAndNoDups(canonicalCollectionList, 
zkStateReader);
+  holder.applyModificationAndExportToZk(aliases -> 
aliases.cloneWithCollectionAlias(aliasName, canonicalCollectionsString));
+} else {
+  final String routedField = message.getStr(ROUTING_FIELD);
+  final String routingType = message.getStr(ROUTING_TYPE);
+  final String tz = message.getStr(TZ);
+  final String start = message.getStr(START);
+  final String increment = message.getStr(ROUTING_INCREMENT);
+  final String maxFutureMs = message.getStr(ROUTING_MAX_FUTURE);
+
+  try {
+if (0 > Long.valueOf(maxFutureMs)) {
+  throw new NumberFormatException("Negative value not allowed 
here");
+}
+  } catch (NumberFormatException e) {
+throw new SolrException(BAD_REQUEST, ROUTING_MAX_FUTURE + " must 
be a valid long integer representing a number " +
+"of milliseconds greater than or equal to zero");
+  }
 
-zkStateReader.aliasesHolder.applyModificationAndExportToZk(aliases -> 
aliases.cloneWithCollectionAlias(aliasName, canonicalCollectionsString));
+  // Validate we got everything we need
+  if (routedField == null || routingType == null || start == null || 
increment == null) {
+SolrException solrException = new SolrException(BAD_REQUEST, "If 
any of " + CREATE_ROUTED_ALIAS_PARAMS +
+" are supplied, then all of " + REQUIRED_ROUTING_PARAMS + " 
must be present.");
+log.error("Could not create routed alias",solrException);
+throw solrException;
+  }
+
+  if (!"time".equals(routingType)) {
+SolrException solrException = new SolrException(BAD_REQUEST, "Only 
time based routing is supported at this time");
+log.error("Could not create routed alias",solrException);
+throw solrException;
+  }
+  // Check for invalid timezone
+  if(tz != null && !TimeZoneUtils.KNOWN_TIMEZONE_IDS.contains(tz)) {
+SolrException solrException = new SolrException(BAD_REQUEST, 
"Invalid timezone:" + tz);
+log.error("Could not create routed alias",solrException);
+throw solrException;
+
+  }
+  TimeZone zone;
+  if (tz != null) {
+zone = TimeZoneUtils.getTimeZone(tz);
+  } else {
+zone = TimeZoneUtils.getTimeZone("UTC");
+  }
+  DateTimeFormatter fmt = 
DATE_TIME_FORMATTER.withZone(zone.toZoneId());
--- End diff --

In reading the issue referenced in the above referenced comment again, it 
occurs to me that what I hadn't thought about is that by having TZ parameter at 
the top level it does sort of imply that it applies to the whole request, and 
we need to handle all dates supplied either as the specified UTC or if the Z is 
omitted handle them as the timezone specified. I had mostly been thinking in 
terms of the effect of TZ on updates and users trying to optimize by querying 
directly to the correct collection and the calculation of "next" collections. 
In theory, there could be 2 TZ's one for the routing configuration router.tz 
and one that applies to the dates in this request (the top level TZ)... but 
that is probably be overkill. So now I think I understand what you've been 
trying to tell me... you want TZ to adjust any dates that don't include a 
timezone, and not adjust any that do (where the only acceptable date included 
TZ is utc via the Z 

[jira] [Commented] (SOLR-11722) API to create a Time Routed Alias and first collection

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-11722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16323256#comment-16323256
 ] 

ASF GitHub Bot commented on SOLR-11722:
---

Github user fsparv commented on a diff in the pull request:

https://github.com/apache/lucene-solr/pull/304#discussion_r161110384
  
--- Diff: solr/core/src/java/org/apache/solr/cloud/CreateAliasCmd.java ---
@@ -45,13 +147,92 @@ public CreateAliasCmd(OverseerCollectionMessageHandler 
ocmh) {
   public void call(ClusterState state, ZkNodeProps message, NamedList 
results)
   throws Exception {
 final String aliasName = message.getStr(NAME);
-final List canonicalCollectionList = 
parseCollectionsParameter(message.get("collections"));
-final String canonicalCollectionsString = 
StrUtils.join(canonicalCollectionList, ',');
-
 ZkStateReader zkStateReader = ocmh.zkStateReader;
-validateAllCollectionsExistAndNoDups(canonicalCollectionList, 
zkStateReader);
+ZkStateReader.AliasesManager holder = zkStateReader.aliasesHolder;
+if (!anyRoutingParams(message)) {
+  final List canonicalCollectionList = 
parseCollectionsParameter(message.get("collections"));
+  final String canonicalCollectionsString = 
StrUtils.join(canonicalCollectionList, ',');
+  validateAllCollectionsExistAndNoDups(canonicalCollectionList, 
zkStateReader);
+  holder.applyModificationAndExportToZk(aliases -> 
aliases.cloneWithCollectionAlias(aliasName, canonicalCollectionsString));
+} else {
+  final String routedField = message.getStr(ROUTING_FIELD);
+  final String routingType = message.getStr(ROUTING_TYPE);
+  final String tz = message.getStr(TZ);
+  final String start = message.getStr(START);
+  final String increment = message.getStr(ROUTING_INCREMENT);
+  final String maxFutureMs = message.getStr(ROUTING_MAX_FUTURE);
+
+  try {
+if (0 > Long.valueOf(maxFutureMs)) {
+  throw new NumberFormatException("Negative value not allowed 
here");
+}
+  } catch (NumberFormatException e) {
+throw new SolrException(BAD_REQUEST, ROUTING_MAX_FUTURE + " must 
be a valid long integer representing a number " +
+"of milliseconds greater than or equal to zero");
+  }
 
-zkStateReader.aliasesHolder.applyModificationAndExportToZk(aliases -> 
aliases.cloneWithCollectionAlias(aliasName, canonicalCollectionsString));
+  // Validate we got everything we need
+  if (routedField == null || routingType == null || start == null || 
increment == null) {
+SolrException solrException = new SolrException(BAD_REQUEST, "If 
any of " + CREATE_ROUTED_ALIAS_PARAMS +
+" are supplied, then all of " + REQUIRED_ROUTING_PARAMS + " 
must be present.");
+log.error("Could not create routed alias",solrException);
+throw solrException;
+  }
+
+  if (!"time".equals(routingType)) {
+SolrException solrException = new SolrException(BAD_REQUEST, "Only 
time based routing is supported at this time");
+log.error("Could not create routed alias",solrException);
+throw solrException;
+  }
+  // Check for invalid timezone
+  if(tz != null && !TimeZoneUtils.KNOWN_TIMEZONE_IDS.contains(tz)) {
+SolrException solrException = new SolrException(BAD_REQUEST, 
"Invalid timezone:" + tz);
+log.error("Could not create routed alias",solrException);
+throw solrException;
+
+  }
+  TimeZone zone;
+  if (tz != null) {
+zone = TimeZoneUtils.getTimeZone(tz);
+  } else {
+zone = TimeZoneUtils.getTimeZone("UTC");
+  }
+  DateTimeFormatter fmt = 
DATE_TIME_FORMATTER.withZone(zone.toZoneId());
--- End diff --

(and that format has no timzeone in it so in theory we could treat it as 
local time based on TZ but at the moment I treat it as utc).


> API to create a Time Routed Alias and first collection
> --
>
> Key: SOLR-11722
> URL: https://issues.apache.org/jira/browse/SOLR-11722
> Project: Solr
>  Issue Type: Sub-task
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: SolrCloud
>Reporter: David Smiley
> Attachments: SOLR-11722.patch, SOLR-11722.patch
>
>
> This issue is about creating a single API command to create a "Time Routed 
> Alias" along with its first collection.  Need to decide what endpoint URL it 
> is and parameters.
> Perhaps in v2 it'd be {{/api/collections?command=create-routed-alias}} or 
> alternatively piggy-back off of command=create-alias but we add more options, 

[jira] [Commented] (SOLR-11722) API to create a Time Routed Alias and first collection

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-11722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16323235#comment-16323235
 ] 

ASF GitHub Bot commented on SOLR-11722:
---

Github user fsparv commented on a diff in the pull request:

https://github.com/apache/lucene-solr/pull/304#discussion_r161107544
  
--- Diff: solr/core/src/java/org/apache/solr/cloud/CreateAliasCmd.java ---
@@ -68,34 +249,100 @@ public void call(ClusterState state, ZkNodeProps 
message, NamedList results)
 Thread.sleep(100);
   }
 
+  private Map buildAliasMap(String routedField, String 
routingType, String tz, String increment, String maxFutureMs, ZkNodeProps 
collectionProps) {
+Map properties = collectionProps.getProperties();
+Map cleanMap = properties.entrySet().stream()
+.filter(stringObjectEntry ->
+!"fromApi".equals(stringObjectEntry.getKey())
--- End diff --

yeah, this is stuff that the collection create code adds. I'll look if I 
can use constants from there and make a set out of them.


> API to create a Time Routed Alias and first collection
> --
>
> Key: SOLR-11722
> URL: https://issues.apache.org/jira/browse/SOLR-11722
> Project: Solr
>  Issue Type: Sub-task
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: SolrCloud
>Reporter: David Smiley
> Attachments: SOLR-11722.patch, SOLR-11722.patch
>
>
> This issue is about creating a single API command to create a "Time Routed 
> Alias" along with its first collection.  Need to decide what endpoint URL it 
> is and parameters.
> Perhaps in v2 it'd be {{/api/collections?command=create-routed-alias}} or 
> alternatively piggy-back off of command=create-alias but we add more options, 
> perhaps with a prefix like "router"?
> Inputs:
> * alias name
> * misc collection creation metadata (e.g. config, numShards, ...) perhaps in 
> this context with a prefix like "collection."
> * metadata for TimeRoutedAliasUpdateProcessor, currently: router.field
> * date specifier for first collection; can include "date math".
> We'll certainly add more options as future features unfold.
> I believe the collection needs to be created first (referring to the alias 
> name via a core property), and then the alias pointing to it which demands 
> collections exist first.  When figuring the collection name, you'll need to 
> reference the format in TimeRoutedAliasUpdateProcessor.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-11722) API to create a Time Routed Alias and first collection

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-11722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16323186#comment-16323186
 ] 

ASF GitHub Bot commented on SOLR-11722:
---

Github user fsparv commented on a diff in the pull request:

https://github.com/apache/lucene-solr/pull/304#discussion_r161103620
  
--- Diff: solr/core/src/java/org/apache/solr/cloud/CreateAliasCmd.java ---
@@ -45,13 +147,92 @@ public CreateAliasCmd(OverseerCollectionMessageHandler 
ocmh) {
   public void call(ClusterState state, ZkNodeProps message, NamedList 
results)
   throws Exception {
 final String aliasName = message.getStr(NAME);
-final List canonicalCollectionList = 
parseCollectionsParameter(message.get("collections"));
-final String canonicalCollectionsString = 
StrUtils.join(canonicalCollectionList, ',');
-
 ZkStateReader zkStateReader = ocmh.zkStateReader;
-validateAllCollectionsExistAndNoDups(canonicalCollectionList, 
zkStateReader);
+ZkStateReader.AliasesManager holder = zkStateReader.aliasesHolder;
+if (!anyRoutingParams(message)) {
+  final List canonicalCollectionList = 
parseCollectionsParameter(message.get("collections"));
+  final String canonicalCollectionsString = 
StrUtils.join(canonicalCollectionList, ',');
+  validateAllCollectionsExistAndNoDups(canonicalCollectionList, 
zkStateReader);
+  holder.applyModificationAndExportToZk(aliases -> 
aliases.cloneWithCollectionAlias(aliasName, canonicalCollectionsString));
+} else {
+  final String routedField = message.getStr(ROUTING_FIELD);
+  final String routingType = message.getStr(ROUTING_TYPE);
+  final String tz = message.getStr(TZ);
+  final String start = message.getStr(START);
+  final String increment = message.getStr(ROUTING_INCREMENT);
+  final String maxFutureMs = message.getStr(ROUTING_MAX_FUTURE);
+
+  try {
+if (0 > Long.valueOf(maxFutureMs)) {
+  throw new NumberFormatException("Negative value not allowed 
here");
+}
+  } catch (NumberFormatException e) {
+throw new SolrException(BAD_REQUEST, ROUTING_MAX_FUTURE + " must 
be a valid long integer representing a number " +
+"of milliseconds greater than or equal to zero");
+  }
 
-zkStateReader.aliasesHolder.applyModificationAndExportToZk(aliases -> 
aliases.cloneWithCollectionAlias(aliasName, canonicalCollectionsString));
+  // Validate we got everything we need
+  if (routedField == null || routingType == null || start == null || 
increment == null) {
+SolrException solrException = new SolrException(BAD_REQUEST, "If 
any of " + CREATE_ROUTED_ALIAS_PARAMS +
+" are supplied, then all of " + REQUIRED_ROUTING_PARAMS + " 
must be present.");
+log.error("Could not create routed alias",solrException);
+throw solrException;
+  }
+
+  if (!"time".equals(routingType)) {
+SolrException solrException = new SolrException(BAD_REQUEST, "Only 
time based routing is supported at this time");
+log.error("Could not create routed alias",solrException);
+throw solrException;
+  }
+  // Check for invalid timezone
+  if(tz != null && !TimeZoneUtils.KNOWN_TIMEZONE_IDS.contains(tz)) {
+SolrException solrException = new SolrException(BAD_REQUEST, 
"Invalid timezone:" + tz);
+log.error("Could not create routed alias",solrException);
+throw solrException;
+
+  }
+  TimeZone zone;
+  if (tz != null) {
+zone = TimeZoneUtils.getTimeZone(tz);
+  } else {
+zone = TimeZoneUtils.getTimeZone("UTC");
+  }
+  DateTimeFormatter fmt = 
DATE_TIME_FORMATTER.withZone(zone.toZoneId());
--- End diff --

actually amending that we do accept the format used for the collection 
names, but since a time stamp is valid date math we need to handle this too.


> API to create a Time Routed Alias and first collection
> --
>
> Key: SOLR-11722
> URL: https://issues.apache.org/jira/browse/SOLR-11722
> Project: Solr
>  Issue Type: Sub-task
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: SolrCloud
>Reporter: David Smiley
> Attachments: SOLR-11722.patch, SOLR-11722.patch
>
>
> This issue is about creating a single API command to create a "Time Routed 
> Alias" along with its first collection.  Need to decide what endpoint URL it 
> is and parameters.
> Perhaps in v2 it'd be {{/api/collections?command=create-routed-alias}} or 
> alternatively piggy-back off of command=create-alias but we add 

[jira] [Commented] (SOLR-11722) API to create a Time Routed Alias and first collection

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-11722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16323171#comment-16323171
 ] 

ASF GitHub Bot commented on SOLR-11722:
---

Github user fsparv commented on a diff in the pull request:

https://github.com/apache/lucene-solr/pull/304#discussion_r161101824
  
--- Diff: solr/core/src/java/org/apache/solr/cloud/CreateAliasCmd.java ---
@@ -45,13 +147,92 @@ public CreateAliasCmd(OverseerCollectionMessageHandler 
ocmh) {
   public void call(ClusterState state, ZkNodeProps message, NamedList 
results)
   throws Exception {
 final String aliasName = message.getStr(NAME);
-final List canonicalCollectionList = 
parseCollectionsParameter(message.get("collections"));
-final String canonicalCollectionsString = 
StrUtils.join(canonicalCollectionList, ',');
-
 ZkStateReader zkStateReader = ocmh.zkStateReader;
-validateAllCollectionsExistAndNoDups(canonicalCollectionList, 
zkStateReader);
+ZkStateReader.AliasesManager holder = zkStateReader.aliasesHolder;
+if (!anyRoutingParams(message)) {
+  final List canonicalCollectionList = 
parseCollectionsParameter(message.get("collections"));
+  final String canonicalCollectionsString = 
StrUtils.join(canonicalCollectionList, ',');
+  validateAllCollectionsExistAndNoDups(canonicalCollectionList, 
zkStateReader);
+  holder.applyModificationAndExportToZk(aliases -> 
aliases.cloneWithCollectionAlias(aliasName, canonicalCollectionsString));
+} else {
+  final String routedField = message.getStr(ROUTING_FIELD);
+  final String routingType = message.getStr(ROUTING_TYPE);
+  final String tz = message.getStr(TZ);
+  final String start = message.getStr(START);
+  final String increment = message.getStr(ROUTING_INCREMENT);
+  final String maxFutureMs = message.getStr(ROUTING_MAX_FUTURE);
+
+  try {
+if (0 > Long.valueOf(maxFutureMs)) {
+  throw new NumberFormatException("Negative value not allowed 
here");
+}
+  } catch (NumberFormatException e) {
+throw new SolrException(BAD_REQUEST, ROUTING_MAX_FUTURE + " must 
be a valid long integer representing a number " +
+"of milliseconds greater than or equal to zero");
+  }
 
-zkStateReader.aliasesHolder.applyModificationAndExportToZk(aliases -> 
aliases.cloneWithCollectionAlias(aliasName, canonicalCollectionsString));
+  // Validate we got everything we need
+  if (routedField == null || routingType == null || start == null || 
increment == null) {
+SolrException solrException = new SolrException(BAD_REQUEST, "If 
any of " + CREATE_ROUTED_ALIAS_PARAMS +
+" are supplied, then all of " + REQUIRED_ROUTING_PARAMS + " 
must be present.");
+log.error("Could not create routed alias",solrException);
+throw solrException;
+  }
+
+  if (!"time".equals(routingType)) {
+SolrException solrException = new SolrException(BAD_REQUEST, "Only 
time based routing is supported at this time");
+log.error("Could not create routed alias",solrException);
+throw solrException;
+  }
+  // Check for invalid timezone
+  if(tz != null && !TimeZoneUtils.KNOWN_TIMEZONE_IDS.contains(tz)) {
+SolrException solrException = new SolrException(BAD_REQUEST, 
"Invalid timezone:" + tz);
+log.error("Could not create routed alias",solrException);
+throw solrException;
+
+  }
+  TimeZone zone;
+  if (tz != null) {
+zone = TimeZoneUtils.getTimeZone(tz);
+  } else {
+zone = TimeZoneUtils.getTimeZone("UTC");
+  }
+  DateTimeFormatter fmt = 
DATE_TIME_FORMATTER.withZone(zone.toZoneId());
--- End diff --

So if someone specifies start as an iso instant, we don't want to adjust 
that to match the specified zone? This would lead to 
start=2018-01-14T20:00:00:00.Z=EST=+1DAY it would seem 
that the first bucket should be 2018-01-15 but this way we would be creating a 
first bucket for 2018-01-14 instead. That seems confusing. Solr fairly 
consistently takes the position that times you speak to it must be in UTC, 
changing that here seems awkward. We could allow more general time format and 
take the timezone from the entered time, so that they can enter 
2018-01-15T00:00:00:00.EST which is unlike the rest of how solr handles 
time AFAIK but would make the correspondence between the collection name and 
the time value entered intuitive (and maybe complicates combining this with the 
datemath) 

I had originally proposed that collections names for this be UTC and making 
them full time stamps to avoid all this TZ stuff, but the response 

[jira] [Commented] (SOLR-11722) API to create a Time Routed Alias and first collection

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-11722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16323124#comment-16323124
 ] 

ASF GitHub Bot commented on SOLR-11722:
---

Github user fsparv commented on a diff in the pull request:

https://github.com/apache/lucene-solr/pull/304#discussion_r161096191
  
--- Diff: solr/core/src/java/org/apache/solr/cloud/CreateAliasCmd.java ---
@@ -45,13 +147,92 @@ public CreateAliasCmd(OverseerCollectionMessageHandler 
ocmh) {
   public void call(ClusterState state, ZkNodeProps message, NamedList 
results)
   throws Exception {
 final String aliasName = message.getStr(NAME);
-final List canonicalCollectionList = 
parseCollectionsParameter(message.get("collections"));
-final String canonicalCollectionsString = 
StrUtils.join(canonicalCollectionList, ',');
-
 ZkStateReader zkStateReader = ocmh.zkStateReader;
-validateAllCollectionsExistAndNoDups(canonicalCollectionList, 
zkStateReader);
+ZkStateReader.AliasesManager holder = zkStateReader.aliasesHolder;
+if (!anyRoutingParams(message)) {
+  final List canonicalCollectionList = 
parseCollectionsParameter(message.get("collections"));
+  final String canonicalCollectionsString = 
StrUtils.join(canonicalCollectionList, ',');
+  validateAllCollectionsExistAndNoDups(canonicalCollectionList, 
zkStateReader);
+  holder.applyModificationAndExportToZk(aliases -> 
aliases.cloneWithCollectionAlias(aliasName, canonicalCollectionsString));
+} else {
+  final String routedField = message.getStr(ROUTING_FIELD);
+  final String routingType = message.getStr(ROUTING_TYPE);
+  final String tz = message.getStr(TZ);
+  final String start = message.getStr(START);
+  final String increment = message.getStr(ROUTING_INCREMENT);
+  final String maxFutureMs = message.getStr(ROUTING_MAX_FUTURE);
+
+  try {
+if (0 > Long.valueOf(maxFutureMs)) {
+  throw new NumberFormatException("Negative value not allowed 
here");
+}
+  } catch (NumberFormatException e) {
+throw new SolrException(BAD_REQUEST, ROUTING_MAX_FUTURE + " must 
be a valid long integer representing a number " +
+"of milliseconds greater than or equal to zero");
+  }
 
-zkStateReader.aliasesHolder.applyModificationAndExportToZk(aliases -> 
aliases.cloneWithCollectionAlias(aliasName, canonicalCollectionsString));
+  // Validate we got everything we need
+  if (routedField == null || routingType == null || start == null || 
increment == null) {
--- End diff --

in this area of the code I implement checks for the following:
name
start
router.field
router.name
router.interval
router.max-future-ms
TZ
collection-create.config

Only that last one could possibly be left to the collection-create 
infrastructure. The rest are ours (note that the router ones are not the 
collection level routing but the alias level routing).


> API to create a Time Routed Alias and first collection
> --
>
> Key: SOLR-11722
> URL: https://issues.apache.org/jira/browse/SOLR-11722
> Project: Solr
>  Issue Type: Sub-task
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: SolrCloud
>Reporter: David Smiley
> Attachments: SOLR-11722.patch, SOLR-11722.patch
>
>
> This issue is about creating a single API command to create a "Time Routed 
> Alias" along with its first collection.  Need to decide what endpoint URL it 
> is and parameters.
> Perhaps in v2 it'd be {{/api/collections?command=create-routed-alias}} or 
> alternatively piggy-back off of command=create-alias but we add more options, 
> perhaps with a prefix like "router"?
> Inputs:
> * alias name
> * misc collection creation metadata (e.g. config, numShards, ...) perhaps in 
> this context with a prefix like "collection."
> * metadata for TimeRoutedAliasUpdateProcessor, currently: router.field
> * date specifier for first collection; can include "date math".
> We'll certainly add more options as future features unfold.
> I believe the collection needs to be created first (referring to the alias 
> name via a core property), and then the alias pointing to it which demands 
> collections exist first.  When figuring the collection name, you'll need to 
> reference the format in TimeRoutedAliasUpdateProcessor.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-11722) API to create a Time Routed Alias and first collection

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-11722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16323095#comment-16323095
 ] 

ASF GitHub Bot commented on SOLR-11722:
---

Github user fsparv commented on a diff in the pull request:

https://github.com/apache/lucene-solr/pull/304#discussion_r161093191
  
--- Diff: solr/core/src/java/org/apache/solr/cloud/CreateAliasCmd.java ---
@@ -30,13 +44,101 @@
 import org.apache.solr.common.cloud.ZkStateReader;
 import org.apache.solr.common.util.NamedList;
 import org.apache.solr.common.util.StrUtils;
+import org.apache.solr.update.processor.TimeRoutedAliasUpdateProcessor;
+import org.apache.solr.util.DateMathParser;
+import org.apache.solr.util.TimeZoneUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
 
+import static java.time.format.DateTimeFormatter.ISO_INSTANT;
+import static 
org.apache.solr.cloud.OverseerCollectionMessageHandler.COLL_CONF;
+import static org.apache.solr.common.SolrException.ErrorCode.BAD_REQUEST;
 import static org.apache.solr.common.params.CommonParams.NAME;
+import static org.apache.solr.common.params.CommonParams.TZ;
+import static 
org.apache.solr.handler.admin.CollectionsHandler.ROUTED_ALIAS_COLLECTION_PROP_PFX;
+import static 
org.apache.solr.update.processor.TimeRoutedAliasUpdateProcessor.DATE_TIME_FORMATTER;
 
 
 public class CreateAliasCmd implements Cmd {
+
+  private static final Logger log = 
LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+  public static final String ROUTING_TYPE = "router.name";
+  public static final String ROUTING_FIELD = "router.field";
+  public static final String ROUTING_INCREMENT = "router.interval";
+  public static final String ROUTING_MAX_FUTURE = "router.max-future-ms";
+  public static final String START = "start";
+  // Collection constants should all reflect names in the v2 structured 
input for this command, not v1
+  // names used for CREATE
+  public static final String CREATE_COLLECTION_CONFIG = 
"create-collection.config";
--- End diff --

I agree, I don't like having them here and in the collection admin request  
but solrj can't see core. Is there a constants class I can move them to that 
solrj can see?


> API to create a Time Routed Alias and first collection
> --
>
> Key: SOLR-11722
> URL: https://issues.apache.org/jira/browse/SOLR-11722
> Project: Solr
>  Issue Type: Sub-task
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: SolrCloud
>Reporter: David Smiley
> Attachments: SOLR-11722.patch, SOLR-11722.patch
>
>
> This issue is about creating a single API command to create a "Time Routed 
> Alias" along with its first collection.  Need to decide what endpoint URL it 
> is and parameters.
> Perhaps in v2 it'd be {{/api/collections?command=create-routed-alias}} or 
> alternatively piggy-back off of command=create-alias but we add more options, 
> perhaps with a prefix like "router"?
> Inputs:
> * alias name
> * misc collection creation metadata (e.g. config, numShards, ...) perhaps in 
> this context with a prefix like "collection."
> * metadata for TimeRoutedAliasUpdateProcessor, currently: router.field
> * date specifier for first collection; can include "date math".
> We'll certainly add more options as future features unfold.
> I believe the collection needs to be created first (referring to the alias 
> name via a core property), and then the alias pointing to it which demands 
> collections exist first.  When figuring the collection name, you'll need to 
> reference the format in TimeRoutedAliasUpdateProcessor.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-11722) API to create a Time Routed Alias and first collection

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-11722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16322995#comment-16322995
 ] 

ASF GitHub Bot commented on SOLR-11722:
---

Github user dsmiley commented on a diff in the pull request:

https://github.com/apache/lucene-solr/pull/304#discussion_r161076165
  
--- Diff: solr/core/src/java/org/apache/solr/cloud/CreateAliasCmd.java ---
@@ -45,13 +147,92 @@ public CreateAliasCmd(OverseerCollectionMessageHandler 
ocmh) {
   public void call(ClusterState state, ZkNodeProps message, NamedList 
results)
   throws Exception {
 final String aliasName = message.getStr(NAME);
-final List canonicalCollectionList = 
parseCollectionsParameter(message.get("collections"));
-final String canonicalCollectionsString = 
StrUtils.join(canonicalCollectionList, ',');
-
 ZkStateReader zkStateReader = ocmh.zkStateReader;
-validateAllCollectionsExistAndNoDups(canonicalCollectionList, 
zkStateReader);
+ZkStateReader.AliasesManager holder = zkStateReader.aliasesHolder;
+if (!anyRoutingParams(message)) {
+  final List canonicalCollectionList = 
parseCollectionsParameter(message.get("collections"));
+  final String canonicalCollectionsString = 
StrUtils.join(canonicalCollectionList, ',');
+  validateAllCollectionsExistAndNoDups(canonicalCollectionList, 
zkStateReader);
+  holder.applyModificationAndExportToZk(aliases -> 
aliases.cloneWithCollectionAlias(aliasName, canonicalCollectionsString));
+} else {
+  final String routedField = message.getStr(ROUTING_FIELD);
+  final String routingType = message.getStr(ROUTING_TYPE);
+  final String tz = message.getStr(TZ);
+  final String start = message.getStr(START);
+  final String increment = message.getStr(ROUTING_INCREMENT);
+  final String maxFutureMs = message.getStr(ROUTING_MAX_FUTURE);
+
+  try {
+if (0 > Long.valueOf(maxFutureMs)) {
+  throw new NumberFormatException("Negative value not allowed 
here");
+}
+  } catch (NumberFormatException e) {
+throw new SolrException(BAD_REQUEST, ROUTING_MAX_FUTURE + " must 
be a valid long integer representing a number " +
+"of milliseconds greater than or equal to zero");
+  }
 
-zkStateReader.aliasesHolder.applyModificationAndExportToZk(aliases -> 
aliases.cloneWithCollectionAlias(aliasName, canonicalCollectionsString));
+  // Validate we got everything we need
+  if (routedField == null || routingType == null || start == null || 
increment == null) {
+SolrException solrException = new SolrException(BAD_REQUEST, "If 
any of " + CREATE_ROUTED_ALIAS_PARAMS +
+" are supplied, then all of " + REQUIRED_ROUTING_PARAMS + " 
must be present.");
+log.error("Could not create routed alias",solrException);
+throw solrException;
+  }
+
+  if (!"time".equals(routingType)) {
+SolrException solrException = new SolrException(BAD_REQUEST, "Only 
time based routing is supported at this time");
+log.error("Could not create routed alias",solrException);
+throw solrException;
+  }
+  // Check for invalid timezone
+  if(tz != null && !TimeZoneUtils.KNOWN_TIMEZONE_IDS.contains(tz)) {
+SolrException solrException = new SolrException(BAD_REQUEST, 
"Invalid timezone:" + tz);
+log.error("Could not create routed alias",solrException);
+throw solrException;
+
+  }
+  TimeZone zone;
+  if (tz != null) {
+zone = TimeZoneUtils.getTimeZone(tz);
+  } else {
+zone = TimeZoneUtils.getTimeZone("UTC");
+  }
+  DateTimeFormatter fmt = 
DATE_TIME_FORMATTER.withZone(zone.toZoneId());
+
+  // check that the increment is valid date math
+  String checkIncrement = ISO_INSTANT.format(Instant.now()) + 
increment;
+  DateMathParser.parseMath(new Date(), checkIncrement); // exception 
if invalid increment
--- End diff --

Maybe I wasn't clear before.  I mean create a new DateMathParser using the 
constructor that accept the timezone (and we can supply ours; null is okay).  
Then call the instance method `parseMath(str)` on it (not the static with 
'now').  the arg should be the start metadata.  The "now" will default to the 
current instant in time which is what we want.


> API to create a Time Routed Alias and first collection
> --
>
> Key: SOLR-11722
> URL: https://issues.apache.org/jira/browse/SOLR-11722
> Project: Solr
>  Issue Type: Sub-task
>  Security Level: Public(Default Security Level. Issues are 

[jira] [Commented] (SOLR-11722) API to create a Time Routed Alias and first collection

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-11722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16322993#comment-16322993
 ] 

ASF GitHub Bot commented on SOLR-11722:
---

Github user dsmiley commented on a diff in the pull request:

https://github.com/apache/lucene-solr/pull/304#discussion_r161071581
  
--- Diff: 
solr/core/src/java/org/apache/solr/handler/admin/CollectionsHandler.java ---
@@ -962,6 +982,76 @@ public static CollectionOperation get(CollectionAction 
action) {
 }
   }
 
+  private static SolrParams convertToV1WhenRequired(SolrQueryRequest req, 
ModifiableSolrParams params) {
+SolrParams v1Params = params; // (maybe...)
+
+// in the v2 world we get a data map based on the json request, from 
the CommandOperation associated
+// with the request, so locate that if we can.. if we find it we have 
to translate the v2 request
+// properties to v1 params, otherwise we're already good to go.
+List cmds = req.getCommands(true);
+if (cmds.size() > 1) {
+  // todo: not sure if this is the right thing to do here, but also 
not sure what to do if there is more than one...
+  throw new SolrException(BAD_REQUEST, "Only one command is allowed 
when creating a routed alias");
+}
+CommandOperation c = cmds.size() == 0 ? null : cmds.get(0);
+if (c != null) {  // v2 api, do conversion to v1
+  v1Params = new 
BaseHandlerApiSupport.V2ToV1SolrParams(CollectionApiMapping.Meta.CREATE_COLLECTION,
+  req.getPathTemplateValues(), true, params,
+  new CommandOperation("create", 
c.getDataMap().get("create-collection")));
+}
+return v1Params;
+  }
+
+  private static Map 
parseColletionCreationProps(CollectionsHandler h, SolrParams params, String 
prefix)
--- End diff --

typo


> API to create a Time Routed Alias and first collection
> --
>
> Key: SOLR-11722
> URL: https://issues.apache.org/jira/browse/SOLR-11722
> Project: Solr
>  Issue Type: Sub-task
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: SolrCloud
>Reporter: David Smiley
> Attachments: SOLR-11722.patch, SOLR-11722.patch
>
>
> This issue is about creating a single API command to create a "Time Routed 
> Alias" along with its first collection.  Need to decide what endpoint URL it 
> is and parameters.
> Perhaps in v2 it'd be {{/api/collections?command=create-routed-alias}} or 
> alternatively piggy-back off of command=create-alias but we add more options, 
> perhaps with a prefix like "router"?
> Inputs:
> * alias name
> * misc collection creation metadata (e.g. config, numShards, ...) perhaps in 
> this context with a prefix like "collection."
> * metadata for TimeRoutedAliasUpdateProcessor, currently: router.field
> * date specifier for first collection; can include "date math".
> We'll certainly add more options as future features unfold.
> I believe the collection needs to be created first (referring to the alias 
> name via a core property), and then the alias pointing to it which demands 
> collections exist first.  When figuring the collection name, you'll need to 
> reference the format in TimeRoutedAliasUpdateProcessor.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-11722) API to create a Time Routed Alias and first collection

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-11722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16322997#comment-16322997
 ] 

ASF GitHub Bot commented on SOLR-11722:
---

Github user dsmiley commented on a diff in the pull request:

https://github.com/apache/lucene-solr/pull/304#discussion_r161074349
  
--- Diff: 
solr/core/src/java/org/apache/solr/handler/admin/CollectionsHandler.java ---
@@ -476,6 +451,31 @@ private static void 
addStatusToResponse(NamedList results, RequestStatus
   SolrIdentifierValidator.validateAliasName(req.getParams().get(NAME));
   return req.getParams().required().getAll(null, NAME, "collections");
 }),
+CREATEROUTEDALIAS_OP(CREATEROUTEDALIAS, (req, rsp, h) -> {
+  String alias = req.getParams().get(NAME);
+  SolrIdentifierValidator.validateAliasName(alias);
+  Map params = req.getParams().required()
+  .getAll(null, REQUIRED_ROUTING_PARAMS.toArray(new 
String[REQUIRED_ROUTING_PARAMS.size()]));
+  req.getParams().getAll(params, NONREQUIRED_ROUTING_PARAMS);
+  // subset the params to reuse the collection creation/parsing code
+  ModifiableSolrParams collectionParams = 
extractPrefixedParams("create-collection.", req.getParams());
+  if (collectionParams.get(NAME) != null) {
+SolrException solrException = new SolrException(BAD_REQUEST, 
"routed aliases calculate names for their " +
+"dependent collections, you cannot specify the name.");
+log.error("Could not create routed alias",solrException);
+throw solrException;
+  }
+  SolrParams v1Params = convertToV1WhenRequired(req, collectionParams);
+
+  // We need to add this temporary name just to pass validation.
--- End diff --

We need to insist that the collection config name is passed here.  It's 
very problematic to depend on automatic creation.  On a related note, when it's 
time to document TRA's, we may want to warn against data driven schema.


> API to create a Time Routed Alias and first collection
> --
>
> Key: SOLR-11722
> URL: https://issues.apache.org/jira/browse/SOLR-11722
> Project: Solr
>  Issue Type: Sub-task
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: SolrCloud
>Reporter: David Smiley
> Attachments: SOLR-11722.patch, SOLR-11722.patch
>
>
> This issue is about creating a single API command to create a "Time Routed 
> Alias" along with its first collection.  Need to decide what endpoint URL it 
> is and parameters.
> Perhaps in v2 it'd be {{/api/collections?command=create-routed-alias}} or 
> alternatively piggy-back off of command=create-alias but we add more options, 
> perhaps with a prefix like "router"?
> Inputs:
> * alias name
> * misc collection creation metadata (e.g. config, numShards, ...) perhaps in 
> this context with a prefix like "collection."
> * metadata for TimeRoutedAliasUpdateProcessor, currently: router.field
> * date specifier for first collection; can include "date math".
> We'll certainly add more options as future features unfold.
> I believe the collection needs to be created first (referring to the alias 
> name via a core property), and then the alias pointing to it which demands 
> collections exist first.  When figuring the collection name, you'll need to 
> reference the format in TimeRoutedAliasUpdateProcessor.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-11722) API to create a Time Routed Alias and first collection

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-11722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16322994#comment-16322994
 ] 

ASF GitHub Bot commented on SOLR-11722:
---

Github user dsmiley commented on a diff in the pull request:

https://github.com/apache/lucene-solr/pull/304#discussion_r161077363
  
--- Diff: solr/core/src/java/org/apache/solr/cloud/CreateAliasCmd.java ---
@@ -45,13 +147,92 @@ public CreateAliasCmd(OverseerCollectionMessageHandler 
ocmh) {
   public void call(ClusterState state, ZkNodeProps message, NamedList 
results)
   throws Exception {
 final String aliasName = message.getStr(NAME);
-final List canonicalCollectionList = 
parseCollectionsParameter(message.get("collections"));
-final String canonicalCollectionsString = 
StrUtils.join(canonicalCollectionList, ',');
-
 ZkStateReader zkStateReader = ocmh.zkStateReader;
-validateAllCollectionsExistAndNoDups(canonicalCollectionList, 
zkStateReader);
+ZkStateReader.AliasesManager holder = zkStateReader.aliasesHolder;
+if (!anyRoutingParams(message)) {
+  final List canonicalCollectionList = 
parseCollectionsParameter(message.get("collections"));
+  final String canonicalCollectionsString = 
StrUtils.join(canonicalCollectionList, ',');
+  validateAllCollectionsExistAndNoDups(canonicalCollectionList, 
zkStateReader);
+  holder.applyModificationAndExportToZk(aliases -> 
aliases.cloneWithCollectionAlias(aliasName, canonicalCollectionsString));
+} else {
+  final String routedField = message.getStr(ROUTING_FIELD);
+  final String routingType = message.getStr(ROUTING_TYPE);
+  final String tz = message.getStr(TZ);
+  final String start = message.getStr(START);
+  final String increment = message.getStr(ROUTING_INCREMENT);
+  final String maxFutureMs = message.getStr(ROUTING_MAX_FUTURE);
+
+  try {
+if (0 > Long.valueOf(maxFutureMs)) {
+  throw new NumberFormatException("Negative value not allowed 
here");
+}
+  } catch (NumberFormatException e) {
+throw new SolrException(BAD_REQUEST, ROUTING_MAX_FUTURE + " must 
be a valid long integer representing a number " +
+"of milliseconds greater than or equal to zero");
+  }
 
-zkStateReader.aliasesHolder.applyModificationAndExportToZk(aliases -> 
aliases.cloneWithCollectionAlias(aliasName, canonicalCollectionsString));
+  // Validate we got everything we need
+  if (routedField == null || routingType == null || start == null || 
increment == null) {
--- End diff --

I think much of these param validation checks are supposed to be handled at 
the CollectionsHandler level; no?  I'm not 100% sure.


> API to create a Time Routed Alias and first collection
> --
>
> Key: SOLR-11722
> URL: https://issues.apache.org/jira/browse/SOLR-11722
> Project: Solr
>  Issue Type: Sub-task
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: SolrCloud
>Reporter: David Smiley
> Attachments: SOLR-11722.patch, SOLR-11722.patch
>
>
> This issue is about creating a single API command to create a "Time Routed 
> Alias" along with its first collection.  Need to decide what endpoint URL it 
> is and parameters.
> Perhaps in v2 it'd be {{/api/collections?command=create-routed-alias}} or 
> alternatively piggy-back off of command=create-alias but we add more options, 
> perhaps with a prefix like "router"?
> Inputs:
> * alias name
> * misc collection creation metadata (e.g. config, numShards, ...) perhaps in 
> this context with a prefix like "collection."
> * metadata for TimeRoutedAliasUpdateProcessor, currently: router.field
> * date specifier for first collection; can include "date math".
> We'll certainly add more options as future features unfold.
> I believe the collection needs to be created first (referring to the alias 
> name via a core property), and then the alias pointing to it which demands 
> collections exist first.  When figuring the collection name, you'll need to 
> reference the format in TimeRoutedAliasUpdateProcessor.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-11722) API to create a Time Routed Alias and first collection

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-11722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16323000#comment-16323000
 ] 

ASF GitHub Bot commented on SOLR-11722:
---

Github user dsmiley commented on a diff in the pull request:

https://github.com/apache/lucene-solr/pull/304#discussion_r161078867
  
--- Diff: solr/core/src/java/org/apache/solr/cloud/CreateAliasCmd.java ---
@@ -68,34 +249,100 @@ public void call(ClusterState state, ZkNodeProps 
message, NamedList results)
 Thread.sleep(100);
   }
 
+  private Map buildAliasMap(String routedField, String 
routingType, String tz, String increment, String maxFutureMs, ZkNodeProps 
collectionProps) {
+Map properties = collectionProps.getProperties();
+Map cleanMap = properties.entrySet().stream()
+.filter(stringObjectEntry ->
+!"fromApi".equals(stringObjectEntry.getKey())
+&& !"stateFormat".equals(stringObjectEntry.getKey())
+&& !"name".equals(stringObjectEntry.getKey()))
+.collect(Collectors.toMap((e) -> "collection-create." + 
e.getKey(), e -> String.valueOf(e.getValue(;
+cleanMap.put(ROUTING_FIELD, routedField);
+cleanMap.put(ROUTING_TYPE, routingType);
+cleanMap.put(ROUTING_INCREMENT, increment);
+cleanMap.put(ROUTING_MAX_FUTURE, maxFutureMs);
+cleanMap.put(TZ, tz);
+return cleanMap;
+  }
+
+  private Instant validateStart(TimeZone zone, DateTimeFormatter fmt, 
String start) {
--- End diff --

This appears redundant with using DateMathParser earlier and I'm sure we 
don't need so much code for this matter (ignore millisecond truncation; in 
practice the user is going to do something like `/MONTH` or `/DAY` for start.


> API to create a Time Routed Alias and first collection
> --
>
> Key: SOLR-11722
> URL: https://issues.apache.org/jira/browse/SOLR-11722
> Project: Solr
>  Issue Type: Sub-task
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: SolrCloud
>Reporter: David Smiley
> Attachments: SOLR-11722.patch, SOLR-11722.patch
>
>
> This issue is about creating a single API command to create a "Time Routed 
> Alias" along with its first collection.  Need to decide what endpoint URL it 
> is and parameters.
> Perhaps in v2 it'd be {{/api/collections?command=create-routed-alias}} or 
> alternatively piggy-back off of command=create-alias but we add more options, 
> perhaps with a prefix like "router"?
> Inputs:
> * alias name
> * misc collection creation metadata (e.g. config, numShards, ...) perhaps in 
> this context with a prefix like "collection."
> * metadata for TimeRoutedAliasUpdateProcessor, currently: router.field
> * date specifier for first collection; can include "date math".
> We'll certainly add more options as future features unfold.
> I believe the collection needs to be created first (referring to the alias 
> name via a core property), and then the alias pointing to it which demands 
> collections exist first.  When figuring the collection name, you'll need to 
> reference the format in TimeRoutedAliasUpdateProcessor.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-11722) API to create a Time Routed Alias and first collection

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-11722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16322999#comment-16322999
 ] 

ASF GitHub Bot commented on SOLR-11722:
---

Github user dsmiley commented on a diff in the pull request:

https://github.com/apache/lucene-solr/pull/304#discussion_r161075027
  
--- Diff: solr/core/src/java/org/apache/solr/cloud/CreateAliasCmd.java ---
@@ -45,13 +147,92 @@ public CreateAliasCmd(OverseerCollectionMessageHandler 
ocmh) {
   public void call(ClusterState state, ZkNodeProps message, NamedList 
results)
   throws Exception {
 final String aliasName = message.getStr(NAME);
-final List canonicalCollectionList = 
parseCollectionsParameter(message.get("collections"));
-final String canonicalCollectionsString = 
StrUtils.join(canonicalCollectionList, ',');
-
 ZkStateReader zkStateReader = ocmh.zkStateReader;
-validateAllCollectionsExistAndNoDups(canonicalCollectionList, 
zkStateReader);
+ZkStateReader.AliasesManager holder = zkStateReader.aliasesHolder;
+if (!anyRoutingParams(message)) {
+  final List canonicalCollectionList = 
parseCollectionsParameter(message.get("collections"));
+  final String canonicalCollectionsString = 
StrUtils.join(canonicalCollectionList, ',');
+  validateAllCollectionsExistAndNoDups(canonicalCollectionList, 
zkStateReader);
+  holder.applyModificationAndExportToZk(aliases -> 
aliases.cloneWithCollectionAlias(aliasName, canonicalCollectionsString));
+} else {
+  final String routedField = message.getStr(ROUTING_FIELD);
+  final String routingType = message.getStr(ROUTING_TYPE);
+  final String tz = message.getStr(TZ);
+  final String start = message.getStr(START);
+  final String increment = message.getStr(ROUTING_INCREMENT);
+  final String maxFutureMs = message.getStr(ROUTING_MAX_FUTURE);
+
+  try {
+if (0 > Long.valueOf(maxFutureMs)) {
+  throw new NumberFormatException("Negative value not allowed 
here");
+}
+  } catch (NumberFormatException e) {
+throw new SolrException(BAD_REQUEST, ROUTING_MAX_FUTURE + " must 
be a valid long integer representing a number " +
+"of milliseconds greater than or equal to zero");
+  }
 
-zkStateReader.aliasesHolder.applyModificationAndExportToZk(aliases -> 
aliases.cloneWithCollectionAlias(aliasName, canonicalCollectionsString));
+  // Validate we got everything we need
+  if (routedField == null || routingType == null || start == null || 
increment == null) {
+SolrException solrException = new SolrException(BAD_REQUEST, "If 
any of " + CREATE_ROUTED_ALIAS_PARAMS +
+" are supplied, then all of " + REQUIRED_ROUTING_PARAMS + " 
must be present.");
+log.error("Could not create routed alias",solrException);
+throw solrException;
+  }
+
+  if (!"time".equals(routingType)) {
+SolrException solrException = new SolrException(BAD_REQUEST, "Only 
time based routing is supported at this time");
+log.error("Could not create routed alias",solrException);
+throw solrException;
+  }
+  // Check for invalid timezone
+  if(tz != null && !TimeZoneUtils.KNOWN_TIMEZONE_IDS.contains(tz)) {
+SolrException solrException = new SolrException(BAD_REQUEST, 
"Invalid timezone:" + tz);
+log.error("Could not create routed alias",solrException);
+throw solrException;
+
+  }
+  TimeZone zone;
+  if (tz != null) {
+zone = TimeZoneUtils.getTimeZone(tz);
+  } else {
+zone = TimeZoneUtils.getTimeZone("UTC");
+  }
+  DateTimeFormatter fmt = 
DATE_TIME_FORMATTER.withZone(zone.toZoneId());
--- End diff --

No we don't want to use the TZ to format the collection name


> API to create a Time Routed Alias and first collection
> --
>
> Key: SOLR-11722
> URL: https://issues.apache.org/jira/browse/SOLR-11722
> Project: Solr
>  Issue Type: Sub-task
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: SolrCloud
>Reporter: David Smiley
> Attachments: SOLR-11722.patch, SOLR-11722.patch
>
>
> This issue is about creating a single API command to create a "Time Routed 
> Alias" along with its first collection.  Need to decide what endpoint URL it 
> is and parameters.
> Perhaps in v2 it'd be {{/api/collections?command=create-routed-alias}} or 
> alternatively piggy-back off of command=create-alias but we add more options, 
> perhaps with a prefix like "router"?
> Inputs:
> * alias name
> * 

[jira] [Commented] (SOLR-11722) API to create a Time Routed Alias and first collection

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-11722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16322996#comment-16322996
 ] 

ASF GitHub Bot commented on SOLR-11722:
---

Github user dsmiley commented on a diff in the pull request:

https://github.com/apache/lucene-solr/pull/304#discussion_r161079147
  
--- Diff: solr/core/src/java/org/apache/solr/cloud/CreateAliasCmd.java ---
@@ -68,34 +249,100 @@ public void call(ClusterState state, ZkNodeProps 
message, NamedList results)
 Thread.sleep(100);
   }
 
+  private Map buildAliasMap(String routedField, String 
routingType, String tz, String increment, String maxFutureMs, ZkNodeProps 
collectionProps) {
+Map properties = collectionProps.getProperties();
+Map cleanMap = properties.entrySet().stream()
+.filter(stringObjectEntry ->
+!"fromApi".equals(stringObjectEntry.getKey())
+&& !"stateFormat".equals(stringObjectEntry.getKey())
+&& !"name".equals(stringObjectEntry.getKey()))
+.collect(Collectors.toMap((e) -> "collection-create." + 
e.getKey(), e -> String.valueOf(e.getValue(;
+cleanMap.put(ROUTING_FIELD, routedField);
+cleanMap.put(ROUTING_TYPE, routingType);
+cleanMap.put(ROUTING_INCREMENT, increment);
+cleanMap.put(ROUTING_MAX_FUTURE, maxFutureMs);
+cleanMap.put(TZ, tz);
+return cleanMap;
+  }
+
+  private Instant validateStart(TimeZone zone, DateTimeFormatter fmt, 
String start) {
+// This is the normal/easy case, if we can get away with this great!
+TemporalAccessor startTime = attemptTimeStampParsing(start, 
zone.toZoneId());
+if (startTime == null) {
+  // No luck, they gave us either date math, or garbage, so we have to 
do more work to figure out which and
+  // to make sure it's valid date math and that it doesn't encode any 
millisecond precision.
+  ZonedDateTime now = 
ZonedDateTime.now(zone.toZoneId()).truncatedTo(ChronoUnit.MILLIS);
+  try {
+Date date = DateMathParser.parseMath(Date.from(now.toInstant()), 
start);
+String reformatted = 
fmt.format(date.toInstant().truncatedTo(ChronoUnit.MILLIS));
+Date reparse = 
Date.from(Instant.from(DATE_TIME_FORMATTER.parse(reformatted)));
+if (!reparse.equals(date)) {
+  throw new SolrException(BAD_REQUEST,
+  "Formatted time did not have the same milliseconds as 
original: " + date.getTime() + " vs. " +
+  reparse.getTime() + " This indicates that you used date 
math that includes milliseconds. " +
+  "(Hint: 'NOW' used without rounding always has this 
problem)" );
+}
+return date.toInstant();
+  } catch (SolrException e) {
+throw new SolrException(BAD_REQUEST,
+"Start Time for the first collection must be a timestamp of 
the format -MM-dd_HH_mm_ss, " +
+"or a valid date math expression not containing specific 
milliseconds", e);
+  }
+}
+return Instant.from(startTime);
+  }
+
+  private TemporalAccessor attemptTimeStampParsing(String start, ZoneId 
zone) {
+try {
+  DATE_TIME_FORMATTER.withZone(zone);
+  return DATE_TIME_FORMATTER.parse(start);
+} catch (DateTimeParseException e) {
+  return null;
+}
+  }
+
+  private boolean anyRoutingParams(ZkNodeProps message) {
+
+return message.containsKey(ROUTING_FIELD) || 
message.containsKey(ROUTING_TYPE) || message.containsKey(START)
+|| message.containsKey(ROUTING_INCREMENT) || 
message.containsKey(TZ);
+  }
+
   private void validateAllCollectionsExistAndNoDups(List 
collectionList, ZkStateReader zkStateReader) {
 final String collectionStr = StrUtils.join(collectionList, ',');
 
 if (new HashSet<>(collectionList).size() != collectionList.size()) {
-  throw new SolrException(SolrException.ErrorCode.BAD_REQUEST,
+  throw new SolrException(BAD_REQUEST,
   String.format(Locale.ROOT,  "Can't create collection alias for 
collections='%s', since it contains duplicates", collectionStr));
 }
 ClusterState clusterState = zkStateReader.getClusterState();
 Set aliasNames = 
zkStateReader.getAliases().getCollectionAliasListMap().keySet();
 for (String collection : collectionList) {
   if (clusterState.getCollectionOrNull(collection) == null && 
!aliasNames.contains(collection)) {
-throw new SolrException(SolrException.ErrorCode.BAD_REQUEST,
+throw new SolrException(BAD_REQUEST,
 String.format(Locale.ROOT,  "Can't create collection alias for 
collections='%s', '%s' 

[jira] [Commented] (SOLR-11722) API to create a Time Routed Alias and first collection

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-11722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16322992#comment-16322992
 ] 

ASF GitHub Bot commented on SOLR-11722:
---

Github user dsmiley commented on a diff in the pull request:

https://github.com/apache/lucene-solr/pull/304#discussion_r161074912
  
--- Diff: solr/core/src/java/org/apache/solr/cloud/CreateAliasCmd.java ---
@@ -45,13 +147,92 @@ public CreateAliasCmd(OverseerCollectionMessageHandler 
ocmh) {
   public void call(ClusterState state, ZkNodeProps message, NamedList 
results)
   throws Exception {
 final String aliasName = message.getStr(NAME);
-final List canonicalCollectionList = 
parseCollectionsParameter(message.get("collections"));
-final String canonicalCollectionsString = 
StrUtils.join(canonicalCollectionList, ',');
-
 ZkStateReader zkStateReader = ocmh.zkStateReader;
-validateAllCollectionsExistAndNoDups(canonicalCollectionList, 
zkStateReader);
+ZkStateReader.AliasesManager holder = zkStateReader.aliasesHolder;
+if (!anyRoutingParams(message)) {
+  final List canonicalCollectionList = 
parseCollectionsParameter(message.get("collections"));
+  final String canonicalCollectionsString = 
StrUtils.join(canonicalCollectionList, ',');
+  validateAllCollectionsExistAndNoDups(canonicalCollectionList, 
zkStateReader);
+  holder.applyModificationAndExportToZk(aliases -> 
aliases.cloneWithCollectionAlias(aliasName, canonicalCollectionsString));
+} else {
+  final String routedField = message.getStr(ROUTING_FIELD);
+  final String routingType = message.getStr(ROUTING_TYPE);
+  final String tz = message.getStr(TZ);
+  final String start = message.getStr(START);
+  final String increment = message.getStr(ROUTING_INCREMENT);
+  final String maxFutureMs = message.getStr(ROUTING_MAX_FUTURE);
+
+  try {
+if (0 > Long.valueOf(maxFutureMs)) {
+  throw new NumberFormatException("Negative value not allowed 
here");
+}
+  } catch (NumberFormatException e) {
+throw new SolrException(BAD_REQUEST, ROUTING_MAX_FUTURE + " must 
be a valid long integer representing a number " +
+"of milliseconds greater than or equal to zero");
+  }
 
-zkStateReader.aliasesHolder.applyModificationAndExportToZk(aliases -> 
aliases.cloneWithCollectionAlias(aliasName, canonicalCollectionsString));
+  // Validate we got everything we need
+  if (routedField == null || routingType == null || start == null || 
increment == null) {
+SolrException solrException = new SolrException(BAD_REQUEST, "If 
any of " + CREATE_ROUTED_ALIAS_PARAMS +
+" are supplied, then all of " + REQUIRED_ROUTING_PARAMS + " 
must be present.");
+log.error("Could not create routed alias",solrException);
+throw solrException;
+  }
+
+  if (!"time".equals(routingType)) {
+SolrException solrException = new SolrException(BAD_REQUEST, "Only 
time based routing is supported at this time");
+log.error("Could not create routed alias",solrException);
+throw solrException;
+  }
+  // Check for invalid timezone
+  if(tz != null && !TimeZoneUtils.KNOWN_TIMEZONE_IDS.contains(tz)) {
--- End diff --

You needn't explicitly validate TZ here; call TimeZoneUtils.parseTimezone 
instead which does so (which I added recently).


> API to create a Time Routed Alias and first collection
> --
>
> Key: SOLR-11722
> URL: https://issues.apache.org/jira/browse/SOLR-11722
> Project: Solr
>  Issue Type: Sub-task
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: SolrCloud
>Reporter: David Smiley
> Attachments: SOLR-11722.patch, SOLR-11722.patch
>
>
> This issue is about creating a single API command to create a "Time Routed 
> Alias" along with its first collection.  Need to decide what endpoint URL it 
> is and parameters.
> Perhaps in v2 it'd be {{/api/collections?command=create-routed-alias}} or 
> alternatively piggy-back off of command=create-alias but we add more options, 
> perhaps with a prefix like "router"?
> Inputs:
> * alias name
> * misc collection creation metadata (e.g. config, numShards, ...) perhaps in 
> this context with a prefix like "collection."
> * metadata for TimeRoutedAliasUpdateProcessor, currently: router.field
> * date specifier for first collection; can include "date math".
> We'll certainly add more options as future features unfold.
> I believe the collection needs to be created first (referring to the alias 
> name via a core property), and then 

[jira] [Commented] (SOLR-11722) API to create a Time Routed Alias and first collection

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-11722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16322998#comment-16322998
 ] 

ASF GitHub Bot commented on SOLR-11722:
---

Github user dsmiley commented on a diff in the pull request:

https://github.com/apache/lucene-solr/pull/304#discussion_r161079495
  
--- Diff: 
solr/core/src/test/org/apache/solr/cloud/CreateRoutedAliasTest.java ---
@@ -0,0 +1,351 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.solr.cloud;
+
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.time.Instant;
+import java.time.format.DateTimeFormatter;
+import java.time.temporal.ChronoUnit;
+import java.util.Date;
+import java.util.Locale;
+import java.util.Map;
+
+import com.fasterxml.jackson.core.type.TypeReference;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.http.HttpEntity;
+import org.apache.http.client.methods.CloseableHttpResponse;
+import org.apache.http.client.methods.HttpGet;
+import org.apache.http.client.methods.HttpPost;
+import org.apache.http.entity.ContentType;
+import org.apache.http.entity.InputStreamEntity;
+import org.apache.http.impl.client.CloseableHttpClient;
+import org.apache.http.impl.client.HttpClients;
+import org.apache.lucene.util.IOUtils;
+import org.apache.solr.SolrTestCaseJ4;
+import org.apache.solr.client.solrj.impl.CloudSolrClient;
+import org.apache.solr.client.solrj.request.CollectionAdminRequest;
+import org.apache.solr.common.cloud.Aliases;
+import org.apache.solr.common.cloud.ZkStateReader;
+import org.apache.solr.update.processor.TimeRoutedAliasUpdateProcessor;
+import org.apache.solr.util.DateMathParser;
+import org.junit.After;
+import org.junit.AfterClass;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.Test;
+
+/**
+ * Direct http tests of the CreateRoutedAlias functionality.
+ */
+public class CreateRoutedAliasTest extends SolrCloudTestCase {
--- End diff --

Hmm; it'd be nice to combine this with the existing test to just have one 
TimeRoutedAliases test.  It can wait.


> API to create a Time Routed Alias and first collection
> --
>
> Key: SOLR-11722
> URL: https://issues.apache.org/jira/browse/SOLR-11722
> Project: Solr
>  Issue Type: Sub-task
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: SolrCloud
>Reporter: David Smiley
> Attachments: SOLR-11722.patch, SOLR-11722.patch
>
>
> This issue is about creating a single API command to create a "Time Routed 
> Alias" along with its first collection.  Need to decide what endpoint URL it 
> is and parameters.
> Perhaps in v2 it'd be {{/api/collections?command=create-routed-alias}} or 
> alternatively piggy-back off of command=create-alias but we add more options, 
> perhaps with a prefix like "router"?
> Inputs:
> * alias name
> * misc collection creation metadata (e.g. config, numShards, ...) perhaps in 
> this context with a prefix like "collection."
> * metadata for TimeRoutedAliasUpdateProcessor, currently: router.field
> * date specifier for first collection; can include "date math".
> We'll certainly add more options as future features unfold.
> I believe the collection needs to be created first (referring to the alias 
> name via a core property), and then the alias pointing to it which demands 
> collections exist first.  When figuring the collection name, you'll need to 
> reference the format in TimeRoutedAliasUpdateProcessor.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-11722) API to create a Time Routed Alias and first collection

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-11722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16322990#comment-16322990
 ] 

ASF GitHub Bot commented on SOLR-11722:
---

Github user dsmiley commented on a diff in the pull request:

https://github.com/apache/lucene-solr/pull/304#discussion_r161074624
  
--- Diff: solr/core/src/java/org/apache/solr/cloud/CreateAliasCmd.java ---
@@ -45,13 +147,92 @@ public CreateAliasCmd(OverseerCollectionMessageHandler 
ocmh) {
   public void call(ClusterState state, ZkNodeProps message, NamedList 
results)
   throws Exception {
 final String aliasName = message.getStr(NAME);
-final List canonicalCollectionList = 
parseCollectionsParameter(message.get("collections"));
-final String canonicalCollectionsString = 
StrUtils.join(canonicalCollectionList, ',');
-
 ZkStateReader zkStateReader = ocmh.zkStateReader;
-validateAllCollectionsExistAndNoDups(canonicalCollectionList, 
zkStateReader);
+ZkStateReader.AliasesManager holder = zkStateReader.aliasesHolder;
+if (!anyRoutingParams(message)) {
+  final List canonicalCollectionList = 
parseCollectionsParameter(message.get("collections"));
+  final String canonicalCollectionsString = 
StrUtils.join(canonicalCollectionList, ',');
+  validateAllCollectionsExistAndNoDups(canonicalCollectionList, 
zkStateReader);
+  holder.applyModificationAndExportToZk(aliases -> 
aliases.cloneWithCollectionAlias(aliasName, canonicalCollectionsString));
+} else {
+  final String routedField = message.getStr(ROUTING_FIELD);
+  final String routingType = message.getStr(ROUTING_TYPE);
+  final String tz = message.getStr(TZ);
+  final String start = message.getStr(START);
+  final String increment = message.getStr(ROUTING_INCREMENT);
+  final String maxFutureMs = message.getStr(ROUTING_MAX_FUTURE);
+
+  try {
+if (0 > Long.valueOf(maxFutureMs)) {
--- End diff --

use parseLong


> API to create a Time Routed Alias and first collection
> --
>
> Key: SOLR-11722
> URL: https://issues.apache.org/jira/browse/SOLR-11722
> Project: Solr
>  Issue Type: Sub-task
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: SolrCloud
>Reporter: David Smiley
> Attachments: SOLR-11722.patch, SOLR-11722.patch
>
>
> This issue is about creating a single API command to create a "Time Routed 
> Alias" along with its first collection.  Need to decide what endpoint URL it 
> is and parameters.
> Perhaps in v2 it'd be {{/api/collections?command=create-routed-alias}} or 
> alternatively piggy-back off of command=create-alias but we add more options, 
> perhaps with a prefix like "router"?
> Inputs:
> * alias name
> * misc collection creation metadata (e.g. config, numShards, ...) perhaps in 
> this context with a prefix like "collection."
> * metadata for TimeRoutedAliasUpdateProcessor, currently: router.field
> * date specifier for first collection; can include "date math".
> We'll certainly add more options as future features unfold.
> I believe the collection needs to be created first (referring to the alias 
> name via a core property), and then the alias pointing to it which demands 
> collections exist first.  When figuring the collection name, you'll need to 
> reference the format in TimeRoutedAliasUpdateProcessor.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-11722) API to create a Time Routed Alias and first collection

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-11722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16322991#comment-16322991
 ] 

ASF GitHub Bot commented on SOLR-11722:
---

Github user dsmiley commented on a diff in the pull request:

https://github.com/apache/lucene-solr/pull/304#discussion_r161078056
  
--- Diff: solr/core/src/java/org/apache/solr/cloud/CreateAliasCmd.java ---
@@ -68,34 +249,100 @@ public void call(ClusterState state, ZkNodeProps 
message, NamedList results)
 Thread.sleep(100);
   }
 
+  private Map buildAliasMap(String routedField, String 
routingType, String tz, String increment, String maxFutureMs, ZkNodeProps 
collectionProps) {
+Map properties = collectionProps.getProperties();
+Map cleanMap = properties.entrySet().stream()
+.filter(stringObjectEntry ->
+!"fromApi".equals(stringObjectEntry.getKey())
--- End diff --

These 3 constants here are unfortunate maybe they could be in a 
constant Set and checked here?


> API to create a Time Routed Alias and first collection
> --
>
> Key: SOLR-11722
> URL: https://issues.apache.org/jira/browse/SOLR-11722
> Project: Solr
>  Issue Type: Sub-task
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: SolrCloud
>Reporter: David Smiley
> Attachments: SOLR-11722.patch, SOLR-11722.patch
>
>
> This issue is about creating a single API command to create a "Time Routed 
> Alias" along with its first collection.  Need to decide what endpoint URL it 
> is and parameters.
> Perhaps in v2 it'd be {{/api/collections?command=create-routed-alias}} or 
> alternatively piggy-back off of command=create-alias but we add more options, 
> perhaps with a prefix like "router"?
> Inputs:
> * alias name
> * misc collection creation metadata (e.g. config, numShards, ...) perhaps in 
> this context with a prefix like "collection."
> * metadata for TimeRoutedAliasUpdateProcessor, currently: router.field
> * date specifier for first collection; can include "date math".
> We'll certainly add more options as future features unfold.
> I believe the collection needs to be created first (referring to the alias 
> name via a core property), and then the alias pointing to it which demands 
> collections exist first.  When figuring the collection name, you'll need to 
> reference the format in TimeRoutedAliasUpdateProcessor.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-11722) API to create a Time Routed Alias and first collection

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-11722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16322989#comment-16322989
 ] 

ASF GitHub Bot commented on SOLR-11722:
---

Github user dsmiley commented on a diff in the pull request:

https://github.com/apache/lucene-solr/pull/304#discussion_r161071261
  
--- Diff: 
solr/core/src/java/org/apache/solr/cloud/OverseerCollectionMessageHandler.java 
---
@@ -218,6 +218,7 @@ public OverseerCollectionMessageHandler(ZkStateReader 
zkStateReader, String myId
 .put(RELOAD, this::reloadCollection)
 .put(DELETE, new DeleteCollectionCmd(this))
 .put(CREATEALIAS, new CreateAliasCmd(this))
+.put(CREATEROUTEDALIAS, new CreateAliasCmd(this))
--- End diff --

Hmm; is there a point to this?  I guess it's debatable.


> API to create a Time Routed Alias and first collection
> --
>
> Key: SOLR-11722
> URL: https://issues.apache.org/jira/browse/SOLR-11722
> Project: Solr
>  Issue Type: Sub-task
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: SolrCloud
>Reporter: David Smiley
> Attachments: SOLR-11722.patch, SOLR-11722.patch
>
>
> This issue is about creating a single API command to create a "Time Routed 
> Alias" along with its first collection.  Need to decide what endpoint URL it 
> is and parameters.
> Perhaps in v2 it'd be {{/api/collections?command=create-routed-alias}} or 
> alternatively piggy-back off of command=create-alias but we add more options, 
> perhaps with a prefix like "router"?
> Inputs:
> * alias name
> * misc collection creation metadata (e.g. config, numShards, ...) perhaps in 
> this context with a prefix like "collection."
> * metadata for TimeRoutedAliasUpdateProcessor, currently: router.field
> * date specifier for first collection; can include "date math".
> We'll certainly add more options as future features unfold.
> I believe the collection needs to be created first (referring to the alias 
> name via a core property), and then the alias pointing to it which demands 
> collections exist first.  When figuring the collection name, you'll need to 
> reference the format in TimeRoutedAliasUpdateProcessor.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-11722) API to create a Time Routed Alias and first collection

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-11722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16322988#comment-16322988
 ] 

ASF GitHub Bot commented on SOLR-11722:
---

Github user dsmiley commented on a diff in the pull request:

https://github.com/apache/lucene-solr/pull/304#discussion_r161071815
  
--- Diff: solr/core/src/java/org/apache/solr/cloud/CreateAliasCmd.java ---
@@ -30,13 +44,101 @@
 import org.apache.solr.common.cloud.ZkStateReader;
 import org.apache.solr.common.util.NamedList;
 import org.apache.solr.common.util.StrUtils;
+import org.apache.solr.update.processor.TimeRoutedAliasUpdateProcessor;
+import org.apache.solr.util.DateMathParser;
+import org.apache.solr.util.TimeZoneUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
 
+import static java.time.format.DateTimeFormatter.ISO_INSTANT;
+import static 
org.apache.solr.cloud.OverseerCollectionMessageHandler.COLL_CONF;
+import static org.apache.solr.common.SolrException.ErrorCode.BAD_REQUEST;
 import static org.apache.solr.common.params.CommonParams.NAME;
+import static org.apache.solr.common.params.CommonParams.TZ;
+import static 
org.apache.solr.handler.admin.CollectionsHandler.ROUTED_ALIAS_COLLECTION_PROP_PFX;
+import static 
org.apache.solr.update.processor.TimeRoutedAliasUpdateProcessor.DATE_TIME_FORMATTER;
 
 
 public class CreateAliasCmd implements Cmd {
+
+  private static final Logger log = 
LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+  public static final String ROUTING_TYPE = "router.name";
+  public static final String ROUTING_FIELD = "router.field";
+  public static final String ROUTING_INCREMENT = "router.interval";
+  public static final String ROUTING_MAX_FUTURE = "router.max-future-ms";
+  public static final String START = "start";
+  // Collection constants should all reflect names in the v2 structured 
input for this command, not v1
+  // names used for CREATE
+  public static final String CREATE_COLLECTION_CONFIG = 
"create-collection.config";
--- End diff --

Ugh, all those CREATE_COLLECTION_* constants are a maintenance burden; 
surely we can avoid them?  I really want to avoid double-specificity of the 
collection-creation stuff in any shape or form (not in docs/constants/etc.)


> API to create a Time Routed Alias and first collection
> --
>
> Key: SOLR-11722
> URL: https://issues.apache.org/jira/browse/SOLR-11722
> Project: Solr
>  Issue Type: Sub-task
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: SolrCloud
>Reporter: David Smiley
> Attachments: SOLR-11722.patch, SOLR-11722.patch
>
>
> This issue is about creating a single API command to create a "Time Routed 
> Alias" along with its first collection.  Need to decide what endpoint URL it 
> is and parameters.
> Perhaps in v2 it'd be {{/api/collections?command=create-routed-alias}} or 
> alternatively piggy-back off of command=create-alias but we add more options, 
> perhaps with a prefix like "router"?
> Inputs:
> * alias name
> * misc collection creation metadata (e.g. config, numShards, ...) perhaps in 
> this context with a prefix like "collection."
> * metadata for TimeRoutedAliasUpdateProcessor, currently: router.field
> * date specifier for first collection; can include "date math".
> We'll certainly add more options as future features unfold.
> I believe the collection needs to be created first (referring to the alias 
> name via a core property), and then the alias pointing to it which demands 
> collections exist first.  When figuring the collection name, you'll need to 
> reference the format in TimeRoutedAliasUpdateProcessor.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-11722) API to create a Time Routed Alias and first collection

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-11722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16322987#comment-16322987
 ] 

ASF GitHub Bot commented on SOLR-11722:
---

Github user dsmiley commented on a diff in the pull request:

https://github.com/apache/lucene-solr/pull/304#discussion_r161070349
  
--- Diff: 
solr/core/src/java/org/apache/solr/update/processor/TimeRoutedAliasUpdateProcessor.java
 ---
@@ -99,6 +101,8 @@
   .parseDefaulting(ChronoField.HOUR_OF_DAY, 0)
   .parseDefaulting(ChronoField.MINUTE_OF_HOUR, 0)
   .parseDefaulting(ChronoField.SECOND_OF_MINUTE, 0)
+  // Setting a timezone here is fine as a default, but generally need 
to clone with user's timezone, so that
+  // truncation of milliseconds is consistent.
--- End diff --

As I indicated in a previous comment on JIRA, this comment isn't true.  We 
deliberately make the formatted collection name *insensitive* to the timezone.  
Heck we ought to say that here.


> API to create a Time Routed Alias and first collection
> --
>
> Key: SOLR-11722
> URL: https://issues.apache.org/jira/browse/SOLR-11722
> Project: Solr
>  Issue Type: Sub-task
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: SolrCloud
>Reporter: David Smiley
> Attachments: SOLR-11722.patch, SOLR-11722.patch
>
>
> This issue is about creating a single API command to create a "Time Routed 
> Alias" along with its first collection.  Need to decide what endpoint URL it 
> is and parameters.
> Perhaps in v2 it'd be {{/api/collections?command=create-routed-alias}} or 
> alternatively piggy-back off of command=create-alias but we add more options, 
> perhaps with a prefix like "router"?
> Inputs:
> * alias name
> * misc collection creation metadata (e.g. config, numShards, ...) perhaps in 
> this context with a prefix like "collection."
> * metadata for TimeRoutedAliasUpdateProcessor, currently: router.field
> * date specifier for first collection; can include "date math".
> We'll certainly add more options as future features unfold.
> I believe the collection needs to be created first (referring to the alias 
> name via a core property), and then the alias pointing to it which demands 
> collections exist first.  When figuring the collection name, you'll need to 
> reference the format in TimeRoutedAliasUpdateProcessor.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-11722) API to create a Time Routed Alias and first collection

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-11722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16322985#comment-16322985
 ] 

ASF GitHub Bot commented on SOLR-11722:
---

Github user dsmiley commented on a diff in the pull request:

https://github.com/apache/lucene-solr/pull/304#discussion_r161068781
  
--- Diff: 
solr/core/src/java/org/apache/solr/handler/admin/CollectionsHandler.java ---
@@ -476,6 +451,31 @@ private static void 
addStatusToResponse(NamedList results, RequestStatus
   SolrIdentifierValidator.validateAliasName(req.getParams().get(NAME));
   return req.getParams().required().getAll(null, NAME, "collections");
 }),
+CREATEROUTEDALIAS_OP(CREATEROUTEDALIAS, (req, rsp, h) -> {
+  String alias = req.getParams().get(NAME);
+  SolrIdentifierValidator.validateAliasName(alias);
+  Map params = req.getParams().required()
+  .getAll(null, REQUIRED_ROUTING_PARAMS.toArray(new 
String[REQUIRED_ROUTING_PARAMS.size()]));
+  req.getParams().getAll(params, NONREQUIRED_ROUTING_PARAMS);
+  // subset the params to reuse the collection creation/parsing code
+  ModifiableSolrParams collectionParams = 
extractPrefixedParams("create-collection.", req.getParams());
+  if (collectionParams.get(NAME) != null) {
+SolrException solrException = new SolrException(BAD_REQUEST, 
"routed aliases calculate names for their " +
+"dependent collections, you cannot specify the name.");
+log.error("Could not create routed alias",solrException);
--- End diff --

I believe the general practice for BAD_REQUEST param validation is for Solr 
to not log it.  Besides, it's redundant with throwing the exception, and any 
exception thrown could be chosen to log or not centrally at some entrypoint 
juncture or by a thread handler for uncaught exceptions.  So lets not litter 
log statements that are superfluous unless there's some interesting/useful 
context.



> API to create a Time Routed Alias and first collection
> --
>
> Key: SOLR-11722
> URL: https://issues.apache.org/jira/browse/SOLR-11722
> Project: Solr
>  Issue Type: Sub-task
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: SolrCloud
>Reporter: David Smiley
> Attachments: SOLR-11722.patch, SOLR-11722.patch
>
>
> This issue is about creating a single API command to create a "Time Routed 
> Alias" along with its first collection.  Need to decide what endpoint URL it 
> is and parameters.
> Perhaps in v2 it'd be {{/api/collections?command=create-routed-alias}} or 
> alternatively piggy-back off of command=create-alias but we add more options, 
> perhaps with a prefix like "router"?
> Inputs:
> * alias name
> * misc collection creation metadata (e.g. config, numShards, ...) perhaps in 
> this context with a prefix like "collection."
> * metadata for TimeRoutedAliasUpdateProcessor, currently: router.field
> * date specifier for first collection; can include "date math".
> We'll certainly add more options as future features unfold.
> I believe the collection needs to be created first (referring to the alias 
> name via a core property), and then the alias pointing to it which demands 
> collections exist first.  When figuring the collection name, you'll need to 
> reference the format in TimeRoutedAliasUpdateProcessor.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-11722) API to create a Time Routed Alias and first collection

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-11722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16322986#comment-16322986
 ] 

ASF GitHub Bot commented on SOLR-11722:
---

Github user dsmiley commented on a diff in the pull request:

https://github.com/apache/lucene-solr/pull/304#discussion_r161070053
  
--- Diff: 
solr/core/src/java/org/apache/solr/update/processor/TimeRoutedAliasUpdateProcessor.java
 ---
@@ -252,6 +256,7 @@ public void processAdd(AddUpdateCommand cmd) throws 
IOException {
   /** Computes the timestamp of the next collection given the timestamp of 
the one before. */
   public static Instant computeNextCollTimestamp(Instant fromTimestamp, 
String intervalDateMath, TimeZone intervalTimeZone) {
 //TODO overload DateMathParser.parseMath to take tz and "now"
+// GH: I don't think that's necessary, you can set the TZ on now when 
you pass it in?
--- End diff --

True that it's not necessary; it's just a proposed convenience.


> API to create a Time Routed Alias and first collection
> --
>
> Key: SOLR-11722
> URL: https://issues.apache.org/jira/browse/SOLR-11722
> Project: Solr
>  Issue Type: Sub-task
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: SolrCloud
>Reporter: David Smiley
> Attachments: SOLR-11722.patch, SOLR-11722.patch
>
>
> This issue is about creating a single API command to create a "Time Routed 
> Alias" along with its first collection.  Need to decide what endpoint URL it 
> is and parameters.
> Perhaps in v2 it'd be {{/api/collections?command=create-routed-alias}} or 
> alternatively piggy-back off of command=create-alias but we add more options, 
> perhaps with a prefix like "router"?
> Inputs:
> * alias name
> * misc collection creation metadata (e.g. config, numShards, ...) perhaps in 
> this context with a prefix like "collection."
> * metadata for TimeRoutedAliasUpdateProcessor, currently: router.field
> * date specifier for first collection; can include "date math".
> We'll certainly add more options as future features unfold.
> I believe the collection needs to be created first (referring to the alias 
> name via a core property), and then the alias pointing to it which demands 
> collections exist first.  When figuring the collection name, you'll need to 
> reference the format in TimeRoutedAliasUpdateProcessor.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-11722) API to create a Time Routed Alias and first collection

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-11722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16322880#comment-16322880
 ] 

ASF GitHub Bot commented on SOLR-11722:
---

GitHub user nsoft opened a pull request:

https://github.com/apache/lucene-solr/pull/304

SOLR-11722

patch #3 including docs, test SSL fix and precommit fixes

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/nsoft/lucene-solr SOLR-11722

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/lucene-solr/pull/304.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #304


commit aadae0c0e03bff019eac819919430b7c6eb7aabd
Author: Gus Heck 
Date:   2018-01-11T20:09:25Z

SOLR-11722 patch #3 including docs, test SSL fix and precommit fixes




> API to create a Time Routed Alias and first collection
> --
>
> Key: SOLR-11722
> URL: https://issues.apache.org/jira/browse/SOLR-11722
> Project: Solr
>  Issue Type: Sub-task
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: SolrCloud
>Reporter: David Smiley
> Attachments: SOLR-11722.patch, SOLR-11722.patch
>
>
> This issue is about creating a single API command to create a "Time Routed 
> Alias" along with its first collection.  Need to decide what endpoint URL it 
> is and parameters.
> Perhaps in v2 it'd be {{/api/collections?command=create-routed-alias}} or 
> alternatively piggy-back off of command=create-alias but we add more options, 
> perhaps with a prefix like "router"?
> Inputs:
> * alias name
> * misc collection creation metadata (e.g. config, numShards, ...) perhaps in 
> this context with a prefix like "collection."
> * metadata for TimeRoutedAliasUpdateProcessor, currently: router.field
> * date specifier for first collection; can include "date math".
> We'll certainly add more options as future features unfold.
> I believe the collection needs to be created first (referring to the alias 
> name via a core property), and then the alias pointing to it which demands 
> collections exist first.  When figuring the collection name, you'll need to 
> reference the format in TimeRoutedAliasUpdateProcessor.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-8126) Spatial prefix tree based on S2 geometry

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-8126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16321726#comment-16321726
 ] 

ASF GitHub Bot commented on LUCENE-8126:


Github user iverase commented on a diff in the pull request:

https://github.com/apache/lucene-solr/pull/302#discussion_r160868387
  
--- Diff: 
lucene/spatial-extras/src/java/org/apache/lucene/spatial/prefix/tree/S2PrefixTreeCell.java
 ---
@@ -0,0 +1,285 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.lucene.spatial.prefix.tree;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import com.google.common.geometry.S2CellId;
+import org.apache.lucene.util.BytesRef;
+import org.locationtech.spatial4j.shape.Shape;
+import org.locationtech.spatial4j.shape.SpatialRelation;
+
+/**
+ * This class represents a S2 pixel in the RPT.
+ *
+ * @lucene.internal
+ */
+class S2PrefixTreeCell implements Cell {
+
+//Faces of S2 Geometry
+private static S2CellId[] FACES = new S2CellId[6];
+static {
+FACES[0] = S2CellId.fromFacePosLevel(0, 0, 0);
+FACES[1] = S2CellId.fromFacePosLevel(1, 0, 0);
+FACES[2] = S2CellId.fromFacePosLevel(2, 0, 0);
+FACES[3] = S2CellId.fromFacePosLevel(3, 0, 0);
+FACES[4] = S2CellId.fromFacePosLevel(4, 0, 0);
+FACES[5] = S2CellId.fromFacePosLevel(5, 0, 0);
+}
+
+/*Special character to define a cell leaf*/
+private static final byte LEAF = '+';
+
+/*Tokens are used to serialize cells*/
+private static final byte[] TOKENS;
+/*Map containing mapping between tokens and integer values*/
+private static final Map PIXELS;
+static {
+TOKENS = new byte[]{'0', '1', '2', '3', '4', '5'};
+PIXELS = new HashMap<>(6);
+PIXELS.put(TOKENS[0], 0);
+PIXELS.put(TOKENS[1], 1);
+PIXELS.put(TOKENS[2], 2);
+PIXELS.put(TOKENS[3], 3);
+PIXELS.put(TOKENS[4], 4);
+PIXELS.put(TOKENS[5], 5);
+}
+
+S2CellId cellId;
+int level; //cache level
+S2PrefixTree tree;
+
+SpatialRelation shapeRel= null;
+boolean isLeaf;
+Shape shape = null;
+
+S2PrefixTreeCell(S2PrefixTree tree, S2CellId cellId){
+this.cellId= cellId;
+this.tree = tree;
+setLevel();
+if (getLevel() == tree.getMaxLevels()) {
+setLeaf();
+}
+}
+
+void readCell(S2PrefixTree tree, BytesRef ref){
+isLeaf = false;
+shape = null;
+shapeRel = null;
+this.tree = tree;
+cellId = getS2CellIdFromBytesRef(ref);
+setLevel();
+if (isLeaf(ref) || getLevel() == tree.getMaxLevels()){
+setLeaf();
+}
+}
+
+@Override
+public SpatialRelation getShapeRel() {
+return shapeRel;
+}
+
+@Override
+public void setShapeRel(SpatialRelation rel) {
+shapeRel = rel;
+}
+
+@Override
+public boolean isLeaf() {
+return isLeaf;
+}
+
+@Override
+public void setLeaf() {
+isLeaf = true;
+}
+
+@Override
+public BytesRef getTokenBytesWithLeaf(BytesRef result) {
+result = getTokenBytesNoLeaf(result);
+//max levels do not have leaf
+if (isLeaf() && !(getLevel() == tree.getMaxLevels())){
+//Add leaf byte
+result.bytes[result.offset + result.length] = LEAF;
+result.length++;
+}
+return result;
+}
+
+@Override
+public BytesRef getTokenBytesNoLeaf(BytesRef result) {
+if (result == null){
+result = 

[jira] [Commented] (LUCENE-8126) Spatial prefix tree based on S2 geometry

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-8126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16320916#comment-16320916
 ] 

ASF GitHub Bot commented on LUCENE-8126:


Github user dsmiley commented on a diff in the pull request:

https://github.com/apache/lucene-solr/pull/302#discussion_r160769685
  
--- Diff: 
lucene/spatial-extras/src/java/org/apache/lucene/spatial/prefix/tree/S2PrefixTreeCell.java
 ---
@@ -0,0 +1,285 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.lucene.spatial.prefix.tree;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import com.google.common.geometry.S2CellId;
+import org.apache.lucene.util.BytesRef;
+import org.locationtech.spatial4j.shape.Shape;
+import org.locationtech.spatial4j.shape.SpatialRelation;
+
+/**
+ * This class represents a S2 pixel in the RPT.
+ *
+ * @lucene.internal
+ */
+class S2PrefixTreeCell implements Cell {
+
+//Faces of S2 Geometry
+private static S2CellId[] FACES = new S2CellId[6];
+static {
+FACES[0] = S2CellId.fromFacePosLevel(0, 0, 0);
+FACES[1] = S2CellId.fromFacePosLevel(1, 0, 0);
+FACES[2] = S2CellId.fromFacePosLevel(2, 0, 0);
+FACES[3] = S2CellId.fromFacePosLevel(3, 0, 0);
+FACES[4] = S2CellId.fromFacePosLevel(4, 0, 0);
+FACES[5] = S2CellId.fromFacePosLevel(5, 0, 0);
+}
+
+/*Special character to define a cell leaf*/
+private static final byte LEAF = '+';
+
+/*Tokens are used to serialize cells*/
+private static final byte[] TOKENS;
+/*Map containing mapping between tokens and integer values*/
+private static final Map PIXELS;
+static {
+TOKENS = new byte[]{'0', '1', '2', '3', '4', '5'};
+PIXELS = new HashMap<>(6);
+PIXELS.put(TOKENS[0], 0);
+PIXELS.put(TOKENS[1], 1);
+PIXELS.put(TOKENS[2], 2);
+PIXELS.put(TOKENS[3], 3);
+PIXELS.put(TOKENS[4], 4);
+PIXELS.put(TOKENS[5], 5);
+}
+
+S2CellId cellId;
+int level; //cache level
+S2PrefixTree tree;
+
+SpatialRelation shapeRel= null;
+boolean isLeaf;
+Shape shape = null;
+
+S2PrefixTreeCell(S2PrefixTree tree, S2CellId cellId){
+this.cellId= cellId;
+this.tree = tree;
+setLevel();
+if (getLevel() == tree.getMaxLevels()) {
+setLeaf();
+}
+}
+
+void readCell(S2PrefixTree tree, BytesRef ref){
+isLeaf = false;
+shape = null;
+shapeRel = null;
+this.tree = tree;
+cellId = getS2CellIdFromBytesRef(ref);
+setLevel();
+if (isLeaf(ref) || getLevel() == tree.getMaxLevels()){
+setLeaf();
+}
+}
+
+@Override
+public SpatialRelation getShapeRel() {
+return shapeRel;
+}
+
+@Override
+public void setShapeRel(SpatialRelation rel) {
+shapeRel = rel;
+}
+
+@Override
+public boolean isLeaf() {
+return isLeaf;
+}
+
+@Override
+public void setLeaf() {
+isLeaf = true;
+}
+
+@Override
+public BytesRef getTokenBytesWithLeaf(BytesRef result) {
+result = getTokenBytesNoLeaf(result);
+//max levels do not have leaf
+if (isLeaf() && !(getLevel() == tree.getMaxLevels())){
+//Add leaf byte
+result.bytes[result.offset + result.length] = LEAF;
+result.length++;
+}
+return result;
+}
+
+@Override
+public BytesRef getTokenBytesNoLeaf(BytesRef result) {
+if (result == null){
+result = 

[jira] [Commented] (LUCENE-8126) Spatial prefix tree based on S2 geometry

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-8126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16320915#comment-16320915
 ] 

ASF GitHub Bot commented on LUCENE-8126:


Github user dsmiley commented on a diff in the pull request:

https://github.com/apache/lucene-solr/pull/302#discussion_r160769274
  
--- Diff: 
lucene/spatial-extras/src/java/org/apache/lucene/spatial/prefix/tree/S2PrefixTreeCell.java
 ---
@@ -0,0 +1,285 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.lucene.spatial.prefix.tree;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import com.google.common.geometry.S2CellId;
+import org.apache.lucene.util.BytesRef;
+import org.locationtech.spatial4j.shape.Shape;
+import org.locationtech.spatial4j.shape.SpatialRelation;
+
+/**
+ * This class represents a S2 pixel in the RPT.
+ *
+ * @lucene.internal
+ */
+class S2PrefixTreeCell implements Cell {
+
+//Faces of S2 Geometry
+private static S2CellId[] FACES = new S2CellId[6];
+static {
+FACES[0] = S2CellId.fromFacePosLevel(0, 0, 0);
+FACES[1] = S2CellId.fromFacePosLevel(1, 0, 0);
+FACES[2] = S2CellId.fromFacePosLevel(2, 0, 0);
+FACES[3] = S2CellId.fromFacePosLevel(3, 0, 0);
+FACES[4] = S2CellId.fromFacePosLevel(4, 0, 0);
+FACES[5] = S2CellId.fromFacePosLevel(5, 0, 0);
+}
+
+/*Special character to define a cell leaf*/
+private static final byte LEAF = '+';
+
+/*Tokens are used to serialize cells*/
+private static final byte[] TOKENS;
+/*Map containing mapping between tokens and integer values*/
+private static final Map PIXELS;
+static {
+TOKENS = new byte[]{'0', '1', '2', '3', '4', '5'};
+PIXELS = new HashMap<>(6);
+PIXELS.put(TOKENS[0], 0);
+PIXELS.put(TOKENS[1], 1);
+PIXELS.put(TOKENS[2], 2);
+PIXELS.put(TOKENS[3], 3);
+PIXELS.put(TOKENS[4], 4);
+PIXELS.put(TOKENS[5], 5);
+}
+
+S2CellId cellId;
+int level; //cache level
+S2PrefixTree tree;
+
+SpatialRelation shapeRel= null;
+boolean isLeaf;
+Shape shape = null;
+
+S2PrefixTreeCell(S2PrefixTree tree, S2CellId cellId){
+this.cellId= cellId;
+this.tree = tree;
+setLevel();
+if (getLevel() == tree.getMaxLevels()) {
+setLeaf();
+}
+}
+
+void readCell(S2PrefixTree tree, BytesRef ref){
+isLeaf = false;
+shape = null;
+shapeRel = null;
+this.tree = tree;
+cellId = getS2CellIdFromBytesRef(ref);
+setLevel();
+if (isLeaf(ref) || getLevel() == tree.getMaxLevels()){
+setLeaf();
+}
+}
+
+@Override
+public SpatialRelation getShapeRel() {
+return shapeRel;
+}
+
+@Override
+public void setShapeRel(SpatialRelation rel) {
+shapeRel = rel;
+}
+
+@Override
+public boolean isLeaf() {
+return isLeaf;
+}
+
+@Override
+public void setLeaf() {
+isLeaf = true;
+}
+
+@Override
+public BytesRef getTokenBytesWithLeaf(BytesRef result) {
+result = getTokenBytesNoLeaf(result);
+//max levels do not have leaf
+if (isLeaf() && !(getLevel() == tree.getMaxLevels())){
+//Add leaf byte
+result.bytes[result.offset + result.length] = LEAF;
+result.length++;
+}
+return result;
+}
+
+@Override
+public BytesRef getTokenBytesNoLeaf(BytesRef result) {
+if (result == null){
+result = 

[jira] [Commented] (LUCENE-8126) Spatial prefix tree based on S2 geometry

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-8126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16320909#comment-16320909
 ] 

ASF GitHub Bot commented on LUCENE-8126:


Github user dsmiley commented on a diff in the pull request:

https://github.com/apache/lucene-solr/pull/302#discussion_r160766117
  
--- Diff: 
lucene/spatial-extras/src/java/org/apache/lucene/spatial/prefix/tree/S2PrefixTreeCell.java
 ---
@@ -0,0 +1,285 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.lucene.spatial.prefix.tree;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import com.google.common.geometry.S2CellId;
+import org.apache.lucene.util.BytesRef;
+import org.locationtech.spatial4j.shape.Shape;
+import org.locationtech.spatial4j.shape.SpatialRelation;
+
+/**
+ * This class represents a S2 pixel in the RPT.
+ *
+ * @lucene.internal
+ */
+class S2PrefixTreeCell implements Cell {
+
+//Faces of S2 Geometry
+private static S2CellId[] FACES = new S2CellId[6];
+static {
+FACES[0] = S2CellId.fromFacePosLevel(0, 0, 0);
+FACES[1] = S2CellId.fromFacePosLevel(1, 0, 0);
+FACES[2] = S2CellId.fromFacePosLevel(2, 0, 0);
+FACES[3] = S2CellId.fromFacePosLevel(3, 0, 0);
+FACES[4] = S2CellId.fromFacePosLevel(4, 0, 0);
+FACES[5] = S2CellId.fromFacePosLevel(5, 0, 0);
+}
+
+/*Special character to define a cell leaf*/
+private static final byte LEAF = '+';
+
+/*Tokens are used to serialize cells*/
+private static final byte[] TOKENS;
+/*Map containing mapping between tokens and integer values*/
+private static final Map PIXELS;
+static {
+TOKENS = new byte[]{'0', '1', '2', '3', '4', '5'};
+PIXELS = new HashMap<>(6);
+PIXELS.put(TOKENS[0], 0);
+PIXELS.put(TOKENS[1], 1);
+PIXELS.put(TOKENS[2], 2);
+PIXELS.put(TOKENS[3], 3);
+PIXELS.put(TOKENS[4], 4);
+PIXELS.put(TOKENS[5], 5);
+}
+
+S2CellId cellId;
+int level; //cache level
+S2PrefixTree tree;
+
+SpatialRelation shapeRel= null;
+boolean isLeaf;
+Shape shape = null;
+
+S2PrefixTreeCell(S2PrefixTree tree, S2CellId cellId){
+this.cellId= cellId;
+this.tree = tree;
+setLevel();
+if (getLevel() == tree.getMaxLevels()) {
+setLeaf();
+}
+}
+
+void readCell(S2PrefixTree tree, BytesRef ref){
+isLeaf = false;
+shape = null;
+shapeRel = null;
+this.tree = tree;
+cellId = getS2CellIdFromBytesRef(ref);
+setLevel();
+if (isLeaf(ref) || getLevel() == tree.getMaxLevels()){
+setLeaf();
+}
+}
+
+@Override
+public SpatialRelation getShapeRel() {
+return shapeRel;
+}
+
+@Override
+public void setShapeRel(SpatialRelation rel) {
+shapeRel = rel;
+}
+
+@Override
+public boolean isLeaf() {
+return isLeaf;
+}
+
+@Override
+public void setLeaf() {
+isLeaf = true;
+}
+
+@Override
+public BytesRef getTokenBytesWithLeaf(BytesRef result) {
+result = getTokenBytesNoLeaf(result);
+//max levels do not have leaf
+if (isLeaf() && !(getLevel() == tree.getMaxLevels())){
+//Add leaf byte
+result.bytes[result.offset + result.length] = LEAF;
+result.length++;
+}
+return result;
+}
+
+@Override
+public BytesRef getTokenBytesNoLeaf(BytesRef result) {
+if (result == null){
+result = 

[jira] [Commented] (LUCENE-8126) Spatial prefix tree based on S2 geometry

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-8126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16320913#comment-16320913
 ] 

ASF GitHub Bot commented on LUCENE-8126:


Github user dsmiley commented on a diff in the pull request:

https://github.com/apache/lucene-solr/pull/302#discussion_r160768405
  
--- Diff: 
lucene/spatial-extras/src/java/org/apache/lucene/spatial/prefix/tree/S2PrefixTreeCell.java
 ---
@@ -0,0 +1,285 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.lucene.spatial.prefix.tree;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import com.google.common.geometry.S2CellId;
+import org.apache.lucene.util.BytesRef;
+import org.locationtech.spatial4j.shape.Shape;
+import org.locationtech.spatial4j.shape.SpatialRelation;
+
+/**
+ * This class represents a S2 pixel in the RPT.
+ *
+ * @lucene.internal
+ */
+class S2PrefixTreeCell implements Cell {
+
+//Faces of S2 Geometry
+private static S2CellId[] FACES = new S2CellId[6];
+static {
+FACES[0] = S2CellId.fromFacePosLevel(0, 0, 0);
+FACES[1] = S2CellId.fromFacePosLevel(1, 0, 0);
+FACES[2] = S2CellId.fromFacePosLevel(2, 0, 0);
+FACES[3] = S2CellId.fromFacePosLevel(3, 0, 0);
+FACES[4] = S2CellId.fromFacePosLevel(4, 0, 0);
+FACES[5] = S2CellId.fromFacePosLevel(5, 0, 0);
+}
+
+/*Special character to define a cell leaf*/
+private static final byte LEAF = '+';
+
+/*Tokens are used to serialize cells*/
+private static final byte[] TOKENS;
+/*Map containing mapping between tokens and integer values*/
+private static final Map PIXELS;
+static {
+TOKENS = new byte[]{'0', '1', '2', '3', '4', '5'};
+PIXELS = new HashMap<>(6);
+PIXELS.put(TOKENS[0], 0);
+PIXELS.put(TOKENS[1], 1);
+PIXELS.put(TOKENS[2], 2);
+PIXELS.put(TOKENS[3], 3);
+PIXELS.put(TOKENS[4], 4);
+PIXELS.put(TOKENS[5], 5);
+}
+
+S2CellId cellId;
+int level; //cache level
+S2PrefixTree tree;
+
+SpatialRelation shapeRel= null;
+boolean isLeaf;
+Shape shape = null;
+
+S2PrefixTreeCell(S2PrefixTree tree, S2CellId cellId){
+this.cellId= cellId;
+this.tree = tree;
+setLevel();
+if (getLevel() == tree.getMaxLevels()) {
+setLeaf();
+}
+}
+
+void readCell(S2PrefixTree tree, BytesRef ref){
+isLeaf = false;
+shape = null;
+shapeRel = null;
+this.tree = tree;
+cellId = getS2CellIdFromBytesRef(ref);
+setLevel();
+if (isLeaf(ref) || getLevel() == tree.getMaxLevels()){
+setLeaf();
+}
+}
+
+@Override
+public SpatialRelation getShapeRel() {
+return shapeRel;
+}
+
+@Override
+public void setShapeRel(SpatialRelation rel) {
+shapeRel = rel;
+}
+
+@Override
+public boolean isLeaf() {
+return isLeaf;
+}
+
+@Override
+public void setLeaf() {
+isLeaf = true;
+}
+
+@Override
+public BytesRef getTokenBytesWithLeaf(BytesRef result) {
+result = getTokenBytesNoLeaf(result);
+//max levels do not have leaf
+if (isLeaf() && !(getLevel() == tree.getMaxLevels())){
+//Add leaf byte
+result.bytes[result.offset + result.length] = LEAF;
+result.length++;
+}
+return result;
+}
+
+@Override
+public BytesRef getTokenBytesNoLeaf(BytesRef result) {
+if (result == null){
+result = 

[jira] [Commented] (LUCENE-8126) Spatial prefix tree based on S2 geometry

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-8126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16320914#comment-16320914
 ] 

ASF GitHub Bot commented on LUCENE-8126:


Github user dsmiley commented on a diff in the pull request:

https://github.com/apache/lucene-solr/pull/302#discussion_r160773175
  
--- Diff: 
lucene/spatial-extras/src/java/org/apache/lucene/spatial/prefix/tree/S2PrefixTree.java
 ---
@@ -0,0 +1,111 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.lucene.spatial.prefix.tree;
+
+import java.util.ArrayList;
+import java.util.List;
+
+import com.google.common.geometry.S2CellId;
+import com.google.common.geometry.S2LatLng;
+import com.google.common.geometry.S2Projections;
+import org.apache.lucene.util.BytesRef;
+import org.locationtech.spatial4j.context.SpatialContext;
+import org.locationtech.spatial4j.distance.DistanceUtils;
+import org.locationtech.spatial4j.shape.Point;
+import org.locationtech.spatial4j.shape.Shape;
+
+/**
+ * Spatial prefix tree for S2 Geometry. Shape factories for the given 
{@link SpatialContext} must
+ * implement the interface {@link S2ShapeFactory}.
+ *
+ * @lucene.experimental
+ */
+public class S2PrefixTree extends SpatialPrefixTree {
+
+/**
+ * Factory for creating {@link S2PrefixTree} instances with useful 
defaults
+ */
+public static class Factory extends SpatialPrefixTreeFactory {
+
+@Override
+protected int getLevelForDistance(double degrees) {
+S2PrefixTree grid = new S2PrefixTree(ctx, 
S2PrefixTree.MAX_LEVELS);
+return grid.getLevelForDistance(degrees);
+}
+
+@Override
+protected SpatialPrefixTree newSPT() {
+return new S2PrefixTree(ctx,
+maxLevels != null ? maxLevels : S2PrefixTree.MAX_LEVELS);
+}
+}
+
+//factory to generate S2 cell shapes
+protected final S2ShapeFactory s2ShapeFactory;
+public static final int MAX_LEVELS = S2CellId.MAX_LEVEL + 1;
+
+public S2PrefixTree(SpatialContext ctx, int maxLevels) {
+super(ctx, maxLevels);
+if (!(ctx.getShapeFactory() instanceof S2ShapeFactory)) {
+throw new IllegalArgumentException("Spatial context does not 
support S2 spatial index.");
+}
+this.s2ShapeFactory = (S2ShapeFactory) ctx.getShapeFactory();
+}
+
+@Override
+public int getLevelForDistance(double dist) {
+if (dist ==0){
+return maxLevels;
+}
+return Math.min(maxLevels, 
S2Projections.MAX_WIDTH.getClosestLevel(dist * 
DistanceUtils.DEGREES_TO_RADIANS) +1);
+}
+
+@Override
+public double getDistanceForLevel(int level) {
+return S2Projections.MAX_WIDTH.getValue(level -1) * 
DistanceUtils.RADIANS_TO_DEGREES;
--- End diff --

nitpick: put space after that minus operator


> Spatial prefix tree based on S2 geometry
> 
>
> Key: LUCENE-8126
> URL: https://issues.apache.org/jira/browse/LUCENE-8126
> Project: Lucene - Core
>  Issue Type: New Feature
>  Components: modules/spatial-extras
>Reporter: Ignacio Vera
>
> Hi [~dsmiley],
> I have been working on a prefix tree based on goggle S2 geometry 
> (https://s2geometry.io/) to be used mainly with Geo3d shapes with very 
> promising results, in particular for complex shapes (e.g polygons). Using 
> this pixelization scheme reduces the size of the index, improves the 
> performance of the queries and reduces the loading time for non-point shapes. 
> If you are ok with this contribution and before providing any code I would 
> like to understand what is the correct/prefered approach:
> 1) Add new depency to the S2 library 
> 

[jira] [Commented] (LUCENE-8126) Spatial prefix tree based on S2 geometry

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-8126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16320917#comment-16320917
 ] 

ASF GitHub Bot commented on LUCENE-8126:


Github user dsmiley commented on a diff in the pull request:

https://github.com/apache/lucene-solr/pull/302#discussion_r160773587
  
--- Diff: 
lucene/spatial-extras/src/java/org/apache/lucene/spatial/prefix/tree/S2PrefixTree.java
 ---
@@ -0,0 +1,111 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.lucene.spatial.prefix.tree;
+
+import java.util.ArrayList;
+import java.util.List;
+
+import com.google.common.geometry.S2CellId;
+import com.google.common.geometry.S2LatLng;
+import com.google.common.geometry.S2Projections;
+import org.apache.lucene.util.BytesRef;
+import org.locationtech.spatial4j.context.SpatialContext;
+import org.locationtech.spatial4j.distance.DistanceUtils;
+import org.locationtech.spatial4j.shape.Point;
+import org.locationtech.spatial4j.shape.Shape;
+
+/**
+ * Spatial prefix tree for S2 Geometry. Shape factories for the given 
{@link SpatialContext} must
+ * implement the interface {@link S2ShapeFactory}.
+ *
+ * @lucene.experimental
+ */
+public class S2PrefixTree extends SpatialPrefixTree {
+
+/**
+ * Factory for creating {@link S2PrefixTree} instances with useful 
defaults
+ */
+public static class Factory extends SpatialPrefixTreeFactory {
+
+@Override
+protected int getLevelForDistance(double degrees) {
+S2PrefixTree grid = new S2PrefixTree(ctx, 
S2PrefixTree.MAX_LEVELS);
+return grid.getLevelForDistance(degrees);
+}
+
+@Override
+protected SpatialPrefixTree newSPT() {
+return new S2PrefixTree(ctx,
+maxLevels != null ? maxLevels : S2PrefixTree.MAX_LEVELS);
+}
+}
+
+//factory to generate S2 cell shapes
+protected final S2ShapeFactory s2ShapeFactory;
+public static final int MAX_LEVELS = S2CellId.MAX_LEVEL + 1;
+
+public S2PrefixTree(SpatialContext ctx, int maxLevels) {
+super(ctx, maxLevels);
+if (!(ctx.getShapeFactory() instanceof S2ShapeFactory)) {
+throw new IllegalArgumentException("Spatial context does not 
support S2 spatial index.");
+}
+this.s2ShapeFactory = (S2ShapeFactory) ctx.getShapeFactory();
+}
+
+@Override
+public int getLevelForDistance(double dist) {
+if (dist ==0){
+return maxLevels;
+}
+return Math.min(maxLevels, 
S2Projections.MAX_WIDTH.getClosestLevel(dist * 
DistanceUtils.DEGREES_TO_RADIANS) +1);
+}
+
+@Override
+public double getDistanceForLevel(int level) {
+return S2Projections.MAX_WIDTH.getValue(level -1) * 
DistanceUtils.RADIANS_TO_DEGREES;
+}
+
+@Override
+public Cell getWorldCell() {
+return  new S2PrefixTreeCell(this, null);
+}
+
+@Override
+public Cell readCell(BytesRef term, Cell scratch) {
+S2PrefixTreeCell cell = (S2PrefixTreeCell) scratch;
+if (cell == null)
+cell = (S2PrefixTreeCell) getWorldCell();
+cell.readCell(this, term);
+return cell;
+}
+
+@Override
+public CellIterator getTreeCellIterator(Shape shape, int detailLevel) {
+if (!(shape instanceof Point)) {
+return  super.getTreeCellIterator(shape, detailLevel);
+}
+Point p = (Point) shape;
+S2CellId id = S2CellId.fromLatLng(S2LatLng.fromDegrees(p.getY(), 
p.getX())).parent(detailLevel-1);
+List cells = new ArrayList<>(detailLevel);
+for (int i=0; i < detailLevel -1; i++) {
--- End diff --

nitpick: put a space after that minus operator


> Spatial 

[jira] [Commented] (LUCENE-8126) Spatial prefix tree based on S2 geometry

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-8126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16320912#comment-16320912
 ] 

ASF GitHub Bot commented on LUCENE-8126:


Github user dsmiley commented on a diff in the pull request:

https://github.com/apache/lucene-solr/pull/302#discussion_r160768053
  
--- Diff: 
lucene/spatial-extras/src/java/org/apache/lucene/spatial/prefix/tree/S2PrefixTreeCell.java
 ---
@@ -0,0 +1,285 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.lucene.spatial.prefix.tree;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import com.google.common.geometry.S2CellId;
+import org.apache.lucene.util.BytesRef;
+import org.locationtech.spatial4j.shape.Shape;
+import org.locationtech.spatial4j.shape.SpatialRelation;
+
+/**
+ * This class represents a S2 pixel in the RPT.
+ *
+ * @lucene.internal
+ */
+class S2PrefixTreeCell implements Cell {
+
+//Faces of S2 Geometry
+private static S2CellId[] FACES = new S2CellId[6];
+static {
+FACES[0] = S2CellId.fromFacePosLevel(0, 0, 0);
+FACES[1] = S2CellId.fromFacePosLevel(1, 0, 0);
+FACES[2] = S2CellId.fromFacePosLevel(2, 0, 0);
+FACES[3] = S2CellId.fromFacePosLevel(3, 0, 0);
+FACES[4] = S2CellId.fromFacePosLevel(4, 0, 0);
+FACES[5] = S2CellId.fromFacePosLevel(5, 0, 0);
+}
+
+/*Special character to define a cell leaf*/
+private static final byte LEAF = '+';
+
+/*Tokens are used to serialize cells*/
+private static final byte[] TOKENS;
+/*Map containing mapping between tokens and integer values*/
+private static final Map PIXELS;
--- End diff --

Since this map has a small set of fixed values that have numeric 
equivalents, perhaps we can do direct addressing into an array?


> Spatial prefix tree based on S2 geometry
> 
>
> Key: LUCENE-8126
> URL: https://issues.apache.org/jira/browse/LUCENE-8126
> Project: Lucene - Core
>  Issue Type: New Feature
>  Components: modules/spatial-extras
>Reporter: Ignacio Vera
>
> Hi [~dsmiley],
> I have been working on a prefix tree based on goggle S2 geometry 
> (https://s2geometry.io/) to be used mainly with Geo3d shapes with very 
> promising results, in particular for complex shapes (e.g polygons). Using 
> this pixelization scheme reduces the size of the index, improves the 
> performance of the queries and reduces the loading time for non-point shapes. 
> If you are ok with this contribution and before providing any code I would 
> like to understand what is the correct/prefered approach:
> 1) Add new depency to the S2 library 
> (https://mvnrepository.com/artifact/io.sgr/s2-geometry-library-java). It has 
> Apache 2.0 license so it should be ok.
> 2) Create a utility class with all methods necessary to navigate the S2 tree 
> and create shapes from S2 cells (basically port what we need from the library 
> into Lucene).
> What do you think?



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-8126) Spatial prefix tree based on S2 geometry

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-8126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16320910#comment-16320910
 ] 

ASF GitHub Bot commented on LUCENE-8126:


Github user dsmiley commented on a diff in the pull request:

https://github.com/apache/lucene-solr/pull/302#discussion_r160768230
  
--- Diff: 
lucene/spatial-extras/src/java/org/apache/lucene/spatial/prefix/tree/S2PrefixTree.java
 ---
@@ -0,0 +1,111 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.lucene.spatial.prefix.tree;
+
+import java.util.ArrayList;
+import java.util.List;
+
+import com.google.common.geometry.S2CellId;
+import com.google.common.geometry.S2LatLng;
+import com.google.common.geometry.S2Projections;
+import org.apache.lucene.util.BytesRef;
+import org.locationtech.spatial4j.context.SpatialContext;
+import org.locationtech.spatial4j.distance.DistanceUtils;
+import org.locationtech.spatial4j.shape.Point;
+import org.locationtech.spatial4j.shape.Shape;
+
+/**
+ * Spatial prefix tree for S2 Geometry. Shape factories for the given 
{@link SpatialContext} must
+ * implement the interface {@link S2ShapeFactory}.
+ *
+ * @lucene.experimental
+ */
+public class S2PrefixTree extends SpatialPrefixTree {
+
+/**
+ * Factory for creating {@link S2PrefixTree} instances with useful 
defaults
+ */
+public static class Factory extends SpatialPrefixTreeFactory {
+
+@Override
+protected int getLevelForDistance(double degrees) {
+S2PrefixTree grid = new S2PrefixTree(ctx, 
S2PrefixTree.MAX_LEVELS);
+return grid.getLevelForDistance(degrees);
+}
+
+@Override
+protected SpatialPrefixTree newSPT() {
+return new S2PrefixTree(ctx,
+maxLevels != null ? maxLevels : S2PrefixTree.MAX_LEVELS);
+}
+}
+
+//factory to generate S2 cell shapes
+protected final S2ShapeFactory s2ShapeFactory;
+public static final int MAX_LEVELS = S2CellId.MAX_LEVEL + 1;
+
+public S2PrefixTree(SpatialContext ctx, int maxLevels) {
+super(ctx, maxLevels);
+if (!(ctx.getShapeFactory() instanceof S2ShapeFactory)) {
+throw new IllegalArgumentException("Spatial context does not 
support S2 spatial index.");
+}
+this.s2ShapeFactory = (S2ShapeFactory) ctx.getShapeFactory();
+}
+
+@Override
+public int getLevelForDistance(double dist) {
+if (dist ==0){
+return maxLevels;
+}
+return Math.min(maxLevels, 
S2Projections.MAX_WIDTH.getClosestLevel(dist * 
DistanceUtils.DEGREES_TO_RADIANS) +1);
+}
+
+@Override
+public double getDistanceForLevel(int level) {
+return S2Projections.MAX_WIDTH.getValue(level -1) * 
DistanceUtils.RADIANS_TO_DEGREES;
+}
+
+@Override
+public Cell getWorldCell() {
+return  new S2PrefixTreeCell(this, null);
+}
+
+@Override
+public Cell readCell(BytesRef term, Cell scratch) {
+S2PrefixTreeCell cell = (S2PrefixTreeCell) scratch;
+if (cell == null)
+cell = (S2PrefixTreeCell) getWorldCell();
--- End diff --

nitpick: our code style in Lucene/Solr is to always use braces


> Spatial prefix tree based on S2 geometry
> 
>
> Key: LUCENE-8126
> URL: https://issues.apache.org/jira/browse/LUCENE-8126
> Project: Lucene - Core
>  Issue Type: New Feature
>  Components: modules/spatial-extras
>Reporter: Ignacio Vera
>
> Hi [~dsmiley],
> I have been working on a prefix tree based on goggle S2 geometry 
> (https://s2geometry.io/) to be used mainly with Geo3d shapes with very 
> promising results, in particular for complex shapes (e.g 

[jira] [Commented] (LUCENE-8126) Spatial prefix tree based on S2 geometry

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-8126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16320911#comment-16320911
 ] 

ASF GitHub Bot commented on LUCENE-8126:


Github user dsmiley commented on a diff in the pull request:

https://github.com/apache/lucene-solr/pull/302#discussion_r160766597
  
--- Diff: 
lucene/spatial-extras/src/java/org/apache/lucene/spatial/prefix/tree/S2PrefixTreeCell.java
 ---
@@ -0,0 +1,285 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.lucene.spatial.prefix.tree;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import com.google.common.geometry.S2CellId;
+import org.apache.lucene.util.BytesRef;
+import org.locationtech.spatial4j.shape.Shape;
+import org.locationtech.spatial4j.shape.SpatialRelation;
+
+/**
+ * This class represents a S2 pixel in the RPT.
+ *
+ * @lucene.internal
+ */
+class S2PrefixTreeCell implements Cell {
+
+//Faces of S2 Geometry
+private static S2CellId[] FACES = new S2CellId[6];
+static {
+FACES[0] = S2CellId.fromFacePosLevel(0, 0, 0);
+FACES[1] = S2CellId.fromFacePosLevel(1, 0, 0);
+FACES[2] = S2CellId.fromFacePosLevel(2, 0, 0);
+FACES[3] = S2CellId.fromFacePosLevel(3, 0, 0);
+FACES[4] = S2CellId.fromFacePosLevel(4, 0, 0);
+FACES[5] = S2CellId.fromFacePosLevel(5, 0, 0);
+}
+
+/*Special character to define a cell leaf*/
+private static final byte LEAF = '+';
+
+/*Tokens are used to serialize cells*/
+private static final byte[] TOKENS;
+/*Map containing mapping between tokens and integer values*/
+private static final Map PIXELS;
+static {
+TOKENS = new byte[]{'0', '1', '2', '3', '4', '5'};
+PIXELS = new HashMap<>(6);
+PIXELS.put(TOKENS[0], 0);
+PIXELS.put(TOKENS[1], 1);
+PIXELS.put(TOKENS[2], 2);
+PIXELS.put(TOKENS[3], 3);
+PIXELS.put(TOKENS[4], 4);
+PIXELS.put(TOKENS[5], 5);
+}
+
+S2CellId cellId;
+int level; //cache level
+S2PrefixTree tree;
+
+SpatialRelation shapeRel= null;
+boolean isLeaf;
+Shape shape = null;
+
+S2PrefixTreeCell(S2PrefixTree tree, S2CellId cellId){
+this.cellId= cellId;
+this.tree = tree;
+setLevel();
+if (getLevel() == tree.getMaxLevels()) {
+setLeaf();
+}
+}
+
+void readCell(S2PrefixTree tree, BytesRef ref){
+isLeaf = false;
+shape = null;
+shapeRel = null;
+this.tree = tree;
+cellId = getS2CellIdFromBytesRef(ref);
+setLevel();
+if (isLeaf(ref) || getLevel() == tree.getMaxLevels()){
+setLeaf();
+}
+}
+
+@Override
+public SpatialRelation getShapeRel() {
+return shapeRel;
+}
+
+@Override
+public void setShapeRel(SpatialRelation rel) {
+shapeRel = rel;
+}
+
+@Override
+public boolean isLeaf() {
+return isLeaf;
+}
+
+@Override
+public void setLeaf() {
+isLeaf = true;
+}
+
+@Override
+public BytesRef getTokenBytesWithLeaf(BytesRef result) {
+result = getTokenBytesNoLeaf(result);
+//max levels do not have leaf
+if (isLeaf() && !(getLevel() == tree.getMaxLevels())){
+//Add leaf byte
+result.bytes[result.offset + result.length] = LEAF;
+result.length++;
+}
+return result;
+}
+
+@Override
+public BytesRef getTokenBytesNoLeaf(BytesRef result) {
+if (result == null){
+result = 

[jira] [Commented] (LUCENE-8126) Spatial prefix tree based on S2 geometry

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-8126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16320448#comment-16320448
 ] 

ASF GitHub Bot commented on LUCENE-8126:


GitHub user iverase opened a pull request:

https://github.com/apache/lucene-solr/pull/302

LUCENE-8126:  Spatial prefix tree based on S2 geometry



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/iverase/lucene-solr master

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/lucene-solr/pull/302.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #302


commit ed57d35c3896c61b8e7af31bce66650800225a34
Author: ivera 
Date:   2018-01-10T15:29:40Z

LUCENE-8126: first commit of S2 RPT

commit 9d00f003a0c980ca9eccaff8d4b2e9eebf1e77e2
Author: ivera 
Date:   2018-01-10T15:31:39Z

LUCENE-8126: first commit of S2 RPT

commit 6aa0199f07f7c8e23be8ff550fc3fd0aefc5554a
Author: ivera 
Date:   2018-01-10T15:33:36Z

LUCENE-8126: Performance test classes. They are included to show the 
increae of performance using the new RPT.




> Spatial prefix tree based on S2 geometry
> 
>
> Key: LUCENE-8126
> URL: https://issues.apache.org/jira/browse/LUCENE-8126
> Project: Lucene - Core
>  Issue Type: New Feature
>  Components: modules/spatial-extras
>Reporter: Ignacio Vera
>
> Hi [~dsmiley],
> I have been working on a prefix tree based on goggle S2 geometry 
> (https://s2geometry.io/) to be used mainly with Geo3d shapes with very 
> promising results, in particular for complex shapes (e.g polygons). Using 
> this pixelization scheme reduces the size of the index, improves the 
> performance of the queries and reduces the loading time for non-point shapes. 
> If you are ok with this contribution and before providing any code I would 
> like to understand what is the correct/prefered approach:
> 1) Add new depency to the S2 library 
> (https://mvnrepository.com/artifact/io.sgr/s2-geometry-library-java). It has 
> Apache 2.0 license so it should be ok.
> 2) Create a utility class with all methods necessary to navigate the S2 tree 
> and create shapes from S2 cells (basically port what we need from the library 
> into Lucene).
> What do you think?



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-8086) G3d wrapper: Improve circles for non spherical planets

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-8086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16320139#comment-16320139
 ] 

ASF GitHub Bot commented on LUCENE-8086:


Github user iverase closed the pull request at:

https://github.com/apache/lucene-solr/pull/288


> G3d wrapper: Improve circles for non spherical planets
> --
>
> Key: LUCENE-8086
> URL: https://issues.apache.org/jira/browse/LUCENE-8086
> Project: Lucene - Core
>  Issue Type: Improvement
>  Components: modules/spatial-extras
>Reporter: Ignacio Vera
>Assignee: David Smiley
> Fix For: 7.3
>
>
> Hi [~dsmiley],
> The purpose of this ticket is to add a new circle shape (GeoExactCircle) for 
> non-spherical planets and therefore remove the method relate from 
> Geo3dCircleShape. The patch will include some simplifications on the wrapper 
> and some refactoring of the tests.
> I will open shortly a pull request.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-3089) Make ResponseBuilder.isDistrib public

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-3089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16319073#comment-16319073
 ] 

ASF GitHub Bot commented on SOLR-3089:
--

Github user abhidemon closed the pull request at:

https://github.com/apache/lucene-solr/pull/301


> Make ResponseBuilder.isDistrib public
> -
>
> Key: SOLR-3089
> URL: https://issues.apache.org/jira/browse/SOLR-3089
> Project: Solr
>  Issue Type: Improvement
>  Components: Response Writers
>Affects Versions: 4.0-ALPHA
>Reporter: Rok Rejc
> Fix For: 4.9, 6.0
>
> Attachments: Solr-3089.patch
>
>
> Hi,
> i have posted this issue on a mailing list but i didn't get any response.
> I am trying to write distributed search component (a class that extends 
> SearchComponent). I have checked FacetComponent and TermsComponent. If I want 
> that search component works in a distributed environment I have to set 
> ResponseBuilder's isDistrib to true, like this (this is also done in 
> TermsComponent for example):
>   public void prepare(ResponseBuilder rb) throws IOException {
>   SolrParams params = rb.req.getParams();
>   String shards = params.get(ShardParams.SHARDS);
>   if (shards != null) {
>   List lst = StrUtils.splitSmart(shards, ",", 
> true);
>   rb.shards = lst.toArray(new String[lst.size()]);
>   rb.isDistrib = true;
>   }
>   }
> If I have my component outside the package org.apache.solr.handler.component 
> this doesn't work. Is it possible to make isDistrib public (or is this the 
> wrong procedure/behaviour/design)?
> Many thanks,
> Rok



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-3089) Make ResponseBuilder.isDistrib public

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-3089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16319070#comment-16319070
 ] 

ASF GitHub Bot commented on SOLR-3089:
--

GitHub user abhidemon opened a pull request:

https://github.com/apache/lucene-solr/pull/301

test: Add test case for ResponseBuilder.isDistributed()

This is being done for https://issues.apache.org/jira/browse/SOLR-3089

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/abhidemon/lucene-solr SOLR-3089

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/lucene-solr/pull/301.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #301


commit 767bfbcb143a26bf325a00888c350d06a2486662
Author: abhidemon 
Date:   2018-01-09T20:14:34Z

test: Add test case for ResponseBuilder.isDistributed




> Make ResponseBuilder.isDistrib public
> -
>
> Key: SOLR-3089
> URL: https://issues.apache.org/jira/browse/SOLR-3089
> Project: Solr
>  Issue Type: Improvement
>  Components: Response Writers
>Affects Versions: 4.0-ALPHA
>Reporter: Rok Rejc
> Fix For: 4.9, 6.0
>
> Attachments: Solr-3089.patch
>
>
> Hi,
> i have posted this issue on a mailing list but i didn't get any response.
> I am trying to write distributed search component (a class that extends 
> SearchComponent). I have checked FacetComponent and TermsComponent. If I want 
> that search component works in a distributed environment I have to set 
> ResponseBuilder's isDistrib to true, like this (this is also done in 
> TermsComponent for example):
>   public void prepare(ResponseBuilder rb) throws IOException {
>   SolrParams params = rb.req.getParams();
>   String shards = params.get(ShardParams.SHARDS);
>   if (shards != null) {
>   List lst = StrUtils.splitSmart(shards, ",", 
> true);
>   rb.shards = lst.toArray(new String[lst.size()]);
>   rb.isDistrib = true;
>   }
>   }
> If I have my component outside the package org.apache.solr.handler.component 
> this doesn't work. Is it possible to make isDistrib public (or is this the 
> wrong procedure/behaviour/design)?
> Many thanks,
> Rok



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-8075) Possible null pointer dereference in core/src/java/org/apache/lucene/codecs/blocktree/IntersectTermsEnum.java

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-8075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16318432#comment-16318432
 ] 

ASF GitHub Bot commented on LUCENE-8075:


Github user asfgit closed the pull request at:

https://github.com/apache/lucene-solr/pull/286


> Possible null pointer dereference in 
> core/src/java/org/apache/lucene/codecs/blocktree/IntersectTermsEnum.java
> -
>
> Key: LUCENE-8075
> URL: https://issues.apache.org/jira/browse/LUCENE-8075
> Project: Lucene - Core
>  Issue Type: Bug
>  Components: core/codecs
>Affects Versions: 7.1
>Reporter: Xiaoshan Sun
>  Labels: easyfix
>   Original Estimate: 10m
>  Remaining Estimate: 10m
>
> Possible null pointer dereference in 
> core/src/java/org/apache/lucene/codecs/blocktree/IntersectTermsEnum.java.
> at line 119. The fr.index may be NULL. This result is based on static 
> analysis tools and the details are shown below:
> *
> {code:java}
> 106: if (fr.index == null) {
> 107:  fstReader = null;  // fr.index is Known NULL here.
> } else {
>   fstReader = fr.index.getBytesReader();
> }
> // TODO: if the automaton is "smallish" we really
> // should use the terms index to seek at least to
> // the initial term and likely to subsequent terms
> // (or, maybe just fallback to ATE for such cases).
> // Else the seek cost of loading the frames will be
> // too costly.
> 119:final FST.Arc arc = fr.index.getFirstArc(arcs[0]); 
> //  fr.index is dereferenced here and fr.index can be NULL if 107 is arrived.
> {code}
> *
> It is not sure if fr.index can be NULL in runtime.
> We think it is reasonable to fix it by a test if fr.index is NULL and an 
> error handling.
> --
> Please Refer to "Trusted Operating System and System Assurance Working Group, 
> TCA, Institute of Software, Chinese Academy of Sciences" in the 
> acknowledgement if applicable.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-11831) Skip second grouping step if group.limit is 1 (aka Las Vegas patch)

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-11831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16316636#comment-16316636
 ] 

ASF GitHub Bot commented on SOLR-11831:
---

GitHub user mjosephidou opened a pull request:

https://github.com/apache/lucene-solr/pull/300

SOLR-11831: Skip second grouping step if group.limit is 1 (aka Las Vegas 
Patch)

Summary:
In cases where we do grouping and ask for  {{group.limit=1}} only it is 
possible to skip the second grouping step. In our test datasets it improved 
speed by around 40%.

Essentially, in the first grouping step each shard returns the top K groups 
based on the highest scoring document in each group. The top K groups from each 
shard are merged in the federator and in the second step we ask all the shards 
to return the top documents from each of the top ranking groups.

If we only want to return the highest scoring document per group we can 
return the top document id in the first step, merge results in the federator to 
retain the top K groups and then skip the second grouping step entirely.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/bloomberg/lucene-solr SOLR-11831

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/lucene-solr/pull/300.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #300


commit 6b918c86cd0f37320c32eb669eca722a9e74f768
Author: Malvina Josephidou 
Date:   2018-01-04T15:00:35Z

SOLR-11831: Skip second grouping step if group.limit is 1 (aka Las Vegas 
patch)

Summary:
In cases where we do grouping and ask for  {{group.limit=1}} only it is 
possible to skip the second grouping step. In our test datasets it improved 
speed by around 40%.

Essentially, in the first grouping step each shard returns the top K groups 
based on the highest scoring document in each group. The top K groups from each 
shard are merged in the federator and in the second step we ask all the shards 
to return the top documents from each of the top ranking groups.

If we only want to return the highest scoring document per group we can 
return the top document id in the first step, merge results in the federator to 
retain the top K groups and then skip the second grouping step entirely.




>  Skip second grouping step if group.limit is 1 (aka Las Vegas patch)
> 
>
> Key: SOLR-11831
> URL: https://issues.apache.org/jira/browse/SOLR-11831
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Malvina Josephidou
>Priority: Minor
>
> In cases where we do grouping and ask for  {{group.limit=1}} only it is 
> possible to skip the second grouping step. In our test datasets it improved 
> speed by around 40%.
> Essentially, in the first grouping step each shard returns the top K groups 
> based on the highest scoring document in each group. The top K groups from 
> each shard are merged in the federator and in the second step we ask all the 
> shards to return the top documents from each of the top ranking groups.
> If we only want to return the highest scoring document per group we can 
> return the top document id in the first step, merge results in the federator 
> to retain the top K groups and then skip the second grouping step entirely. 
> This is possible provided that:
> a) We do not need to know the total number of matching documents per group
> b) Within group sort and between group sort is the same. 
> c) We are not doing reranking (this is because this is done in the second 
> grouping step. It is also possible to get this to work with reranking but 
> more work and some additional assumptions are required)
>  
> This patch applies the grouping optimisation in cases where a)-c) apply and 
> we are only sorting by relevance. It is also possible to extend this work to 
> handle multiple sorting criteria and also reranking. 
> P.S. Diego and I called this patch "las vegas" because we started to write it 
> on the flight to Las Vegas for Lucene/Solr revolution. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-8075) Possible null pointer dereference in core/src/java/org/apache/lucene/codecs/blocktree/IntersectTermsEnum.java

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-8075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16314891#comment-16314891
 ] 

ASF GitHub Bot commented on LUCENE-8075:


Github user imgpulak commented on the issue:

https://github.com/apache/lucene-solr/pull/286
  
@ jpountz @mikemccand I have updated as per your comment. Please review the 
same. 
Thanks !


> Possible null pointer dereference in 
> core/src/java/org/apache/lucene/codecs/blocktree/IntersectTermsEnum.java
> -
>
> Key: LUCENE-8075
> URL: https://issues.apache.org/jira/browse/LUCENE-8075
> Project: Lucene - Core
>  Issue Type: Bug
>  Components: core/codecs
>Affects Versions: 7.1
>Reporter: Xiaoshan Sun
>  Labels: easyfix
>   Original Estimate: 10m
>  Remaining Estimate: 10m
>
> Possible null pointer dereference in 
> core/src/java/org/apache/lucene/codecs/blocktree/IntersectTermsEnum.java.
> at line 119. The fr.index may be NULL. This result is based on static 
> analysis tools and the details are shown below:
> *
> {code:java}
> 106: if (fr.index == null) {
> 107:  fstReader = null;  // fr.index is Known NULL here.
> } else {
>   fstReader = fr.index.getBytesReader();
> }
> // TODO: if the automaton is "smallish" we really
> // should use the terms index to seek at least to
> // the initial term and likely to subsequent terms
> // (or, maybe just fallback to ATE for such cases).
> // Else the seek cost of loading the frames will be
> // too costly.
> 119:final FST.Arc arc = fr.index.getFirstArc(arcs[0]); 
> //  fr.index is dereferenced here and fr.index can be NULL if 107 is arrived.
> {code}
> *
> It is not sure if fr.index can be NULL in runtime.
> We think it is reasonable to fix it by a test if fr.index is NULL and an 
> error handling.
> --
> Please Refer to "Trusted Operating System and System Assurance Working Group, 
> TCA, Institute of Software, Chinese Academy of Sciences" in the 
> acknowledgement if applicable.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-11484) CloudSolrClient's cache of collection clusterstate can cause RouteExceptions when attempting directUpdates after collection modifications

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-11484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16307051#comment-16307051
 ] 

ASF GitHub Bot commented on SOLR-11484:
---

Github user SerCeMan commented on the issue:

https://github.com/apache/lucene-solr/pull/264
  
Closing as this might be fixed by 
https://issues.apache.org/jira/browse/SOLR-11484


> CloudSolrClient's cache of collection clusterstate can cause RouteExceptions 
> when attempting directUpdates after collection modifications
> -
>
> Key: SOLR-11484
> URL: https://issues.apache.org/jira/browse/SOLR-11484
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Hoss Man
>Assignee: Noble Paul
> Fix For: 7.2, master (8.0)
>
> Attachments: SOLR-11484.patch, SOLR-11484.patch, 
> jenkins.thetaphi.20662.txt
>
>
> This was discovered while auditing jenkins failures from 
> {{TestCollectionsAPIViaSolrCloudCluster.testCollectionCreateSearchDelete}} 
> (where a test explicitly deletes and then recreates a collection with the 
> same name), but as noted in a comment below, SOLR-11392 is another example of 
> non-obvious test failures that can pop up because of this bug.
> In practice, it can affect any CloudSolrClient user after changes have been 
> made to a collection (to add/move replicas, etc...)
> 
> Original jira notes...
> {{TestCollectionsAPIViaSolrCloudCluster.testCollectionCreateSearchDelete}}
> seems to fail with non-trivial frequency, so I grabbed the logs from a recent 
> failure and starting trying to follow along with the actions to figure out 
> what exactly is happening
> https://jenkins.thetaphi.de/job/Lucene-Solr-master-Linux/20662/
> {noformat}
>[junit4] ERROR   20.3s J1 | 
> TestCollectionsAPIViaSolrCloudCluster.testCollectionCreateSearchDelete <<<
>[junit4]> Throwable #1: 
> org.apache.solr.client.solrj.impl.CloudSolrClient$RouteException: Error from 
> server at https://127.0.0.1:42959/solr/testcollection_shard1_replica_n3: 
> Expected mime type a
> pplication/octet-stream but got text/html. 
>[junit4]> 
>[junit4]>  content="text/html;charset=ISO-8859-1"/>
>[junit4]> Error 404 
> {noformat}
> The crux of this failure appears to be a genuine bug in how CloudSolrClient 
> uses it's cached ClusterState info when doing (direct) updates.  The key bits 
> seem to be:
> * CloudSolrClient does _something_ (update,query,etc...) with a collection 
> causing the current cluster state for the collection to be cached
> * The actual collection changes such that a Solr node/core no longer exists 
> as part of the collection
> * CloudSolrClient is asked to process an UpdateRequest which triggers the 
> code paths for the {{directUpdate()}} method -- which attempts to route the 
> updates directly to a replica of the appropriate shard using the (cache) 
> collection state info
> * CloudSolrClient (may) attempt to send that UpdateRequest to a node/core 
> that doesn't exist, getting a 404 -- which does not (seem to) trigger a state 
> refresh, or retry to find a correct URL to resend the update to.
> Details to follow in comment



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-11624) collection creation should not also overwrite/delete any configset but it can!

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-11624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16306661#comment-16306661
 ] 

ASF GitHub Bot commented on SOLR-11624:
---

GitHub user abhidemon opened a pull request:

https://github.com/apache/lucene-solr/pull/297

fix: https://issues.apache.org/jira/browse/SOLR-11624



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/abhidemon/lucene-solr SOLR-11624

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/lucene-solr/pull/297.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #297


commit 01eac2e19d0badf095816fa8f8a546653c0e4556
Author: abhidemon 
Date:   2017-12-12T20:44:56Z

fix: https://issues.apache.org/jira/browse/SOLR-11624




> collection creation should not also overwrite/delete any configset but it can!
> --
>
> Key: SOLR-11624
> URL: https://issues.apache.org/jira/browse/SOLR-11624
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>Affects Versions: 7.2
>Reporter: Erick Erickson
>Assignee: Ishan Chattopadhyaya
> Attachments: SOLR-11624-2.patch, SOLR-11624.3.patch, 
> SOLR-11624.4.patch, SOLR-11624.patch
>
>
> Looks like a problem that crept in when we changed the _default configset 
> stuff.
> setup:
> upload a configset named "wiki"
> collections?action=CREATE=wiki&.
> My custom configset "wiki" gets overwritten by _default and then used by the 
> "wiki" collection.
> Assigning to myself only because it really needs to be fixed IMO and I don't 
> want to lose track of it. Anyone else please feel free to take it.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-11804) Test RankQuery in distributed mode

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-11804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16306291#comment-16306291
 ] 

ASF GitHub Bot commented on SOLR-11804:
---

GitHub user diegoceccarelli opened a pull request:

https://github.com/apache/lucene-solr/pull/296

SOLR-11804: Test RankQuery in distributed mode

Currently `RankQuery` is not tested in distributed mode. I added a few 
tests in `TestDistributedSearch` to check that it works properly.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/bloomberg/lucene-solr SOLR-11804

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/lucene-solr/pull/296.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #296


commit 1591ec8e3c5e0cd670890642b34aba4eb22e0a99
Author: Diego Ceccarelli 
Date:   2017-12-29T13:51:04Z

SOLR-11804: Test RankQuery in distributed mode




> Test RankQuery in distributed mode
> --
>
> Key: SOLR-11804
> URL: https://issues.apache.org/jira/browse/SOLR-11804
> Project: Solr
>  Issue Type: Test
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Diego Ceccarelli
>Priority: Minor
>
> Currently {{RankQuery}} is not tested in distributed mode. I added a few 
> tests in `TestDistributedSearch` to check that it works properly.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-11800) Improve error message when parsing RankQuery

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-11800?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16305583#comment-16305583
 ] 

ASF GitHub Bot commented on SOLR-11800:
---

Github user diegoceccarelli commented on a diff in the pull request:

https://github.com/apache/lucene-solr/pull/295#discussion_r158969387
  
--- Diff: solr/core/src/java/org/apache/solr/search/QParser.java ---
@@ -360,6 +360,10 @@ public static QParser getParser(String qstr, String 
parserName, boolean allowLoc
 }
 
 QParserPlugin qplug = req.getCore().getQueryPlugin(parserName);
+if (qplug == null){
+  // error: log ?
--- End diff --

I would log an error here, but `QParser` doesn't have a logger, should we 
add one? 


> Improve error message when parsing RankQuery
> 
>
> Key: SOLR-11800
> URL: https://issues.apache.org/jira/browse/SOLR-11800
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Diego Ceccarelli
>Priority: Minor
>
> When a user specifies something wrong for the parameter {{rq}} sometimes it 
> is hard to understand where is the problem, this patch attempts to improve 
> the error message returned in the response.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-11800) Improve error message when parsing RankQuery

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-11800?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16305580#comment-16305580
 ] 

ASF GitHub Bot commented on SOLR-11800:
---

GitHub user diegoceccarelli opened a pull request:

https://github.com/apache/lucene-solr/pull/295

SOLR-11800: Improve error message when parsing RankQuery

When a user specifies something wrong for the parameter `rq` sometimes it 
is hard to understand where is the problem, this patch attempts to improve the 
error message returned in the response.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/bloomberg/lucene-solr SOLR-11800

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/lucene-solr/pull/295.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #295


commit 64928e43710eb86be0f7e23adbc8107da124efb8
Author: Diego Ceccarelli 
Date:   2017-12-28T16:22:55Z

SOLR-11800: Improve error message when parsing RankQuery




> Improve error message when parsing RankQuery
> 
>
> Key: SOLR-11800
> URL: https://issues.apache.org/jira/browse/SOLR-11800
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Diego Ceccarelli
>Priority: Minor
>
> when a user specify something wrong for the parameter rq sometimes it is hard 
> to understand where is the problem, this patch attempts to improve the error 
> message returned in the response.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-11622) Bundled mime4j library not sufficient for Tika requirement

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-11622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16304573#comment-16304573
 ] 

ASF GitHub Bot commented on SOLR-11622:
---

Github user mrkarthik closed the pull request at:

https://github.com/apache/lucene-solr/pull/273


> Bundled mime4j library not sufficient for Tika requirement
> --
>
> Key: SOLR-11622
> URL: https://issues.apache.org/jira/browse/SOLR-11622
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: Build
>Affects Versions: 7.1, 6.6.2
>Reporter: Karim Malhas
>Assignee: Karthik Ramachandran
>Priority: Minor
>  Labels: build
> Fix For: 7.3
>
> Attachments: SOLR-11622.patch, SOLR-11622.patch
>
>
> The version 7.2 of Apache James Mime4j bundled with the Solr binary releases 
> does not match what is required by Apache Tika for parsing rfc2822 messages. 
> The master branch for james-mime4j seems to contain the missing Builder class
> [https://github.com/apache/james-mime4j/blob/master/core/src/main/java/org/apache/james/mime4j/stream/MimeConfig.java
> ]
> This prevents import of rfc2822 formatted messages. For example like so:
> {{./bin/post -c dovecot -type 'message/rfc822' 'testdata/email_01.txt'
> }}
> And results in the following stacktrace:
> java.lang.NoClassDefFoundError: 
> org/apache/james/mime4j/stream/MimeConfig$Builder
> at 
> org.apache.tika.parser.mail.RFC822Parser.parse(RFC822Parser.java:63)
> at 
> org.apache.tika.parser.CompositeParser.parse(CompositeParser.java:280)
> at 
> org.apache.tika.parser.CompositeParser.parse(CompositeParser.java:280)
> at 
> org.apache.tika.parser.AutoDetectParser.parse(AutoDetectParser.java:135)
> at 
> org.apache.solr.handler.extraction.ExtractingDocumentLoader.load(ExtractingDocumentLoader.java:228)
> at 
> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:68)
> at 
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:173)
> at org.apache.solr.core.SolrCore.execute(SolrCore.java:2477)
> at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:723)
> at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:529)
> at 
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:361)
> at 
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:305)
> at 
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1691)
> at 
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
> at 
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
> at 
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
> at 
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
> at 
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
> at 
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
> at 
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
> at 
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
> at 
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
> at 
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
> at 
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
> at 
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
> at 
> org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)
> at 
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
> at org.eclipse.jetty.server.Server.handle(Server.java:534)
> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
> at 
> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
> at 
> org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
> at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
> at 
> org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
> at 
> org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
> at 
> org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
> 

[jira] [Commented] (SOLR-11620) Occasional NullPointerException in CloudSolrClient

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-11620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16299301#comment-16299301
 ] 

ASF GitHub Bot commented on SOLR-11620:
---

Github user tflobbe commented on the issue:

https://github.com/apache/lucene-solr/pull/264
  
I believe this is the same as reported in SOLR-11620


> Occasional NullPointerException in CloudSolrClient 
> ---
>
> Key: SOLR-11620
> URL: https://issues.apache.org/jira/browse/SOLR-11620
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: SolrJ
>Affects Versions: 6.6.1
>Reporter: Rob Trickey
>Priority: Minor
>
> When sending document updates to Solr, we occasionally see the following 
> error:
> {noformat}
> java.lang.NullPointerException
>   at 
> org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:1182)
>   at 
> org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:1073)
>   at 
> org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:160)
>   at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:173)
>   at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:138)
>   at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:152)
> {noformat}
> Looking at the code, there is a lack of null check on requestedCollections 
> around the for loop
> for (DocCollection ext : requestedCollections) 
> which causes the error. Wrapping this loop with if(requestedCollections != 
> null) would solve the problem.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-8327) SolrDispatchFilter is not caching new state format, which results in live fetch from ZK per request if node does not contain core from collection

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-8327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16299071#comment-16299071
 ] 

ASF GitHub Bot commented on SOLR-8327:
--

Github user slackhappy commented on the issue:

https://github.com/apache/lucene-solr/pull/294
  
I updated my PR to target SOLR-8327 more specifically, and got the tests to 
pass.  I think a smarter approach like that used by CloudSolrClient would be 
great. My understanding of the change in SOLR-10524 is that even the 
smartest/fastest updates of zookeeper data won't match the real-world state of 
the cluster in many situations, such as replica state changes, because those 
will be batched, but certainly a smarter approach would narrow that gap as much 
as possible, in addition to reducing the amount of state fetching.


> SolrDispatchFilter is not caching new state format, which results in live 
> fetch from ZK per request if node does not contain core from collection
> -
>
> Key: SOLR-8327
> URL: https://issues.apache.org/jira/browse/SOLR-8327
> Project: Solr
>  Issue Type: Bug
>  Components: SolrCloud
>Affects Versions: 5.3
>Reporter: Jessica Cheng Mallet
>Assignee: Varun Thacker
>  Labels: solrcloud
> Attachments: SOLR-8327.patch
>
>
> While perf testing with non-solrj client (request can be sent to any solr 
> node), we noticed a huge amount of data from Zookeeper in our tcpdump (~1G 
> for 20 second dump). From the thread dump, we noticed this:
> java.lang.Object.wait (Native Method)
> java.lang.Object.wait (Object.java:503)
> org.apache.zookeeper.ClientCnxn.submitRequest (ClientCnxn.java:1309)
> org.apache.zookeeper.ZooKeeper.getData (ZooKeeper.java:1152)
> org.apache.solr.common.cloud.SolrZkClient$7.execute (SolrZkClient.java:345)
> org.apache.solr.common.cloud.SolrZkClient$7.execute (SolrZkClient.java:342)
> org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation 
> (ZkCmdExecutor.java:61)
> org.apache.solr.common.cloud.SolrZkClient.getData (SolrZkClient.java:342)
> org.apache.solr.common.cloud.ZkStateReader.getCollectionLive 
> (ZkStateReader.java:841)
> org.apache.solr.common.cloud.ZkStateReader$7.get (ZkStateReader.java:515)
> org.apache.solr.common.cloud.ClusterState.getCollectionOrNull 
> (ClusterState.java:175)
> org.apache.solr.common.cloud.ClusterState.getLeader (ClusterState.java:98)
> org.apache.solr.servlet.HttpSolrCall.getCoreByCollection 
> (HttpSolrCall.java:784)
> org.apache.solr.servlet.HttpSolrCall.init (HttpSolrCall.java:272)
> org.apache.solr.servlet.HttpSolrCall.call (HttpSolrCall.java:417)
> org.apache.solr.servlet.SolrDispatchFilter.doFilter 
> (SolrDispatchFilter.java:210)
> org.apache.solr.servlet.SolrDispatchFilter.doFilter 
> (SolrDispatchFilter.java:179)
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter 
> (ServletHandler.java:1652)
> org.eclipse.jetty.servlet.ServletHandler.doHandle (ServletHandler.java:585)
> org.eclipse.jetty.server.handler.ScopedHandler.handle (ScopedHandler.java:143)
> org.eclipse.jetty.security.SecurityHandler.handle (SecurityHandler.java:577)
> org.eclipse.jetty.server.session.SessionHandler.doHandle 
> (SessionHandler.java:223)
> org.eclipse.jetty.server.handler.ContextHandler.doHandle 
> (ContextHandler.java:1127)
> org.eclipse.jetty.servlet.ServletHandler.doScope (ServletHandler.java:515)
> org.eclipse.jetty.server.session.SessionHandler.doScope 
> (SessionHandler.java:185)
> org.eclipse.jetty.server.handler.ContextHandler.doScope 
> (ContextHandler.java:1061)
> org.eclipse.jetty.server.handler.ScopedHandler.handle (ScopedHandler.java:141)
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle 
> (ContextHandlerCollection.java:215)
> org.eclipse.jetty.server.handler.HandlerCollection.handle 
> (HandlerCollection.java:110)
> org.eclipse.jetty.server.handler.HandlerWrapper.handle 
> (HandlerWrapper.java:97)
> org.eclipse.jetty.server.Server.handle (Server.java:499)
> org.eclipse.jetty.server.HttpChannel.handle (HttpChannel.java:310)
> org.eclipse.jetty.server.HttpConnection.onFillable (HttpConnection.java:257)
> org.eclipse.jetty.io.AbstractConnection$2.run (AbstractConnection.java:540)
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob 
> (QueuedThreadPool.java:635)
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run 
> (QueuedThreadPool.java:555)
> java.lang.Thread.run (Thread.java:745)
> Looks like SolrDispatchFilter doesn't have caching similar to the 
> collectionStateCache in CloudSolrClient, so if the node doesn't know about a 
> collection in the new state format, it just live-fetch it from Zookeeper on 
> every request.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (SOLR-8327) SolrDispatchFilter is not caching new state format, which results in live fetch from ZK per request if node does not contain core from collection

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-8327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16297686#comment-16297686
 ] 

ASF GitHub Bot commented on SOLR-8327:
--

Github user slackhappy commented on the issue:

https://github.com/apache/lucene-solr/pull/294
  
I'll look into the test failures.  I actually didn't mean to create the PR 
yet  


> SolrDispatchFilter is not caching new state format, which results in live 
> fetch from ZK per request if node does not contain core from collection
> -
>
> Key: SOLR-8327
> URL: https://issues.apache.org/jira/browse/SOLR-8327
> Project: Solr
>  Issue Type: Bug
>  Components: SolrCloud
>Affects Versions: 5.3
>Reporter: Jessica Cheng Mallet
>Assignee: Varun Thacker
>  Labels: solrcloud
> Attachments: SOLR-8327.patch
>
>
> While perf testing with non-solrj client (request can be sent to any solr 
> node), we noticed a huge amount of data from Zookeeper in our tcpdump (~1G 
> for 20 second dump). From the thread dump, we noticed this:
> java.lang.Object.wait (Native Method)
> java.lang.Object.wait (Object.java:503)
> org.apache.zookeeper.ClientCnxn.submitRequest (ClientCnxn.java:1309)
> org.apache.zookeeper.ZooKeeper.getData (ZooKeeper.java:1152)
> org.apache.solr.common.cloud.SolrZkClient$7.execute (SolrZkClient.java:345)
> org.apache.solr.common.cloud.SolrZkClient$7.execute (SolrZkClient.java:342)
> org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation 
> (ZkCmdExecutor.java:61)
> org.apache.solr.common.cloud.SolrZkClient.getData (SolrZkClient.java:342)
> org.apache.solr.common.cloud.ZkStateReader.getCollectionLive 
> (ZkStateReader.java:841)
> org.apache.solr.common.cloud.ZkStateReader$7.get (ZkStateReader.java:515)
> org.apache.solr.common.cloud.ClusterState.getCollectionOrNull 
> (ClusterState.java:175)
> org.apache.solr.common.cloud.ClusterState.getLeader (ClusterState.java:98)
> org.apache.solr.servlet.HttpSolrCall.getCoreByCollection 
> (HttpSolrCall.java:784)
> org.apache.solr.servlet.HttpSolrCall.init (HttpSolrCall.java:272)
> org.apache.solr.servlet.HttpSolrCall.call (HttpSolrCall.java:417)
> org.apache.solr.servlet.SolrDispatchFilter.doFilter 
> (SolrDispatchFilter.java:210)
> org.apache.solr.servlet.SolrDispatchFilter.doFilter 
> (SolrDispatchFilter.java:179)
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter 
> (ServletHandler.java:1652)
> org.eclipse.jetty.servlet.ServletHandler.doHandle (ServletHandler.java:585)
> org.eclipse.jetty.server.handler.ScopedHandler.handle (ScopedHandler.java:143)
> org.eclipse.jetty.security.SecurityHandler.handle (SecurityHandler.java:577)
> org.eclipse.jetty.server.session.SessionHandler.doHandle 
> (SessionHandler.java:223)
> org.eclipse.jetty.server.handler.ContextHandler.doHandle 
> (ContextHandler.java:1127)
> org.eclipse.jetty.servlet.ServletHandler.doScope (ServletHandler.java:515)
> org.eclipse.jetty.server.session.SessionHandler.doScope 
> (SessionHandler.java:185)
> org.eclipse.jetty.server.handler.ContextHandler.doScope 
> (ContextHandler.java:1061)
> org.eclipse.jetty.server.handler.ScopedHandler.handle (ScopedHandler.java:141)
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle 
> (ContextHandlerCollection.java:215)
> org.eclipse.jetty.server.handler.HandlerCollection.handle 
> (HandlerCollection.java:110)
> org.eclipse.jetty.server.handler.HandlerWrapper.handle 
> (HandlerWrapper.java:97)
> org.eclipse.jetty.server.Server.handle (Server.java:499)
> org.eclipse.jetty.server.HttpChannel.handle (HttpChannel.java:310)
> org.eclipse.jetty.server.HttpConnection.onFillable (HttpConnection.java:257)
> org.eclipse.jetty.io.AbstractConnection$2.run (AbstractConnection.java:540)
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob 
> (QueuedThreadPool.java:635)
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run 
> (QueuedThreadPool.java:555)
> java.lang.Thread.run (Thread.java:745)
> Looks like SolrDispatchFilter doesn't have caching similar to the 
> collectionStateCache in CloudSolrClient, so if the node doesn't know about a 
> collection in the new state format, it just live-fetch it from Zookeeper on 
> every request.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-8327) SolrDispatchFilter is not caching new state format, which results in live fetch from ZK per request if node does not contain core from collection

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-8327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16297495#comment-16297495
 ] 

ASF GitHub Bot commented on SOLR-8327:
--

Github user chatman commented on the issue:

https://github.com/apache/lucene-solr/pull/294
  
Seems like there are some test failures due to this change:

   [junit4] Tests with failures [seed: DE1D5337E38D2C32]:
   [junit4]   - 
org.apache.solr.cloud.TestPullReplica.testRemoveAllWriterReplicas
   [junit4]   - 
org.apache.solr.cloud.TestPullReplica.testAddRemovePullReplica
   [junit4]   - 
org.apache.solr.cloud.CollectionTooManyReplicasTest.testAddTooManyReplicas
   [junit4]   - 
org.apache.solr.cloud.CollectionsAPIDistributedZkTest.addReplicaTest
   [junit4]   - 
org.apache.solr.cloud.DeleteShardTest.testDirectoryCleanupAfterDeleteShard
   [junit4]   - org.apache.solr.cloud.TestCloudRecovery.corruptedLogTest
   [junit4]   - 
org.apache.solr.cloud.TestCloudRecovery.leaderRecoverFromLogOnStartupTest
   [junit4]   - org.apache.solr.cloud.TestUtilizeNode.test
   [junit4]   - 
org.apache.solr.cloud.TestTlogReplica.testAddRemoveTlogReplica

Haven't looked into them yet, though.


> SolrDispatchFilter is not caching new state format, which results in live 
> fetch from ZK per request if node does not contain core from collection
> -
>
> Key: SOLR-8327
> URL: https://issues.apache.org/jira/browse/SOLR-8327
> Project: Solr
>  Issue Type: Bug
>  Components: SolrCloud
>Affects Versions: 5.3
>Reporter: Jessica Cheng Mallet
>Assignee: Varun Thacker
>  Labels: solrcloud
> Attachments: SOLR-8327.patch
>
>
> While perf testing with non-solrj client (request can be sent to any solr 
> node), we noticed a huge amount of data from Zookeeper in our tcpdump (~1G 
> for 20 second dump). From the thread dump, we noticed this:
> java.lang.Object.wait (Native Method)
> java.lang.Object.wait (Object.java:503)
> org.apache.zookeeper.ClientCnxn.submitRequest (ClientCnxn.java:1309)
> org.apache.zookeeper.ZooKeeper.getData (ZooKeeper.java:1152)
> org.apache.solr.common.cloud.SolrZkClient$7.execute (SolrZkClient.java:345)
> org.apache.solr.common.cloud.SolrZkClient$7.execute (SolrZkClient.java:342)
> org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation 
> (ZkCmdExecutor.java:61)
> org.apache.solr.common.cloud.SolrZkClient.getData (SolrZkClient.java:342)
> org.apache.solr.common.cloud.ZkStateReader.getCollectionLive 
> (ZkStateReader.java:841)
> org.apache.solr.common.cloud.ZkStateReader$7.get (ZkStateReader.java:515)
> org.apache.solr.common.cloud.ClusterState.getCollectionOrNull 
> (ClusterState.java:175)
> org.apache.solr.common.cloud.ClusterState.getLeader (ClusterState.java:98)
> org.apache.solr.servlet.HttpSolrCall.getCoreByCollection 
> (HttpSolrCall.java:784)
> org.apache.solr.servlet.HttpSolrCall.init (HttpSolrCall.java:272)
> org.apache.solr.servlet.HttpSolrCall.call (HttpSolrCall.java:417)
> org.apache.solr.servlet.SolrDispatchFilter.doFilter 
> (SolrDispatchFilter.java:210)
> org.apache.solr.servlet.SolrDispatchFilter.doFilter 
> (SolrDispatchFilter.java:179)
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter 
> (ServletHandler.java:1652)
> org.eclipse.jetty.servlet.ServletHandler.doHandle (ServletHandler.java:585)
> org.eclipse.jetty.server.handler.ScopedHandler.handle (ScopedHandler.java:143)
> org.eclipse.jetty.security.SecurityHandler.handle (SecurityHandler.java:577)
> org.eclipse.jetty.server.session.SessionHandler.doHandle 
> (SessionHandler.java:223)
> org.eclipse.jetty.server.handler.ContextHandler.doHandle 
> (ContextHandler.java:1127)
> org.eclipse.jetty.servlet.ServletHandler.doScope (ServletHandler.java:515)
> org.eclipse.jetty.server.session.SessionHandler.doScope 
> (SessionHandler.java:185)
> org.eclipse.jetty.server.handler.ContextHandler.doScope 
> (ContextHandler.java:1061)
> org.eclipse.jetty.server.handler.ScopedHandler.handle (ScopedHandler.java:141)
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle 
> (ContextHandlerCollection.java:215)
> org.eclipse.jetty.server.handler.HandlerCollection.handle 
> (HandlerCollection.java:110)
> org.eclipse.jetty.server.handler.HandlerWrapper.handle 
> (HandlerWrapper.java:97)
> org.eclipse.jetty.server.Server.handle (Server.java:499)
> org.eclipse.jetty.server.HttpChannel.handle (HttpChannel.java:310)
> org.eclipse.jetty.server.HttpConnection.onFillable (HttpConnection.java:257)
> org.eclipse.jetty.io.AbstractConnection$2.run (AbstractConnection.java:540)
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob 
> (QueuedThreadPool.java:635)
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run 

[jira] [Commented] (SOLR-8327) SolrDispatchFilter is not caching new state format, which results in live fetch from ZK per request if node does not contain core from collection

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-8327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16297410#comment-16297410
 ] 

ASF GitHub Bot commented on SOLR-8327:
--

Github user elyograg commented on the issue:

https://github.com/apache/lucene-solr/pull/294
  
Java programs are migrating to nanoTime instead of currentTimeMillis for 
elapsed time because many people have found that the latter will go *backwards* 
on occasion.  It is not monotonic.

Using nanoTime should be far less likely to go backwards.  That undesirable 
behavior has been observed in the wild, but should be rare.  Supposedly 
nanoTime is monotonic if the OS properly supports a monotonic clock.  There's a 
lot of info out there about it:

https://www.google.com/search?q=java+nanotime+monotonic

The fact that nanoTime *might* produce elapsed times with greater accuracy 
than one millisecond is a bonus.


> SolrDispatchFilter is not caching new state format, which results in live 
> fetch from ZK per request if node does not contain core from collection
> -
>
> Key: SOLR-8327
> URL: https://issues.apache.org/jira/browse/SOLR-8327
> Project: Solr
>  Issue Type: Bug
>  Components: SolrCloud
>Affects Versions: 5.3
>Reporter: Jessica Cheng Mallet
>Assignee: Varun Thacker
>  Labels: solrcloud
> Attachments: SOLR-8327.patch
>
>
> While perf testing with non-solrj client (request can be sent to any solr 
> node), we noticed a huge amount of data from Zookeeper in our tcpdump (~1G 
> for 20 second dump). From the thread dump, we noticed this:
> java.lang.Object.wait (Native Method)
> java.lang.Object.wait (Object.java:503)
> org.apache.zookeeper.ClientCnxn.submitRequest (ClientCnxn.java:1309)
> org.apache.zookeeper.ZooKeeper.getData (ZooKeeper.java:1152)
> org.apache.solr.common.cloud.SolrZkClient$7.execute (SolrZkClient.java:345)
> org.apache.solr.common.cloud.SolrZkClient$7.execute (SolrZkClient.java:342)
> org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation 
> (ZkCmdExecutor.java:61)
> org.apache.solr.common.cloud.SolrZkClient.getData (SolrZkClient.java:342)
> org.apache.solr.common.cloud.ZkStateReader.getCollectionLive 
> (ZkStateReader.java:841)
> org.apache.solr.common.cloud.ZkStateReader$7.get (ZkStateReader.java:515)
> org.apache.solr.common.cloud.ClusterState.getCollectionOrNull 
> (ClusterState.java:175)
> org.apache.solr.common.cloud.ClusterState.getLeader (ClusterState.java:98)
> org.apache.solr.servlet.HttpSolrCall.getCoreByCollection 
> (HttpSolrCall.java:784)
> org.apache.solr.servlet.HttpSolrCall.init (HttpSolrCall.java:272)
> org.apache.solr.servlet.HttpSolrCall.call (HttpSolrCall.java:417)
> org.apache.solr.servlet.SolrDispatchFilter.doFilter 
> (SolrDispatchFilter.java:210)
> org.apache.solr.servlet.SolrDispatchFilter.doFilter 
> (SolrDispatchFilter.java:179)
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter 
> (ServletHandler.java:1652)
> org.eclipse.jetty.servlet.ServletHandler.doHandle (ServletHandler.java:585)
> org.eclipse.jetty.server.handler.ScopedHandler.handle (ScopedHandler.java:143)
> org.eclipse.jetty.security.SecurityHandler.handle (SecurityHandler.java:577)
> org.eclipse.jetty.server.session.SessionHandler.doHandle 
> (SessionHandler.java:223)
> org.eclipse.jetty.server.handler.ContextHandler.doHandle 
> (ContextHandler.java:1127)
> org.eclipse.jetty.servlet.ServletHandler.doScope (ServletHandler.java:515)
> org.eclipse.jetty.server.session.SessionHandler.doScope 
> (SessionHandler.java:185)
> org.eclipse.jetty.server.handler.ContextHandler.doScope 
> (ContextHandler.java:1061)
> org.eclipse.jetty.server.handler.ScopedHandler.handle (ScopedHandler.java:141)
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle 
> (ContextHandlerCollection.java:215)
> org.eclipse.jetty.server.handler.HandlerCollection.handle 
> (HandlerCollection.java:110)
> org.eclipse.jetty.server.handler.HandlerWrapper.handle 
> (HandlerWrapper.java:97)
> org.eclipse.jetty.server.Server.handle (Server.java:499)
> org.eclipse.jetty.server.HttpChannel.handle (HttpChannel.java:310)
> org.eclipse.jetty.server.HttpConnection.onFillable (HttpConnection.java:257)
> org.eclipse.jetty.io.AbstractConnection$2.run (AbstractConnection.java:540)
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob 
> (QueuedThreadPool.java:635)
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run 
> (QueuedThreadPool.java:555)
> java.lang.Thread.run (Thread.java:745)
> Looks like SolrDispatchFilter doesn't have caching similar to the 
> collectionStateCache in CloudSolrClient, so if the node doesn't know about a 
> collection in the new state format, it just live-fetch it from Zookeeper on 

[jira] [Commented] (SOLR-8327) SolrDispatchFilter is not caching new state format, which results in live fetch from ZK per request if node does not contain core from collection

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-8327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16297372#comment-16297372
 ] 

ASF GitHub Bot commented on SOLR-8327:
--

Github user dragonsinth commented on the issue:

https://github.com/apache/lucene-solr/pull/294
  
This approach seems fine to me.  Remind me why we use nanoTime vs. normal 
clock?  I'm sure you're right I just want to refresh my brain.


> SolrDispatchFilter is not caching new state format, which results in live 
> fetch from ZK per request if node does not contain core from collection
> -
>
> Key: SOLR-8327
> URL: https://issues.apache.org/jira/browse/SOLR-8327
> Project: Solr
>  Issue Type: Bug
>  Components: SolrCloud
>Affects Versions: 5.3
>Reporter: Jessica Cheng Mallet
>Assignee: Varun Thacker
>  Labels: solrcloud
> Attachments: SOLR-8327.patch
>
>
> While perf testing with non-solrj client (request can be sent to any solr 
> node), we noticed a huge amount of data from Zookeeper in our tcpdump (~1G 
> for 20 second dump). From the thread dump, we noticed this:
> java.lang.Object.wait (Native Method)
> java.lang.Object.wait (Object.java:503)
> org.apache.zookeeper.ClientCnxn.submitRequest (ClientCnxn.java:1309)
> org.apache.zookeeper.ZooKeeper.getData (ZooKeeper.java:1152)
> org.apache.solr.common.cloud.SolrZkClient$7.execute (SolrZkClient.java:345)
> org.apache.solr.common.cloud.SolrZkClient$7.execute (SolrZkClient.java:342)
> org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation 
> (ZkCmdExecutor.java:61)
> org.apache.solr.common.cloud.SolrZkClient.getData (SolrZkClient.java:342)
> org.apache.solr.common.cloud.ZkStateReader.getCollectionLive 
> (ZkStateReader.java:841)
> org.apache.solr.common.cloud.ZkStateReader$7.get (ZkStateReader.java:515)
> org.apache.solr.common.cloud.ClusterState.getCollectionOrNull 
> (ClusterState.java:175)
> org.apache.solr.common.cloud.ClusterState.getLeader (ClusterState.java:98)
> org.apache.solr.servlet.HttpSolrCall.getCoreByCollection 
> (HttpSolrCall.java:784)
> org.apache.solr.servlet.HttpSolrCall.init (HttpSolrCall.java:272)
> org.apache.solr.servlet.HttpSolrCall.call (HttpSolrCall.java:417)
> org.apache.solr.servlet.SolrDispatchFilter.doFilter 
> (SolrDispatchFilter.java:210)
> org.apache.solr.servlet.SolrDispatchFilter.doFilter 
> (SolrDispatchFilter.java:179)
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter 
> (ServletHandler.java:1652)
> org.eclipse.jetty.servlet.ServletHandler.doHandle (ServletHandler.java:585)
> org.eclipse.jetty.server.handler.ScopedHandler.handle (ScopedHandler.java:143)
> org.eclipse.jetty.security.SecurityHandler.handle (SecurityHandler.java:577)
> org.eclipse.jetty.server.session.SessionHandler.doHandle 
> (SessionHandler.java:223)
> org.eclipse.jetty.server.handler.ContextHandler.doHandle 
> (ContextHandler.java:1127)
> org.eclipse.jetty.servlet.ServletHandler.doScope (ServletHandler.java:515)
> org.eclipse.jetty.server.session.SessionHandler.doScope 
> (SessionHandler.java:185)
> org.eclipse.jetty.server.handler.ContextHandler.doScope 
> (ContextHandler.java:1061)
> org.eclipse.jetty.server.handler.ScopedHandler.handle (ScopedHandler.java:141)
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle 
> (ContextHandlerCollection.java:215)
> org.eclipse.jetty.server.handler.HandlerCollection.handle 
> (HandlerCollection.java:110)
> org.eclipse.jetty.server.handler.HandlerWrapper.handle 
> (HandlerWrapper.java:97)
> org.eclipse.jetty.server.Server.handle (Server.java:499)
> org.eclipse.jetty.server.HttpChannel.handle (HttpChannel.java:310)
> org.eclipse.jetty.server.HttpConnection.onFillable (HttpConnection.java:257)
> org.eclipse.jetty.io.AbstractConnection$2.run (AbstractConnection.java:540)
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob 
> (QueuedThreadPool.java:635)
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run 
> (QueuedThreadPool.java:555)
> java.lang.Thread.run (Thread.java:745)
> Looks like SolrDispatchFilter doesn't have caching similar to the 
> collectionStateCache in CloudSolrClient, so if the node doesn't know about a 
> collection in the new state format, it just live-fetch it from Zookeeper on 
> every request.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-10524) Better ZkStateWriter batching

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-10524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16297192#comment-16297192
 ] 

ASF GitHub Bot commented on SOLR-10524:
---

GitHub user slackhappy opened a pull request:

https://github.com/apache/lucene-solr/pull/294

ZkStateReader: cache LazyCollectionRef

SOLR-10524 introduced zk state update batching, with
a default interval of 2 seconds.  That opens
the door for a simple, time-based cache on the read side
to address the issue described in SOLR-8327

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/slackhappy/lucene-solr 
cloud_cache_lazy_collection_ref

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/lucene-solr/pull/294.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #294


commit e7c6a6773f1d01d2ddb0dbce6cdbaeff52e78376
Author: John Gallagher 
Date:   2017-12-19T16:57:25Z

ZkStateReader: cache LazyCollectionRef

SOLR-10524 introduced zk state update batching, with
a default interval of 2 seconds.  That opens
the door for a simple, time-based cache on the read side
to address the issue described in SOLR-8327




> Better ZkStateWriter batching
> -
>
> Key: SOLR-10524
> URL: https://issues.apache.org/jira/browse/SOLR-10524
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Erick Erickson
>Assignee: Cao Manh Dat
> Fix For: 6.6, 7.0
>
> Attachments: SOLR-10524-NPE-fix.patch, SOLR-10524-dragonsinth.patch, 
> SOLR-10524.patch, SOLR-10524.patch, SOLR-10524.patch, SOLR-10524.patch
>
>
> There are several JIRAs (I'll link in a second) about trying to be more 
> efficient about processing overseer messages as the overseer can become a 
> bottleneck, especially with very large numbers of replicas in a cluster. One 
> of the approaches mentioned near the end of SOLR-5872 (15-Mar) was to "read 
> large no:of items say 1. put them into in memory buckets and feed them 
> into overseer".
> This JIRA is to break out that part of the discussion as it might be an easy 
> win whereas "eliminating the Overseer queue" would be quite an undertaking.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-11701) Upgrade to Tika 1.17 when available

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-11701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16291932#comment-16291932
 ] 

ASF GitHub Bot commented on SOLR-11701:
---

GitHub user tballison opened a pull request:

https://github.com/apache/lucene-solr/pull/291

jira/solr-11701

SOLR-11701 upgrade to Tika 1.17

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tballison/lucene-solr jira/solr-11701

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/lucene-solr/pull/291.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #291


commit c5d4e37de782b2491b3e71cfbb004e5022b55f6b
Author: Karthik Ramachandran 
Date:   2017-11-14T00:21:44Z

SOLR-11622: Fix mime4j library dependency for Tika

commit 40b246b12e8fc6455e023d9d60b8edcfab9b184e
Author: Karthik Ramachandran 
Date:   2017-12-01T22:12:15Z

Merge remote-tracking branch 'upstream/master' into jira/solr-11622

commit 21f2ab483f356fad9b89233e544457a07540afd1
Author: Karthik Ramachandran 
Date:   2017-12-03T03:50:01Z

SOLR-11622: Fix bundled mime4j library not sufficient for Tika requirement

commit a40ca80ed7036732a332f5508589ae32eb4b
Author: Karthik Ramachandran 
Date:   2017-12-04T15:33:18Z

Merge remote-tracking branch 'upstream/master' into jira/solr-11622

commit a0d6fba8c2e85565a02a8565882a627fa7ceccc4
Author: Karthik Ramachandran 
Date:   2017-12-14T16:24:45Z

Merge remote-tracking branch 'upstream/master' into jira/SOLR-11622

commit c2c885f8a2e2c49fab6f737b13f0ff9a1346714c
Author: Karthik Ramachandran 
Date:   2017-12-14T20:45:09Z

SOLR-11622: Fix mime4j library dependency for Tika

commit e834693a31d0b410a7e0205e1eecda55206a44fa
Author: tballison 
Date:   2017-12-15T02:20:51Z

SOLR-11701 - upgrade to Tika 1.17




> Upgrade to Tika 1.17 when available
> ---
>
> Key: SOLR-11701
> URL: https://issues.apache.org/jira/browse/SOLR-11701
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Tim Allison
>
> Kicking off release process for Tika 1.17 in the next few days.  Please let 
> us know if you have any requests.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-2857) Multi-content-type /update handler

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-2857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16289242#comment-16289242
 ] 

ASF GitHub Bot commented on SOLR-2857:
--

GitHub user steeveb972 opened a pull request:

https://github.com/apache/lucene-solr/pull/290

Removed deprecated reference

XmlUpdateRequestHandler has been replaced by UpdateRequestHandler as of 
SOLR-2857

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/steeveb972/lucene-solr master

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/lucene-solr/pull/290.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #290


commit 43dece91ffdac55344ff2fbf65eb1207dff6bec7
Author: steeveb972 
Date:   2017-12-13T13:12:50Z

Removed deprecated reference

XmlUpdateRequestHandler has been replaced by UpdateRequestHandler




> Multi-content-type /update handler
> --
>
> Key: SOLR-2857
> URL: https://issues.apache.org/jira/browse/SOLR-2857
> Project: Solr
>  Issue Type: Improvement
>Reporter: Erik Hatcher
>Assignee: Ryan McKinley
> Fix For: 4.0-ALPHA
>
> Attachments: SOLR-2857-content-type-refactor.patch, 
> SOLR-2857-content-type-refactor.patch, SOLR-2857-content-type-refactor.patch, 
> SOLR-2857-update-content-type.patch, SOLR-2857-update-content-type.patch, 
> SOLR-2857-update-content-type.patch, SOLR-2857_oldUrls.patch
>
>
> Something I've been thinking about lately... it'd be great to get rid of all 
> the specific update handlers like /update/csv, /update/extract, and 
> /update/json and collapse them all into a single /update that underneath uses 
> the content-type(s) to hand off to specific content handlers.  This would 
> make it much easier to toss content at Solr and provide a single entry point 
> for updates.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-8011) Improve similarity explanations

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-8011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16288923#comment-16288923
 ] 

ASF GitHub Bot commented on LUCENE-8011:


Github user jpountz commented on the issue:

https://github.com/apache/lucene-solr/pull/280
  
Merged, thank you @mayya-sharipova.


> Improve similarity explanations
> ---
>
> Key: LUCENE-8011
> URL: https://issues.apache.org/jira/browse/LUCENE-8011
> Project: Lucene - Core
>  Issue Type: Improvement
>Reporter: Robert Muir
>  Labels: newdev
> Fix For: master (8.0)
>
>
> LUCENE-7997 improves BM25 and Classic explains to better explain:
> {noformat}
> product of:
>   2.2 = scaling factor, k1 + 1
>   9.388654 = idf, computed as log(1 + (N - n + 0.5) / (n + 0.5)) from:
> 1.0 = n, number of documents containing term
> 17927.0 = N, total number of documents with field
>   0.9987758 = tf, computed as freq / (freq + k1 * (1 - b + b * dl / avgdl)) 
> from:
> 979.0 = freq, occurrences of term within document
> 1.2 = k1, term saturation parameter
> 0.75 = b, length normalization parameter
> 1.0 = dl, length of field
> 1.0 = avgdl, average length of field
> {noformat}
> Previously it was pretty cryptic and used confusing terminology like 
> docCount/docFreq without explanation: 
> {noformat}
> product of:
>   0.016547536 = idf, computed as log(1 + (docCount - docFreq + 0.5) / 
> (docFreq + 0.5)) from:
> 449.0 = docFreq
> 456.0 = docCount
>   2.1920826 = tfNorm, computed as (freq * (k1 + 1)) / (freq + k1 * (1 - b + b 
> * fieldLength / avgFieldLength)) from:
> 113659.0 = freq=113658
> 1.2 = parameter k1
> 0.75 = parameter b
> 2300.5593 = avgFieldLength
> 1048600.0 = fieldLength
> {noformat}
> We should fix other similarities too in the same way, they should be more 
> practical.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-8011) Improve similarity explanations

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-8011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16288907#comment-16288907
 ] 

ASF GitHub Bot commented on LUCENE-8011:


Github user asfgit closed the pull request at:

https://github.com/apache/lucene-solr/pull/280


> Improve similarity explanations
> ---
>
> Key: LUCENE-8011
> URL: https://issues.apache.org/jira/browse/LUCENE-8011
> Project: Lucene - Core
>  Issue Type: Improvement
>Reporter: Robert Muir
>  Labels: newdev
>
> LUCENE-7997 improves BM25 and Classic explains to better explain:
> {noformat}
> product of:
>   2.2 = scaling factor, k1 + 1
>   9.388654 = idf, computed as log(1 + (N - n + 0.5) / (n + 0.5)) from:
> 1.0 = n, number of documents containing term
> 17927.0 = N, total number of documents with field
>   0.9987758 = tf, computed as freq / (freq + k1 * (1 - b + b * dl / avgdl)) 
> from:
> 979.0 = freq, occurrences of term within document
> 1.2 = k1, term saturation parameter
> 0.75 = b, length normalization parameter
> 1.0 = dl, length of field
> 1.0 = avgdl, average length of field
> {noformat}
> Previously it was pretty cryptic and used confusing terminology like 
> docCount/docFreq without explanation: 
> {noformat}
> product of:
>   0.016547536 = idf, computed as log(1 + (docCount - docFreq + 0.5) / 
> (docFreq + 0.5)) from:
> 449.0 = docFreq
> 456.0 = docCount
>   2.1920826 = tfNorm, computed as (freq * (k1 + 1)) / (freq + k1 * (1 - b + b 
> * fieldLength / avgFieldLength)) from:
> 113659.0 = freq=113658
> 1.2 = parameter k1
> 0.75 = parameter b
> 2300.5593 = avgFieldLength
> 1048600.0 = fieldLength
> {noformat}
> We should fix other similarities too in the same way, they should be more 
> practical.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-8075) Possible null pointer dereference in core/src/java/org/apache/lucene/codecs/blocktree/IntersectTermsEnum.java

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-8075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16288541#comment-16288541
 ] 

ASF GitHub Bot commented on LUCENE-8075:


Github user mikemccand commented on a diff in the pull request:

https://github.com/apache/lucene-solr/pull/286#discussion_r156538059
  
--- Diff: 
lucene/core/src/java/org/apache/lucene/codecs/blocktree/IntersectTermsEnum.java 
---
@@ -106,37 +106,37 @@ public IntersectTermsEnum(FieldReader fr, Automaton 
automaton, RunAutomaton runA
 if (fr.index == null) {
   fstReader = null;
--- End diff --

Sorry for the slow respons ehere @imgpulak and @jpountz but Adrien is 
right: `fr.index` can never be null anymore.  So I think we should change the 
code to `assert` it's never null and only do the `else` clause of the current 
`if` statement?


> Possible null pointer dereference in 
> core/src/java/org/apache/lucene/codecs/blocktree/IntersectTermsEnum.java
> -
>
> Key: LUCENE-8075
> URL: https://issues.apache.org/jira/browse/LUCENE-8075
> Project: Lucene - Core
>  Issue Type: Bug
>  Components: core/codecs
>Affects Versions: 7.1
>Reporter: Xiaoshan Sun
>  Labels: easyfix
>   Original Estimate: 10m
>  Remaining Estimate: 10m
>
> Possible null pointer dereference in 
> core/src/java/org/apache/lucene/codecs/blocktree/IntersectTermsEnum.java.
> at line 119. The fr.index may be NULL. This result is based on static 
> analysis tools and the details are shown below:
> *
> {code:java}
> 106: if (fr.index == null) {
> 107:  fstReader = null;  // fr.index is Known NULL here.
> } else {
>   fstReader = fr.index.getBytesReader();
> }
> // TODO: if the automaton is "smallish" we really
> // should use the terms index to seek at least to
> // the initial term and likely to subsequent terms
> // (or, maybe just fallback to ATE for such cases).
> // Else the seek cost of loading the frames will be
> // too costly.
> 119:final FST.Arc arc = fr.index.getFirstArc(arcs[0]); 
> //  fr.index is dereferenced here and fr.index can be NULL if 107 is arrived.
> {code}
> *
> It is not sure if fr.index can be NULL in runtime.
> We think it is reasonable to fix it by a test if fr.index is NULL and an 
> error handling.
> --
> Please Refer to "Trusted Operating System and System Assurance Working Group, 
> TCA, Institute of Software, Chinese Academy of Sciences" in the 
> acknowledgement if applicable.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-8011) Improve similarity explanations

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-8011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16288340#comment-16288340
 ] 

ASF GitHub Bot commented on LUCENE-8011:


Github user jpountz commented on the issue:

https://github.com/apache/lucene-solr/pull/280
  
No need to be sorry!


> Improve similarity explanations
> ---
>
> Key: LUCENE-8011
> URL: https://issues.apache.org/jira/browse/LUCENE-8011
> Project: Lucene - Core
>  Issue Type: Improvement
>Reporter: Robert Muir
>  Labels: newdev
>
> LUCENE-7997 improves BM25 and Classic explains to better explain:
> {noformat}
> product of:
>   2.2 = scaling factor, k1 + 1
>   9.388654 = idf, computed as log(1 + (N - n + 0.5) / (n + 0.5)) from:
> 1.0 = n, number of documents containing term
> 17927.0 = N, total number of documents with field
>   0.9987758 = tf, computed as freq / (freq + k1 * (1 - b + b * dl / avgdl)) 
> from:
> 979.0 = freq, occurrences of term within document
> 1.2 = k1, term saturation parameter
> 0.75 = b, length normalization parameter
> 1.0 = dl, length of field
> 1.0 = avgdl, average length of field
> {noformat}
> Previously it was pretty cryptic and used confusing terminology like 
> docCount/docFreq without explanation: 
> {noformat}
> product of:
>   0.016547536 = idf, computed as log(1 + (docCount - docFreq + 0.5) / 
> (docFreq + 0.5)) from:
> 449.0 = docFreq
> 456.0 = docCount
>   2.1920826 = tfNorm, computed as (freq * (k1 + 1)) / (freq + k1 * (1 - b + b 
> * fieldLength / avgFieldLength)) from:
> 113659.0 = freq=113658
> 1.2 = parameter k1
> 0.75 = parameter b
> 2300.5593 = avgFieldLength
> 1048600.0 = fieldLength
> {noformat}
> We should fix other similarities too in the same way, they should be more 
> practical.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-8011) Improve similarity explanations

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-8011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16288338#comment-16288338
 ] 

ASF GitHub Bot commented on LUCENE-8011:


Github user mayya-sharipova commented on the issue:

https://github.com/apache/lucene-solr/pull/280
  
@jpountz Adrien thanks for your help. Sorry, I will make sure to run `ant 
precommit` before committing next time. I have pushed another change to address 
this.


> Improve similarity explanations
> ---
>
> Key: LUCENE-8011
> URL: https://issues.apache.org/jira/browse/LUCENE-8011
> Project: Lucene - Core
>  Issue Type: Improvement
>Reporter: Robert Muir
>  Labels: newdev
>
> LUCENE-7997 improves BM25 and Classic explains to better explain:
> {noformat}
> product of:
>   2.2 = scaling factor, k1 + 1
>   9.388654 = idf, computed as log(1 + (N - n + 0.5) / (n + 0.5)) from:
> 1.0 = n, number of documents containing term
> 17927.0 = N, total number of documents with field
>   0.9987758 = tf, computed as freq / (freq + k1 * (1 - b + b * dl / avgdl)) 
> from:
> 979.0 = freq, occurrences of term within document
> 1.2 = k1, term saturation parameter
> 0.75 = b, length normalization parameter
> 1.0 = dl, length of field
> 1.0 = avgdl, average length of field
> {noformat}
> Previously it was pretty cryptic and used confusing terminology like 
> docCount/docFreq without explanation: 
> {noformat}
> product of:
>   0.016547536 = idf, computed as log(1 + (docCount - docFreq + 0.5) / 
> (docFreq + 0.5)) from:
> 449.0 = docFreq
> 456.0 = docCount
>   2.1920826 = tfNorm, computed as (freq * (k1 + 1)) / (freq + k1 * (1 - b + b 
> * fieldLength / avgFieldLength)) from:
> 113659.0 = freq=113658
> 1.2 = parameter k1
> 0.75 = parameter b
> 2300.5593 = avgFieldLength
> 1048600.0 = fieldLength
> {noformat}
> We should fix other similarities too in the same way, they should be more 
> practical.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-11711) distributed pivot & field facets can processes excessive docs unneccessarily due to internal mincount=0

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-11711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16287988#comment-16287988
 ] 

ASF GitHub Bot commented on SOLR-11711:
---

Github user HoustonPutman closed the pull request at:

https://github.com/apache/lucene-solr/pull/279


> distributed pivot & field facets can processes excessive docs unneccessarily 
> due to internal mincount=0
> ---
>
> Key: SOLR-11711
> URL: https://issues.apache.org/jira/browse/SOLR-11711
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: faceting
>Affects Versions: master (8.0)
>Reporter: Houston Putman
>Assignee: Hoss Man
>  Labels: pull-request-available
> Fix For: master (8.0), 7.3
>
>
> Currently while sending pivot facet requests to each shard, the 
> {{facet.pivot.mincount}} is set to {{0}} if the facet is sorted by count with 
> a specified limit > 0. However with a mincount of 0, the pivot facet will use 
> exponentially more wasted memory for every pivot field added. This is because 
> there will be a total of {{limit^(# of pivots)}} pivot values created in 
> memory, even though the vast majority of them will have counts of 0, and are 
> therefore useless.
> Imagine the scenario of a pivot facet with 3 levels, and 
> {{facet.limit=1000}}. There will be a billion pivot values created, and there 
> will almost definitely be nowhere near a billion pivot values with counts > 0.
> This likely due to the reasoning mentioned in [this comment in the original 
> distributed pivot facet 
> ticket|https://issues.apache.org/jira/browse/SOLR-2894?focusedCommentId=13979898].
>  Basically it was thought that the refinement code would need to know that a 
> count was 0 for a shard so that a refinement request wasn't sent to that 
> shard. However this is checked in the code, [in this part of the refinement 
> candidate 
> checking|https://github.com/apache/lucene-solr/blob/releases/lucene-solr/7.1.0/solr/core/src/java/org/apache/solr/handler/component/PivotFacetField.java#L275].
>  Therefore if the {{pivot.mincount}} was set to 1, the non-existent values 
> would either:
> * Not be known, because the {{facet.limit}} was smaller than the number of 
> facet values with positive counts. This isn't an issue, because they wouldn't 
> have been returned with {{pivot.mincount}} set to 0.
> * Would be known, because the {{facet.limit}} would be larger than the number 
> of facet values returned. therefore this conditional would return false 
> (since we are only talking about pivot facets sorted by count).
> The solution, is to use the same pivot mincount as would be used if no limit 
> was specified. 
> This also relates to a similar problem in field faceting that was "fixed" in 
> [SOLR-8988|https://issues.apache.org/jira/browse/SOLR-8988#13324]. The 
> solution was to add a flag, {{facet.distrib.mco}}, which would enable not 
> choosing a mincount of 0 when unnessesary. Since this flag can only increase 
> performance, and doesn't break any queries I have removed it as an option and 
> replaced the code to use the feature always. 
> There was one code change necessary to fix the MCO option, since the 
> refinement candidate selection logic had a bug. The bug only occured with a 
> minCount > 0 and limit > 0 specified. When a shard replied with less than the 
> limit requested, it would assume the next maximum count on that shard was the 
> {{mincount}}, where it would actually be the {{mincount-1}} (because a facet 
> value with a count of mincount would have been returned). Therefore the MCO 
> didn't cause any errors, but with a mincount of 1 the refinement logic always 
> assumed that the shard had more values with a count of 1.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-8011) Improve similarity explanations

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-8011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16287881#comment-16287881
 ] 

ASF GitHub Bot commented on LUCENE-8011:


Github user jpountz commented on the issue:

https://github.com/apache/lucene-solr/pull/280
  
Thanks @mayya-sharipova, this looks great. `ant precommit` complains from 
some missing docs (the build requires that all public/protected APIs have some 
minimal documentation), could you fix it?


> Improve similarity explanations
> ---
>
> Key: LUCENE-8011
> URL: https://issues.apache.org/jira/browse/LUCENE-8011
> Project: Lucene - Core
>  Issue Type: Improvement
>Reporter: Robert Muir
>  Labels: newdev
>
> LUCENE-7997 improves BM25 and Classic explains to better explain:
> {noformat}
> product of:
>   2.2 = scaling factor, k1 + 1
>   9.388654 = idf, computed as log(1 + (N - n + 0.5) / (n + 0.5)) from:
> 1.0 = n, number of documents containing term
> 17927.0 = N, total number of documents with field
>   0.9987758 = tf, computed as freq / (freq + k1 * (1 - b + b * dl / avgdl)) 
> from:
> 979.0 = freq, occurrences of term within document
> 1.2 = k1, term saturation parameter
> 0.75 = b, length normalization parameter
> 1.0 = dl, length of field
> 1.0 = avgdl, average length of field
> {noformat}
> Previously it was pretty cryptic and used confusing terminology like 
> docCount/docFreq without explanation: 
> {noformat}
> product of:
>   0.016547536 = idf, computed as log(1 + (docCount - docFreq + 0.5) / 
> (docFreq + 0.5)) from:
> 449.0 = docFreq
> 456.0 = docCount
>   2.1920826 = tfNorm, computed as (freq * (k1 + 1)) / (freq + k1 * (1 - b + b 
> * fieldLength / avgFieldLength)) from:
> 113659.0 = freq=113658
> 1.2 = parameter k1
> 0.75 = parameter b
> 2300.5593 = avgFieldLength
> 1048600.0 = fieldLength
> {noformat}
> We should fix other similarities too in the same way, they should be more 
> practical.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-8086) G3d wrapper: Improve circles for non spherical planets

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-8086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16287651#comment-16287651
 ] 

ASF GitHub Bot commented on LUCENE-8086:


Github user dsmiley commented on a diff in the pull request:

https://github.com/apache/lucene-solr/pull/288#discussion_r155938328
  
--- Diff: 
lucene/spatial-extras/src/java/org/apache/lucene/spatial/spatial4j/Geo3dShapeFactory.java
 ---
@@ -150,10 +176,21 @@ public Rectangle rect(double minX, double maxX, 
double minY, double maxY) {
 
   @Override
   public Circle circle(double x, double y, double distance) {
-GeoCircle circle = GeoCircleFactory.makeGeoCircle(planetModel,
-y * DistanceUtils.DEGREES_TO_RADIANS,
-x * DistanceUtils.DEGREES_TO_RADIANS,
-distance * DistanceUtils.DEGREES_TO_RADIANS);
+GeoCircle circle;
+if (planetModel.ab == planetModel.c) {
--- End diff --

@DaddyWri comments?


> G3d wrapper: Improve circles for non spherical planets
> --
>
> Key: LUCENE-8086
> URL: https://issues.apache.org/jira/browse/LUCENE-8086
> Project: Lucene - Core
>  Issue Type: Improvement
>  Components: modules/spatial-extras
>Reporter: Ignacio Vera
>
> Hi [~dsmiley],
> The purpose of this ticket is to add a new circle shape (GeoExactCircle) for 
> non-spherical planets and therefore remove the method relate from 
> Geo3dCircleShape. The patch will include some simplifications on the wrapper 
> and some refactoring of the tests.
> I will open shortly a pull request.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-8086) G3d wrapper: Improve circles for non spherical planets

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-8086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16287650#comment-16287650
 ] 

ASF GitHub Bot commented on LUCENE-8086:


Github user dsmiley commented on a diff in the pull request:

https://github.com/apache/lucene-solr/pull/288#discussion_r155938315
  
--- Diff: 
lucene/spatial-extras/src/test/org/apache/lucene/spatial/spatial4j/ShapeRectRelationTestCase.java
 ---
@@ -28,15 +28,15 @@
 
 import static 
org.locationtech.spatial4j.distance.DistanceUtils.DEGREES_TO_RADIANS;
 
-public abstract class Geo3dShapeRectRelationTestCase extends 
RandomizedShapeTestCase {
+public abstract class ShapeRectRelationTestCase extends 
RandomizedShapeTestCase {
   protected final static double RADIANS_PER_DEGREE = Math.PI/180.0;
 
   @Rule
   public final TestLog testLog = TestLog.instance;
 
   protected int maxRadius = 180;
 
-  public Geo3dShapeRectRelationTestCase() {
+  public ShapeRectRelationTestCase() {
 super(SpatialContext.GEO);
--- End diff --

I think the subclass should pass in the context.


> G3d wrapper: Improve circles for non spherical planets
> --
>
> Key: LUCENE-8086
> URL: https://issues.apache.org/jira/browse/LUCENE-8086
> Project: Lucene - Core
>  Issue Type: Improvement
>  Components: modules/spatial-extras
>Reporter: Ignacio Vera
>
> Hi [~dsmiley],
> The purpose of this ticket is to add a new circle shape (GeoExactCircle) for 
> non-spherical planets and therefore remove the method relate from 
> Geo3dCircleShape. The patch will include some simplifications on the wrapper 
> and some refactoring of the tests.
> I will open shortly a pull request.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-11745) SolrCore doesn't log core if too many closes called on it

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-11745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16286633#comment-16286633
 ] 

ASF GitHub Bot commented on SOLR-11745:
---

GitHub user millerjeff0 opened a pull request:

https://github.com/apache/lucene-solr/pull/289

SOLR-11745: change logging references to getName

SOLR-11745

this doesn't resolve to anything by object name, Eg: 
org.apache.solr.core.SolrCore@4812a0d7

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/millerjeff0/lucene-solr SOLR-11745

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/lucene-solr/pull/289.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #289


commit 6f87979e088b49ccb12d74af9a4352cb920665c9
Author: Jeff 
Date:   2017-12-11T21:54:42Z

SOLR-11745: change logging references to getName




> SolrCore doesn't log core if too many closes called on it
> -
>
> Key: SOLR-11745
> URL: https://issues.apache.org/jira/browse/SOLR-11745
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: logging
>Affects Versions: 7.1
>Reporter: Jeff Miller
>Priority: Trivial
>   Original Estimate: 5m
>  Remaining Estimate: 5m
>
>   log.error("Too many close [count:{}] on {}. Please report this 
> exception to solr-u...@lucene.apache.org", count, this );
> Calling this just prints
> org.apache.solr.core.SolrCore@4812a0d7
> Suggest changing to getName



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-8086) G3d wrapper: Improve circles for non spherical planets

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-8086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16285077#comment-16285077
 ] 

ASF GitHub Bot commented on LUCENE-8086:


Github user dsmiley commented on a diff in the pull request:

https://github.com/apache/lucene-solr/pull/288#discussion_r155938309
  
--- Diff: 
lucene/spatial-extras/src/test/org/apache/lucene/spatial/spatial4j/ShapeRectRelationTestCase.java
 ---
@@ -75,9 +75,9 @@ public void testGeoCircleRect() {
 new Geo3dRectIntersectionTestHelper(ctx) {
--- End diff --

Should be renamed as this class is no longer Geo3d specific


> G3d wrapper: Improve circles for non spherical planets
> --
>
> Key: LUCENE-8086
> URL: https://issues.apache.org/jira/browse/LUCENE-8086
> Project: Lucene - Core
>  Issue Type: Improvement
>  Components: modules/spatial-extras
>Reporter: Ignacio Vera
>
> Hi [~dsmiley],
> The purpose of this ticket is to add a new circle shape (GeoExactCircle) for 
> non-spherical planets and therefore remove the method relate from 
> Geo3dCircleShape. The patch will include some simplifications on the wrapper 
> and some refactoring of the tests.
> I will open shortly a pull request.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-8075) Possible null pointer dereference in core/src/java/org/apache/lucene/codecs/blocktree/IntersectTermsEnum.java

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-8075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16284972#comment-16284972
 ] 

ASF GitHub Bot commented on LUCENE-8075:


Github user imgpulak commented on a diff in the pull request:

https://github.com/apache/lucene-solr/pull/286#discussion_r155930642
  
--- Diff: 
lucene/core/src/java/org/apache/lucene/codecs/blocktree/IntersectTermsEnum.java 
---
@@ -106,37 +106,37 @@ public IntersectTermsEnum(FieldReader fr, Automaton 
automaton, RunAutomaton runA
 if (fr.index == null) {
   fstReader = null;
--- End diff --

@jpountz, @mikemccand any update here?


> Possible null pointer dereference in 
> core/src/java/org/apache/lucene/codecs/blocktree/IntersectTermsEnum.java
> -
>
> Key: LUCENE-8075
> URL: https://issues.apache.org/jira/browse/LUCENE-8075
> Project: Lucene - Core
>  Issue Type: Bug
>  Components: core/codecs
>Affects Versions: 7.1
>Reporter: Xiaoshan Sun
>  Labels: easyfix
>   Original Estimate: 10m
>  Remaining Estimate: 10m
>
> Possible null pointer dereference in 
> core/src/java/org/apache/lucene/codecs/blocktree/IntersectTermsEnum.java.
> at line 119. The fr.index may be NULL. This result is based on static 
> analysis tools and the details are shown below:
> *
> {code:java}
> 106: if (fr.index == null) {
> 107:  fstReader = null;  // fr.index is Known NULL here.
> } else {
>   fstReader = fr.index.getBytesReader();
> }
> // TODO: if the automaton is "smallish" we really
> // should use the terms index to seek at least to
> // the initial term and likely to subsequent terms
> // (or, maybe just fallback to ATE for such cases).
> // Else the seek cost of loading the frames will be
> // too costly.
> 119:final FST.Arc arc = fr.index.getFirstArc(arcs[0]); 
> //  fr.index is dereferenced here and fr.index can be NULL if 107 is arrived.
> {code}
> *
> It is not sure if fr.index can be NULL in runtime.
> We think it is reasonable to fix it by a test if fr.index is NULL and an 
> error handling.
> --
> Please Refer to "Trusted Operating System and System Assurance Working Group, 
> TCA, Institute of Software, Chinese Academy of Sciences" in the 
> acknowledgement if applicable.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-8086) G3d wrapper: Improve circles for non spherical planets

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-8086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16284004#comment-16284004
 ] 

ASF GitHub Bot commented on LUCENE-8086:


Github user iverase commented on a diff in the pull request:

https://github.com/apache/lucene-solr/pull/288#discussion_r155844958
  
--- Diff: 
lucene/spatial-extras/src/test/org/apache/lucene/spatial/spatial4j/Geo3dShapeRectRelationTestCase.java
 ---
@@ -16,29 +16,15 @@
  */
 package org.apache.lucene.spatial.spatial4j;
 
-import java.util.ArrayList;
-import java.util.List;
-
-import org.apache.lucene.spatial3d.geom.GeoPath;
-import org.apache.lucene.spatial3d.geom.GeoPolygon;
+import org.junit.Rule;
+import org.junit.Test;
 import org.locationtech.spatial4j.TestLog;
 import org.locationtech.spatial4j.context.SpatialContext;
-import org.locationtech.spatial4j.distance.DistanceUtils;
 import org.locationtech.spatial4j.shape.Circle;
 import org.locationtech.spatial4j.shape.Point;
 import org.locationtech.spatial4j.shape.RectIntersectionTestHelper;
-import org.apache.lucene.spatial3d.geom.LatLonBounds;
--- End diff --

I didn't dear to change it but that was the idea of the effort, I will 
remove Geo3d. 


> G3d wrapper: Improve circles for non spherical planets
> --
>
> Key: LUCENE-8086
> URL: https://issues.apache.org/jira/browse/LUCENE-8086
> Project: Lucene - Core
>  Issue Type: Improvement
>  Components: modules/spatial-extras
>Reporter: Ignacio Vera
>
> Hi [~dsmiley],
> The purpose of this ticket is to add a new circle shape (GeoExactCircle) for 
> non-spherical planets and therefore remove the method relate from 
> Geo3dCircleShape. The patch will include some simplifications on the wrapper 
> and some refactoring of the tests.
> I will open shortly a pull request.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-8086) G3d wrapper: Improve circles for non spherical planets

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-8086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16283996#comment-16283996
 ] 

ASF GitHub Bot commented on LUCENE-8086:


Github user iverase commented on a diff in the pull request:

https://github.com/apache/lucene-solr/pull/288#discussion_r155844229
  
--- Diff: 
lucene/spatial-extras/src/test/org/apache/lucene/spatial/spatial4j/Geo3dShapeRectRelationTestCase.java
 ---
@@ -155,16 +107,12 @@ protected Geo3dShape generateRandomShape(Point nearP) 
{
   ulhcPoint = lrhcPoint;
   lrhcPoint = temp;
 }
-final GeoBBox shape = GeoBBoxFactory.makeGeoBBox(planetModel, 
ulhcPoint.getY() * DEGREES_TO_RADIANS,
-lrhcPoint.getY() * DEGREES_TO_RADIANS,
-ulhcPoint.getX() * DEGREES_TO_RADIANS,
-lrhcPoint.getX() * DEGREES_TO_RADIANS);
-return new Geo3dShape(shape, ctx);
+return (Geo3dShape) ctx.getShapeFactory().rect(lrhcPoint, 
ulhcPoint);
--- End diff --

Indeed!


> G3d wrapper: Improve circles for non spherical planets
> --
>
> Key: LUCENE-8086
> URL: https://issues.apache.org/jira/browse/LUCENE-8086
> Project: Lucene - Core
>  Issue Type: Improvement
>  Components: modules/spatial-extras
>Reporter: Ignacio Vera
>
> Hi [~dsmiley],
> The purpose of this ticket is to add a new circle shape (GeoExactCircle) for 
> non-spherical planets and therefore remove the method relate from 
> Geo3dCircleShape. The patch will include some simplifications on the wrapper 
> and some refactoring of the tests.
> I will open shortly a pull request.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-8086) G3d wrapper: Improve circles for non spherical planets

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-8086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16283991#comment-16283991
 ] 

ASF GitHub Bot commented on LUCENE-8086:


Github user iverase commented on a diff in the pull request:

https://github.com/apache/lucene-solr/pull/288#discussion_r155843938
  
--- Diff: 
lucene/spatial-extras/src/java/org/apache/lucene/spatial/spatial4j/Geo3dShapeFactory.java
 ---
@@ -150,10 +176,21 @@ public Rectangle rect(double minX, double maxX, 
double minY, double maxY) {
 
   @Override
   public Circle circle(double x, double y, double distance) {
-GeoCircle circle = GeoCircleFactory.makeGeoCircle(planetModel,
-y * DistanceUtils.DEGREES_TO_RADIANS,
-x * DistanceUtils.DEGREES_TO_RADIANS,
-distance * DistanceUtils.DEGREES_TO_RADIANS);
+GeoCircle circle;
+if (planetModel.ab == planetModel.c) {
--- End diff --

I think spatial3d is a low level library in that respect, so it shouldn't 
have such a method. Karl Wright has the last word, comment would be fine   


> G3d wrapper: Improve circles for non spherical planets
> --
>
> Key: LUCENE-8086
> URL: https://issues.apache.org/jira/browse/LUCENE-8086
> Project: Lucene - Core
>  Issue Type: Improvement
>  Components: modules/spatial-extras
>Reporter: Ignacio Vera
>
> Hi [~dsmiley],
> The purpose of this ticket is to add a new circle shape (GeoExactCircle) for 
> non-spherical planets and therefore remove the method relate from 
> Geo3dCircleShape. The patch will include some simplifications on the wrapper 
> and some refactoring of the tests.
> I will open shortly a pull request.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-8086) G3d wrapper: Improve circles for non spherical planets

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-8086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16283993#comment-16283993
 ] 

ASF GitHub Bot commented on LUCENE-8086:


Github user iverase commented on a diff in the pull request:

https://github.com/apache/lucene-solr/pull/288#discussion_r155844074
  
--- Diff: 
lucene/spatial-extras/src/java/org/apache/lucene/spatial/spatial4j/Geo3dShapeFactory.java
 ---
@@ -55,6 +55,13 @@
   private SpatialContext context;
   private PlanetModel planetModel;
 
+  /**
+   * Default accuracy for circles when not using the unit sphere.
+   * It is equivalent to 10m on the surface of the earth.
+   */
+  private static double DEFAULT_CIRCLE_ACCURACY = 1.6e-6;
--- End diff --

Indeed!


> G3d wrapper: Improve circles for non spherical planets
> --
>
> Key: LUCENE-8086
> URL: https://issues.apache.org/jira/browse/LUCENE-8086
> Project: Lucene - Core
>  Issue Type: Improvement
>  Components: modules/spatial-extras
>Reporter: Ignacio Vera
>
> Hi [~dsmiley],
> The purpose of this ticket is to add a new circle shape (GeoExactCircle) for 
> non-spherical planets and therefore remove the method relate from 
> Geo3dCircleShape. The patch will include some simplifications on the wrapper 
> and some refactoring of the tests.
> I will open shortly a pull request.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-8086) G3d wrapper: Improve circles for non spherical planets

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-8086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16283985#comment-16283985
 ] 

ASF GitHub Bot commented on LUCENE-8086:


Github user iverase commented on a diff in the pull request:

https://github.com/apache/lucene-solr/pull/288#discussion_r155843390
  
--- Diff: 
lucene/spatial-extras/src/java/org/apache/lucene/spatial/spatial4j/Geo3dShapeFactory.java
 ---
@@ -67,6 +74,25 @@ public SpatialContext getSpatialContext() {
 return context;
   }
 
+  /**
+   * Set the accuracy for circles.
+   *
+   * "Accuracy" is defined as the maximum linear distance between any 
point on the
+   * surface circle and planes that describe the circle. Therefore on 
WSG84, since the
+   * radius of earth is 6,371,000 meters, an accuracy of 1e-6 corresponds 
to 6.3 meters.
+   * For an accuracy of 1.0 meters, the value of 1.6e-7.
+   *
+   * The default value is set to 10m (1.6e-6).
+   *
+   * Note that accuracy has no effect when the planet model is a sphere. 
In that case circles
+   * are always fully precise.
+   *
+   * @param circleAccuracy the provided accuracy as a linear distance.
--- End diff --

I need to ask Karl Wright if that is what it means, but I guess so. I will 
update accordingly.


> G3d wrapper: Improve circles for non spherical planets
> --
>
> Key: LUCENE-8086
> URL: https://issues.apache.org/jira/browse/LUCENE-8086
> Project: Lucene - Core
>  Issue Type: Improvement
>  Components: modules/spatial-extras
>Reporter: Ignacio Vera
>
> Hi [~dsmiley],
> The purpose of this ticket is to add a new circle shape (GeoExactCircle) for 
> non-spherical planets and therefore remove the method relate from 
> Geo3dCircleShape. The patch will include some simplifications on the wrapper 
> and some refactoring of the tests.
> I will open shortly a pull request.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-8086) G3d wrapper: Improve circles for non spherical planets

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-8086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16283854#comment-16283854
 ] 

ASF GitHub Bot commented on LUCENE-8086:


Github user dsmiley commented on a diff in the pull request:

https://github.com/apache/lucene-solr/pull/288#discussion_r155822251
  
--- Diff: 
lucene/spatial-extras/src/java/org/apache/lucene/spatial/spatial4j/Geo3dShapeFactory.java
 ---
@@ -67,6 +74,25 @@ public SpatialContext getSpatialContext() {
 return context;
   }
 
+  /**
+   * Set the accuracy for circles.
+   *
+   * "Accuracy" is defined as the maximum linear distance between any 
point on the
+   * surface circle and planes that describe the circle. Therefore on 
WSG84, since the
+   * radius of earth is 6,371,000 meters, an accuracy of 1e-6 corresponds 
to 6.3 meters.
+   * For an accuracy of 1.0 meters, the value of 1.6e-7.
+   *
+   * The default value is set to 10m (1.6e-6).
+   *
+   * Note that accuracy has no effect when the planet model is a sphere. 
In that case circles
+   * are always fully precise.
+   *
+   * @param circleAccuracy the provided accuracy as a linear distance.
--- End diff --

by "linear distance" do you mean decimal degrees as is used in other parts 
of the Spatial4j API? If so please say "decimal degrees".  If not, perhaps it 
should be in that unit?


> G3d wrapper: Improve circles for non spherical planets
> --
>
> Key: LUCENE-8086
> URL: https://issues.apache.org/jira/browse/LUCENE-8086
> Project: Lucene - Core
>  Issue Type: Improvement
>  Components: modules/spatial-extras
>Reporter: Ignacio Vera
>
> Hi [~dsmiley],
> The purpose of this ticket is to add a new circle shape (GeoExactCircle) for 
> non-spherical planets and therefore remove the method relate from 
> Geo3dCircleShape. The patch will include some simplifications on the wrapper 
> and some refactoring of the tests.
> I will open shortly a pull request.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-8086) G3d wrapper: Improve circles for non spherical planets

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-8086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16283853#comment-16283853
 ] 

ASF GitHub Bot commented on LUCENE-8086:


Github user dsmiley commented on a diff in the pull request:

https://github.com/apache/lucene-solr/pull/288#discussion_r155825386
  
--- Diff: 
lucene/spatial-extras/src/test/org/apache/lucene/spatial/spatial4j/Geo3dShapeRectRelationTestCase.java
 ---
@@ -16,29 +16,15 @@
  */
 package org.apache.lucene.spatial.spatial4j;
 
-import java.util.ArrayList;
-import java.util.List;
-
-import org.apache.lucene.spatial3d.geom.GeoPath;
-import org.apache.lucene.spatial3d.geom.GeoPolygon;
+import org.junit.Rule;
+import org.junit.Test;
 import org.locationtech.spatial4j.TestLog;
 import org.locationtech.spatial4j.context.SpatialContext;
-import org.locationtech.spatial4j.distance.DistanceUtils;
 import org.locationtech.spatial4j.shape.Circle;
 import org.locationtech.spatial4j.shape.Point;
 import org.locationtech.spatial4j.shape.RectIntersectionTestHelper;
-import org.apache.lucene.spatial3d.geom.LatLonBounds;
--- End diff --

If I get this right, you've removed the Geo3D dependencies of this test.  
Yet it's still named to be Geo3d?


> G3d wrapper: Improve circles for non spherical planets
> --
>
> Key: LUCENE-8086
> URL: https://issues.apache.org/jira/browse/LUCENE-8086
> Project: Lucene - Core
>  Issue Type: Improvement
>  Components: modules/spatial-extras
>Reporter: Ignacio Vera
>
> Hi [~dsmiley],
> The purpose of this ticket is to add a new circle shape (GeoExactCircle) for 
> non-spherical planets and therefore remove the method relate from 
> Geo3dCircleShape. The patch will include some simplifications on the wrapper 
> and some refactoring of the tests.
> I will open shortly a pull request.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-8086) G3d wrapper: Improve circles for non spherical planets

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-8086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16283852#comment-16283852
 ] 

ASF GitHub Bot commented on LUCENE-8086:


Github user dsmiley commented on a diff in the pull request:

https://github.com/apache/lucene-solr/pull/288#discussion_r155826444
  
--- Diff: 
lucene/spatial-extras/src/java/org/apache/lucene/spatial/spatial4j/Geo3dDistanceCalculator.java
 ---
@@ -73,62 +74,20 @@ public boolean within(Point from, double toX, double 
toY, double distance) {
 
   @Override
   public Point pointOnBearing(Point from, double distDEG, double 
bearingDEG, SpatialContext ctx, Point reuse) {
-// Algorithm using Vincenty's formulae 
(https://en.wikipedia.org/wiki/Vincenty%27s_formulae)
-// which takes into account that planets may not be spherical.
-//Code adaptation from 
http://www.movable-type.co.uk/scripts/latlong-vincenty.html
 Geo3dPointShape geoFrom = (Geo3dPointShape) from;
 GeoPoint point = (GeoPoint) geoFrom.shape;
-double lat = point.getLatitude();
-double lon = point.getLongitude();
 double dist = DistanceUtils.DEGREES_TO_RADIANS * distDEG;
 double bearing = DistanceUtils.DEGREES_TO_RADIANS * bearingDEG;
-
--- End diff --

Yay


> G3d wrapper: Improve circles for non spherical planets
> --
>
> Key: LUCENE-8086
> URL: https://issues.apache.org/jira/browse/LUCENE-8086
> Project: Lucene - Core
>  Issue Type: Improvement
>  Components: modules/spatial-extras
>Reporter: Ignacio Vera
>
> Hi [~dsmiley],
> The purpose of this ticket is to add a new circle shape (GeoExactCircle) for 
> non-spherical planets and therefore remove the method relate from 
> Geo3dCircleShape. The patch will include some simplifications on the wrapper 
> and some refactoring of the tests.
> I will open shortly a pull request.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-8086) G3d wrapper: Improve circles for non spherical planets

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-8086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16283848#comment-16283848
 ] 

ASF GitHub Bot commented on LUCENE-8086:


Github user dsmiley commented on a diff in the pull request:

https://github.com/apache/lucene-solr/pull/288#discussion_r155821440
  
--- Diff: 
lucene/spatial-extras/src/java/org/apache/lucene/spatial/spatial4j/Geo3dShapeFactory.java
 ---
@@ -55,6 +55,13 @@
   private SpatialContext context;
   private PlanetModel planetModel;
 
+  /**
+   * Default accuracy for circles when not using the unit sphere.
+   * It is equivalent to 10m on the surface of the earth.
+   */
+  private static double DEFAULT_CIRCLE_ACCURACY = 1.6e-6;
--- End diff --

should be final


> G3d wrapper: Improve circles for non spherical planets
> --
>
> Key: LUCENE-8086
> URL: https://issues.apache.org/jira/browse/LUCENE-8086
> Project: Lucene - Core
>  Issue Type: Improvement
>  Components: modules/spatial-extras
>Reporter: Ignacio Vera
>
> Hi [~dsmiley],
> The purpose of this ticket is to add a new circle shape (GeoExactCircle) for 
> non-spherical planets and therefore remove the method relate from 
> Geo3dCircleShape. The patch will include some simplifications on the wrapper 
> and some refactoring of the tests.
> I will open shortly a pull request.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-8086) G3d wrapper: Improve circles for non spherical planets

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-8086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16283851#comment-16283851
 ] 

ASF GitHub Bot commented on LUCENE-8086:


Github user dsmiley commented on a diff in the pull request:

https://github.com/apache/lucene-solr/pull/288#discussion_r155824604
  
--- Diff: 
lucene/spatial-extras/src/test/org/apache/lucene/spatial/spatial4j/Geo3dShapeRectRelationTestCase.java
 ---
@@ -155,16 +107,12 @@ protected Geo3dShape generateRandomShape(Point nearP) 
{
   ulhcPoint = lrhcPoint;
   lrhcPoint = temp;
 }
-final GeoBBox shape = GeoBBoxFactory.makeGeoBBox(planetModel, 
ulhcPoint.getY() * DEGREES_TO_RADIANS,
-lrhcPoint.getY() * DEGREES_TO_RADIANS,
-ulhcPoint.getX() * DEGREES_TO_RADIANS,
-lrhcPoint.getX() * DEGREES_TO_RADIANS);
-return new Geo3dShape(shape, ctx);
+return (Geo3dShape) ctx.getShapeFactory().rect(lrhcPoint, 
ulhcPoint);
--- End diff --

change is good but the variable names are wrong.  `rect(lowerLeft, 
upperRight)`


> G3d wrapper: Improve circles for non spherical planets
> --
>
> Key: LUCENE-8086
> URL: https://issues.apache.org/jira/browse/LUCENE-8086
> Project: Lucene - Core
>  Issue Type: Improvement
>  Components: modules/spatial-extras
>Reporter: Ignacio Vera
>
> Hi [~dsmiley],
> The purpose of this ticket is to add a new circle shape (GeoExactCircle) for 
> non-spherical planets and therefore remove the method relate from 
> Geo3dCircleShape. The patch will include some simplifications on the wrapper 
> and some refactoring of the tests.
> I will open shortly a pull request.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org