[jira] [Commented] (SOLR-4735) Improve Solr metrics reporting
[ https://issues.apache.org/jira/browse/SOLR-4735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15731809#comment-15731809 ] Ramkumar Aiyengar commented on SOLR-4735: - Didn't Shalin already add a /admin/metrics endpoint for reporting metrics? > Improve Solr metrics reporting > -- > > Key: SOLR-4735 > URL: https://issues.apache.org/jira/browse/SOLR-4735 > Project: Solr > Issue Type: Improvement >Reporter: Alan Woodward >Assignee: Andrzej Bialecki >Priority: Minor > Attachments: SOLR-4735.patch, SOLR-4735.patch, SOLR-4735.patch, > SOLR-4735.patch, screenshot-1.png > > > Following on from a discussion on the mailing list: > http://search-lucene.com/m/IO0EI1qdyJF1/codahale=Solr+metrics+in+Codahale+metrics+and+Graphite+ > It would be good to make Solr play more nicely with existing devops > monitoring systems, such as Graphite or Ganglia. Stats monitoring at the > moment is poll-only, either via JMX or through the admin stats page. I'd > like to refactor things a bit to make this more pluggable. > This patch is a start. It adds a new interface, InstrumentedBean, which > extends SolrInfoMBean to return a > [[Metrics|http://metrics.codahale.com/manual/core/]] MetricRegistry, and a > couple of MetricReporters (which basically just duplicate the JMX and admin > page reporting that's there at the moment, but which should be more > extensible). The patch includes a change to RequestHandlerBase showing how > this could work. The idea would be to eventually replace the getStatistics() > call on SolrInfoMBean with this instead. > The next step would be to allow more MetricReporters to be defined in > solrconfig.xml. The Metrics library comes with ganglia and graphite > reporting modules, and we can add contrib plugins for both of those. > There's some more general cleanup that could be done around SolrInfoMBean > (we've got two plugin handlers at /mbeans and /plugins that basically do the > same thing, and the beans themselves have some weirdly inconsistent data on > them - getVersion() returns different things for different impls, and > getSource() seems pretty useless), but maybe that's for another issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-8946) bin/post script stdin detection is faulty
[
https://issues.apache.org/jira/browse/SOLR-8946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15227043#comment-15227043
]
Ramkumar Aiyengar commented on SOLR-8946:
-
+1
> bin/post script stdin detection is faulty
> -
>
> Key: SOLR-8946
> URL: https://issues.apache.org/jira/browse/SOLR-8946
> Project: Solr
> Issue Type: Bug
> Components: SimplePostTool
>Affects Versions: 6.0
> Environment: Ubuntu
>Reporter: David Smiley
>
> Summary: I think the {{post}} bash script tries to detect stdin the wrong
> way.
> If I run "post -h" I see some examples, to include this:
> {code}
> echo '{commit: {}}' | bin/post -c my_collection -type application/json -out
> yes -d
> {code}
> However if I try it, Solr returns this error: {{Cannot parse provided JSON:
> JSON Parse Error: char=<,position=0 BEFORE='<' AFTER='add/>'}}
> I see that {{post}} is sending an {{}} in certain cases, but it's not
> supposed to when the mode is stdin. Looking closer, it appears to be because
> of this line:
> {{if [[ -s /dev/stdin ]]; then}}
> If I change it to this (which I figured out after googling around; I'm no
> bash expert!):
> {{if [[ ! -t 0 ]]; then}}
> Apparently '0' represents stdin. It works for me. [~thelabdude] might you
> take a look? I'd be happy to add a formal patch & commit for this trivial
> change but it really needs other eyes.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-6359) Allow customization of the number of records and logs kept by UpdateLog
[
https://issues.apache.org/jira/browse/SOLR-6359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15213679#comment-15213679
]
Ramkumar Aiyengar commented on SOLR-6359:
-
Yeah, you are right that you get better control by tweaking one vs the other.
With num logs, you can get an approximation to amount of time for which you
keep logs (ie number of commits).. I agree that's not exact, and it would be
good to have the time itself as a config option.
.
The other option is to leave the num logs at unlimited, and tweak the number of
records, which probably helps to protect against going berserk with log sizes
if your indexing rates vary wildly. We do this in our setups, ie leave the
number of files unlimited, while set the records to keep..
> Allow customization of the number of records and logs kept by UpdateLog
> ---
>
> Key: SOLR-6359
> URL: https://issues.apache.org/jira/browse/SOLR-6359
> Project: Solr
> Issue Type: Improvement
>Reporter: Ramkumar Aiyengar
>Assignee: Ramkumar Aiyengar
>Priority: Minor
> Fix For: 5.1, master
>
> Attachments: SOLR-6359.patch
>
>
> Currently {{UpdateLog}} hardcodes the number of logs and records it keeps,
> and the hardcoded numbers (100 records, 10 logs) can be quite low (esp. the
> records) in an heavily indexing setup, leading to full recovery even if Solr
> was just stopped and restarted.
> These values should be customizable (even if only present as expert options).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-8371) Try and prevent too many recovery requests from stacking up and clean up some faulty logic.
[ https://issues.apache.org/jira/browse/SOLR-8371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15146242#comment-15146242 ] Ramkumar Aiyengar commented on SOLR-8371: - Sorry Mark, I completely missed your comments on my patch. The intent there was to have at least one in the queue after the currently processing one (hence a fixed queue of size 1), so yes, there will be at least one more attempt after the last request. And any other attempt will just be rejected. But as I said, I haven't tested it -- I know the ArrayBlockingQueue will block any queueing when it hits capacity, but the ThreadpoolExecutor docs say that when a bound queue size is hit, an exception is thrown. The exception was my intention (so all jobs won't be run), perhaps testing will show otherwise.. > Try and prevent too many recovery requests from stacking up and clean up some > faulty logic. > --- > > Key: SOLR-8371 > URL: https://issues.apache.org/jira/browse/SOLR-8371 > Project: Solr > Issue Type: Improvement > Components: SolrCloud >Reporter: Mark Miller >Assignee: Mark Miller > Fix For: 5.5, master > > Attachments: SOLR-8371-2.patch, SOLR-8371.patch, SOLR-8371.patch, > SOLR-8371.patch, SOLR-8371.patch, SOLR-8371.patch, SOLR-8371.patch, > SOLR-8371.patch, SOLR-8371.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-8619) A new replica should not become leader when all current replicas are down as it leads to data loss
[ https://issues.apache.org/jira/browse/SOLR-8619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15124331#comment-15124331 ] Ramkumar Aiyengar commented on SOLR-8619: - Typical cloud issues lead to all replicas of a shard going to recovery/down/recovery_failed, and the only way is to cold start it by shutting all down and bringing them back up. Will checking lastPublished for ACTIVE interfere with that? An another cue which can be useful is the index generation being zero.. > A new replica should not become leader when all current replicas are down as > it leads to data loss > -- > > Key: SOLR-8619 > URL: https://issues.apache.org/jira/browse/SOLR-8619 > Project: Solr > Issue Type: Bug >Reporter: Anshum Gupta > > Here's what I'm talking about: > * Start a 2 node solrcloud cluster > * Create a 1 shard/1 replica collection > * Add documents > * Shut down the node that has the only active shard > * ADDREPLICA for the shard/collection, so Solr would attempt to add a new > replica on the other node > * Solr waits for a while before this replica becomes an active leader. > * Index a few new docs > * Bring up the old node > * The replica comes up, with it's old index and then syncs to only contain > the docs from the new leader. > All old documents are lost in this case > Here are a few things that might work here: > 1. Reject an ADDREPLICA call if all current replicas for the shard are down. > Considering the new replica can not sync from anyone, it doesn't make sense > for this replica to even come up > 2. The replica shouldn't become active/leader unless either it was the last > known leader or active before it went into recovering state > unless there are no other replicas in the clusterstate. > This might very well be related to SOLR-8173 but we should add a check to > ADDREPLICA as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Reopened] (SOLR-8418) BoostQuery cannot be cast to TermQuery
[
https://issues.apache.org/jira/browse/SOLR-8418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ramkumar Aiyengar reopened SOLR-8418:
-
Reopening to backport to 5.4.1
> BoostQuery cannot be cast to TermQuery
> --
>
> Key: SOLR-8418
> URL: https://issues.apache.org/jira/browse/SOLR-8418
> Project: Solr
> Issue Type: Bug
> Components: MoreLikeThis
>Affects Versions: 5.4
>Reporter: Jens Wille
>Assignee: Ramkumar Aiyengar
> Fix For: 5.5, Trunk
>
> Attachments: SOLR-8418.patch, SOLR-8418.patch
>
>
> As a consequence of LUCENE-6590, {{MoreLikeThisHandler}} was changed in
> r1701621 to use the new API. In SOLR-7912, I adapted that code for
> {{CloudMLTQParser}} and {{SimpleMLTQParser}}. However, setting the {{boost}}
> parameter just failed for me after updating to 5.4 with the following error
> message:
> {code}
> java.lang.ClassCastException: org.apache.lucene.search.BoostQuery cannot be
> cast to org.apache.lucene.search.TermQuery
> at
> org.apache.solr.search.mlt.SimpleMLTQParser.parse(SimpleMLTQParser.java:139)
> at org.apache.solr.search.QParser.getQuery(QParser.java:141)
> at
> org.apache.solr.handler.component.QueryComponent.prepare(QueryComponent.java:160)
> at
> org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:254)
> at
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:156)
> at org.apache.solr.core.SolrCore.execute(SolrCore.java:2073)
> at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:658)
> at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:457)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:222)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:181)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
> at
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
> at
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577)
> at
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
> at
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
> at
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
> at
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215)
> at
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110)
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
> at org.eclipse.jetty.server.Server.handle(Server.java:499)
> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
> at
> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
> at
> org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
> at java.lang.Thread.run(Thread.java:745)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Resolved] (SOLR-8418) BoostQuery cannot be cast to TermQuery
[
https://issues.apache.org/jira/browse/SOLR-8418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ramkumar Aiyengar resolved SOLR-8418.
-
Resolution: Fixed
Fix Version/s: (was: 5.5)
5.4.1
> BoostQuery cannot be cast to TermQuery
> --
>
> Key: SOLR-8418
> URL: https://issues.apache.org/jira/browse/SOLR-8418
> Project: Solr
> Issue Type: Bug
> Components: MoreLikeThis
>Affects Versions: 5.4
>Reporter: Jens Wille
>Assignee: Ramkumar Aiyengar
> Fix For: Trunk, 5.4.1
>
> Attachments: SOLR-8418.patch, SOLR-8418.patch
>
>
> As a consequence of LUCENE-6590, {{MoreLikeThisHandler}} was changed in
> r1701621 to use the new API. In SOLR-7912, I adapted that code for
> {{CloudMLTQParser}} and {{SimpleMLTQParser}}. However, setting the {{boost}}
> parameter just failed for me after updating to 5.4 with the following error
> message:
> {code}
> java.lang.ClassCastException: org.apache.lucene.search.BoostQuery cannot be
> cast to org.apache.lucene.search.TermQuery
> at
> org.apache.solr.search.mlt.SimpleMLTQParser.parse(SimpleMLTQParser.java:139)
> at org.apache.solr.search.QParser.getQuery(QParser.java:141)
> at
> org.apache.solr.handler.component.QueryComponent.prepare(QueryComponent.java:160)
> at
> org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:254)
> at
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:156)
> at org.apache.solr.core.SolrCore.execute(SolrCore.java:2073)
> at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:658)
> at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:457)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:222)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:181)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
> at
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
> at
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577)
> at
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
> at
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
> at
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
> at
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215)
> at
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110)
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
> at org.eclipse.jetty.server.Server.handle(Server.java:499)
> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
> at
> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
> at
> org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
> at java.lang.Thread.run(Thread.java:745)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-8539) Solr queries swallows up OutOfMemoryErrors
[
https://issues.apache.org/jira/browse/SOLR-8539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15098769#comment-15098769
]
Ramkumar Aiyengar commented on SOLR-8539:
-
I don't think there is a sensible way to "limp along"? Even if you did so and
returned 5xx errors, you aren't doing any favours, without any other special
handling, you pretty much quite probably would just be returning that over and
over again. Or worse, find weird bugs in application code which potentially do
more damage than a simple exit. With an exit due to an exception, at least any
decent process manager would respawn it, or a load balancer reroute requests
elsewhere.. I would suggest not handling Errors in general, unless there are
specific Errors you can whitelist as safe to trap..
> Solr queries swallows up OutOfMemoryErrors
> --
>
> Key: SOLR-8539
> URL: https://issues.apache.org/jira/browse/SOLR-8539
> Project: Solr
> Issue Type: Bug
>Reporter: Varun Thacker
> Fix For: 5.5, Trunk
>
> Attachments: SOLR-8539.patch
>
>
> I was testing a crazy surround query and was hitting OOMs easily with the
> query. However I saw that the OOM killer wasn't triggered. Here is the stack
> trace of the error on solr 5.4:
> {code}
> WARN - 2016-01-12 18:37:03.920; [ x:techproducts]
> org.eclipse.jetty.util.thread.QueuedThreadPool$3;
> java.lang.OutOfMemoryError: Java heap space
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.addConditionWaiter(AbstractQueuedSynchronizer.java:1855)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2068)
> at
> org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590)
> at java.lang.Thread.run(Thread.java:745)
> ERROR - 2016-01-12 18:37:03.922; [ x:techproducts]
> org.apache.solr.common.SolrException; null:java.lang.RuntimeException:
> java.lang.OutOfMemoryError: Java heap space
> at
> org.apache.solr.servlet.HttpSolrCall.sendError(HttpSolrCall.java:611)
> at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:472)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:222)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:181)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
> at
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
> at
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577)
> at
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
> at
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
> at
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
> at
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215)
> at
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110)
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
> at org.eclipse.jetty.server.Server.handle(Server.java:499)
> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
> at
> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
> at
> org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.OutOfMemoryError: Java heap space
> at
> org.apache.lucene.codecs.lucene50.Lucene50PostingsReader.newTermState(Lucene50PostingsReader.java:149)
> at
> org.apache.lucene.codecs.blocktree.SegmentTermsEnumFrame.(SegmentTermsEnumFrame.java:100)
> at
>
[jira] [Commented] (SOLR-7683) Introduce support to identify Solr internal request types
[
https://issues.apache.org/jira/browse/SOLR-7683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15091751#comment-15091751
]
Ramkumar Aiyengar commented on SOLR-7683:
-
Thanks [~hgadre]..
bq. In case of DirectXmlRequest, we don't know the actual type of request until
after the object is created (and hence the value can not be hardcoded). Also
since existing clients/applications may be using this class, it may not a good
idea to enforce the requirement to specify a type. Hence may be we should keep
"Unspecified" type for backwards compatibility?
We should keep backward compatibility for 5x, but could break it in trunk if it
takes us forward. [~ryantxu], [~shalinmangar], this seems to be hardly
used/touched in the last eight years (asking since you guys seem to have dealt
with it years back), and is only internally used by DIH -- but is a public
interface. Is there any value in keeping it any longer? If not, I would propose
just repurposing it as {{DIHRequest}} in trunk, else we could add a type..
bq. The other thing is we need to make sure that for server <-> server
communication, we are using the "server" context. Since we are creating
HttpSolrClient instances at number of places, we need to figure out a good way
to implement this. I propose that we should use a factory pattern for this.
i.e. we should define two factory implementations each dedicated for a specific
use-case (e.g. as a client or as a server) and refactor the code to use them
appropriately. This may involve considerable amount of refactoring. But the
resulting code would be clean and extensible for future requirements.
+1
> Introduce support to identify Solr internal request types
> -
>
> Key: SOLR-7683
> URL: https://issues.apache.org/jira/browse/SOLR-7683
> Project: Solr
> Issue Type: Sub-task
> Components: SolrCloud
>Reporter: Hrishikesh Gadre
>Assignee: Ramkumar Aiyengar
> Attachments: SOLR-7683.patch, SOLR-7683.patch
>
>
> SOLR-7344 is introducing support to partition the Jetty worker pool to
> enforce the number of concurrent requests for various types (e.g.
> Internal_Querying, Internal_Indexing, External etc.). For this we need to
> identify requests sent between Solr servers and their types (i.e.
> Querying/Indexing etc.).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-8488) Add support for leading wildcards to ComplexPhraseQParserPlugin
[
https://issues.apache.org/jira/browse/SOLR-8488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15088100#comment-15088100
]
Ramkumar Aiyengar commented on SOLR-8488:
-
I get that consistency with the normal phrase queries will be good, do keep in
mind that cpq makes it even easier to shoot yourself in your foot, "*a *b *c"
for example. Also enabling it by default would also be a compatibility break
for 5x at least.
For trunk, fine, I am not too opinionated about the default, as long as there's
an option to block it. And if we are looking to be consistent, should other
parsers also honor this option?
> Add support for leading wildcards to ComplexPhraseQParserPlugin
> ---
>
> Key: SOLR-8488
> URL: https://issues.apache.org/jira/browse/SOLR-8488
> Project: Solr
> Issue Type: Improvement
>Reporter: Varun Thacker
>Priority: Minor
> Fix For: 5.5, Trunk
>
> Attachments: SOLR-8488.patch, SOLR-8488.patch
>
>
> It would be useful to support leading wildcards in phrase searches as well.
> Currently we support this query -
> {code}{!complexphrase inOrder=true}name:"Jo* Smith"{code}
> It would be useful to be support a query like -
> {code}{!complexphrase inOrder=true\}name:"*Jo* Smith"{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-8488) Add support for leading wildcards to ComplexPhraseQParserPlugin
[
https://issues.apache.org/jira/browse/SOLR-8488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15085188#comment-15085188
]
Ramkumar Aiyengar commented on SOLR-8488:
-
Functionally looks okay, but this should be opted in. Leading wildcards can be
very costly and the user should consciously opt in to allow them..
> Add support for leading wildcards to ComplexPhraseQParserPlugin
> ---
>
> Key: SOLR-8488
> URL: https://issues.apache.org/jira/browse/SOLR-8488
> Project: Solr
> Issue Type: Improvement
>Reporter: Varun Thacker
>Priority: Minor
> Fix For: 5.5, Trunk
>
> Attachments: SOLR-8488.patch
>
>
> It would be useful to support leading wildcards in phrase searches as well.
> Currently we support this query -
> {code}{!complexphrase inOrder=true}name:"Jo* Smith"{code}
> It would be useful to be support a query like -
> {code}{!complexphrase inOrder=true\}name:"*Jo* Smith"{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-8418) BoostQuery cannot be cast to TermQuery
[
https://issues.apache.org/jira/browse/SOLR-8418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15082866#comment-15082866
]
Ramkumar Aiyengar commented on SOLR-8418:
-
Cool, thanks for confirming!
> BoostQuery cannot be cast to TermQuery
> --
>
> Key: SOLR-8418
> URL: https://issues.apache.org/jira/browse/SOLR-8418
> Project: Solr
> Issue Type: Bug
> Components: MoreLikeThis
>Affects Versions: 5.4
>Reporter: Jens Wille
>Assignee: Ramkumar Aiyengar
> Fix For: 5.5, Trunk
>
> Attachments: SOLR-8418.patch, SOLR-8418.patch
>
>
> As a consequence of LUCENE-6590, {{MoreLikeThisHandler}} was changed in
> r1701621 to use the new API. In SOLR-7912, I adapted that code for
> {{CloudMLTQParser}} and {{SimpleMLTQParser}}. However, setting the {{boost}}
> parameter just failed for me after updating to 5.4 with the following error
> message:
> {code}
> java.lang.ClassCastException: org.apache.lucene.search.BoostQuery cannot be
> cast to org.apache.lucene.search.TermQuery
> at
> org.apache.solr.search.mlt.SimpleMLTQParser.parse(SimpleMLTQParser.java:139)
> at org.apache.solr.search.QParser.getQuery(QParser.java:141)
> at
> org.apache.solr.handler.component.QueryComponent.prepare(QueryComponent.java:160)
> at
> org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:254)
> at
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:156)
> at org.apache.solr.core.SolrCore.execute(SolrCore.java:2073)
> at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:658)
> at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:457)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:222)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:181)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
> at
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
> at
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577)
> at
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
> at
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
> at
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
> at
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215)
> at
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110)
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
> at org.eclipse.jetty.server.Server.handle(Server.java:499)
> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
> at
> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
> at
> org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
> at java.lang.Thread.run(Thread.java:745)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Resolved] (SOLR-8418) BoostQuery cannot be cast to TermQuery
[
https://issues.apache.org/jira/browse/SOLR-8418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ramkumar Aiyengar resolved SOLR-8418.
-
Resolution: Fixed
Fix Version/s: Trunk
5.5
Thanks Jens!
> BoostQuery cannot be cast to TermQuery
> --
>
> Key: SOLR-8418
> URL: https://issues.apache.org/jira/browse/SOLR-8418
> Project: Solr
> Issue Type: Bug
> Components: MoreLikeThis
>Affects Versions: 5.4
>Reporter: Jens Wille
>Assignee: Ramkumar Aiyengar
> Fix For: 5.5, Trunk
>
> Attachments: SOLR-8418.patch, SOLR-8418.patch
>
>
> As a consequence of LUCENE-6590, {{MoreLikeThisHandler}} was changed in
> r1701621 to use the new API. In SOLR-7912, I adapted that code for
> {{CloudMLTQParser}} and {{SimpleMLTQParser}}. However, setting the {{boost}}
> parameter just failed for me after updating to 5.4 with the following error
> message:
> {code}
> java.lang.ClassCastException: org.apache.lucene.search.BoostQuery cannot be
> cast to org.apache.lucene.search.TermQuery
> at
> org.apache.solr.search.mlt.SimpleMLTQParser.parse(SimpleMLTQParser.java:139)
> at org.apache.solr.search.QParser.getQuery(QParser.java:141)
> at
> org.apache.solr.handler.component.QueryComponent.prepare(QueryComponent.java:160)
> at
> org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:254)
> at
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:156)
> at org.apache.solr.core.SolrCore.execute(SolrCore.java:2073)
> at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:658)
> at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:457)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:222)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:181)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
> at
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
> at
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577)
> at
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
> at
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
> at
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
> at
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215)
> at
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110)
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
> at org.eclipse.jetty.server.Server.handle(Server.java:499)
> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
> at
> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
> at
> org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
> at java.lang.Thread.run(Thread.java:745)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Assigned] (SOLR-8418) BoostQuery cannot be cast to TermQuery
[
https://issues.apache.org/jira/browse/SOLR-8418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ramkumar Aiyengar reassigned SOLR-8418:
---
Assignee: Ramkumar Aiyengar
> BoostQuery cannot be cast to TermQuery
> --
>
> Key: SOLR-8418
> URL: https://issues.apache.org/jira/browse/SOLR-8418
> Project: Solr
> Issue Type: Bug
> Components: MoreLikeThis
>Affects Versions: 5.4
>Reporter: Jens Wille
>Assignee: Ramkumar Aiyengar
> Attachments: SOLR-8418.patch
>
>
> As a consequence of LUCENE-6590, {{MoreLikeThisHandler}} was changed in
> r1701621 to use the new API. In SOLR-7912, I adapted that code for
> {{CloudMLTQParser}} and {{SimpleMLTQParser}}. However, setting the {{boost}}
> parameter just failed for me after updating to 5.4 with the following error
> message:
> {code}
> java.lang.ClassCastException: org.apache.lucene.search.BoostQuery cannot be
> cast to org.apache.lucene.search.TermQuery
> at
> org.apache.solr.search.mlt.SimpleMLTQParser.parse(SimpleMLTQParser.java:139)
> at org.apache.solr.search.QParser.getQuery(QParser.java:141)
> at
> org.apache.solr.handler.component.QueryComponent.prepare(QueryComponent.java:160)
> at
> org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:254)
> at
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:156)
> at org.apache.solr.core.SolrCore.execute(SolrCore.java:2073)
> at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:658)
> at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:457)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:222)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:181)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
> at
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
> at
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577)
> at
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
> at
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
> at
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
> at
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215)
> at
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110)
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
> at org.eclipse.jetty.server.Server.handle(Server.java:499)
> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
> at
> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
> at
> org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
> at java.lang.Thread.run(Thread.java:745)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-8418) BoostQuery cannot be cast to TermQuery
[
https://issues.apache.org/jira/browse/SOLR-8418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ramkumar Aiyengar updated SOLR-8418:
Attachment: SOLR-8418.patch
Attached is a slightly revised patch, please test and let me know if it works.
> BoostQuery cannot be cast to TermQuery
> --
>
> Key: SOLR-8418
> URL: https://issues.apache.org/jira/browse/SOLR-8418
> Project: Solr
> Issue Type: Bug
> Components: MoreLikeThis
>Affects Versions: 5.4
>Reporter: Jens Wille
>Assignee: Ramkumar Aiyengar
> Attachments: SOLR-8418.patch, SOLR-8418.patch
>
>
> As a consequence of LUCENE-6590, {{MoreLikeThisHandler}} was changed in
> r1701621 to use the new API. In SOLR-7912, I adapted that code for
> {{CloudMLTQParser}} and {{SimpleMLTQParser}}. However, setting the {{boost}}
> parameter just failed for me after updating to 5.4 with the following error
> message:
> {code}
> java.lang.ClassCastException: org.apache.lucene.search.BoostQuery cannot be
> cast to org.apache.lucene.search.TermQuery
> at
> org.apache.solr.search.mlt.SimpleMLTQParser.parse(SimpleMLTQParser.java:139)
> at org.apache.solr.search.QParser.getQuery(QParser.java:141)
> at
> org.apache.solr.handler.component.QueryComponent.prepare(QueryComponent.java:160)
> at
> org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:254)
> at
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:156)
> at org.apache.solr.core.SolrCore.execute(SolrCore.java:2073)
> at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:658)
> at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:457)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:222)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:181)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
> at
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
> at
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577)
> at
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
> at
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
> at
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
> at
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215)
> at
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110)
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
> at org.eclipse.jetty.server.Server.handle(Server.java:499)
> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
> at
> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
> at
> org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
> at java.lang.Thread.run(Thread.java:745)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-8371) Try and prevent too many recovery requests from stacking up and clean up some faulty logic.
[ https://issues.apache.org/jira/browse/SOLR-8371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ramkumar Aiyengar updated SOLR-8371: Attachment: SOLR-8371-2.patch Here's a patch of what I meant, on top of your changes. I haven't tested it extensively though, beyond a run or two of the existing TestRecovery.. > Try and prevent too many recovery requests from stacking up and clean up some > faulty logic. > --- > > Key: SOLR-8371 > URL: https://issues.apache.org/jira/browse/SOLR-8371 > Project: Solr > Issue Type: Improvement >Reporter: Mark Miller >Assignee: Mark Miller > Attachments: SOLR-8371-2.patch, SOLR-8371.patch, SOLR-8371.patch, > SOLR-8371.patch, SOLR-8371.patch, SOLR-8371.patch, SOLR-8371.patch, > SOLR-8371.patch, SOLR-8371.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-8371) Try and prevent too many recovery requests from stacking up and clean up some faulty logic.
[ https://issues.apache.org/jira/browse/SOLR-8371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15071983#comment-15071983 ] Ramkumar Aiyengar commented on SOLR-8371: - Looks good functionally. But I don't fully understand why you need the locking/queue-checks.. Can't you have a bounded recovery queue (of say 1 or 2 -- not sure of the semantics), and just return if the queue limit is exceeded? > Try and prevent too many recovery requests from stacking up and clean up some > faulty logic. > --- > > Key: SOLR-8371 > URL: https://issues.apache.org/jira/browse/SOLR-8371 > Project: Solr > Issue Type: Improvement >Reporter: Mark Miller >Assignee: Mark Miller > Attachments: SOLR-8371.patch, SOLR-8371.patch, SOLR-8371.patch, > SOLR-8371.patch, SOLR-8371.patch, SOLR-8371.patch, SOLR-8371.patch, > SOLR-8371.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-8354) RecoveryStrategy retry timing is innaccurate
[
https://issues.apache.org/jira/browse/SOLR-8354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15033577#comment-15033577
]
Ramkumar Aiyengar commented on SOLR-8354:
-
Yeah, just complicates things and leads to unexpected delays if our assumptions
on the root cause of the failure was wrong. I do favour a simple retry delay..
> RecoveryStrategy retry timing is innaccurate
>
>
> Key: SOLR-8354
> URL: https://issues.apache.org/jira/browse/SOLR-8354
> Project: Solr
> Issue Type: Improvement
>Reporter: Mike Drob
> Attachments: SOLR-8354.patch
>
>
> At the end of {{RecoveryStrategy::doRecovery}} there is a retry segment, with
> a comment that suggests the code will {{// start at 1 sec and work up to a
> min}}. The code will actually start at 10 seconds, and work up to 5 minutes.
> Additionally, the log statement incorrectly reports how long the next wait
> will be. Either the comment and log should be corrected or the logic adjusted.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Comment Edited] (SOLR-7344) Allow Jetty thread pool limits while still avoiding distributed deadlock.
[ https://issues.apache.org/jira/browse/SOLR-7344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14961970#comment-14961970 ] Ramkumar Aiyengar edited comment on SOLR-7344 at 10/17/15 5:10 PM: --- [~ysee...@gmail.com], [~markrmil...@gmail.com] and I got a chance to discuss this at Revolution. One approach we seemed to have consensus on (correct me guys if I am wrong) is: - We use a single, small, internal queue for non-streaming APIs. - We leave the limit high for streaming APIs and raise a new issue for (1) The ability to control the amount of nesting in Streaming APIs, and (2) To dynamically have N pools (Internal.1, Internal.2, , Internal.N), each with a small limit, where N is the amount of maximum nesting configured for Streaming APIs.. was (Author: andyetitmoves): [~ysee...@gmail.com], [~markrmil...@gmail.com] and me got a chance to discuss this at Revolution. One approach we seem to have consensus on (correct me guys if I am wrong) is: - We use a single, small, internal queue for non-streaming APIs. - We leave the limit high for streaming APIs and raise a new issue for (1) The ability to control the amount of nesting in Streaming APIs, and (2) To dynamically have N pools (Internal.1, Internal.2, , Internal.N), each with a small limit, where N is the amount of maximum nesting configured for Streaming APIs.. > Allow Jetty thread pool limits while still avoiding distributed deadlock. > - > > Key: SOLR-7344 > URL: https://issues.apache.org/jira/browse/SOLR-7344 > Project: Solr > Issue Type: Improvement > Components: SolrCloud >Reporter: Mark Miller > Attachments: SOLR-7344.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-7344) Allow Jetty thread pool limits while still avoiding distributed deadlock.
[ https://issues.apache.org/jira/browse/SOLR-7344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14961970#comment-14961970 ] Ramkumar Aiyengar commented on SOLR-7344: - [~ysee...@gmail.com], [~markrmil...@gmail.com] and me got a chance to discuss this at Revolution. One approach we seem to have consensus on (correct me guys if I am wrong) is: - We use a single, small, internal queue for non-streaming APIs. - We leave the limit high for streaming APIs and raise a new issue for (1) The ability to control the amount of nesting in Streaming APIs, and (2) To dynamically have N pools (Internal.1, Internal.2, , Internal.N), each with a small limit, where N is the amount of maximum nesting configured for Streaming APIs.. > Allow Jetty thread pool limits while still avoiding distributed deadlock. > - > > Key: SOLR-7344 > URL: https://issues.apache.org/jira/browse/SOLR-7344 > Project: Solr > Issue Type: Improvement > Components: SolrCloud >Reporter: Mark Miller > Attachments: SOLR-7344.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Assigned] (SOLR-7683) Introduce support to identify Solr internal request types
[ https://issues.apache.org/jira/browse/SOLR-7683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ramkumar Aiyengar reassigned SOLR-7683: --- Assignee: Ramkumar Aiyengar > Introduce support to identify Solr internal request types > - > > Key: SOLR-7683 > URL: https://issues.apache.org/jira/browse/SOLR-7683 > Project: Solr > Issue Type: Sub-task > Components: SolrCloud >Reporter: Hrishikesh Gadre >Assignee: Ramkumar Aiyengar > Attachments: SOLR-7683.patch > > > SOLR-7344 is introducing support to partition the Jetty worker pool to > enforce the number of concurrent requests for various types (e.g. > Internal_Querying, Internal_Indexing, External etc.). For this we need to > identify requests sent between Solr servers and their types (i.e. > Querying/Indexing etc.). -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-7683) Introduce support to identify Solr internal request types
[ https://issues.apache.org/jira/browse/SOLR-7683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ramkumar Aiyengar updated SOLR-7683: Assignee: (was: Ramkumar Aiyengar) > Introduce support to identify Solr internal request types > - > > Key: SOLR-7683 > URL: https://issues.apache.org/jira/browse/SOLR-7683 > Project: Solr > Issue Type: Sub-task > Components: SolrCloud >Reporter: Hrishikesh Gadre > Attachments: SOLR-7683.patch > > > SOLR-7344 is introducing support to partition the Jetty worker pool to > enforce the number of concurrent requests for various types (e.g. > Internal_Querying, Internal_Indexing, External etc.). For this we need to > identify requests sent between Solr servers and their types (i.e. > Querying/Indexing etc.). -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Assigned] (SOLR-7683) Introduce support to identify Solr internal request types
[ https://issues.apache.org/jira/browse/SOLR-7683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ramkumar Aiyengar reassigned SOLR-7683: --- Assignee: Ramkumar Aiyengar > Introduce support to identify Solr internal request types > - > > Key: SOLR-7683 > URL: https://issues.apache.org/jira/browse/SOLR-7683 > Project: Solr > Issue Type: Sub-task > Components: SolrCloud >Reporter: Hrishikesh Gadre >Assignee: Ramkumar Aiyengar > Attachments: SOLR-7683.patch > > > SOLR-7344 is introducing support to partition the Jetty worker pool to > enforce the number of concurrent requests for various types (e.g. > Internal_Querying, Internal_Indexing, External etc.). For this we need to > identify requests sent between Solr servers and their types (i.e. > Querying/Indexing etc.). -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-7683) Introduce support to identify Solr internal request types
[ https://issues.apache.org/jira/browse/SOLR-7683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14961974#comment-14961974 ] Ramkumar Aiyengar commented on SOLR-7683: - [~hgadre], I am happy to pick this up, could you please bring the patch up to date? The only change I would probably do is to remove unspecified in trunk alone so that any request handler is forced to specify it. But you don't have to update the patch for that, I will do it for trunk alone, and keep your patch for branch_5x where the backward compat is useful. > Introduce support to identify Solr internal request types > - > > Key: SOLR-7683 > URL: https://issues.apache.org/jira/browse/SOLR-7683 > Project: Solr > Issue Type: Sub-task > Components: SolrCloud >Reporter: Hrishikesh Gadre >Assignee: Ramkumar Aiyengar > Attachments: SOLR-7683.patch > > > SOLR-7344 is introducing support to partition the Jetty worker pool to > enforce the number of concurrent requests for various types (e.g. > Internal_Querying, Internal_Indexing, External etc.). For this we need to > identify requests sent between Solr servers and their types (i.e. > Querying/Indexing etc.). -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-6828) Speed up requests for many rows
[ https://issues.apache.org/jira/browse/LUCENE-6828?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14945860#comment-14945860 ] Ramkumar Aiyengar commented on LUCENE-6828: --- You still need to do old fashioned deep paging if you are paging with grouping. Grouping requires you to have context of groups and docs with any higher sort value than what you are returning. > Speed up requests for many rows > --- > > Key: LUCENE-6828 > URL: https://issues.apache.org/jira/browse/LUCENE-6828 > Project: Lucene - Core > Issue Type: Improvement > Components: core/search >Affects Versions: 4.10.4, 5.3 >Reporter: Toke Eskildsen >Priority: Minor > Labels: memory, performance > > Standard relevance ranked searches for top-X results uses the HitQueue class > to keep track of the highest scoring documents. The HitQueue is a binary heap > of ScoreDocs and is pre-filled with sentinel objects upon creation. > Binary heaps of Objects in Java does not scale well: The HitQueue uses 28 > bytes/element and memory access is scattered due to the binary heap algorithm > and the use of Objects. To make matters worse, the use of sentinel objects > means that even if only a tiny number of documents matches, the full amount > of Objects is still allocated. > As long as the HitQueue is small (< 1000), it performs very well. If top-1M > results are requested, it performs poorly and leaves 1M ScoreDocs to be > garbage collected. > An alternative is to replace the ScoreDocs with a single array of packed > longs, each long holding the score and the document ID. This strategy > requires only 8 bytes/element and is a lot lighter on the GC. > Some preliminary tests has been done and published at > https://sbdevel.wordpress.com/2015/10/05/speeding-up-core-search/ > These indicate that a long[]-backed implementation is at least 3x faster than > vanilla HitDocs for top-1M requests. > For smaller requests, such as top-10, the packed version also seems > competitive, when the amount of matched documents exceeds 1M. This needs to > be investigated further. > Going forward with this idea requires some refactoring as Lucene is currently > hardwired to the abstract PriorityQueue. Before attempting this, it seems > prudent to discuss whether speeding up large top-X requests has any value? > Paging seems an obvious contender for requesting large result sets, but I > guess the two could work in tandem, opening up for efficient large pages. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-8069) Leader Initiated Recovery can put the replica with the latest data into LIR and a shard will have no leader even on restart.
[ https://issues.apache.org/jira/browse/SOLR-8069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14876205#comment-14876205 ] Ramkumar Aiyengar commented on SOLR-8069: - That makes sense, but for my understanding, why is it a bad idea even if not everyone is participating? > Leader Initiated Recovery can put the replica with the latest data into LIR > and a shard will have no leader even on restart. > > > Key: SOLR-8069 > URL: https://issues.apache.org/jira/browse/SOLR-8069 > Project: Solr > Issue Type: Bug >Reporter: Mark Miller > Attachments: SOLR-8069.patch, SOLR-8069.patch > > > I've seen this twice now. Need to work on a test. > When some issues hit all the replicas at once, you can end up in a situation > where the rightful leader was put or put itself into LIR. Even on restart, > this rightful leader won't take leadership and you have to manually clear the > LIR nodes. > It seems that if all the replicas participate in election on startup, LIR > should just be cleared. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-8069) Leader Initiated Recovery can put the replica with the latest data into LIR and a shard will have no leader even on restart.
[ https://issues.apache.org/jira/browse/SOLR-8069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14876697#comment-14876697 ] Ramkumar Aiyengar commented on SOLR-8069: - R1 was not in LIR, but it came up while R2 was still at the lead and decided to recover, before R2 stepped down due to being in LIR. > Leader Initiated Recovery can put the replica with the latest data into LIR > and a shard will have no leader even on restart. > > > Key: SOLR-8069 > URL: https://issues.apache.org/jira/browse/SOLR-8069 > Project: Solr > Issue Type: Bug >Reporter: Mark Miller > Attachments: SOLR-8069.patch, SOLR-8069.patch > > > I've seen this twice now. Need to work on a test. > When some issues hit all the replicas at once, you can end up in a situation > where the rightful leader was put or put itself into LIR. Even on restart, > this rightful leader won't take leadership and you have to manually clear the > LIR nodes. > It seems that if all the replicas participate in election on startup, LIR > should just be cleared. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-8069) Leader Initiated Recovery can put the replica with the latest data into LIR and a shard will have no leader even on restart.
[ https://issues.apache.org/jira/browse/SOLR-8069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14876133#comment-14876133 ] Ramkumar Aiyengar commented on SOLR-8069: - Late to the party here.. We experienced the same issue, and [~cpoerschke] was trying to create a test case for this. My initial thought was why we even check LIR when we are about to become the leader? Shouldn't the double way sync cover us even if we are behind due to losing documents? > Leader Initiated Recovery can put the replica with the latest data into LIR > and a shard will have no leader even on restart. > > > Key: SOLR-8069 > URL: https://issues.apache.org/jira/browse/SOLR-8069 > Project: Solr > Issue Type: Bug >Reporter: Mark Miller > Attachments: SOLR-8069.patch, SOLR-8069.patch > > > I've seen this twice now. Need to work on a test. > When some issues hit all the replicas at once, you can end up in a situation > where the rightful leader was put or put itself into LIR. Even on restart, > this rightful leader won't take leadership and you have to manually clear the > LIR nodes. > It seems that if all the replicas participate in election on startup, LIR > should just be cleared. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-8069) Leader Initiated Recovery can put the replica with the latest data into LIR and a shard will have no leader even on restart.
[ https://issues.apache.org/jira/browse/SOLR-8069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14876156#comment-14876156 ] Ramkumar Aiyengar commented on SOLR-8069: - The case we hit was when we cold stopped/started the cloud. This was on 4.10.4, so may not be valid now. Let's say you have R1 and R2. * R1 is the leader and both R1 and R2 are stopped at the same time. * R2's stops accepting requests but hasn't updated ZK as yet, when R1 sends a update to R2, it fails and puts R2 in LIR. * R2 shuts down first, then R1. * R1 starts up first, finds it should be the leader. * R2 decides it should follow and tries to recover. * R1 decides it can't be leader due to LIR and steps down. But by then R2 is in recovery, doesn't step up, and we have no one stepping forward. > Leader Initiated Recovery can put the replica with the latest data into LIR > and a shard will have no leader even on restart. > > > Key: SOLR-8069 > URL: https://issues.apache.org/jira/browse/SOLR-8069 > Project: Solr > Issue Type: Bug >Reporter: Mark Miller > Attachments: SOLR-8069.patch, SOLR-8069.patch > > > I've seen this twice now. Need to work on a test. > When some issues hit all the replicas at once, you can end up in a situation > where the rightful leader was put or put itself into LIR. Even on restart, > this rightful leader won't take leadership and you have to manually clear the > LIR nodes. > It seems that if all the replicas participate in election on startup, LIR > should just be cleared. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-8069) Leader Initiated Recovery can put the replica with the latest data into LIR and a shard will have no leader even on restart.
[ https://issues.apache.org/jira/browse/SOLR-8069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14876271#comment-14876271 ] Ramkumar Aiyengar commented on SOLR-8069: - Got it. You've to include those in recovery along with the participants since the ones which have gone into recovery are not going to help in anyway (an alternative would be for them to abort recovery and rejoin of no one is around). But in your case, if I understand it right, detecting that there are down replicas (which might come back as good leaders) would certainly be a good idea. > Leader Initiated Recovery can put the replica with the latest data into LIR > and a shard will have no leader even on restart. > > > Key: SOLR-8069 > URL: https://issues.apache.org/jira/browse/SOLR-8069 > Project: Solr > Issue Type: Bug >Reporter: Mark Miller > Attachments: SOLR-8069.patch, SOLR-8069.patch > > > I've seen this twice now. Need to work on a test. > When some issues hit all the replicas at once, you can end up in a situation > where the rightful leader was put or put itself into LIR. Even on restart, > this rightful leader won't take leadership and you have to manually clear the > LIR nodes. > It seems that if all the replicas participate in election on startup, LIR > should just be cleared. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-8029) Modernize and standardize Solr APIs
[
https://issues.apache.org/jira/browse/SOLR-8029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14746087#comment-14746087
]
Ramkumar Aiyengar commented on SOLR-8029:
-
bq. Do you mean ,the introspect API returns the schema of the output?
Schema for both the input and output, yes.
> Modernize and standardize Solr APIs
> ---
>
> Key: SOLR-8029
> URL: https://issues.apache.org/jira/browse/SOLR-8029
> Project: Solr
> Issue Type: Improvement
>Affects Versions: 6.0
>Reporter: Noble Paul
>Assignee: Noble Paul
> Labels: API, EaseOfUse
> Fix For: 6.0
>
>
> Solr APIs have organically evolved and they are sometimes inconsistent with
> each other or not in sync with the widely followed conventions of HTTP
> protocol. Trying to make incremental changes to make them modern is like
> applying band-aid. So, we have done a complete rethink of what the APIs
> should be. The most notable aspects of the API are as follows:
> The new set of APIs will be placed under a new path {{/solr2}}. The legacy
> APIs will continue to work under the {{/solr}} path as they used to and they
> will be eventually deprecated.
> There are 3 types of requests in the new API
> * {{/solr2//*}} : Operations on specific collections
> * {{/solr2/_cluster/*}} : Cluster-wide operations which are not specific to
> any collections.
> * {{/solr2/_node/*}} : Operations on the node receiving the request. This is
> the counter part of the core admin API
> This will be released as part of a major release. Check the link given below
> for the full specification. Your comments are welcome
> [Solr API version 2 Specification | http://bit.ly/1JYsBMQ]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-8029) Modernize and standardize Solr APIs
[
https://issues.apache.org/jira/browse/SOLR-8029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14745220#comment-14745220
]
Ramkumar Aiyengar commented on SOLR-8029:
-
Right, my point was to see if we can use it beyond just being a doc, and use it
to format the response. The {{wt}} param will still drive the encoding/decoding
(i.e. json/xml/hocon whatever..), but instead of all APIs just populating and
serializing a generic NamedList, it would help to make sure what the APIs
return conform to the doc we make available. That way, something like Avro for
example, can use the doc as configuration to encode/decode it's data.
> Modernize and standardize Solr APIs
> ---
>
> Key: SOLR-8029
> URL: https://issues.apache.org/jira/browse/SOLR-8029
> Project: Solr
> Issue Type: Improvement
>Affects Versions: 6.0
>Reporter: Noble Paul
>Assignee: Noble Paul
> Labels: API, EaseOfUse
> Fix For: 6.0
>
>
> Solr APIs have organically evolved and they are sometimes inconsistent with
> each other or not in sync with the widely followed conventions of HTTP
> protocol. Trying to make incremental changes to make them modern is like
> applying band-aid. So, we have done a complete rethink of what the APIs
> should be. The most notable aspects of the API are as follows:
> The new set of APIs will be placed under a new path {{/solr2}}. The legacy
> APIs will continue to work under the {{/solr}} path as they used to and they
> will be eventually deprecated.
> There are 3 types of requests in the new API
> * {{/solr2//*}} : Operations on specific collections
> * {{/solr2/_cluster/*}} : Cluster-wide operations which are not specific to
> any collections.
> * {{/solr2/_node/*}} : Operations on the node receiving the request. This is
> the counter part of the core admin API
> This will be released as part of a major release. Check the link given below
> for the full specification. Your comments are welcome
> [Solr API version 2 Specification | http://bit.ly/1JYsBMQ]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-8029) Modernize and standardize Solr APIs
[
https://issues.apache.org/jira/browse/SOLR-8029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14742298#comment-14742298
]
Ramkumar Aiyengar commented on SOLR-8029:
-
The introspect API probably requires this anyway, but all API should be able to
statically declare their schema in some format. This would make it possible to
plug in encodings which require a schema, like Avro.
> Modernize and standardize Solr APIs
> ---
>
> Key: SOLR-8029
> URL: https://issues.apache.org/jira/browse/SOLR-8029
> Project: Solr
> Issue Type: Improvement
>Affects Versions: 6.0
>Reporter: Noble Paul
>Assignee: Noble Paul
> Labels: API, EaseOfUse
> Fix For: 6.0
>
>
> Solr APIs have organically evolved and they are sometimes inconsistent with
> each other or not in sync with the widely followed conventions of HTTP
> protocol. Trying to make incremental changes to make them modern is like
> applying band-aid. So, we have done a complete rethink of what the APIs
> should be. The most notable aspects of the API are as follows:
> The new set of APIs will be placed under a new path {{/solr2}}. The legacy
> APIs will continue to work under the {{/solr}} path as they used to and they
> will be eventually deprecated.
> There are 3 types of requests in the new API
> * {{/solr2//*}} : Operations on specific collections
> * {{/solr2/_cluster/*}} : Cluster-wide operations which are not specific to
> any collections.
> * {{/solr2/_node/*}} : Operations on the node receiving the request. This is
> the counter part of the core admin API
> This will be released as part of a major release. Check the link given below
> for the full specification. Your comments are welcome
> [Solr API version 2 Specification | http://bit.ly/1JYsBMQ]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-7955) Auto create .system collection on first start if it does not exist
[
https://issues.apache.org/jira/browse/SOLR-7955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14708976#comment-14708976
]
Ramkumar Aiyengar commented on SOLR-7955:
-
Should certainly be possible. Actually there's the concept of lazy cores
already which probably does this already, but I am not sure..
Auto create .system collection on first start if it does not exist
--
Key: SOLR-7955
URL: https://issues.apache.org/jira/browse/SOLR-7955
Project: Solr
Issue Type: Improvement
Reporter: Jan Høydahl
Why should a user need to create the {{.system}} collection manually? It
would simplify instructions related to BLOB store if user could assume it is
always there.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-7932) Solr replication relies on timestamps to sync across machines
[
https://issues.apache.org/jira/browse/SOLR-7932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ramkumar Aiyengar updated SOLR-7932:
Attachment: SOLR-7932.patch
Okay, attached here is a smaller patch which was my biggest concern here with
the timestamps to begin with. These two places use timestamps *instead* of
generations, and can lead to data loss. The other place where both are used,
worst case you will end up doing a full-copy, not a big deal either way (and I
understand the case with master-slave in such a situation).
*First change:*
{code}
if (latestVersion == 0L) {
if (forceReplication commit.getGeneration() != 0) {
// since we won't get the files for an empty index,
// we just clear ours and commit
RefCountedIndexWriter iw =
solrCore.getUpdateHandler().getSolrCoreState().getIndexWriter(solrCore);
try {
iw.get().deleteAll();
} finally {
iw.decref();
}
SolrQueryRequest req = new LocalSolrQueryRequest(solrCore, new
ModifiableSolrParams());
solrCore.getUpdateHandler().commit(new CommitUpdateCommand(req,
false));
}
//there is nothing to be replicated
successfulInstall = true;
return true;
}
{code}
Proposing replacing {{latestVersion}} with {{latestGeneration}}. Is there a
case where one would be 0 and the other not?
*Second change:*
{code}
if (!forceReplication
IndexDeletionPolicyWrapper.getCommitTimestamp(commit) == latestVersion) {
//master and slave are already in sync just return
LOG.info(Slave in sync with master.);
successfulInstall = true;
return true;
}
{code}
I am proposing:
{code}
- if (!forceReplication
IndexDeletionPolicyWrapper.getCommitTimestamp(commit) == latestVersion) {
+ if (!forceReplication commit.getGeneration() == latestGeneration) {
{code}
- If generations are not the same, but timestamps are, no harm done really,
yes, you will go through replication unnecessarily, but that's it.
- Will master-slave result in the same situation? If you blow away the index,
I would expect the generations to not match (would be much lesser).
Alternatively, we could OR the two checks here as well (i.e. check both
versions and generations here), to be absolutely sure we don't skip replication
if we shouldn't.
[~ysee...@gmail.com], [~markrmil...@gmail.com], could you clarify how this
change would affect this adversely in such a case?
Solr replication relies on timestamps to sync across machines
-
Key: SOLR-7932
URL: https://issues.apache.org/jira/browse/SOLR-7932
Project: Solr
Issue Type: Bug
Components: replication (java)
Reporter: Ramkumar Aiyengar
Attachments: SOLR-7932.patch, SOLR-7932.patch
Spinning off SOLR-7859, noticed there that wall time recorded as commit data
on a commit to check if replication needs to be done. In IndexFetcher, there
is this code:
{code}
if (!forceReplication
IndexDeletionPolicyWrapper.getCommitTimestamp(commit) == latestVersion) {
//master and slave are already in sync just return
LOG.info(Slave in sync with master.);
successfulInstall = true;
return true;
}
{code}
It appears as if we are checking wall times across machines to check if we
are in sync, this could go wrong.
Once a decision is made to replicate, we do seem to use generations instead,
except for this place below checks both generations and timestamps to see if
a full copy is needed..
{code}
// if the generation of master is older than that of the slave , it
means they are not compatible to be copied
// then a new index directory to be created and all the files need to
be copied
boolean isFullCopyNeeded = IndexDeletionPolicyWrapper
.getCommitTimestamp(commit) = latestVersion
|| commit.getGeneration() = latestGeneration || forceReplication;
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-7955) Auto create .system collection on first start if it does not exist
[
https://issues.apache.org/jira/browse/SOLR-7955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14708132#comment-14708132
]
Ramkumar Aiyengar commented on SOLR-7955:
-
I agree, there's a footprint to adding a collection, and people who don't need
this feature shouldn't have to bear it. If we want to make the feature easier,
may be lazily on first use (if possible) is a better alternative than on first
start..
Auto create .system collection on first start if it does not exist
--
Key: SOLR-7955
URL: https://issues.apache.org/jira/browse/SOLR-7955
Project: Solr
Issue Type: Improvement
Reporter: Jan Høydahl
Why should a user need to create the {{.system}} collection manually? It
would simplify instructions related to BLOB store if user could assume it is
always there.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-6760) New optimized DistributedQueue implementation for overseer
[ https://issues.apache.org/jira/browse/SOLR-6760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14704564#comment-14704564 ] Ramkumar Aiyengar commented on SOLR-6760: - In the output above, 'success' new way is 3 instead of 20k, some test bug? New optimized DistributedQueue implementation for overseer -- Key: SOLR-6760 URL: https://issues.apache.org/jira/browse/SOLR-6760 Project: Solr Issue Type: Bug Reporter: Noble Paul Assignee: Shalin Shekhar Mangar Attachments: SOLR-6760.patch, SOLR-6760.patch, SOLR-6760.patch, SOLR-6760.patch, deadlock.patch Currently the DQ works as follows * read all items in the directory * sort them all * take the head and return it and discard everything else * rinse and repeat This works well when we have only a handful of items in the Queue. If the items in the queue is much larger (in tens of thousands) , this is counterproductive As the overseer queue is a multiple producers + single consumer queue, We can read them all in bulk and before processing each item , just do a zk.exists(itemname) and if all is well we don't need to do the fetch all + sort thing again -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Comment Edited] (SOLR-6760) New optimized DistributedQueue implementation for overseer
[ https://issues.apache.org/jira/browse/SOLR-6760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14704564#comment-14704564 ] Ramkumar Aiyengar edited comment on SOLR-6760 at 8/20/15 9:17 AM: -- In the output above, 'success' new way is 3 instead of 20k, some test bug? Or is it counting with the batching as one op? was (Author: andyetitmoves): In the output above, 'success' new way is 3 instead of 20k, some test bug? New optimized DistributedQueue implementation for overseer -- Key: SOLR-6760 URL: https://issues.apache.org/jira/browse/SOLR-6760 Project: Solr Issue Type: Bug Reporter: Noble Paul Assignee: Shalin Shekhar Mangar Attachments: SOLR-6760.patch, SOLR-6760.patch, SOLR-6760.patch, SOLR-6760.patch, deadlock.patch Currently the DQ works as follows * read all items in the directory * sort them all * take the head and return it and discard everything else * rinse and repeat This works well when we have only a handful of items in the Queue. If the items in the queue is much larger (in tens of thousands) , this is counterproductive As the overseer queue is a multiple producers + single consumer queue, We can read them all in bulk and before processing each item , just do a zk.exists(itemname) and if all is well we don't need to do the fetch all + sort thing again -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-6760) New optimized DistributedQueue implementation for overseer
[ https://issues.apache.org/jira/browse/SOLR-6760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14704610#comment-14704610 ] Ramkumar Aiyengar commented on SOLR-6760: - Ah, got it, my bad.. New optimized DistributedQueue implementation for overseer -- Key: SOLR-6760 URL: https://issues.apache.org/jira/browse/SOLR-6760 Project: Solr Issue Type: Bug Reporter: Noble Paul Assignee: Shalin Shekhar Mangar Attachments: SOLR-6760.patch, SOLR-6760.patch, SOLR-6760.patch, SOLR-6760.patch, deadlock.patch Currently the DQ works as follows * read all items in the directory * sort them all * take the head and return it and discard everything else * rinse and repeat This works well when we have only a handful of items in the Queue. If the items in the queue is much larger (in tens of thousands) , this is counterproductive As the overseer queue is a multiple producers + single consumer queue, We can read them all in bulk and before processing each item , just do a zk.exists(itemname) and if all is well we don't need to do the fetch all + sort thing again -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-7932) Solr replication relies on timestamps to sync across machines
[
https://issues.apache.org/jira/browse/SOLR-7932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14702927#comment-14702927
]
Ramkumar Aiyengar commented on SOLR-7932:
-
bq. Why? Don't both times come from the master?
A clock skew could cause two different commits to have the same time (commit 1
happens at time X, NTP sets the clock back by 200ms. 200ms later, commit 2
happens). It's not exactly what's in this title (i.e. relying on timestamps
across machines), and you have to be a lot more unlucky, but you can't rely on
wall time even in the same machine.
bq. I don't think it can just be removed in either case without better
replacement logic.
How does the timestamp help currently in the first case? We are anyway using
generations immediately following, so won't you be better off comparing
generations instead to check if replication can be skipped?
Solr replication relies on timestamps to sync across machines
-
Key: SOLR-7932
URL: https://issues.apache.org/jira/browse/SOLR-7932
Project: Solr
Issue Type: Bug
Components: replication (java)
Reporter: Ramkumar Aiyengar
Attachments: SOLR-7932.patch
Spinning off SOLR-7859, noticed there that wall time recorded as commit data
on a commit to check if replication needs to be done. In IndexFetcher, there
is this code:
{code}
if (!forceReplication
IndexDeletionPolicyWrapper.getCommitTimestamp(commit) == latestVersion) {
//master and slave are already in sync just return
LOG.info(Slave in sync with master.);
successfulInstall = true;
return true;
}
{code}
It appears as if we are checking wall times across machines to check if we
are in sync, this could go wrong.
Once a decision is made to replicate, we do seem to use generations instead,
except for this place below checks both generations and timestamps to see if
a full copy is needed..
{code}
// if the generation of master is older than that of the slave , it
means they are not compatible to be copied
// then a new index directory to be created and all the files need to
be copied
boolean isFullCopyNeeded = IndexDeletionPolicyWrapper
.getCommitTimestamp(commit) = latestVersion
|| commit.getGeneration() = latestGeneration || forceReplication;
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-7932) Solr replication relies on timestamps to sync across machines
[
https://issues.apache.org/jira/browse/SOLR-7932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14702773#comment-14702773
]
Ramkumar Aiyengar commented on SOLR-7932:
-
Thanks for the comments [~ysee...@gmail.com]. With master-slave replication,
yes, this is less of a problem (you still have to deal with clock skew though).
There are two places where the index time is used..
- To compare if they are equal to skip replication. Unless I am mistaken, the
timestamp check is not useful to detect index re-creation in this case.
- To check if full index replication should be forced. I see the use here
(though I don't see an easy way you can do this in a cloud without stopping the
full cloud, blowing an index on one but not all replicas, and making sure it
comes up first)
I am more concerned really about the first case, as you can lose data if you
are unlucky. Do you agree that the timestamp check can be removed there?
For the second, probably the index creation time is a better thing to check
against rather than the last commit time, as it is less subject to skew? I
don't know if Lucene even provides a way to know when the index was initially
created though. And this could be tackled as a different issue..
Solr replication relies on timestamps to sync across machines
-
Key: SOLR-7932
URL: https://issues.apache.org/jira/browse/SOLR-7932
Project: Solr
Issue Type: Bug
Components: replication (java)
Reporter: Ramkumar Aiyengar
Attachments: SOLR-7932.patch
Spinning off SOLR-7859, noticed there that wall time recorded as commit data
on a commit to check if replication needs to be done. In IndexFetcher, there
is this code:
{code}
if (!forceReplication
IndexDeletionPolicyWrapper.getCommitTimestamp(commit) == latestVersion) {
//master and slave are already in sync just return
LOG.info(Slave in sync with master.);
successfulInstall = true;
return true;
}
{code}
It appears as if we are checking wall times across machines to check if we
are in sync, this could go wrong.
Once a decision is made to replicate, we do seem to use generations instead,
except for this place below checks both generations and timestamps to see if
a full copy is needed..
{code}
// if the generation of master is older than that of the slave , it
means they are not compatible to be copied
// then a new index directory to be created and all the files need to
be copied
boolean isFullCopyNeeded = IndexDeletionPolicyWrapper
.getCommitTimestamp(commit) = latestVersion
|| commit.getGeneration() = latestGeneration || forceReplication;
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-5043) hostanme lookup in SystemInfoHandler should be refactored to not block core (re)load
[
https://issues.apache.org/jira/browse/SOLR-5043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14701849#comment-14701849
]
Ramkumar Aiyengar commented on SOLR-5043:
-
Ah, my bad, I interpreted the comment to mean not static (for other reasons),
so core reload will (unfortunately) refresh (, but that's not a big deal) :)
Makes sense overall..
hostanme lookup in SystemInfoHandler should be refactored to not block core
(re)load
Key: SOLR-5043
URL: https://issues.apache.org/jira/browse/SOLR-5043
Project: Solr
Issue Type: Improvement
Reporter: Hoss Man
Attachments: SOLR-5043-lazy.patch, SOLR-5043.patch
SystemInfoHandler currently lookups the hostname of the machine on it's init,
and caches for it's lifecycle -- there is a comment to the effect that the
reason for this is because on some machines (notably ones with wacky DNS
settings) looking up the hostname can take a long ass time in some JVMs...
{noformat}
// on some platforms, resolving canonical hostname can cause the thread
// to block for several seconds if nameservices aren't available
// so resolve this once per handler instance
//(ie: not static, so core reload will refresh)
{noformat}
But as we move forward with a lot more multi-core, solr-cloud, dynamically
updated instances, even paying this cost per core-reload is expensive.
we should refactoring this so that SystemInfoHandler instances init
immediately, with some kind of lazy loading of the hostname info in a
background thread, (especially since hte only real point of having that info
here is for UI use so you cna keep track of what machine you are looking at)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-7884) Omit reverse lookups in SystemInfoHandler
[
https://issues.apache.org/jira/browse/SOLR-7884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14700370#comment-14700370
]
Ramkumar Aiyengar commented on SOLR-7884:
-
Robert, I think Hoss' suggestion of using SOLR-5043 works best, as it decouples
the functionality which needs the hostname (SystemInfoHandler) from core load.
If you need system info to be shown without hanging on hostname, that's
something you can discuss as an enhancement to that handler, or you could even
write your own handler which avoids these lookups. But in any case, it
shouldn't hold up core load..
Omit reverse lookups in SystemInfoHandler
-
Key: SOLR-7884
URL: https://issues.apache.org/jira/browse/SOLR-7884
Project: Solr
Issue Type: Improvement
Affects Versions: 5.2.1
Reporter: Robert Krüger
Attachments: SOLR-7884.patch
The method SystemInfoHandler.init contains this code:
{code:java}
try {
InetAddress addr = InetAddress.getLocalHost();
hostname = addr.getCanonicalHostName();
} catch (UnknownHostException e) {
//default to null
}
}
{code}
The call to getCanonicalHostName triggers a DNS reverse lookup, that on Mac
OSX times out after 30 seconds if the current network setup has a problem
with reverse lookup and then returns the IP address as host name. In our
product this leads to a hang of 2x30 seconds (one for the SystemInfoHandler
of the container and another one for that of the core) during startup under
these conditions. Unfortunately I have found no way to make this reproducible
by simulating it as it depends on local network config, os and whatever else.
As Solr is shipped bundled with a desktop application, there is currently no
workaround because the user network configuration is beyond our reach.
The suggested fix for this is to replace the call to getCanonicalHostName by
one to getHostName(), which is done for exactly this purpose (obtaining a
host string for logging) in the popular Logback framework (see
http://logback.qos.ch/xref/ch/qos/logback/core/util/ContextUtil.html, how
they do it).
Otherwise Solr has been working perfectly in this setup (bundled with a
desktop app) for a long time and it would be great to remove that last
obstacle to make it a nicer citizen bundlingwise.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-5872) Eliminate overseer queue
[ https://issues.apache.org/jira/browse/SOLR-5872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14700181#comment-14700181 ] Ramkumar Aiyengar commented on SOLR-5872: - (1) sounds like a good idea.. On (2), to either have people decide on the approach, or have Solr do it, we would need to know the perf characteristics of both approaches. So maintaining two implementations or not really comes down to what we are trading off against. Again, only some serious benchmarking can answer that.. Eliminate overseer queue - Key: SOLR-5872 URL: https://issues.apache.org/jira/browse/SOLR-5872 Project: Solr Issue Type: Improvement Components: SolrCloud Reporter: Noble Paul Assignee: Noble Paul The overseer queue is one of the busiest points in the entire system. The raison d'être of the queue is * Provide batching of operations for the main clusterstate,json so that state updates are minimized * Avoid race conditions and ensure order Now , as we move the individual collection states out of the main clusterstate.json, the batching is not useful anymore. Race conditions can easily be solved by using a compare and set in Zookeeper. The proposed solution is , whenever an operation is required to be performed on the clusterstate, the same thread (and of course the same JVM) # read the fresh state and version of zk node # construct the new state # perform a compare and set # if compare and set fails go to step 1 This should be limited to all operations performed on external collections because batching would be required for others -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-7932) Solr replication relies on timestamps to sync across machines
[
https://issues.apache.org/jira/browse/SOLR-7932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14698653#comment-14698653
]
Ramkumar Aiyengar commented on SOLR-7932:
-
Thanks [~varunthacker], that's what I would have thought, good to have
confirmation. Also good point regarding all this being applicable only to
master-slave replication.
[~elyograg], I can see why detecting this would be good for log analysis, could
you raise a separate ticket for this? I would like to keep this to solve the
replication issue.
Solr replication relies on timestamps to sync across machines
-
Key: SOLR-7932
URL: https://issues.apache.org/jira/browse/SOLR-7932
Project: Solr
Issue Type: Bug
Components: replication (java)
Reporter: Ramkumar Aiyengar
Spinning off SOLR-7859, noticed there that wall time recorded as commit data
on a commit to check if replication needs to be done. In IndexFetcher, there
is this code:
{code}
if (!forceReplication
IndexDeletionPolicyWrapper.getCommitTimestamp(commit) == latestVersion) {
//master and slave are already in sync just return
LOG.info(Slave in sync with master.);
successfulInstall = true;
return true;
}
{code}
It appears as if we are checking wall times across machines to check if we
are in sync, this could go wrong.
Once a decision is made to replicate, we do seem to use generations instead,
except for this place below checks both generations and timestamps to see if
a full copy is needed..
{code}
// if the generation of master is older than that of the slave , it
means they are not compatible to be copied
// then a new index directory to be created and all the files need to
be copied
boolean isFullCopyNeeded = IndexDeletionPolicyWrapper
.getCommitTimestamp(commit) = latestVersion
|| commit.getGeneration() = latestGeneration || forceReplication;
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-7932) Solr replication relies on timestamps to sync across machines
[
https://issues.apache.org/jira/browse/SOLR-7932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ramkumar Aiyengar updated SOLR-7932:
Attachment: SOLR-7932.patch
Attached is a patch to remove the use of versions. This however fails
{{TestReplicationHandler}} tests, need to dig into that further..
Solr replication relies on timestamps to sync across machines
-
Key: SOLR-7932
URL: https://issues.apache.org/jira/browse/SOLR-7932
Project: Solr
Issue Type: Bug
Components: replication (java)
Reporter: Ramkumar Aiyengar
Attachments: SOLR-7932.patch
Spinning off SOLR-7859, noticed there that wall time recorded as commit data
on a commit to check if replication needs to be done. In IndexFetcher, there
is this code:
{code}
if (!forceReplication
IndexDeletionPolicyWrapper.getCommitTimestamp(commit) == latestVersion) {
//master and slave are already in sync just return
LOG.info(Slave in sync with master.);
successfulInstall = true;
return true;
}
{code}
It appears as if we are checking wall times across machines to check if we
are in sync, this could go wrong.
Once a decision is made to replicate, we do seem to use generations instead,
except for this place below checks both generations and timestamps to see if
a full copy is needed..
{code}
// if the generation of master is older than that of the slave , it
means they are not compatible to be copied
// then a new index directory to be created and all the files need to
be copied
boolean isFullCopyNeeded = IndexDeletionPolicyWrapper
.getCommitTimestamp(commit) = latestVersion
|| commit.getGeneration() = latestGeneration || forceReplication;
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Resolved] (SOLR-7859) Clamp down on use of System.currentTimeMillis
[ https://issues.apache.org/jira/browse/SOLR-7859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ramkumar Aiyengar resolved SOLR-7859. - Resolution: Fixed Fix Version/s: 5.4 Clamp down on use of System.currentTimeMillis - Key: SOLR-7859 URL: https://issues.apache.org/jira/browse/SOLR-7859 Project: Solr Issue Type: Improvement Reporter: Ramkumar Aiyengar Assignee: Ramkumar Aiyengar Fix For: 5.4 Attachments: SOLR-7859.patch, SOLR-7859.patch, SOLR-7859.patch We did one round of this in SOLR-5734, but more places seem to keep cropping up. We should do one more round, and start whitelisting places which really need it using forbidden-apis. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-5043) hostanme lookup in SystemInfoHandler should be refactored to not block core (re)load
[
https://issues.apache.org/jira/browse/SOLR-5043?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ramkumar Aiyengar updated SOLR-5043:
Attachment: SOLR-5043-lazy.patch
Attached is a patch describing what I meant (untested)
hostanme lookup in SystemInfoHandler should be refactored to not block core
(re)load
Key: SOLR-5043
URL: https://issues.apache.org/jira/browse/SOLR-5043
Project: Solr
Issue Type: Improvement
Reporter: Hoss Man
Attachments: SOLR-5043-lazy.patch, SOLR-5043.patch
SystemInfoHandler currently lookups the hostname of the machine on it's init,
and caches for it's lifecycle -- there is a comment to the effect that the
reason for this is because on some machines (notably ones with wacky DNS
settings) looking up the hostname can take a long ass time in some JVMs...
{noformat}
// on some platforms, resolving canonical hostname can cause the thread
// to block for several seconds if nameservices aren't available
// so resolve this once per handler instance
//(ie: not static, so core reload will refresh)
{noformat}
But as we move forward with a lot more multi-core, solr-cloud, dynamically
updated instances, even paying this cost per core-reload is expensive.
we should refactoring this so that SystemInfoHandler instances init
immediately, with some kind of lazy loading of the hostname info in a
background thread, (especially since hte only real point of having that info
here is for UI use so you cna keep track of what machine you are looking at)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-5043) hostanme lookup in SystemInfoHandler should be refactored to not block core (re)load
[
https://issues.apache.org/jira/browse/SOLR-5043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14698447#comment-14698447
]
Ramkumar Aiyengar commented on SOLR-5043:
-
Having this on a per-instance basis like it does currently would also mean that
you have one more thread running per core, even if temporarily, that might
cause issues if you have lots of cores starting up at the same time in a JVM.
Is it a big deal to just do this lazily and synchronously the first time the
handler is requested? I know it just moves the delay elsewhere, but on the flip
side, with this patch, you might sometimes not get the hostname when you expect
it (so technically it's a functional difference)
hostanme lookup in SystemInfoHandler should be refactored to not block core
(re)load
Key: SOLR-5043
URL: https://issues.apache.org/jira/browse/SOLR-5043
Project: Solr
Issue Type: Improvement
Reporter: Hoss Man
Attachments: SOLR-5043.patch
SystemInfoHandler currently lookups the hostname of the machine on it's init,
and caches for it's lifecycle -- there is a comment to the effect that the
reason for this is because on some machines (notably ones with wacky DNS
settings) looking up the hostname can take a long ass time in some JVMs...
{noformat}
// on some platforms, resolving canonical hostname can cause the thread
// to block for several seconds if nameservices aren't available
// so resolve this once per handler instance
//(ie: not static, so core reload will refresh)
{noformat}
But as we move forward with a lot more multi-core, solr-cloud, dynamically
updated instances, even paying this cost per core-reload is expensive.
we should refactoring this so that SystemInfoHandler instances init
immediately, with some kind of lazy loading of the hostname info in a
background thread, (especially since hte only real point of having that info
here is for UI use so you cna keep track of what machine you are looking at)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-7932) Solr replication relies on timestamps to sync across machines
[
https://issues.apache.org/jira/browse/SOLR-7932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14698475#comment-14698475
]
Ramkumar Aiyengar commented on SOLR-7932:
-
My point was that we shouldn't have to worry about if the clocks are out of
sync.. This (replication) is one place where we do currently rely on clocks
being sync, and my question is if it needs to be. We do have a replication bug
here regardless of how sync'd up the clocks are, it's a race condition waiting
to happen..
Solr replication relies on timestamps to sync across machines
-
Key: SOLR-7932
URL: https://issues.apache.org/jira/browse/SOLR-7932
Project: Solr
Issue Type: Bug
Components: replication (java)
Reporter: Ramkumar Aiyengar
Spinning off SOLR-7859, noticed there that wall time recorded as commit data
on a commit to check if replication needs to be done. In IndexFetcher, there
is this code:
{code}
if (!forceReplication
IndexDeletionPolicyWrapper.getCommitTimestamp(commit) == latestVersion) {
//master and slave are already in sync just return
LOG.info(Slave in sync with master.);
successfulInstall = true;
return true;
}
{code}
It appears as if we are checking wall times across machines to check if we
are in sync, this could go wrong.
Once a decision is made to replicate, we do seem to use generations instead,
except for this place below checks both generations and timestamps to see if
a full copy is needed..
{code}
// if the generation of master is older than that of the slave , it
means they are not compatible to be copied
// then a new index directory to be created and all the files need to
be copied
boolean isFullCopyNeeded = IndexDeletionPolicyWrapper
.getCommitTimestamp(commit) = latestVersion
|| commit.getGeneration() = latestGeneration || forceReplication;
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-7859) Clamp down on use of System.currentTimeMillis
[ https://issues.apache.org/jira/browse/SOLR-7859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14698232#comment-14698232 ] Ramkumar Aiyengar commented on SOLR-7859: - bq. There are also a couple of cases where the usage is suspect, but I haven't got to it as yet. One is around stats, but the more worrying thing is that we use the wall time recorded as commit data on a commit to check if replication needs to be done. In IndexFetcher, there is this code: I have raised SOLR-7932 for this.. Clamp down on use of System.currentTimeMillis - Key: SOLR-7859 URL: https://issues.apache.org/jira/browse/SOLR-7859 Project: Solr Issue Type: Improvement Reporter: Ramkumar Aiyengar Assignee: Ramkumar Aiyengar Attachments: SOLR-7859.patch, SOLR-7859.patch, SOLR-7859.patch We did one round of this in SOLR-5734, but more places seem to keep cropping up. We should do one more round, and start whitelisting places which really need it using forbidden-apis. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Created] (SOLR-7932) Solr replication relies on timestamps to sync across machines
Ramkumar Aiyengar created SOLR-7932:
---
Summary: Solr replication relies on timestamps to sync across
machines
Key: SOLR-7932
URL: https://issues.apache.org/jira/browse/SOLR-7932
Project: Solr
Issue Type: Bug
Components: replication (java)
Reporter: Ramkumar Aiyengar
Spinning off SOLR-7859, noticed there that wall time recorded as commit data on
a commit to check if replication needs to be done. In IndexFetcher, there is
this code:
{code}
if (!forceReplication
IndexDeletionPolicyWrapper.getCommitTimestamp(commit) == latestVersion) {
//master and slave are already in sync just return
LOG.info(Slave in sync with master.);
successfulInstall = true;
return true;
}
{code}
It appears as if we are checking wall times across machines to check if we are
in sync, this could go wrong.
Once a decision is made to replicate, we do seem to use generations instead,
except for this place below checks both generations and timestamps to see if a
full copy is needed..
{code}
// if the generation of master is older than that of the slave , it means
they are not compatible to be copied
// then a new index directory to be created and all the files need to be
copied
boolean isFullCopyNeeded = IndexDeletionPolicyWrapper
.getCommitTimestamp(commit) = latestVersion
|| commit.getGeneration() = latestGeneration || forceReplication;
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-7932) Solr replication relies on timestamps to sync across machines
[
https://issues.apache.org/jira/browse/SOLR-7932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14698390#comment-14698390
]
Ramkumar Aiyengar commented on SOLR-7932:
-
Personally I would rather remove any place relying on absolute timestamps. The
feature you describe IMO doesn't belong to Solr, and there are well known
mechanisms to keep clocks in sync already (ntpd on Linux for example)
Solr replication relies on timestamps to sync across machines
-
Key: SOLR-7932
URL: https://issues.apache.org/jira/browse/SOLR-7932
Project: Solr
Issue Type: Bug
Components: replication (java)
Reporter: Ramkumar Aiyengar
Spinning off SOLR-7859, noticed there that wall time recorded as commit data
on a commit to check if replication needs to be done. In IndexFetcher, there
is this code:
{code}
if (!forceReplication
IndexDeletionPolicyWrapper.getCommitTimestamp(commit) == latestVersion) {
//master and slave are already in sync just return
LOG.info(Slave in sync with master.);
successfulInstall = true;
return true;
}
{code}
It appears as if we are checking wall times across machines to check if we
are in sync, this could go wrong.
Once a decision is made to replicate, we do seem to use generations instead,
except for this place below checks both generations and timestamps to see if
a full copy is needed..
{code}
// if the generation of master is older than that of the slave , it
means they are not compatible to be copied
// then a new index directory to be created and all the files need to
be copied
boolean isFullCopyNeeded = IndexDeletionPolicyWrapper
.getCommitTimestamp(commit) = latestVersion
|| commit.getGeneration() = latestGeneration || forceReplication;
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-5872) Eliminate overseer queue
[ https://issues.apache.org/jira/browse/SOLR-5872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14698435#comment-14698435 ] Ramkumar Aiyengar commented on SOLR-5872: - Though I haven't done serious experiments on this as yet, I see the lack of batching in stateFormat=2 is a potential blocker to it's adoption. We need some benchmarks on a single collection with lots of cores (at least 1000), and see how it works with stateFormat=1, stateFormat=2, and this new approach. Keep in mind that hundreds of cores might change state at the same time, that's the real benefit to batching. I fear that without a batching approach, the system might choke due to the contention at that point. Eliminate overseer queue - Key: SOLR-5872 URL: https://issues.apache.org/jira/browse/SOLR-5872 Project: Solr Issue Type: Improvement Components: SolrCloud Reporter: Noble Paul Assignee: Noble Paul The overseer queue is one of the busiest points in the entire system. The raison d'être of the queue is * Provide batching of operations for the main clusterstate,json so that state updates are minimized * Avoid race conditions and ensure order Now , as we move the individual collection states out of the main clusterstate.json, the batching is not useful anymore. Race conditions can easily be solved by using a compare and set in Zookeeper. The proposed solution is , whenever an operation is required to be performed on the clusterstate, the same thread (and of course the same JVM) # read the fresh state and version of zk node # construct the new state # perform a compare and set # if compare and set fails go to step 1 This should be limited to all operations performed on external collections because batching would be required for others -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Comment Edited] (SOLR-5872) Eliminate overseer queue
[ https://issues.apache.org/jira/browse/SOLR-5872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14698435#comment-14698435 ] Ramkumar Aiyengar edited comment on SOLR-5872 at 8/15/15 8:29 PM: -- Though I haven't done serious experiments on this as yet, I see the lack of batching in stateFormat=2 is a potential blocker to it's adoption. We need some benchmarks on a single collection with lots of cores (at least 1000), and see how it works with stateFormat=1, stateFormat=2, and this new approach. Keep in mind that hundreds of cores might change state at the same time, that's the real benefit to batching. I fear that without a batching approach, the system might choke due to the contention at that point. My point here being that stateFormat=2 not doing batching isn't a convincing enough argument to eliminate overseer queue, may be the effort should be directed more towards getting batching for stateFormat=2 if that's more useful. was (Author: andyetitmoves): Though I haven't done serious experiments on this as yet, I see the lack of batching in stateFormat=2 is a potential blocker to it's adoption. We need some benchmarks on a single collection with lots of cores (at least 1000), and see how it works with stateFormat=1, stateFormat=2, and this new approach. Keep in mind that hundreds of cores might change state at the same time, that's the real benefit to batching. I fear that without a batching approach, the system might choke due to the contention at that point. Eliminate overseer queue - Key: SOLR-5872 URL: https://issues.apache.org/jira/browse/SOLR-5872 Project: Solr Issue Type: Improvement Components: SolrCloud Reporter: Noble Paul Assignee: Noble Paul The overseer queue is one of the busiest points in the entire system. The raison d'être of the queue is * Provide batching of operations for the main clusterstate,json so that state updates are minimized * Avoid race conditions and ensure order Now , as we move the individual collection states out of the main clusterstate.json, the batching is not useful anymore. Race conditions can easily be solved by using a compare and set in Zookeeper. The proposed solution is , whenever an operation is required to be performed on the clusterstate, the same thread (and of course the same JVM) # read the fresh state and version of zk node # construct the new state # perform a compare and set # if compare and set fails go to step 1 This should be limited to all operations performed on external collections because batching would be required for others -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-6737) Add DecimalDigitFilter
[ https://issues.apache.org/jira/browse/LUCENE-6737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14697719#comment-14697719 ] Ramkumar Aiyengar commented on LUCENE-6737: --- ICU folding does this right? This patch is still useful even if so, in case you don't want to do the full folding, or don't want to use ICU, just curious really.. Add DecimalDigitFilter -- Key: LUCENE-6737 URL: https://issues.apache.org/jira/browse/LUCENE-6737 Project: Lucene - Core Issue Type: New Feature Reporter: Robert Muir Fix For: Trunk, 5.4 Attachments: LUCENE-6737.patch TokenFilter that folds all unicode digits (http://unicode.org/cldr/utility/list-unicodeset.jsp?a=[:General_Category=Decimal_Number:]) to 0-9. Historically a lot of the impacted analyzers couldn't even tokenize numbers at all, but now they use standardtokenizer for numbers/alphanum tokens. But its usually the case you will find e.g. a mix of both ascii digits and native digits, and today that makes searching difficult. Note this only impacts *decimal* digits, hence the name DecimalDigitFilter. So no processing of chinese numerals or anything crazy like that. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Created] (SOLR-7910) Build SimplePostTool as a separate module outside solr-core
Ramkumar Aiyengar created SOLR-7910: --- Summary: Build SimplePostTool as a separate module outside solr-core Key: SOLR-7910 URL: https://issues.apache.org/jira/browse/SOLR-7910 Project: Solr Issue Type: Bug Reporter: Ramkumar Aiyengar Priority: Minor Spun off SOLR-7859. Currently SimplePostTool is a part of solr-core, but has restrictions on the dependencies it can use (deps need to be added to bin/post, post.jar should be usable by itself etc.) which are not enforced in any way. SOLR-7901 is for adding some tests for this purpose, but really this should be checked at compile-time by the build system by needing dependencies to be whitelisted (if at all). -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-7859) Clamp down on use of System.currentTimeMillis
[ https://issues.apache.org/jira/browse/SOLR-7859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14680499#comment-14680499 ] Ramkumar Aiyengar commented on SOLR-7859: - [~ehatcher], I have already committed the RTimer refactoring, so the SolrJ dep is no longer needed. The addition of RTimer to post.jar is a one line fix, which I will soon add, so no reversion should be necessary (would be simpler than that really). I have raised SOLR-7910 for the modularization, that applies even if post.cmd is created, as we can't assume that all the dependencies of solr-core are available to use. Clamp down on use of System.currentTimeMillis - Key: SOLR-7859 URL: https://issues.apache.org/jira/browse/SOLR-7859 Project: Solr Issue Type: Improvement Reporter: Ramkumar Aiyengar Assignee: Ramkumar Aiyengar Attachments: SOLR-7859.patch, SOLR-7859.patch, SOLR-7859.patch We did one round of this in SOLR-5734, but more places seem to keep cropping up. We should do one more round, and start whitelisting places which really need it using forbidden-apis. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Resolved] (SOLR-7902) Split out use of child timers from RTimer to a sub-class
[ https://issues.apache.org/jira/browse/SOLR-7902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ramkumar Aiyengar resolved SOLR-7902. - Resolution: Fixed Fix Version/s: 5.4 Split out use of child timers from RTimer to a sub-class Key: SOLR-7902 URL: https://issues.apache.org/jira/browse/SOLR-7902 Project: Solr Issue Type: Improvement Reporter: Ramkumar Aiyengar Assignee: Ramkumar Aiyengar Priority: Minor Fix For: 5.4 RTimer is now heavily used by Solr, but it's sub-timer functionality is limited to one place, factor it out to a separate class to make it more lightweight and reduce its dependencies. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-7859) Clamp down on use of System.currentTimeMillis
[ https://issues.apache.org/jira/browse/SOLR-7859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14680841#comment-14680841 ] Ramkumar Aiyengar commented on SOLR-7859: - Thanks for the offer Erik. This commit should hopefully address Uwe's post.jar concern. I have also now reverted your bin/post commit as it should not be needed any longer (and per my smoke test with the tool), but let me know if you see otherwise.. Clamp down on use of System.currentTimeMillis - Key: SOLR-7859 URL: https://issues.apache.org/jira/browse/SOLR-7859 Project: Solr Issue Type: Improvement Reporter: Ramkumar Aiyengar Assignee: Ramkumar Aiyengar Attachments: SOLR-7859.patch, SOLR-7859.patch, SOLR-7859.patch We did one round of this in SOLR-5734, but more places seem to keep cropping up. We should do one more round, and start whitelisting places which really need it using forbidden-apis. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-7859) Clamp down on use of System.currentTimeMillis
[ https://issues.apache.org/jira/browse/SOLR-7859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14679145#comment-14679145 ] Ramkumar Aiyengar commented on SOLR-7859: - Thanks Erik! Clamp down on use of System.currentTimeMillis - Key: SOLR-7859 URL: https://issues.apache.org/jira/browse/SOLR-7859 Project: Solr Issue Type: Improvement Reporter: Ramkumar Aiyengar Assignee: Ramkumar Aiyengar Attachments: SOLR-7859.patch, SOLR-7859.patch, SOLR-7859.patch We did one round of this in SOLR-5734, but more places seem to keep cropping up. We should do one more round, and start whitelisting places which really need it using forbidden-apis. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Created] (SOLR-7902) Split out use of child timers from RTimer to a sub-class
Ramkumar Aiyengar created SOLR-7902: --- Summary: Split out use of child timers from RTimer to a sub-class Key: SOLR-7902 URL: https://issues.apache.org/jira/browse/SOLR-7902 Project: Solr Issue Type: Improvement Reporter: Ramkumar Aiyengar Assignee: Ramkumar Aiyengar Priority: Minor RTimer is now heavily used by Solr, but it's sub-timer functionality is limited to one place, factor it out to a separate class to make it more lightweight and reduce its dependencies. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-7859) Clamp down on use of System.currentTimeMillis
[ https://issues.apache.org/jira/browse/SOLR-7859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14679347#comment-14679347 ] Ramkumar Aiyengar commented on SOLR-7859: - bq.let SimplePostTool use System.currentTimeMillies, add @SuppressForbidden You can use nanoTime without RTimer anyway. But hold off in any case.. I am anyway making a change to RTimer to break the NamedList dependency, RTimer is now used in many more places, and the child timer functionality is used in only one place. Clamp down on use of System.currentTimeMillis - Key: SOLR-7859 URL: https://issues.apache.org/jira/browse/SOLR-7859 Project: Solr Issue Type: Improvement Reporter: Ramkumar Aiyengar Assignee: Ramkumar Aiyengar Attachments: SOLR-7859.patch, SOLR-7859.patch, SOLR-7859.patch We did one round of this in SOLR-5734, but more places seem to keep cropping up. We should do one more round, and start whitelisting places which really need it using forbidden-apis. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-7859) Clamp down on use of System.currentTimeMillis
[ https://issues.apache.org/jira/browse/SOLR-7859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14679363#comment-14679363 ] Ramkumar Aiyengar commented on SOLR-7859: - Created SOLR-7902 to split out RTimer. Clamp down on use of System.currentTimeMillis - Key: SOLR-7859 URL: https://issues.apache.org/jira/browse/SOLR-7859 Project: Solr Issue Type: Improvement Reporter: Ramkumar Aiyengar Assignee: Ramkumar Aiyengar Attachments: SOLR-7859.patch, SOLR-7859.patch, SOLR-7859.patch We did one round of this in SOLR-5734, but more places seem to keep cropping up. We should do one more round, and start whitelisting places which really need it using forbidden-apis. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Comment Edited] (SOLR-7859) Clamp down on use of System.currentTimeMillis
[
https://issues.apache.org/jira/browse/SOLR-7859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14679415#comment-14679415
]
Ramkumar Aiyengar edited comment on SOLR-7859 at 8/10/15 12:05 AM:
---
Committed SOLR-7902. I haven't tried as yet, but we should be able to revert
the {{bin/post}} change (without any reversion/changes to SimplePostTool, it
can now use RTimer without additional deps).
was (Author: andyetitmoves):
Committed SOLR-7902. I haven't tried as yet, but we should be able to revert
the post.jar change..
Clamp down on use of System.currentTimeMillis
-
Key: SOLR-7859
URL: https://issues.apache.org/jira/browse/SOLR-7859
Project: Solr
Issue Type: Improvement
Reporter: Ramkumar Aiyengar
Assignee: Ramkumar Aiyengar
Attachments: SOLR-7859.patch, SOLR-7859.patch, SOLR-7859.patch
We did one round of this in SOLR-5734, but more places seem to keep cropping
up. We should do one more round, and start whitelisting places which really
need it using forbidden-apis.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-7859) Clamp down on use of System.currentTimeMillis
[ https://issues.apache.org/jira/browse/SOLR-7859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14679415#comment-14679415 ] Ramkumar Aiyengar commented on SOLR-7859: - Committed SOLR-7902. I haven't tried as yet, but we should be able to revert the post.jar change.. Clamp down on use of System.currentTimeMillis - Key: SOLR-7859 URL: https://issues.apache.org/jira/browse/SOLR-7859 Project: Solr Issue Type: Improvement Reporter: Ramkumar Aiyengar Assignee: Ramkumar Aiyengar Attachments: SOLR-7859.patch, SOLR-7859.patch, SOLR-7859.patch We did one round of this in SOLR-5734, but more places seem to keep cropping up. We should do one more round, and start whitelisting places which really need it using forbidden-apis. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-7859) Clamp down on use of System.currentTimeMillis
[ https://issues.apache.org/jira/browse/SOLR-7859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14679432#comment-14679432 ] Ramkumar Aiyengar commented on SOLR-7859: - On second thought, I might still need to change the post.jar generation in solr/build.xml to include RTimer.class to satisfy Uwe's requirement. (If this is really a requirement, ideally SPT should move outside solr-core to a separate place where this requirement can be verified at compile time, though I find my build-system-fu lacking to do that..) Clamp down on use of System.currentTimeMillis - Key: SOLR-7859 URL: https://issues.apache.org/jira/browse/SOLR-7859 Project: Solr Issue Type: Improvement Reporter: Ramkumar Aiyengar Assignee: Ramkumar Aiyengar Attachments: SOLR-7859.patch, SOLR-7859.patch, SOLR-7859.patch We did one round of this in SOLR-5734, but more places seem to keep cropping up. We should do one more round, and start whitelisting places which really need it using forbidden-apis. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-7859) Clamp down on use of System.currentTimeMillis
[ https://issues.apache.org/jira/browse/SOLR-7859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ramkumar Aiyengar updated SOLR-7859: Attachment: SOLR-7859.patch Latest version, I will commit this soon before this goes out of date as well.. Clamp down on use of System.currentTimeMillis - Key: SOLR-7859 URL: https://issues.apache.org/jira/browse/SOLR-7859 Project: Solr Issue Type: Improvement Reporter: Ramkumar Aiyengar Assignee: Ramkumar Aiyengar Attachments: SOLR-7859.patch, SOLR-7859.patch, SOLR-7859.patch We did one round of this in SOLR-5734, but more places seem to keep cropping up. We should do one more round, and start whitelisting places which really need it using forbidden-apis. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-7855) OverseerCollectionProcessor: separate general task management from collection message handling
[ https://issues.apache.org/jira/browse/SOLR-7855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14663117#comment-14663117 ] Ramkumar Aiyengar commented on SOLR-7855: - [~gchanan], SOLR-7766 is now done, any plans to commit this soon to branch_5x? Trying to avoid conflicts with SOLR-7859.. OverseerCollectionProcessor: separate general task management from collection message handling -- Key: SOLR-7855 URL: https://issues.apache.org/jira/browse/SOLR-7855 Project: Solr Issue Type: Bug Components: SolrCloud Reporter: Gregory Chanan Assignee: Gregory Chanan Attachments: SOLR-7855.patch, SOLR-7855.patch, SOLR-7855.patch While working on SOLR-7789, I realized it would be easier if I split up the OverseerCollectionProcessor into two parts: 1) General handling of tasks (work queues, etc) 2) Processing a collection handler request I haven't decided whether the ConfigSet should have its own processor, i.e. OverseerConfigSetProcessor or reuse at least the thread for the OverseerCollectionProcessor, but in either case this refactoring will be helpful. That is, if the ConfigSet processing has its own processing, I can reuse the general handling of tasks part. If the ConfigSet processing reuses the OverseerCollectionProcessing thread, I won't complicate the implementation with ConfigSet operations. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-6273) Cross Data Center Replication
[ https://issues.apache.org/jira/browse/SOLR-6273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14663130#comment-14663130 ] Ramkumar Aiyengar commented on SOLR-6273: - [~erickerickson], any plans to move this to branch_5x soon? I am aware that this needs to be bedded a bit, so no big deal either way, but if you follow my merge for SOLR-7859, you might have to merge in [a change to CdcrReplicatorState|https://svn.apache.org/viewvc/lucene/dev/trunk/solr/core/src/java/org/apache/solr/handler/CdcrReplicatorState.java?r1=1694798r2=1694797pathrev=1694798] to avoid failing on forbidden-apis/precommit. Cross Data Center Replication - Key: SOLR-6273 URL: https://issues.apache.org/jira/browse/SOLR-6273 Project: Solr Issue Type: New Feature Reporter: Yonik Seeley Assignee: Erick Erickson Attachments: SOLR-6273-trunk-testfix1.patch, SOLR-6273-trunk-testfix2.patch, SOLR-6273-trunk-testfix3.patch, SOLR-6273-trunk.patch, SOLR-6273-trunk.patch, SOLR-6273.patch, SOLR-6273.patch, SOLR-6273.patch, SOLR-6273.patch This is the master issue for Cross Data Center Replication (CDCR) described at a high level here: http://heliosearch.org/solr-cross-data-center-replication/ -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Comment Edited] (SOLR-6273) Cross Data Center Replication
[ https://issues.apache.org/jira/browse/SOLR-6273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14663130#comment-14663130 ] Ramkumar Aiyengar edited comment on SOLR-6273 at 8/8/15 7:15 PM: - [~erickerickson], any plans to move this to branch_5x soon? I am aware that this needs to be bedded in a bit, so no big deal either way, but if you follow my merge for SOLR-7859, you might have to merge in [a change to CdcrReplicatorState|https://svn.apache.org/viewvc/lucene/dev/trunk/solr/core/src/java/org/apache/solr/handler/CdcrReplicatorState.java?r1=1694798r2=1694797pathrev=1694798] to avoid failing on forbidden-apis/precommit. was (Author: andyetitmoves): [~erickerickson], any plans to move this to branch_5x soon? I am aware that this needs to be bedded a bit, so no big deal either way, but if you follow my merge for SOLR-7859, you might have to merge in [a change to CdcrReplicatorState|https://svn.apache.org/viewvc/lucene/dev/trunk/solr/core/src/java/org/apache/solr/handler/CdcrReplicatorState.java?r1=1694798r2=1694797pathrev=1694798] to avoid failing on forbidden-apis/precommit. Cross Data Center Replication - Key: SOLR-6273 URL: https://issues.apache.org/jira/browse/SOLR-6273 Project: Solr Issue Type: New Feature Reporter: Yonik Seeley Assignee: Erick Erickson Attachments: SOLR-6273-trunk-testfix1.patch, SOLR-6273-trunk-testfix2.patch, SOLR-6273-trunk-testfix3.patch, SOLR-6273-trunk.patch, SOLR-6273-trunk.patch, SOLR-6273.patch, SOLR-6273.patch, SOLR-6273.patch, SOLR-6273.patch This is the master issue for Cross Data Center Replication (CDCR) described at a high level here: http://heliosearch.org/solr-cross-data-center-replication/ -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-7877) TestAuthenticationFramework.testBasics to preserve/restore the original request(Username|Password)
[
https://issues.apache.org/jira/browse/SOLR-7877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14660786#comment-14660786
]
Ramkumar Aiyengar commented on SOLR-7877:
-
I agree, I think in general, deriving tests is a bad idea. I once pulled out
`test` virtual method for this very reason from AbstractZkTestCase.. If many
tests extend a particular test suite, then they should be changed to not do it.
TestAuthenticationFramework.testBasics to preserve/restore the original
request(Username|Password)
--
Key: SOLR-7877
URL: https://issues.apache.org/jira/browse/SOLR-7877
Project: Solr
Issue Type: Bug
Affects Versions: 5.3, Trunk
Reporter: Noble Paul
Assignee: Christine Poerschke
Priority: Blocker
Attachments: SOLR-7877.patch
{code}
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/13742/
Java: 32bit/jdk1.8.0_60-ea-b24 -server -XX:+UseG1GC
1 tests failed.
FAILED:
org.apache.solr.cloud.TestAuthenticationFramework.testCollectionCreateWithoutCoresThenDelete
Error Message:
Error from server at http://127.0.0.1:51573/solr: Expected mime type
application/octet-stream but got text/html. html head meta
http-equiv=Content-Type content=text/html;charset=ISO-8859-1/
titleError 401 /title /head body h2HTTP ERROR: 401/h2 pProblem
accessing /solr/admin/collections. Reason: preUnauthorized
request/pre/p hr /ismallPowered by Jetty:///small/i /body
/html
Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error
from server at http://127.0.0.1:51573/solr: Expected mime type
application/octet-stream but got text/html. html
head
meta http-equiv=Content-Type content=text/html;charset=ISO-8859-1/
titleError 401 /title
/head
body
h2HTTP ERROR: 401/h2
pProblem accessing /solr/admin/collections. Reason:
preUnauthorized request/pre/p
hr /ismallPowered by Jetty:///small/i
/body
/html
at
__randomizedtesting.SeedInfo.seed([A454441B503006EB:17918BDA5F48D5AA]:0)
at
org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:528)
at
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:234)
at
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:226)
at
org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:376)
at
org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:328)
at
org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1086)
at
org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:856)
at
org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:799)
at
org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1220)
at
org.apache.solr.cloud.MiniSolrCloudCluster.makeCollectionsRequest(MiniSolrCloudCluster.java:349)
at
org.apache.solr.cloud.MiniSolrCloudCluster.createCollection(MiniSolrCloudCluster.java:333)
at
org.apache.solr.cloud.TestMiniSolrCloudCluster.createCollection(TestMiniSolrCloudCluster.java:115)
at
org.apache.solr.cloud.TestMiniSolrCloudCluster.testCollectionCreateWithoutCoresThenDelete(TestMiniSolrCloudCluster.java:298)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-7859) Clamp down on use of System.currentTimeMillis
[ https://issues.apache.org/jira/browse/SOLR-7859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14653491#comment-14653491 ] Ramkumar Aiyengar commented on SOLR-7859: - Updated patch, now fixes a few places, removes use of core open timestamp for caching purposes in join queries in favour of nanoTime. Clamp down on use of System.currentTimeMillis - Key: SOLR-7859 URL: https://issues.apache.org/jira/browse/SOLR-7859 Project: Solr Issue Type: Improvement Reporter: Ramkumar Aiyengar Attachments: SOLR-7859.patch We did one round of this in SOLR-5734, but more places seem to keep cropping up. We should do one more round, and start whitelisting places which really need it using forbidden-apis. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Issue Comment Deleted] (SOLR-7859) Clamp down on use of System.currentTimeMillis
[ https://issues.apache.org/jira/browse/SOLR-7859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ramkumar Aiyengar updated SOLR-7859: Comment: was deleted (was: Updated patch, now fixes a few places, removes use of core open timestamp for caching purposes in join queries in favour of nanoTime.) Clamp down on use of System.currentTimeMillis - Key: SOLR-7859 URL: https://issues.apache.org/jira/browse/SOLR-7859 Project: Solr Issue Type: Improvement Reporter: Ramkumar Aiyengar Attachments: SOLR-7859.patch We did one round of this in SOLR-5734, but more places seem to keep cropping up. We should do one more round, and start whitelisting places which really need it using forbidden-apis. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-7859) Clamp down on use of System.currentTimeMillis
[ https://issues.apache.org/jira/browse/SOLR-7859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ramkumar Aiyengar updated SOLR-7859: Attachment: SOLR-7859.patch Updated patch, now fixes a few more places, removes use of core open timestamp for caching purposes in join queries in favour of nanoTime. Clamp down on use of System.currentTimeMillis - Key: SOLR-7859 URL: https://issues.apache.org/jira/browse/SOLR-7859 Project: Solr Issue Type: Improvement Reporter: Ramkumar Aiyengar Attachments: SOLR-7859.patch, SOLR-7859.patch We did one round of this in SOLR-5734, but more places seem to keep cropping up. We should do one more round, and start whitelisting places which really need it using forbidden-apis. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Assigned] (SOLR-7859) Clamp down on use of System.currentTimeMillis
[ https://issues.apache.org/jira/browse/SOLR-7859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ramkumar Aiyengar reassigned SOLR-7859: --- Assignee: Ramkumar Aiyengar Clamp down on use of System.currentTimeMillis - Key: SOLR-7859 URL: https://issues.apache.org/jira/browse/SOLR-7859 Project: Solr Issue Type: Improvement Reporter: Ramkumar Aiyengar Assignee: Ramkumar Aiyengar Attachments: SOLR-7859.patch, SOLR-7859.patch We did one round of this in SOLR-5734, but more places seem to keep cropping up. We should do one more round, and start whitelisting places which really need it using forbidden-apis. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-7860) Standardize use of HttpSolrClients in Solr
[
https://issues.apache.org/jira/browse/SOLR-7860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14654747#comment-14654747
]
Ramkumar Aiyengar commented on SOLR-7860:
-
This is a good start Ishan! There's also IndexFetcher which has it's own http
client..
There is one question still here which I am in two minds about myself. One
thing we do get by separate select and update httpclients is that you can
configure limits/parallelism on each one separately. By extension, should we be
aiming to consolidate the clients or just standardizing clients, for example,
by just providing a client factory which can provide different clients for
different kinds of operations within Solr, and have an unified configuration?
So, the configuration describes different clients for different purposes with
different tunables (with defaults obviously), and we instantiate a factory with
that config, which can then hand out a client for anything which needs one for
one of these purposes?
Standardize use of HttpSolrClients in Solr
--
Key: SOLR-7860
URL: https://issues.apache.org/jira/browse/SOLR-7860
Project: Solr
Issue Type: Improvement
Reporter: Ramkumar Aiyengar
Priority: Minor
Attachments: SOLR-7860.patch
{{HttpShardHandlerFactory}} and {{UpdateShardHandler}} already provide two
places where one can get {{HttpSolrClient}}s with timeouts set up properly
etc., but there are lots of miscellaneous places which instantiate their own,
with hardcoded timeouts. These are just waiting for some environment to
realize the timeouts are not suitable, and not configurable as well. We
should standardize this (anyone knows why there two to begin with?), ideally
this hardcoding shouldn't exist at all..
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Comment Edited] (SOLR-7859) Clamp down on use of System.currentTimeMillis
[
https://issues.apache.org/jira/browse/SOLR-7859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14652972#comment-14652972
]
Ramkumar Aiyengar edited comment on SOLR-7859 at 8/4/15 2:38 AM:
-
Attached is a patch to remove most occurrences in solr code (boy there are a
few!) and SuppressForbidden where the usage is legitimate.
There are also a couple of cases where the usage is suspect, but I haven't got
to it as yet. One is around stats, but the more worrying thing is that we use
the wall time recorded as commit data on a commit to check if replication needs
to be done. In IndexFetcher, there is this code:
{code}
if (!forceReplication
IndexDeletionPolicyWrapper.getCommitTimestamp(commit) == latestVersion) {
//master and slave are already in sync just return
LOG.info(Slave in sync with master.);
successfulInstall = true;
return true;
}
{code}
We are checking wall times across machines to check if we are in sync? That
sounds wrong.. Or I am mistaken here? Why can't we just check generations?
Another place below checks both generations and timestamps to see if a full
copy is needed..
{code}
// if the generation of master is older than that of the slave , it means
they are not compatible to be copied
// then a new index directory to be created and all the files need to be
copied
boolean isFullCopyNeeded = IndexDeletionPolicyWrapper
.getCommitTimestamp(commit) = latestVersion
|| commit.getGeneration() = latestGeneration || forceReplication;
{code}
was (Author: andyetitmoves):
Attached is a patch to remove most occurrences in solr code (boy there are a
few!) and SuppressForbidden where the usage is legitimate.
There are also a couple of cases where the usage is suspect, but I haven't got
to it as yet. One is around stats, but the more worrying thing is that we use
the wall time recorded as commit data on a commit to check if replication needs
to be done. In IndexFetcher, there is this code:
{code}
if (!forceReplication
IndexDeletionPolicyWrapper.getCommitTimestamp(commit) == latestVersion) {
//master and slave are already in sync just return
LOG.info(Slave in sync with master.);
successfulInstall = true;
return true;
}
{code}
We are checking wall times across machines to check if we are in sync, that
sounds wrong. Or I am mistaken here?
Clamp down on use of System.currentTimeMillis
-
Key: SOLR-7859
URL: https://issues.apache.org/jira/browse/SOLR-7859
Project: Solr
Issue Type: Improvement
Reporter: Ramkumar Aiyengar
Attachments: SOLR-7859.patch
We did one round of this in SOLR-5734, but more places seem to keep cropping
up. We should do one more round, and start whitelisting places which really
need it using forbidden-apis.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-7859) Clamp down on use of System.currentTimeMillis
[
https://issues.apache.org/jira/browse/SOLR-7859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ramkumar Aiyengar updated SOLR-7859:
Attachment: SOLR-7859.patch
Attached is a patch to remove most occurrences in solr code (boy there are a
few!) and SuppressForbidden where the usage is legitimate.
There are also a couple of cases where the usage is suspect, but I haven't got
to it as yet. One is around stats, but the more worrying thing is that we use
the wall time recorded as commit data on a commit to check if replication needs
to be done. In IndexFetcher, there is this code:
{code}
if (!forceReplication
IndexDeletionPolicyWrapper.getCommitTimestamp(commit) == latestVersion) {
//master and slave are already in sync just return
LOG.info(Slave in sync with master.);
successfulInstall = true;
return true;
}
{code}
We are checking wall times across machines to check if we are in sync, that
sounds wrong. Or I am mistaken here?
Clamp down on use of System.currentTimeMillis
-
Key: SOLR-7859
URL: https://issues.apache.org/jira/browse/SOLR-7859
Project: Solr
Issue Type: Improvement
Reporter: Ramkumar Aiyengar
Attachments: SOLR-7859.patch
We did one round of this in SOLR-5734, but more places seem to keep cropping
up. We should do one more round, and start whitelisting places which really
need it using forbidden-apis.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-7819) ZkController.ensureReplicaInLeaderInitiatedRecovery does not respect retryOnConnLoss
[
https://issues.apache.org/jira/browse/SOLR-7819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14650674#comment-14650674
]
Ramkumar Aiyengar commented on SOLR-7819:
-
A couple of comments, looks sensible overall..
{code}
log.info(Node + replicaNodeName +
is not live, so skipping leader-initiated recovery for replica:
core={} coreNodeName={},
replicaCoreName, replicaCoreNodeName);
// publishDownState will be false to avoid publishing the down state
too many times
// as many errors can occur together and will each call into this method
(SOLR-6189)
{code}
It goes ahead and does `publishDownState` still if `forcePublishState` is true,
is that intentional? The caller does check for if the replica is live, but
there could a race. Similarly, if our state is suspect due to zk
disconnect/session (the block before this), should the force be respected?
{code}
// if the replica's state is not DOWN right now, make it so ...
// we only really need to try to send the recovery command if the node
itself is live
if
(getZkStateReader().getClusterState().liveNodesContain(replicaNodeName)) {
LeaderInitiatedRecoveryThread lirThread =
{code}
The comment doesn't make sense as the code has moved to LIRT.
ZkController.ensureReplicaInLeaderInitiatedRecovery does not respect
retryOnConnLoss
Key: SOLR-7819
URL: https://issues.apache.org/jira/browse/SOLR-7819
Project: Solr
Issue Type: Bug
Components: SolrCloud
Affects Versions: 5.2, 5.2.1
Reporter: Shalin Shekhar Mangar
Labels: Jepsen
Fix For: 5.3, Trunk
Attachments: SOLR-7819.patch, SOLR-7819.patch
SOLR-7245 added a retryOnConnLoss parameter to
ZkController.ensureReplicaInLeaderInitiatedRecovery so that indexing threads
do not hang during a partition on ZK operations. However, some of those
changes were unintentionally reverted by SOLR-7336 in 5.2.
I found this while running Jepsen tests on 5.2.1 where a hung update managed
to put a leader into a 'down' state (I'm still investigating and will open a
separate issue about this problem).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Created] (SOLR-7859) Clamp down on use of System.currentTimeMillis
Ramkumar Aiyengar created SOLR-7859: --- Summary: Clamp down on use of System.currentTimeMillis Key: SOLR-7859 URL: https://issues.apache.org/jira/browse/SOLR-7859 Project: Solr Issue Type: Improvement Reporter: Ramkumar Aiyengar We did one round of this in SOLR-5734, but more places seem to keep cropping up. We should do one more round, and start whitelisting places which really need it using forbidden-apis. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Created] (SOLR-7860) Standardize use of HttpSolrClients in Solr
Ramkumar Aiyengar created SOLR-7860:
---
Summary: Standardize use of HttpSolrClients in Solr
Key: SOLR-7860
URL: https://issues.apache.org/jira/browse/SOLR-7860
Project: Solr
Issue Type: Improvement
Reporter: Ramkumar Aiyengar
Priority: Minor
{{HttpShardHandlerFactory}} and {{UpdateShardHandler}} already provide two
places where one can get {{HttpSolrClient}}s with timeouts set up properly
etc., but there are lots of miscellaneous places which instantiate their own,
with hardcoded timeouts. These are just waiting for some environment to realize
the timeouts are not suitable, and not configurable as well. We should
standardize this (anyone knows why there two to begin with?), ideally this
hardcoding shouldn't exist at all..
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-7820) IndexFetcher should calculate ahead of time how much space is needed for full snapshot based recovery and cleanly abort instead of trying and running out of space on a n
[ https://issues.apache.org/jira/browse/SOLR-7820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14643329#comment-14643329 ] Ramkumar Aiyengar commented on SOLR-7820: - In Yonik's words.. https://issues.apache.org/jira/browse/SOLR-6460?focusedCommentId=14117807page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14117807 IndexFetcher should calculate ahead of time how much space is needed for full snapshot based recovery and cleanly abort instead of trying and running out of space on a node Key: SOLR-7820 URL: https://issues.apache.org/jira/browse/SOLR-7820 Project: Solr Issue Type: Improvement Components: replication (java) Reporter: Timothy Potter When a replica is trying to recover and it's IndexFetcher decides it needs to pull the full index from a peer (isFullCopyNeeded == true), then the existing index directory should be deleted before the full copy is started to free up disk to pull a fresh index, otherwise the server will potentially need 2x the disk space (old + incoming new). Currently, the IndexFetcher removes the index directory after the new is downloaded; however, once the fetcher decides a full copy is needed, what is the value of the existing index? It's clearly out-of-date and should not serve queries. Since we're deleting data preemptively, maybe this should be an advanced configuration property, only to be used by those that are disk-space constrained (which I'm seeing more and more with people deploying high-end SSDs - they typically don't have 2x the disk capacity required by an index). -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Resolved] (SOLR-7714) reduce SearchHandler's overall use of ShardHandler objects (from N+1+x to just 1)
[ https://issues.apache.org/jira/browse/SOLR-7714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ramkumar Aiyengar resolved SOLR-7714. - Resolution: Fixed reduce SearchHandler's overall use of ShardHandler objects (from N+1+x to just 1) - Key: SOLR-7714 URL: https://issues.apache.org/jira/browse/SOLR-7714 Project: Solr Issue Type: Improvement Reporter: Christine Poerschke Assignee: Ramkumar Aiyengar Priority: Minor Fix For: 5.3 github pull request with proposed change to follow. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-7820) IndexFetcher should delete the current index directory before downloading the new index when isFullCopyNeeded==true
[
https://issues.apache.org/jira/browse/SOLR-7820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14641543#comment-14641543
]
Ramkumar Aiyengar commented on SOLR-7820:
-
I agree there are a few issues here, just that the deleting the current index
just brushes them all under the carpet and adds risk.
- The current default of 100 updates for {{UpdateLog}} is often insufficient
for many cases. I made that number configurable, if it's a few thousand
updates, just tweaking it might work. But {{UpdateLog}} has scaling limitations
I think, so YMMV. I thought {{CdcrUpdateLog}} came about to overcome this
scaling limitation -- but I haven't looked at it enough to know if it can
replace {{UpdateLog}}, perhaps [~erickerickson] or [~ysee...@gmail.com] know..
- The other thing which could vastly improve this situation, even if a full
recovery was needed, was synchronizing commits across replicas, since recovery
skips segments already present in the current index. I believe [~varunthacker]
was looking at this, but I can't find the issue now.
- Regardless, I agree that it would be a good enhancement to calculate ahead
of time how much space is needed for recovery and cleanly abort instead of
trying and running out of space.
IndexFetcher should delete the current index directory before downloading the
new index when isFullCopyNeeded==true
---
Key: SOLR-7820
URL: https://issues.apache.org/jira/browse/SOLR-7820
Project: Solr
Issue Type: Improvement
Components: replication (java)
Reporter: Timothy Potter
When a replica is trying to recover and it's IndexFetcher decides it needs to
pull the full index from a peer (isFullCopyNeeded == true), then the existing
index directory should be deleted before the full copy is started to free up
disk to pull a fresh index, otherwise the server will potentially need 2x the
disk space (old + incoming new). Currently, the IndexFetcher removes the
index directory after the new is downloaded; however, once the fetcher
decides a full copy is needed, what is the value of the existing index? It's
clearly out-of-date and should not serve queries. Since we're deleting data
preemptively, maybe this should be an advanced configuration property, only
to be used by those that are disk-space constrained (which I'm seeing more
and more with people deploying high-end SSDs - they typically don't have 2x
the disk capacity required by an index).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-7820) IndexFetcher should delete the current index directory before downloading the new index when isFullCopyNeeded==true
[ https://issues.apache.org/jira/browse/SOLR-7820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14640204#comment-14640204 ] Ramkumar Aiyengar commented on SOLR-7820: - -0. As Mark mentions, its a scary place to be in already. However advanced you make this, it's still going to be one more optimize like feature. Ooh, saves space, let's turn this on! Yes, it will, but it carries a risk which is not easily quantifiable. We still have cases where the entire cloud can go to recovery. They are rare with all our fixes over time, but they still exist. If you are serious/demanding enough about your search index to procure SSDs, you don't want to wake up finding your index gone just because you were feeling wise with your dollars one day in the past. I am still -0 with this only because I don't foresee any *technical* issues with this, but still doesn't sound like a feature we want to have. IndexFetcher should delete the current index directory before downloading the new index when isFullCopyNeeded==true --- Key: SOLR-7820 URL: https://issues.apache.org/jira/browse/SOLR-7820 Project: Solr Issue Type: Improvement Components: replication (java) Reporter: Timothy Potter When a replica is trying to recover and it's IndexFetcher decides it needs to pull the full index from a peer (isFullCopyNeeded == true), then the existing index directory should be deleted before the full copy is started to free up disk to pull a fresh index, otherwise the server will potentially need 2x the disk space (old + incoming new). Currently, the IndexFetcher removes the index directory after the new is downloaded; however, once the fetcher decides a full copy is needed, what is the value of the existing index? It's clearly out-of-date and should not serve queries. Since we're deleting data preemptively, maybe this should be an advanced configuration property, only to be used by those that are disk-space constrained (which I'm seeing more and more with people deploying high-end SSDs - they typically don't have 2x the disk capacity required by an index). -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-7819) ZkController.ensureReplicaInLeaderInitiatedRecovery does not respect retryOnConnLoss
[ https://issues.apache.org/jira/browse/SOLR-7819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14639404#comment-14639404 ] Ramkumar Aiyengar commented on SOLR-7819: - Duh, this is why we need a good test for this (I gave up after trying a bit in the original ticket), and I need to pay attention to automated merges more. Looks like my initial patch had the change, but when I merged with [your changes|https://svn.apache.org/viewvc?view=revisionrevision=1666825] for SOLR-7109, looks like the local variable use just got removed :( I get your concern, I think we already do this, look at DistributedUpdateProcessor.java around line 883, if we are unable to set the LIR node, we start a thread to keep retrying the node set. We just need to return false in the connection loss case as well, we currently do it only if the node is not live (and hence we didnt even bother setting the node). ZkController.ensureReplicaInLeaderInitiatedRecovery does not respect retryOnConnLoss Key: SOLR-7819 URL: https://issues.apache.org/jira/browse/SOLR-7819 Project: Solr Issue Type: Bug Components: SolrCloud Affects Versions: 5.2, 5.2.1 Reporter: Shalin Shekhar Mangar Labels: Jepsen Fix For: 5.3, Trunk SOLR-7245 added a retryOnConnLoss parameter to ZkController.ensureReplicaInLeaderInitiatedRecovery so that indexing threads do not hang during a partition on ZK operations. However, some of those changes were unintentionally reverted by SOLR-7336 in 5.2. I found this while running Jepsen tests on 5.2.1 where a hung update managed to put a leader into a 'down' state (I'm still investigating and will open a separate issue about this problem). -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-7766) support creation of a coreless collection
[
https://issues.apache.org/jira/browse/SOLR-7766?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ramkumar Aiyengar updated SOLR-7766:
Attachment: SOLR-7766.patch
Attaching a updated patch with conflicts resolved in trunk..
support creation of a coreless collection
-
Key: SOLR-7766
URL: https://issues.apache.org/jira/browse/SOLR-7766
Project: Solr
Issue Type: New Feature
Reporter: Christine Poerschke
Assignee: Ramkumar Aiyengar
Priority: Minor
Attachments: SOLR-7766.patch
By supplying a deliberately empty create node set (via
{{.../solr/admin/collections?action=CREATEcreateNodeSet=name=myCollection...}})
the collection can be created in the usual manner but it will just not yet
contain any cores for any of its shards. Later on
{{.../solr/admin/collections?wt=jsonaction=ADDREPLICAcollection=myCollection...}}
calls can create cores as and when and where required.
github pull request with proposed changes to follow.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Assigned] (SOLR-7766) support creation of a coreless collection
[
https://issues.apache.org/jira/browse/SOLR-7766?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ramkumar Aiyengar reassigned SOLR-7766:
---
Assignee: Ramkumar Aiyengar
support creation of a coreless collection
-
Key: SOLR-7766
URL: https://issues.apache.org/jira/browse/SOLR-7766
Project: Solr
Issue Type: New Feature
Reporter: Christine Poerschke
Assignee: Ramkumar Aiyengar
Priority: Minor
By supplying a deliberately empty create node set (via
{{.../solr/admin/collections?action=CREATEcreateNodeSet=name=myCollection...}})
the collection can be created in the usual manner but it will just not yet
contain any cores for any of its shards. Later on
{{.../solr/admin/collections?wt=jsonaction=ADDREPLICAcollection=myCollection...}}
calls can create cores as and when and where required.
github pull request with proposed changes to follow.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (LUCENE-6673) Maven build fails for target javadoc:jar on trunk/Java8
[ https://issues.apache.org/jira/browse/LUCENE-6673?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ramkumar Aiyengar updated LUCENE-6673: -- Attachment: LUCENE-6673.patch I am not Maven expert, but this seems to fix it. If there are no objections, will go ahead and commit it.. Maven build fails for target javadoc:jar on trunk/Java8 --- Key: LUCENE-6673 URL: https://issues.apache.org/jira/browse/LUCENE-6673 Project: Lucene - Core Issue Type: Bug Components: general/build Reporter: Daniel Collins Assignee: Ramkumar Aiyengar Priority: Minor Attachments: LUCENE-6673.patch We currently disable missing checks for doclint, but the maven poms don't have it, as a result javadoc:jar fails. (thanks to [~dancollins] for spotting this) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Created] (LUCENE-6673) Maven build fails for target javadoc:jar on trunk/Java8
Ramkumar Aiyengar created LUCENE-6673: - Summary: Maven build fails for target javadoc:jar on trunk/Java8 Key: LUCENE-6673 URL: https://issues.apache.org/jira/browse/LUCENE-6673 Project: Lucene - Core Issue Type: Bug Components: general/build Reporter: Daniel Collins Assignee: Ramkumar Aiyengar Priority: Minor We currently disable missing checks for doclint, but the maven poms don't have it, as a result javadoc:jar fails. (thanks to [~dancollins] for spotting this) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-6563) MockFileSystemTestCase.testURI should be improved to handle cases where OS/JVM cannot create non-ASCII filenames
[
https://issues.apache.org/jira/browse/LUCENE-6563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14616300#comment-14616300
]
Ramkumar Aiyengar commented on LUCENE-6563:
---
My point is that the layer which stops caring about the sequence of bytes being
a string is different in OSs. In MacOSX, right up to the FS understands it as
UTF-8. In Linux, its at the application layer.
Actually in this case, my terminal emulator was still in UTF-8, so it accepted
Chinese chars. Which then got passed to touch as a stream of utf-8 bytes (a
simple char* to void main()), and then touch, regardless of the locale, just
created a filename with those bytes as input. Similarly ls just read these
bytes from the directory, and on display alone tried to interpret it using the
locale. In both these cases, the interface used with the OS passed in/out bytes
and nothing more. Java on the other hand uses Strings with an implicit encoding
for these inputs and hence is forced to interpret these bytes even if they are
not being displayed or input.
MockFileSystemTestCase.testURI should be improved to handle cases where
OS/JVM cannot create non-ASCII filenames
Key: LUCENE-6563
URL: https://issues.apache.org/jira/browse/LUCENE-6563
Project: Lucene - Core
Issue Type: Wish
Reporter: Christine Poerschke
Assignee: Dawid Weiss
Priority: Minor
{{ant test -Dtestcase=TestVerboseFS -Dtests.method=testURI
-Dtests.file.encoding=UTF-8}} fails (for example) with 'Oracle Corporation
1.8.0_45 (64-bit)' when the default {{sun.jnu.encoding}} system property is
(for example) {{ANSI_X3.4-1968}}
[details to follow]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Resolved] (SOLR-7750) change TestConfig.testDefaults to cover all SolrIndexConfig fields
[ https://issues.apache.org/jira/browse/SOLR-7750?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ramkumar Aiyengar resolved SOLR-7750. - Resolution: Fixed Fix Version/s: 5.3 Thanks Christine! change TestConfig.testDefaults to cover all SolrIndexConfig fields -- Key: SOLR-7750 URL: https://issues.apache.org/jira/browse/SOLR-7750 Project: Solr Issue Type: Test Reporter: Christine Poerschke Assignee: Ramkumar Aiyengar Priority: Minor Fix For: 5.3 github pull request with proposed changes to follow -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-6563) MockFileSystemTestCase.testURI should be improved to handle cases where OS/JVM cannot create non-ASCII filenames
[
https://issues.apache.org/jira/browse/LUCENE-6563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14616538#comment-14616538
]
Ramkumar Aiyengar commented on LUCENE-6563:
---
I think it's one patch per commit, and the patch file has multiple commits in
it. git tools probably understand this multi-patch format, and may be svn
doesn't..
MockFileSystemTestCase.testURI should be improved to handle cases where
OS/JVM cannot create non-ASCII filenames
Key: LUCENE-6563
URL: https://issues.apache.org/jira/browse/LUCENE-6563
Project: Lucene - Core
Issue Type: Wish
Reporter: Christine Poerschke
Assignee: Uwe Schindler
Priority: Minor
Attachments: LUCENE-6563.patch, LUCENE-6563.patch
{{ant test -Dtestcase=TestVerboseFS -Dtests.method=testURI
-Dtests.file.encoding=UTF-8}} fails (for example) with 'Oracle Corporation
1.8.0_45 (64-bit)' when the default {{sun.jnu.encoding}} system property is
(for example) {{ANSI_X3.4-1968}}
[details to follow]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-6563) MockFileSystemTestCase.testURI should be improved to handle cases where OS/JVM cannot create non-ASCII filenames
[
https://issues.apache.org/jira/browse/LUCENE-6563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14616475#comment-14616475
]
Ramkumar Aiyengar commented on LUCENE-6563:
---
[~thetaphi], looks like this is the original patch and not the one without the
Charset assume etc.?
MockFileSystemTestCase.testURI should be improved to handle cases where
OS/JVM cannot create non-ASCII filenames
Key: LUCENE-6563
URL: https://issues.apache.org/jira/browse/LUCENE-6563
Project: Lucene - Core
Issue Type: Wish
Reporter: Christine Poerschke
Assignee: Uwe Schindler
Priority: Minor
Attachments: LUCENE-6563.patch
{{ant test -Dtestcase=TestVerboseFS -Dtests.method=testURI
-Dtests.file.encoding=UTF-8}} fails (for example) with 'Oracle Corporation
1.8.0_45 (64-bit)' when the default {{sun.jnu.encoding}} system property is
(for example) {{ANSI_X3.4-1968}}
[details to follow]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Assigned] (SOLR-7750) change TestConfig.testDefaults to cover all SolrIndexConfig fields
[ https://issues.apache.org/jira/browse/SOLR-7750?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ramkumar Aiyengar reassigned SOLR-7750: --- Assignee: Ramkumar Aiyengar change TestConfig.testDefaults to cover all SolrIndexConfig fields -- Key: SOLR-7750 URL: https://issues.apache.org/jira/browse/SOLR-7750 Project: Solr Issue Type: Test Reporter: Christine Poerschke Assignee: Ramkumar Aiyengar Priority: Minor github pull request with proposed changes to follow -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Resolved] (SOLR-7751) tweak QueryComponent.process
[ https://issues.apache.org/jira/browse/SOLR-7751?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ramkumar Aiyengar resolved SOLR-7751. - Resolution: Fixed Assignee: Ramkumar Aiyengar Fix Version/s: 5.3 tweak QueryComponent.process Key: SOLR-7751 URL: https://issues.apache.org/jira/browse/SOLR-7751 Project: Solr Issue Type: Improvement Reporter: Christine Poerschke Assignee: Ramkumar Aiyengar Priority: Minor Fix For: 5.3 github pull request with proposed small changes/optimisations to follow -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-6563) sun.jnu.encoding to match file.encoding system property (MockFileSystemTestCase.testURI)
[
https://issues.apache.org/jira/browse/LUCENE-6563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14614842#comment-14614842
]
Ramkumar Aiyengar commented on LUCENE-6563:
---
I see why we shouldn't modify the internal prop -- it sounded hacky to begin
with, but I still think our tests shouldn't fail for some supported locale on a
supported OS (I was able to reproduce this on the latest Ubuntu on my laptop).
To the best of my knowledge, Lucene doesn't need non-ASCII file names for
itself -- so if we are not able to create non-ASCII filenames to begin with,
shouldn't the mock filesystem not try to use them? That way we continue to test
that if the OS was able to create non-ASCII parent directories, Lucene should
continue to work fine, but that shouldn't be a requirement for Lucene per se.
sun.jnu.encoding to match file.encoding system property
(MockFileSystemTestCase.testURI)
Key: LUCENE-6563
URL: https://issues.apache.org/jira/browse/LUCENE-6563
Project: Lucene - Core
Issue Type: Wish
Reporter: Christine Poerschke
Assignee: Dawid Weiss
Priority: Minor
{{ant test -Dtestcase=TestVerboseFS -Dtests.method=testURI
-Dtests.file.encoding=UTF-8}} fails (for example) with 'Oracle Corporation
1.8.0_45 (64-bit)' when the default {{sun.jnu.encoding}} system property is
(for example) {{ANSI_X3.4-1968}}
[details to follow]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (LUCENE-6563) MockFileSystemTestCase.testURI should be improved to handle cases where OS/JVM cannot create non-ASCII files
[
https://issues.apache.org/jira/browse/LUCENE-6563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ramkumar Aiyengar updated LUCENE-6563:
--
Summary: MockFileSystemTestCase.testURI should be improved to handle cases
where OS/JVM cannot create non-ASCII files (was: sun.jnu.encoding to match
file.encoding system property (MockFileSystemTestCase.testURI))
MockFileSystemTestCase.testURI should be improved to handle cases where
OS/JVM cannot create non-ASCII files
Key: LUCENE-6563
URL: https://issues.apache.org/jira/browse/LUCENE-6563
Project: Lucene - Core
Issue Type: Wish
Reporter: Christine Poerschke
Assignee: Dawid Weiss
Priority: Minor
{{ant test -Dtestcase=TestVerboseFS -Dtests.method=testURI
-Dtests.file.encoding=UTF-8}} fails (for example) with 'Oracle Corporation
1.8.0_45 (64-bit)' when the default {{sun.jnu.encoding}} system property is
(for example) {{ANSI_X3.4-1968}}
[details to follow]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-6563) MockFileSystemTestCase.testURI should be improved to handle cases where OS/JVM cannot create non-ASCII filenames
[
https://issues.apache.org/jira/browse/LUCENE-6563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14615766#comment-14615766
]
Ramkumar Aiyengar commented on LUCENE-6563:
---
bq. I honestly think this is a quirk/bug in the JVM... and perhaps should be
reported. Setting LANG=C shouldn't be affecting how filenames are (mis)handled
(and it currently does).
I digged a bit, and it appears that Java is kind of doing the right thing given
it's current API. Certain newer versions/FSs of Windows and MacOSX guarantee
that all files are in UTF-16/8 respectively. Linux/Solaris etc. (aka the more
traditional Unix systems) tend not to care about the encoding at all. As a
matter of fact, a filename is just a sequence of bytes, it's not even a string.
How that byte array is displayed comes down to the locale. This is probably why
this works.
{code}
$ LANG=C touch 中国
{code}
touch doesn't care about the input, the shell maps it into a set of UTF-8
bytes, and is stored as a filename. ls, then, in an UTF-8 locale shows the
correct thing
{code}
$ ls
build.xml ivy.xml lib lucene-test-framework.iml src 中国
{code}
And if I try to read in the C locale, I get a bunch of unreadable chars
{code}
$ LANG=C ls
build.xml ivy.xml lib lucene-test-framework.iml src ??
{code}
Java APIs on the other hand treats filenames as strings in all it's APIs, and
as a result is forced to need an encoding even when it is using it only for I/O
and not for display. As a result, it is forced to choose some encoding, and it
goes with the locale.. In platforms where the filename encoding is guaranteed
to be UTF-8, it goes with that -- see
http://bugs.java.com/bugdatabase/view_bug.do?bug_id=8003228 for MacOSX.
Looks like this issue is not specific to Java -- see
https://bugs.python.org/issue19846 this for example.
MockFileSystemTestCase.testURI should be improved to handle cases where
OS/JVM cannot create non-ASCII filenames
Key: LUCENE-6563
URL: https://issues.apache.org/jira/browse/LUCENE-6563
Project: Lucene - Core
Issue Type: Wish
Reporter: Christine Poerschke
Assignee: Dawid Weiss
Priority: Minor
{{ant test -Dtestcase=TestVerboseFS -Dtests.method=testURI
-Dtests.file.encoding=UTF-8}} fails (for example) with 'Oracle Corporation
1.8.0_45 (64-bit)' when the default {{sun.jnu.encoding}} system property is
(for example) {{ANSI_X3.4-1968}}
[details to follow]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
