[jira] [Commented] (SOLR-11216) Make PeerSync more robust
[
https://issues.apache.org/jira/browse/SOLR-11216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16514102#comment-16514102
]
hamada commented on SOLR-11216:
---
General review comments, some not related to the patch but relevant In general.
PeerSyncWithLeader
use
startingVersions.isEmpty() rather than size() == 0, same for 215
The following try/finally can return, in which case proc is not closed, Is this
intentional, and if so please add a comment to the effect
line 299, consider sizing the List properly to avoid garbage side effect from
growing the list, same applies to line 317
HttpShardHandler.java
if (urls.size()==0) { with if (urls.isEmpty()) {
RecoveryStrategy.java
line 223 and 613, 235 (on
core.getDeletionPolicy().getLatestCommit().getGeneration()) may result in an
NPE
line 436
SolrQueryRequest req = new LocalSolrQueryRequest(core,
new ModifiableSolrParams());
request is not safely closed, is this intentional? won't this break the
reference count mechanism?
> Make PeerSync more robust
> -
>
> Key: SOLR-11216
> URL: https://issues.apache.org/jira/browse/SOLR-11216
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Cao Manh Dat
>Priority: Major
> Attachments: SOLR-11216.patch, SOLR-11216.patch, SOLR-11216.patch
>
>
> First of all, I will change the issue's title with a better name when I have.
> When digging into SOLR-10126. I found a case that can make peerSync fail.
> * leader and replica receive update from 1 to 4
> * replica stop
> * replica miss updates 5, 6
> * replica start recovery
> ## replica buffer updates 7, 8
> ## replica request versions from leader,
> ## in the same time leader receive update 9, so it will return updates from 1
> to 9 (for request versions) when replica get recent versions ( so it will be
> 1,2,3,4,5,6,7,8,9 )
> ## replica do peersync and request updates 5, 6, 9 from leader
> ## replica apply updates 5, 6, 9. Its index does not have update 7, 8 and
> maxVersionSpecified for fingerprint is 9, therefore compare fingerprint will
> fail
> My idea here is why replica request update 9 (step 6) while it knows that
> updates with lower version ( update 7, 8 ) are on its buffering tlog. Should
> we request only updates that lower than the lowest update in its buffering
> tlog ( < 7 )?
> Someone my ask that what if replica won't receive update 9. In that case,
> leader will put the replica into LIR state, so replica will run recovery
> process again.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-11216) Make PeerSync more robust
[ https://issues.apache.org/jira/browse/SOLR-11216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16509971#comment-16509971 ] hamada commented on SOLR-11216: --- from 20,000 foot level. Any time based solution is just brittle, solution 2 sounds like a workaround. solution 3 seems to fit the bill. > Make PeerSync more robust > - > > Key: SOLR-11216 > URL: https://issues.apache.org/jira/browse/SOLR-11216 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Cao Manh Dat >Priority: Major > Attachments: SOLR-11216.patch > > > First of all, I will change the issue's title with a better name when I have. > When digging into SOLR-10126. I found a case that can make peerSync fail. > * leader and replica receive update from 1 to 4 > * replica stop > * replica miss updates 5, 6 > * replica start recovery > ## replica buffer updates 7, 8 > ## replica request versions from leader, > ## in the same time leader receive update 9, so it will return updates from 1 > to 9 (for request versions) when replica get recent versions ( so it will be > 1,2,3,4,5,6,7,8,9 ) > ## replica do peersync and request updates 5, 6, 9 from leader > ## replica apply updates 5, 6, 9. Its index does not have update 7, 8 and > maxVersionSpecified for fingerprint is 9, therefore compare fingerprint will > fail > My idea here is why replica request update 9 (step 6) while it knows that > updates with lower version ( update 7, 8 ) are on its buffering tlog. Should > we request only updates that lower than the lowest update in its buffering > tlog ( < 7 )? > Someone my ask that what if replica won't receive update 9. In that case, > leader will put the replica into LIR state, so replica will run recovery > process again. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Created] (SOLR-12354) org.apache.solr.security.PKIAuthenticationPlugin does not check response code when retrieving remotePublicKey
hamada created SOLR-12354: - Summary: org.apache.solr.security.PKIAuthenticationPlugin does not check response code when retrieving remotePublicKey Key: SOLR-12354 URL: https://issues.apache.org/jira/browse/SOLR-12354 Project: Solr Issue Type: Bug Security Level: Public (Default Security Level. Issues are Public) Components: Authentication Affects Versions: 6.6.3, 6.6.2 Reporter: hamada in decipherHeader(), if keyCache does not contain the key of interest, then a remote call is made to retrieve the key from the remote host, by calling getRemotePublicKey, which fails if the server returns an html error page. e.g.: org.noggit.JSONParser$ParseException: JSON Parse Error: char=<,position=0 BEFORE='<' AFTER='html>
[jira] [Comment Edited] (SOLR-11892) Avoid unnecessary exceptions in FSDirectory and RAMDirectory
[
https://issues.apache.org/jira/browse/SOLR-11892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16343976#comment-16343976
]
hamada edited comment on SOLR-11892 at 1/29/18 8:42 PM:
specifically the code of interest where an IOException is thrown RAMDirectory :
@Override
public void deleteFile(String name) throws IOException {
ensureOpen();
RAMFile file = fileMap.remove(name);
if (file != null)
{ file.directory = null; sizeInBytes.addAndGet(-file.sizeInBytes); }
else
{ // FIXME there are no file operations here this method removes fileName entry
from a map, it isn't per se deleting a file!
throw new FileNotFoundException(name); }
}
was (Author: hamadaca):
specifically the code of interest where an IOException is thrown :
@Override
public void deleteFile(String name) throws IOException {
ensureOpen();
RAMFile file = fileMap.remove(name);
if (file != null) {
file.directory = null;
sizeInBytes.addAndGet(-file.sizeInBytes);
} else {
// FIXME there are no file operations here this method removes fileName entry
from a map, it isn't per se removing deleting a file!
throw new FileNotFoundException(name);
}
}
> Avoid unnecessary exceptions in FSDirectory and RAMDirectory
>
>
> Key: SOLR-11892
> URL: https://issues.apache.org/jira/browse/SOLR-11892
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Erick Erickson
>Assignee: Erick Erickson
>Priority: Minor
> Attachments: Screen Shot 2018-01-24 at 9.09.55 PM.png, Screen Shot
> 2018-01-24 at 9.10.47 PM.png
>
>
> In privateDeleteFile, just use deleteIfExists.
> in RamDirectory we can declare a static exception and create it once.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-11892) Avoid unnecessary exceptions in FSDirectory and RAMDirectory
[
https://issues.apache.org/jira/browse/SOLR-11892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16343976#comment-16343976
]
hamada commented on SOLR-11892:
---
specifically the code of interest where an IOException is thrown :
@Override
public void deleteFile(String name) throws IOException {
ensureOpen();
RAMFile file = fileMap.remove(name);
if (file != null) {
file.directory = null;
sizeInBytes.addAndGet(-file.sizeInBytes);
} else {
// FIXME there are no file operations here this method removes fileName entry
from a map, it isn't per se removing deleting a file!
throw new FileNotFoundException(name);
}
}
> Avoid unnecessary exceptions in FSDirectory and RAMDirectory
>
>
> Key: SOLR-11892
> URL: https://issues.apache.org/jira/browse/SOLR-11892
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Erick Erickson
>Assignee: Erick Erickson
>Priority: Minor
> Attachments: Screen Shot 2018-01-24 at 9.09.55 PM.png, Screen Shot
> 2018-01-24 at 9.10.47 PM.png
>
>
> In privateDeleteFile, just use deleteIfExists.
> in RamDirectory we can declare a static exception and create it once.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-11892) Avoid unnecessary exceptions in FSDirectory and RAMDirectory
[
https://issues.apache.org/jira/browse/SOLR-11892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16343843#comment-16343843
]
hamada commented on SOLR-11892:
---
Few points about Exception cost:
* It can not be assumed that all JVM's are all optimized the same, therefore,
performance cost should not be ignored. General assumptions on cost should not
be based on one VM's optimization.
* Exception creation has associated cost, follow the stack on an exception
creation :
** public synchronized Throwable fillInStackTrace() {
if (stackTrace != null ||
backtrace != null /* Out of protocol state */ ) {
fillInStackTrace(0);
stackTrace = UNASSIGNED_STACK;
}
return this;
}
private native Throwable fillInStackTrace(int dummy);
** In addition to being synchronized, it has a memory cost, the exception, and
all stackelement strings
* Exceptions in the critical path will affect performance, and add to memory
pressure.
** In this use case it appears to be logic flow control, and not of the
Exception
In addition to the cpu and memory cost, the frequency of such an exception, the
absence of file name map entry is indicative of a race condition, a bug, or a
concurrency issue.
The general rule about exceptions, as they're named, they should be of the
exception, and not the norm.
> Avoid unnecessary exceptions in FSDirectory and RAMDirectory
>
>
> Key: SOLR-11892
> URL: https://issues.apache.org/jira/browse/SOLR-11892
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Erick Erickson
>Assignee: Erick Erickson
>Priority: Minor
> Attachments: Screen Shot 2018-01-24 at 9.09.55 PM.png, Screen Shot
> 2018-01-24 at 9.10.47 PM.png
>
>
> In privateDeleteFile, just use deleteIfExists.
> in RamDirectory we can declare a static exception and create it once.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
