[jira] [Commented] (SOLR-6920) During replication use checksums to verify if files are the same
[ https://issues.apache.org/jira/browse/SOLR-6920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14312629#comment-14312629 ] ASF subversion and git services commented on SOLR-6920: --- Commit 1658526 from [~markrmil...@gmail.com] in branch 'dev/branches/lucene_solr_5_0' [ https://svn.apache.org/r1658526 ] SOLR-6920, SOLR-6640: Make constant and fix logging. > During replication use checksums to verify if files are the same > > > Key: SOLR-6920 > URL: https://issues.apache.org/jira/browse/SOLR-6920 > Project: Solr > Issue Type: Bug > Components: replication (java) >Reporter: Varun Thacker >Assignee: Mark Miller >Priority: Critical > Fix For: 5.0, Trunk > > Attachments: SOLR-6920-5x.patch, SOLR-6920-5x.patch, > SOLR-6920-5x.patch, SOLR-6920-5x.patch, SOLR-6920.patch, SOLR-6920.patch, > SOLR-6920.patch, SOLR-6920.patch, SOLR-6920.patch, SOLR-6920.patch > > > Currently we check if an index file on the master and slave is the same by > checking if it's name and file length match. > With LUCENE-2446 we now have a checksums for each index file in the segment. > We should leverage this to verify if two files are the same. > Places like SnapPuller.isIndexStale and SnapPuller.downloadIndexFiles should > check against the checksum also. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-6920) During replication use checksums to verify if files are the same
[ https://issues.apache.org/jira/browse/SOLR-6920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14312611#comment-14312611 ] ASF subversion and git services commented on SOLR-6920: --- Commit 1658524 from [~markrmil...@gmail.com] in branch 'dev/branches/branch_5x' [ https://svn.apache.org/r1658524 ] SOLR-6920, SOLR-6640: Make constant and fix logging. > During replication use checksums to verify if files are the same > > > Key: SOLR-6920 > URL: https://issues.apache.org/jira/browse/SOLR-6920 > Project: Solr > Issue Type: Bug > Components: replication (java) >Reporter: Varun Thacker >Assignee: Mark Miller >Priority: Critical > Attachments: SOLR-6920-5x.patch, SOLR-6920-5x.patch, > SOLR-6920-5x.patch, SOLR-6920-5x.patch, SOLR-6920.patch, SOLR-6920.patch, > SOLR-6920.patch, SOLR-6920.patch, SOLR-6920.patch, SOLR-6920.patch > > > Currently we check if an index file on the master and slave is the same by > checking if it's name and file length match. > With LUCENE-2446 we now have a checksums for each index file in the segment. > We should leverage this to verify if two files are the same. > Places like SnapPuller.isIndexStale and SnapPuller.downloadIndexFiles should > check against the checksum also. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-6920) During replication use checksums to verify if files are the same
[ https://issues.apache.org/jira/browse/SOLR-6920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14312601#comment-14312601 ] ASF subversion and git services commented on SOLR-6920: --- Commit 1658519 from [~markrmil...@gmail.com] in branch 'dev/trunk' [ https://svn.apache.org/r1658519 ] SOLR-6920, SOLR-6640: Make constant and fix logging. > During replication use checksums to verify if files are the same > > > Key: SOLR-6920 > URL: https://issues.apache.org/jira/browse/SOLR-6920 > Project: Solr > Issue Type: Bug > Components: replication (java) >Reporter: Varun Thacker >Assignee: Mark Miller >Priority: Critical > Attachments: SOLR-6920-5x.patch, SOLR-6920-5x.patch, > SOLR-6920-5x.patch, SOLR-6920-5x.patch, SOLR-6920.patch, SOLR-6920.patch, > SOLR-6920.patch, SOLR-6920.patch, SOLR-6920.patch, SOLR-6920.patch > > > Currently we check if an index file on the master and slave is the same by > checking if it's name and file length match. > With LUCENE-2446 we now have a checksums for each index file in the segment. > We should leverage this to verify if two files are the same. > Places like SnapPuller.isIndexStale and SnapPuller.downloadIndexFiles should > check against the checksum also. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-6920) During replication use checksums to verify if files are the same
[ https://issues.apache.org/jira/browse/SOLR-6920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14311491#comment-14311491 ] ASF subversion and git services commented on SOLR-6920: --- Commit 1658240 from [~markrmil...@gmail.com] in branch 'dev/branches/lucene_solr_5_0' [ https://svn.apache.org/r1658240 ] SOLR-6920, SOLR-6640: A replicated index can end up corrupted when small files end up with the same file name and size. > During replication use checksums to verify if files are the same > > > Key: SOLR-6920 > URL: https://issues.apache.org/jira/browse/SOLR-6920 > Project: Solr > Issue Type: Bug > Components: replication (java) >Reporter: Varun Thacker >Assignee: Mark Miller >Priority: Critical > Attachments: SOLR-6920-5x.patch, SOLR-6920-5x.patch, > SOLR-6920-5x.patch, SOLR-6920-5x.patch, SOLR-6920.patch, SOLR-6920.patch, > SOLR-6920.patch, SOLR-6920.patch, SOLR-6920.patch, SOLR-6920.patch > > > Currently we check if an index file on the master and slave is the same by > checking if it's name and file length match. > With LUCENE-2446 we now have a checksums for each index file in the segment. > We should leverage this to verify if two files are the same. > Places like SnapPuller.isIndexStale and SnapPuller.downloadIndexFiles should > check against the checksum also. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-6920) During replication use checksums to verify if files are the same
[ https://issues.apache.org/jira/browse/SOLR-6920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14311100#comment-14311100 ] ASF subversion and git services commented on SOLR-6920: --- Commit 1658130 from [~markrmil...@gmail.com] in branch 'dev/branches/branch_5x' [ https://svn.apache.org/r1658130 ] SOLR-6920, SOLR-6640: Rename method to filesToAlwaysDownloadIfNoChecksums. > During replication use checksums to verify if files are the same > > > Key: SOLR-6920 > URL: https://issues.apache.org/jira/browse/SOLR-6920 > Project: Solr > Issue Type: Bug > Components: replication (java) >Reporter: Varun Thacker >Assignee: Mark Miller >Priority: Critical > Attachments: SOLR-6920-5x.patch, SOLR-6920-5x.patch, > SOLR-6920-5x.patch, SOLR-6920-5x.patch, SOLR-6920.patch, SOLR-6920.patch, > SOLR-6920.patch, SOLR-6920.patch, SOLR-6920.patch, SOLR-6920.patch > > > Currently we check if an index file on the master and slave is the same by > checking if it's name and file length match. > With LUCENE-2446 we now have a checksums for each index file in the segment. > We should leverage this to verify if two files are the same. > Places like SnapPuller.isIndexStale and SnapPuller.downloadIndexFiles should > check against the checksum also. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-6920) During replication use checksums to verify if files are the same
[ https://issues.apache.org/jira/browse/SOLR-6920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14311095#comment-14311095 ] ASF subversion and git services commented on SOLR-6920: --- Commit 1658129 from [~markrmil...@gmail.com] in branch 'dev/trunk' [ https://svn.apache.org/r1658129 ] SOLR-6920, SOLR-6640: Rename method to filesToAlwaysDownloadIfNoChecksums. > During replication use checksums to verify if files are the same > > > Key: SOLR-6920 > URL: https://issues.apache.org/jira/browse/SOLR-6920 > Project: Solr > Issue Type: Bug > Components: replication (java) >Reporter: Varun Thacker >Assignee: Mark Miller >Priority: Critical > Attachments: SOLR-6920-5x.patch, SOLR-6920-5x.patch, > SOLR-6920-5x.patch, SOLR-6920-5x.patch, SOLR-6920.patch, SOLR-6920.patch, > SOLR-6920.patch, SOLR-6920.patch, SOLR-6920.patch, SOLR-6920.patch > > > Currently we check if an index file on the master and slave is the same by > checking if it's name and file length match. > With LUCENE-2446 we now have a checksums for each index file in the segment. > We should leverage this to verify if two files are the same. > Places like SnapPuller.isIndexStale and SnapPuller.downloadIndexFiles should > check against the checksum also. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-6920) During replication use checksums to verify if files are the same
[ https://issues.apache.org/jira/browse/SOLR-6920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14310856#comment-14310856 ] ASF subversion and git services commented on SOLR-6920: --- Commit 1658090 from [~markrmil...@gmail.com] in branch 'dev/branches/branch_5x' [ https://svn.apache.org/r1658090 ] SOLR-6920, SOLR-6640: Do not log an error when a file does not exist for comparison. > During replication use checksums to verify if files are the same > > > Key: SOLR-6920 > URL: https://issues.apache.org/jira/browse/SOLR-6920 > Project: Solr > Issue Type: Bug > Components: replication (java) >Reporter: Varun Thacker >Assignee: Mark Miller >Priority: Critical > Attachments: SOLR-6920-5x.patch, SOLR-6920-5x.patch, > SOLR-6920-5x.patch, SOLR-6920-5x.patch, SOLR-6920.patch, SOLR-6920.patch, > SOLR-6920.patch, SOLR-6920.patch, SOLR-6920.patch, SOLR-6920.patch > > > Currently we check if an index file on the master and slave is the same by > checking if it's name and file length match. > With LUCENE-2446 we now have a checksums for each index file in the segment. > We should leverage this to verify if two files are the same. > Places like SnapPuller.isIndexStale and SnapPuller.downloadIndexFiles should > check against the checksum also. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-6920) During replication use checksums to verify if files are the same
[ https://issues.apache.org/jira/browse/SOLR-6920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14310853#comment-14310853 ] ASF subversion and git services commented on SOLR-6920: --- Commit 1658089 from [~markrmil...@gmail.com] in branch 'dev/trunk' [ https://svn.apache.org/r1658089 ] SOLR-6920, SOLR-6640: Do not log an error when a file does not exist for comparison. > During replication use checksums to verify if files are the same > > > Key: SOLR-6920 > URL: https://issues.apache.org/jira/browse/SOLR-6920 > Project: Solr > Issue Type: Bug > Components: replication (java) >Reporter: Varun Thacker >Assignee: Mark Miller >Priority: Critical > Attachments: SOLR-6920-5x.patch, SOLR-6920-5x.patch, > SOLR-6920-5x.patch, SOLR-6920-5x.patch, SOLR-6920.patch, SOLR-6920.patch, > SOLR-6920.patch, SOLR-6920.patch, SOLR-6920.patch, SOLR-6920.patch > > > Currently we check if an index file on the master and slave is the same by > checking if it's name and file length match. > With LUCENE-2446 we now have a checksums for each index file in the segment. > We should leverage this to verify if two files are the same. > Places like SnapPuller.isIndexStale and SnapPuller.downloadIndexFiles should > check against the checksum also. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-6920) During replication use checksums to verify if files are the same
[ https://issues.apache.org/jira/browse/SOLR-6920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14310835#comment-14310835 ] ASF subversion and git services commented on SOLR-6920: --- Commit 1658083 from [~markrmil...@gmail.com] in branch 'dev/branches/branch_5x' [ https://svn.apache.org/r1658083 ] SOLR-6920, SOLR-6640: A replicated index can end up corrupted when small files end up with the same file name and size. > During replication use checksums to verify if files are the same > > > Key: SOLR-6920 > URL: https://issues.apache.org/jira/browse/SOLR-6920 > Project: Solr > Issue Type: Bug > Components: replication (java) >Reporter: Varun Thacker >Assignee: Mark Miller >Priority: Critical > Attachments: SOLR-6920-5x.patch, SOLR-6920-5x.patch, > SOLR-6920-5x.patch, SOLR-6920-5x.patch, SOLR-6920.patch, SOLR-6920.patch, > SOLR-6920.patch, SOLR-6920.patch, SOLR-6920.patch, SOLR-6920.patch > > > Currently we check if an index file on the master and slave is the same by > checking if it's name and file length match. > With LUCENE-2446 we now have a checksums for each index file in the segment. > We should leverage this to verify if two files are the same. > Places like SnapPuller.isIndexStale and SnapPuller.downloadIndexFiles should > check against the checksum also. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-6920) During replication use checksums to verify if files are the same
[ https://issues.apache.org/jira/browse/SOLR-6920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14310818#comment-14310818 ] ASF subversion and git services commented on SOLR-6920: --- Commit 1658078 from [~markrmil...@gmail.com] in branch 'dev/trunk' [ https://svn.apache.org/r1658078 ] SOLR-6920, SOLR-6640: When we so not have a checksum for a file, always download files under 100kb and other small improvements. > During replication use checksums to verify if files are the same > > > Key: SOLR-6920 > URL: https://issues.apache.org/jira/browse/SOLR-6920 > Project: Solr > Issue Type: Bug > Components: replication (java) >Reporter: Varun Thacker >Assignee: Mark Miller >Priority: Critical > Attachments: SOLR-6920-5x.patch, SOLR-6920-5x.patch, > SOLR-6920-5x.patch, SOLR-6920-5x.patch, SOLR-6920.patch, SOLR-6920.patch, > SOLR-6920.patch, SOLR-6920.patch, SOLR-6920.patch, SOLR-6920.patch > > > Currently we check if an index file on the master and slave is the same by > checking if it's name and file length match. > With LUCENE-2446 we now have a checksums for each index file in the segment. > We should leverage this to verify if two files are the same. > Places like SnapPuller.isIndexStale and SnapPuller.downloadIndexFiles should > check against the checksum also. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-6920) During replication use checksums to verify if files are the same
[ https://issues.apache.org/jira/browse/SOLR-6920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14310785#comment-14310785 ] Mark Miller commented on SOLR-6920: --- Yeah, good catch. I'd almost like to do that on trunk as well just for consistency. > During replication use checksums to verify if files are the same > > > Key: SOLR-6920 > URL: https://issues.apache.org/jira/browse/SOLR-6920 > Project: Solr > Issue Type: Bug > Components: replication (java) >Reporter: Varun Thacker >Assignee: Mark Miller >Priority: Critical > Attachments: SOLR-6920-5x.patch, SOLR-6920-5x.patch, SOLR-6920.patch, > SOLR-6920.patch, SOLR-6920.patch, SOLR-6920.patch, SOLR-6920.patch, > SOLR-6920.patch > > > Currently we check if an index file on the master and slave is the same by > checking if it's name and file length match. > With LUCENE-2446 we now have a checksums for each index file in the segment. > We should leverage this to verify if two files are the same. > Places like SnapPuller.isIndexStale and SnapPuller.downloadIndexFiles should > check against the checksum also. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-6920) During replication use checksums to verify if files are the same
[
https://issues.apache.org/jira/browse/SOLR-6920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14310565#comment-14310565
]
Varun Thacker commented on SOLR-6920:
-
Hi Mark,
In the 5.x patch, should we wrap {{fileMeta.put(CHECKSUM,
CodecUtil.retrieveChecksum(in));}} for the segments_n file in a try-catch
block?
> During replication use checksums to verify if files are the same
>
>
> Key: SOLR-6920
> URL: https://issues.apache.org/jira/browse/SOLR-6920
> Project: Solr
> Issue Type: Bug
> Components: replication (java)
>Reporter: Varun Thacker
>Assignee: Mark Miller
>Priority: Critical
> Attachments: SOLR-6920-5x.patch, SOLR-6920-5x.patch, SOLR-6920.patch,
> SOLR-6920.patch, SOLR-6920.patch, SOLR-6920.patch, SOLR-6920.patch,
> SOLR-6920.patch
>
>
> Currently we check if an index file on the master and slave is the same by
> checking if it's name and file length match.
> With LUCENE-2446 we now have a checksums for each index file in the segment.
> We should leverage this to verify if two files are the same.
> Places like SnapPuller.isIndexStale and SnapPuller.downloadIndexFiles should
> check against the checksum also.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-6920) During replication use checksums to verify if files are the same
[ https://issues.apache.org/jira/browse/SOLR-6920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14309976#comment-14309976 ] ASF subversion and git services commented on SOLR-6920: --- Commit 1657969 from [~markrmil...@gmail.com] in branch 'dev/trunk' [ https://svn.apache.org/r1657969 ] SOLR-6920, SOLR-6640: A replicated index can end up corrupted when small files end up with the same file name and size. > During replication use checksums to verify if files are the same > > > Key: SOLR-6920 > URL: https://issues.apache.org/jira/browse/SOLR-6920 > Project: Solr > Issue Type: Bug > Components: replication (java) >Reporter: Varun Thacker >Assignee: Mark Miller >Priority: Critical > Attachments: SOLR-6920.patch, SOLR-6920.patch, SOLR-6920.patch, > SOLR-6920.patch, SOLR-6920.patch, SOLR-6920.patch > > > Currently we check if an index file on the master and slave is the same by > checking if it's name and file length match. > With LUCENE-2446 we now have a checksums for each index file in the segment. > We should leverage this to verify if two files are the same. > Places like SnapPuller.isIndexStale and SnapPuller.downloadIndexFiles should > check against the checksum also. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-6920) During replication use checksums to verify if files are the same
[ https://issues.apache.org/jira/browse/SOLR-6920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14309904#comment-14309904 ] Mark Miller commented on SOLR-6920: --- Okay - I think we should put this in trunk and 5x to start jenkins on it. I've moved my local machine to only running the 9 chaosmonkey jobs I have. If anyone is following along, more review would be good. I'll commit to trunk and let's work on a 5x patch. > During replication use checksums to verify if files are the same > > > Key: SOLR-6920 > URL: https://issues.apache.org/jira/browse/SOLR-6920 > Project: Solr > Issue Type: Bug > Components: replication (java) >Reporter: Varun Thacker >Assignee: Mark Miller >Priority: Critical > Attachments: SOLR-6920.patch, SOLR-6920.patch, SOLR-6920.patch, > SOLR-6920.patch, SOLR-6920.patch, SOLR-6920.patch > > > Currently we check if an index file on the master and slave is the same by > checking if it's name and file length match. > With LUCENE-2446 we now have a checksums for each index file in the segment. > We should leverage this to verify if two files are the same. > Places like SnapPuller.isIndexStale and SnapPuller.downloadIndexFiles should > check against the checksum also. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-6920) During replication use checksums to verify if files are the same
[
https://issues.apache.org/jira/browse/SOLR-6920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14309895#comment-14309895
]
Varun Thacker commented on SOLR-6920:
-
You're right. I got confused a bit with this portion {{!compareResult.equal}}
in the statement and replied too early.
> During replication use checksums to verify if files are the same
>
>
> Key: SOLR-6920
> URL: https://issues.apache.org/jira/browse/SOLR-6920
> Project: Solr
> Issue Type: Bug
> Components: replication (java)
>Reporter: Varun Thacker
>Assignee: Mark Miller
>Priority: Critical
> Attachments: SOLR-6920.patch, SOLR-6920.patch, SOLR-6920.patch,
> SOLR-6920.patch, SOLR-6920.patch, SOLR-6920.patch
>
>
> Currently we check if an index file on the master and slave is the same by
> checking if it's name and file length match.
> With LUCENE-2446 we now have a checksums for each index file in the segment.
> We should leverage this to verify if two files are the same.
> Places like SnapPuller.isIndexStale and SnapPuller.downloadIndexFiles should
> check against the checksum also.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-6920) During replication use checksums to verify if files are the same
[
https://issues.apache.org/jira/browse/SOLR-6920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14309788#comment-14309788
]
Mark Miller commented on SOLR-6920:
---
bq. 3
I'm not following - what are you seeing exactly? I read it as 'if a file was
not checksummed, download anyway if it matches this file list'.
bq. || (!compareResult.checkSummed && (((String)
file.get(NAME)).endsWith(".si") || ((String) file.get(NAME)).endsWith(".liv")
|| ((String) file.get(NAME)).startsWith("segments_")))
If the file is checksummed, we can rely on the checksum. What am I missing that
you are seeing?
> During replication use checksums to verify if files are the same
>
>
> Key: SOLR-6920
> URL: https://issues.apache.org/jira/browse/SOLR-6920
> Project: Solr
> Issue Type: Bug
> Components: replication (java)
>Reporter: Varun Thacker
>Assignee: Mark Miller
>Priority: Critical
> Attachments: SOLR-6920.patch, SOLR-6920.patch, SOLR-6920.patch,
> SOLR-6920.patch, SOLR-6920.patch, SOLR-6920.patch
>
>
> Currently we check if an index file on the master and slave is the same by
> checking if it's name and file length match.
> With LUCENE-2446 we now have a checksums for each index file in the segment.
> We should leverage this to verify if two files are the same.
> Places like SnapPuller.isIndexStale and SnapPuller.downloadIndexFiles should
> check against the checksum also.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-6920) During replication use checksums to verify if files are the same
[ https://issues.apache.org/jira/browse/SOLR-6920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14309701#comment-14309701 ] Mark Miller commented on SOLR-6920: --- That patch did pick up cruft (TopGroupsShardResponseProcessor) from fooling around trying to fix a Solr fail triggered by a Lucene change. I'll put up a clean one in a moment. > During replication use checksums to verify if files are the same > > > Key: SOLR-6920 > URL: https://issues.apache.org/jira/browse/SOLR-6920 > Project: Solr > Issue Type: Bug > Components: replication (java) >Reporter: Varun Thacker >Assignee: Mark Miller >Priority: Critical > Attachments: SOLR-6920.patch, SOLR-6920.patch, SOLR-6920.patch, > SOLR-6920.patch, SOLR-6920.patch > > > Currently we check if an index file on the master and slave is the same by > checking if it's name and file length match. > With LUCENE-2446 we now have a checksums for each index file in the segment. > We should leverage this to verify if two files are the same. > Places like SnapPuller.isIndexStale and SnapPuller.downloadIndexFiles should > check against the checksum also. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-6920) During replication use checksums to verify if files are the same
[ https://issues.apache.org/jira/browse/SOLR-6920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14309538#comment-14309538 ] Mark Miller commented on SOLR-6920: --- bq. In your patch if the file fails to retrieve a checksum... Feel free to put up a follow on patch - this one is just a first draft. I'll be taking another pass when I get a chance as well. bq. I think you have an extra file from HdfsTestUtil in the patch. No, that's fixes an error I see in the test that shalin mentions in SOLR-6640 and led me back to this issue. > During replication use checksums to verify if files are the same > > > Key: SOLR-6920 > URL: https://issues.apache.org/jira/browse/SOLR-6920 > Project: Solr > Issue Type: Bug > Components: replication (java) >Reporter: Varun Thacker >Assignee: Mark Miller >Priority: Critical > Attachments: SOLR-6920.patch, SOLR-6920.patch, SOLR-6920.patch, > SOLR-6920.patch > > > Currently we check if an index file on the master and slave is the same by > checking if it's name and file length match. > With LUCENE-2446 we now have a checksums for each index file in the segment. > We should leverage this to verify if two files are the same. > Places like SnapPuller.isIndexStale and SnapPuller.downloadIndexFiles should > check against the checksum also. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-6920) During replication use checksums to verify if files are the same
[ https://issues.apache.org/jira/browse/SOLR-6920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14309489#comment-14309489 ] Varun Thacker commented on SOLR-6920: - Mark, In your patch if the file fails to retrieve a checksum and if its a .si/segments_n file then we will have to re-download the complete index. the reason being isStaleIndex will return true. Instead if we moved to the check into the if statement in downloadIndexFiles() to always re-download those files won't it be better? I think you have an extra file from HdfsTestUtil in the patch. > During replication use checksums to verify if files are the same > > > Key: SOLR-6920 > URL: https://issues.apache.org/jira/browse/SOLR-6920 > Project: Solr > Issue Type: Bug > Components: replication (java) >Reporter: Varun Thacker >Assignee: Mark Miller >Priority: Critical > Attachments: SOLR-6920.patch, SOLR-6920.patch, SOLR-6920.patch, > SOLR-6920.patch > > > Currently we check if an index file on the master and slave is the same by > checking if it's name and file length match. > With LUCENE-2446 we now have a checksums for each index file in the segment. > We should leverage this to verify if two files are the same. > Places like SnapPuller.isIndexStale and SnapPuller.downloadIndexFiles should > check against the checksum also. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-6920) During replication use checksums to verify if files are the same
[ https://issues.apache.org/jira/browse/SOLR-6920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14309438#comment-14309438 ] Mark Miller commented on SOLR-6920: --- For 5x, we will want to put the version checks back in. > During replication use checksums to verify if files are the same > > > Key: SOLR-6920 > URL: https://issues.apache.org/jira/browse/SOLR-6920 > Project: Solr > Issue Type: Bug > Components: replication (java) >Reporter: Varun Thacker >Assignee: Mark Miller >Priority: Critical > Attachments: SOLR-6920.patch, SOLR-6920.patch, SOLR-6920.patch, > SOLR-6920.patch > > > Currently we check if an index file on the master and slave is the same by > checking if it's name and file length match. > With LUCENE-2446 we now have a checksums for each index file in the segment. > We should leverage this to verify if two files are the same. > Places like SnapPuller.isIndexStale and SnapPuller.downloadIndexFiles should > check against the checksum also. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-6920) During replication use checksums to verify if files are the same
[ https://issues.apache.org/jira/browse/SOLR-6920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14309393#comment-14309393 ] Varun Thacker commented on SOLR-6920: - bq. I guess it would be fairly expensive for large files. For critical small files (the ones that always seem to come as the problem), perhaps we can do the checksum ourselves, or more likely, just always down .si and segments_n files regardless. Nice idea! So .si/segments_n/.liv should be downloaded again regardless. > During replication use checksums to verify if files are the same > > > Key: SOLR-6920 > URL: https://issues.apache.org/jira/browse/SOLR-6920 > Project: Solr > Issue Type: Bug > Components: replication (java) >Reporter: Varun Thacker >Priority: Critical > Attachments: SOLR-6920.patch, SOLR-6920.patch, SOLR-6920.patch > > > Currently we check if an index file on the master and slave is the same by > checking if it's name and file length match. > With LUCENE-2446 we now have a checksums for each index file in the segment. > We should leverage this to verify if two files are the same. > Places like SnapPuller.isIndexStale and SnapPuller.downloadIndexFiles should > check against the checksum also. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-6920) During replication use checksums to verify if files are the same
[ https://issues.apache.org/jira/browse/SOLR-6920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14309370#comment-14309370 ] Mark Miller commented on SOLR-6920: --- bq. can't we just calculate the checksum ourselves if we don't find it? I guess it would be fairly expensive for large files. For critical small files (the ones that always seem to come as the problem), perhaps we can do the checksum ourselves, or more likely, just always down .si and segments_n files regardless. > During replication use checksums to verify if files are the same > > > Key: SOLR-6920 > URL: https://issues.apache.org/jira/browse/SOLR-6920 > Project: Solr > Issue Type: Bug > Components: replication (java) >Reporter: Varun Thacker >Priority: Critical > Attachments: SOLR-6920.patch, SOLR-6920.patch, SOLR-6920.patch > > > Currently we check if an index file on the master and slave is the same by > checking if it's name and file length match. > With LUCENE-2446 we now have a checksums for each index file in the segment. > We should leverage this to verify if two files are the same. > Places like SnapPuller.isIndexStale and SnapPuller.downloadIndexFiles should > check against the checksum also. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-6920) During replication use checksums to verify if files are the same
[ https://issues.apache.org/jira/browse/SOLR-6920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14309353#comment-14309353 ] Mark Miller commented on SOLR-6920: --- bq. Any idea on how to tackle it It seems like we can't be sure every file will have a checksum (at least in tests) and we have to recover properly if we find that it doesn't? > During replication use checksums to verify if files are the same > > > Key: SOLR-6920 > URL: https://issues.apache.org/jira/browse/SOLR-6920 > Project: Solr > Issue Type: Bug > Components: replication (java) >Reporter: Varun Thacker >Priority: Critical > Attachments: SOLR-6920.patch, SOLR-6920.patch, SOLR-6920.patch > > > Currently we check if an index file on the master and slave is the same by > checking if it's name and file length match. > With LUCENE-2446 we now have a checksums for each index file in the segment. > We should leverage this to verify if two files are the same. > Places like SnapPuller.isIndexStale and SnapPuller.downloadIndexFiles should > check against the checksum also. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-6920) During replication use checksums to verify if files are the same
[ https://issues.apache.org/jira/browse/SOLR-6920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14309351#comment-14309351 ] Mark Miller commented on SOLR-6920: --- [~varunthacker], can't we just calculate the checksum ourselves if we don't find it? So that we treat old segments right as well? This issue is critical for avoiding corrupted indexes on replication. > During replication use checksums to verify if files are the same > > > Key: SOLR-6920 > URL: https://issues.apache.org/jira/browse/SOLR-6920 > Project: Solr > Issue Type: Bug > Components: replication (java) >Reporter: Varun Thacker > Attachments: SOLR-6920.patch, SOLR-6920.patch, SOLR-6920.patch > > > Currently we check if an index file on the master and slave is the same by > checking if it's name and file length match. > With LUCENE-2446 we now have a checksums for each index file in the segment. > We should leverage this to verify if two files are the same. > Places like SnapPuller.isIndexStale and SnapPuller.downloadIndexFiles should > check against the checksum also. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-6920) During replication use checksums to verify if files are the same
[ https://issues.apache.org/jira/browse/SOLR-6920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14277707#comment-14277707 ] Mark Miller commented on SOLR-6920: --- But isnt this just checking what solrconfig is set to? And that doesn't have to match what the index was built with? > During replication use checksums to verify if files are the same > > > Key: SOLR-6920 > URL: https://issues.apache.org/jira/browse/SOLR-6920 > Project: Solr > Issue Type: Bug > Components: replication (java) >Reporter: Varun Thacker > Attachments: SOLR-6920.patch, SOLR-6920.patch > > > Currently we check if an index file on the master and slave is the same by > checking if it's name and file length match. > With LUCENE-2446 we now have a checksums for each index file in the segment. > We should leverage this to verify if two files are the same. > Places like SnapPuller.isIndexStale and SnapPuller.downloadIndexFiles should > check against the checksum also. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-6920) During replication use checksums to verify if files are the same
[
https://issues.apache.org/jira/browse/SOLR-6920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14276158#comment-14276158
]
Varun Thacker commented on SOLR-6920:
-
Hi [~markrmil...@gmail.com],
Thanks for the review!
bq. We should not need the back compat < 5.0 code AFAICT though.
>From what I understand an index built pre LUCENE-2446 would not have the
>checksum , so {{CodecUtil.retrieveChecksum(in)}} would throw an error as the
>validation would fail. I guess the check should be 4_8 and not 5_0 though.
> During replication use checksums to verify if files are the same
>
>
> Key: SOLR-6920
> URL: https://issues.apache.org/jira/browse/SOLR-6920
> Project: Solr
> Issue Type: Bug
> Components: replication (java)
>Reporter: Varun Thacker
> Attachments: SOLR-6920.patch, SOLR-6920.patch
>
>
> Currently we check if an index file on the master and slave is the same by
> checking if it's name and file length match.
> With LUCENE-2446 we now have a checksums for each index file in the segment.
> We should leverage this to verify if two files are the same.
> Places like SnapPuller.isIndexStale and SnapPuller.downloadIndexFiles should
> check against the checksum also.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-6920) During replication use checksums to verify if files are the same
[ https://issues.apache.org/jira/browse/SOLR-6920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14275909#comment-14275909 ] Mark Miller commented on SOLR-6920: --- This looks good. We should not need the back compat < 5.0 code AFAICT though. > During replication use checksums to verify if files are the same > > > Key: SOLR-6920 > URL: https://issues.apache.org/jira/browse/SOLR-6920 > Project: Solr > Issue Type: Bug > Components: replication (java) >Reporter: Varun Thacker > Attachments: SOLR-6920.patch, SOLR-6920.patch > > > Currently we check if an index file on the master and slave is the same by > checking if it's name and file length match. > With LUCENE-2446 we now have a checksums for each index file in the segment. > We should leverage this to verify if two files are the same. > Places like SnapPuller.isIndexStale and SnapPuller.downloadIndexFiles should > check against the checksum also. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-6920) During replication use checksums to verify if files are the same
[
https://issues.apache.org/jira/browse/SOLR-6920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14267406#comment-14267406
]
Varun Thacker commented on SOLR-6920:
-
When using a try-finally block code in SOLR-6440 the threads don't hang
anymore.
The test still fails because of errors like this -
{code}
391305 T11 C196 P59014 oasu.SolrIndexWriter.close ERROR Error closing
IndexWriter java.lang.AssertionError: file _2_2.liv does not exist;
files=[segments_2, _1.cfs, _3.si, _2.cfe, _1.si, _1.cfe, _3.cfe, _0.cfs,
_0.cfe, _2.si, _0.si, _3.cfs, _2.cfs]
at org.apache.lucene.index.IndexWriter.filesExist(IndexWriter.java:4232)
at
org.apache.lucene.index.IndexWriter.startCommit(IndexWriter.java:4303)
at
org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:2785)
at
org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:2888)
at org.apache.lucene.index.IndexWriter.shutdown(IndexWriter.java:965)
at org.apache.lucene.index.IndexWriter.close(IndexWriter.java:1008)
at
org.apache.solr.update.SolrIndexWriter.close(SolrIndexWriter.java:129)
at
org.apache.solr.update.DirectUpdateHandler2.closeWriter(DirectUpdateHandler2.java:804)
at
org.apache.solr.update.DefaultSolrCoreState.closeIndexWriter(DefaultSolrCoreState.java:68)
at
org.apache.solr.update.DefaultSolrCoreState.close(DefaultSolrCoreState.java:359)
at
org.apache.solr.update.SolrCoreState.decrefSolrCoreState(SolrCoreState.java:72)
at org.apache.solr.core.SolrCore.close(SolrCore.java:1110)
at org.apache.solr.core.SolrCores.close(SolrCores.java:117)
at org.apache.solr.core.CoreContainer.shutdown(CoreContainer.java:366)
at
org.apache.solr.servlet.SolrDispatchFilter.destroy(SolrDispatchFilter.java:194)
at
org.apache.solr.cloud.ChaosMonkey.stopJettySolrRunner(ChaosMonkey.java:197)
at org.apache.solr.cloud.ChaosMonkey.stop(ChaosMonkey.java:550)
at
org.apache.solr.cloud.AbstractFullDistribZkTestBase.destroyServers(AbstractFullDistribZkTestBase.java:1568)
at
org.apache.solr.BaseDistributedSearchTestCase.tearDown(BaseDistributedSearchTestCase.java:283)
at
org.apache.solr.cloud.AbstractDistribZkTestBase.tearDown(AbstractDistribZkTestBase.java:231)
at
org.apache.solr.cloud.AbstractFullDistribZkTestBase.tearDown(AbstractFullDistribZkTestBase.java:1546)
at
org.apache.solr.cloud.BasicDistributedZkTest.tearDown(BasicDistributedZkTest.java:1166)
at org.apache.solr.cloud.ShardSplitTest.tearDown(ShardSplitTest.java:91)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
at
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:885)
at
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at
com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
at
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
at
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
at
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
at
com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
at
com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
at
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at
com.carrotsea
